Fix evaluation bias to unlock better models on Amazon Bedrock

Property Finder, a digital real estate application operating across the Middle East and North Africa, discovered that the evaluation dataset it used to compare vision models was largely written by the model it was trying to replace. Of its labeled samples, 96.8 percent came directly from the predictions of its incumbent vision language model rather than from independent human verification. The company surfaced the problem while migrating that workload to a cheaper option on Amazon Bedrock, working with the AWS Generative AI Innovation Center. The resulting account, published on the AWS Machine Learning Blog and co-authored with Khaled Thabit, Meriem Boughrara and Mohamed Desouky of Property Finder, reports a final choice of Meta's Llama 4 Maverick 17B Instruct, with higher accuracy and an estimated 86 percent cost reduction.
The production workload is substantial. Property Finder assesses tens of thousands of property images a day, checking 11 quality attributes per image covering watermark detection and text extraction, room classification, furnishing status, photo quality and construction indicators. Listings average 15 images each, and every image needs a quality verdict before going live.
The bias is easy to see once stated. When labels are model outputs, a challenger model is scored on how closely it imitates the incumbent's behavior, not on whether its answers are right. Property Finder's labels also contained errors that the bias had masked: watermark text extraction carried character-level mistakes marked as correct, including invented brand names and dropped characters, and photo classification forced a single room label onto images showing several rooms. In the small human-labeled remainder, annotators had introduced categories such as Office that were never in the original specification. The team's conclusion is blunt: a model comparison loses validity when the answer key was written by the model being replaced.
To rebuild ground truth, the team had one strategy and analytics expert with real estate listing quality knowledge annotate fresh, independently. Annotators had no sight of previous model outputs. The team hand-annotated 356 images from the UAE, drawn from unlabeled production listings across representative industry segments, plus a separate Egypt set of 230 images used as a cross-regional check. The first 256 UAE images formed a development set for tuning; the full 356, including 100 images annotated after the tuning runs, formed the evaluation set. The post notes that 356 images are roughly enough to detect moderate differences between models, while rare classes within individual attributes have too few examples to catch smaller gaps.
Two methodology changes mattered alongside the new labels. The task moved from single-label to multi-label classification, reflecting that a hallway photo with a bedroom visible through a doorway is genuinely both. And metrics were chosen deliberately: Micro F1 as the primary score because it weights each prediction equally and matches the business goal of overall label correctness, Macro F1 alongside it because Micro F1 is dominated by frequent classes and can hide weak performance on rare attributes, plus per-label precision and recall, per-sample intersection over union over the label sets, and exact-match accuracy for the nine single-label attributes such as rotation or watermark detection.
With a trustworthy score in place, prompt development ran from v0 to v27 in a loop the authors compare to reinforcement learning: each prompt version acts as an agent, Micro F1 is the reward, and error analysis plays the role of gradient. The idea resembles GEPA, a genetic-Pareto style reflective prompt evolution approach, where prompt components are mutated, scored and selected. Most versions were refined by hand from error analysis with AI assistants reviewing failed images; MIPROv2, an automated prompt optimizer in the DSPy library, handled the photo classification task in v27.
Several specific fixes came out of looking at failures image by image. Early single-label prompts misassigned rooms glimpsed through doorways, so the team instructed the model to classify by the room occupying most of the frame. Regional rules for decorative Arabic text overlays removed a class of false watermark positives. Structured JSON output was added. The v21 prompt, which moved photo and room classification to multi-label output, became the shared starting point for every candidate model; with it, Llama 4 Maverick 17B Instruct scored 0.75 Micro F1 on the development set. Subsequent versions tuned separately for that model: v24 targeted room size overestimation, multi-label handling and output format; v25, automatically optimized, moved critical tasks earlier; v27 restructured photo classification via MIPROv2. The shipped prompt, v27_PF, adds Property Finder's rules for renders, bordered or framed images and corridors, and a normalization step cleans up residual format differences after each call.
Numbers from the 356-image set: Llama 4 Maverick 17B Instruct with v27_PF reached 0.80 Micro F1, against 0.56 for the proprietary baseline using its own tuned v22 prompt. On the development set, each model on its own tuned prompt moved the score from 0.55 to 0.81, a 47 percent relative gain; prompt tuning alone on Llama 4 Maverick lifted it from 0.75 to 0.81. The authors flag that the evaluation set contains the development images and both prompts were tuned on them, so these figures are optimistic. A caveat worth noting: the baseline received one tuning round while the Llama model got several plus edge-case rules, so effort was not equal. Exact-match results for the open-weight model were 100 percent on rotation detection, 97 percent on people and boxed-image detection, 94 percent on watermark detection, 93 percent on render detection, 91 percent on furnished status, 81 percent on room under construction, 65 percent on room size and 55 percent on watermark text extraction. It met or beat the baseline on all 11 attributes and tied on photo classification.
On cost, the estimate is $2.35 per 1,000 images, 86 percent below Property Finder's previous setup, based on December 2025 US East (Ohio) pricing and an assumption of 9,000 input and 200 output tokens per image inherited from an earlier two-call analysis; the single production call uses roughly 7,500–8,000 input tokens, so the figure is an upper bound. Property Finder reports a 40–60 percent latency reduction in its own monitoring. Failed experiments are included: splitting watermark attributes into a separate API call produced no meaningful accuracy gain and sharply hurt boxed-image detection, because without watermark attributes in the same prompt the model confused overlays with borders, and generic prompts underperformed on every model given the domain specificity of real estate imagery.
Why it matters: any team that regressed models against labels generated by the incumbent has been measuring agreement, not quality, and this case shows the bias can be near-total and invisible. For developers and product teams, the practical sequence is to audit where evaluation labels came from, commission a few hundred domain-expert annotations with a held-out split never used for tuning, allow multi-label outputs where attributes overlap, and iterate prompts with error analysis. Inference: the 86 percent saving and latency gain suggest open-weight vision models on a managed platform are now viable replacements for a proprietary production model in high-volume classification, provided teams also budget review capacity, since the pipeline produces no calibrated confidence scores and the authors suggest model disagreement on a sample as a triage signal for human inspection.
Based on reporting from the original publisher. Visit the source for full context and later updates.
Publisher excerpt
Property Finder found that 96.8% of their vision-AI evaluation labels were inherited from the incumbent model, hiding its true accuracy. See how they built unbiased ground truth, applied evolutionary prompt optimization, and selected a more accurate, cost-effective model on Amazon Bedrock, cutting cost by 86% across 11 image-quality attributes.