AWS traces Property Finder’s evaluation bias to incumbent labels

In a case study published October 27, 2026, AWS reported that 96.8% of Property Finder’s earlier evaluation labels came from predictions by the incumbent vision model. That made the model being evaluated a major part of the answer key used to judge alternatives.

How the incumbent became the benchmark

When labels come from a model’s own predictions, a competing model is scored against the incumbent’s behavior rather than an independently established reference. The remaining 3.2% of Property Finder’s earlier labels were human-made, but AWS also described errors in that portion: watermark text mistakes were accepted as correct, and an “Office” category appeared even though it was absent from the prompt. A single room label also could not represent images showing more than one room type.

Property Finder’s workload assesses 11 attributes of property photos, including watermark detection, room classification, furnishing status and photo quality. AWS described it as processing tens of thousands of images each day across the Middle East and North Africa, with listings averaging 15 images.

How Property Finder rebuilt the labels

A domain expert annotated 356 property images from the United Arab Emirates without referring to the previous model’s outputs. AWS Generative AI Innovation Center staff reviewed label quality and clarified ambiguous cases. The first 256 images formed the development set; 100 images annotated later brought the final evaluation set to 356. The case study also described a separate 230-image check using Egyptian images.

The team changed room and photo classification to allow multiple labels when an image contained overlapping content. It used Micro F1 as its primary metric, alongside Macro F1, precision and recall for each label, intersection over union for each sample, and exact-match accuracy for nine single-label attributes. These measures let the team examine performance beyond one overall score.

What the Micro F1 scores show

On the 356-image UAE evaluation set, AWS reported Micro F1 of 0.80 for Llama 4 Maverick 17B Instruct with prompt v27_PF, compared with 0.56 for proprietary model A with v22.

The comparison was shaped by how the models were tuned. The final set included the same 256 images used to develop prompts, and Llama received several rounds of tuning plus additional Property Finder rules. Proprietary model A received one tuning round. AWS called the scores optimistic, so they describe this UAE property-image evaluation rather than a general ranking of vision models.

Prompt changes and a failed two-call approach

The prompt work addressed specific edge cases, including rooms visible through doorways, decorative Arabic text that could be mistaken for a watermark, renders, borders or frames, and corridors. The v27_PF prompt included Property Finder rules for renders, borders or frames, and corridors. AWS also reported using MIPROv2 for photo classification in v27.

A separate “2-bucket” design split watermark attributes from other attributes across two model calls. AWS said it produced no meaningful accuracy gain and reduced boxed-image detection: without watermark attributes in the same prompt, watermark overlays could be mistaken for borders.

Cost and latency in the case study

AWS estimated a cost of $2.35 per 1,000 images for Llama 4 Maverick 17B Instruct, with an estimated 86% reduction against Property Finder’s previous setup. The estimate was for December 2025 in US East (Ohio) and assumed 9,000 input tokens and 200 output tokens per image. AWS described $2.35 as an upper estimate because the production prompt uses about 7,500 to 8,000 input tokens in a single call. The cost comparison used the previous setup’s v21 estimate; the model-accuracy comparison used proprietary model A with v22.

Property Finder reported that its own production monitoring found inference latency 40% to 60% lower than with its previous setup.