How Amazon Marketplace evaluates content quality with Amazon SageMaker AI

Amazon's Marketplace team published an account of a production system that grades the quality of A+ Content — the rich media modules on Amazon product detail pages — using a Vision-Language Model (VLM) paired with optical character recognition (OCR) and a small gradient-boosted classifier, all running on Amazon SageMaker AI. The company says selling partners who use high-quality A+ Content see sales gains of 8–20 percent versus those without it, which helps explain why inconsistent quality across a very wide range of brands is worth automating. The pipeline spans rendering, annotation, scoring, explainability and production monitoring, with each stage mapped to a managed SageMaker capability.
The core design decision is to render an entire A+ page as one snapshot and let the VLM judge it as a shopper would, rather than scoring each image or text block separately. Amazon's own illustration: on a blender page, the hero image can be sharp, the bullet list grammatical and the lifestyle photo well lit, yet the bullets might describe a 64-oz pitcher while the photo shows a single-serve model, or the lifestyle image may sit above the hero shot. Those are cross-asset mismatches that per-asset checks miss. Conventional computer vision can already flag a blurry image or a grammar error; the argument here is that a VLM adds little when grading assets one at a time, but does add something when it sees the assembled page.
The VLM is prompted with a fixed rubric covering four dimensions — Visual Storytelling, Feature Presentation, A+ Layout and Conversion Optimization — each with explicit criteria and point-deduction rules. For every dimension it returns a 0–10 score, a confidence value and a natural-language rationale. Rather than fine-tuning the foundation model, which Amazon says would need GPU training infrastructure and risks overfitting on only a few thousand labeled examples, the team keeps the VLM and prompt frozen and feeds its structured outputs as features into CatBoost, a gradient-boosted decision tree library that works well on small tabular datasets. Retraining when annotations grow or criteria shift is a short CPU-only job rather than a foundation-model run.
Labels came from one ecommerce marketing expert and four professional annotators who independently rated 5,478 product detail pages across 10 categories — including Furniture, Shoes, Grocery, Beauty and Apparel — as high or low quality. Majority vote produced a consensus label. The split is nearly even at 49.6 percent high quality, so a model cannot win by always predicting the majority class. Agreement against the consensus label, measured with Cohen's kappa, ranged from 0.845 to 0.915 per annotator; kappa above 0.8 is generally treated as strong agreement, and the metric corrects for agreement expected by chance.
On a 20 percent holdout of roughly 1,100 pages, a baseline that simply averages the four VLM scores and applies a fixed threshold reached 0.80 accuracy and 0.84 F1 on the high-quality class. CatBoost trained on the same scores reached 0.89 accuracy and 0.90 F1, because it learns how much each dimension matters and how dimensions interact rather than weighting them equally.
The OCR leg addresses a known VLM weakness: fine-grained readability. Amazon notes that VLMs can misjudge how crowded text is or how much space separates lines, and that their language-model foundation can favor long, detailed descriptions even when shoppers prefer scannable copy. An OCR model hosted on SageMaker AI Inference extracts each text block and its position, producing four layout-based signals: Text Density (share of page area covered by text), Character Density (characters per unit area), Spacing (consistency and sufficiency of whitespace between words, lines and paragraphs), and Overall Readability, a combined score. Because these depend on layout rather than wording, two pages with identical text can score differently.
Feeding all eight features — four VLM, four OCR — into CatBoost produced the best results: 0.94 accuracy and 0.95 F1, versus 0.82/0.83 for OCR-only and 0.89/0.90 for VLM-only. That is a 14-point accuracy and 11-point F1 gain over the original baseline. Amazon also swapped the VLM backbone on a single-category dataset of 499 content projects: with a lighter VLM the baseline hit 0.74 accuracy and 0.72 F1 while the fusion model hit 0.90 on both; with a more capable VLM the figures were 0.79/0.84 and 0.94/0.95. Gains of 0.16 and 0.15 accuracy respectively suggest the approach does not depend on a particular foundation model — a useful property for a system meant to stay portable as models change.
Explanation runs through SHAP (SHapley Additive exPlanations), which assigns each feature a value indicating how far it pushed a prediction toward high or low quality. Because VLM and OCR scores enter the same classifier, SHAP places them on a common scale — something raw scores cannot do, since a Visual Storytelling score of 6 and a high Character Density are not directly comparable. Across the holdout set the most influential features were OCR Character Density (mean absolute SHAP value 1.1), OCR Overall Readability (1.0) and VLM Visual Storytelling (1.0), followed by A+ Layout (0.8), Feature Presentation (0.6) and Conversion Optimization (0.4). Both signal sources rank near the top, which supports the fusion claim.
For an individual low-quality page, the system ranks the eight features by SHAP value and passes the most negative ones, the product category, ecommerce best practices and the VLM rationales to an LLM, which turns them into prioritized, specific fixes. If A+ Layout scores -1.2 and Visual Storytelling -0.3, layout restructuring and rebalancing text against images is recommended ahead of imagery work. Amazon says it validated this loop: after selling partners revised their content, average quality rose 1.4 points on a 10-point scale on a statistically significant sample. It also reports a worldwide A/B experiment that adjusted visibility of low-quality A+ Content based on the model's decisions produced a statistically meaningful improvement in Ordered Product Sales.
On SageMaker, the stages map to managed services: Processing Jobs for containerized snapshot rendering at scale; Inference for hosting the VLM, with Real-Time Endpoints for low-latency single listings, Asynchronous Endpoints for large payloads such as high-resolution snapshots, and Batch Transform for catalog-wide offline scoring, plus automatic scaling and GPU instance selection; and Training Jobs optionally for LoRA adapters or distillation into a smaller, faster variant. The architecture keeps the VLM frozen and retrains only the classifier to hold down development cost.
This suggests a pattern worth borrowing for any team that needs to judge composite content: use a foundation model for holistic, semantic judgment, add cheap deterministic signals for the things it is weak at, and put a small interpretable model on top so you can explain and recalibrate without GPU spend. A likely trade-off is that the rubric and consensus labels encode one marketplace's quality standards, and the reported gains come from Amazon's own catalog and annotators. Teams adopting the pattern would need to build comparable labeled sets before trusting the classifier's weights.
Why it matters: Sellers and brands get measurable, prioritized feedback on page content instead of a bare score, and Amazon says revisions driven by it lifted quality by 1.4 points on average. For developers, the design shows a cost-conscious way to combine frozen VLMs, OCR heuristics and a retrainable classifier — and to explain the result — for any content that is more than the sum of its assets. The inference, clearly labeled as such, is that similar fusion pipelines are likely to spread to listings, ads and other visual merchandising where consistency across modules matters more than any single asset.
Based on reporting from the original publisher. Visit the source for full context and later updates.
Publisher excerpt
Learn how Amazon Marketplace evaluates A+ Content quality at scale on Amazon SageMaker AI by rendering each product page as a single snapshot, scoring it with a vision-language model fused with OCR readability signals in a CatBoost classifier, and using SHAP to turn each decision into prioritized, actionable recommendations for selling partners.