NarrateAI: production-ready LLM quality assurance on Amazon Bedrock

AWS published a technical post describing five quality assurance techniques in NarrateAI, a conversational agentic AI assistant built on Amazon Bedrock AgentCore that serves over 4,000 AWS executive leaders. The post is the second in the NarrateAI series.

The techniques are adaptive pipeline orchestration, cross-account multi-model failover, real-time streaming evaluation, composite evaluation framework, and data accuracy verification. AWS says they work as a coordinated system to achieve approximately 99 percent numerical accuracy while streaming responses in real time. NarrateAI uses a two-layer architecture: an Automated Narrative Generation Layer for batch processing and a Conversational AI Interface Layer for real-time interaction.

Adaptive pipeline orchestration routes queries by aggregated section volume against an empirically calibrated threshold, concatenating sections into one chunk on a fast path and packing them into batches on a normal path. The post states that approximately 90 percent of queries take the fast path, with time-to-first-token within a few seconds and total latency typically under 25 seconds. Normal-path queries take approximately 50-75 seconds. With N=4 typical batches, blended cost is approximately 1.4 LLM invocations per query, a 72 percent reduction versus always using multi-pass processing.

Cross-account multi-model failover treats each model-account pair as an independent Amazon Bedrock quota space; a 3-model by 3-account configuration provides nine such spaces. The implementation is a custom Strands model provider. AWS reports a six-month production deployment across over 4,000 users, and Locust load testing supporting over 100 concurrent users streaming responses without failed requests.

Real-time streaming evaluation uses a producer-consumer pattern to validate each paragraph as it is produced. AWS measured 1,000 production queries (approximately 10,439 paragraphs) using Anthropic Claude Sonnet on Amazon Bedrock, reporting an 86.8 percent latency reduction versus sequential evaluation. The post attributes deterministic checks at roughly 79ms median and LLM-based numerical verification at roughly 2,025ms median.