AI in Production: Panellists Highlight Silent Failures, Harness Engineering, and Agent Governance

InfoQ hosted a webinar titled "AI in Production: What Breaks, What Works, and Who Approves It?" in which a panel of senior engineering practitioners discussed the operational realities of deploying agentic systems. The session was moderated by Hien Luu and featured Zichuan Xiong and Prem Chandrasekaran of ThoughtWorks, privacy specialist Katharine Jarmul, and financial AI practitioner Lan Chu. Olimpiu Pop reported on the discussion.
Luu framed the conversation around three drivers. First, AI tends to fail differently from conventional software: instead of crashing, it produces convincing but inaccurate output. Second, coding agents are generating production code faster than humans can review it. Third, sensitive data has new places to escape through prompts, context windows, and retrieved documents.
Xiong, who leads AI Ops for Managed Services at ThoughtWorks, described semantic drift as a widespread problem in LLM pipelines. Text-to-SQL translation agents, he said, often flatten domain-specific concepts, for example treating "member" the same way across sales and customer support contexts. The resulting errors are subtle, hide in downstream business analytics, and pass standard tracing. Chu, working in financial compliance, described a failure rooted in embedding drift. An agent designed to block prohibited investment advice used vector similarity thresholds. An unannounced background update to the embedding model shifted the vector distance distribution, producing sharp increases in both false positives and false negatives in live traffic, even though every infrastructure health check stayed green.
The panel also covered retrieval and repository risks. Chandrasekaran noted that RAG pipelines can retrieve sensitive documents unintentionally, and Jarmul observed that models memorize training data and can be used to de-anonymize information or expose API keys. Chandrasekaran cited the GitSpawn vulnerability, in which malicious repository configuration can cause unauthorized code execution during routine commands.
On review, the panel agreed that line-by-line manual code review cannot keep pace with agent-generated code volume. Chandrasekaran argued teams should automate tactical style and syntax checks inside CI/CD pipelines and reserve human attention for architecture, business intent, and non-functional requirements. Xiong added that evaluation should shift left, to the harness level, before a pull request is created. Chandrasekaran pointed to mutation testing as a revived practice: deliberately introducing faults shows whether a test suite exercises underlying logic or just satisfies coverage numbers.
Chu treated evals as permanent, living infrastructure rather than a pre-release checklist. In her account, effective evaluation needs continuous online tracing at the span level, log filtering, and the use of LLMs to categorize failure patterns, with production edge cases fed back into offline benchmark suites. This is a useful framing for teams that currently treat evaluation as a gate before launch: the panel's position is that the same signals must keep flowing after deployment, because model and data distributions do not stay fixed. Embedding drift is a case in point, since a silent upstream change moves the ground beneath a fixed threshold.
On privacy and security, Jarmul warned against depending on prompt engineering as a defence. She recommended systematic red teaming, continuous threat modeling, strict functional scopes for autonomous agents, and observability scaled to handle larger event volumes. Chandrasekaran compared each AI agent to a new engineering hire: give it a distinct non-human identity, apply tiered access controls that separate read-only operations from reversible writes, version third-party Model Context Protocol dependencies, and keep API secrets outside the agent's runtime environment in sandboxes. The shared logic here is blast-radius reduction. An agent with its own identity and narrowly scoped credentials resembles a service account or a limited-privilege user, which makes revocation and auditing tractable rather than speculative. Reversible writes matter because autonomy plus irreversible actions is where small mistakes become incidents. Deploying agents onto the Model Context Protocol, an emerging standard for connecting models to external tools and data, adds third-party dependencies that teams should pin and track like any other supply-chain component.
A recurring theme is that serious agent failures are quiet. The green dashboards in Chu's example were not wrong about the infrastructure; they simply measured something that could not reveal a distributional shift in a model dependency. Similarly, semantic drift in text-to-SQL passes tracing because the query executes successfully. This suggests that evaluation harnesses need to test meaning and business intent, not just execution health, and that model dependencies need the same versioning discipline teams already apply to libraries.
The panel converged on a division of responsibility: engineering, security, and privacy teams validate safety guardrails, while the decision to deploy remains a strategic business call. In practice, that split asks technical teams to produce evidence and constraints, and business owners to accept the residual risk. It also implies that approval is not a one-time event, since guardrails can decay as models and data shift underneath them.
Why it matters: teams building agents are being told that the failure modes to watch are silent and statistical, appearing in production after all health checks pass. Developers should expect to invest in continuous tracing, mutation testing, and automated checks so that human review can focus on architecture and intent, and should give agents scoped non-human identities with versioned tool dependencies and isolated secrets. Product and engineering leaders should treat evaluation infrastructure and agent governance as ongoing costs rather than launch-time tasks, and should note, as inference from the panel's framing, that approving deployment means owning risks that may not surface in conventional monitoring.
Based on reporting from the original publisher. Visit the source for full context and later updates.
Publisher excerpt
A recent InfoQ webinar examined operational challenges in deploying agentic systems. Key points included issues like semantic and embedding drift, which can lead to significant errors. Experts discussed the need for automated code reviews, continuous evaluation of models, and enhanced privacy measures. The session emphasised a strategic approach to deployment and security management. By Olimpiu Pop