AivexaNewsSearch
AI news for builders and product teamsChecked every hour

Search agent traces by behavior with AI Search in LangSmith

Collected Oct 13, 2026

LangChain has added AI Search to LangSmith, a new capability that lets developers query agent traces by describing the behavior they want to find in plain language rather than by tagging or full-text search. Announced as a public beta, the feature is available on LangSmith Cloud in the GCP US, EU, and APAC regions at no additional cost. Organization admins can enable it in settings, after which it appears on the tracing page, and LangChain points users to the AI Search docs for details.

The motivation is that certain questions about agent behavior do not map onto any single trace attribute. Examples LangChain gives include whether personally identifiable information leaked, whether a user became frustrated, or whether an agent claimed to finish a task it had not completed. Those questions require reading and understanding trace contents, which previously meant either configuring an evaluator in advance or reviewing traces by hand. AI Search instead takes a behavioral description and returns the traces in which that behavior occurred, which LangChain frames as a way to find needle-in-haystack examples and ad hoc troublesome patterns.

What can be searched is anything appearing in trace content. LangChain lists agents that claim completion without doing the work, users who seem frustrated or push back, model requests that leaked PII, and users attempting to cancel a subscription. Product teams are also pointed at queries such as requests for unsupported integrations, mentions of competitors, or confusion about onboarding, so they can see shortcomings in users' own words. AI Search composes with existing filters, so a search can be restricted to traces that errored or that carry a specific tag.

A central design decision is that AI Search does not use vector search as its primary mechanism. Vector search embeds a query and each trace into numeric vectors and returns the closest matches, which works for topic-level retrieval such as finding billing conversations. LangChain argues this compression is lossy: a trace where an agent actually did the work and one where it merely said it did can land at nearly the same embedding. Distinguishing them requires a model that reads the trace text alongside the question and reasons about what happened. Running such a model at search time was previously too slow and costly, and LangChain credits fast decision models, which answer one narrow question about an input quickly and cheaply, with making it viable. AI Search pairs that inference with SmithDB, LangChain's database purpose-built for trace data, which it previously introduced alongside a Custom Apps feature for tailored trace views.

The pipeline works on trajectories rather than raw traces. Traces arrive from many sources, including the OpenAI and Anthropic SDKs, LangChain, LangGraph, the Vercel AI SDK, and OpenTelemetry, and each records messages slightly differently. Traces also repeat content, because agents typically resend conversation history with every model call. As traces arrive, SmithDB distills each session into a trajectory: each message, tool call, and tool result listed once, in a common shape across all supported formats. Because the shape is shared, one search works across agents built on different SDKs, duplicate removal cuts context size, and the trajectory spans the whole session, so a search for something like a cancellation attempt can see what the user asked, what the agent did, and how the conversation ended.

Trajectories remain long even after deduplication. LangChain sampled 150 random conversations from Open SWE, its open source coding agent used internally, where each task runs in its own sandbox with shell access and can delegate to subagents. A couple of conversations exceeded a million tokens even in that small sample. To cope, each trajectory is given a fixed token budget filled in priority order: user messages and the agent's replies go in first, then intermediate AI messages covering reasoning and tool calls, tool results, and system messages, with a partial message trimmed to fit and the remainder dropped. LangChain reports that in the median conversation of its sample, roughly 2K of 72K tokens came from user messages and agent replies, while about 85% was tool output, mostly file reads and shell commands. AI Search fills an 8K token budget and sends the kept messages to the decision model in their original order.

For speed at scale, structured user filters narrow candidate traces first, then the decision model checks each remaining trajectory, many in parallel. While one batch is being checked, SmithDB fetches and prepares the next, and matches stream back as they are found rather than waiting for a complete scan.

LangChain is explicit that the approach has open edges. It says the hardest conversations to classify are those with subtle differences, giving the example of telling a user who gave up apart from one who got what they needed and left, which can hinge on a few turns in a long trajectory, and it wants to test whether a model tuned on traces separates them more reliably. The token budget also assumes the answer lives in what the user and agent said; an agent ignoring a tool error would only appear in tool results. LangChain says it is exploring question-dependent context selection and signals for model uncertainty so a second pass can run with more context, and it wants to move beyond a plain list of matches toward grouping similar results or summarizing what they share.

Why it matters: Teams running agents in production often have large volumes of trace data but limited ways to interrogate it beyond tags, metadata, and exact-match search. Semantic behavioral search changes what debugging and product analysis can look like, since questions about user sentiment or policy violations become queries rather than evaluator projects or manual review. The caveats are real, though: the 8K budget that prioritizes user and agent messages over tool results suggests behavior visible only in dense tool output may be missed, which is my inference from the stated design. Developers evaluating this in beta should test whether their target behaviors survive that prioritization before relying on it.

Read at LangChain

Based on reporting from the original publisher. Visit the source for full context and later updates.

Publisher excerpt

AI Search in LangSmith is a new way to query traces using natural language. Describe the behavior you're looking for, and LangSmith returns the traces where it happened.