New in LangSmith: Engine v2, Managed Deep Agents, Fine-Tuning, and more

LangChain announced a set of LangSmith updates at its Interrupt NYC event. The releases span what the company describes as runtime and observability plus evaluations for the agent development lifecycle.

LangSmith Engine v2 adds red teaming, expanded issue detection, and automated fix validation. The company says Engine, launched in May, has helped engineers analyze more than 60 million traces to diagnose tens of thousands of issues. Red Teaming uses production traces and repositories to generate hypotheses about issues that may not have appeared in production, test those hypotheses, and surface confirmed failures for review. Engine can now also identify performance trends across error rate, latency, and cost, as well as inefficient agent work such as repetitive tool calls or unnecessarily long trajectories. For agents running on LangSmith Deployment, Engine can validate proposed fixes before human review and deployment, running offending inputs against the agent, testing candidate fixes against a broader eval set, and presenting a satisfactory fix that users can turn into a pull request. LangChain said the next release of self-hosted LangSmith will support BYOK for Engine.

Managed Deep Agents v0.8 adds identity-scoped authentication and memory, new channels, and built-in web search. The release introduces user-level memory so agents can store context scoped to each authenticated user, kept separate from shared agent-level memory, with access policies definable at each level. It also supports agent and user-owned credentials for external connections such as GitHub and Notion. Slack support now includes file transfer, and new HTTP channel support lets teams connect agents to any service that can send a JSON webhook. Built-in web search, powered by Parallel, is offered as a prebuilt tool.

LangChain also launched LangSmith Trajectories, a conversational view of an agent session that aggregates messages from humans, AI, and tools across the main agent and subagents in chronological order. LangSmith online evaluators can score trajectories.

LangSmith Fine-Tuning enables supervised fine-tuning of open models on task examples, with a CLI called smithtune that builds and prepares datasets from trajectories, trains models with Baseten or Fireworks, evaluates results with LangSmith, and serves tuned models and connects them to applications. LangChain said it benchmarked SmithTune internally.

Custom Apps lets teams build, publish, and run custom interfaces on top of LangSmith data for workflows such as annotation queues and experiments.