Pushing LangSmith to new limits with Replit Agent's complex workflows

Replit integrated LangSmith into its Replit Agent, which was built atop LangGraph, to gain visibility into agent interactions and debug issues, according to LangChain. Replit's platform serves over 30 million developers, and Replit Agent went viral after release.

Replit Agent's workflow is described as highly custom and agentic, with substantial control and parallel execution. Its traces were very large, involving hundreds of steps, which challenged data ingestion and display. LangChain said it improved ingestion to process and store large volumes of trace data and improved LangSmith's frontend rendering to display long-running agent traces seamlessly.

LangSmith previously supported search between traces. As Replit Agent's traces grew longer, Replit needed to search within traces for specific events, often issues reported by alpha testers. LangSmith added a search pattern for searching within traces, allowing users to filter on criteria such as keywords in the inputs or outputs of a run. LangChain said this reduced Replit's time to debug agent steps within a trace.

For human-in-the-loop workflows, Replit Agent was designed for AI agents to collaborate with human developers who can edit and correct agent trajectories. With separate agents handling roles such as managing, editing, and verifying generated code, interactions with users continued over long, multi-turn conversations and each session generated disjoint traces. LangSmith's thread view collated related traces from one conversation, providing a logical view of all agent-user interactions across a multi-turn conversation. LangChain said this helped Replit find bottlenecks where users got stuck and pinpoint areas where human intervention could be beneficial.