How Podium optimized agent behavior and reduced engineering intervention by 90% with LangSmith

Podium, a communication platform for small businesses, used LangSmith to optimize its AI Employee agent and reduced the need for engineering intervention by 90%, according to a LangChain case study. The company also improved its agent's F1 response quality from 91.7% to 98.6%, exceeding its 98% quality threshold.

Podium launched its AI Employee agentic application in January to engage local business customers, schedule appointments, and close sales. The company initially used the LangChain framework for single-turn interactions but turned to LangSmith for LLM testing and observability as its agentic use cases grew more complex.

A specific challenge was the agent failing to recognize when a conversation had naturally ended, resulting in repeated goodbyes. Podium addressed this by creating a dataset in LangSmith with various conversation scenarios. The team upgraded to a larger model and used model distillation to curate outputs into a smaller model. Because inputs and outputs were automatically captured in LangSmith traces, the team enriched traces with metadata on customer profiles and business types to create a balanced dataset and avoid overfitting. Pairwise evaluations compared the fine-tuned model against the original larger model.

Podium gave its Technical Product Specialists access to LangSmith traces, allowing them to identify whether customer-reported issues stemmed from application bugs, incomplete context, misaligned instructions, or LLM issues. This guided appropriate interventions: bugs and LLM issues required engineering intervention, while incomplete context and misaligned instructions could be remediated by the TPS team. For example, a car dealership using the AI Employee received an incorrect response about oil changes; the TPS team could use LangSmith's playground to edit the system output and determine if an Admin interface setting change would resolve it.

Previously, troubleshooting agent behavior often required engineering intervention, involving engineers reviewing model inputs and outputs and rewriting code. With LangSmith, Podium reduced that need by 90%. The company reported increased efficiency of its support team and improved customer satisfaction scores for both support interactions and its AI-powered services.

Podium is also integrating LangGraph into its workflow to reduce complexity in agent orchestration while increasing controllability over agent conversations.