How Factory used LangSmith to automate their feedback loop and improve iteration speed by 2x

Factory, which builds a secure AI platform for software development lifecycle automation, used self-hosted LangSmith to address observability requirements for its autonomous LLM systems while maintaining enterprise-level security and privacy, according to LangChain.

Factory integrated LangSmith to export traces to AWS CloudWatch logs, allowing engineers to track data flow through stages of its LLM pipeline and pinpoint their position in the agentic stage. LangSmith's first-party API provided custom tracing, which Factory needed because its custom LLM tooling made most LLM observability tools difficult to set up.

For debugging context-awareness issues, Factory linked feedback directly to each LLM call in LangSmith, which helped the team identify and resolve issues such as hallucinations without a proprietary logging system.

Factory also used LangSmith's Feedback API to append feedback to workflow stages, export it to datasets, and analyze patterns. Its feedback loop begins with a Droid posting a comment and collecting positive or negative feedback; LangSmith analyzes the data, then engineers use custom LangChain tooling to optimize the prompt, re-prompt the LLM, and improve accuracy and reduce errors. Factory had the LLM examine a prompt and state why it may have caused a bad example rather than a good one.

Compared with manual data collection and human-driven prompt iteration, Factory was able to 2x its iteration speed. Factory also reports its average customer experienced a roughly 20% reduction in open-to-merge time and a 3x reduction in code churn on code impacted by Droids in the first 90 days. Clients report an average reduction in cycle time of up to 20%, with over 550,000 hours of development time saved across organizations, according to the post.

Factory recently publicly launched its AI Droids and raised $15 million in Series A funding led by Sequoia Capital. Eno Reyes, CTO of Factory, said the collaboration with LangChain has been critical to deploying enterprise LLM-based systems and that the company is significantly more confident in its decision making and operational capabilities because of LangChain's observability and orchestration-layer tooling.