AivexaNewsSearch
AI news for builders and product teamsChecked every hour

How We Made Our Docs Test Themselves

Collected Oct 13, 2026

LangChain published a walkthrough of how it converted inline code samples in its documentation into standalone, testable files using its Deep Agents CLI. The company described the migration as a way to stop tutorials, API guides and integration examples from silently breaking as dependencies and APIs change — a problem it says is made worse by how quickly models, SDKs and best practices move in the AI space.

The core idea is that code samples should be treated like application code: run them in continuous integration, assert they execute correctly, and fail the build when they break. The catch LangChain identifies is setup cost. Making docs samples testable means extracting inline code into standalone files, adding setup and teardown so snippets are complete runnable examples, adding markup that designates which lines are the snippet, extracting those snippets with tooling, including them back into the docs, and running the standalone files in CI both on a schedule and whenever samples change. That sequence is exactly the kind of work teams tend to deprioritize, which is why stale samples persist.

LangChain's answer was to delegate the mechanical part to an agent rather than write bespoke tooling by hand. The Deep Agents CLI is a command line agent you chat with, and one of its capabilities is loading information from skills. Skills are reusable instructions the agent pulls in when a task matches the skill's description — effectively a written procedure that a person could follow.

The skill LangChain wrote is stored in the docs repository inside a hidden folder at .agents/skills/docs-code-samples/SKILL.md. Its frontmatter carries a description that tells the agent when the skill applies: migrating inline code samples from MDX documentation files into external, testable code files, extracted with Bluehawk and used as Mintlify snippets. The full SKILL.md body contains when to use the skill, the directory structure and file layout, step-by-step migration instructions, the commands to run and their order, and conventions for naming, tagging and imports.

The instructions the agent executes are concrete. Code moves into standalone files under src/code-samples/{product}, organized by product area. The agent adds setup and teardown so each snippet becomes a complete runnable example, then lints the code with the project's configured linters. Snippet boundaries are marked with :snippet-start: and :snippet-end: tags, and lines that should be excluded from the visible snippet can be wrapped with :remove-start: and :remove-end:. The agent then runs the samples to test them, generates snippets from the markup, and includes the generated files in the docs.

On top of the agent workflow, LangChain pairs the setup with a GitHub action that runs the tests regularly and opens tickets when a test fails. In practice that closes the loop: the agent does the one-time migration, and the scheduled CI job provides ongoing detection so breakage surfaces as a ticket rather than as a reader hitting a broken example.

The intended usage is conversational and requires no coding. Anyone can open the Deep Agents CLI inside the docs repo and ask the agent to make one or more code samples testable — for example, requesting that the inline code in a specific .mdx file be migrated to testable code samples. Because the skill is committed to the repository, the procedure is shared rather than living in one engineer's head, and the same request should produce consistent file placement, tags and commands across contributors.

Bluehawk, the tool named in the skill description, is the piece that reads the markup and emits snippet files, which Mintlify then includes in the rendered docs. That split matters: the source of truth is a runnable file that CI can execute, while the docs merely reference a derived snippet. A likely trade-off is that contributors must keep markup tags accurate and keep files under src/code-samples organized by product, since the generated docs now depend on that structure. Another practical consequence, inferred from the described workflow, is that the agent-written files still pass through the repo's linters and test runs, so review effort shifts from writing boilerplate to checking that generated samples genuinely exercise the documented behavior rather than merely executing.

LangChain frames this as one example of using skills with deep agents in a repository, and points readers to its CLI documentation to get started. The company also mentions Managed Deep Agents for building out-of-the-box user experiences, alongside its LangSmith agent engineering platform for debugging agent decisions, evaluating changes and deploying, though those references are separate from the docs-migration workflow itself.

Why it matters: documentation teams and developer-facing product groups at any company shipping tutorials or API guides face the same drift problem, and this approach suggests the migration can be described once as a skill and then executed by an agent against individual files. For developers, the immediate value is that turning a docs page into testable samples becomes a repeatable request rather than a multi-step project, with the GitHub action providing the ongoing safety net. The caveat, as inference, is that the workflow's reliability depends on the quality of the skill instructions and the rigor of the CI checks, so teams adopting it should expect to iterate on both.

Read at LangChain

Based on reporting from the original publisher. Visit the source for full context and later updates.

Publisher excerpt

Stale code samples are a docs problem nobody prioritizes. Here's how we used the Deep Agents CLI to automate migrating inline code into testable, CI-ready examples