AivexaNewsSearch
AI news for builders and product teamsChecked every hour

How Numan Built an Eval-First AI Health Coach with LangGraph and LangSmith

Collected Oct 19, 2026

Numan, a UK digital health company that supports patients managing long-term conditions across weight, hormones and sexual health, has published an account of how it built Nu, an AI health coach, on LangGraph and LangSmith. The company says its central lesson was that the quality of evaluation criteria mattered more than the agent itself. Nu is positioned as a wellness product for coaching and education, deliberately kept away from diagnosis, prescribing and clinical decision-making — boundaries that matter because, under MHRA guidance, software with a medical purpose becomes a medical device.

Nu is a LangGraph graph. A supervisor model routes conversations to domain-specialist sub-agents, each responsible for one coaching area with its own prompt and tools. Numan argues this composition makes a new specialist cheap to add and, just as importantly, keeps the evaluation surface uniform across agents. Because evaluation targets the graph, a test can exercise the whole system or a single sub-agent, and new specialists inherit the existing test coverage.

Numan started with the conventional setup: a dataset of inputs and expected outputs scored by an LLM judge. It found this worked poorly for coaching, which is a conversation rather than a single answer. Two failures stood out. Single-turn evals on static fixtures could not judge whether a reply was safe, since safety often depends on conversational context, and averaging one score across a dataset hid pockets of failure. In one run, a cluster of low scores turned out to be a flaw in the rubric: it expected empathy, exploration and advice within a two-turn exchange, while a human coach would pace those across a longer conversation. The other approach, scripting a patient's replies in advance, broke down whenever Nu said something the script did not anticipate, leaving later turns responding to a conversation that never happened.

The current framework separates what a single reply can demonstrate from what requires a whole conversation. Themed datasets cover coaching quality, safety, medication queries, routing and summarisation, and each is scored per category rather than as one average, so a cluster of failures stays visible. Rubrics are scoped so a judge only assesses what one reply can show, which stops it penalising Nu for pacing. Numan also emphasises that no separate test harness exists: every eval runs through the same LangGraph graph patients talk to, on the principle that the thing shipped is the thing tested.

That portability had a practical payoff. Nu runs on two frontier model providers, one for the supervisor and another for the specialists, chosen per agent because the roles involve different cognitive work. Because the eval suite is built as open, themed datasets, Numan could point the same datasets at a different provider configuration and compare results side by side in LangSmith. A migration it would normally expect to take weeks of manual testing took a day.

Safety is enforced in two places. Offline, safety probes run as per-category metrics and gate releases, so a regression blocks automatic shipping. Numan describes tuning toward "safe-completion" — a redirect that holds the line while still sounding like a coach — so out-of-scope questions get a warm, useful answer rather than a blunt refusal. Online, guard nodes run inside the graph: a safety guard reviews each outgoing response against a fixed set of clinical safety rules before the patient sees it, unsafe drafts are sent back to the supervisor to regenerate, and persistent failures are replaced with a safe fallback. One themed dataset exists specifically to test that the guard catches what it should. Separately, human health coaches review a sample of real conversations daily and flag apparent safety misses, calibrating what counts as safe.

The newest layer, still in prototype, simulates conversations rather than scripting them. Each eval supplies a patient persona, a scenario, the problems Nu should draw out, and a turn limit; a cheaper model plays the patient, reacting to whatever Nu actually says, and a judge then scores the transcript per category. This lets Numan probe adversarial cases such as clinical requests disguised as ordinary questions, and catch behaviour like rushing to advice instead of investigating. Numan reports that this prompt hardening reduced serious scope violations in jailbreak scenarios against the regulatory boundary, with parallel medication-query evals acting as a regression gate so prompt improvements did not break trusted behaviour.

Getting this layer right is, in Numan's telling, a rubric problem rather than an engineering one. Its first draft criteria contained two themes that contradicted each other on missed-dose handling: one said decline and signpost to a clinician, the other said safe-complete with general facts from the patient information leaflet. No single prompt can satisfy both. From this Numan concludes that prompts are not the scarce asset — a capable model drafts them well from limited context — but the criteria are. On a regulated boundary, those criteria must be authored and signed off by a clinician who also understands the regulatory line. Until that sign-off happens, results from the simulated-conversation layer are judged against engineering-drafted criteria and do not gate releases or support any safety claim.

The stack splits cleanly. LangGraph is the agent; LangSmith is where every real and simulated conversation lands as a trace, where reviewers annotate those traces to drive continuous improvement, and where datasets and experiments live. Numan says the integration keeps evaluation cheap because there is no glue code joining separate tools. This is a useful pattern for teams building conversational agents: the same instrumented graph serves production and testing, so scores reflect the deployed system rather than a stand-in.

Looking ahead, Numan plans more adversarial personas and scenarios, additional themes as specialists multiply, and wider clinician-in-the-loop rubric coverage wherever safety or scope is at stake. The next step is online evaluators running the same rubrics continuously against real conversations as traces arrive in LangSmith, so human reviewers start from flagged conversations instead of a daily sample. Early experiments also suggest an offline prompt-optimisation loop pointed at a single behavioural benchmark improved its score without hand-tuning.

Why it matters: Teams building agents in regulated or safety-sensitive domains should note that Numan treats evaluation criteria, not prompts or orchestration, as the main engineering investment and as a clinical governance artefact requiring clinician sign-off. Its approach of per-category scoring, running evals through the production graph, and blocking releases on boundary regressions is a template that product and platform teams could adopt without the health-specific parts. The one-day provider migration is a plausible inference about how much rework a portable, open eval suite can save.

Read at LangChain

Based on reporting from the original publisher. Visit the source for full context and later updates.

Publisher excerpt

How Numan built Nu, an AI health coach, on LangGraph and LangSmith with per-category evals, simulated conversations, and safety gates on a regulatory line.