AivexaNewsSearch
AI news for builders and product teamsChecked every hour

ALTK-Evolve: How learned memory tops the τ²-bench leaderboard on customer service tasks without a frontier model

Collected Oct 20, 2026

ALTK-Evolve, described in a Hugging Face blog post, is a method for improving customer service agents without retraining a model. Using GPT-4.1, a non-reasoning model, a basic agent on τ²-bench's stock ReAct scaffold reached 67.5% Pass^4 on the retail domain, above every retail entry in the leaderboard snapshot the authors used from October 12, 2026. The only modification to the benchmark setup is a block of learned guidelines appended to the system prompt, placed after the domain policy as a plain markdown list. Model weights are untouched.

The guidelines are produced by the same model that drives the agent. It reviews the agent's attempts on training tasks and condenses successes and failures into short behavioral rules, such as authenticating a customer before order actions, confirming intended changes with a structured summary and explicit approval before any database write, and taking one action per turn without pairing a tool call with a user-facing message. The rules are then applied unchanged to the held-out test set. No human annotation is involved, and the authors say there is no test-set leakage.

The benchmark, τ²-bench from Sierra, pairs an agent with an LLM-simulated customer who holds a goal, a set of domain tools, and a written policy. A task counts as solved only when the final database state matches the expected outcome, a strict binary reward. The authors ran the benchmark's own two-phase THINK → ACT ReAct agent with gpt-4.1 as the agent model and gpt-5.5 as the customer simulator, four trials per task, across the retail and airline domains.

Two metrics are reported. Mean Accuracy is the average success rate across the four trials and reflects capability. Pass^4 is the share of tasks solved on all four attempts and reflects reliability; it is the figure the leaderboard tracks. The authors argue the second matters more for support work, since an agent that resolves a refund three times in four still fails a quarter of customers.

Against baseline, Mean Accuracy rose on both domains: retail from 78.1 to 82.5 (+4.4 points) and airline from 57.5 to 65.0 (+7.5). Pass^4 moved further, up 10 points in each, from 57.5 to 67.5 in retail and 35.0 to 45.0 in airline.

How much guidance to keep depends on the domain. Each guideline carries a support count, the number of distinct training tasks it was mined from. The authors compared baseline, a sup3 core (guidelines with support of at least 3), and a lossless set (everything retained after consolidation). In retail, more was better on every metric, and the real jump required the full lossless set, reported at 292 guidelines. In airline the picture flipped at the reliability end: lossless led on accuracy (65% vs 60%), but a compact 14-guideline sup3 core took Pass^4 to 45% against lossless's 40%. The authors treat support count as a per-domain knob rather than a fixed value.

The extra performance does not come from longer conversations. Comparing runs solved under both baseline and guideline settings with identical scenario and seed, lossless trimmed about 0.3 to 0.4 assistant turns in each domain (retail 10.59 to 10.16, airline 9.15 to 8.82), and no variant lengthened them. The cost instead lands in input tokens, since the guideline block rides on every call: retail's large prompt pushes cost from roughly $0.084 to roughly $0.139 per conversation, while airline's smaller playbook is cheaper. The authors note that in production most of that is reclaimable through prompt caching, because the block is identical across calls. Fewer turns also means fewer sequential round-trips and lower end-to-end latency.

The method was also tested with a frontier model, rerunning airline with gpt-5.5 for both the agent and guideline extraction and re-mining rules from the agent's own runs. In this setup no guideline reached support 3, topping out at 2, so the high-support core became sup2. Appending the learned rules lifted accuracy 10 points (72.5 to 82.5) and Pass^4 15 points (55.0 to 70.0), a larger reliability gain than GPT-4.1 saw on the same tasks and starting from a higher floor. The lossless configuration also reproduced exactly across reruns, where the baseline's reliability swung.

On the full base split (train plus test), which the benchmark requires for submissions, the retail agent with learned guidelines reached Pass^4 of 67.5%, ahead of RAFT-30B-A3B at 61.4%, a model fine-tuned on synthetic τ²-bench retail data, a margin of 6.1 points. On airline, the sup3 core with gpt-4.1 gave 44%, eighth place, above GPT-5.2 with no reasoning and close to Claude Sonnet 4.5. With gpt-5.5 as the base model the same approach reached 82.0%, above every frontier entry in the snapshot, which ranged from 66% to 72%. The authors point out that base-split figures differ from the held-out test numbers above because they cover a different task set.

This suggests the appeal for teams is largely operational. Fine-tuning is expensive, can be brittle, and does not carry across domains, whereas prompt-level guidelines are cheap to generate, easy to inspect and trivial to roll back. A rule list is also legible in a way weights are not, which fits domains governed by written policy. The trade-offs are real, though: a 292-guideline prompt is a large fixed input cost on every call and depends on caching to stay cheap, and the volume-versus-reliability result in airline is my reading of a caution that more rules can sometimes hurt consistency rather than help it. There is also a ceiling, since guidelines shape behavior but cannot add capabilities the base model lacks, which likely explains why gpt-4.1 places eighth on airline while gpt-5.5 takes first. The approach also assumes enough representative training attempts to mine from; the authors generated two leaderboard submissions, referenced as τ²-bench PR #542 and PR #551, and published the library as open code.

Why it matters: Teams building customer service agents now have evidence that a stock scaffold plus an automatically learned playbook can match or beat fine-tuned and frontier baselines on a strict reliability metric, without weight updates, human annotation or a bigger model. The practical work shifts toward curating and versioning guideline sets per domain and tuning how many rules to keep, with prompt caching handling much of the added cost. For product teams, fewer turns alongside higher accuracy hints at faster resolutions rather than the usual quality-for-latency trade, though that is inference from the reported turn counts rather than a production measurement.

Read at Hugging Face Blog

Based on reporting from the original publisher. Visit the source for full context and later updates.

Publisher excerpt