Turning Database Grading into RL Rewards: Post-Training Agents with ThinkingBox

Microsoft and Hugging Face published a joint follow-up to their earlier ThinkingBox post and released ThinkingBox-Training on GitHub. The central idea: the same executable check that grades whether an agent left the right database records can be used as a reward signal for reinforcement learning. The pairing is neat because the grader already exists for evaluation, so the training reward needs no separate annotation pipeline or LLM judge for most tasks. The authors credit Mirco Milletari and Xiang Jiang from Microsoft, Sergio Paniego from Hugging Face, and former intern Zhuochun Li of the University of Pittsburgh.
The benchmark at the center is ThinkingBox-Bench, 507 tasks. Each one spins up a real backend, gives the agent a set of MCP tools and a simulated LLM user with private context, lets the agent work, and then runs deterministic checks against the final database state. Grading targets the resulting record rather than the specific sequence of tool calls, which the authors say lets models find new valid solution paths instead of imitating a human reference solution. In the earlier post, eighteen models each ran the benchmark twenty times, and the reported gap was between average success and consistent success. Kimi-K3 solved 93.89% of tasks at least once but completed only 68 of 507 on all twenty attempts. Claude Opus 5 was narrower on coverage but far more consistent.
For this round, Qwen3.8-27B was full-parameter fine-tuned with GRPO for 50 steps and then evaluated on the sealed 507-task benchmark, which it never trained on. Average success (pass@1) went from 51.70% to 60.78%. Observed 20/20—tasks passing every one of the twenty recorded attempts—rose from 38 to 107, roughly a threefold gain. Pass@20, the share solved at least once, slipped slightly: up in neobank and consulting, down in retail, travel and auto insurance, with auto insurance losing 7.0 points of breadth and retail gaining 24.5 points on observed 20/20. That pattern, repeatable success up and coverage down, is consistent with sharpening: probability mass moves toward trajectories that work reliably while some low-probability paths disappear.
One worked example shows the behavior change. A corporate hotel-booking task requires manager approval before a room downgrade. Before training, the agent changed the booking and issued refunds without approval in 15 of 20 attempts; after fine-tuning it preserved the booking and payment state in all 20, with all expected records matching in 19. The results also complicate model selection by headline accuracy. GPT-5.6 Sol scored slightly higher on pass@1 than the tuned 27B (61.91% versus 60.78%) but passed fewer tasks on all twenty attempts (82 versus 107). Claude Sonnet 4.6 was lower on pass@1 at 59.19% but passed about as many tasks every time, at 102. This suggests product teams choosing models for work that touches production records should look at the everything-passes column, not just average success.
The method's most interesting contribution is how training tasks were chosen. GRPO computes advantage within a group of rollouts for each task; if all eight rollouts fail, or all eight succeed, the group carries no contrast and teaches nothing. Since every rollout here is a full stateful episode with its own backend and simulated user, wasted groups are expensive. So the pipeline screens candidate tasks offline against the base policy. Each task is probed n times, producing k successes, and the latent success probability is modeled with a Jeffreys posterior, Beta(k + ½, n − k + ½). The unit interval is split at 0.08, 0.40, 0.60 and 0.90; tasks whose posterior mass lands mostly in the lowest or highest bucket are dropped, and the middle bands are kept. The authors call this offline policy-relative difficulty, because hard is relative to the current policy and is estimated once, with the pool fixed for the run. A GPT-5.4 screen filtered for solvability first. The final Qwen3.8-27B pool held 187 tasks, disjoint from the 507 benchmark tasks.
The reference run used token-mean GRPO with 18 groups of 8 rollouts per update, 144 episodes, across 24 H100s on three nodes of eight, synchronous training with parameter and optimizer offload, AdamW at 1e-6 with zero weight decay, and 50 steps. Zero-variance groups were replaced DAPO-style, and backend timeouts were treated as system errors and refilled rather than scored as failures—a sensible distinction, since infrastructure failure says nothing about the agent's competence. Rewards are mostly deterministic terminal-state checks, returning 1 or 0; a minority of narrow rubric requirements invoke a configured LLM judge (GPT-5 Chat) for a binary answer. Nothing about the trajectory itself earns reward, only the final state.
The release includes data-selection, rollout, reward and Verl integration code in microsoft/thinkingbox-training, alongside pinned checkouts of thinkingbox and thinkingbox-data (tag thinkingbox-bench-v1.0). Requirements are Linux or WSL, Python 3.12 with uv, Verl 0.9.0, NVIDIA GPUs with a compatible CUDA/NCCL stack, a local Qwen3.8-27B checkpoint, and OpenAI-compatible endpoints for the simulated user and judge. Documentation covers data preparation, the validated 24-H100 configuration, and merging a checkpoint to rerun the 507 by 20 evaluation. A public sample task list demonstrates the interface but is not the private 187-task pool. The team also flags a subtle requirement: training and evaluation must parse the model's reasoning, tool calls, argument types and rendering identically, via shared code in trainer/core/tb_interpret.py, or the reward measures a different agent than the one evaluated. The benchmark is out-of-distribution relative to training—domains, databases, policies and most tools differ—though both share the ThinkingBox framework, simulated user and grading machinery, a few common tools and a similar task style.
The authors also price consistency: an estimated cost per dependable task (campaign cost divided by tasks passing 20/20) put the tuned model at $8.00, near GPT-5.4 at $6.80, GPT-6 Astra at $7.45 and Claude Opus 5.5 at $7.80 in the comparison. Cost per successful attempt fell from $0.177 to $0.139. Those figures are a serving index, not invoices; training and probing costs are excluded.
Why it matters: teams building agents that mutate real systems now have a public recipe for turning outcome checks into training signal, which is useful because most of that software already has tests or database assertions to borrow. The results suggest a smaller open model can be made more dependable on out-of-distribution backend work, at the cost of some breadth—inference, but a likely trade-off for anyone optimizing for production reliability. The harder lesson may be the selection step: RL budget is wasted on tasks the model always passes or always fails, so instrumenting difficulty before training is probably as valuable as the reward itself.
Based on reporting from the original publisher. Visit the source for full context and later updates.