Build efficient agents with knowledge distillation on Amazon SageMaker AI

AWS's Machine Learning Blog published a practical walkthrough for cutting the cost of tool-using agents by distilling a large "teacher" model into a smaller "student" that runs the same multi-step loop. The worked example is a shopping assistant built on the ShoppingBench benchmark: a shopper asks for something like a black waist bag under twenty dollars from a shop rated 4.5 stars or higher, and the agent searches a catalog, inspects candidates, checks constraints and recommends. The argument is economic rather than academic, since each completed task is several billed calls to a frontier model, repeated for every user.
The method described is sequence-level knowledge distillation, or SeqKD. The teacher generates full trajectories, the student is fine-tuned on the successful ones with ordinary supervised fine-tuning, and no teacher weights, logits or gradients are needed at training time. This matters for frontier models reachable only through an API: logit-based and on-policy distillation both require internal access that closed models do not expose, while SeqKD consumes nothing but generated text. The saved trajectories persist as a reusable dataset, so different students can be trained or tested on them without regenerating data.
For agents, the sequence being copied is not a block of prose but a trajectory: an ordered run of reasoning, tool calls and the observations those calls return, with each step conditioned on everything before it. Training on the whole sequence teaches the reasoning behind each action as well as the course of action itself. The post cites DeepSeek-R1's distilled models, where reasoning from a large teacher was fine-tuned into smaller Qwen and Llama students, as a well-known precedent.
The pipeline uses GLM-5.3 on Amazon Bedrock as the teacher, with reasoning enabled and temperature 1.0, acting as the ShoppingBench agent over a catalog of roughly 2.7 million products. ShoppingBench publishes 750 synthesized shopping problems, 250 each for product, shop and voucher intents, each with a hidden gold answer and constraints. The authors split these into 426 training and 324 held-out evaluation problems (103 product, 105 shop, 116 voucher) using fixed indices so the sets never overlap.
The teacher is sampled twice per training problem, producing 850 trajectories, and ShoppingBench's own programmatic scorer keeps only those that solve the task. Temperature 1.0 is deliberate: sampling rather than greedy decoding pushes the two attempts toward different queries and reasoning paths, raising the chance that at least one run succeeds and diversifying what the student learns from. This rejection-sampling step is what turns a capable-but-imperfect teacher into a clean training set, since the teacher only needs to be right often enough that the solved runs are worth imitating.
Encoding works one training row per step. Each row is a messages list of chat turns rendered with the student's own chat template, so training tokens match inference tokens. The user turn holds the system prompt plus the conversation so far; the assistant turn holds that step's reasoning block and its tool call in the harness's native form. Loss falls only on the assistant turn, teaching the model to produce the next reasoning-and-action rather than predict returned observations. The solved trajectories expand to about 2,600 training rows and 130 validation rows.
Two practical traps are flagged. Splits must be made by problem, not by row, because multiple rows share a problem and a row-level split would leak near-identical context into validation; roughly five percent of whole problems are held out. And per-step encoding repeats the growing context, while serverless fine-tuning truncates anything past dataset_max_len. The recommended fix is compressing tool observations, which dominate token counts: keep only fields the next decision depends on, such as id, price, rating and the constraints under check; drop raw listing text; and cap results per search. If rows still overflow, cap steps per trajectory. In this run the limit is 16,384 tokens, which fits most trajectories after trimming.
Datasets land as JSONL in Amazon S3 and are registered with DataSet.create from the SageMaker Python SDK v3, returning train and validation dataset objects. The training job uses SageMaker AI model customization with serverless supervised fine-tuning starting from Qwen3.5-4B in the JumpStart model hub, with LoRA chosen as the training type. Because the data is already formatted, there is no training script, container image or instance to manage; the service provisions compute, runs the job and returns a customized model, with metrics tracked in Amazon SageMaker with MLflow. Output goes into a model package group that versions each model, created ahead of time with ModelPackageGroup.create and passed to the trainer; a documented default hyperparameter set can be inspected and overridden before launch.
The published settings raise dataset_max_len to 16,384, use LoRA rank 64 with alpha 256, train three epochs at a global batch size of 64 and a learning rate of 0.0001, apply a 0.1 warmup ratio and a 0.95 train/validation split. A dry run validates the configuration without launching, then the job runs to completion and returns a model package ARN that evaluation and deployment consume.
Evaluation happens in two stages, cheapest first. The full end-to-end test is resource intensive, standing up a search server over millions of products, deploying an endpoint and running the agent loop on every held-out problem. Before that, a managed pre-check with SageMaker AI model evaluation scores one generated step at a time against the teacher's reference action through a custom scorer supplied by the user. The scorer grades tool selection rather than argument syntax, parsing tool names leniently from both sides, because selection is the behavior distillation most needs to transfer. The rule is simple: the student must beat the base model on this pre-check before the expensive end-to-end run is worth doing; if it does not, training needs fixing first.
Serving deserves attention. Fine-tuning teaches the student to emit the right tool call as text, but it does not execute anything. At inference the model is wrapped in a small deterministic harness that prompts the model, parses the emitted call, runs the tool, appends the observation and prompts again until a final answer appears. That harness is described as open-ended work left to the reader; the distilled student is deployed behind a SageMaker AI Inference real-time endpoint. Prerequisites include an IAM role with S3 read/write and teacher-invocation rights, access to model customization in a supported region, a Python 3.12+ kernel with pinned SDK versions, and at least an ml.m5.4xlarge notebook instance for CPU-bound data prep and local scoring.
Why it matters: teams running agentic products face the same compounding per-task bill, and this recipe shows a path to a smaller model that keeps much of the teacher's behavior without access to its internals. Developers should note that the heavy lifting is data engineering, not training: a reliable programmatic grader, problem-level splits, observation compression and a cheap per-step pre-check are what make the loop work. My inference is that the approach pays off most where tasks are verifiable automatically, since the rejection-sampling step depends entirely on a scorer that can tell solved runs from failed ones.
Based on reporting from the original publisher. Visit the source for full context and later updates.
Publisher excerpt
Agentic assistants built on frontier models run a multi-step tool loop, and paying frontier prices on every step is hard to sustain at production scale. This post shows how to use sequence-level knowledge distillation (SeqKD) to transfer a frontier teacher's agentic behavior into a small, lower-cost student on Amazon SageMaker AI.