AivexaNewsSearch
AI news for builders and product teamsChecked every hour

Build a model improvement flywheel with Amazon SageMaker HyperPod Data Capture

Collected Nov 13, 2026

Amazon Web Services has added Data Capture to Amazon SageMaker HyperPod, a built-in capability that writes live inference inputs and outputs to Amazon Simple Storage Service (Amazon S3). The AWS Machine Learning Blog post describes it as a way to keep records of production traffic on models that meet inputs no pre-curated dataset anticipated, such as domain jargon, new products and shifting user behavior. AWS states the capture runs with minimal latency impact and zero impact on model availability. The post walks through a complete example on a single HyperPod cluster: a support-ticket router captures production traffic, Amazon Athena queries reveal a new-product blind spot, a SQL policy labels the records, and a fine-tune on that data teaches the router the new product. The example uses synthetic data, and the scripts, manifests, SQL and datasets are in an accompanying GitHub repository in the awslabs/awsome-distributed-ai project.

Three capture tiers sit at different points in the request path, and any combination can be enabled in one dataCapture block. Tier 1 is the SageMaker endpoint, managed by SageMaker's native data capture, storing full request and response payloads; AWS positions it for compliance, Model Monitor and audit work. Tier 2 sits at the load balancer and records Application Load Balancer access-log fields: request line, status codes, timings, client IP and user agent, aimed at low-cost traffic and error-rate monitoring. Tier 3 runs in the model pod through the HyperPod inference operator, which adds NGINX, an OpenTelemetry collector and an S3 uploader as sidecars, and writes full payloads plus event metadata for debugging, analytics and training-data collection. Tiers 1 and 3 both store complete payloads, so turning on both duplicates every record.

Tier 3 configuration covers a sampling percentage, batch size and flush interval for the upload buffer, a maximum payload size for truncation, AWS KMS key for encryption at rest and allowed content types. Defaults are 100% sampling with both input and output captured, a batch size of 10 (the walkthrough sets 50), a 60-second flush interval, and no payload size limit (the walkthrough caps at 1024 KB). Because Tier 3 holds full payloads that can contain names, addresses and order numbers, AWS advises sampling, KMS encryption via kmsKeyId, S3 lifecycle rules and PII redaction, for example with Amazon Comprehend, before the data becomes training material.

Prerequisites are a HyperPod cluster orchestrated by Amazon EKS with the inference operator and the amazon-sagemaker-hyperpod-training-operator EKS add-on installed, two NVIDIA GPUs (one for serving, one for training), kubectl access and an S3 bucket in the same Region. The example used an ml.g6e.8xlarge for serving and an ml.g6e.xlarge for training, each carrying a 48 GB NVIDIA L40S. This is a practical constraint: the serving pod holds its GPU for the life of the endpoint, so with only one GPU the training pod stays Pending.

Enabling capture takes a dataCapture block inside an InferenceEndpointConfig with modelPod.enabled set to true; the surrounding spec is an ordinary vLLM deployment, in this case vllm/vllm-openai:v0.10.0 serving Qwen2.5-3B-Instruct from S3. AWS chose that model because it is small enough to fine-tune in about a minute and fits on one L40S. A capture-enabled pod runs four containers: the model, sidecar-reverse-proxy (NGINX), otel-collector and s3-uploader. In the walkthrough the pod was ready about five minutes after applying the manifest, most of that spent downloading weights.

Requests must go through the capture proxy on port 8081; the model itself listens on 8000, and traffic sent there is not captured. After the flush interval, records appear as .jsonl objects under a path of the form {s3Uri}/{uniqueID}/pod/{variant}/{YYYY}/{MM}/{DD}/{HH}/, where uniqueID is generated per endpoint. Each line uses the SageMaker data-capture format: captureData.endpointInput and endpointOutput each hold the content type, a mode, base64-encoded data and an encoding field, alongside eventMetadata with an eventId and inferenceTime and an eventVersion. Reading the bodies back means base64-decoding and parsing JSON.

Permissions span four identities. The Mountpoint-S3 CSI driver role writes the Tier 3 capture files and needs s3:PutObject, s3:AbortMultipartUpload and s3:DeleteObject on the bucket, plus kms:Decrypt, kms:GenerateDataKey and kms:DescribeKey when a KMS key is set. The inference operator execution role needs s3:PutObject and, with KMS, kms:Decrypt and kms:GenerateDataKey. The training pod's service account reads the base model and training data and writes the merged model through EKS Pod Identity. Athena needs read access to the capture prefix and a query-results location. A useful operational detail: if the CSI driver role is misconfigured, the endpoint stays healthy and requests succeed, but no captures appear, so that role is the first place to look when records go missing.

In the scenario, the router classifies tickets into ProductFridge, ProductWasher, ProductOven, ProductDishwasher, ProductMicrowave or Other using vLLM guided JSON decoding, which constrains output to a valid object with one of those labels and a confidence number. After deployment the retailer launched its own Arctis air conditioner line (9K, 12K, Mini and so on), which has no category. The model cannot flag novelty; it picks the nearest known label. The synthetic traffic is 900 tickets replayed over 15 minutes: 400 before the launch with no air conditioners, then 500 in which the Arctis share ramps from 15% to 55%. The held-out evaluation set is 280 labeled tickets, 40 per category including ProductAC, built from symptoms, sentence templates and Arctis model names that never appear in the traffic.

On the analysis side, AWS notes that offline benchmarks look unchanged after the launch because they are frozen in the past, and that the shift shows up only in captured traffic. Athena reads the S3 files in place via an external table over the record envelope using the OpenX JsonSerDe, with partition projection over year, month, day and hour so hourly prefixes map to partitions without a crawler or ADD PARTITION. The base64 fields are declared as plain strings, so queries decode them in SQL.

A likely trade-off is cost and duplication: full payload capture multiplies storage and downstream scan volume, which is why sampling, truncation and lifecycle rules matter, and why capturing at both endpoint and pod tier is wasteful. Capture through sidecars also means the proxy port is the boundary, so any client that reaches port 8000 directly bypasses the record entirely. The pattern AWS describes is a flywheel rather than a one-off job: capture, analyze, label, fine-tune as a HyperPodPyTorchJob on the second GPU, then serve the new model from a second endpoint with capture still on, so the next drift is caught the same way.

Why it matters: Teams running open-weight models on HyperPod get a supported path to turn production traffic into training data without building their own logging pipeline, which matters most when new products or vocabulary appear after a model ships. Developers should budget for GPU capacity beyond serving, plan IAM carefully around the CSI driver role, and treat captured payloads as sensitive data requiring redaction and encryption. Whether the measured accuracy gain generalizes beyond synthetic traffic is an open question the walkthrough alone cannot answer.

Read at AWS Machine Learning Blog

Based on reporting from the original publisher. Visit the source for full context and later updates.

Publisher excerpt

Amazon SageMaker HyperPod Data Capture records live inference inputs and outputs to Amazon S3. This post walks through one end-to-end example: capturing a support router's production traffic, surfacing a new-product blind spot with Amazon Athena, fine-tuning on the captured data with the HyperPod training operator, and measuring the before-and-after lift.