AivexaNewsSearch
AI news for builders and product teamsChecked every hour

Introducing usage-based pricing for Amazon SageMaker AI bidirectional streaming

Collected Nov 11, 2026

AWS announced usage-based metering for Amazon SageMaker AI bidirectional streaming inference, letting model providers report actual consumption during a live stream so that the AWS Marketplace software charge tracks usage rather than availability. Deepgram, the real-time speech AI company, is the launch partner, bringing usage-based Speech-to-Text and Text-to-Speech billing to AWS Marketplace. SageMaker AI endpoint infrastructure charges continue to apply separately while an endpoint is provisioned.

The previous marketplace metering mechanism fit request-response inference: a model provider reports a unit count in an HTTP response header, and AWS Marketplace bills that quantity. Streaming sessions break this model because billable usage accumulates over time. A bidirectional session runs through an HTTP/2 event stream from the client to SageMaker AI, which connects to the model server over WebSockets, and can last anywhere from 30 seconds to 30 minutes. The final total is unknown until the session ends, so a single response header cannot report usage as it accrues. Hourly software pricing, the alternative, can charge customers for quiet periods when the model is merely available.

The implementation adds a second WebSocket alongside the inference stream. At session start SageMaker AI opens the normal data connection and, if the model server opts in, a companion metadata connection dedicated to metering. The opt-in is a single header on the data WebSocket upgrade response: X-Amzn-SageMaker-Metadata-Stream-Supported set to true. SageMaker AI then connects to the model server at the default path /invocations-bidirectional-stream-metadata, or a custom path set via X-Amzn-SageMaker-Metadata-Invocation-Path. Both WebSocket handshakes carry the same X-Amzn-SageMaker-Request-Id header, which the model server uses to associate metering records with the active inference session.

Metering data travels as JSON text frames on the metadata connection, one record per frame, never arrays. Each record nests a Metering object with a Dimension field that must be inference.count, a ConsumedUnits number of zero or greater, and an optional ClientToken string of up to 64 characters for idempotency. A duplicate ClientToken on the same connection is dropped silently. The dimension name is a label, not a rule: the provider defines what counts as a unit, and the marketplace offer maps that unit to a price. For a product priced per second of audio, twelve minutes of metered audio is sent as ConsumedUnits 720. AWS recommends sending records incrementally as usage accrues, either per unit or as periodic batches since the prior record, and sending any final record as soon as the session ends, to limit unreported usage if a session terminates unexpectedly.

The metadata stream has a rate quota of one message per second per connection and a 512-byte maximum message size. The connection relies on TCP for ordered, reliable transport, but transport delivery does not by itself establish billing; AWS advises providers to reconcile reported units against Marketplace reports and buyer charges, including in interrupted-session tests.

A second header, X-Amzn-SageMaker-Metadata-Stream-Required, governs failure behavior and defaults to false. With the default, if the metadata connection fails to establish or drops mid-session, inference continues in data-only mode without metering. Setting it to true terminates the data stream when metering is unavailable, trading inference availability for metering continuity. Malformed records are dropped while the metadata connection stays open, under either setting. When the data connection breaks, the metadata stream is closed. In practice, this is a policy decision for each provider: a per-second audio product may prefer the strict setting so customers are not served unbilled audio, while a real-time transcription product may prefer to keep talking to end users.

Integration is three steps for a model server: return the support header on the data WebSocket upgrade, accept the upgrade on the metadata path and correlate sessions by request ID, and emit metering frames as usage accrues. AWS also points to the SageMaker Marketplace Onboarding Agent Skill, whose starter templates advertise metadata-stream support and build validated incremental ConsumedUnits frames, with a warning to review and production-harden generated code before submission. Listing a model on AWS Marketplace follows the existing path: register as a seller, complete tax and banking details, create a SageMaker AI model package and an IAM role Marketplace can assume, then create the product in AWS Partner Central. Real-time inference uses per-inference pricing with one per-unit price across all instance types, displayed per request with up to three decimal places, plus an hourly price for each batch transform instance type. The pricing model must be chosen at listing creation, since a released listing cannot switch between hourly and per-inference.

For customers, the change means the software charge becomes per unit of processed usage, consolidated into the AWS bill, with private offers available for negotiated per-unit rates or fixed-price contracts and no separate billing relationship with the model provider. A twelve-minute open connection does not by itself equal twelve minutes of billable audio; the provider's counting rules define the quantity. Getting started with Deepgram requires deploying a real-time endpoint from its Marketplace subscription and streaming audio through the SageMaker AI bidirectional streaming API, with no client-side metering configuration. AWS advises deleting test endpoints and subscriptions afterward to avoid ongoing infrastructure charges.

This suggests the design mainly matters where streaming sessions are long-lived and variable, such as contact centers, voice agents, and captioning, because that is where header-based billing and hourly pricing both misalign with cost. A likely trade-off is added complexity in the model server, which now must manage a second connection, idempotency tokens, and a billing-correlation ID alongside its inference loop, plus a new metric to watch: unreported usage when metadata streams fail while inference deliberately continues.

Why it matters: model providers listing streaming models on AWS Marketplace can now bill per unit of actual consumption instead of per hour, while buyers get metered charges folded into their normal AWS bill. Developers deploying speech models like Deepgram's should expect per-second-style software charges separate from endpoint costs. The main engineering work falls on providers, who must opt in, emit metering frames within strict rate and size limits, and test failure cases before release, since a lost metadata stream can leave audio served but unbilled.

Read at AWS Machine Learning Blog

Based on reporting from the original publisher. Visit the source for full context and later updates.

Publisher excerpt

AWS Marketplace now supports usage-based metering for Amazon SageMaker bidirectional streaming inference, so model providers can charge for what customers actually consume during a live stream. Deepgram is the launch partner, bringing usage-based speech-to-text and text-to-speech billing to AWS Marketplace.