AivexaNewsSearch
AI news for builders and product teamsChecked every hour

Cost-optimizing AssemblyAI speech-to-text inference on AWS with Amazon EC2 Spot

Collected Nov 13, 2026

AssemblyAI detailed how it runs speech-to-text inference on Amazon Web Services in a post co-authored with AWS solutions staff, published on the AWS Machine Learning Blog. The transcription provider, whose API is sold through AWS Marketplace, runs production inference on a fleet of thousands of NVIDIA GPUs that mixes Amazon EC2 Spot Instances with On-Demand capacity. The company reports 70 percent savings on inference compute compared with running everything on On-Demand pricing, while meeting the availability and latency commitments its service level agreements promise. Compute demand on the platform grew nearly four times year over year, according to the post.

Each audio file triggers a pipeline of models — acoustic models, language models and post-processing stages — and AssemblyAI runs a single inference model per GPU. The orchestration layer is Amazon Elastic Container Service (ECS), with capacity managed through ECS capacity providers and Amazon EC2 Auto Scaling groups. Jobs are decoupled from workers using Amazon Simple Queue Service (SQS): the platform writes each transcription request to a queue, and GPU-backed ECS tasks poll the queue, pull audio from Amazon S3, run the pipeline and write results to a store that delivers output to the customer. Individual jobs typically take between 15 seconds and several minutes depending on audio length.

The architecture rests on four design choices. The first is a capacity provider strategy inside each ECS cluster: an On-Demand provider acts as an always-on baseline, while a Spot provider absorbs demand spikes. ECS managed scaling adds or removes GPU instances based on capacity provider weights and the number of pending tasks, so the Spot pool grows when the queue grows and scales in gracefully when demand falls.

The second is diversification across GPU instance families. Rather than standardizing on one type, AssemblyAI spreads work over three EC2 families: g4dn, built on NVIDIA T4 GPUs, which serves as a cost-effective baseline for standard speech recognition models; g5, with NVIDIA A10G GPUs, for higher throughput on more demanding model architectures; and g6, with NVIDIA L4 GPUs, which the post describes as offering better performance per dollar for larger models, including LLM-based inference. The fleet spans multiple sizes within those families, such as g4dn.xlarge, g4dn.2xlarge, g5.xlarge, g5.4xlarge and g6.2xlarge. AssemblyAI uses EC2's price-capacity optimized allocation strategy, which draws from Spot pools with the most available capacity at the lowest price, to reduce the odds of interruption. The company says this diversification is the most effective lever it found for cutting interruption rates, because each family draws on separate capacity pools.

The third is a retry mechanism built around Spot's two-minute interruption notice. When a Spot Instance is reclaimed, Amazon EventBridge invokes a Lambda function that returns the in-flight job to the main SQS queue. Jobs that can finish inside the two-minute window are allowed to run to completion; the rest go back to the queue and become immediately available to another healthy worker, possibly on a different instance type or in a different Availability Zone. Jobs that exhaust their retries route to a dead-letter SQS queue for investigation. To shorten recovery, AssemblyAI builds custom Amazon Machine Images containing model artifacts and container images that can boot in roughly two minutes, and it tracks fleet health and interruption events with Amazon CloudWatch metrics. From the customer's side, the post says, an interruption shows up only as a small increase in processing time.

The fourth is regional isolation for data residency. Each AWS Region runs a fully independent copy of the stack — its own SQS queues, ECS clusters, capacity providers and GPU fleets. The post names US West (Oregon), us-west-2, and Europe (London), eu-west-2, and says the three GPU families are available in those European Regions at competitive Spot prices. That matters for customers in healthcare, financial services and legal work, who need audio to stay within geographic boundaries.

For developers, the pattern is a familiar one applied to GPU inference: treat Spot capacity as the default and On-Demand as a small safety margin, keep jobs short and independently retriable, and make warm-up fast enough that a replacement worker is productive quickly. This is inference plumbing rather than a new model, so the transferable lesson is architectural. The design depends on the workload's shape — jobs of seconds to minutes that can be rerun from scratch — and would be harder to apply to long-running training runs or stateful sessions that cannot be cheaply restarted. A likely trade-off is added latency variance under heavy Spot pressure, which AssemblyAI frames as a small processing-time increase rather than a failure. Another consideration is that save rates depend on current Spot prices and interruption frequency, which vary by family, size and Region; the 70 percent figure reflects AssemblyAI's mix and history rather than a guarantee for other workloads.

Why it matters: Teams running GPU-heavy batch or near-real-time AI workloads now have a concrete, published production reference for using Spot capacity without breaking SLAs, and the biggest reported saving comes from breadth across instance families rather than from any single clever trick. For product teams, the takeaway is that queue-based retry plus fast image warm-up can turn interruptions into routine events, though the approach presumes short, restartable jobs. For teams with data residency obligations, the regional-isolation model suggests Spot savings and strict geographic boundaries can coexist, at the cost of operating independent stacks per Region.

Read at AWS Machine Learning Blog

Based on reporting from the original publisher. Visit the source for full context and later updates.

Publisher excerpt

Learn how AssemblyAI runs cost-optimized speech-to-text inference at scale on AWS, using Amazon ECS, Amazon EC2 Spot GPU instances, and an Amazon SQS-based retry architecture to achieve 70% cost savings while maintaining SLA commitments across European Regions.