AivexaNewsSearch
AI news for builders and product teamsChecked every hour

How Deepgram optimized voice AI at scale with AWS Trainium

Collected Oct 19, 2026

Deepgram, working with the AWS Forward Deployed Engineering (FDE) team, has brought its Nova-3 speech-to-text model to AWS Trainium2 for offline batch workloads such as contact-center analytics and medical transcription. The port began with Deepgram's standard PyTorch code and took several months of reconstruction in PyTorch plus hardware-specific tuning. In its final form the configuration runs eight model replicas on a single Amazon EC2 trn2.3xlarge instance. On Deepgram's offline, throughput-oriented batch workloads, that setup processed 11–70 percent more audio per dollar than the compute options Deepgram had been running before, measured using instance pricing at the time of the evaluation. Deepgram says it processes quadrillions of audio tokens a year for more than 200,000 developers and 1,400 organizations, spanning speech-to-text, text-to-speech and voice agents, delivered through its API or run in customer environments including Amazon SageMaker AI.

For a company whose economics hinge on serving efficiency, the relevant metric is how much audio each accelerator hour yields, since high-volume batch transcription puts steady pressure on infrastructure budgets. AWS positions the result as validation that Trainium is a viable and efficient option for inference workloads, and notes the same benefits can extend to customers who self-host Deepgram models in their own VPC on SageMaker AI, where the two companies continue work on setup ease, reliability and observability. The cost figures apply to offline, throughput-oriented scenarios; nothing in the reported results speaks to real-time streaming latency, which is a separate constraint for voice-agent products.

The starting point was TorchNeuron, the native PyTorch integration of the Neuron Compiler, which supports both eager execution and torch.compile. Running Nova-3 in eager mode first let the team identify operations that fell back to the CPU; compiling with torch.compile produced a path where every model operation ran on Trainium2 with no detected CPU fallbacks. Encoder and decoder were compiled as separate static-shape graphs. Accuracy was checked at two levels throughout: layer-level output consistency against CPU, and end-to-end Word Error Rate, which had to stay in line with Deepgram's reference values and target.

From that baseline, several layers were replaced with fused kernels from the public NKI Library (NKILib), a set of pre-built reference kernels written with the Neuron Kernel Interface, covering fused QKV projections, fused feed-forward network layers and depthwise 1D convolution. The team swept batch sizes from 1 to 128 to find the throughput-maximizing point, reporting throughput as a relative gain over a baseline rather than as absolute audio processed per second.

Enabling kernels was only the first step. The team describes an optimization loop: enable an NKI kernel, profile the workload with Neuron Explorer, then tune or replace the kernel based on what the profile shows. The Neuron Agentic Development (NAD) toolkit sped up the analysis, with its profiling skills tracing a profile to a root cause and its kernel-writing skills helping produce replacements. Three interventions came out of this.

The feed-forward network reference kernel reloaded the full set of projection weights from off-chip memory for every token tile, so memory bandwidth limited throughput as token counts grew. The hand-tuned replacement loads each weight tile once, reuses it across multiple token tiles and pipelines weight reads with on-chip compute, yielding 2.0x lower latency than the reference kernel at batch size 96. The depthwise conv1d kernel, framed as an implicit general matrix multiply, left Tensor Engine utilization low; rewriting it as a broadcast multiply-accumulate loop on the Vector Engine reached full engine utilization, enough to more than offset that engine's lower peak throughput, for a 1.5x speedup at batch size 96. The model's conv2d layers had blocked compilation as a single graph beyond batch size 32, and NKILib had no conv2d kernel, so one was written from scratch in NKI and iterated with profiler feedback, unlocking larger batch sizes and delivering up to 50x on the convolution module at batch sizes of 32 or below.

Serving required as much work as compilation. Each Trainium2 device has eight physical NeuronCores, exposed by default as four logical cores (LNC2). Because one Nova-3 instance fits comfortably on a single physical core, the team moved to one physical core per logical core (LNC1), hosting eight replicas (DP8) per device. Deepgram's production serving stack is written in Rust, so the model had to run without Python or PyTorch in the serving path. The result is a lightweight runtime where Rust coordinates replicas through a shared work-stealing queue with one thread per replica, while a C layer loads and executes ahead-of-time-compiled Neuron Executable File Format files directly through the Neuron Runtime API (libnrt). It reproduced the reference token sequences exactly at every tested batch size.

Benchmarked against multithreaded and multiprocessing Python data-parallel runtimes, the Rust runtime removes Python interpreter overhead entirely and, versus the multithreaded baseline, reduces global interpreter lock contention on dispatch calls. This matters because Nova-3 issues a sequence of small decoder calls per audio chunk, making host-side dispatch the bottleneck at low batch sizes; gains are largest there and still meaningful at the throughput-maximizing batch size, where the Rust runtime delivered a 7.5 percent throughput gain over the multiprocessing Python baseline. My reading is that this is the pattern to watch for anyone serving many small sequential calls: once the GPU or accelerator work is efficient, the orchestrating language starts to set the ceiling.

Why it matters: Teams running large speech workloads now have a published reference for moving a non-trivial ASR model to AWS's own silicon, including the kernel-level and runtime-level tricks that produced the gains. The 11–70 percent audio-per-dollar range is a wide band, so product and platform teams should treat it as a signal to benchmark their own batch pipelines rather than a guaranteed figure. Inference from the reported scope: the numbers cover offline batch transcription, not real-time streaming, and customers self-hosting Deepgram on SageMaker AI stand to inherit the efficiency work as it matures.

Read at AWS Machine Learning Blog

Based on reporting from the original publisher. Visit the source for full context and later updates.

Publisher excerpt

Deepgram worked with the AWS Forward Deployed Engineering team to port its Nova-3 speech-to-text model to AWS Trainium2 for offline batch inference. This post covers porting to the AWS Neuron stack, kernel-level tuning, a data-parallel Rust runtime, and the throughput and cost results on batch transcription.