[AINews] DeepSeek DeepInfer: Open Source SOTA Inference running 155T toks/day/cluster making $22m/day at ~80% margin for a 81 day payback

DeepSeek released DeepInfer, the inference engine it uses to serve its V4.1 Flash model, timed to the first day of PyTorch Conf. The release covers the inference engine itself along with deepseek-recipe, a component handling production token encoding and decoding and processing compatible with DeepSeek's official API. DeepSeek said future model inference implementations and optimizations will also land in the same repository, and the engine is published under Apache 2.0.
DeepInfer supports distributed inference, speculative decoding, SWA Bounded Replay and structured output on both NVIDIA and Huawei Ascend hardware. The company credits vLLM, SGLang and TRT-LLM, and describes the code as a customization of vLLM rather than a from-scratch engine.
The accompanying engineering report gives production numbers: 155 trillion tokens served per day from a single cluster, a 96% input-cache hit rate, 400 ms average time to first token, 230 tokens/s generation overall and 300 tokens/s for coding workloads. DeepSeek also reports $38 in revenue per accelerator-hour at off-peak prices, against API pricing of $0.15 per million input tokens and $0.60 per million output tokens. These are first-party figures from DeepSeek's operational data and have not been independently audited.
The open-source label comes with a significant caveat. KV-cache management and load balancing are not fully released, because they remain tied to DeepSeek's internal infrastructure; 3FS, the storage system, was open-sourced earlier. In practice, teams can run the engine, but reproducing the full production service is a deployment problem they own.
Architecturally, DeepInfer splits vision encoding, prefill and decode so each stage scales against its own bottleneck. The recommended shapes are standard eight-accelerator machines for prefill and 64–128-accelerator supernodes for decode, where wide expert parallelism spreads Mixture-of-Experts weights while keeping large effective batches. These are deployment recommendations, not a confirmed map of DeepSeek's own cluster.
The scheduler is where much of the claimed advantage sits. Prefill work is allocated by estimated attention-compute cost rather than raw request length, and chunk sizes vary to hit a time budget instead of being fixed. Because cached prefixes live in shared storage, prefill routing does not need cache affinity; a request can go to whichever worker is free. Decode routing balances both running-request count and accumulated KV length, since request count alone misses memory pressure. Popular experts get replicated and the replicas are periodically redistributed to reduce waiting caused by routing skew, at the cost of extra HBM.
The cache design ties the throughput figures together. Persistent block KV state lives on a separate 3FS cluster with roughly 100 PiB of SSD capacity, retained for three days for agent traffic and seven days for chatbot traffic. Any prefill worker can pull a cached prefix, which decouples cache locality from compute placement and lets storage and compute scale separately. Loads are transferred asynchronously and overlapped with layer execution, and the implementation uses GDR/RDMA transfers rather than the CUDA copy engine, which performs poorly on fine-grained KV block copies.
Sliding-window attention state is a shorter-lived second tier, held in host memory. SWA Bounded Replay rebuilds missing state from only the latest 128 tokens, restricting attention to that segment while reusing the persistent block cache. DeepSeek reports this nearly halves total prefill computation. That is an approximation, so its validity depends on the model architecture.
The headline 155 trillion tokens per day is therefore a cached-inclusive figure, not a count of newly generated tokens. With a 96% input-cache hit rate, most of that volume is cheap re-reads rather than fresh prefill or autoregressive output, which matters when comparing it with raw decode benchmarks. One community analysis puts the long retention tail at only about 0.5–1 percentage point of additional cache hits, which frames the storage-versus-recompute trade-off.
DSpark, the speculative decoding path, predicts draft-token acceptance probabilities and adjusts verification length to maximize expected throughput. DeepSeek reports a 6% production throughput gain over always verifying five draft tokens; that is a comparison against a specific speculative baseline, not against non-speculative decoding. The deepseek-recipe component covers Chat Completions, Responses and Messages APIs, parallelizes tokenization, and distinguishes real control tokens from user text that merely looks like them.
vLLM offered the most direct counterpoint. The team acknowledged that DeepInfer was built largely on vLLM and released under Apache 2.0, then reported decode throughput within 6% of DeepSeek's published results on GB200 under a matched configuration, publishing reproduction recipes and AgentX results alongside it. This suggests the gap to leading open engines may be narrower than the framing implied. The comparison is still self-reported by a competing team and covers decode-stage parity, not the cache fleet, routing or end-to-end latency under DeepSeek's traffic mix.
The economics depend heavily on assumptions. One widely circulated estimate put inference margins at roughly 88–97% using a token mix of about 97% cached input, 2% uncached input and 1% output, then was corrected: the idealized scenario implied about $49 per accelerator-hour against DeepSeek's reported $38, or roughly 80% of benchmark performance, giving a revised range of 84–96%. At $7 per hour against $38, the arithmetic gives about 81.6% before other costs. At $5–10 per hour it spans roughly 74–87%. Latent Space's own cluster math using $5–8 all-in per card-hour lands above 80% margin, about $166B per GW of capacity and an 81-day payback, against DeepSeek's stated 10-month planning target.
Every one of those figures counts revenue, not profit, and omits the SSD cache fleet, CPUs, networking, idle and reserve capacity, failures, engineering and support. They also rest on an unusually cache-heavy workload. High contribution margin on inference can coexist with large losses at the company level.
Why it matters: teams serving LLMs now have DeepSeek's production scheduler and speculative decoding design as a readable public reference, while vLLM's near-parity result means the obvious default remains competitive. The lesson likely worth copying is less the code than the cache hierarchy: long-lived prefix storage separated from compute, and scheduling that tracks compute cost and KV length rather than token counts. Expect infrastructure vendors to treat the $38-per-hour disclosure as a benchmark to argue with, since the margins only hold under a workload with very high cache reuse.
Based on reporting from the original publisher. Visit the source for full context and later updates.
Publisher excerpt
An open source “bomb” drops from China