AivexaNewsSearch
AI news for builders and product teamsChecked every hour

[AINews] DeepSeek DeepInfer: Open Source SOTA Inference running 155T toks/day/cluster making $22m/day at ~80% margin for a 81 day payback

Collected Oct 21, 2026

DeepSeek released DeepInfer, the inference engine that serves its V4.1 Flash model, on the opening day of PyTorch Conf. The company published the engine under Apache 2.0 along with a production engineering report, and said future model inference work and optimizations will also land in the same repository. The timing put the release directly into conversation with the conference and with OpenAI's recently announced 12-hour API prompt caching; DeepSeek's stack advertises prompt caching lasting up to 72 hours.

DeepInfer is a customization of vLLM, with credit given to SGLang and TRT-LLM, and it carries a substantial amount of DeepSeek-specific engineering on top of the base engine. It supports distributed inference, speculative decoding, a technique called SWA Bounded Replay, and structured output on both NVIDIA and Huawei Ascend hardware. A companion component, deepseek-recipe, handles production token encoding and decoding and implements processing compatible with the official API, supporting Chat Completions, Responses and Messages interfaces. It also parallelizes tokenization and separates genuine control tokens from identical-looking strings supplied as user text, so literal user content is not accidentally parsed as a structural delimiter.

The headline production numbers are large. DeepSeek reports 155 trillion tokens per day from a single cluster, a 96% input-cache hit rate, an average time to first token of 400 milliseconds, and generation at 230 tokens per second, rising to 300 tokens per second for coding. It also reports $38 in revenue per accelerator-hour at off-peak prices, against current API pricing of $0.15 per million input tokens and $0.60 per million output tokens. These are first-party production claims, not independently audited measurements, and the throughput figure counts cache hits as served tokens rather than newly generated ones. That distinction is central: with a 96% input-cache hit rate, the mix of billed tokens is heavily weighted toward cheap cached reads, and newly computed prefill plus autoregressive output carry very different costs.

The open sourcing has boundaries. DeepSeek describes DeepInfer as fully open source, but its KV-cache management and load-balancing components are not fully released because they remain coupled to internal infrastructure. The earlier-open-sourced 3FS storage system covers part of that gap. In practice, the engine is available while important service-level orchestration remains a deployment responsibility, so a team cannot reproduce DeepSeek's full production service directly from the repository.

The disclosed architecture separates vision encoding, prefill and decode so each stage scales against its own bottleneck. Recommended deployment shapes are ordinary eight-accelerator machines for prefill and 64 to 128-accelerator supernodes for decode, where wide expert parallelism spreads mixture-of-experts weights across devices while keeping large effective batches. These are recommendations, not a description of DeepSeek's exact production hardware.

Scheduling is where much of the difference from a conventional engine lives. Prefill scheduling estimates attention-compute cost rather than using raw request length, and varies chunk sizes against a time budget instead of fixing one chunk size across heterogeneous requests. Because cached prefixes are pooled in shared storage, prefill routing does not need cache affinity: a request can go to any available worker rather than the one that previously served its prefix, and data-parallel groups balance on computed tokens. Decode routing balances both the number of running requests and accumulated KV-cache length, since request count alone misses memory pressure. Expert-parallel load balancing periodically redistributes replicas of popular experts to reduce inter-device waiting, at the cost of extra HBM consumption, with diminishing returns as replication grows. Host-side scheduling is asynchronous so decode steps do not leave idle gaps.

One notable observation is the apparent absence of tensor parallelism in the disclosed Blackwell configuration, which commentators contrast with default vLLM and SGLang recipes. That reads as an interpretation of the configuration rather than evidence that tensor parallelism is generally worse. Practitioner commentary also argues that wide expert-parallel prefill needs enough traffic and workers to pay off, and that small-to-medium deployments with only a few prefill workers are better served by pipeline parallelism.

The cache hierarchy ties the throughput, the agent workloads and the revenue figure together. Persistent block KV lives in a separate 3FS storage cluster with roughly 100 PiB of SSD capacity, retained for three days for agent traffic and seven days for chatbot traffic. Any prefill worker can retrieve a cached prefix, decoupling cache locality from compute placement so storage and compute scale independently. Host-to-device transfers overlap layer execution, and the described path uses GDR/RDMA-based transfers rather than the CUDA copy engine, which performs poorly on fine-grained KV block copies. Sliding-window attention state stays in host memory with a shorter lifetime, and SWA Bounded Replay reconstructs missing short-lived state from only the latest 128 tokens, reusing the persistent block cache to produce an approximation rather than a full rebuild. DeepSeek reports this nearly halves total prefill computation. Community commentary puts the benefit of the long retention tail at roughly 0.5 to 1 percentage point of extra cache hits, illustrating a storage-versus-recomputation trade-off.

DSpark, DeepSeek's speculative decoding scheme, predicts draft-token acceptance probabilities and adjusts verification length to maximize expected throughput. DeepSeek reports a 6% production throughput gain over always verifying five draft tokens, which is an improvement over that specific baseline, not over non-speculative decoding.

vLLM offered the most concrete counterweight. After acknowledging that DeepInfer was initially built mainly on vLLM, the vLLM team reported decode throughput within 6% of DeepSeek's published results on GB200 under a matched configuration, and published AgentX results and reproduction recipes. That result is decode-stage parity, not a reproduction of the whole service: it does not establish equivalent 100 PiB cache operations, retention economics, routing or end-to-end latency under DeepSeek's traffic distribution. The comparison is also self-reported by a competing engine team, even if its recipes improve reproducibility. vLLM credited DeepSeek as an early adopter and contributor, which makes DeepInfer better understood as production learnings flowing back into an existing ecosystem than as a clean replacement for it.

On economics, accelerator-hour revenue is not profit, and accelerators are only part of a serving cost base. One widely shared estimate assumed a token mix of 97% cached input, 2% uncached input and 1% output, with 80K uncached input tokens per second per GPU and 16K output tokens per second per GPU, and hardware at $3 to $6 per GPU-hour. That model implied roughly $49 per accelerator-hour, about 80% of benchmark performance versus the reported $38; the corrected margin range quoted was 84.2% to 96.1%. At $7 per hour against $38 the arithmetic gives about 81.6% before other costs, though DeepSeek did not disclose which GPU underlies the revenue figure. At $5 to $10 per hour the same accelerator-only calculation spans roughly 73.7% to 86.8%. All of these calculations leave out or only partly allocate the SSD cache fleet, CPUs, networking, idle and reserve capacity, failures, engineering and support, and they depend on a heavily cache-favored token mix.

A community explanation attributes the efficiency to model and runtime co-design: a smaller prefill backbone activating roughly 8B parameters per token versus 16B at decode, wide expert parallelism, highly compressed KV, and speculative decoding, alongside figures such as 552B expert parameters and mostly four-bit weights. Those are secondary claims that cannot be independently reconstructed from the released material, and they underline that these gains are model-specific rather than automatically transferable to arbitrary checkpoints.

Reactions split accordingly. Some practitioners rate the engineering write-up as more valuable than the code for experienced inference developers, partly because key components are absent. Others see value in a per-model repository and kernel strategy, and suggest adapting the implementation to other models, while a proposed path of 3FS plus possibly Dynamo has been estimated to need roughly 3,000 to 4,000 GB200s to match cluster-scale throughput; that is a suggested deployment, not a demonstrated reproduction.

Why it matters: Teams building agent products with long, repeatedly resent contexts now have a detailed public reference for serving them, from persistent prefix caching to decoupled prefill and decode, and vLLM's within-6% decode result suggests parts of that efficiency are reachable on existing open-source engines. Inference providers should treat the $38 per accelerator-hour and the margin figures as revenue-level disclosures whose profitability depends on cache-heavy traffic and undisclosed cost assumptions. The practical near-term change is that prompt caching measured in days, rather than minutes or hours, becomes a realistic design target and a competitive pressure point across API vendors.

Read at Latent Space

Based on reporting from the original publisher. Visit the source for full context and later updates.

Publisher excerpt

An open source “bomb” drops from China