DeepSeek opens DeepInfer for offline inference on SM100 and Ascend 950

DeepSeek announced on October 20, 2026, that it was open-sourcing DeepInfer, an inference framework for DeepSeek-V4.1-Flash. The New Stack reported that the release itself took place on October 19. The public version supports offline inference on NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs, rather than providing a complete online serving service. DeepInfer is released under the Apache License 2.0, and its official repository lists the project’s code and setup requirements.

What DeepSeek released

DeepInfer’s initial public release lists DeepSeek-V4.1-Flash as its supported model. DeepSeek also released deepseek-recipe, a companion library that converts between common large-language-model API protocols and model token sequences.

The framework includes methods for attention and expert parallelism, asynchronous scheduling, structured outputs, and speculative decoding. DeepSeek identifies vLLM as its primary foundation, with SGLang and TensorRT-LLM as additional inspirations.

What offline inference includes and a full service needs

The public release runs inference offline. A complete online service also requires KV-cache management, prefill/decode disaggregation, and load balancing across data-parallel replicas, which DeepSeek lists as additional components. The KV cache stores attention values from earlier tokens so they can be reused during generation; prefill processes the prompt, while decode generates the response tokens.

Supported accelerators and build requirements

DeepSeek lists two accelerator platforms, each with its own software backend:

Both builds require Linux, GCC 13 or later, and Python 3.12. The repository marks batch-invariant inference and deterministic sampling as NVIDIA-only. Its published benchmark setup uses four SM100 GPUs for prefill and 64 for decode, with each setup within one scale-up domain.

DSpark estimates vary with concurrency

DSpark is DeepSeek’s confidence-scheduled speculative decoding method. It predicts how many draft tokens are likely to be accepted, then chooses how many to verify. In DeepSeek’s EP64/DP64 estimates, dynamic verification is compared with fixed-five verification, which checks five draft tokens at a time.

DeepSeek describes about 60 concurrent sequences per rank as its production operating point. Its estimate at that level is a 5.4% throughput gain; at 16 sequences per rank, the estimate is a 1.5% decrease.

DeepSeek’s efficient deployment uses 64 prefill and 64 decode GPUs

DeepSeek says a setup with one prefill GPU and one decode GPU is deployable. It describes 64 prefill GPUs plus 64 decode GPUs as the smallest efficient deployment under its recommended decode configuration. That distinction gives developers a sense of the scale DeepSeek associates with efficient operation, beyond a setup it says can be deployed.

DeepSeek’s reported operating figures

In its deployment Q&A, DeepSeek says one cluster serves 155 trillion tokens per day and reports overall revenue of US$38 per GPU-hour at “valley pricing.” Those figures describe DeepSeek’s own operation and pricing condition. The deployment Q&A also contains the company’s DSpark method and workload estimates.