On October 24, 2026, the SGLang Team published its comparison of SGLang and DeepInfer running DeepSeek-V4.1-Flash. It reported prefill throughput at parity or ahead at the tested points, with a peak of 1.04× DeepInfer’s measured throughput, and shorter decode steps at most tested request counts. The results apply to specific NVIDIA GPU configurations and workloads.
For background on the framework’s release, read NeoTeo’s earlier report on DeepInfer’s offline-inference release.
What the reported benchmarks show
Prefill is the stage that processes a prompt; decode generates the response tokens. The SGLang Team reported separate results for prefill on 4× GB300 and 4× B300, and decode on 64× GB300. The team says it ran DeepInfer’s benchmark workloads on the same hardware, using DeepSeek-V4.1-Flash.
| Benchmark phase | Hardware and workload | SGLang result reported against DeepInfer |
| Prefill | 4× GB300; 128K-token context, 4K new tokens per request, and 4K–64K new tokens per GPU per step | Ahead at 4K, 32K, and 64K tokens per GPU per step; at parity at 8K and 16K; up to 1.04× DeepInfer’s measured throughput |
| Prefill | 4× B300; 128K-token context, 4K new tokens per request, and 4K–64K new tokens per GPU per step | Ahead at 4K, 8K, and 32K tokens per GPU per step; at parity at 16K and 64K; up to 1.04× DeepInfer’s measured throughput |
| Decode | 64 GB300 GPUs across 16 nodes; 128K-token context with a shared cached prefix | Steps reported 1.1%–7.4% shorter at 16–256 requests per GPU; at parity at 8 requests per GPU |
Prefill results depend on the test point
For prefill, the SGLang Team reports a lead at some batch sizes and parity at others on both GPU setups. A single peak figure therefore does not describe every tested point: the reported maximum was 1.04× DeepInfer’s measured throughput, while the team reported parity at multiple batch sizes.
The workload used a 128K-token context and 4K new tokens per request. The tested steps ranged from 4K to 64K new tokens per GPU. Both frameworks used the same model weights, according to the SGLang Team.
The team also disclosed differences in how it collected the prefill measurements. DeepInfer measured about nine batches on GPU 0 and repeated the same prompt across requests on each GPU. SGLang averaged complete batches across every GPU while between 20% and 90% of requests finished, and used different numbers of new tokens per request. For each point, the comparison selected DeepInfer’s fastest tested Engram placement. SGLang sharded Engram tables across GPUs and enabled some optimizations that were not on by default.
Decode results used 64 GB300 GPUs
The decode comparison ran across 16 nodes, with 64 GB300 GPUs and DP64/EP64 parallelism. Its workload used 128K-token contexts with a shared cached prefix. The SGLang Team reported parity at 8 requests per GPU and steps 1.1%–7.4% shorter at 16–256 requests per GPU. Its chart converted step time to token throughput using an assumption of 3.25 accepted tokens per step.
The two decode configurations differed as well. SGLang kept Engram tables in GPU memory, while DeepInfer used host memory. DeepInfer’s draft model had 64 redundant experts; SGLang’s had none. Those settings are part of the reported comparison, alongside the step-duration results.
Serving features are a separate comparison
The SGLang Team described its serving stack as including a unified radix KV cache, an optional host-memory HiCache tier, prefill/decode disaggregation, replica load balancing, and an API server. The team characterized those components as absent from the DeepInfer release. These are serving-system features, distinct from the prefill throughput and decode-step measurements.
The reported benchmarks measure throughput and decode-step duration for DeepSeek-V4.1-Flash. They do not compare model sizes or quality, measure energy use or cost per token, or benchmark vLLM.
