NVIDIA reports a 42.5% gain in DGX Spark MoE decoding

On October 27, 2026, NVIDIA published results reporting 42.5% higher decode throughput for Ling-3.0-flash-fp4 on DGX Spark across 1,319 GSM8K questions at batch size one. Throughput rose from 37.81 to 53.89 tokens per second with three optimizations enabled. The result applies to that model and evaluation, not to every MoE workload.

What NVIDIA measured on DGX Spark

NVIDIA evaluated all 1,319 GSM8K questions using five-shot prompts, temperature 0.6, top-p 0.95, top-k 29, chain-of-thought enabled, and a maximum of 32,768 generated tokens. The baseline used its W4A8 configuration, FlashInfer MXFP4 MoE kernels, NVIDIA CUTLASS dense GEMM, and the --enable-fp32-lm-head option.

The tested model, Ling-3.0-flash-fp4, has 124 billion total parameters and 5.1 billion active parameters per token, according to NVIDIA. In a mixture-of-experts (MoE) model, a token activates selected experts rather than all of them. NVIDIA says each MoE layer routes a token to eight of 512 routed experts, plus one shared expert.

The throughput gain was accompanied by lower mean latency: NVIDIA reported 15.21 seconds at baseline and 11.49 seconds with all three optimizations. Mean time to first token, the wait before generation begins, was 0.207 seconds at baseline and 0.214 seconds with the optimizations. Those measurements describe different parts of the response experience.

Three changes to the decoding path

The optimizations target different parts of token generation. Humming replaces the baseline MoE kernel backend with Humming MXFP4 kernels. NVIDIA says Humming uses NVRTC, a runtime compiler, to build kernels for the relevant dimensions, data types, and GPU architecture, then caches them in ~/.humming/cache.

The second change applies online FP8 quantization to the language-model head, the layer that produces next-token scores. NVIDIA says the LM head’s weights are stored in BF16 and that online FP8 halves the weight-read volume for this layer relative to BF16. Its SGLang setting is SGLANG_ENABLE_FP8_LM_HEAD=1, which is mutually exclusive with the baseline’s --enable-fp32-lm-head option.

The third change uses NEXTN for multi-token prediction (MTP), a form of speculative decoding in which draft tokens are checked together. NVIDIA’s reported setup uses three draft steps and a four-token verification window. The combined configuration therefore changes both the MoE kernels and the work involved in generating and checking tokens.

Why the 185.1% synthetic result is different

NVIDIA reported a separate 185.1% throughput increase on a synthetic batch-one workload with 32K input tokens and 2K output tokens. That test used greedy decoding and fixed output length. Its baseline was 36.09 tokens per second; all three optimizations reached 102.88 tokens per second.

The synthetic run accepted an average of 3.90 draft tokens per MTP round, compared with 2.68 on GSM8K. NVIDIA cautions that repetitive output and greedy decoding can overstate speculative-decoding gains on other workloads. The two percentages describe different tests, not competing estimates of one result.

The distinction between prompt processing and token decoding also matters when judging inference speed. A Spanish-language video by Codigo gris explains that prompt processing can be compute-bound, while token generation can be limited by memory bandwidth because the model’s weights must be read repeatedly. The NVIDIA figures here measure decoding throughput, so they answer a different question from a prompt-processing measurement.

The FP8 and MTP integration requirement

When online FP8 quantization and MTP are combined, NVIDIA says the draft model must share the complete LM-head module with the target model. Sharing only its weights can leave the draft interpreting FP8 weights as BF16. NVIDIA reports that this mismatch can silently trigger a fallback, lower MTP’s accept length to 1.00, and push throughput below the unoptimized baseline.