NVIDIA mentioned it has achieved a file massive language mannequin (LLM) inference pace, asserting that an NVIDIA DGX B200 node with eight NVIDIA Blackwell GPUs achieved greater than 1,000 tokens per second (TPS) per consumer on the 400-billion-parameter Llama 4 Maverick mannequin.
NVIDIA mentioned the mannequin is the most important and strongest within the Llama 4 assortment and that the pace was independently measured by the AI benchmarking service Artificial Analysis.
NVIDIA added that Blackwell reaches 72,000 TPS/server at their highest throughput configuration.
The corporate mentioned it made software program optimizations utilizing TensorRT-LLM and educated a speculative decoding draft mannequin utilizing EAGLE-3 techniques. Combining these approaches, NVIDIA has achieved a 4x speed-up relative to the perfect prior Blackwell baseline, NVIDIA mentioned.
“The optimizations described under considerably improve efficiency whereas preserving response accuracy,” NVIDIA mentioned in a weblog posted yesterday. “We leveraged FP8 knowledge varieties for GEMMs, Combination of Specialists (MoE), and Consideration operations to cut back the mannequin dimension and make use of the excessive FP8 throughput attainable with Blackwell Tensor Core technology. Accuracy when utilizing the FP8 knowledge format matches that of Artificial Analysis BF16 across many metrics….”Most generative AI utility contexts require a stability of throughput and latency, guaranteeing that many purchasers can concurrently get pleasure from a “adequate” expertise. Nonetheless, for crucial purposes that should make vital choices at pace, minimizing latency for a single consumer turns into paramount. Because the TPS/consumer file exhibits, Blackwell {hardware} is the only option for any process—whether or not you want to maximize throughput, stability throughput and latency, or decrease latency for a single consumer (the main target of this publish).
Under is an summary of the kernel optimizations and fusions (denoted in red-dashed squares) NVIDIA utilized throughout the inference. NVIDIA applied a number of low-latency GEMM kernels, and utilized numerous kernel fusions (like FC13 + SwiGLU, FC_QKV + attn_scaling and AllReduce + RMSnorm) to verify Blackwell excels on the minimal latency situation.
Overview of the kernel optimizations & fusions used for Llama 4 Maverick
NVIDIA optimized the CUDA kernels for GEMMs, MoE, and Consideration operations to attain the perfect efficiency on the Blackwell GPUs.
- Utilized spatial partitioning (also called warp specialization) and designed the GEMM kernels to load knowledge from reminiscence in an environment friendly method to maximise utilization of the large reminiscence bandwidth that the NVIDIA DGX system gives—64TB/s HBM3e bandwidth in whole.
- Shuffled the GEMM weight in a swizzled format to permit higher format when loading the computation consequence from Tensor Memory after the matrix multiplication computations utilizing Blackwell’s fifth-generation Tensor Cores.
- Optimized the efficiency of the eye kernels by dividing the computations alongside the sequence size dimension of the Okay and V tensors, permitting computations to run in parallel throughout a number of CUDA thread blocks. As well as, NVIDIA utilized distributed shared memory to effectively scale back outcomes throughout the thread blocks in the identical thread block cluster with out the necessity to entry the worldwide reminiscence.
The rest of the weblog will be found here.
