VC-Attention: Faster Low-Bit Attention Without Retraining

VC-Attention: Faster Low-Bit Attention Without Retraining

Attention speedup over BF16 FlashAttention-4 [1] on B200 and B300. We benchmark the attention workload in MiniMax-H3 when generating 243 frames at 1344×768. Attn-QAT [3] uses QK4 · PV8; VC-Attention uses 8-bit attention with ExpCast-FP8; the B200 bar also runs V-Smooth, time-weighted by the deployed schedule (on for the first quarter of the denoising steps, off for the rest). Nunchux Attention is our proprietary extension of VC-Attention.

Show as a table

FlashAttention-4 · BF16

1.00× attention speedup

SageAttention2 · 8-bit

VC-Attention · 8-bit

1.59× attention speedup

FlashAttention-4 · BF16

1.00× attention speedup

SageAttention2 · 8-bit

VC-Attention · 8-bit

1.59× attention speedup

Two MiniMax-H3 examples generated on an NVIDIA B200 at 1344×768, 243 frames. Speedups are for the attention kernel relative to BF16 FlashAttention-4. Across 100 prompts, VC-Attention achieves 20.2 dB mean PSNR against the BF16 reference, compared with 19.9 dB for SageAttention2. Higher PSNR means the output is closer to the BF16 reference.

Nunchux is building the frontier of multimodal inference: the fastest, cheapest, and highest quality inference for image, video, and world models. For video, the operator that matters most is attention.

The 10 second clips above were generated with MiniMax-H3. On one B200 with BF16 FlashAttention-4, about two thirds of every denoising step is attention. Attention cost grows with the square of the token count, so longer clips make it worse. Low-bit attention offers a path to faster video generation, but on B200 and B300, realizing those gains while preserving fidelity requires addressing both quantization error and the softmax bottleneck.

Today we introduce VC-Attention, a faster and more accurate training-free low-bit attention. VC-Attention combines two innovations, V-Smooth and ExpCast-FP8. V-Smooth reduces the quantization error; ExpCast-FP8 removes the softmax bottleneck. On B200, VC-Attention runs the attention kernel 1.6× faster than BF16 FlashAttention-4 [1] on MiniMax-H3, and it is more faithful to the BF16 output than SageAttention2 [2].

V-Smooth

Existing methods already keep the query and key product accurate at low precision, for example with a Hadamard transform that spreads outliers across channels. With that in place, value quantization becomes a major source of error. V-Smooth groups the value tokens with a lightweight k-means so each hardware block holds similar tokens, then subtracts the block mean and quantizes only the residual. Across the four models in the report [4], this alone adds 1.1 to 2.8 dB of PSNR over SageAttention2.

ExpCast-FP8

Low-bit compute speeds up the two matrix products, but the softmax between them still runs in high precision and becomes the longest stage of the kernel. ExpCast-FP8 replaces that stage with a linear approximation that maps each score directly to its FP8 probability code.

Given the log-domain score , the standard path evaluates in FP32 and then casts to E4M3. An E4M3 byte stores roughly , so ExpCast-FP8 computes the byte directly with one linear map, , rounded to an integer.

From VC-Attention to Nunchux Attention

Nunchux Attention is our proprietary extension of VC-Attention. It adds algorithm and kernel optimizations developed for our inference stack. On the MiniMax-H3 attention workload, it runs 1.91× faster than BF16 FlashAttention-4 on B200 and 1.83× faster on B300.

What this makes possible

VC-Attention speeds up attention in existing video models without retraining. It can be combined with sparse attention [5,6,7], few-step distillation, and multi-GPU execution to further reduce the cost of video generation. Together, these improvements make higher-resolution and longer videos more practical to generate.

Read the report: VC-Attention: Value Smoothing and Softmax Casting for Low-bit Attention.

Free access to MiniMax-H3 arrives very soon. Join the Nunchux Modelverse waiting list today to be among the first to use it.

If you run visual models at scale, contact sales. We are hiring. Please visit our careers page for more details.

References

- [1]Ted Zadouri, Markus Hoehnerbach, Jay Shah, Vijay Thakkar, and Tri Dao. FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling. MLSys 2026.

- [2]Jintao Zhang, Haofeng Huang, Pengle Zhang, Jia Wei, Jun Zhu, and Jianfei Chen. SageAttention2: Efficient Attention with Thorough Outlier Smoothing and Per-thread INT4 Quantization. ICML 2025.

- [3]Peiyuan Zhang, Matthew Noto, Wenxuan Tan, Chengquan Jiang, Will Lin, Wei Zhou, and Hao Zhang. Attn-QAT: 4-Bit Attention with Quantization-Aware Training. arXiv:2603.00040, 2026.

- [4]Xingyang Li, Dongyun Zou, Shining Zhang, Jiacheng Chen, Haocheng Xi, Lvmin Zhang, Jun-Yan Zhu, Song Han, Zhekai Zhang, Yujun Lin, and Muyang Li. VC-Attention: Value Smoothing and Softmax Casting for Low-bit Attention. arXiv:2609.15810, 2026.

- [5]Haocheng Xi, Shuo Yang, Yilong Zhao, Chenfeng Xu, Muyang Li, Xiuyu Li, Yujun Lin, Han Cai, Jintao Zhang, Dacheng Li, Jianfei Chen, Ion Stoica, Kurt Keutzer, and Song Han. Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity. ICML 2025.

- [6]Xingyang Li, Muyang Li, Tianle Cai, Haocheng Xi, Shuo Yang, Yujun Lin, Lvmin Zhang, Songlin Yang, Jinbo Hu, Kelly Peng, Maneesh Agrawala, Ion Stoica, Kurt Keutzer, and Song Han. Radial Attention: O(n log n) Sparse Attention with Energy Decay for Long Video Generation. NeurIPS 2025.

- [7]Jintao Zhang, Chendong Xiang, Haofeng Huang, Jia Wei, Haocheng Xi, Jun Zhu, and Jianfei Chen. SpargeAttn: Accurate Sparse Attention Accelerating Any Model Inference. ICML 2025.