MiMo-V3 introduces HySparse2, a new architecture designed for agentic inference that reduces prefill FLOPs by 5× and KV cache size by 4.5× compared to MiMo-V2.6 while improving long-context retrieval. HySparse2 uses two levels of KV sharing—KV Bridging and KV Reuse—along with token-level selection and a unified cache for local and global tokens to optimize the workload of processing short actions followed by long observations.
HySparse2 is a hybrid sparse attention architecture designed for long-context language models that improves efficiency through two-level KV sharing between self-decoder and cross-decoder components. It replaces block-level sparsity with token-level sparsity and enables prefill computation to exit early, reducing computational cost and KV-cache storage while maintaining performance on long-context retrieval and multi-turn agent tasks.
This article explains Transformer inference, contrasting it with training by introducing latency as a key consideration. It describes how naive token sampling is computationally expensive (O(n²) to O(n³)), but can be optimized using a KV cache to reduce complexity to O(n) to O(n²), enabling efficient sequence generation through separate forward passes for each token.
AI infrastructure optimization is shifting focus from token generation to prefill processing, which handles input context before model output begins. Prefill and decode have different computational needs, leading companies like Lumai to advocate for specialized hardware architectures rather than using the same processors for both tasks. As context lengths grow and agentic workflows increase, prefill efficiency becomes critical to managing power budgets and inference economics in data centers.
A system successfully prefills a 284B parameter DeepSeek-V4-Flash model on NVIDIA DGX hardware and decodes it on Apple Silicon Mac Studio over standard 10GbE ethernet, achieving 1.5x to 3.7x speedup on prompts up to 241K tokens by computing the decoder's cache on the prefill machine rather than transferring incompatible KV cache formats.
Deep Seek v4.1 M5 Max achieves 17 tokens/s by optimizing mixture-of-experts model execution from SSDs on a 128GB laptop. The system reads only routed experts (187 of 384 per layer) instead of all experts, achieving 1.73× speedup on prefill; adding multiple drives reduces read latency rather than increasing bandwidth, with time-to-first-token improving from 31.5s (baseline) to 18.3s (one drive) to 11.7s (three drives).