A high-performance LLM serving scheduler written in Rust achieves 191k tokens/second on a single vCPU using continuous batching, paged KV cache allocation, and lock-free scheduling. The repository includes formal SMT Z3 mathematical verification, an independent anti-cheat auditor, and a reproducible benchmark harness under strict hardware isolation.
Cohere introduces a serving engine for North Mini Code using a decode megakernel, achieving 1.25–1.41× speedup over vLLM on H100 GPUs by consolidating multiple kernel launches into a single persistent kernel that runs the entire forward pass, eliminating GPU idle time during autoregressive decoding.