SGLang and Miles add day-0 support for DeepSeek-V4.1, a model featuring low-ratio compression, sliding-window attention, manifold hyper-connections, and Engram memory for efficient serving. The implementation includes cross-layer sharing, sparse retrieval mechanisms, and host-memory placement optimizations that increase KV cache capacity by 36% while maintaining comparable throughput.
A researcher replayed real Claude Code and Mooncake requests through a prefix-cache simulator to test whether alternative eviction policies could beat LRU, but failed across three approaches. The analysis reveals that under capacity pressure, most recomputation stems from tool-calling loops seconds apart rather than idle sessions, and the production LRU baseline proves surprisingly difficult to improve upon.
A developer transitioning from B2B SaaS to inference engineering outlines the core technical stack: coordinating work across GPU model replicas using tools like Nvidia Dynamo or llm-d, running models with inference engines like vLLM or SGLang, and reusing cached computations through prefix caching services. The role emphasizes systems engineering—stitching together existing APIs and services rather than building net-new code.