A high-performance LLM serving scheduler written in Rust achieves 191k tokens/second on a single vCPU using continuous batching, paged KV cache allocation, and lock-free scheduling. The repository includes formal SMT Z3 mathematical verification, an independent anti-cheat auditor, and a reproducible benchmark harness under strict hardware isolation.
A developer transitioning from B2B SaaS to inference engineering outlines the core technical stack: coordinating work across GPU model replicas using tools like Nvidia Dynamo or llm-d, running models with inference engines like vLLM or SGLang, and reusing cached computations through prefix caching services. The role emphasizes systems engineering—stitching together existing APIs and services rather than building net-new code.
Broadway is an Elixir library for building data processing pipelines. It uses producers to consume from data sources, processors to handle individual messages, batchers to group messages, and batch processors to work on batches, with optional partitioning for message ordering.