Floria is a high-throughput serving engine for large language models that eliminates GPU starvation by replacing Python-based scheduling with native hardware dispatching. Running on a single vCPU, it achieves 191k tokens/sec and 100% GPU utilization, compared to conventional systems like vLLM that leave GPUs idle 25-40% of the time due to host scheduling latency.
Halo is a new post-training framework for open-source models that achieves 2.8x higher throughput than TRL while using less peak memory and maintaining HuggingFace compatibility.
This article explains LLM inference optimization techniques for faster, cheaper production deployments. It covers the two-phase inference process (prefill and decode), memory management strategies like KV caching and PagedAttention, and methods such as model compression and speculative decoding to reduce cost and improve throughput.
A system-one model called Jev significantly improves entity resolution in high-throughput data pipelines, reducing costs by 99.56% and increasing throughput 7.35× while maintaining near-baseline accuracy when integrated into multi-stage review workflows. The authors demonstrate this on Ohio campaign finance data, where Jev resolves entities across donors, committees, and companies by adjudicating bundles of equivalent evidence.
ShapeLearn released optimized GGUF quantizations of Qwen 3.8 27B, with full models outperforming their earlier Lite versions. GPU-5 is recommended for best quality-speed tradeoff at 99.63% of BF16 performance, while GPU-4 offers competitive results in smaller 11.0 GB size. Speculative decoding with MTP or DFlash2 further improves throughput across all tested GPUs.
Union Alpha is Unbiased's Pareto, a multimodal composite model hosted by OpenRouter with 262,144 token context window, priced at $2.50/M input and $7.50/M output tokens. The model delivers 34 tokens/second throughput and 4.52 seconds latency, supporting function calling and JSON output for research, coding, and agentic workflows.
A systems characterization study examines executing the ~375 GB Kimi K2.5 model on a single 128 GB AMD Ryzen AI MAX+ 395 PC using storage-backed expert caching. The bounded locality cache achieved 7.7% hit rate, reducing expert traffic by ~70 GB and decreasing package energy from 117.50 J to 113.47 J per generated token while maintaining ~0.438 tokens/s throughput.
VoltDB is a distributed database that optimizes performance by partitioning data and stored procedures across multiple sites, enabling parallel query execution without traditional locking overhead. Transactions are serialized within partitions for consistency, while multi-partition operations use coordination to maintain throughput and integrity.
Dense and Mixture-of-Experts (MoE) model architectures differ fundamentally in parameter activation: dense models activate all parameters for every token, while MoE models route each token through only a subset of expert networks. The choice between them depends on deployment constraints like throughput, memory cost, and serving complexity rather than raw parameter count alone.
Rivvr automates vLLM deployment at production scale by automatically tuning configurations, monitoring metrics, and adjusting cluster topology to maintain latency and throughput SLO targets while reducing AWS costs by 40-70%.