Floria is a high-throughput serving engine for large language models that eliminates GPU starvation by replacing Python-based scheduling with native hardware dispatching. Running on a single vCPU, it achieves 191k tokens/sec and 100% GPU utilization, compared to conventional systems like vLLM that leave GPUs idle 25-40% of the time due to host scheduling latency.
Halo is a new post-training framework for open-source models that achieves 2.8x higher throughput than TRL while using less peak memory and maintaining HuggingFace compatibility.