Native High-Throughput Serving & Continuous Batching Acceleration Engine for Dense LLMs
Publisher & Authors: The Floria Authors | Floria Systems Research
CONVENTIONAL SERVING (vLLM / TGI) FLORIA SERVING ARCHITECTURE
+--------------------------------------+ +--------------------------------------+
| Host CPU (Python GIL Contention) | | Host CPU (Zero-Latency Dispatcher) |
| - Python Scheduler: ~1 500 us/step | | - Native Hardware Core: 8.5 us/step |
| - Heavy DRAM Pointer Chasing | | - Low-Latency Native Scheduling |
+------------------+-------------------+ +------------------+-------------------+
| |
(PCIe Transfer Stall) (Zero-Bubble Host Queue)
v v
+--------------------------------------+ +--------------------------------------+
| GPU Accelerator (25-40% Idle Wait) | | GPU Accelerator (100% Saturation) |
| - GPU executes in 4.5 ms | | - GPU executes in 3.9 ms |
| - GPU STALLS 1.5 ms per step | | - Next step pre-queued in hardware |
| - Real step cycle: 6.0 ms | | - Real step cycle: 3.9 ms |
+--------------------------------------+ +--------------------------------------+
./scripts/run_audit.shExecutes the standalone, zero-dependency binary bin/floria-serving-audit (489 KB). Verifies 7/7 formal invariants in < 2s: FP64 ground truth concordance, zero GPU starvation over 1 200 requests, 100% recycled KV blocks (16 384 / 16 384 blocks), and sub-microsecond dispatch.
# Standard benchmark (batch 64, prompt 64, 180 steps)
./scripts/run_bench.sh 64 64 180
# Multi-core NUMA sharded benchmark
./scripts/run_bench.sh --sharded --shards-range 1,2,4,8,16 -c 256,512,1024,2048,4096# Direct editable install with zero external dependencies
pip install -e .# Continuous batching quickstart (64 streams)
python examples/01_quickstart_batching.py
# Shared-nothing NUMA-aware multi-sharded serving
python examples/02_numa_sharded_serving.py
# Prompt-lookup speculative decoding acceleration
python examples/03_speculative_batching.py#include "floria_serving.h"
int main(void) {
FloriaServingEngine* engine = cortex_serving_init(512);
int32_t slot = cortex_serving_add_request(engine, 1, 64, 32);
uint32_t tokens[256];
size_t count = cortex_serving_step(engine, tokens, 256);
cortex_serving_free(engine);
return 0;
}In conventional serving stacks (vLLM, TGI, Triton), cloud accelerators spend 25% to 40% of their operational runtime completely idle, bottlenecked by Python GIL contention and host scheduling latency:
- Conventional Reality (vLLM): The GPU finishes compute in 4.5 ms, then stalls for 1.5 ms while the Python host scheduler resolves KV-cache block tables. Real step cycle: 6.0 ms (25% compute wasted).
- Floria Hardware Reality: The GPU executes in 3.9 ms, the host CPU arbitrates in 0.0085 ms, and the GPU queues the next step instantly. Real step cycle: 3.9 ms (0% compute wasted).
- Elimination of Host GPU Starvation: The CPU host dispatch plane sustains 5.61 Million request steps/sec across 8 192 concurrent streams, guaranteeing 100% GPU compute duty cycle without host bottlenecks.
- Elimination of Host CPU Over-Provisioning: Expensive 64-to-128-vCPU host instances babysitting GPUs are obsolete; a single pinned vCPU core easily delivers 191k tokens/sec.
- Zero-Jitter Real-Time SLAs: The P99/P50 jitter ratio is locked at 1.04 under full saturation, eliminating latency tail spikes in multi-tenant production.
- 100% Duty-Cycle Hardware: Every megawatt drawn by the datacenter goes directly into tensor compute, not idle waiting states.
- Zero-Overhead Hardware Dispatcher: Request arbitration resolved at native silicon speed, eliminating host scheduling latency without requiring dedicated ASICs or hardware modifications.
- Fused Epilogue Pipeline: Attention computation and normalization fused directly on-chip (0 DRAM round-trips).
- Shared-Nothing NUMA Sharding: Physical core pinning avoiding cross-socket bus contention, holding P99/P50 jitter at 1.04.
- WORLD_RANKING.md — Comprehensive comparative leaderboard across 1-vCPU scheduling, NUMA sharding, Llama-3.1 70B attention, and vision.
- MLPERF_DENSE_INFERENCE_REPORT.md — Formal metrology report, in-situ physical telemetry, MLCommons compliance, and Z3 SMT UNSAT proofs.
- PATENT_NOTICE.md — Constructive patent disclosure and protected claims under Canadian Patent Application CIPO CA 3,322,620.
Distributed under the Floria Community License (LICENSE.md):
- Free for Personal, Academic & Evaluation Use: Unrestricted benchmarking, research, and auditing. Zero telemetry, zero external runtimes.
- Commercial Exploitation Reserved: Production deployments, SaaS hosting, or hyperscaler cluster distribution require an enterprise licensing agreement.
- Patent Protection: Canadian Patent Application CIPO CA 3,322,620 (Priority Date: August 25, 2026). See PATENT_NOTICE.md.
- Contact & Secretariat: cortex.lab1@outlook.com(The Floria Authors).