Reflex is a high-performance Rust and CUDA inference engine designed to minimize cold-start latency for running GGUF-based language models on NVIDIA GPUs. It compiles all CUDA kernels ahead-of-time into the binary to eliminate runtime JIT overhead, enabling fast System 1 decision loops with support for token generation and candidate scoring on models like Qwen3 and DeepSeek.
Hugging Face's Transformers library now supports running llama.cpp quantized models (GGUF format) locally on Apple Silicon Macs, making it easier to run AI models on personal machines. The integration reuses llama.cpp's ggml kernels for performance and supports various quantization levels like Q4_K_M to balance model size and quality.
Ternary-Bonsai-8B is a 1.58-bit quantized language model in GGUF Q2_0 format, using ternary weights ({-1, 0, +1}) with shared FP16 scales. The model ranks 2nd among compared 6B-9B parameter models despite being 1/8th their size, with implementation support in a custom llama.cpp fork.