AI infrastructure optimization is shifting focus from token generation to prefill processing, which handles input context before model output begins. Prefill and decode have different computational needs, leading companies like Lumai to advocate for specialized hardware architectures rather than using the same processors for both tasks. As context lengths grow and agentic workflows increase, prefill efficiency becomes critical to managing power budgets and inference economics in data centers.
Lumai has unveiled an optical computing system for AI inference that uses 3D optical components to achieve 10-100x performance and efficiency gains over traditional silicon accelerators, with 90% power reduction and 10% of typical total cost of ownership while supporting large language models at datacenter scale.