01 / Chutes · One year of requests
One Year in LLM Serving
A year of production request traces metadata from Chutes, covering model usage, token counts, and prefix-cache reuse.
- 6.12Brequests
- 9,174models
- 1 yearof traffic
Research uses
- Benchmark batching and scheduling with recorded request arrivals and input/output token lengths.
- Reconstruct user sessions from request metadata to analyze usage patterns and behavior.
- Explore GPU colocation for low-traffic models using their busy and idle periods.
user_id values rotate every three months, so one person appears under different IDs across the year.
Findings
-
01
Requests exhibit high temporal locality. 99% of repeat requests from the same user to the same model arrive within 15 minutes.
-
02
LRU often matches or outperforms more complex cache eviction policies.
-
03
Cache-aware routing raises the token hit ratio, with 5–7% load imbalance in simulation.