Self-taught from zero to disaggregated DeepSeek deployment
Contracted R&D for a high-traffic streaming chat product. No prior background in transformers,
KV cache, or GPU serving. I learned the stack from scratch, rented GPUs, and open-sourced the runbooks.
Problem
What had to be solved
Self-hosted DeepSeek-class inference for real chat traffic: long prompts, heavy prefix reuse,
streaming API, TTFT matters more than end-to-end latency.
~80%Prefix cache hits
~7.9kInput tokens (P50)
~60Peak QPS target
~2sProd TTFT P50 ballpark
What I tried
Eight setups, in the order I actually ran them
Chat traffic reuses the same system prompt about 80% of the time. The question was how to stop
recomputing that prefix on every request. I started on the production-class model, burned money,
then dropped to a smaller model and compared three ways of moving the KV cache between GPUs.
Shorthand: 1P2D means 1 GPU does prefill (read the prompt) and 2 GPUs do decode
(stream tokens). LMCache MP is a shared KV store in host RAM that every GPU talks to,
instead of handing cache privately from one GPU to another.
What worked
Shared L1 via LMCache MP + ZMQ
One central LMCache Multiprocess server (ZMQ port 6000) as a shared L1 KV layer in host RAM.
Every prefiller and decoder uses LMCacheMPConnector against the same store.
Prefiller writes prefix KV once; later requests with the same prefix skip full prefill.
Results
Asked vs delivered
Best V2-Lite stack: 1 prefiller + 2 decoders on 3× A100, shared LMCache store.
Production wanted ~2s time-to-first-token and ~60 req/s. Here is what that stack actually did.
0.52sTTFT P50 when asked for 10 req/svs ~2s production ballpark
8.5 / 10Req/s delivered vs askedSustained at the headline point
~11 / 60Ceiling vs peak targetSaturates well below 60 QPS
Measured on DeepSeek V2-Lite (3× A100), not a production-class model.
On V3.2 NVFP4 (8× B200) the same architecture held sub-second TTFT only to ~2 req/s.
This proves a shared L1 KV layer is the right shape for prefix-heavy chat, not production capacity.
Could it keep up?
Gold: load we requested. Blue: load the stack actually served. After 10 they split:
served stalls near 11.* That is this box’s ceiling, not 60.
How long until the first token?
Wait time in seconds, not speed. Lower is better. Green: typical request (median).
Red: slow tail. Dashed lines: production (~2s / ~3.5s). At 10 requested, median is 0.52s.
* One 1P2D box (3 GPUs). ~11 req/s is that hardware ceiling, not a failed design.
More QPS means more replicas at about 1:2 to 1:3 prefill:decode, e.g. 3 prefiller + 6 decoder GPUs.
Cost
Total GPU spend
~$1,205RunPod + Vast.ai (spot rentals)
Insights
What I would do again
Small firstV2-Lite on cheap A100 before V3.2 on B200
Move fastExperiments per week beat perfect plans
Shrink itOne request, one role, one path, then scale up
Technical depth
Everything else is in the repo
Full write-ups, sweep tables, per-phase runbooks, and the resource bibliography.