- 01Production SGLang changesWe select real inference-system work from release notes, roadmap issues, and maintainer suggestions.

- 02Build the taskBenchmark authors, assisted by coding agents, prepare the instruction, a reproducible environment, a reference solution, and tests that check the requested behavior.

- 03Qualification runsOn the task's assigned hardware, the no-op must fail every new-behavior test while passing the regression tests; the oracle must pass every test. no-op: no new tests passoracle: all tests pass

- 04Review and decisionAgent-assisted adversarial probing challenges the instruction and tests; human reviewers inspect the qualification results and make the final inclusion decision.

SWE-Serve

Loading benchmark results…

How SGLang work becomes benchmark tasks

SWE-Serve turns recent SGLang engineering work into coding tasks. Before including a task, we run its tests against both the unchanged repository (no-op) and the frozen reference solution (oracle). The no-op must fail every new-behavior test while passing the regression tests, and the oracle must pass every test. Human reviewers inspect the instruction, tests, and results before including the task.

During evaluation, the agent receives the task instruction and starting repository. It can access the task environment, the model endpoint, and Hugging Face for model artifacts, but not the public web or upstream repositories. Hidden functional and regression tests score what its patch does, not whether its code matches the reference solution.

- 01Pinned task environmentEach attempt starts from its task's pinned SGLang revision. The repository and verifier run in that task's CPU-only or single-H100 environment; model inference is separate.

- 02Closed-book runmini-SWE-agent receives the instruction and starting repository and can inspect, edit, build, and test locally. It can access the task environment, the model endpoint, and Hugging Face for model artifacts, but not the public web or upstream repositories. public web blocked

- 03Hidden functional and regression testsHidden functional tests check the requested behavior; regression tests check selected existing behaviors that should remain unchanged. They do not compare the patch with the reference solution.

- 04ResultPass or fail is recorded with cost, output tokens, agent steps, and total wall time. Pass/fail reflects the verifier only; it does not assess code quality or merge readiness.

Consistent setup. Same mini-SWE-agent harness and 53 tasks, repeated three times. Scores apply only to this setup.

Pass@1. We divide verifier passes by 159 attempts per configuration. Agent-caused missing scores count as failures; ± is the 95% confidence interval across three runs.

Cost. API costs use fixed prices; open-weight model costs are hosted-rate estimates, not GPU operating costs.