Conversation
…ting step The diffusion sampler stashes a request's top-k logprobs on the step it converges and attaches them to the SamplerOutput on its commit step. The attach loop popped the stash for every decoding request in the batch, not only the committing ones, so a request that converged in the same step another request committed had its logprobs emitted under the wrong request. Its own commit then found no stash, the API received fewer logprobs than tokens, and chat completions failed with "IndexError: list index out of range" in _create_chat_logprobs. Any two concurrent requests whose schedules line up one step apart hit this. Restrict the pop to slots that are committing this step. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…s via extra_args
SamplingParams.extra_args (vllm_xargs on the OpenAI server) gains three
optional fields for structured reads on a DiffusionGemma model:
diffusion_seed_canvas list[int], exactly canvas_length ids. Replaces the
all-noise initial canvas after prefill; the caller
puts its answer template in known positions and
random ids in the slots it wants the model to fill.
diffusion_max_steps int. Denoise steps before the request commits.
diffusion_read_only bool. Commit the argmax canvas as soon as the step
cap is reached instead of waiting for convergence,
so a single-step request returns one forward's
logprobs at every canvas position.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The sampler read diffusion_seed_canvas, diffusion_max_steps and diffusion_read_only straight out of extra_args on the GPU worker. A seed id outside the vocabulary reached the embedding gather as a device-side assert, and a wrong type or length raised inside the engine loop, both of which take the engine down rather than the one request. SamplingParams.verify now checks the three fields and returns a 400. The input processor passes the served DiffusionConfig so the seed length is checked against the configured canvas, not the model's default. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A read-only request emitted its argmax canvas and then, unless the client had set max_tokens at or below the canvas length and ignore_eos, rolled into a second block from fresh noise with the seed gone. The request now carries that contract itself: verify caps max_tokens at the served canvas length and sets ignore_eos, so the scheduler ends the request on the first canvas and an end-of-turn token drawn into a noise slot cannot cut it short. A smaller client max_tokens still trims the output. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Every denoise step ran a GPU sync for the read-only mask (`.any()` and `.tolist()`) whether or not any read-only request was in the batch, and each finished prefill uploaded its seed canvas in a Python loop, one host-to-device copy per slot. DiffusionGemmaRequestStates now keeps host-side sets of the seeded and read-only slots next to the GPU flags. The sampler checks the sets first, so ordinary generation pays nothing. Seeds live in a [max_num_reqs, canvas] tensor uploaded once at add_request and applied to a whole prefill batch with one masked copy. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The logprobs stashed on the converging step came from the sampler's schedule-tempered logits (temperature 0.8 on a first step), so a caller after calibrated probabilities had to know the schedule and rescale. A read-only request now stashes logprobs from the model's own logits at temperature 1. The argmax, and so the emitted canvas, is unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The per-slot cap was a float tensor compared against a float cast of the int32 step counter inside the compiled step. Both are now int32. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A read-only slot never enters the encoder phase, so `~is_committing` was always true for it. Also state why remove_request leaves the GPU flags to add_request. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The schema-to-canvas work (template rendering, single-token label slots, noise seeding, averaging over draws) lives behind an OpenAI-shaped server so clients never build a canvas by hand. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.