Mini-Jev – typesafe's Jev implemented on top of an LLM locally

What a Jev-style interface looks like on a frozen model: classify against a schema that arrives with the request by reading next-token logits, not by generating JSON. Measured on Qwen3-4B.

TypeSafe's Jev is trained for typed decisions without open generation. mini-Jev asks how much of that interface an ordinary frozen model already provides, and measures it against today's path, grammar-constrained JSON.

A JSON schema arrives with the request. Today the model writes a JSON object under a grammar, token by token. This project measures the alternative for closed-choice fields: turn each field into a lettered multiple-choice question, run one forward pass, and read the model's scores for the option letters at the answer position. No token is generated. Strings and numbers are still generated.

Everything here runs on Qwen/Qwen3-4B-Instruct-2507 (bf16, greedy) with xgrammar for constrained generation and CLINC150 as the task. The experiment was preregistered (PREREG.md, amendments v1.1–v1.3) and every number below is recomputed from the stored run records.

In TypeSafe's own vocabulary (agent skill): their Choice and Noul primitives are what this study measures on a frozen model, as the letter read and the boolean; Score (an ordered scale) was not measured; their pattern select over generate is the extraction-as-choice recommendation above, also not measured here. This is a correspondence of terms, not a reproduction of their model.

Rule of thumb from the whole study: on this model, quality is the same either way for closed choices; the single lever that changed accuracy was giving the model a one-token identifier to answer with. Reading the logits then removes the decode cost and hands you a distribution.

demo/ is a local page that takes a text and a flat JSON schema, answers it with both methods on one model, and shows every step with the model's own numbers: the exact prompt and constraint, the top next tokens at the answer position, the letter scores, what the grammar allowed and overrode at each written token, and the timing.

Requirements: Python 3.11, uv, ~9 GB of disk for the weights, and either Apple Silicon (MPS) or an NVIDIA GPU (CUDA). The 4B model needs about 8.5 GB of memory.

git clone https://github.com/r-ms/mini-jev.git && cd mini-jev

uv sync # torch 2.5.1, transformers 4.57.6, xgrammar 0.2.7, pinned

MINIJEV_DEVICE=mps uv run python demo/server.py # or MINIJEV_DEVICE=cuda

# first start downloads Qwen/Qwen3-4B-Instruct-2507 (revision pinned) and warms up, then:

open http://127.0.0.1:8765/Edit the text or the schema in the page and press Run. Supported field types: string with enum, boolean, string (with maxLength), integer, number. Nested objects and arrays are refused on purpose.

What you will see, on the default example:

Hover any written token in the page to see the five tokens the model considered at that step, which of them the grammar blocked, and the best tokens it allowed.

uv run python scripts/build_data.py # CLINC150 sample, pinned revision, manifest with hashes

MINIJEV_DEVICE=cuda uv run python scripts/smoke_positions.py # two-sided guard for the generation path

MINIJEV_DEVICE=cuda uv run python scripts/run.py --arm B1 --subset main --run-id r1

MINIJEV_DEVICE=cuda uv run python scripts/run.py --arm A1joint --subset main --run-id r1

uv run python scripts/report.py runs/r1 # mechanism counters first, then accuracies

uv run python scripts/run_cost.py --run-id r1 --lengths 32,128,512

uv run python analysis/figures.py # redraws docs/figures from runs/

uv run pytest -q tests/ # 29 harness guards (each has a placebo)Arms: B1 (read letters), B1cache (read letters on a shared prefix), A1joint (one JSON, all fields), A1split (one JSON per field), A1letter (one letter under a grammar), A1label (option name under a grammar), A1prob (model writes probabilities), A0 (no grammar), B1free (unconstrained text control), B1perm (option-order rotation), plus scripts/run_score.py for sequence likelihood of the names. Every record carries the raw response, candidate logits, prompt and schema hashes, the gold position and a run-mode string; a run refuses to resume if the mode changed.

README.md this file

docs/ARTICLE.md the write-up, with figures

PREREG.md preregistration and its four amendments (v1 … v1.3)

DEFERRED.md what was deliberately not done (second model, calibration)

minijev/ engine, prompts, schema and grammar builders, scoring, report

scripts/ run.py (all arms), run_cost.py, run_score.py, smoke_*.py, report.py, build_data.py

analysis/ follow-up analyses (dependent fields, controls) and figures.py

demo/ the teaching bench: server.py + index.html

tests/ 29 guards for the harness

runs/ run records (not committed; regenerate or ask)

One model (Qwen3-4B-Instruct), one dataset (CLINC150, English, short utterances). Bench times are one request at a time on one GPU. The letter shares are a ranking with a confidence gap, not calibrated probabilities.