jevperis an independent implementation of the documented System One wire format. It is not affiliated with, endorsed by, or supported by TypeSafe AI — questions about the API itself belong in their docs.
The Jev interface — state in, typed questions (noul, choice, score) out,
answers carrying probabilities and confidence — on top of any OpenAI-compatible model.
Same call as typesafe-sdk, different backend: point jevper at a hosted LLM or a self-hosted llama.cpp
server and code written for Jev keeps working, unchanged.
It does not call the hosted TypeSafe API and does not depend on typesafe-sdk or openai at runtime — the
client object is duck-typed. Any object exposing responses.create or chat.completions.create works,
including a self-hosted llama.cpp server.
The responses surface speaks both OpenAI's Responses API and the OpenResponses
specification served at the same /v1/responses path — LM Studio (0.3.39+), llama.cpp, vLLM and SGLang all
implement it, and what each one does with the fields jevper sends is measured in
docs/local-servers.md.
from openai import OpenAI
from jevper import Choice, SystemOneClient
client = SystemOneClient(OpenAI(), model="gpt-5.6-terra")
response = client.system_one(
state="I was charged twice for the same subscription this month.",
questions={
"intent": Choice(
instructions="Pick the intent of the message.",
criteria={
"billing": "money, invoices, refunds, charges",
"technical": "errors, crashes, login or performance problems",
"sales": "pricing, plans, purchasing, upgrades",
},
)
},
)
answer = response.answers["intent"]
answer.choice # "billing"
answer.probabilities # {"billing": 0.88, "technical": 0.08, "sales": 0.03}
answer.confidence # 0.83method defaults to auto: it asks for logprobs where the provider has them and answers in JSON where it
does not, remembering the verdict per model and surface. gpt-5.6-terra is a reasoning model and returns
none, so the probabilities above arrive as JSON. Point method="logprobs" at a provider that does return
them — a local ollama, llama.cpp or vLLM server, or a non-reasoning OpenAI model — to read the model's
real distribution instead of its self-report.
pip install jevperPython 3.10+. The only runtime dependency is pydantic>=2.7.
Agent skill for using this package:
npx skills add zhulinchng/jevper-skillFor development:
git clone https://github.com/zhulinchng/jevper && cd jevper
uv venv && uv pip install -e '.[test]'
pytest -qflowchart LR
A["state + questions"] --> B["build_parts + assemble: system prompt, few-shot turns, question block, state turns"]
B --> C{"method (auto resolves first)"}
C -->|logprobs| D["logprobs=true, top_logprobs=20"]
C -->|grammar| E["+ GBNF grammar in extra_body"]
C -->|structured| F["strict JSON schema: probabilities"]
C -->|discrete| G["strict JSON schema: one label"]
D --> H["first label token -> softmax over the labels"]
E --> H
F --> I["probability dict from JSON"]
G --> J["one-hot from the chosen label"]
H --> K["Answer: choice / noul / score"]
I --> K
J --> K
Each question becomes its own provider call, so questions are independent and run concurrently
(max_concurrency, default 8). Answers come back keyed by your question ids, in insertion order.
Three types, mirroring the Jev API — Noul answers yes/no with one probability, Choice picks one of your
labelled options, Score rates on an ordered scale:
Score.score is the probability-weighted level index (Σ i·pᵢ, levels zero-based), as in the Jev API —
read off the distribution rescaled to sum 1, so with normalize_probabilities=False the reported
probabilities stay the model's own numbers while the score stays on the 0..N-1 line.
Choice takes up to 255 options, the Jev API limit, and the API documents no minimum.
The two methods that read a label token — logprobs and grammar — stop at 26, because the first token
of "AA" is "A"; past 26 options they
raise InvalidQuestionError pointing at structured and discrete, which answer in JSON and use
two-letter labels. The default method="auto" never hits that error: it answers a wide Choice in JSON.
Questions can also be passed as raw mappings ({"type": "choice", "criteria": {...}}) and are validated the
same way.
method= decides how the decision is elicited. All four share the same label→option mapping, so switching
methods does not change your types; only the label alphabet differs (logprobs and grammar need
single-letter labels, so they cap at 26 options).
auto is the default because logprobs are not universal: OpenAI's reasoning models — the GPT-5.6 family
included — do not offer them, Anthropic and Gemini's OpenAI-compatibility endpoints never had them, and a
model that returns a logprob with no alternatives gives you no distribution at all. A gateway in front of
one says so in as many words (logprobs are not supported with reasoning models.), and so does a
Responses endpoint that refuses the include list the carrier travels in.
auto reads the logprobs where they exist — one short call, and the model's real distribution rather than a
self-report — and answers in JSON where they do not, remembering the verdict per model and surface. See
docs/methods.md for the
provider table, the exact request bodies, the readout rules and the failure modes.
The same holds for the request fields jevper adds: a server that refuses structured output, the reasoning
parameters, the Responses include list, the cache key or the Messages output_config gets that field
dropped and the call re-asked, so a partially implemented server answers instead of failing.
debug["server_limits"] reports what it refused.
Hosted providers need no adapter of their own. Gemini speaks the OpenAI API at
https://generativelanguage.googleapis.com/v1beta/openai/, so OpenAI(base_url=…, api_key=…) is the whole
integration; what it lacks in logprobs, auto answers around.
Pass reasoning=ReasoningConfig(...) to make the model think before it classifies:
from jevper import ReasoningConfig, reasoning_text
client = SystemOneClient(OpenAI(), model="gpt-5.6-terra", reasoning=ReasoningConfig(effort="medium"))
response = client.system_one(state=..., questions=...)
reasoning_text(response.reasoning) # the trace, as textmode="auto" (the default) uses native provider reasoning on the Responses surface and a two-step
think-then-classify path on Chat Completions, where the analysis text is replayed as an assistant turn before
the answer. The trace always lands on response.reasoning, and the two-step analysis call's usage is counted
in response.usage. See docs/reasoning.md.
Examples are chat turns (question block + example state, then the expected answer), so the demonstration is always in the format the active method expects. They can be attached at three levels:
from jevper import Choice, Example, SystemOneClient
question = Choice(
criteria={"billing": "...", "technical": "..."},
examples=[Example(state="Charged twice for one order", answer="billing")],
)
client = SystemOneClient(OpenAI(), model="gpt-5.6-terra",
examples=[Example(state="Login fails", answer="technical")]) # fallback for every question
client.system_one(state=..., questions={"intent": question},
examples={"intent": [...]}) # or a bare sequence for all questionsPrecedence is question → per call → constructor, and the first non-empty level wins. examples is excluded
from model_dump(), so question dumps keep exactly the Jev wire keys. See
docs/few-shot.md.
response.model # the model id jevper asked for
response.answers # {"intent": ChoiceAnswer(...)}
response.nouls / .choices / .scores # filtered views
response.usage # input_tokens, output_tokens, reasoning_tokens, cached_tokens, n_calls, n_retries, latency
response.reasoning # tuple[ReasoningContentPart, ...]
response.debug # per-attempt requests/responses, retry reasons, normalization notesresponse.model_dump_json() serializes to the Jev answer shape — the answer field names and JSON keys match
POST /v1/systemone. Token counts are None when any constituent call omitted them; n_calls counts the
provider calls that returned a result, including analysis passes and corrective retries, while n_retries
counts transient-failure retries only. A failed attempt appears in debug["llm_attempts"] but not in usage.
Confidence is a share in [0, 1] and the call counters are counts, because jevper computes them; the
probabilities beside them are the model's own numbers, passed through verbatim when
normalize_probabilities=False. See docs/api.md
for the full reference.
Every provider that serves these calls caches the prefix of a prompt and reuses it for the next request that starts the same way, and jevper is shaped for it: the state comes last, so the system prompt, the few-shot examples and the question block are identical across every state classified with one rubric.
client = SystemOneClient(OpenAI(), model="gpt-5.6")
client.system_one(state=record_a, questions=rubric)
client.system_one(state=record_b, questions=rubric) # the shared prefix is reusedTwo things make it steerable and observable:
- prompt_cache_keyis sent with every request, derived per question from the parts of the prompt that do not change between calls — model, method, examples, question block — so a rubric's requests are routed together, and a- logprobsrequest is not routed with a- structuredone whose prefix differs. Pass your own to group or account for them your way, on the client (- prompt_cache_key="tenant-42") or per call. A server that refuses the field gets it dropped and the call re-asked, like the other optional fields.
- usage.cached_tokensis the prompt tokens the provider read from its cache, summed over the call.- Nonemeans the provider said nothing — vLLM needs- --enable-prompt-tokens-details, and SGLang's Chat Completions route needs- --enable-cache-report— while a reported- 0means a cold or disabled cache.
Measured on one 2388-token prompt carrying two examples, second call differing only in the state: reused
tokens went from 40 — the system prompt alone — to 1010 on llama.cpp, 528 on vLLM and 896 on SGLang once the
state moved to the end. Per-server flags, what each server accepts or ignores, and how to isolate a cache with
cache_salt: docs/local-servers.md.
Every server in the local fleet serves the Anthropic Messages API at /v1/messages as well as the OpenAI
ones. jevper speaks it with api="messages": point the anthropic client at the server and pass it in place
of the OpenAI one.
from anthropic import Anthropic
from jevper import SystemOneClient
client = SystemOneClient(Anthropic(base_url="http://127.0.0.1:1234"), model="qwen3-4b-instruct")
client.system_one(state=record, questions=rubric, api="messages", method="structured")Five things differ from the OpenAI surfaces, and all five come from the protocol rather than from any server:
- No logprobs exist in it. Not withheld by some servers — absent from the API. method="logprobs"andmethod="grammar"raiseUnsupportedMethodErrorbefore a request is sent, andmethod="auto"answers in JSON without spending a call to find out.structuredanddiscretework exactly as they do elsewhere: the prompt already asks for one JSON object.
- Its schema field is Anthropic's own, output_config.format, the counterpart ofresponse_format:structured/discretesend it wherever the server takes it, and a server that refuses it gets it dropped and the call re-asked, reported indebug["server_limits"]["output_config"]. It travels in the request body rather than as an SDK keyword, because the oldest Anthropic SDK jevper supports has no such parameter, and the schema is rewritten for the API's documented subset first — Anthropic rejects numerical constraints, so eachminimum/maximummoves into the description of the field it bounded and the wire schema saysMust be at least 0.where the prompt still saysminimum: 0. The JSON Schema also stays in the system prompt: vLLM implements that field — a schema naming a constant the prompt never mentions comes back with that constant in the answer — while llama.cpp and LM Studio accept it and ignore it, and a server that discards a field it accepted looks exactly like one that never read it.
- max_tokenshas no server-side default. jevper sends- 1024— or- 1024plus the caller's thinking budget, because Anthropic requires the budget to be strictly below- max_tokensand would otherwise refuse the 1024 its own docs call the floor.- extra_body={"max_tokens": n}overrides both, and a value that cannot hold the budget you asked for raises- JevperErrorlocally, naming both numbers, rather than being sent to earn the 400.
- temperatureis not a typed parameter of the current SDK and is left out of a request that enables thinking, which the API refuses alongside a non-default one. On the other requests it travels in the request body, so a local server still reads it.
- Thinking is asked for with a budget, not an effort name. ReasoningConfig(budget_tokens=2048)sendsthinking={"type": "enabled", "budget_tokens": 2048}: a budget is the only reason to ask for this surface's own thinking, so it selects it even undermode="auto". A server that refuses the field gets it dropped and the call re-asked, reported indebug["server_limits"]["thinking"]; a server that refuses the value — SGLang answersbudget_tokens: must be at least 1024— gets its own error back instead, because a bad number is not a missing field.
Thinking blocks come back as ordinary response.reasoning parts with the block's signature kept, and
usage.cached_tokens is read from cache_read_input_tokens. Which servers implement the route, and since
which version: docs/local-servers.md.
Local problems fail before any request is sent: an invalid question, an empty questions mapping, an
unusable state, a model that is not a non-empty string, a count option that is not an integer, or
grammar on a surface that cannot carry a grammar.
Transient failures (HTTP 408, 409, 429 and any 5xx — the set both official SDKs retry — plus the
transport and timeout errors the SDKs and the standard library raise, and whatever an x-should-retry
header says, which outranks the status) are retried per call with
RetryPolicy(n_retries=2, base_delay=0.5, max_delay=8.0, respect_retry_after=True). The wait is the
provider's own instruction when it sent one: a Retry-After (delta-seconds or an HTTP date) or the
millisecond retry-after-ms replaces the exponential backoff min(base_delay · 3ⁿ, max_delay), which is
what the TypeSafe clients do — coming back sooner than a rate limit asked only extends it. max_delay caps
jevper's curve, not the server's number; a header past a day (jevper.client.MAX_RETRY_AFTER) is not an
instruction any client should carry out, so the curve answers instead; respect_retry_after=False goes back
to the curve alone. Unreadable answers get one corrective retry (n_retry_malformed) with the failure
appended to the conversation. ProviderError propagates after all questions have settled, in question
insertion order.
Every provider call goes through the SDK client you hand in, so mlflow.openai.autolog(),
mlflow.anthropic.autolog() or OpenTelemetry instrumentation sees them without extra wiring — including the
attempts made before settling on a request shape the provider accepts. The per-question worker threads run a
copy of your context, so a span you open around system_one parents every one of them: one trace per call,
one child span per question, however many questions it carried. Span names, attributes and the hosting
recipes: docs/mlflow.md.
pytest -q # the whole suite runs against a local stub HTTP server; no network, no API keys
ruff check src tests # clean except three PYI034 hints (see docs/internals.md)The suite drives a real openai SDK client at a stdlib ThreadingHTTPServer stub, so the SDK's own
serialization path is exercised; see docs/internals.md.
Optional live check, skipped unless both variables are set:
LLM_MODEL=gpt-5.6-terra OPENAI_API_KEY=... pytest -q tests/test_live.pyOptional MLflow check, skipped unless MLflow is installed — tracing, hosting jevper as a model, the AI
Gateway, and mlflow.genai.evaluate (see docs/mlflow.md):
uv pip install -e '.[test,mlflow]' && pytest -q tests/test_mlflow.pyRead the jevper documentation site for the quick start, method and
surface selection, public API reference, local-server compatibility, reasoning, few-shot examples, internals,
and MLflow integration. Contributors can edit the Markdown under docs/ and run
uv run --extra docs mkdocs build --strict.
Apache-2.0 — see LICENSE.