Routing a support ticket, assessing an agent’s proposed action, or rating a request’s urgency calls for a decision your code can act on. Language models can return structured decisions, but still produce them through text generation.
Jev, a System One model from TypeSafe AI, evaluates supplied state against typed questions and returns choices, scores, and boolean probabilities without generating prose. The AI SDK’s experimental_evaluate API makes these answers available directly in TypeScript via Vercel AI Gateway.
Your application decides what happens next: a department choice can select a support queue, a severity score can influence priority, and an uncertain result can trigger review. Keeping those rules in code lets you change how the application responds without redefining what you ask the model to assess.
Copy link to headingOverview
In this guide, you'll learn how to:
- Ask a single yes-or-no question about a piece of state
- Answer several typed questions in one request, including choice, score, and boolean questions against structured state
- Branch on probabilities and confidence so clear cases route automatically and uncertain cases go to review
- Unit-test that branching with a mock evaluation model
- Call Jev through the Gateway provider instance when your application needs an explicit provider object
Copy link to headingPrerequisites
Before you begin, make sure you have:
- A Vercel account.
- Vercel CLI installed (npm i -g vercel)
- Node.js 22+ and a package manager (e.g., pnpm)
- An existing Next.js project using the App Router
Copy link to headingHow Jev differs from a language model
Jev is a probabilistic decision model, not a chat model. Language models can generate structured answers, but any probability included in that output is itself a generated estimate.
Jev evaluates each question independently against the same state, returning typed answers with probabilities over the defined outcomes. Your schema constrains those answers, but doesn’t guarantee they’re correct.
Copy link to headingJev on AI Gateway at a glance
Copy link to headingThree question types
AI SDK exposes three question types, each mapped to a TypeSafe AI primitive:
Every answer keeps its question ID and matches the question’s type. For Boolean answers, probability estimates how likely the statement is to be true:
- 0.98indicates a strong yes.
- 0.02indicates a strong no.
- 0.5indicates uncertainty between the two outcomes.
Copy link to headingSteps
Copy link to heading1. Install the AI SDK
Install AI SDK 7.0.105 or later, which adds experimental_evaluate:
pnpm i ainpm i aiyarn add aibun add aiLink your local directory to its Vercel project and pull your environment variables. This writes a VERCEL_OIDC_TOKEN to your environment file.
When you pass a model ID as a plain string (e.g., typesafe-ai/jev), AI SDK routes the call through AI Gateway and authenticates with the OIDC token.
Deployments on Vercel automatically receive the token. Locally, the token expires after 12 hours, so re-run vercel env pull when a request returns a 401.
Copy link to heading2. Ask one boolean question
Start with a single yes-or-no decision. The state is whatever you want the model to look at, and each key in questions becomes a key in answers.
Calling wasRefunded('The support agent issued a full refund to the customer.') returns a number close to 1.
One example of result.answers:
The criteria on a boolean question are optional, but they sharpen the decision by telling the model exactly what counts as true and false.
Copy link to heading3. Answer several questions against structured state in one request
Jev evaluates all questions in a request in parallel, so adding questions barely changes latency. You can mix all three question types, and state accepts a JSON object or array as well as a string, which means you can pass a record or a message history without serializing it yourself.
The following route handler triages a support ticket into a department, a severity level, and a refund flag in one round trip.
Start the dev server and send a ticket:
pnpm devnpm run devyarn devbun devThe response contains a typed answer and probabilities for each question under its original ID, as shown in the example below:
Reading the response:
- department.choiceis inferred as a union of your option keys (- 'billing' | 'technical' | 'account' | 'other'), so TypeScript flags a typo like- choice === 'tech'at compile time.
- department.probabilitiescovers every option, and the selected choice always has the highest value.
- severity.scoreis the probability-weighted mean across the rubric levels, indexed from zero.- 2.86sits between "Blocking with no workaround" and "Blocking and causing financial or data loss".
- severity.probabilitiesuses string keys for the level indices, so- "3"is the fourth rubric entry.
- requestsRefund.probabilityis the estimated probability that the customer wants money back, not a confidence in the answer.
The providerOptions.gateway object is optional. Jev supports Zero Data Retention and No Training per request, and evaluation calls appear in AI Gateway logs and count toward budgets like any other model call.
Copy link to heading4. Branch on probabilities and confidence
Use the returned probabilities to decide when your application should act automatically or request review, setting stricter thresholds where an incorrect decision would have greater consequences.
TypeSafe AI also returns a separate confidence statistic for choice and score answers in result.providerMetadata.typesafe.confidence, keyed by question ID. Confidence summarizes how concentrated the probability distribution is, from 0 (spread evenly across options) to 1 (all on one option). It differs from the selected option's probability, and it isn't returned for boolean answers.
The routing logic below uses two paths:
Move the evaluate call into a function that returns a decision, keeping the questions from step three. Its model parameter defaults to Jev and lets the tests in step five substitute a mock.
Then update the route handler from step three to call routeTicket and return the routing decision instead of the raw model answers:
Sending the same curl request from step three now returns a decision:
Keep two distinctions in mind when applying this pattern:
- Probability distributions are optional in the AI SDK. Jev returns probabilitiesfor Choice and Score answers, but other providers may omit them. Use optional chaining (?.) and handle a missing distribution explicitly, such as by sending the ticket to review.
- refundRequestedrecords intent, not approval. It identifies whether the customer asked for money back. The billing team or application rules must check eligibility against the account, plan, and refund policy before approving a refund.
Treat the example thresholds as starting points. Calibration describes how predicted probabilities match observed outcomes across many examples; it doesn’t guarantee an individual answer is correct. Evaluate labeled tickets using the same questions, then choose cutoffs based on the errors your workflow can tolerate.
Copy link to heading5. Test the routing logic without calling Jev
The thresholds in routeTicket are application logic, so test them like any other function. The AI SDK ships Experimental_EvaluationMockModelV4 in ai/test, which returns whatever answers you give it. Pass it as the model argument to check each branch with fixed inputs and no network call.
Install Vitest as a development dependency:
pnpm add -D vitestnpm install -D vitestyarn add -D vitestbun add -d vitestThe mockJev helper derives its answers type from the mock’s doEvaluate return type, letting TypeScript catch incompatible answer shapes at compile time.
Run the tests:
pnpm vitest run lib/route-ticket.test.tsnpx vitest run lib/route-ticket.test.tsyarn vitest run lib/route-ticket.test.tsbunx vitest run lib/route-ticket.test.tsExpected output:
Testing the thresholds this way is separate from checking whether the thresholds are right for your data. For that, run real tickets with known outcomes through Jev and compare its probabilities to what actually happened.
Copy link to heading6. Use the Gateway provider instance
Use @ai-sdk/gateway when you need an explicit provider object or want to configure custom headers, a custom fetch, or a different Gateway base URL.
pnpm i @ai-sdk/gatewaynpm i @ai-sdk/gatewayyarn add @ai-sdk/gatewaybun add @ai-sdk/gatewayThe string and provider-instance forms are interchangeable.
Copy link to headingBest practices
Copy link to headingAsk atomic questions and combine them in code
Jev works best when each question asks one well-scoped thing that a knowledgeable person could answer in a few seconds. If a question would require extended reasoning or weighs several independent factors, split it into one question per factor and combine the answers with your own logic.
Copy link to headingDescribe options and levels instead of labeling them
Choice criteria are a map of option keys to descriptions, and score criteria are ordered descriptions from lowest to highest.
Writing 'Blocking with no workaround' gives the model far more to match against than 'high'. Descriptions can be strings, JSON objects, or arrays, so you can pass a list of example phrases for each option.
Copy link to headingKeep state focused
Input tokens are the only thing you pay for, so pass the fields the decision depends on rather than an entire record. Adding questions to a request doesn't degrade the answers to existing ones, because each question is evaluated independently.
Copy link to headingSet thresholds per action, not per model
Read-only actions like showing a screen can tolerate a wrong guess, so a probability of 0.7 might be enough. Destructive actions need a higher bar, closer to 0.9 or above, and a confirmation step below it. Encode that risk tolerance in your code, and keep a review path for anything under your floor.
Keep classification separate from authorization. Jev can tell you that a customer asked for a refund or that a command looks destructive. Whether to grant the refund or run the command depends on rules Jev doesn't see, such as account status, policy, and permissions, so make that a second check in your code.
Copy link to headingRead distributions defensively
TypeSafe AI rounds probabilities and scores to two decimal places, and result.rounding reports that precision. Because of this rounding, a choice distribution may sum to 0.99 rather than exactly 1. The AI SDK accounts for this during validation, so don't renormalize the values yourself.
Copy link to headingTroubleshooting
Copy link to headingAuthentication errors (401 or 403)
Expired local credentials or an unlinked project can prevent authentication. Run vercel env pull to refresh your credentials, using vercel link first if the directory isn’t linked to a project. If a 403 persists, check your access to the linked project and AI Gateway.
Copy link to headingUnsupported question type
The evaluation model doesn’t support one of the requested question types. Check the model’s supported types and either adjust the questions or select a model that supports them. Jev supports Choice, Score, and Boolean questions.
Copy link to headingNoSuchModelError
When modelType is 'evaluationModel', the provider couldn’t resolve the requested evaluation model or doesn’t support evaluation. Check that the model ID is typesafe-ai/jev and that the call resolves through AI Gateway or another evaluation-capable provider.
Copy link to headingInvalidResponseDataError
The provider returned an invalid answer, such as a distribution with a missing option or a score outside the rubric range. Retry the request. If the error persists, report the failing questions and response to the provider.
Copy link to headingEvaluation is unavailable through an OpenAI-compatible client
AI Gateway exposes evaluation through the AI SDK, rather than its compatibility endpoints. Call experimental_evaluate from the ai package.
Copy link to headingAnswers seem overconfident on your data
Run labeled examples through the same questions and compare predicted probabilities with observed outcomes. Inspect where errors cluster, revise unclear criteria, and evaluate the updated questions before choosing new thresholds.
Copy link to headingNext steps
- Read the Evaluation guide and the experimental_evaluatereference in the AI SDK docs for the full API, includingabortSignal,headers, andExperimental_EvaluationMockModelV4for tests
- See the AI Gateway evaluation docs and the Jev model page for pricing, limits, and the Gateway provider instance
- Learn more about OIDC authentication for AI Gateway
- Explore confidence-gated routing, speculative fan-out, and composite scoring in the TypeSafe AI patterns docs
- Run the evaluate examples from the AI SDK repository