The best feature of Jev is not the speed or low cost but the way it's honest when it doesn't know something whereas an LLM will often confidently provide an incorrect answer. This, combined with the high efficiency at classification problems makes it a great gatekeeper in front of an LLM.
Jev with a selective Gemini fallback made my evaluation 6.24x faster and 8.7x cheaper than Grok 4.6, matching its 89.6% accuracy. That's an 84.0% reduction in mean latency and an 88.5% reduction in mean cost.
The part I find interesting is how little Gemini had to do. Jev is a constrained 'decision' model that handles common questions and exposes uncertainty. It scored 81.3% on its own. Sending low-confidence questions to Gemini added 8.3 percentage points, with Gemini invoked for just 14.6% of questions. I wanted to understand when a smaller model could do enough, and when it should ask for help.
That is why Jev feels like an honest game changer. It does not pretend to replace a general-purpose LLM. It handles the common path, admits uncertainty, and makes the expensive model optional.
A response matches when its label is in the manually defined acceptable set. Valid responses count final logical answers, not total calls. Gemini retry rows include the cost and sequential wall time of both the failed first attempt and its one permitted retry.
Why Jev fits this problem
Sleuth the Truth is a Guess Who-style game built from Wikipedia subjects. A player asks a question and the system returns yes, no, sort-of, or maybe.
Instead of generating prose, it chooses from the allowed answers and returns probabilities. That gives me a typed answer and a useful measure of uncertainty.
The production route is simple:
- use deterministic date or Wikidata tools when they apply
- otherwise ask Jev
- return high and medium-confidence answers directly
- send low-confidence answers to Gemini 3.8 Flash
Jev handles the common path. Gemini handles the edge cases.
The test
I sampled 48 privacy-scrubbed questions from 48 production sessions. Thirty-six follow the normal traffic mix. Twelve are deliberately difficult, including questions that received substantive negative feedback and questions with genuinely ambiguous wording.
I labelled the questions before seeing the model outputs. Some questions accept more than one label where the wording supports it. This measures agreement with a small human-authored answer set, not general intelligence.
The replay starts at Jev and does not exercise the deterministic routes that run before it in production.
Jev matched 39 of 48 labels on its own. It marked seven questions as low confidence. Sending those seven to Gemini raised the result to 43 of 48, while keeping the median response time at 379 ms.
Sending medium-confidence answers to Gemini as well did not improve accuracy. It only added latency and cost. Jev's confidence was useful enough to identify the right boundary.
The retry caveat
Twelve standalone Gemini calls exhausted the original 256-token completion budget before returning valid JSON. I retried only those failures once with a 2,000-token cap. All 12 retries returned valid responses and 10 matched the expected label.
That is a selective-retry policy, not a clean one-pass comparison. The displayed cost and latency include both the failed first attempt and the retry. The Jev route pays that overhead only for questions it actually sends to Gemini.
What changed
Jev is now the semantic middle layer in production. It is fast enough for the common path and honest enough about uncertainty to know when it needs help.
The result is not that a small constrained model beat every general-purpose LLM. Standalone Gemini scored higher. The useful result is that Jev avoided most generative calls while matching Grok's accuracy at a fraction of its latency and cost.
That is the game changer: not replacing the expensive model, but making it optional.