Everybody is excited about Jev and i just got
access to it so i thought it’d be fun to play around with things it’s probably not intended to be the best at.
i’m not claiming this to be a serious eval, and watchful eyes will find some nitpicks and things to complain about,
so i just put together a bunch of questions that came to my mind late at night and had Jev run through each one N=50 times.
What is Jev
Jev IMHO sits right in the middle between LLM (too slow and relatively expensive), and training your own specialized classifier (relatively expensive and takes time to train / collect data etc.). i kinda treat it as a classifier on steroids - with primitives for binary (noul), multiclass (choice), and continuous (score).
In any case, the thing that makes it fun to poke at is that you don’t get text back, you get a distribution over the options you handed it. So i can ask the same question 50 times and actually watch which parts move and which parts don’t.
The setup
Each benchmark is a question and a list of options (choice) or true/false (noul). i send
the identical request 50 times, record the whole distribution, and then look at the spread.
The rest of the parameters were set to default. Running the Python SDK against jev-1.13.0, 3200 calls in total.
i also kept the very first one-shot answer for each question, from before i started repeating them. 6 of the 64
came out more than 2 standard deviations off the 50-run mean, which is basically why i stopped trusting single runs.
Here’s the list of all of them below. i put some of my comments in there, take them with a grain of salt.
The checkmarks and x’s are my own scoring and a few of them are arguable, which i’ve said on the cards where
it matters. For the noul ones i’m calling anything above 0.5 a yes.
A few that stuck out
i’m not drawing conclusions from any of this, these are just the ones i kept going back to.
The ids are searchable in the box above if you want to pull one up.
b61 and b62. It says “there are 2 r’s in the word strawberry” is true on all 50 runs, and it says
“there are 3 r’s” is true on all 50 runs, and those two can’t both hold. Then b64 asks for
both at once and it comes back false, also on all 50. So each predicate on its own gets a yes and
the conjunction gets a no.
b44 and b46 are the same question with the options typed differently. With the ASCII
stand-ins it won’t settle, N on 34 runs and other on 16. Write the options properly as
ω and ℵ₀ and it picks ℵ₀ on all 50, with confidence going from 0.315 to 0.982. That is
a lot of movement for a change of characters, though it can probably be explained.
b31 next to b15. The 25th digit of pi comes back at 0.033 confidence, the lowest anywhere
in the set, and it still picks the same digit on 49 of 50 runs. The next digit after 3.1415
comes back at 0.584, much higher, and flips 7 times out of 50. So it being unsure doesn’t
really tell you the answer is going to move around.
The primes block, b57 through b60. Counting them works, 4 on all 50 runs. Ask for the
same count as a letter from a to h and it says a on all 50, which is just wrong. Give it
a1 through a6 and it outputs correctly, but perhaps because they’re numbered. Then in
b60, where i write out a = 7, b = 4, c = 5, d = 3, the correct label b is last of the
four at 0.154, behind every distractor.
b41 is the one i set out to trick, by leaving “continuous” out of the compact to Hausdorff
bijection. It answers homeomorphism on all 50 runs at 0.998, which is about the top of the
range for this set. It doesn’t waver at all on a question where none of the options is right.
The implications. Everything with a false antecedent, b07 through b11, comes back true on
all 50 runs, which is correct and pretty solid. The controls with a true antecedent are the
ones that wobble: b28 and b29 sit around 0.41 to 0.45 and cross the line on 2 runs out
of 50.