Jev Ran 132,422 Independent Test Requests This Week. None of Them Measured Calibration.

Three days ago we published that nobody outside TypeSafe had tested what Jev is actually selling. Two independent benchmarks landed on Monday. The Decision Index is a reproduction kit running 132,422 requests across 37 benchmarks, 19 of which form a scored panel, released MIT so a stranger can run the whole suite on a laptop or as one cloud job. Cua-Bench-S1 targets decision models for computer use, shipping two open checkpoints alongside it. Both are real work plus a genuine improvement on a vendor-run test. Neither measures calibration. The Decision Index requires a full probability distribution across every supplied option, then scores with accuracy metrics and chance-normalised skill. Whether an 80% confidence means right 80% of the time remains untested by anyone. Best for anyone choosing a decision model on evidence. Not ideal if you wanted the calibration question settled.

On Friday we wrote that Jev’s central claim had never been checked by anyone outside the company selling it, plus that the test would take about a day.

On Monday two teams published benchmarks.

That is a faster turnaround than this industry usually manages. Both efforts improve on what they replace. One of them is fully reproducible by a stranger, which is more than TypeSafe’s own numbers offered.

Then you look at what gets scored.

What Actually Shipped

Verified against the repository plus the Hugging Face listing on September 22.

Alongside it, Cua-Bench-S1 arrived from the team behind the trycua computer-use project, described as a benchmark for decision models built for computer use, with two open checkpoints named Cua-S1-Nano-0.1 plus Cua-S1-4B-0.1. Those figures come from the launch post rather than a repository we could verify.

The Decision Index Is Properly Built

Credit where it belongs. This is serious methodology, plus the design choices are the ones an honest evaluator makes.

Reproducible by a stranger

The repository exists so somebody with no connection to the author can obtain the frozen suite, run any engine through it, score every benchmark with the leaderboard’s own metrics, then compute the index.

It runs on a laptop or as a single Hugging Face job on an RTX PRO 6000, with a live leaderboard space carrying the results. Submissions arrive by pull request, with the results file re-scored on review rather than taken on trust.

Compare that to the benchmark TypeSafe published. We covered the three caveats they disclosed themselves, including that their own model team wrote the tests plus that competing models ran through a TypeSafe wrapper the company admits is slower. Nobody outside could rerun any of it.

The scoring is deliberately unkind

Three design decisions stand out, all of them harder on the model than the easy alternative.

Unanswered counts as wrong. Unsupported, errored, abstained plus pending requests all score zero against the full frozen denominator. A model that refuses long or wide inputs pays for it in the score.

No truncation. An engine that cannot fit a request raises an error and the row counts against it. Nothing gets cut down to fit.

No option filtering. Every option in the criteria gets scored, plus the response must carry a probability for each one or validation rejects it.

Parity is demonstrated, which almost nobody bothers with

The kit reproduces the leaderboard’s own reference run exactly, matching its benchmark summary plus leaderboard entry at 55.74, 40.86 plus 39.45 across the three formulas, along with every area and per-benchmark value.

Publishing a reproduction that lands on the same numbers is how you show the pipeline works. Most benchmark releases skip it, then wonder why nobody trusts the figures.

The Number Nobody Is Checking

Now the part that keeps this story open.

What Jev is sold on

Structured output has been available for years through constrained decoding. TypeSafe’s actual pitch, plus the reason RLCD exists as a training method, is calibration. When the model reports 80% confidence, it should be correct roughly 80% of the time.

That property governs whether you can build on the output. Software acting on a confidence score needs that score to mean something numerically.

What the Decision Index measures

Every benchmark gets scored with its native metric. Those are accuracy measures plus conservative F1 with a fixed label universe. A chance-normalised skill figure is computed as raw minus random over one minus random, alongside a breadth variant.

All three formulas reward getting the top answer right. None of them examines whether the probability attached to that answer corresponds to a real hit rate.

Why that gap is strange

The suite requires a full probability distribution over the supplied options. Validation rejects any response missing a probability for any option.

So the data needed to measure calibration is collected on all 132,422 requests. It simply is not scored that way. No Brier score, no expected calibration error, no reliability curve.

The distributions are sitting there in the results files. Somebody could compute calibration from a completed run without rerunning anything.

How the Index Is Actually Computed

Worth walking through, since the formula decides what the number rewards.

Five areas, equal weight

Every benchmark uses its native metric. Those scores average into five equal-weight areas: knowledge plus reasoning, language understanding, retrieval plus classification, tool use, along with human judgment.

The Decision Index is the mean of those five areas on a 0 to 100 scale. Equal weighting means a model strong at retrieval cannot buy its way past weakness in judgment.

The area reshuffle

Area membership follows the leaderboard rather than the raw suite, so six interactive environments sit outside the scored panel. Those six are MiniWoB++, ScienceWorld, Boxoban, RTFM, Hanabi plus Codenames. ChessBench folds into knowledge and reasoning. The former single language area splits into language understanding against retrieval plus classification.

Those choices are documented, which lets an outsider argue with them. That is the useful part of publishing a method rather than a chart.

Chance normalisation

The kit draws random baselines from frozen values, using exact expectations where possible plus Monte Carlo with standard error below 0.001 for F1 metrics.

Skill takes raw minus random, divides by one minus random, then clips the result. So a model scoring 55 on a benchmark where guessing gets you 50 earns very little credit.

The F1 penalty

Multi-label benchmarks use a conservative F1 against a fixed label universe. The missing-label penalty reads as two true positives over two true positives plus false positives plus false negatives plus missing.

A model that skips labels gets charged for it. Again, harder than the easy alternative.

What a Calibration Test Would Look Like

Since this keeps coming up, here is the actual procedure, because it is not complicated.

The method

Take a completed run. For every request, record the probability the engine assigned to the option it picked, alongside whether that pick was correct.

Sort the predictions into buckets by reported confidence, meaning everything between 50 and 60%, then 60 to 70%, upward. For each bucket, compute what share came back correct.

Plot bucket confidence against bucket accuracy. A perfectly calibrated model traces the diagonal.

The single number

Expected calibration error takes the weighted average gap between confidence plus accuracy across buckets. Brier score does something similar while rewarding sharpness, meaning confident predictions that turn out right.

Either gives you one figure comparable across engines, exactly as the Decision Index gives one figure for accuracy.

Why it would change the conversation

A model at 60 on the Decision Index with tight calibration is more useful in software than a model at 70 whose confidence numbers wander. The first one you can build thresholds on. The second one you cannot.

That distinction is the entire argument for typed decisions over a chat model with a parser bolted on, plus it remains the thing no published benchmark scores.

Who could run it

Anyone holding a finished results file. The probabilities are recorded per request, so the analysis is a script over data that already exists rather than another 132,422 API calls.

The Tweet Rounded, the Repo Is Precise

Small thing, worth getting right, because the rounding runs in a flattering direction.

What the announcement said

The launch post describes 35-plus benchmarks plus 130,000 questions asked of each model.

What the repository says

Thirty-seven benchmarks in the suite, of which nineteen form the scored panel. The remainder are display-only, meaning they get scored plus reported without entering the index.

132,422 requests, which rounds to 130,000 fairly enough.

Why the panel number matters more

Nineteen scored benchmarks is still a substantial evaluation. Describing it as 35-plus invites a reader to picture roughly double the panel actually driving the headline figure.

The repository documents this openly. Anyone reading past the tweet finds it immediately, which is the same pattern we keep running into: the qualifier exists, it just does not travel.

Cua-Bench-S1 Takes a Different Angle

The second benchmark targets a narrower question.

What it covers

The announcement covers decision models for computer use specifically, rather than general typed decisions. The team shipped two checkpoints alongside the benchmark, a nano model plus a 4B.

Releasing your own models with a benchmark you designed invites the same criticism TypeSafe attracted, which is that the test author has a horse in the race. Shipping the models openly at least lets others check.

Why computer use is the right narrow test

Computer use is the clearest place decision speed matters. An agent classifying the next click needs an answer in milliseconds, plus it needs to be right often enough that the loop makes progress.

We saw community builds land on exactly this shape within days of Jev’s launch, including one costing roughly $0.0002 a step.

Still no calibration

Nothing in the announcement suggests calibration measurement either. A computer-use benchmark scores task completion, which is the correct thing to score for that use case, plus it leaves the confidence question where it was.

Why This Keeps Not Getting Tested

Two benchmarks in a day, neither touching the central claim. That pattern deserves an explanation rather than a complaint.

Calibration is boring to measure

Accuracy produces a leaderboard. People share leaderboards. A reliability curve produces a diagram that says your 70% bucket came back at 62%, which is useful plus unshareable.

It needs the right dataset

Measuring calibration well means enough examples in each confidence bucket to say anything. A model confident on nearly everything gives you a mountain of data at 95% plus almost nothing at 40%.

That is solvable with a big enough suite. The Decision Index has 132,422 requests, which is comfortably big enough.

Nobody’s job

TypeSafe has no reason to publish a test that might undercut its own framing. The clone authors are competing rather than auditing. Leaderboard operators optimise for rankings people want to read.

So the gap persists by default. We wrote the same thing on Friday plus it remains true after two benchmarks.

The shortest path

Whoever has a completed Decision Index run already holds every probability distribution needed. Bucket the predictions by reported confidence, then check what share of each bucket came back correct.

That is an afternoon of work against data that already exists on disk.

Three Days, Two Benchmarks, One Pattern

Step back from Jev for a moment, because the timing says something about how this category evaluates itself.

The sequence

TypeSafe launched on September 15 with a benchmark written in house. By the 19th that benchmark’s figures had reached roughly 901,000 views across eight posts, with none of them carrying the three caveats the company published. By the 22nd, two outside teams had built their own.

Seven days from launch to independent evaluation is fast by any standard. The open-source reflex in this category works.

What it fixed plus what it did not

The reproducibility problem is solved. Anyone can now run a frozen suite against any engine, submit results, then have them re-scored by somebody else.

The framing problem is untouched. Both new benchmarks measure what benchmarks conventionally measure, which is whether the answer was right. Neither asks the question specific to this product category.

Why the convention wins

Evaluation tooling inherits its shape from what came before. Decision engines are new, so the natural move is to score them like classifiers, since that is what the existing harnesses do.

Calibration needs somebody to decide the category deserves a different measurement. That is a judgment call rather than an engineering problem, plus it costs a weekend rather than a research budget.

The same shape as the licence problem

We keep finding this. A community moves fast, builds real things, then leaves the boring verification step for somebody else. Four of six Jev clones shipped without a usable licence. Two benchmarks shipped without a calibration metric.

None of it is carelessness exactly. It is what happens when building is more interesting than checking, which it always is.

What the Benchmarks Do Settle

Plenty, actually. This is not a nothing result.

Comparison becomes possible

Before Monday, comparing Jev against alternatives meant trusting vendor figures. Now there is a frozen suite anyone can run against any engine, with a documented metric plus a submission process.

The eight or so open reimplementations we counted last week can now be measured against the thing they copy, on the same panel, by whoever cares to run it.

The scoring is harsh in useful ways

Counting unanswered as wrong plus refusing to truncate means a model that quietly fails on long inputs cannot hide it in a footnote. That is precisely the kind of failure a vendor benchmark tends to smooth over.

Reproduction has a reference point

The parity check gives anyone a way to confirm their pipeline matches before submitting a result. Without that, every independent run is arguing about tooling rather than models.

It is MIT

The kit ships under MIT, in a category where four of six Jev clones shipped without a usable licence. Anyone can fork this, add a calibration metric, then publish.

What You Should Actually Do

If you are choosing a decision model

Use the Decision Index for accuracy comparison. It is the best available evidence on which engines answer correctly across a broad panel.

Then run your own calibration check before you trust a confidence number in production. The benchmark does not do it for you.

If you have API access plus an afternoon

Run the suite, then compute calibration from the results file. Bucket by reported confidence, compare against actual hit rate, publish the curve.

That single output would settle a question four articles plus two benchmarks have now left open. Whoever publishes it first owns the answer.

If you are building on typed decisions

Test thresholds against your own labelled examples. TypeSafe advises this themselves, which is worth repeating because almost nobody quotes it.

Our Jev explainer covers what the model does plus what it costs. The clone roundup covers the open alternatives.

If you run a leaderboard

Add the metric. The Decision Index already collects everything needed, so a calibration column costs a scoring function rather than a redesign.

Whoever adds it first becomes the reference for a property this entire product category is sold on, which is a better position than being one more accuracy table.

If you are repeating a benchmark number

Say which panel it came from. Nineteen scored benchmarks is not 37, plus the repository is clear about the difference even when a summary is not.

The Part Worth Keeping

Two independent benchmarks in one day, arriving three days after we wrote that none existed. That is the system working. Both better built than the vendor test they improve on. One fully reproducible by a stranger with a laptop.

Neither answers the question Jev is sold on.

The Decision Index asks every engine for a probability on every option, records 775,202 answer fields, then scores whether the top pick was correct. The data needed to check calibration is sitting in those files right now.

Somebody with a completed run could produce the reliability curve this afternoon without rerunning a single request. It needs a scoring function plus a plot. Until they do, the fastest decision model in the industry still has an unverified answer to the one question that decides whether you can build on it.

Charts and Blocks

What the Decision Index covers

Vendor benchmark against independent benchmark