CLM-8B Beat Jev on Speed. Only 40 Million of Those Parameters Are Trained.

A Stanford-affiliated team released CLM-8B on September 23, a System One model claiming up to 9x lower latency than Jev at comparable zero-shot performance. The weights are Apache 2.0, the repository took 766 stars in a day, plus it serves behind a TypeSafe-compatible API so an existing Jev request replays against it. Three things the launch posts skip. The 8B refers to a frozen Qwen3-8B encoder, with roughly 40 million trainable parameters across two projection heads and a reference checkpoint around 75 megabytes. The speedup varies by condition, running 9x zero-shot, 13x at about a thousand candidates, then 4.1 to 5.7x fine-tuned. And the headline coding scores come from fine-tuned heads on held-out subsets of 38 plus 30 tasks. Best for agent loops scoring many candidates against a changing state. Not ideal if you wanted a drop-in model file.

Nine days after Jev launched, a team including Christopher Ré plus Azalia Mirhoseini published a competitor claiming up to 9x lower latency at comparable performance.

It ships Apache 2.0 with weights, data plus serving code. It answers the same three question shapes. An existing Jev request replays against it through a compatible API.

It also contains one striking claim about Jev that nobody repeated, plus a name that oversells what you are actually downloading.

What CLM Actually Is

Verified against the repository, the model card plus the GitHub API on September 24.

The author list runs Jacky Kwok, Hangoo Kang, Tarun Suresh, Jon Saad-Falcon, Marco Pavone, Christopher Ré plus Azalia Mirhoseini, described as joint work with Hazy Research.

That is a serious group. It separates this release from the anonymous reimplementations we counted last week. Four of those shipped with no usable licence at all.

The 8B Is Doing Something Misleading

Start here, because the name sets an expectation the architecture does not meet.

What you actually train

Per the model card, CLM-8B is two small projection heads sitting on a frozen Qwen3-8B encoder. One head embeds states, one embeds actions. Roughly 20 million trainable parameters each.

The published reference head checkpoint runs about 75 megabytes. The 8 billion parameters in the name belong to a Qwen model released by Alibaba, which the team did not train plus does not modify.

Why that matters practically

Two consequences follow. The thing you download is tiny, which helps a lot. Fine-tuning it for your own task is cheap, since you are training 40 million parameters rather than 8 billion.

The second consequence is less comfortable. The heads require Qwen3-8B last-token-pooled embeddings, so the backbone cannot be swapped. You are locked to that specific encoder.

Is the name unfair

Not exactly. Plenty of adapter releases name themselves after the base they attach to, plus the model card states the architecture plainly in its first paragraph.

The launch posts do not. Somebody reading only X comes away thinking an 8 billion parameter model was trained to beat Jev, when the trained artefact fits on a floppy disk’s worth of modern storage twenty times over.

How the Contrastive Objective Works

Worth explaining, since it is the actual difference between these two models plus it is simpler than the name suggests.

Pulling and pushing

CLM trains two encoders on a large dataset with a contrastive objective. Each state gets pulled toward the action actually taken, then pushed away from every other action in the batch.

Do that across enough examples and the geometry does the work. Correct actions sit close to their states in embedding space. Wrong ones sit far away.

Scoring becomes arithmetic

At inference, CLM scores each candidate by taking the dot product of the state embedding plus the action embedding. That is a multiply-and-add over two vectors.

Compare that with a generative judge, which reads the state plus the candidates together, then produces tokens describing its verdict. One approach runs a model. The other runs a similarity comparison.

Why caching falls out of it

The two encoders are separate, so an action’s embedding never depends on the state. Embed your action set once, keep the vectors. Every subsequent turn then costs one state embedding plus a handful of dot products.

A generative judge cannot do that. It has to read state plus candidates together on every single call, which is why the authors describe Jev as supporting state caching only.

The trade you are making

Contrastive scoring gives you a ranking over options you supplied. It cannot reason about an option it was never given, explain itself, or handle a question that does not reduce to picking from a set.

That is fine for the job. It is worth knowing the ceiling is structural rather than a matter of scale.

Scaling Laws Are the Quiet Part

One finding in the release matters more for the category than for this model.

What they report

The authors report that test contrastive loss follows a power law against training compute, dataset size, projection head size plus encoder size. They add that these dimensions must be scaled jointly for optimal performance.

Why that is notable

Scaling laws are what turned language modelling from craft into engineering. Once you can predict what more compute buys, you can plan.

Nobody had published them for this model class. TypeSafe describes RLCD without publishing enough to evaluate it, so there was no public curve for decision models at all.

The practical consequence

If the relationship holds, a larger encoder with proportionally larger heads plus more data should improve predictably. That makes CLM a starting point rather than a finished artefact.

It also means the 75 megabyte checkpoint available today is the small version of something, which changes how you read its parity with Jev.

Hold it loosely

These curves come from one team, in a blog post, on their own training runs. Scaling law claims are notoriously sensitive to how the axes are chosen.

Independent replication would settle it, which is the same sentence this category keeps needing.

Three Speedups, One Number Travelled

The 9x figure is real. It is also one of three, with different conditions behind each.

The zero-shot claim

Up to 9x lower latency than Jev on computer-use, gaming plus tool-calling tasks, at comparable performance. Latency measured on an H100.

The caching claim

With around 1,000 candidate actions, CLM runs 13x faster than Jev.

That larger figure comes from the actual mechanism, which is worth understanding. CLM embeds states plus actions separately, so if your action set is fixed and only the state changes, you embed each action once, keep those vectors, then pay for one fresh state embedding plus a cheap similarity comparison per candidate.

The speedup therefore grows with the number of candidates. Few candidates, small gain. A thousand candidates, large gain.

The fine-tuned claim

As a fine-tuned verifier on agentic coding, CLM runs 4.1 to 5.7x faster than Jev.

So the advantage shrinks in the configuration producing the headline accuracy scores.

What to take from that

The speedup is conditional on how many candidates you score against how often your state changes. An agent loop re-ranking a large fixed action set benefits enormously. One making a single yes-or-no call each turn benefits far less.

That is a legitimate architectural advantage rather than a benchmark trick. It just is not one number.

The Training Data Tells You Its Limits

The recipe is published, which is more than Jev offers, plus it explains where CLM should be strong.

Three stages

Pre-training ran on roughly 60 million Nemotron question and answer pairs. Mid-training added about 30 million synthetic hard negatives. Post-training used around 1 million agentic trajectories, all of it documented.

Hard negatives are the interesting middle stage. Synthesising wrong answers that look plausible is how you stop a contrastive model learning shortcuts, since an easy negative teaches it nothing.

What that predicts

Heavy question-and-answer pre-training plus agentic post-training should produce a model good at picking among plausible options in tasks resembling its trajectories.

The reported results match that. Parity with Jev on computer use, gaming plus tool calling, which are all selection problems over a bounded set.

Where it should be weaker

Anything far from the trajectory distribution. A domain-specific decision your agent faces that looks nothing like the post-training data is exactly where a zero-shot contrastive model tends to degrade.

The authors provide fine-tuning documentation for that reason, plus at 40 million trainable parameters tuning is realistic for a small team.

What nobody has tested

Generalisation outside the published task families. Every number reported sits inside computer use, gaming, tool calling or agentic coding.

That is four categories from one team. Anyone deploying this somewhere else is the first person measuring it there.

The Claim About Jev Nobody Repeated

Buried in the repository is a finding sharper than the speed numbers.

What it says

On long-horizon agentic coding tasks, used as a verifier picking the best of several sampled solutions, Jev scores below pass@1.

Pass@1 means taking one sample without selecting. So on those tasks, using Jev to choose among candidates produced worse results than not choosing at all.

Why that is serious

Verification is exactly what people are building on Jev right now. The compaction plugin that took 4,914 stars picks which tool history to keep. The model router decides how much reasoning each turn needs. Both are selection problems on long-horizon work.

If Jev underperforms random selection in that setting, a chunk of the ecosystem we documented is built on a weaker foundation than assumed.

Why to hold it loosely

This is a competitor’s measurement, published on their own benchmark subsets, in a blog rather than a reviewed venue. TypeSafe has not responded.

The tasks are also specific. Long-horizon agentic coding is not computer use or tool calling, where the same document reports parity. Jev failing at one job says nothing about the others.

Somebody independent should check it. That is the third time this month we have written that sentence about this category.

The SOTA Numbers Need Their Conditions

81.6% on DeepSWE plus 87.6% on Terminal-Bench 2.1 are the figures spreading. Both carry qualifiers the repository states openly.

Fine-tuned, not zero-shot

Those scores come from lightweight fine-tuned heads, not the released zero-shot checkpoint. Download CLM-8B, run it untuned, then expect the parity numbers rather than the record ones.

Held-out subsets, not leaderboards

Evaluation ran on 38 held-out DeepSWE tasks plus 30 held-out Terminal-Bench 2.1 tasks. These are not full leaderboard submissions.

Thirty-eight tasks means a single task is worth 2.6 percentage points. The 81.6% figure is 31 correct out of 38.

The candidates came from frontier models

CLM acted as verifier over candidate solutions sampled from Opus 5 for DeepSWE plus Fable 5 for Terminal-Bench 2.1.

So the pipeline is a frontier model generating options with a cheap model picking between them. That is a sensible architecture. It also means the score reflects both halves.

None of this is hidden

Every qualifier above sits in the project’s own documentation. We keep finding the same shape, most recently when TypeSafe published three caveats that roughly 901,000 views of their benchmark failed to carry.

The researchers documented their conditions. The tweet had a character limit.

Still Nobody Is Measuring Calibration

Third article in a week ending at the same place.

What is missing again

CLM scores candidates by dot product between state plus action embeddings, returning probabilities across three question types covering a boolean, a choice from a set plus a score on an ordered rubric.

Same output shape as Jev. Same absence of any published measure of whether those probabilities track real accuracy rates.

Why a contrastive objective makes it more interesting

InfoNCE trains embeddings so the correct action sits closer to the state than the alternatives. That produces a ranking. Whether the resulting probabilities are calibrated is a separate question, plus a contrastive objective gives no particular reason to expect they are.

A model can rank perfectly while reporting 90% confidence on everything. Ranking is what these benchmarks measure.

The tooling now exists

The Decision Index published this week collects a full probability distribution across every option on 132,422 requests, then scores only accuracy. CLM serves behind a compatible API.

So somebody could run CLM through that suite, then compute reliability curves for both models from data the suite already captures. Two afternoons, one comparison, question settled.

What It Means That This Came From a Lab

Nine days from a commercial launch to an academic competitor is fast, plus the shape of the response matters.

Nine days is the number to sit with

Jev entered limited early access on September 15. CLM published on September 23.

Eight days from a funded commercial launch to an open competitor matching its interface. Not matching its product, its reliability or its support, but its interface plus its claimed zero-shot performance.

Open where Jev is closed

Jev sits behind an early access waitlist with an unpublished training method. CLM ships weights, data recipe plus serving code under Apache 2.0, with the architecture documented in the first paragraph of its model card.

A team can self-host it on one GPU today.

Compatible on purpose

Serving behind a TypeSafe-compatible API is a deliberate competitive move. Anyone who built against Jev’s interface can try CLM by changing an endpoint.

We saw the same instinct across the open reimplementations, where the interface was the spec even when the weights were not available.

What it costs TypeSafe

Two years of stealth plus $40 million of funding went into a model whose interface a research group matched in nine days with a 75 megabyte adapter over an open encoder.

That does not mean the products are equivalent. It does mean the moat was never the interface.

What You Should Actually Do

If you score many candidates per turn

Test it. The caching mechanism targets exactly that shape, plus Apache 2.0 with a compatible API means swapping an endpoint rather than rewriting.

Measure on your own candidate counts. The 9x assumes conditions that may not be yours.

If you are already running Jev in a verifier role

Read the repository’s verifier section, then run your own comparison on your tasks. The claim that Jev scores below pass@1 on long-horizon coding is a competitor’s finding, though it is specific enough to check.

If it holds on your workload, you are paying for selection that is hurting you.

If you are running a leaderboard

CLM serves behind a TypeSafe-compatible API, so adding it to an existing decision-model suite costs an endpoint entry rather than an integration.

Two open models with the same interface makes a like-for-like comparison possible for the first time in this category.

If you want the record scores

Fine-tune the heads. The published SOTA figures use tuned heads rather than the zero-shot checkpoint, plus tuning 40 million parameters costs very little.

If you are choosing between these

Neither has published calibration. Both publish accuracy. Our Jev explainer covers what TypeSafe sells, plus the Decision Index piece covers what the independent benchmarks measure.

Run the reliability curve yourself before a confidence number gates anything that matters.

The Part Worth Keeping

A seventy-five megabyte adapter over a frozen open encoder now matches a venture-funded closed model on its own interface, at lower latency, under Apache 2.0.

That 8B in the name belongs to Alibaba. Speed depends on how many candidates you score. Record coding numbers come from tuned heads on thirty-eight and thirty task subsets. Every one of those qualifiers is in the documentation plus absent from the posts carrying the claim.

The finding worth chasing is the other one. A research group says Jev scores below pass@1 as a verifier on long-horizon coding, which means selecting with it beats not selecting at all. Half the ecosystem built on Jev this month does exactly that job.

Somebody outside both camps should check. Nobody has yet, which is how every article about this category has ended so far.

The difference this time is that one side shipped its weights.

Charts and Blocks

Three speedups, three conditions

CLM against Jev