Google Research Says You Can Improve An AI Agent By 11.5 Points Without Touching The Model

Harness² is a framework from Google Research for improving what it calls the agent harness, meaning the system around a model rather than the model itself. It keeps the weights frozen and refines eight things instead, running from the system prompt and guardrails through memory and tool guidance out to subagents and plugins. The reported gains are up to 11.5 points on WorkBuddy-Bench, 4.5 on JobBench and 11.8 percentage points on LAB’s partial pass rate, measured on GLM-5 and Gemini 3.5 Flash. It’s Apache 2.0, it took 11,020 stars in 12 days. Nothing in the 216 posts we scanned this morning mentioned it. The demo runs with no model credentials at all. The README also carries a warning most people will skip, which is that its executor runs with unrestricted network access.

Best for anyone already tuning their own agent setup. Not ideal for anyone hoping for a one-command upgrade.

Everybody is arguing about which model is best. A paper out of Google Research spent its time somewhere else.

It left the model alone and rebuilt the scaffolding around it. On one benchmark that was worth 11.5 points, with weights that never moved.

What An Agent Harness Actually Means Here

The README gives a definition worth borrowing, because the word gets used loosely everywhere else. An agent harness is “the system around a model that enables it to act as an agent.”

That’s the prompt, the rules, the memory, the tools and everything else you assemble so a model can do work rather than answer questions. Harness² treats that assembly as the thing to optimise.

Here’s what the repository actually contains, read off the GitHub API in this morning’s scan and the project’s own README.

The last row matters. This is a Google Research output with a disclaimer saying it “is not an officially supported Google product,” which is the standard phrasing for research code rather than a product launch.

What 11,020 Stars In 12 Days Actually Tells You

The star count is the least interesting number on that list. Here’s the one that matters.

987 forks against 11,020 stars is a fork ratio of 9.0%. That’s the share of people who took a copy rather than just saving the link. On a repo this young it’s the only free signal that separates attention from use.

For scale, here’s where that sits against the other research code we’ve looked at this month.

Under 12% reads as a bookmark pile. Harness² sits at 9.0%, which puts it in the honest middle: a small group has it cloned and most of the eleven thousand are saving it for a weekend that may never come.

One caveat on all four of those. A fork ratio on a 12 day old repo is noisy, because stars arrive faster than forks do in the first week. If anything the 9.0% will drift upward as the people who saved it get around to cloning, so treat it as a floor rather than a verdict.

Which fits the install list. This needs Linux with Python 3.12 and uv to start, then more tooling again depending on which benchmark you want, so the distance between starring it and running it is an afternoon wide.

The Growth Rate, Stated Carefully

Here’s where I’ll be pedantic, because the number invites a mistake. 11,020 stars across 12 days works out to a since-launch average of about 918 a day. That’s a since-launch average rather than a measured daily rate, because we have one reading instead of a series. Research repos usually front-load hard in the first 48 hours, so the real curve is almost certainly steeper at the start and flatter now.

What the average does tell you is how quietly this landed. Eleven thousand people found the repo in under two weeks. None of the 216 posts in this morning’s scan mentioned it, which is a gap between adoption and discussion I don’t think I’ve seen this wide before.

For a pure reporting piece that’s the whole justification for writing it up. Something with eleven thousand stars and no coverage means the people who would benefit most haven’t heard of it yet.

The Eight Things It Treats As Editable

This list is the useful part even if you never install anything.

System prompt. Guardrails. Memory. Tool guidance. Subagents. Skills. Scripts. Plugins.

Read it again with whatever agent you already run in mind. For Claude Code that’s roughly your CLAUDE.md, your permission rules, your saved context, your tool descriptions, your subagent definitions, your skills folder, your hooks and your MCP servers.

Most people tune two of those. The prompt, because it’s obvious. Maybe the tool list too, because something broke once. The other six sit at whatever default they arrived with.

That’s the claim buried in this research. Those six aren’t decoration. They’re parameters. The paper measures what happens when you treat them that way.

Our roundup of GitHub repos that turn Claude into something you use daily is mostly a list of things that modify exactly these components. Nobody framed them as one surface before.

How The Recursion Works

Two loops. The distinction between them is where the design gets interesting.

Task-level recursion refines the execution harness for the task in front of it, through a step the README calls Propose, Probe and Compose. The notable constraint is that it runs without ground-truth feedback. It isn’t being told the right answer and tuning toward it.

Harnessing-level recursion is the longer loop. It updates a persistent improvement harness with the harnesses that got adopted plus what the README calls evidence-supported lessons. So the system keeps what worked and carries it forward.

Both can run in parallel or sequentially. The README says a larger task-level recursion budget produces further gains.

That three step loop is worth unpacking, because the verbs describe something anybody who tunes a setup by hand is already doing badly. First you propose a change. Trying it is the probe. Then whatever survived gets composed into the version you keep.

The difference is that most of us skip the probe. We change a prompt, the output looks better, we move on. No second run, no comparison, no record of what we changed.

The no-ground-truth detail is the one worth sitting with. A system that improves by being told the right answer is doing something closer to fitting. A system that improves by proposing variations and probing them is doing something you could point at your own work, where nobody has an answer key.

That’s also what makes the harnessing-level loop more than bookkeeping. It keeps the harnesses that got adopted plus the lessons the evidence supported, so run twenty tasks and the twenty-first starts from everything the first twenty learned. The improvement compounds in the scaffolding rather than in the weights.

What The Numbers Rest On

A single Harness² step improved both models over their base harnesses, across three benchmarks and four model-harness configurations.

The headline figures are up to 11.5 points on WorkBuddy-Bench, 4.5 points on JobBench and 11.8 percentage points on LAB’s partial pass rate.

Two things to hold alongside those. They’re the authors’ own measurements and nobody outside has reproduced them yet. And they’re measured on GLM-5 and Gemini 3.5 Flash, so nothing here tells you what it does to Claude or GPT.

Testing a third model is the obvious experiment for anybody with a weekend. The executor runs on OpenCode by default and Codex with a flag, so pointing it at a different setup is a configuration question rather than a rewrite.

Model selection is wired through environment variables. The README has you drop a Vertex AI project and model ID into a .env file, which is how GLM-5 and Gemini 3.5 Flash got measured. Swapping that is a text edit. Whether the refinement loop finds the same gains on a model trained differently is the actual open question. That one gets answered by somebody posting a table rather than by another paper.

What The Benchmarks Measure

JobBench, LAB and WorkBuddy-Bench are agent task suites rather than reasoning tests. What’s being scored is whether an agent finished work, not whether a model answered correctly.

LAB reports a partial pass rate, which is why its figure comes in percentage points while the other two come in benchmark points. Those units aren’t interchangeable. The README keeps them separate and so should anybody quoting the numbers.

That distinction matters more than it sounds. A gain on an agent task suite means the scaffolding helped something get done. A gain on a reasoning benchmark would mean the model got smarter, which is explicitly not the claim here.

The size of the number is what makes it credible. An 11.5 point gain from scaffolding sits in the range where a claim is big enough to matter but small enough to be plausible. Papers promising 40 point jumps from prompt engineering have trained everybody to squint. This number lands well below that line.

The Warning In The README

Three sentences in the limits section deserve more attention than they’ll get.

Executor network access is unrestricted. Codex runs with its sandbox disabled. Public benchmark rubrics may be accessible online.

Put those together and the implication is direct: if you evaluate without blocking network access first, the model under test can go and read the grading criteria. The README says to configure network restrictions before evaluation, which is the authors flagging their own measurement hazard in their own documentation.

That’s good practice and it’s also a reminder of what you’re running. An agent framework whose executor can reach anything, with the sandbox off, pointed at your machine.

We ran into the same question from the other direction in the ESP32 iPhone agent piece, where runtime capture operates with your user permissions. Different tool, same structural answer: put it somewhere you don’t mind losing.

There’s a reproducibility note too. Judging and reporting have to match the refinement run exactly: same benchmark and executor, same subset and seed, same --k and --mode. Change one of those and your numbers compare to nothing anybody else published.

What It Costs To Actually Run

The demo is free. harness2-demo runs with no model credentials, which is an unusually honest way to ship research code, since it lets you see the machinery before you spend anything.

Past that, the bill starts. Model-backed runs hit paid APIs. The README recommends starting with small task selections for exactly that reason. Recursion means running a task more than once by design, so the cost scales with your --k budget rather than with the number of tasks.

The dependency list is longer than most. Linux, Python 3.12 and uv to start. Then Node.js 20 or newer, the Google Cloud CLI and benchmark dependencies for anything model-backed. LAB wants Pandoc and Poppler. WorkBuddy-Bench wants Docker. Then gcloud auth application-default login before anything runs.

So budget an afternoon rather than a command. Which is fine for research code and worth knowing before you start.

If you’re pricing the model side of an experiment like this, our Claude Pro review covers what the subscription tiers include, though recursive benchmark runs are an API workload rather than a subscription one.

Why Nobody Posted About It

Our scan flagged this repo SILENT, meaning zero mentions across 216 posts collected this morning. For something with eleven thousand stars that’s unusual enough to explain.

Three reasons, as far as I can tell.

The name is the first one. Harness² is a word most people use loosely and nobody searches for. There’s no demo video, no landing page and no hook. Compare that to anything with “agent” in the title and a gif.

The second is that it’s research code wearing a research-code disclaimer. “Not an officially supported Google product” is accurate and it also reads as a warning label to anybody deciding what to write about.

The third is that the finding is boring in a way that’s hard to post. “We improved the thing around the model” has no villain, no price change and nobody to be angry at. The repos that generate 216 posts worth of noise are the ones where somebody is either winning or getting ripped off.

None of that makes the result smaller. It does mean you’ll hear about it late, which is the pattern we keep running into. The interesting work ships quietly and the loud stuff is usually a pricing page.

Why This Lands Now

Timing is why this one lands now instead of in a citation list.

For two years the answer to “my agent isn’t good enough” has been to wait for a better model. That worked, because better models kept arriving on a schedule fast enough to outrun anybody’s tuning.

What changed is that the frontier models converged. The gap between the top few on most practical work is now small enough that switching costs more attention than it returns. We watched that play out in the prompt cache piece last week, where the thing determining what a session costs was configuration rather than model choice.

So the variable moved. If the models are close, the difference between two people getting different results from the same model is everything wrapped around it, which is precisely what Harness² measures.

The agent harness has been the unexamined half of every setup. Everyone has one. Almost nobody tuned it on purpose.

Verified vs Asserted

What This Means If You Never Install It

Most readers won’t run a benchmark suite. The idea still transfers. Here’s the honest version of how.

Pick one of the eight components you’ve never deliberately changed. Memory is usually the one, because it accumulates rather than gets designed. Tool guidance is the other, since most people write a tool description once and never revisit it after watching the agent misuse it.

Then change it on purpose and run the same task twice. That’s the whole method at human scale, minus the automation.

What the paper adds is evidence that there’s more headroom here than people assume. An 11.5 point swing from scaffolding isn’t a rounding error. If a research team got that out of automated refinement, somebody who knows their own work by heart can probably find a slice of it by hand.

The skills side of this is already a market. We counted it in the piece on the agent skills marketplace, where the bundles people install are doing exactly what Harness² calls the skills component. Nobody was measuring whether they help.

What You Should Actually Do

If You Want To See It Work

Run the demo. It needs Linux, Python 3.12 and uv. It costs nothing because it needs no model credentials. That gets you the machinery without the API bill, which is the cheapest way to decide whether the rest is worth your afternoon.

If You Want To Benchmark It Properly

Block network access first. The README says public benchmark rubrics may be reachable and the executor runs unrestricted, so an unblocked run can quietly measure the wrong thing. Then keep the run settings identical between refinement and judging. Same benchmark and executor, same subset and seed, same --k and --mode.

If You Just Want A Better Agent Today

Take the list of eight and audit your own setup against it. Our OpenClaw deep dive walks through most of these components for a different agent. The exercise is the same whichever one you run. You’ll probably find three you’ve never touched.

If You Run Claude Code Specifically

The mapping is close enough to be useful. CLAUDE.md is the system prompt layer, permission rules are guardrails, skills and subagents are themselves, MCP servers are plugins. Hooks are scripts. Nobody has tested whether Harness² transfers, so that’s an open question rather than a recommendation.

The part I’d actually do first is write down what your eight look like right now. Not change anything, just list them. Most people can’t name their own tool guidance or say when they last edited it, which is the whole problem in one sentence.

Then pick the one with the emptiest answer and give it an afternoon. Memory and tool guidance are usually where the slack is, because both accumulate by accident rather than by design.

The Part Worth Keeping

A research team at Google froze the model and improved the agent by 11.5 points on one benchmark. The weights never changed. Only the scaffolding did.

Whether that exact number holds up is for somebody with a GPU budget to check. The reframe underneath it doesn’t need reproducing, because it just names something everyone running an agent already half knew.

You have a harness. You assembled it out of defaults, one fix at a time. You’ve probably never looked at it as a single thing with eight parts. That’s where the next improvement is sitting. It’s also cheaper than waiting for the next model.

Charts and Blocks

The Eight Components And What They Map To

Reported Gains From One Harness² Step