The scaffold is half the result

SWE-bench Verified has an official leaderboard, maintained by the people who built the benchmark. It runs every submission through mini-SWE-agent, their own reference harness: the same tools, the same loop, the same budget for everybody. Thirty of the models in our table appear on it.

Every single one scores lower there than its maker reported.

Not most of them. Not the ones from the labs you would expect. All thirty, in the same direction, by margins that go from a rounding error to a quarter of the scale.

Gemini 2.5 Flash is the extreme case: 54 in Google's announcement, 28.7 on the reference harness. Nearly half the score disappears. GPT-4.1 loses fifteen points, GLM-4.6 loses thirteen. At the top of the table the gaps are politer — Claude Opus 4.5 gives up four — but they are still there, and they still only go one way.

Thirty out of thirty pointing the same direction is not sampling noise. It is structural, and the structure is not what it looks like.

This is not cheating

The obvious reading is that labs inflate their numbers. That reading is wrong, and taking it means missing what the gap actually tells you.

A SWE-bench result is not a property of a model. It cannot be. The benchmark does not ask a question and grade an answer — it drops the model into a repository with a bug report and waits to see whether the tests pass afterwards. Everything between the issue text and the final patch is machinery somebody had to build.

That loop is the scaffold. How many turns does the model get before it is cut off? Can it run the test suite itself, or does it only find out at the end? When a patch fails, does it see the traceback? Can it revert and start over, or does it dig deeper into the hole? How many parallel attempts are allowed, and who picks the winner?

Every one of those decisions moves the score, and none of them are the model. A lab building the harness for its own model knows how that model behaves when it is confused, what its context window does under pressure, which prompt shape makes it check its work. They tune for it, because that tuning is the product. When Anthropic reports a SWE-bench figure, it reports what its model achieves inside the agent it ships — a legitimate thing to measure, and closer to what you get if you use their tool.

mini-SWE-agent is deliberately the opposite: a small fixed loop that treats every model identically. It is not trying to get the best out of anybody. It asks what a model brings when nothing else is helping it.

Both numbers are honest. They answer different questions, and both questions are worth asking. The gap between them is roughly what the scaffold is worth — twenty-five points on Gemini 2.5 Flash, four on Opus 4.5. That difference between the differences is itself informative: a model that gains enormously from a good harness is telling you it needs one.

Pass, fail, and nothing in between

Some of the volatility comes from how brutally SWE-bench scores. There is no partial credit anywhere in it. Each instance asks two separate questions, and only one answer to both counts.

Because the outcome is binary, small differences in the loop tip whole instances from one side to the other. A model that would have found the fix on turn twelve scores zero in a harness that stops at ten. That is why harness quality converts into points so efficiently, and why the gap is widest for the models that need the most turns.

Where the column sits now

Worth keeping in view: SWE-bench Verified is filling up. Here is every figure the makers have published on it, unnamed, against the full scale.

The marks crowd the right-hand end and the space above the leader has nearly closed. When a benchmark gets this full, the difference between two frontier models stops being a difference in capability and becomes a difference in harness engineering — which is exactly the effect this post is about. Saturation and scaffold sensitivity arrive together.

What we changed

The table now carries both figures in the same cell: the maker's in white, the leaderboard's in blue, with the harness named in the tooltip. Where the two disagree you see the disagreement rather than a curated single number.

There is a bonus. Several models never reported SWE-bench at all — o3, o4-mini, GPT-4o, both Llama 4 sizes, Gemini 2.0 Flash, Qwen2.5-Coder. Their cells used to be empty. The reference harness ran them anyway, so those cells now hold figures nobody had a marketing reason to produce: o3 at 58.4, o4-mini at 45.0, GPT-4o at 21.6, Llama 4 Scout at 9.1.

Those are among the most trustworthy numbers in the column, precisely because nobody chose to publish them.

The habit to build

When you meet a coding benchmark number, the useful question is not is this high but whose harness produced it. A ten-point jump in an announcement can be a better model, a better agent, or a longer leash — and only one of those follows you home if you are calling the API yourself and writing your own loop.

Ask which harness. Ask how many turns. Ask what the budget was. If the answer is not published, treat the number as a claim about a product rather than a measurement of a model, because that is what it is.