AI - Paski.dev

Benchmarks

Seven columns decide the order of the table. This is what each one is trying to find out, what a single item actually looks like, and what to distrust about the number that comes out of it.

Two shapes

Every benchmark here is one of two things, and the difference decides how much of a score belongs to the model at all.

Quiz

  1. question
  2. model answers
  3. grader checks
  4. score

One shot per question, graded automatically. Cheap to run and easy to compare — and the first shape to saturate, because there is a ceiling at 100.

Agentic task

  1. goal + environment
  2. agent acts in a loop
  3. result is executed
  4. score

The model works inside a scaffold — tools, retries, a token budget — and is graded on the outcome rather than the text. The scaffold is part of the result, which is why the same model scores differently in different hands.

A quiz answer is the model's. An agentic result belongs to the model and the scaffold around it — which is why the same model can appear twice in the table with two different numbers.

MMLU-Pro

the paper ↗

Quiz · 12,032 questions across 14 subjects

MMLU had become useless as a discriminator: the frontier was scoring in the high eighties, the remaining gap was partly mislabelled answers, and four options meant a model that knew nothing still got a quarter of them right. MMLU-Pro rebuilds it to keep separating models that have already run out of room on the original.

What an item is made of

  • Ten options per question instead of four, which pushes the reward for blind guessing from 25% down to 10%.
  • Questions filtered and rewritten to need reasoning rather than recall — the ones MMLU could be beaten on by memorising were the first to go.
  • Fourteen subjects, from maths and law to health and engineering, so one strong domain cannot carry the average.
MMLU, 4 options 25%
MMLU-Pro, 10 options 10%
What a model that knows nothing scores, by design.

How one run goes

  1. question
  2. model answers
  3. grader checks
  4. score

Scoring

Plain accuracy over all questions, usually with chain-of-thought prompting. No partial credit, one number.

Where the field sits

0 100

32 published figures · best 88 · 12.0 points of headroom left

Every figure the makers have published on this benchmark, unnamed. A clot against the right edge means the benchmark has stopped separating anyone.

Running it yourself

  1. 1. Load the set: datasets.load_dataset("TIGER-Lab/MMLU-Pro")
  2. 2. Present each question with its ten options and ask for a single letter.
  3. 3. Score exact matches and report the mean.

What to watch

Prompt format moves this benchmark by several points, and labs almost never publish theirs. Two MMLU-Pro numbers from two labs are closer to two different exams than to one shared ruler.

GPQA

the paper ↗

Quiz · 198 questions — the Diamond subset

To find out whether a model holds expert knowledge, rather than whether it can look things up. Every question is written by someone with a PhD in the field and then screened against people who are highly skilled but working outside their speciality — with a search engine and unlimited time. What survives is what the internet does not readily hand over.

What an item is made of

  • Biology, physics and chemistry, at the level of a working researcher.
  • Four options, so the floor is 25% rather than zero.
  • Diamond is the hardest slice: the questions domain experts got right and non-experts got wrong even while searching.
  • Only 198 of them, which is the whole problem with it now.
448 questions written by PhDs in the field
198 Diamond: experts got them right, non-experts with the web did not
Every stage throws work away. What is left is the part where knowing the field beats looking it up.
Domain expert 65%
Non-expert + web 34%
Blind guessing 25%
The validation that defines the Diamond subset: expertise separates, searching does not.

How one run goes

  1. question
  2. model answers
  3. grader checks
  4. score

Scoring

Accuracy, pass@1, normally averaged over several runs because the sample is small enough that one question is worth half a point.

Where the field sits

0 100

67 published figures · best 94.6 · 5.4 points of headroom left

Every figure the makers have published on this benchmark, unnamed. A clot against the right edge means the benchmark has stopped separating anyone.

Running it yourself

  1. 1. Load the set: datasets.load_dataset("Idavidrein/gpqa", "gpqa_diamond") — access is gated, deliberately, to slow down contamination.
  2. 2. Ask for one option, and give the model no search tools.
  3. 3. Average across trials rather than reporting a single run.

What to watch

Saturated. The leaders in our table are separated by about a point, which on 198 questions is two questions — inside the noise. Anthropic has said outright that it considers GPQA Diamond saturated and plans to stop reporting it.

AIME 2025

the 2025 problems ↗

Quiz · 30 problems — AIME 2025, papers I and II

Competition mathematics with nothing to argue about. Every answer is an integer, so grading needs no judge and no rubric, and the problems are hard enough that getting there requires real multi-step work rather than pattern completion.

What an item is made of

  • Two papers of fifteen problems, sat by American high-school students as a qualifier for the olympiad track.
  • Every answer is an integer from 0 to 999 — write it down and it is either right or wrong.
  • Reported with no tools: no calculator, no code execution, no search.
  • Held after the training cutoff of the models that report it, which is what makes a fresh year worth using.
The whole benchmark, at the score most of the frontier is posting. Drag one cell and watch the number move by 3.33 — that single problem is the entire distance between several models in our table.

How one run goes

  1. question
  2. model answers
  3. grader checks
  4. score

Scoring

pass@1, exact match on the integer. Serious reports average several runs, because a single sampling accident costs a whole problem.

Where the field sits

0 100

28 published figures · best 100 · 0.0 points of headroom left

Every figure the makers have published on this benchmark, unnamed. A clot against the right edge means the benchmark has stopped separating anyone.

Running it yourself

  1. 1. Take the 2025 problems from the AoPS wiki. Community mirrors of them disagree in places, which is itself a reason to ask what a lab used.
  2. 2. Ask for the integer only. No calculator, no code interpreter, no search.
  3. 3. Run it several times and report the mean, not the best run.

What to watch

Saturated and noisy at the same time — a rare combination. The top of the field scores at or near 100, and the gap between 96.7 and 100 is one problem out of thirty. A figure produced "with tools" is a different benchmark and never enters this column.

SWE-bench

how Verified was built ↗

Agentic task · 500 real GitHub issues — the Verified subset

To stop asking models to write code in a vacuum. Each task is a bug or feature request that was actually filed against a real Python project, and the model has to work in that repository as it stood at the time. Nobody scores the prose: the project’s own test suite decides.

What an item is made of

  • A snapshot of the repository at the commit before the fix, plus the issue text as the reporter wrote it.
  • A hidden pair of test sets. FAIL_TO_PASS are the tests the fix is supposed to repair; PASS_TO_PASS are the ones that must not break on the way.
  • Verified is the human-screened slice: 500 instances that engineers confirmed are solvable and unambiguously specified, out of a far noisier original.
  • Real instance ids look like astropy__astropy-12907 — a repository, and the issue number it came from.

How one run goes

  1. repo at commit
  2. issue text
  3. agent edits files
  4. run FAIL_TO_PASS
  5. run PASS_TO_PASS
  6. resolved

Scoring

Binary per instance: every FAIL_TO_PASS test must pass and every PASS_TO_PASS test must still pass. No judge, no partial credit. The score is the share of the 500 that resolved.

resolved
Switch either one off. Three of the four states are a failure, and there is no partial credit between them — a patch that fixes the bug and breaks the build scores exactly what a blank patch scores.

Where the field sits

0 100

58 published figures · best 96 · 4.0 points of headroom left

Every figure the makers have published on this benchmark, unnamed. A clot against the right edge means the benchmark has stopped separating anyone.

Running it yourself

  1. 1. Load the set: datasets.load_dataset("princeton-nlp/SWE-bench_Verified")
  2. 2. Run your agent over the instances and save one patch each as a predictions file.
  3. 3. Grade with the official harness, which rebuilds each repo in Docker: python -m swebench.harness.run_evaluation --predictions_path <file> --run_id <name>

What to watch

The scaffold is doing a great deal of the work, and this is measurable rather than theoretical. Run through the authors’ own reference harness, every model in our table scores lower than its maker reported — by three points at best and twenty-five at worst, always in the same direction. Both numbers are in the table, in different colours.

SWE-bench Pro

Scale’s write-up ↗

Agentic task · 1,865 tasks across a public and a held-out split

SWE-bench Verified is public, which means it is in the training data of everything that comes after it. Pro is the answer to that: harder tasks, drawn from repositories still under active development, and a private split that cannot be memorised because it was never published.

What an item is made of

  • The same contract as SWE-bench — repository, issue, hidden tests — with the difficulty turned up.
  • Larger diffs that span several files, so a lucky one-line edit does not resolve the task.
  • A public split anyone can run, and a private split only its maintainers can.
Public split open

Published, so anyone can reproduce a figure on it — and so it can be trained on.

runnable by: anyone

Private split sealed

Never published, so it cannot leak into training data — and cannot be checked by anyone else either.

runnable by: its maintainers only

The trade the whole design rests on: the half you can verify is the half that will end up in the next training run.

How one run goes

  1. repo at commit
  2. issue text
  3. multi-file patch
  4. hidden tests
  5. resolved

Scoring

Identical in principle to SWE-bench: the repository’s tests decide, and the score is the share resolved.

Where the field sits

0 100

31 published figures · best 80.3 · 19.7 points of headroom left

Every figure the makers have published on this benchmark, unnamed. A clot against the right edge means the benchmark has stopped separating anyone.

Running it yourself

  1. 1. The public split runs on Scale’s leaderboard with its published harness.
  2. 2. The private split is not replicable by design. Only its maintainers can produce a number on it.

What to watch

A private split is the honest answer to contamination and an unfalsifiable claim at the same time. Treat public and private figures as two different assertions, and do not average them.

HLE

the project site ↗

Quiz · 2,500 questions across more than a hundred subjects

Built explicitly because the other quizzes ran out of headroom. Contributors were asked for questions at the edge of their own field that current models get wrong — the set was assembled by filtering against frontier models, so difficulty is a design property rather than an accident.

What an item is made of

  • Expert-written, spanning far more subjects than a school curriculum, including text and image questions.
  • Answers are open-ended, so a model grades them against the reference rather than a string comparison.
  • Reported in two regimes — with tools and without — and the same model differs by around eight points between them.
No tools 46.9%
With tools 54.7%
Claude Opus 4.7, reported by Anthropic in the same system card: identical model, identical questions, one allowed to search and run code.

How one run goes

  1. question
  2. model answers
  3. grader checks
  4. score

Scoring

Accuracy as judged by a grader model, since the answers are free text. Which regime the number came from matters as much as the number.

Where the field sits

0 100

35 published figures · best 59 · 41.0 points of headroom left

Every figure the makers have published on this benchmark, unnamed. A clot against the right edge means the benchmark has stopped separating anyone.

Running it yourself

  1. 1. Load the set: datasets.load_dataset("cais/hle")
  2. 2. Decide up front whether the run gets search and code execution, and publish which.
  3. 3. Grade with a judge model, and if browsing is allowed, blocklist the pages that discuss the benchmark.

What to watch

The one column here with real room left, which makes it the one worth watching. Our table only ever carries the no-tools figure: mixing the two would credit a model with a capability it rented.

BrowseComp

OpenAI’s write-up ↗

Agentic task · 1,266 questions

To measure persistence on the live web rather than knowledge. Each question was written backwards from an answer that is short and checkable but buried — reachable only by chaining several searches, discarding wrong leads and reading past the first page of results.

What an item is made of

  • Answers are a name, a number, a date — easy to grade, hard to reach.
  • The model needs a real browsing harness: search, page fetch, and usually code execution.
  • Every published figure sits on a token budget, and the budgets differ by an order of magnitude between labs.

How one run goes

  1. question
  2. search
  3. read + discard
  4. search again
  5. short answer
  6. exact match

Scoring

Exact match against the reference answer. The interesting variable is not the grader but how much the agent was allowed to spend getting there.

Where the field sits

0 100

27 published figures · best 90.8 · 9.2 points of headroom left

Every figure the makers have published on this benchmark, unnamed. A clot against the right edge means the benchmark has stopped separating anyone.

Running it yourself

  1. 1. The questions and the grader live in OpenAI’s simple-evals repository on GitHub.
  2. 2. Give the agent search and fetch tools and a stated token budget.
  3. 3. Blocklist pages discussing the benchmark, or the agent will find the answer key rather than the answer.

What to watch

The token budget is the hidden variable, and vendors publish scaling curves showing accuracy climbing with it. A BrowseComp figure quoted without the budget beside it is half a claim.

Why we do not run any of this

Every figure in the table is one somebody else published: the lab that made the model, or a platform that measures models for a living. Running these ourselves would mean a GPU bill, a scaffold of our own, and a number nobody could check against anything. Instead each cell links the document it came from, and where two parties measured the same thing, the table shows both and lets the gap speak.