On 19 November 2026, the Vals Index put Claude Fable 5.2 first at 71.31%, Gemini 4 Neon second at 70.87%, and Claude Sonnet 5.5 third at 70.85%. The gap between first and third was 0.46 percentage points. The gap between first and second was 0.44.
A leaderboard that compresses the top three models into less than half a point is not necessarily telling you that the models are equally good. It is telling you something about the measurement. Depending on how the index is built, 0.46 points can be a real advantage, a rounding artifact, or a difference that would disappear if the tasks were resampled. The only way to know is to read the methodology.
That is the point of this guide. “AI benchmark” is not a single thing. It is a task set, a harness, a scorer, a reporting convention and an institution, all stacked together. Change any layer and the number changes. The seven benchmarks below are among the most-cited ways to measure frontier models in late 2026. They do not agree because they are not measuring the same thing.
What a benchmark actually is
At minimum, a benchmark has four parts. The task set is the collection of problems, prompts or environments. The harness is the software that presents the task, gives the model tools or time limits, and records what it does. The scorer decides whether the output counts as correct. The reporting convention turns the results into a number: pass@1, average accuracy, a weighted composite, a win rate, a time horizon.
Most leaderboard arguments are really arguments about one of those layers. A model that scores 80% on a coding benchmark may have used a different scaffold than a model that scored 75%. A benchmark that looks contaminated may have been designed for a different purpose. A composite that weights finance at 8.0 and tax at 0.5 is making a claim about what matters, not just measuring what is.
The Vals Index: a GDP-weighted composite
The Vals Index is the most explicitly economic of the major leaderboards. Version 2.2, updated on 19 November 2026, aggregates eight benchmarks across four sectors: finance, coding, legal and tax. Six of the eight are private; two are public. The sector scores are weighted by each sector’s share of U.S. GDP, then combined:
Vals Index = (8.0 × Finance + 5.6 × Coding + 1.2 × Legal + 0.5 × Tax) ÷ 15.3
The weights are the story. Finance is 8.0 of the 15.3 denominator, or 52.3%. Coding is 5.6, or 36.6%. Legal is 1.2 (7.8%), and tax is 0.5 (3.3%). A model that is excellent at tax research but mediocre at coding will not move the index much. A model that is excellent at finance and coding will. That is a deliberate choice about what “economic impact” means, and it is not the only defensible one.
The index is also a composite of composites. Finance is the average of Finance Agent v2.1 and EMB v1.1. Coding is the average of Terminal-Bench 4, Vibe Code Bench v1.1 and Code Migration v1.1. Legal is the average of Legal Research and HLAB-Vals-1.1. Tax is Tax Agent Bench. Each component keeps its own published methodology. The index does not re-run the benchmarks; it combines their scores.
Vals argues that its private benchmarks are an advantage because no other party has full access to them, which reduces the risk that models were trained on the answers. That is plausible. It also means an outsider cannot independently reproduce the component scores. The public/private split is a trade-off, not a free lunch: private tasks are harder to contaminate but harder to audit; public tasks are auditable but more likely to leak into training data.
The leaderboard also reports cost per test, tokens per test and duration. That matters because accuracy alone can mislead. A model that scores 70% at $17.94 per test is not obviously better than a model that scores 70% at $7.40. The Vals page makes the trade-off visible, which is more than many leaderboards do. But the headline number is still an accuracy composite, and the version of the page we read does not show a confidence interval for that composite. Without one, a 0.46-point gap should be treated as a tie until someone shows otherwise.
SWE-bench Verified: what happens when a benchmark becomes a target
SWE-bench Verified is the cleanest case study in benchmark decay. It was released in August 2024 as a human-validated subset of 500 software-engineering problems drawn from 12 open-source Python repositories. Each problem comes from a real GitHub issue and a merged pull request. The model gets the issue and the pre-fix repository; it passes only if its patch makes the tests pass without breaking existing behavior.
For a while, it was the standard coding metric. Then OpenAI audited it. In a 23 February 2026 post, OpenAI said it had examined 138 problems, 27.6% of the 500, that its o3 model did not solve consistently over 64 runs. At least six experienced engineers reviewed each one. OpenAI found that 59.4% of the audited problems had material issues in test design or problem description: 35.5% had narrow tests that demanded a specific implementation detail, 18.8% had wide tests that checked behavior the problem never specified, and 5.1% had miscellaneous issues.
The 59.4% figure is widely misquoted. It is not the defect rate of all 500 tasks. It is the defect rate of a deliberately selected hard subset. OpenAI chose those 138 because its model often failed them. Epoch AI’s review, published on 3 September 2026, translated the audit into a floor: at least 16.4% of all 500 tasks are broken. Epoch rated the benchmark “Flawed,” noting that 100% of the tasks and 100% of the solutions are public.
The contamination finding is the more important one. OpenAI said all frontier models it tested could reproduce the original human-written bug fix, known as the gold patch, or verbatim problem specifics for certain tasks. That is evidence the models had seen the problems during training. A model that has memorized the answer is not demonstrating the ability to solve a new one. OpenAI stopped reporting SWE-bench Verified scores and recommended SWE-bench Pro instead.
The lesson is not that SWE-bench is worthless. Its early scores measured something real. The lesson is that a public benchmark becomes a target, and a target becomes training data. Once that happens, the score increasingly measures exposure rather than capability. Any leaderboard that reports a static public benchmark without a contamination check is asking to be gamed.
ARC-AGI-2: hold out the tasks, control the sandbox
ARC-AGI-2 is designed around the opposite problem. The benchmark consists of 240 previously unseen tasks, split into 120 semi-private and 120 private evaluation tasks. Each submission runs offline inside a sandbox with four NVIDIA L4 GPUs and a 12-hour wall-clock window, with no internet access. The public training set is separate from the evaluation set.
The tasks are visual grid puzzles that require inferring a rule from a handful of examples and applying it to a new case. They are designed to be novel, so memorizing the training set does not help much. The benchmark’s creators calibrated the public, semi-private and private sets so that mean human accuracy differed by no more than one percentage point across partitions. That is a serious attempt to make the held-out sets comparable to the public one.
ARC-AGI-2 is a good measure of abstract reasoning under a tight compute and time budget. It is not a measure of economic value. A model that solves a grid puzzle has demonstrated something about pattern induction; it has not demonstrated that it can draft a contract, debug a production system or run a sales call. The benchmark’s own framing is about AGI progress, not workplace performance. Treating an ARC score as a general capability score is a category error.
METR time horizons: capability expressed as human time
METR’s time-horizon metric asks a different question: at what task duration can an agent succeed with a given reliability? The 50% horizon is the duration at which the agent is predicted to succeed half the time; the 80% horizon is the duration at which it succeeds 80% of the time. The measurements are based on more than a hundred diverse software tasks, with task duration measured by human expert completion time.
The metric is intuitive because it translates capability into a unit people understand: hours of work. It is also narrow. The task suite is software-heavy, so a four-hour horizon means four hours of software work, not four hours of legal research or customer support. METR’s page says it is no longer actively updated, with a last update on 8 September 2026, and warns that some models were measured long after release while others were skipped entirely. A time horizon is a snapshot, not a permanent property of a model.
GDPval: does the deliverable look like professional work?
GDPval, released by OpenAI in September 2025, takes the economic-value question seriously. It spans 44 occupations across nine industries that contribute more than 5% of U.S. GDP. The full set contains 1,320 tasks, 30 per occupation; the open-sourced gold set contains 220 tasks, five per occupation. The tasks were written by experienced professionals, 14 years of experience on average, and are based on real deliverables such as legal briefs, engineering blueprints, customer support conversations and nursing care plans.
Grading is done by expert humans who compare a model’s deliverable with a human-produced one without knowing which is which. They rank the outputs and classify the model’s work as better, as good as, or worse than the human’s. That is closer to how work is actually judged than a unit test. It is also expensive, slower to scale, and limited to one-shot tasks: the current version does not capture multi-turn revision, client feedback or the messy context of a real job.
GDPval is the best available answer to “can the model produce something a professional would accept?” It is not an answer to “will this model replace the professional?” A deliverable can be good and still require a human to scope the work, verify the facts, handle the client and take responsibility for the outcome.
HELM: the case for broad, boring coverage
HELM, the Holistic Evaluation of Language Models, was built to measure many models across many scenarios and many metrics, not to crown a single winner. It entered maintenance mode on 1 June 2026. The code and leaderboards remain available, but the project is no longer actively expanding them.
That is a useful reminder that benchmarks are institutions. They need funding, maintainers, compute and fresh tasks. A benchmark that stops updating does not become wrong overnight, but its scores become historical artifacts. The same is true of any leaderboard: a score without a date is a claim about the past.
LMArena: preference is not correctness
LMArena, now Arena, takes the human-preference route. A visitor sends one prompt, sees two anonymous answers and votes for the better one. The platform fits a Bradley-Terry model to those pairwise votes, reports confidence intervals, and publishes a raw rank plus a rank spread that reflects how much the intervals overlap. Since May 2025, style control has been the default for the text and vision arenas; it adjusts for answer length and counts of markdown headers, bold text and lists.
This is a good measure of which answer people prefer. It is not a measure of which answer is correct. A confident, well-formatted wrong answer can beat a terse correct one. Style control reduces that bias but does not eliminate it, and the adjustment is observational: length may correlate with quality for reasons the model cannot separate. LMArena is useful for chat quality, tone and general helpfulness. It is not a substitute for a task-specific evaluation.
What the seven benchmarks actually measure
| Benchmark | Core question | Main strength | Main limitation |
|---|---|---|---|
| Vals Index | How well do models perform economically weighted tasks? | Composite across finance, coding, legal and tax; cost and latency reported | Private components are hard to audit; GDP weights are a choice |
| SWE-bench Verified | Can a model resolve real GitHub issues? | Real repositories and tests | Contamination and flawed tests; OpenAI stopped reporting |
| ARC-AGI-2 | Can a model infer novel rules from few examples? | Held-out tasks; controlled sandbox | Abstract puzzles, not economic work |
| METR time horizons | How long a task can an agent complete reliably? | Intuitive human-time unit | Software-heavy; no longer actively updated |
| GDPval | Do deliverables match professional work? | Expert-authored, human-graded, 44 occupations | Expensive; one-shot; not a replacement forecast |
| HELM | How do models perform across many scenarios and metrics? | Breadth and transparency | Maintenance mode since June 2026 |
| LMArena | Which answer do people prefer? | Large-scale pairwise votes; intervals and style control | Preference, not correctness |
How to read a leaderboard without being fooled
Seven questions will catch most of the problems.
What is the task set, and is it public? A public set is auditable but can leak into training. A private set is harder to contaminate but harder to verify. The right answer depends on the claim being made.
What is the harness? Tool access, time limits, token budgets, system prompts and scaffolding can change scores by more than the gap between models. A leaderboard that does not disclose the harness is not reproducible.
Who scored it, and how? Unit tests, human graders, model judges and pairwise votes have different failure modes. A model judge can be biased toward its own family. A human grader can be inconsistent. A unit test can be too narrow or too wide.
Is there a contamination check? If the tasks and solutions are public, assume some exposure. Look for canary strings, held-out sets, private test splits or post-hoc audits.
Is there an uncertainty interval? A rank without an interval is a point estimate pretending to be a fact. LMArena publishes rank spreads; many leaderboards do not. If the top three are within half a point and there is no interval, the honest reading is “too close to call.”
What is the date? Models change, harnesses change, and benchmarks change. A score from six months ago may describe a different system.
Who built it, and who paid for it? That does not make a benchmark wrong. It does tell you what question the builders cared about. A benchmark built by a model lab will emphasize the tasks that lab considers important. A benchmark built by an industry consortium will reflect a different set of priorities.
What this means for buyers
If you are choosing a model for a specific job, the best benchmark is your own task set. Take 50 real examples of the work, define what “good” means, and run the candidates under the same harness. That is more work than reading a leaderboard, and it is more likely to predict your results.
If you cannot build your own eval, use a composite like the Vals Index to narrow the field, then check the component benchmarks that match your use case. A model that leads the composite because of finance and coding may not be the best choice for legal research. A model that wins LMArena may not be the most accurate. A model that tops SWE-bench may have memorized the test.
The larger point is that “state of the art” is not a single property. It is a relationship between a model, a task distribution, a harness and a scoring rule. The leaderboard is a map. The territory is your work.
Methodology and caveats
This article is a methodology review, not a hands-on model test. We did not run a 20-prompt, five-model comparison, and we did not independently reproduce any of the benchmark scores. We read the Vals Index methodology and leaderboard, OpenAI’s 23 February 2026 SWE-bench audit, Epoch AI’s 3 September 2026 review, the ARC-AGI-2 technical paper, METR’s time-horizon page, OpenAI’s GDPval release, the HELM maintenance notice and Arena’s ranking methodology. Quotations and figures are attributed to those sources.
We did not verify the private components of the Vals Index, which are not publicly available. We did not audit the ARC-AGI-2 task set or the GDPval gold set. We did not run the LMArena preference model. The table’s “main limitation” column is our editorial judgment based on the documented methodology, not a finding by the benchmark creators.
What we added: We compared seven benchmark methodologies side by side, calculated the Vals sector weights as shares of the denominator, separated SWE-bench’s hard-subset defect rate from its full-set rate, and built the “how to read a leaderboard” checklist. The comparison and the checklist are ours; the underlying methodology documents are linked below.
Last reviewed: 21 November 2026.
Sources: Vals Index methodology and leaderboard (v2.2, 19 November 2026); OpenAI, “Why we no longer evaluate SWE-bench Verified” (23 February 2026); Epoch AI, SWE-bench Verified benchmark review (3 September 2026); ARC-AGI-2 technical paper; ARC Prize, ARC-AGI-2 benchmark page; METR, Task-Completion Time Horizons (last updated 8 September 2026); OpenAI, GDPval (25 September 2025); HELM maintenance mode policy; Arena, ranking method; Arena-Rank methodology.
Have a tip? Email [email protected] or use the contact form and choose Editorial. We do not promise anonymity.
Read next: All AI coverage; What WARN notices say about AI layoffs in 2026; What AI APIs cost per million tokens; How AI export controls actually work; What the D.C. Circuit’s Anthropic ruling actually says.
