In speedrunning there's a category called any%: get to the end any way you can. The player doesn't touch the code or use outside programs. They use the game they were given and find the route the designer didn't see.
It was never cheating. It was reading better.
What I see in teams is that we write metrics assuming there's someone acting in good faith on the other side. And now on the other side there's something that reads them like a speedrunner and runs them thousands of times a day.
What already happened
A team had an agent audit the ten most-cited agent benchmarks. On nine out of ten it got a near-perfect score without solving a single task.
- 9 of 10benchmarks with a near-perfect score without solving a taskBenchJack, arXiv, May 2026
- 219distinct flaws foundnone were the model's, all were the exam's
The uncomfortable part
It's not that the model doesn't understand the intent. In METR's tests on o3, across 128 RE-Bench runs the model took the shortcut in 39 (30.4%), without anyone asking it to.
After it was explicitly told not to, it kept doing it between 70% and 95% of the time. And when it was asked ten times whether that was what the designer wanted, it answered no all ten times. And kept going.
A product decision
In February 2026, OpenAI stopped reporting SWE-bench Verified, the industry's most-used coding benchmark. The reason wasn't that the number was dropping. It was that going up there no longer reflected real capability, but how much the model had seen that exam during training. It recommended reporting SWE-bench Pro.
Where it breaks: four routes
Shortcuts aren't infinite.
| Route | What happens |
|---|---|
| The answer is already there | The test case lives somewhere the system has already read. |
| The judge can be touched | Whoever grades is software, and whoever is being graded can write there. |
| You win without playing | The success criterion is met without the task being done. |
| What isn't measured | Everything left outside the number gets abandoned, and doesn't warn you. |
Five questions before turning a number into a target
The shortcut
What's the fastest way to move this number without doing the work?
Passes if you can name the route. If you can't name it, you haven't closed it: you've left it open.
The judge
Who can write where the result is recorded?
Passes if whoever is being graded can't get there. If they can, it's not a measurement.
The counterweight
What other metric breaks if this one rises alone?
Passes if there's at least one. If there's none, you don't have a dashboard: you have a number.
The date
Could the case have been in the training data?
Passes if it's after the model's cutoff. If not, it doesn't measure capability: it measures memory.
The attempt
Did anyone try to break it on purpose?
Passes if someone tried. If nobody did, the metric isn't tested: it's just premiering.
If it passes all five, only then is it a target. Before that it's a route, and someone out there runs it faster than you.
I haven't tried to break mine either. I look at them every day and never sat down to look for their shortcut.
