What the source published

SlopCodeBench is built to mirror real development: an agent receives a problem in stages, writes a program, gets new requirements, and changes its existing code, with later checkpoints testing both the new behaviour and whether old behaviour still works. This round used a pinned catalog of 36 problems and 196 checkpoints per setting, one stock prompt and one seed. It reports two bars: a strict pass requires every test category including regression to pass, while a core pass only requires the core category.

Across the ten complete campaigns, strict pass rates landed between 12.2% and 16.3%. GPT-6 Astra at extra-high effort took the most strict passes, 32 of 196 (16.33%), at an estimated $758.62; Opus 5.5 at medium effort led on core passes with 145 of 196 (73.98%) and 28 strict passes at about $193.47. The author stresses that the gaps between models are often just two checkpoints, so the ranking itself is the least interesting part.

The headline comes from three complete same-model pairs. Moving from medium to extra-high effort multiplied estimated cost by 2.11, 2.91 and 4.22 times. Strict passes moved from 28 to 25 for Opus 5.5, from 30 to 26 for GPT-6 SOL, and only Astra improved, from 29 to 32. The author's own caution is explicit: this does not prove that more reasoning hurts coding, but in these runs higher effort was not a reliable way to buy more strict passes — with one campaign per setting, run-to-run variation cannot be separated from the effect of effort.

Sources: [1] [2]

Try it: turn the benchmark into cheaper habits

  1. Default routine coding work to medium effort: in two of the three pairs it scored more strict passes than extra-high at a quarter to a half of the cost.
  2. Reserve the top setting for genuinely hard, well-specified problems where failure is expensive — and validate the choice on one of your own tasks before adopting it everywhere.
  3. Run the regression suite after every stage, not just the new tests: every campaign's final code still had large numbers of failing cases, and the overwhelming majority were regressions.
  4. Make cleanup part of the job: require the agent to remove scratch and temporary files before it declares done, because the measured runs left working files behind by the tens of thousands of lines.
  5. Write your acceptance bar in strict terms: delivery means the new behaviour works and the previously passing tests still pass, not merely that the demo path runs.

Sources: [1]

The full picture: how far the final code is from shippable

Passing checkpoints is not the same as ending with a working program. Looking only at each problem's final checkpoint, the ten campaigns left between 1,435 and 2,831 failing test cases, most of them regression failures — old behaviour broken while adding new behaviour. The number of problems whose final checkpoint passed strictly was between zero and three per campaign. The author's summary: agents are getting better at the main behaviour, but keeping all tested behaviour working as requirements evolve lags far behind, and the best strict rate across the field only crept from 14.8% in May to 16.3% now.

The code-quality scan is just as telling. Models left a great deal of junk and scratch code as they worked; on the 28 problems measured for every run, Opus at extra-high effort had the lowest share of flagged lines at 77.75%, while its medium run was the highest at 94.25% — yet the extra-high run passed fewer strict checkpoints. Opus at medium also left the most code overall, about 216,881 source lines, and a cleanup filter alone removed over 41,000 lines sitting in temporary folders. Average function complexity ranged from Fable's 3.57 to GPT-6 SOL medium's 11.70, with the caveat that splitting code into more helpers lowers the average without necessarily making anything easier to change.

Sources: [1]

How to read this

For anyone driving Codex or Claude Code daily, the useful export is not the leaderboard but the budgeting rule: effort is a cost line, not a quality guarantee. Dropping the default back to medium and spending the savings on a full regression run plus an independent review pass is, on this data, the better trade.

Hold the conclusion loosely, as the author does. With a single campaign per setting, a two-or-three-checkpoint gap can easily be noise. Use the method and the numbers as a prior, run a small comparison on your own repository and task mix, and only then set your team's defaults.

Sources: [1] [2]

Key takeaways

  • Strict pass rates across the field were only 12.2% to 16.3%: when requirements arrive in stages, agents break old behaviour far more than they miss new behaviour.
  • Extra-high effort cost 2.11 to 4.22 times medium in the three complete pairs, and strict passes fell in two of them — more spend, no reliable gain.
  • Working default: medium effort for routine tasks, top effort only for well-specified, high-cost-of-failure problems, verified on your own work first.
  • Every campaign's final code still failed many tests, mostly regressions — a full test run after each stage beats an effort upgrade.
  • Make the agent clean up after itself: measured runs left scratch code by the tens of thousands of lines when nobody asked.

Sources: [1]