The same Claude went from 30 to 96: the limiting factor may be the harness, not the model

Conclusion

In ARC Prize's official model-only evaluation, Claude Opus 5 (High) scored 30.16% on ARC-AGI-3 and completed five public environments that no previous model had completed. Independent developer Jeremy Berman then put the same Opus 5 into near-stock Claude Code with a shell, filesystem, action command and logs, reaching 24/25 and 96.2%. A second implementation by Poobesh Gowtham, arc-skill, was replay-verified by ARC at 25/25, 183/183 levels and RHAE 100.00.

This does not mean Claude's official ARC score has been updated from 30% to 100%, and the three numbers do not belong on one leaderboard. The 30.16% result measures the bare model; 96.2% and 100.00 measure a model plus a harness, external memory, tools and feedback loops. Together they show that for tasks requiring exploration of an unfamiliar environment, how a model works may matter as much as the model itself.

Same model, very different results

ARC-AGI-3 gives no game rules. An Agent must observe an unfamiliar environment, try an action, record the change, infer the goal and mechanics, then revise its hypothesis. It is closer to picking up a game with no manual than answering a fully specified question.

ARC Prize's official model-only results are:

| Evaluation setting | ARC-AGI-3 score | | --- | ---: | | Claude Opus 5 (High) | 30.16% | | GPT-5.6 Sol (Max) | 7.78% | | Claude Opus 4.8 (High) | 1.5% |

The test deliberately minimizes tools and engineering scaffolding to examine the model's generalization. It answers “what can this model do on its own,” not “what can the full system do after receiving a persistent workspace.”

First reversal: Claude Code let it write its own Solver

Jeremy Berman did not retrain Opus 5 or provide an ARC-specific solver. He used stock Claude Code with Bash, Read, Write, Edit, Grep and Glob, plus one action command for interacting with the game.

The model could read and write files, inspect logs and run Python programs. It then built whatever each game required: parsers, rule models, simulators, path searches and verification scripts. In one 25-game pass it wrote 269 programs and about 12,700 lines of code, then discarded them when the game ended.

The first complete run:

| System | ARC score | Games won | Actions | Estimated cost | | --- | ---: | ---: | ---: | ---: | | Claude Code + Opus 5 | 96.2% | 24/25 | 8,789 | about $540 | | Claude Code + GPT-5.6 Sol | 73.7% | 19/25 | 26,971 | at least about $424 |

The one failed game succeeded on a retry, bringing Opus 5's pass@2 to 99.3%. This remains a preliminary run rather than a stable mean over dozens of repetitions; the reasoning settings also differ, with Sol at xhigh and Opus at high.

Second reversal: another Skill reached a perfect score

If one person reaching 96.2% were the only evidence, prompt tuning or chance would remain plausible. Poobesh Gowtham then built a different arc-skill, without copying the first implementation and without embedding answers for the 25 games.

Its rule is simple: before spending an action, write down what the action is expected to change on the grid. The harness rejects an action without a prediction; when the prediction fails, the changed board is recorded so the Agent can update its model.

The final result:

• 25/25 games completed;
• 183/183 levels completed;
• RHAE 100.00, the metric's ceiling;
• 7,645 actions, compared with ARC's median human baseline of about 17,135;
• every action replayed through ARC's servers, with public scorecard 24ddb219.

The Skill has 129 lines of instructions plus a 4,343-line command-line tool. The author says it contains no game-specific answers. It does not prove that one prompt solves ARC, but it makes 96.2% look less like an isolated screenshot.

What changed was the working environment

The experiments did not give Opus 5 a mysterious plug-in. They gave it concrete capabilities.

External memory. Logs and notes live in files. The model can recover observations from half an hour ago with grep, Python and text files instead of packing every state into the current context.

Temporary tools. Engineers did not pre-build a universal ARC Solver. The model wrote a parser, simulator or search program for the game in front of it, turning a scratchpad into runnable software.

A feedback loop. Each action returns a new board and log. arc-skill adds a falsifiable prediction: state the expected result, press the button, and revise the hypothesis if reality disagrees.

A stopping condition. Exploration tasks do not only test whether an Agent can guess correctly. They test whether it knows when the rules are understood and planned execution can begin. Berman's record shows Opus using fewer actions for more wins, which makes the switch from exploration to execution part of the capability.

This does not make model leaderboards useless

Bare-model evaluations still matter. They compare basic capability under similar conditions and prevent an engineer-written solver from being counted as model intelligence.

But users buy systems, not one line on a model card. A model scoring 30 bare can do 96-level work with logs, a terminal, action tools and a verification loop. A model with a higher base score may perform worse if it cannot preserve state, write temporary programs or test its own hypotheses.

That is why the same model can feel different in Cursor, Claude Code, Codex or OpenCode. The difference is not only the model version; it is how the harness manages context, tools, permissions, memory and recovery.

A perfect score still does not generalize automatically

ARC-AGI-3 community results and the official model-only leaderboard are separate tracks. ARC's community already includes systems built specifically for ARC that score higher. arc-skill's 100.00 means that this public set and this replay passed completely.

Berman calls his result preliminary. More repetitions, cost accounting, crash recovery, reasoning settings and cross-model portability are still needed. Most importantly, exploration skill on 25 unfamiliar games does not automatically prove transfer to codebases, ERP systems or scientific environments.

Our judgement

The memorable part is not the 30.2 → 96.2 arrow. It is what happened inside the arrow: with a terminal, files, logs and observable feedback, the model began manufacturing a small toolchain for each task at run time.

Agent engineering used to put the intelligence into the harness: preset planners, retrievers, world models, solvers and memories. Stronger models may now absorb part of that scaffolding, deciding which script to write, what state to preserve and which hypothesis to test.

This changes how products should be compared. Asking which model has the highest ARC score still makes sense, but also ask: in what environment does it work? Can it externalize state? Can it build tools? Does it change plans after failure? How much time, action and money does one completed task cost?

What to watch next

• Whether 96.2% and 100.00 hold across more independent repetitions;
• Whether the same thin harness transfers to other unfamiliar tasks;
• Whether model-written temporary tools create new security, permission and audit risks;
• Whether stronger models truly need fewer built-in tools, or simply fit ARC well;
• Whether total cost, recovery and human supervision become more stable system metrics than one benchmark score.

Sources and evidence boundary

• Primary evaluation: ARC Prize, Claude Opus 5 ARC-AGI-3 model-only result
• Independent experiment: Jeremy Berman's arc-code, 96.2% run and code
• Independent Skill / replay: Poobesh Gowtham's arc-skill, 25/25, 100.00 and ARC scorecard

This article separates ARC's official 30.16% from the community harness results of 96.2% and 100.00. The latter are independent developer system experiments, not ARC's official model-only leaderboard scores; cost, transfer and repeatability remain open questions.