A 27B model “beat Claude” — but it actually used GPT-5.5 as its coding specialist

Conclusion

Inherent's Faraday is an AI Scientist built on Qwen3.6-27B and trained specifically to replicate research papers. On Inherent's Replica Benchmark, Faraday scored 0.791 on the held-out AI-for-science split, above Claude Opus 4.8 at 0.748 and GPT-5.5 Codex at 0.729.

But “a 27B model beat Claude” is not an accurate summary. Faraday delegates code implementation and execution to GPT-5.5 Codex. Faraday reads the paper, chooses experimental directions, evaluates results and decides what to try next. The more interesting result is that a smaller Manager Agent may outperform a frontier model working alone after it learns to organize a stronger Specialist Agent.

Replica was designed by Inherent, and its scoring depends on an LLM rubric judge that the team checked against human evaluation. There is not yet an independent replication by outside teams. Faraday demonstrates a promising way to organize Agents, not that a 27B model has universally surpassed Claude or GPT.

Why a 27B model was described as beating Claude

Inherent selected 100 machine-learning and AI-for-science papers and extracted 310 replication tasks. The Agent receives the paper but one key result figure is removed, so it must infer what experiment the authors ran.

It must read the method, infer missing details, design a feasible smaller experiment, write code, run it, and revise the plan from the results. 242 tasks belong to the training distribution, while 68 AI-for-science tasks form the held-out test.

The final results were:

| Agent | 242 in-distribution tasks | 68 held-out tasks | | --- | ---: | ---: | | Qwen3.6-27B Base | 0.678 | 0.554 | | GPT-5.5 Codex | 0.796 | 0.729 | | Claude Opus 4.8 | 0.828 | 0.748 | | Faraday | 0.856 | 0.791 |

Faraday did lead: on held-out tasks it was about 5.7% relatively above Claude and 8.5% above Codex. This is not evidence that 27B parameters overpower a much larger model, but it is enough to show that Faraday learned a useful way of working on this task family.

The real twist: it calls GPT-5.5 to write code

Faraday is closer to a research lead than a standalone model.

It understands the paper, proposes hypotheses, orders experiments and checks whether results are credible. When implementation is needed, it calls GPT-5.5 Codex to write code, modify the implementation and execute engineering work. Faraday then decides whether to continue, backtrack or change the experiment.

The benchmark therefore compares two workflows: GPT-5.5 Codex doing the whole replication alone, and Faraday making research decisions while GPT-5.5 Codex handles coding. The latter scoring higher does not mean Faraday contains more knowledge than GPT-5.5.

It is less like a schoolchild defeating a professor and more like a research lead with strong judgement working with a top programmer. The division of labor becomes part of the capability.

It may have learned how to manage an Agent

During training, Faraday mainly learned to call GPT-5.4-mini, while evaluation could use the stronger GPT-5.5 Codex. If the result holds up, this suggests the Manager learned a general research workflow rather than a fixed prompt for one model.

A Manager can stay online to own goals, context, progress and acceptance. Coding, search, vision, math and database work can go to different Specialists. When the underlying tools improve, the Manager may not need to be retrained from scratch.

This is also the structure Agent products are moving toward: models are employees, while the harness, tools and acceptance loops are company processes. The Manager decides who should work and when.

The same 27B base rose by 43% after training

Faraday's most striking number is not its lead over Claude. It is that the same Qwen3.6-27B base moved from 0.554 to 0.791 on held-out AI-for-science tasks, a relative improvement of about 43%.

The main change was long-horizon reinforcement learning. The model repeatedly tried research decisions and received feedback after experiments completed, gradually learning which hypotheses deserved testing and which errors should be abandoned quickly.

This suggests that many Base Models may not yet have been trained into effective work Agents. The next competition may not only add parameters; it may train models to become better leads.

It has not proved that it can discover new science

Replica reproduces published papers. The answer exists; the result figure is simply hidden. Faraday must recover experimental details that papers often omit, but it has not shown that it can discover a new material, propose a new drug or find an unknown law.

Inherent describes Replica as a step from replication toward innovation. Teaching an Agent to do research reliably before asking it to make discoveries is a reasonable sequence, but the two achievements should not be conflated.

How to read the scores

Replica was built by Inherent, and Faraday was reinforced on this class of tasks. The final score depends on an Inherent-designed rubric-based LLM judge. The team ran human checks and says the rubric judge is more stable and closer to researcher judgement than a basic baseline, but automated scoring and training rewards still share the same rubric, leaving some circularity.

The careful claim is therefore: Faraday beat Claude Opus 4.8 and GPT-5.5 Codex on Inherent's own Replica Benchmark, with human validation of the rubric; the result still needs outside tasks, different judges and independent replication.

Our judgement

Faraday's real significance is not the number 27B. It changes the model contest from “whose brain is smarter?” to “who is better at organizing a group of AIs?”

A relatively cheap Manager can own long-running judgement, then call an expensive model only for coding, search or hard reasoning. That may be more economical than making a frontier model handle every step. For enterprise Agents, this affects latency, cost, recovery from failure and permission boundaries.

It also suggests that the unit of comparison should not always be one model. The more useful future question may be: how much does a Manager plus Specialists spend to complete a set of verifiable research tasks?

What to watch next

• Whether outside teams reproduce Faraday's lead with different papers and judges;
• Whether the Manager still works when paired with other Coding Agents;
• Whether the advantage holds on biology, materials and weather tasks outside training;
• Whether Manager plus Specialist systems reduce the total cost of each successful research task;
• When Faraday moves from reproducing known papers to verifiable new-discovery tasks.

Sources and evidence boundary

• Primary research: Inherent, “Training AI Scientists to Replicate Research”
• Research paper: arXiv 2608.13331
• Independent analysis: AI Adventurer's summary of Replica scores and training
• Paper review: Pith's discussion of the scoring system, circularity and replication limits

This article separates the “Faraday plus GPT-5.5 Codex Agent system” from “GPT-5.5 Codex completing the task alone.” Replica is an Inherent-built evaluation with no large-scale independent replication yet; its results do not establish that Faraday has universally surpassed Claude, GPT or other research systems.