AI writes better code, so why do projects still go wrong? Opus 5 missed 27% of requirements
Datafruit's August 24 requirements discovery benchmark shows Claude Opus 5 recovering 73.1% of gold requirements, leaving about 26.9% missing. Its de-duplicated precision was only 22.4% because it produced about 1.88 times as many requirements as the gold set. GPT-5.6 Terra was more conservative but reached only 37.3% recall. These are not universal accuracy rates; they warn that important Coding Agent work needs clarification, specification review and acceptance criteria before coding begins.
Conclusion
On August 24, 2026, Datafruit published a Requirements Discovery Benchmark built from the structure of real enterprise implementation projects. It tests whether Agents can recover the final business requirements from several discovery-meeting transcripts. Claude Opus 5 led with 73.1% recall, still missing about 26.9% of the gold requirements, while producing enough extra claims that its de-duplicated precision was only 22.4%.
This does not mean Opus 5 is “22.4% accurate,” nor does it prove that one model cannot understand people. The metrics describe different risks: recall measures what was missed; de-duplicated precision asks how many distinct claims match the gold set. Opus 5 searches broadly and writes more. GPT-5.6 Terra is more conservative, but recovers only 37.3% of the gold requirements.
The practical warning for Coding Agent users is that requirements discovery remains a weak link before code generation begins. A model can implement the wrong interpretation beautifully. Important work should pass through clarification, specification review and acceptance criteria before an Agent is allowed to edit code.
The strongest model still missed more than a quarter
Datafruit did not test algorithm exercises. Each task contains discovery-meeting transcripts that simulate a customer, consultants and an implementation team gradually defining a software project over several conversations.
The benchmark deliberately includes three kinds of real-world difficulty: plausible ideas that are discussed and later rejected; implicit requirements that must be inferred across meetings; and “Do what I mean” cases where the customer asks for A but platform experience suggests B is the correct implementation.
The Agent reads the material in a Bash workspace and submits a structured requirements.json. Mean recall was:
| Model | Gold requirement recall | | --- | ---: | | Claude Opus 5 | 73.1% | | Claude Fable 5 | 65.2% | | Kimi K3 | 59.1% | | GPT-5.6 Sol | 55.0% | | GLM 5.2 | 52.0% | | Claude Sonnet 5 | 49.6% | | DeepSeek V4 Pro | 45.9% | | GPT-5.6 Terra | 37.3% | | GPT-5.6 Luna | 33.8% |
Opus 5 leads clearly. The other side of 73.1%, however, is that more than one in four real requirements was not fully recovered on average. Better coding, testing and deployment cannot automatically restore something that never entered the specification.
First reversal: it heard the most and invented the most
Opus 5 wins on recall. Datafruit's de-duplicated precision reverses the order: Terra reaches 36.7%, Sol 32.7%, Sonnet 5 32.5%, and Opus 5 22.4%.
The reason is understandable. Opus 5 outputs an average number of requirements equal to 1.88 times the gold count. It would rather capture more potentially useful signals than miss them, at the cost of turning inferences, elaborations or discussed ideas into additional distinct claims.
Terra takes the opposite route. Its output volume is close to the gold set and it makes fewer extra claims, giving it the highest de-duplicated precision, but it recovers only 37.3% of the gold requirements.
There is therefore no model to choose blindly. Opus risks scope inflation; Terra risks missing requirements. A real project should not optimize one score in isolation. A safer workflow collects candidates with high recall and then asks people to confirm which requirements are valid and which are assumptions.
What customers never say explicitly is hardest
Datafruit also separates explicit and implicit requirements. Opus 5 reaches 91.1% recall on explicit requirements and 68.9% on implicit ones. Every model shows the same pattern, with implicit recall trailing explicit recall by 24.4 to 44.6 percentage points.
That resembles real software work. A customer rarely says, “add an idempotency key and handle webhook retries.” They are more likely to say, “orders are occasionally duplicated.” An engineer must translate the business symptom into data, permissions, concurrency, failure handling and acceptance conditions.
Agents are increasingly good at following an explicit instruction. Discovering contradictions across scattered information, admitting uncertainty and asking the question that removes ambiguity is a separate capability.
More tools did not automatically create business judgement
Datafruit also compared a one-shot Opus prompt with a Coding Agent-style harness. Recall increased from 67.9% in the single call to 73.1% with the Agent harness. De-duplicated precision barely moved: 22.6% versus 22.4%.
Tools let the model read more files and sustain a longer process, and they did recover more requirements. But deciding which discussion became a live requirement remains a judgement problem. Bash, file search and a longer context window do not solve it automatically.
That is why progress on Coding Benchmarks is not identical to complete software delivery. SWE-bench measures fixes and Terminal-Bench measures terminal work; requirements discovery happens before either one.
Another study found that Agents often guess when information is missing
AMBIG-SWE, published at ICLR 2026, creates underspecified versions of real software-engineering tasks. It tests whether Agents detect ambiguity, ask a question and use the answer in the repair. The paper reports that in some experimental settings, interactive clarification improved success by as much as 74% relative to a non-interactive setup.
“As much as” matters. It is not a uniform gain for every model and task. But the two studies point to the same workflow gap. A model is not always incapable of completing the work; sometimes it simply fails to stop and confirm that it understood the work correctly.
Anthropic's 400,000 sessions show that people still decide what to do
Anthropic's privacy-preserving analysis of roughly 400,000 Claude Code sessions provides similar context. In a typical session, people make about 70% of planning decisions, while Claude makes most execution decisions. The more task-specific domain expertise a user displays, the more actions Claude typically completes after each instruction.
This does not mean beginners cannot use Coding Agents. It shows that the present division of labor remains clear: the model searches, edits, runs and iterates; the person supplies the goal, boundaries, business rules and judgement criteria.
The same Agent can feel more reliable for an experienced user, not because the prompt is more decorative, but because that user identifies earlier what cannot change, what must remain compatible and what counts as done.
A workflow change available now: ban coding for the first ten minutes
For tasks that cross pages or services, or involve permissions and business rules, ask the Agent to perform requirements discovery first:
- Restate the goal, users and current pain in its own words;
- Separate confirmed facts, reasonable inferences and unknowns;
- Ask the smallest set of questions that could change the implementation;
- State what is in scope and out of scope;
- Write verifiable acceptance criteria;
- Wait for human confirmation before editing code.
The following rule can go into a project's AGENTS.md or CLAUDE.md:
If missing information could change the data model, permissions, compatibility, external interfaces or acceptance outcome, do not choose an assumption and begin implementation. Inspect the repository and existing documentation first. If several reasonable approaches remain, ask the smallest number of decisive questions. Before editing, restate the goal, constraints, out-of-scope items and acceptance criteria, and label every provisional assumption.
This will not suddenly give an Agent the business experience of a senior consultant. It can reduce one expensive failure mode: working for an hour along the wrong interpretation.
Our judgement
Datafruit's benchmark matters because it turns “the AI built something, but not what I wanted” into a measurable problem. The strongest models still trade missing requirements against adding requirements, suggesting that the bottleneck is moving from execution alone toward how people and models define a problem together.
It remains a Datafruit-built evaluation. The meeting text is generated from the structure of anonymized internal implementation data, while the gold requirements and LLM verifier are designed by the team. Datafruit reports manual checks, gold replay and repeated-judging stability tests, but there is not yet a large outside replication.
The 73.1% figure should therefore not be treated as Opus 5's fixed “understanding rate” on every customer project. The careful conclusion is that frontier models did not reliably complete requirements discovery in this implementation-consulting-style test, and different models failed in very different ways.
What to watch next
- Whether outside teams reproduce these gaps on real, non-generated meeting transcripts;
- Whether allowing Agents to question customers raises recall and precision together;
- Whether a two-stage candidate extraction plus human confirmation workflow reduces rework;
- How models recover implicit security, compliance, permission and failure-mode requirements;
- Whether requirements-discovery scores predict final software-delivery success.
Sources and evidence boundary
- Primary benchmark: Datafruit, “Evaluating Agents on Requirements Discovery,” August 24, 2026
- Peer-reviewed paper: AMBIG-SWE, ICLR 2026
- Primary usage research: Anthropic, “Agentic coding and persistent returns to expertise,” covering roughly 400,000 Claude Code sessions
The 73.1%, 22.4% and 37.3% figures come from Datafruit's own benchmark, not a universal industry accuracy rate. AMBIG-SWE's 74% is the largest relative improvement in selected settings and should not be generalized to every Coding Agent.