The same DeepSeek lost 33 points with a different harness: what model rankings hide

Conclusion

The same DeepSeek-V4-Pro-0813, under the same benchmark name, now has two radically different scorecards.

DeepSeek's model card reports 87.9% on Terminal-Bench 2.1. Vals AI independently reports 54.68% ±1.50, a gap of 33.22 percentage points. Yet on SWE-bench Verified, where Vals uses the same mini-swe-agent across models, DeepSeek scores 96.4% ±0.83, ranking second among 86 systems and only 0.6 points behind Claude Opus 5.

This is not a choice between "the vendor cheated" and "DeepSeek has fully matched Claude." The 87.9% and 54.68% runs are not like-for-like replications. Their difference includes the harness, tools, prompt, reasoning level and run rules. The defensible conclusion is narrower: DeepSeek V4 Pro is extremely strong at repository repair with clear goals and acceptance tests, but one vendor table cannot settle its long-horizon Agent capability.

The 87.9% is striking, but the footnote matters

DeepSeek released V4 Pro 0813 on August 13. Its Hugging Face model card describes an open-weight model with roughly 1.7T parameters, a 1M-token context window and an MIT license. DeepSeek says the 0813 release focuses heavily on improving Agent capability over the Preview checkpoint.

The most striking row is Terminal-Bench 2.1:

• DeepSeek V4 Pro 0813: 87.9;
• Claude Opus 4.8: 85.0;
• Claude Fable 5 with fallback: 88.0.

Read alone, the table puts DeepSeek beside the strongest Claude tier. The card also states the configuration: public coding-agent benchmarks use DeepSeek Harness minimal mode, reasoning_effort = max, temperature = 1.0 and top_p = 0.95.

The accurate description is therefore not "the bare model scored 87.9%." It is: DeepSeek V4 Pro, combined with DeepSeek's selected harness and reasoning configuration, scored 87.9% in the vendor evaluation.

First reversal: the independent run falls to 54.68%

Vals' DeepSeek V4 Pro 0813 model page reports 54.68% ±1.50 on Terminal-Bench 2.1, currently placing it 36th among 57 systems on the page. Vals' update says the result comes from three full trials and includes a 28.89% score on hard tasks.

The move from 87.9% to 54.68% is 33.22 percentage points. That is not a rounding difference or the sort of small movement explained by one random seed or a few tasks.

There is still an important limit. The Vals model page does not demonstrate that it copied DeepSeek's system prompt, harness version, timeout, retry policy and stopping conditions item by item. The gap proves that a different evaluation system can produce a radically different result. By itself, it does not prove the vendor result is irreproducible or intentionally inflated.

What the harness actually does for the model

A conventional question-answer benchmark is close to "give the model a question and grade one answer." Terminal-Bench evaluates a system that keeps acting. Its final score depends on at least the following parts:

• How the system prompt tells the model to plan, verify and stop;
• How shell, file and other tools are exposed, and whether results are easy to interpret;
• What is retained or compressed when context fills up;
• Whether failed commands are retried and repeated behavior is interrupted;
• Token, time and step limits for each task;
• When the harness declares completion and when it continues checking.

These are not decorative details. They decide whether the model finds the right files, remembers failures, changes course and stops after solving the task. A 33-point Terminal-Bench gap is enough to show that the harness cannot remain a leaderboard footnote.

Second reversal: it really is strong on SWE-bench

If DeepSeek's capability existed only inside its own harness, it should collapse across independent shared environments. Vals' SWE-bench Verified results do not support that claim.

Vals gives every model the same mini-swe-agent and only one tool, Bash. Models work on 500 human-verified GitHub issues inside isolated Docker containers. The August 19 leaderboard reads:

• Claude Opus 5: 97.0% ±0.76;
• DeepSeek V4 Pro 0813: 96.4% ±0.83;
• GPT-5.6 Sol: 96.2% ±0.86;
• Grok 4.6: 95.6% ±0.92;
• GPT-5.6 Terra: 95.4% ±0.94.

There is no DeepSeek-specific harness in this comparison. The model still ranks second, and the 0.6-point gap sits within the reported error bars around both scores. On this class of work, reading a repository, locating a problem, producing a patch and passing tests, DeepSeek is already in the top tier.

Why the two benchmarks draw opposite portraits

SWE-bench Verified has relatively clear boundaries: a real issue, a repository and a final patch judged by tests. The model still has to search, edit and run commands, but the success condition is concrete.

Terminal-Bench depends more heavily on continuous decisions in a terminal. An Agent must understand the environment, combine commands, handle unexpected output, recover from failure and decide when it is finished. As the task gets longer, one early reading, memory or planning error has more opportunities to compound.

That is why all three figures can be true:

• 87.9%: a vendor Agent result under DeepSeek's selected system;
• 54.68%: a Vals independent Terminal-Bench system result;
• 96.4%: a Vals shared-harness SWE-bench repair result.

They do not measure one abstract "coding IQ." They measure delivery under three different working arrangements.

The cost is attractive, but the public figures conflict

DeepSeek's price advantage still matters. However, Vals' current pages contain a discrepancy worth preserving: the update text on the model page says $0.02 per SWE-bench test, while the live SWE-bench table currently shows $0.10. The same table lists $1.29 for Claude Opus 5.

Until Vals explains the two DeepSeek figures, this article does not use a precise "60 times cheaper" claim. The current table supports a materially lower cost per test. It does not establish whether $0.02 or $0.10 is the durable comparison point.

How developers should choose

If your work resembles SWE-bench, a clear issue, a known repository, explicit tests and a clean finish, DeepSeek V4 Pro belongs on the shortlist. Do not rely only on public tasks. Run a blind set of your team's historical bugs, refactors and small features, then record success rate, rework time and total cost per mergeable pull request.

If the work requires cross-service exploration, opening websites, reading logs, resolving ambiguous requirements or operating autonomously for hours, do not treat 96.4% as a universal success rate. Test the complete product, including harness memory, permissions, retries, approvals and failure recovery, rather than changing only the API model name.

A practical router may send bounded bulk repairs to the lower-cost model, reserve complex work for a system with stronger long-horizon behavior, and use another model or a human for final review. The unit of selection should be the delivered result, not price per million tokens or the highest leaderboard number.

Our judgment

These results raise our view of DeepSeek V4 Pro's underlying capability. After removing the vendor-specific harness, the model did not collapse across every coding benchmark. It reached 96.4% in a shared SWE-bench environment. That is strong independent evidence.

At the same time, the 87.9% to 54.68% swing shows why the model cannot be reduced to "a cheaper Claude." The same checkpoint can move by 33 points when its execution system changes. Agent leaderboard claims should now name the model, harness, tools, reasoning level, timeout and cost together.

What to watch next

• Whether DeepSeek submits a reproducible configuration to the official Terminal-Bench leaderboard;
• Whether outside teams can reproduce 87.9% with public harness, prompt, timeout and retry rules;
• How Vals reconciles the $0.02 and $0.10 cost figures;
• Whether models clustered around 96% on SWE-bench separate reliably on longer and more ambiguous work;
• Whether different harnesses systematically favor DeepSeek, Claude, GPT or Qwen.

Sources and evidence boundary

• Primary source: DeepSeek's Hugging Face model card supports model specifications, vendor scores and Agent settings;
• Independent evaluation: Vals' DeepSeek model page supports the 54.68% Terminal-Bench result and error bar;
• Independent shared evaluation: Vals' SWE-bench Verified page supports the 96.4% result, ranking, shared mini-swe-agent and current cost table;
• Evaluation methodology: Vals' methodology explains error bars, cost reporting and the scope of Agent-system evaluation.

This article does not present 87.9% and 54.68% as a like-for-like replication, and it does not derive a universal coding success rate from two different benchmarks. The firmest conclusion is that DeepSeek V4 Pro is excellent in a shared repository-repair test, while the enormous Terminal-Bench gap exposes how much the harness can change an Agent score.