The LLM Arms Race Has a Scoreboard. Here’s What It’s Telling You.

TL;DR – The SWE-bench leaderboard is the closest thing we have to an objective measure of which LLM is actually useful for software engineering. Right now, GPT-5.5 leads at 88.7% on Verified, with Claude Opus 4.7 breathing down its neck at 87.6%. But raw scores tell you about a third of the story. Cost per problem, security implications of agentic models, and what happens when you throw harder benchmarks at these things – that’s where the real decisions live. This post breaks it down.

There’s a game being played at the frontier of AI that’s equal parts Formula One and supermarket price war. Every few weeks, a new model lands, claims the top spot on some benchmark, and the press release hits LinkedIn before most engineers have finished their morning coffee. It’s exhausting and, honestly, a bit silly.

But there is one corner of this madness that’s worth paying attention to: SWE-bench. Unlike the benchmarks designed to make every model look brilliant, SWE-bench is built on real GitHub issues from real Python repositories. The model gets a problem description, a codebase, and a bash shell. It either fixes the issue well enough to pass the existing unit tests, or it doesn’t. No partial credit. No vibes. Just pass or fail.

That’s a harder test than it sounds. And the leaderboard it produces is one of the more honest windows we have into the current state of the art.

What SWE-bench actually measures

A quick primer before we get into the numbers. SWE-bench has several variants, but the two you should care about are:

SWE-bench Verified: 500 human-reviewed instances, created in collaboration with OpenAI. Each task has been validated to confirm the problem description is clear, the tests are correct, and the problem is actually solvable. This is the main event. When people quote a percentage score in a press release, this is usually what they mean.

SWE-bench Pro: A newer, harder benchmark from Scale AI with 731 problems designed to reflect complex, enterprise-level software engineering. Multi-file edits, multiple repos, genuinely difficult problems. If Verified is the driving test, Pro is being handed the keys to a lorry in a snowstorm.

The benchmark evaluates the whole system – model plus scaffold (the agent loop, tool access, context management) – which matters enormously. The same underlying model can score very differently depending on how it’s wrapped. More on that shortly.

The leaderboard as of May 2026

Here’s where things currently stand on SWE-bench Verified. These numbers are a snapshot; the leaderboard moves fast.

A few things jump out immediately.

First, the gap between first and fifth is about eight percentage points. In a benchmark where every percentage represents real bugs fixed on real code, that’s meaningful. But it’s not a chasm. The top five models are all genuinely capable of agentic software engineering work.

Second, look at DeepSeek. At 80.6% on Verified, it’s sitting level with Gemini 3.1 Pro and Claude Opus 4.5, and its output tokens cost $0.28 per million compared to $25.00 for Opus 4.7. That’s roughly an 89x difference in output cost for a model within eight percentage points of the current leader. That number should be tattooed somewhere prominent if you’re making architecture decisions.

Third, Grok 4 is a lesson in why you should always look for independent evaluation. xAI self-reported 72-75% on Verified. Independent testing by vals.ai with a standard SWE-agent scaffold came in at 58.6%. That’s a significant gap, and it illustrates something important: scaffold choice matters enormously, and self-reported numbers from model providers should be treated with polite scepticism.

What happens when you make the test harder

SWE-bench Pro is where the real humility kicks in. On the easier Verified benchmark, the best models are cracking 88%. Move to Pro, and the top score drops to around 23%. The same model that fixes nearly nine in ten standard GitHub issues manages less than a quarter of enterprise-level problems. Performance on multi-file edits degrades sharply. Smaller models go from moderate to nearly useless. The benchmark that was approaching saturation suddenly has a lot of headroom again.

This matters practically. If you’re evaluating an LLM for use on a greenfield CRUD app, Verified scores are a reasonable signal. If you’re thinking about agentic coding assistants on a complex enterprise codebase with multi-service dependencies and a ten-year history of technical debt, the Pro numbers are more honest about what you’d actually be getting.

There’s also a subtler issue. A March 2026 preprint found that roughly 20% of patches marked as “solved” by top leaderboard agents are actually semantically incorrect – they pass the existing unit tests but produce wrong behaviour. The tests themselves have coverage gaps. This doesn’t invalidate the benchmark, but it does mean that even the headline Verified scores are somewhat optimistic. The ceiling may be lower than the numbers suggest.

The cost conversation no one wants to have until it’s too late

Let’s talk money, because this is usually where good engineering decisions go to die.

LLM pricing has collapsed dramatically since 2024, but the spread across the current generation is still enormous. Running a coding agent on GPT-5.5 or Claude Opus 4.7 for anything resembling production workloads is expensive if you’re not careful about token management. Agentic tasks are particularly costly because they generate a lot of output tokens – reasoning steps, code, commentary, retry loops – and output tokens are always priced higher than input tokens.

The practical options, roughly tiered:

Frontier performance, frontier cost: GPT-5.5 (~$15/M output), Claude Opus 4.7 (~$25/M output). Use when you need the best, accept the bill, and enable prompt caching aggressively. Both providers offer batch API discounts of around 50% for non-interactive workloads. That’s a real lever.

Very good performance, reasonable cost: Claude Sonnet 4.6 (~$15/M output, comparable Verified scores to Opus), Gemini 3.1 Pro (~$12/M output). The Sonnet line in particular has been closing the gap with Opus for coding tasks. If you’re not doing the most complex work, Sonnet is frequently good enough and meaningfully cheaper.

Surprisingly capable, significantly cheaper: DeepSeek V4 Pro Max, Gemini 3 Flash. DeepSeek at $0.28/M output is not a toy. At 80.6% on Verified it’s in the same tier as models costing 80 times more. The caveats are latency on the public API and questions about data routing if you’re working with sensitive code – more on that below.

The open-weight option: Kimi K2.6, Qwen3-Coder, Llama 4 variants. Free to download, expensive to run yourself, cheap via third-party hosting. If you have GPU infrastructure or are using providers like Together or Fireworks, the economics change. Qwen3-Coder-Next has been putting up competitive Verified scores using a fraction of the parameters of its closed-source competition.

A rough rule: for most coding agent pipelines, start with Sonnet or Gemini Flash, validate quality against your actual tasks, and only reach for Opus or GPT-5.5 if the quality delta is demonstrably worth it for your specific use case. It usually isn’t for straightforward tasks. It sometimes is for complex ones.

The security problem that nobody put in the press release

Here’s where it gets less fun.

Agentic coding models – the kind that sit inside Claude Code, Copilot, Cursor, or your homegrown agent loop – are not just answering questions. They have file I/O, shell access, network access, and the ability to write and execute code. They operate with significantly elevated trust compared to a chat interface. And they have a well-documented vulnerability: prompt injection.

The attack surface is larger than most teams appreciate. A malicious instruction doesn’t have to come from the user. It can come from a README in a repository the agent reads for context, from a JSON document it processes, from a comment in a file it opens, or from a poisoned dependency in the project. The agent reads it, the instruction gets interpreted, and something bad happens. NIST has characterised prompt injection as generative AI’s greatest security flaw, and OWASP ranks it first in their LLM Applications Top 10 for a reason.

The numbers here are uncomfortable. Around 45% of AI-generated code contains security flaws according to Veracode’s 2025 GenAI Code Security Report. Even Claude, which blocks around 88% of prompt injection attempts according to Anthropic’s own system card, leaves a 12% gap. In practice, that gap can be exploited by burying injections inside structured data that confuses detection heuristics.

The specific threats worth understanding:

Prompt injection via repository content: When an agent explores a codebase for context, it often reads documentation, config files, and comments. Any of these can contain injected instructions. This is particularly relevant if your agent is working with third-party or public repositories.

Supply chain poisoning: Researchers have documented attackers planting malicious instructions inside skill files and tool definitions in public registries. An agent that loads a poisoned skill gets compromised. If the skill has file access, that compromise can mean exfiltration. CVE-2025-59536 is a real example of this playing out in the wild.

Excessive agency: OWASP’s Agentic Applications Top 10 (published December 2025) calls this out explicitly. When an agent can take actions – commit code, send requests, modify files – without human review at each step, the blast radius of a successful injection expands dramatically. The “auto-approve” checkboxes in most coding tools exist precisely to hand over this blast radius.

Data routing considerations: This one is less about attacks and more about compliance. When you’re sending code to an LLM API, you’re sending it somewhere. For Anthropic, OpenAI, and Google, you have contracts, terms, and (for enterprise tiers) zero data retention options. For DeepSeek, you’re routing to servers in China. For most teams working on internal tooling or non-sensitive applications, that’s fine. For anyone with GDPR obligations, financial data, or defence-adjacent work, it’s a hard stop. This isn’t a political point; it’s a data governance one.

The practical mitigations aren’t complicated, but they require deliberate implementation: human-in-the-loop review gates for consequential actions, minimal permission scoping for agent tool access, treating everything an agent reads as potentially adversarial, and reading the code that your MCP servers execute before you trust them. The risks are manageable. They’re just not zero, and the speed of adoption is outpacing the security upskilling.

Which model for which job

Given all of the above, here’s a rough decision framework.

You’re building a coding agent for internal use on a standard codebase: Claude Sonnet 4.6 or Gemini 3.1 Pro. Good Verified scores, reasonable cost, solid context windows, established enterprise data agreements. Enable caching. Use the batch API for non-interactive workloads.

You need frontier performance and cost is secondary: GPT-5.5 or Claude Opus 4.7. Both are genuinely excellent. GPT-5.5 has a slight edge on Verified; Claude leads on multi-language tasks (Aider Polyglot, where Opus 4.5 hits 89.4%). Anthropic’s 50% batch discount is meaningful at scale.

You’re optimising for cost and can accept slightly lower performance: DeepSeek V4 Pro Max, if data routing is not a concern. Gemini 3 Flash if you need European data residency. The value per dollar for both is exceptional. For open-weight enthusiasts, Qwen3-Coder-Next is competitive at a fraction of the token cost.

You’re doing complex enterprise engineering work: Check the Pro scores, not just Verified. The models that maintain consistency across repositories and languages on harder tasks are GPT-5.5 and Claude Opus 4.7. The performance cliff on Pro is real for everything below the top tier.

You’re evaluating for security-sensitive environments: Implement human review gates regardless of model choice. Treat the model’s file and shell access as you’d treat any other privileged service. Audit what your MCP servers do before connecting them. And maybe don’t auto-approve.

The benchmark problem in brief

One thing worth holding onto: SWE-bench, like all benchmarks, measures what it measures. The models at the top of the Verified leaderboard are demonstrably good at fixing Python bugs on well-documented open-source codebases with existing unit tests. That’s a real and useful capability. It’s not the same as general software engineering competence across arbitrary languages, architectures, and constraints.

SWE-bench Pro is a better approximation of real work, and the 60+ percentage point drop from Verified to Pro scores tells you that the gap between benchmark performance and production capability is still significant. The models are getting better at an impressive rate – the top Verified score went from around 65% in early 2025 to 88.7% by May 2026 – but the harder the problem, the more headroom remains.

The leaderboard is worth watching. It’s just not worth treating as a complete specification.

The punchline

The LLM field has a habit of generating more heat than light. The SWE-bench leaderboard is one of the places where there’s actual light. Models are measurably getting better at real engineering tasks, the cost curve is collapsing faster than most predictions suggested, and the open-weight community is closing the gap with closed-source frontrunners in ways that would have seemed unlikely twelve months ago.

The security implications of handing agentic systems elevated system privileges are not yet being taken seriously enough relative to the speed of adoption. That will be corrected either by the industry getting ahead of it, or by an incident that forces the issue. The former would be nicer.

Pick your model based on your actual workload. Do the cost maths. Read the security considerations. And keep an eye on what the next Pro benchmark scores look like, because that’s where the real capability ceiling is visible.

The race is not over. It’s just that the easy laps are done.