New DeepSWE Benchmark Upends LLM Coding Performance Rankings, Exposing Flaws in Industry Standards

A new benchmark dubbed DeepSWE, developed by Data Curve, is poised to redefine how large language models (LLMs) are evaluated for software development tasks. This release critically exposes fundamental flaws in widely adopted benchmarks like SWEBench Pro, Code Arena, and Artificial Analysis, which are accused of relying on contaminated problems, unrealistic prompts, and inadequate verification mechanisms. DeepSWE’s initial results present a stark contrast to previous industry perceptions, positioning OpenAI’s GPT-5.5 with a leading 70% success rate, followed by GPT-5.4 at 56%. Claude Opus 4.7 trailed slightly at 54%, with a significant drop to Sonnet 4.6 at 32%, and open-weight models performing substantially worse, often failing to reach half the scores of the top-tier LLMs. These findings strongly challenge the notion that many models are “meaningfully better” than OpenAI’s state-of-the-art for coding, particularly highlighting Gemini 3.5 Flash’s exceptionally poor real-world performance despite its higher scores on traditional benchmarks.

DeepSWE’s methodology emphasizes realism, featuring tasks written from scratch across a diverse set of active open-source repositories in TypeScript, Go, and Python, ensuring novel problems free from contamination. Prompts are behavior-focused, concise, and reflective of how developers interact with agents, allowing models to discover implementation details rather than execute overly prescriptive instructions. Crucially, the benchmark employs hand-written verifiers that assess software behavior, drastically reducing false positive and negative rates compared to AI-driven methods. Performance analysis also reveals significant efficiency disparities: GPT-5.5 processed tasks using approximately 47K tokens per trial at an average cost of $5.80, while Claude Opus consumed 97K tokens at $16 per run for a lower score. Gemini 3.5 Flash, despite being a “flash” model, incurred similar costs to GPT-5.5 but performed significantly worse, consuming 150K tokens per trial. These results confirm anecdotal developer experiences, underscoring a substantial performance and cost-efficiency gap between leading OpenAI models and other LLMs for practical, complex coding challenges. Data Curve acknowledges limitations, including the use of a minimal agent harness and specific language focus, but stresses its commitment to reflecting actual developer workflows.