Results

47 results

DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps results
ProviderModelScoreSourceRegistry No.
OpenAIGPT-6 Astra (max)72.0%mercor.com for GPT-6 Astra (max) on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps (opens in a new tab)Archived copy (opens in a new tab)DeepSWE Pass@1Reported Benchmark leaderboardMercorIndependentRecord for GPT-6 Astra on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps10005
OpenAIGPT-5.4 (xhigh)49.0%mercor.com for GPT-5.4 (xhigh) on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps (opens in a new tab)Archived copy (opens in a new tab)DeepSWE Pass@1Reported Benchmark leaderboardMercorIndependentRecord for GPT-5.4 on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps10007
OpenAIGPT-5.5 (xhigh)66.1%mercor.com for GPT-5.5 (xhigh) on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps (opens in a new tab)Archived copy (opens in a new tab)DeepSWE Pass@1Reported Benchmark leaderboardMercorIndependentRecord for GPT-5.5 on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps10008
OpenAIGPT-5.6 Luna (max)64.3%mercor.com for GPT-5.6 Luna (max) on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps (opens in a new tab)Archived copy (opens in a new tab)DeepSWE Pass@1Reported Benchmark leaderboardMercorIndependentRecord for GPT-5.6 Luna on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps10009
OpenAIGPT-5.6 Sol (max)70.5%mercor.com for GPT-5.6 Sol (max) on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps (opens in a new tab)Archived copy (opens in a new tab)DeepSWE Pass@1Reported Benchmark leaderboardMercorIndependentRecord for GPT-5.6 Sol on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps10010
OpenAIGPT-5.6 Terra (max)70.2%mercor.com for GPT-5.6 Terra (max) on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps (opens in a new tab)Archived copy (opens in a new tab)DeepSWE Pass@1Reported Benchmark leaderboardMercorIndependentRecord for GPT-5.6 Terra on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps10011
OpenAIGPT-6 Luna (max)67.0%mercor.com for GPT-6 Luna (max) on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps (opens in a new tab)Archived copy (opens in a new tab)DeepSWE Pass@1Reported Benchmark leaderboardMercorIndependentRecord for GPT-6 Luna on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps10013
OpenAIGPT-6 Sol (max)70.5%mercor.com for GPT-6 Sol (max) on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps (opens in a new tab)Archived copy (opens in a new tab)DeepSWE Pass@1Reported Benchmark leaderboardMercorIndependentRecord for GPT-6 Sol on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps10014
OpenAIGPT-6.1 Sol (max)72.3%mercor.com for GPT-6.1 Sol (max) on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps (opens in a new tab)Archived copy (opens in a new tab)DeepSWE Pass@1Reported Benchmark leaderboardMercorIndependentRecord for GPT-6.1 Sol on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps10015
DeepSeek AIDeepSeek-V4-Pro (max)9.1%mercor.com for DeepSeek-V4-Pro (max) on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps (opens in a new tab)Archived copy (opens in a new tab)DeepSWE Pass@1Reported Benchmark leaderboardMercorIndependentRecord for DeepSeek-V4-Pro on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps110002
DeepSeek AIDeepSeek-V3.20.6%mercor.com for DeepSeek-V3.2 on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps (opens in a new tab)Archived copy (opens in a new tab)DeepSWE Pass@1Reported Benchmark leaderboardMercorIndependentRecord for DeepSeek-V3.2 on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps110003
Moonshot AIKimi K3 (max)66.4%mercor.com for Kimi K3 (max) on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps (opens in a new tab)Archived copy (opens in a new tab)DeepSWE Pass@1Reported Benchmark leaderboardMercorIndependentRecord for Kimi K3 on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps120003
Alibaba (Tongyi)Qwen3.5-397B-A17B0.0%mercor.com for Qwen3.5-397B-A17B on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps (opens in a new tab)Archived copy (opens in a new tab)DeepSWE Pass@1Reported Benchmark leaderboardMercorIndependentRecord for Qwen3.5-397B-A17B on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps130003
Alibaba (Tongyi)Qwen3.8-Max (xhigh)53.1%mercor.com for Qwen3.8-Max (xhigh) on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps (opens in a new tab)Archived copy (opens in a new tab)DeepSWE Pass@1Reported Benchmark leaderboardMercorIndependentRecord for Qwen3.8-Max on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps130008
MiniMaxMiniMax M2.7 (high)0.0%mercor.com for MiniMax M2.7 (high) on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps (opens in a new tab)Archived copy (opens in a new tab)DeepSWE Pass@1Reported Benchmark leaderboardMercorIndependentRecord for MiniMax M2.7 on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps140002
MiniMaxMiniMax M3 (high)9.1%mercor.com for MiniMax M3 (high) on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps (opens in a new tab)Archived copy (opens in a new tab)DeepSWE Pass@1Reported Benchmark leaderboardMercorIndependentRecord for MiniMax M3 on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps140003
Z.aiGLM-52.1%mercor.com for GLM-5 on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps (opens in a new tab)Archived copy (opens in a new tab)DeepSWE Pass@1Reported Benchmark leaderboardMercorIndependentRecord for GLM-5 on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps150002
Z.aiGLM-5.118.9%mercor.com for GLM-5.1 on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps (opens in a new tab)Archived copy (opens in a new tab)DeepSWE Pass@1Reported Benchmark leaderboardMercorIndependentRecord for GLM-5.1 on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps150003
Z.aiGLM-5.2 (max)38.9%mercor.com for GLM-5.2 (max) on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps (opens in a new tab)Archived copy (opens in a new tab)DeepSWE Pass@1Reported Benchmark leaderboardMercorIndependentRecord for GLM-5.2 on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps150004
Z.aiGLM-5.3 (max)70.5%mercor.com for GLM-5.3 (max) on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps (opens in a new tab)Archived copy (opens in a new tab)DeepSWE Pass@1Reported Benchmark leaderboardMercorIndependentRecord for GLM-5.3 on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps150005
Z.aiGLM-5.3-Flash (max)42.5%mercor.com for GLM-5.3-Flash (max) on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps (opens in a new tab)Archived copy (opens in a new tab)DeepSWE Pass@1Reported Benchmark leaderboardMercorIndependentRecord for GLM-5.3-Flash on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps150006
OpenAIgpt-oss-120b (high)0.0%mercor.com for gpt-oss-120b (high) on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps (opens in a new tab)Archived copy (opens in a new tab)DeepSWE Pass@1Reported Benchmark leaderboardMercorIndependentRecord for gpt-oss-120b on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps15001
Thinking Machines LabInkling (high)5.0%mercor.com for Inkling (high) on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps (opens in a new tab)Archived copy (opens in a new tab)DeepSWE Pass@1Reported Benchmark leaderboardMercorIndependentRecord for Inkling on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps160001
AnthropicClaude Fable 5.1 (max)64.9%mercor.com for Claude Fable 5.1 (max) on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps (opens in a new tab)Archived copy (opens in a new tab)DeepSWE Pass@1Reported Benchmark leaderboardMercorIndependentRecord for Claude Fable 5.1 on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps20004
AnthropicClaude Fable 5.1 (high)67.3%mercor.com for Claude Fable 5.1 (high) on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps (opens in a new tab)Archived copy (opens in a new tab)DeepSWE Pass@1Reported Benchmark leaderboardMercorIndependentRecord for Claude Fable 5.1 on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps20004
AnthropicClaude Opus 4.6 (max)31.9%mercor.com for Claude Opus 4.6 (max) on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps (opens in a new tab)Archived copy (opens in a new tab)DeepSWE Pass@1Reported Benchmark leaderboardMercorIndependentRecord for Claude Opus 4.6 on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps20008
AnthropicClaude Sonnet 4.6 (high)34.2%mercor.com for Claude Sonnet 4.6 (high) on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps (opens in a new tab)Archived copy (opens in a new tab)DeepSWE Pass@1Reported Benchmark leaderboardMercorIndependentRecord for Claude Sonnet 4.6 on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps20009
AnthropicClaude Opus 4.7 (max)49.3%mercor.com for Claude Opus 4.7 (max) on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps (opens in a new tab)Archived copy (opens in a new tab)DeepSWE Pass@1Reported Benchmark leaderboardMercorIndependentRecord for Claude Opus 4.7 on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps20010
AnthropicClaude Opus 4.8 (max)59.6%mercor.com for Claude Opus 4.8 (max) on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps (opens in a new tab)Archived copy (opens in a new tab)DeepSWE Pass@1Reported Benchmark leaderboardMercorIndependentRecord for Claude Opus 4.8 on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps20011
AnthropicClaude Fable 5 (max)67.3%mercor.com for Claude Fable 5 (max) on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps (opens in a new tab)Archived copy (opens in a new tab)DeepSWE Pass@1Reported Benchmark leaderboardMercorIndependentRecord for Claude Fable 5 on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps20012
AnthropicClaude Sonnet 5 (max)49.6%mercor.com for Claude Sonnet 5 (max) on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps (opens in a new tab)Archived copy (opens in a new tab)DeepSWE Pass@1Reported Benchmark leaderboardMercorIndependentRecord for Claude Sonnet 5 on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps20013
AnthropicClaude Opus 5 (max)71.4%mercor.com for Claude Opus 5 (max) on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps (opens in a new tab)Archived copy (opens in a new tab)DeepSWE Pass@1Reported Benchmark leaderboardMercorIndependentRecord for Claude Opus 5 on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps20014
AnthropicClaude Opus 5 (xhigh)70.2%mercor.com for Claude Opus 5 (xhigh) on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps (opens in a new tab)Archived copy (opens in a new tab)DeepSWE Pass@1Reported Benchmark leaderboardMercorIndependentRecord for Claude Opus 5 on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps20014
AnthropicClaude Opus 5.5 (max)72.3%mercor.com for Claude Opus 5.5 (max) on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps (opens in a new tab)Archived copy (opens in a new tab)DeepSWE Pass@1Reported Benchmark leaderboardMercorIndependentRecord for Claude Opus 5.5 on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps20015
AnthropicClaude Sonnet 5.5 (max)70.5%mercor.com for Claude Sonnet 5.5 (max) on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps (opens in a new tab)Archived copy (opens in a new tab)DeepSWE Pass@1Reported Benchmark leaderboardMercorIndependentRecord for Claude Sonnet 5.5 on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps20016
Google (DeepMind)Gemini 3.8 Flash (high)71.4%mercor.com for Gemini 3.8 Flash (high) on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps (opens in a new tab)Archived copy (opens in a new tab)DeepSWE Pass@1Reported Benchmark leaderboardMercorIndependentRecord for Gemini 3.8 Flash on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps30004
Google (DeepMind)Gemini 3.1 Pro (high)7.7%mercor.com for Gemini 3.1 Pro (high) on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps (opens in a new tab)Archived copy (opens in a new tab)DeepSWE Pass@1Reported Benchmark leaderboardMercorIndependentRecord for Gemini 3.1 Pro on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps30005
Google (DeepMind)Gemini 3.5 Flash (high)36.6%mercor.com for Gemini 3.5 Flash (high) on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps (opens in a new tab)Archived copy (opens in a new tab)DeepSWE Pass@1Reported Benchmark leaderboardMercorIndependentRecord for Gemini 3.5 Flash on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps30006
Google (DeepMind)Gemini 3.6 Flash (high)49.3%mercor.com for Gemini 3.6 Flash (high) on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps (opens in a new tab)Archived copy (opens in a new tab)DeepSWE Pass@1Reported Benchmark leaderboardMercorIndependentRecord for Gemini 3.6 Flash on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps30007
Google (DeepMind)Gemini 3.7 Flash (high)65.5%mercor.com for Gemini 3.7 Flash (high) on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps (opens in a new tab)Archived copy (opens in a new tab)DeepSWE Pass@1Reported Benchmark leaderboardMercorIndependentRecord for Gemini 3.7 Flash on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps30010
Google (DeepMind)Gemma 4 31B Instruct0.0%mercor.com for Gemma 4 31B Instruct on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps (opens in a new tab)Archived copy (opens in a new tab)DeepSWE Pass@1Reported Benchmark leaderboardMercorIndependentRecord for Gemma 4 31B Instruct on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps35002
SpaceXAIGrok 4.5 (high)49.3%mercor.com for Grok 4.5 (high) on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps (opens in a new tab)Archived copy (opens in a new tab)DeepSWE Pass@1Reported Benchmark leaderboardMercorIndependentRecord for Grok 4.5 on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps40001
SpaceXAIGrok 4.6 (high)63.7%mercor.com for Grok 4.6 (high) on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps (opens in a new tab)Archived copy (opens in a new tab)DeepSWE Pass@1Reported Benchmark leaderboardMercorIndependentRecord for Grok 4.6 on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps40002
SpaceXAIGrok 4.6 (xhigh)24.8%mercor.com for Grok 4.6 (xhigh) on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps (opens in a new tab)Archived copy (opens in a new tab)DeepSWE Pass@1Reported Benchmark leaderboardMercorIndependentRecord for Grok 4.6 on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps40002
SpaceXAIGrok 4.7 (xhigh)33.3%mercor.com for Grok 4.7 (xhigh) on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps (opens in a new tab)Archived copy (opens in a new tab)DeepSWE Pass@1Reported Benchmark leaderboardMercorIndependentRecord for Grok 4.7 on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps40003
NVIDIANVIDIA Nemotron 3 Ultra 550B-A55B (high)0.6%mercor.com for NVIDIA Nemotron 3 Ultra 550B-A55B (high) on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps (opens in a new tab)Archived copy (opens in a new tab)DeepSWE Pass@1Reported Benchmark leaderboardMercorIndependentRecord for NVIDIA Nemotron 3 Ultra 550B-A55B on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps60003
Meta AI (originally Facebook AI Research)Muse Spark 1.1 (xhigh)40.4%mercor.com for Muse Spark 1.1 (xhigh) on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps (opens in a new tab)Archived copy (opens in a new tab)DeepSWE Pass@1Reported Benchmark leaderboardMercorIndependentRecord for Muse Spark 1.1 on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps80002