Results

34 results

SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps results
ProviderModelScoreSourceRegistry No.
OpenAIGPT-6 Astra (xhigh)48.9%mercor.com for GPT-6 Astra (xhigh) on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps (opens in a new tab)Archived copy (opens in a new tab)Mean pass@1Reported Benchmark leaderboardMercorIndependentRecord for GPT-6 Astra on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps10005
OpenAIGPT-6 Astra (max)47.8%mercor.com for GPT-6 Astra (max) on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps (opens in a new tab)Archived copy (opens in a new tab)Mean pass@1Reported Benchmark leaderboardMercorIndependentRecord for GPT-6 Astra on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps10005
OpenAIGPT-5.4 (xhigh)37.1%mercor.com for GPT-5.4 (xhigh) on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps (opens in a new tab)Archived copy (opens in a new tab)Mean pass@1Reported Benchmark leaderboardMercorIndependentRecord for GPT-5.4 on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps10007
OpenAIGPT-5.5 (xhigh)45.2%mercor.com for GPT-5.5 (xhigh) on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps (opens in a new tab)Archived copy (opens in a new tab)Mean pass@1Reported Benchmark leaderboardMercorIndependentRecord for GPT-5.5 on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps10008
OpenAIGPT-5.6 Luna (max)46.0%mercor.com for GPT-5.6 Luna (max) on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps (opens in a new tab)Archived copy (opens in a new tab)Mean pass@1Reported Benchmark leaderboardMercorIndependentRecord for GPT-5.6 Luna on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps10009
OpenAIGPT-5.6 Terra (max)45.2%mercor.com for GPT-5.6 Terra (max) on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps (opens in a new tab)Archived copy (opens in a new tab)Mean pass@1Reported Benchmark leaderboardMercorIndependentRecord for GPT-5.6 Terra on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps10011
OpenAIGPT-6 Luna (max)33.9%mercor.com for GPT-6 Luna (max) on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps (opens in a new tab)Archived copy (opens in a new tab)Mean pass@1Reported Benchmark leaderboardMercorIndependentRecord for GPT-6 Luna on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps10013
OpenAIGPT-6 Sol (max)44.1%mercor.com for GPT-6 Sol (max) on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps (opens in a new tab)Archived copy (opens in a new tab)Mean pass@1Reported Benchmark leaderboardMercorIndependentRecord for GPT-6 Sol on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps10014
OpenAIGPT-6.1 Sol (max)48.4%mercor.com for GPT-6.1 Sol (max) on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps (opens in a new tab)Archived copy (opens in a new tab)Mean pass@1Reported Benchmark leaderboardMercorIndependentRecord for GPT-6.1 Sol on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps10015
DeepSeek AIDeepSeek-V4-Pro (max)25.0%mercor.com for DeepSeek-V4-Pro (max) on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps (opens in a new tab)Archived copy (opens in a new tab)Mean pass@1Reported Benchmark leaderboardMercorIndependentRecord for DeepSeek-V4-Pro on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps110002
DeepSeek AIDeepSeek-V3.25.9%mercor.com for DeepSeek-V3.2 on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps (opens in a new tab)Archived copy (opens in a new tab)Mean pass@1Reported Benchmark leaderboardMercorIndependentRecord for DeepSeek-V3.2 on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps110003
Moonshot AIKimi K3 (max)52.7%mercor.com for Kimi K3 (max) on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps (opens in a new tab)Archived copy (opens in a new tab)Mean pass@1Reported Benchmark leaderboardMercorIndependentRecord for Kimi K3 on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps120003
Alibaba (Tongyi)Qwen3.8-Max (xhigh)50.5%mercor.com for Qwen3.8-Max (xhigh) on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps (opens in a new tab)Archived copy (opens in a new tab)Mean pass@1Reported Benchmark leaderboardMercorIndependentRecord for Qwen3.8-Max on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps130008
MiniMaxMiniMax M3 (high)37.9%mercor.com for MiniMax M3 (high) on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps (opens in a new tab)Archived copy (opens in a new tab)Mean pass@1Reported Benchmark leaderboardMercorIndependentRecord for MiniMax M3 on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps140003
Z.aiGLM-5.2 (max)46.0%mercor.com for GLM-5.2 (max) on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps (opens in a new tab)Archived copy (opens in a new tab)Mean pass@1Reported Benchmark leaderboardMercorIndependentRecord for GLM-5.2 on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps150004
Z.aiGLM-5.3 (max)42.5%mercor.com for GLM-5.3 (max) on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps (opens in a new tab)Archived copy (opens in a new tab)Mean pass@1Reported Benchmark leaderboardMercorIndependentRecord for GLM-5.3 on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps150005
Thinking Machines LabInkling (high)21.2%mercor.com for Inkling (high) on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps (opens in a new tab)Archived copy (opens in a new tab)Mean pass@1Reported Benchmark leaderboardMercorIndependentRecord for Inkling on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps160001
AnthropicClaude Fable 5.1 (max)43.3%mercor.com for Claude Fable 5.1 (max) on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps (opens in a new tab)Archived copy (opens in a new tab)Mean pass@1Reported Benchmark leaderboardMercorIndependentRecord for Claude Fable 5.1 on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps20004
AnthropicClaude Fable 5.1 (high)42.7%mercor.com for Claude Fable 5.1 (high) on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps (opens in a new tab)Archived copy (opens in a new tab)Mean pass@1Reported Benchmark leaderboardMercorIndependentRecord for Claude Fable 5.1 on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps20004
AnthropicClaude Sonnet 4.6 (high)35.5%mercor.com for Claude Sonnet 4.6 (high) on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps (opens in a new tab)Archived copy (opens in a new tab)Mean pass@1Reported Benchmark leaderboardMercorIndependentRecord for Claude Sonnet 4.6 on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps20009
AnthropicClaude Opus 4.7 (max)47.0%mercor.com for Claude Opus 4.7 (max) on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps (opens in a new tab)Archived copy (opens in a new tab)Mean pass@1Reported Benchmark leaderboardMercorIndependentRecord for Claude Opus 4.7 on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps20010
AnthropicClaude Opus 4.8 (max)44.4%mercor.com for Claude Opus 4.8 (max) on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps (opens in a new tab)Archived copy (opens in a new tab)Mean pass@1Reported Benchmark leaderboardMercorIndependentRecord for Claude Opus 4.8 on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps20011
AnthropicClaude Fable 5 (max)33.1%mercor.com for Claude Fable 5 (max) on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps (opens in a new tab)Archived copy (opens in a new tab)Mean pass@1Reported Benchmark leaderboardMercorIndependentRecord for Claude Fable 5 on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps20012
AnthropicClaude Sonnet 5 (max)39.5%mercor.com for Claude Sonnet 5 (max) on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps (opens in a new tab)Archived copy (opens in a new tab)Mean pass@1Reported Benchmark leaderboardMercorIndependentRecord for Claude Sonnet 5 on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps20013
AnthropicClaude Opus 5 (max)50.3%mercor.com for Claude Opus 5 (max) on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps (opens in a new tab)Archived copy (opens in a new tab)Mean pass@1Reported Benchmark leaderboardMercorIndependentRecord for Claude Opus 5 on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps20014
AnthropicClaude Opus 5.5 (max)48.9%mercor.com for Claude Opus 5.5 (max) on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps (opens in a new tab)Archived copy (opens in a new tab)Mean pass@1Reported Benchmark leaderboardMercorIndependentRecord for Claude Opus 5.5 on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps20015
AnthropicClaude Sonnet 5.5 (max)50.3%mercor.com for Claude Sonnet 5.5 (max) on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps (opens in a new tab)Archived copy (opens in a new tab)Mean pass@1Reported Benchmark leaderboardMercorIndependentRecord for Claude Sonnet 5.5 on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps20016
Google (DeepMind)Gemini 3.1 Pro (high)18.0%mercor.com for Gemini 3.1 Pro (high) on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps (opens in a new tab)Archived copy (opens in a new tab)Mean pass@1Reported Benchmark leaderboardMercorIndependentRecord for Gemini 3.1 Pro on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps30005
Google (DeepMind)Gemini 3.5 Flash (high)19.6%mercor.com for Gemini 3.5 Flash (high) on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps (opens in a new tab)Archived copy (opens in a new tab)Mean pass@1Reported Benchmark leaderboardMercorIndependentRecord for Gemini 3.5 Flash on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps30006
Google (DeepMind)Gemini 3.6 Flash (high)32.0%mercor.com for Gemini 3.6 Flash (high) on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps (opens in a new tab)Archived copy (opens in a new tab)Mean pass@1Reported Benchmark leaderboardMercorIndependentRecord for Gemini 3.6 Flash on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps30007
SpaceXAIGrok 4.5 (high)41.9%mercor.com for Grok 4.5 (high) on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps (opens in a new tab)Archived copy (opens in a new tab)Mean pass@1Reported Benchmark leaderboardMercorIndependentRecord for Grok 4.5 on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps40001
SpaceXAIGrok 4.6 (xhigh)50.3%mercor.com for Grok 4.6 (xhigh) on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps (opens in a new tab)Archived copy (opens in a new tab)Mean pass@1Reported Benchmark leaderboardMercorIndependentRecord for Grok 4.6 on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps40002
SpaceXAIGrok 4.7 (xhigh)47.0%mercor.com for Grok 4.7 (xhigh) on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps (opens in a new tab)Archived copy (opens in a new tab)Mean pass@1Reported Benchmark leaderboardMercorIndependentRecord for Grok 4.7 on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps40003
Meta AI (originally Facebook AI Research)Muse Spark 1.1 (xhigh)3.9%mercor.com for Muse Spark 1.1 (xhigh) on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps (opens in a new tab)Archived copy (opens in a new tab)Mean pass@1Reported Benchmark leaderboardMercorIndependentRecord for Muse Spark 1.1 on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps80002