Benchmarks

17 results across 6 benchmarks

Benchmark results for Claude 3.7 Sonnet
BenchmarkScoreSource
GPQA DiamondEffort: NoneReported as: No extended thinking68.0%anthropic.com for Claude 3.7 Sonnet (No extended thinking) on Graduate-Level Google-Proof Q&A Diamond (opens in a new tab)GPQA Diamond AccuracyReported Launch postAnthropicSelf-reportedRecord for Claude 3.7 Sonnet on Graduate-Level Google-Proof Q&A Diamond
GPQA DiamondReported as: 64K extended thinking78.2%anthropic.com for Claude 3.7 Sonnet (64K extended thinking) on Graduate-Level Google-Proof Q&A Diamond (opens in a new tab)GPQA Diamond AccuracyReported Launch postAnthropicSelf-reportedRecord for Claude 3.7 Sonnet on Graduate-Level Google-Proof Q&A Diamond
GPQA Diamond — Epoch AI harnessConfiguration: Epoch AI harness66.0%epoch.ai for Claude 3.7 Sonnet on Graduate-Level Google-Proof Q&A Diamond — Epoch AI harness (opens in a new tab)Archived copy (opens in a new tab)GPQA Diamond AccuracyReported Evaluated Benchmark leaderboardEpoch AIIndependentRecord for Claude 3.7 Sonnet on Graduate-Level Google-Proof Q&A Diamond — Epoch AI harness
GPQA Diamond — Vals AI harnessConfiguration: Vals AI harness: mean of zero-shot and few-shot CoT67.4%vals.ai for Claude 3.7 Sonnet on Graduate-Level Google-Proof Q&A Diamond — Vals AI harness (opens in a new tab)Archived copy (opens in a new tab)GPQA Diamond AccuracyReported Benchmark leaderboardVals AIIndependentRecord for Claude 3.7 Sonnet on Graduate-Level Google-Proof Q&A Diamond — Vals AI harness
GPQA Diamond — Vals AI harnessReported as: thinkingConfiguration: Vals AI harness: mean of zero-shot and few-shot CoT75.3%vals.ai for Claude 3.7 Sonnet (thinking) on Graduate-Level Google-Proof Q&A Diamond — Vals AI harness (opens in a new tab)Archived copy (opens in a new tab)GPQA Diamond AccuracyReported Benchmark leaderboardVals AIIndependentRecord for Claude 3.7 Sonnet on Graduate-Level Google-Proof Q&A Diamond — Vals AI harness
Humanity's Last Exam Full set — Scale AI evaluationReported as: Thinking8.0%labs.scale.com for Claude 3.7 Sonnet (Thinking) on Humanity's Last Exam Full set — Scale AI evaluation (opens in a new tab)Archived copy (opens in a new tab)Humanity's Last Exam AccuracyReported Benchmark leaderboardScale AIIndependentRecord for Claude 3.7 Sonnet on Humanity's Last Exam Full set — Scale AI evaluation
Humanity's Last Exam Text only — Scale AI evaluationReported as: Thinking7.9%labs.scale.com for Claude 3.7 Sonnet (Thinking) on Humanity's Last Exam Text only — Scale AI evaluation (opens in a new tab)Archived copy (opens in a new tab)Humanity's Last Exam AccuracyReported Benchmark leaderboardScale AIIndependentRecord for Claude 3.7 Sonnet on Humanity's Last Exam Text only — Scale AI evaluation
MMLU-Pro Original — Vals AI harnessConfiguration: Vals AI harness80.7%vals.ai for Claude 3.7 Sonnet on MMLU-Pro Original — Vals AI harness (opens in a new tab)Archived copy (opens in a new tab)AccuracyReported Benchmark leaderboardVals AIIndependentRecord for Claude 3.7 Sonnet on MMLU-Pro Original — Vals AI harness
MMLU-Pro Original — Vals AI harnessReported as: thinkingConfiguration: Vals AI harness82.7%vals.ai for Claude 3.7 Sonnet (thinking) on MMLU-Pro Original — Vals AI harness (opens in a new tab)Archived copy (opens in a new tab)AccuracyReported Benchmark leaderboardVals AIIndependentRecord for Claude 3.7 Sonnet on MMLU-Pro Original — Vals AI harness
MMMU-Pro Vals AI harnessConfiguration: Vals AI harness71.5%vals.ai for Claude 3.7 Sonnet on MMMU-Pro Vals AI harness (opens in a new tab)Archived copy (opens in a new tab)MMMU-Pro accuracySetting: Standard (4 options)Reported Benchmark leaderboardVals AIIndependentRecord for Claude 3.7 Sonnet on MMMU-Pro Vals AI harness
MMMU-Pro Vals AI harnessReported as: thinkingConfiguration: Vals AI harness75.1%vals.ai for Claude 3.7 Sonnet (thinking) on MMMU-Pro Vals AI harness (opens in a new tab)Archived copy (opens in a new tab)MMMU-Pro accuracySetting: Standard (4 options)Reported Benchmark leaderboardVals AIIndependentRecord for Claude 3.7 Sonnet on MMMU-Pro Vals AI harness
OSWorld Verified — 100 stepsConfiguration: 100-step limit35.6%os-world.github.io for Claude 3.7 Sonnet on OSWorld Verified — 100 steps (opens in a new tab)Archived copy (opens in a new tab)OSWorld-Verified success rateReported Benchmark leaderboardOSWorldIndependentXLang LabRecord for Claude 3.7 Sonnet on OSWorld Verified — 100 steps
OSWorld Verified — 15 stepsConfiguration: 15-step limit27.1%os-world.github.io for Claude 3.7 Sonnet on OSWorld Verified — 15 steps (opens in a new tab)Archived copy (opens in a new tab)OSWorld-Verified success rateReported Benchmark leaderboardOSWorldIndependentXLang LabRecord for Claude 3.7 Sonnet on OSWorld Verified — 15 steps
OSWorld Verified — 50 stepsConfiguration: 50-step limit35.8%os-world.github.io for Claude 3.7 Sonnet on OSWorld Verified — 50 steps (opens in a new tab)Archived copy (opens in a new tab)OSWorld-Verified success rateReported Benchmark leaderboardOSWorldIndependentXLang LabRecord for Claude 3.7 Sonnet on OSWorld Verified — 50 steps
SWE-bench VerifiedEffort: NoneReported as: No extended thinking62.3%anthropic.com for Claude 3.7 Sonnet (No extended thinking) on SWE-bench Verified (opens in a new tab)Resolved TasksReported Launch postAnthropicSelf-reportedRecord for Claude 3.7 Sonnet on SWE-bench Verified
SWE-bench Verified — Epoch AI harnessConfiguration: Epoch AI harness61.0%epoch.ai for Claude 3.7 Sonnet on SWE-bench Verified — Epoch AI harness (opens in a new tab)Archived copy (opens in a new tab)Resolved TasksReported Evaluated Benchmark leaderboardEpoch AIIndependentRecord for Claude 3.7 Sonnet on SWE-bench Verified — Epoch AI harness
SWE-bench Verified — mini-swe-agentConfiguration: mini-swe-agent harness52.8%github.com for Claude 3.7 Sonnet on SWE-bench Verified — mini-swe-agent (opens in a new tab)Archived copy (opens in a new tab)Resolved TasksReported Benchmark repositorySWE-benchIndependentSWE-bench TeamRecord for Claude 3.7 Sonnet on SWE-bench Verified — mini-swe-agent