Benchmarks

17 results across 14 benchmarks

Benchmark results for Gemma 4 31B Instruct
BenchmarkScoreSource
AutomationBench 1.0.6Effort: Max1.7%zapier.com for Gemma 4 31B Instruct (max) on AutomationBench 1.0.6 (opens in a new tab)Archived copy (opens in a new tab)Task pass rateReported Benchmark leaderboardZapierIndependentZapier AutomationBench TeamRecord for Gemma 4 31B Instruct on AutomationBench 1.0.6
BIG-Bench Extra Hard Original — micro averageReported as: thinking mode74.4%huggingface.co for Gemma 4 31B Instruct (thinking mode) on BIG-Bench Extra Hard Original — micro average (opens in a new tab)Micro-average accuracyReported Model cardGoogleSelf-reportedGoogle DeepMindRecord for Gemma 4 31B Instruct on BIG-Bench Extra Hard Original — micro average
DeepSWE 1.1 — Mercor mini-swe-agent, 500 stepsConfiguration: Mercor mini-swe-agent, 500 steps, 2 hours0.0%mercor.com for Gemma 4 31B Instruct on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps (opens in a new tab)Archived copy (opens in a new tab)DeepSWE Pass@1Reported Benchmark leaderboardMercorIndependentRecord for Gemma 4 31B Instruct on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps
GPQA DiamondReported as: thinking mode84.3%huggingface.co for Gemma 4 31B Instruct (thinking mode) on Graduate-Level Google-Proof Q&A Diamond (opens in a new tab)GPQA Diamond AccuracyReported Model cardGoogleSelf-reportedGoogle DeepMindRecord for Gemma 4 31B Instruct on Graduate-Level Google-Proof Q&A Diamond
GPQA Diamond — Mercor single-shotConfiguration: Mercor single-shot agent85.5%mercor.com for Gemma 4 31B Instruct on Graduate-Level Google-Proof Q&A Diamond — Mercor single-shot (opens in a new tab)Archived copy (opens in a new tab)GPQA Diamond AccuracyReported Benchmark leaderboardMercorIndependentRecord for Gemma 4 31B Instruct on Graduate-Level Google-Proof Q&A Diamond — Mercor single-shot
Humanity's Last Exam Text only — Mercor single-shotConfiguration: Mercor single-shot agent23.2%mercor.com for Gemma 4 31B Instruct on Humanity's Last Exam Text only — Mercor single-shot (opens in a new tab)Archived copy (opens in a new tab)Humanity's Last Exam AccuracyReported Benchmark leaderboardMercorIndependentRecord for Gemma 4 31B Instruct on Humanity's Last Exam Text only — Mercor single-shot
IFBench OriginalReported as: thinking mode76.0%arxiv.org for Gemma 4 31B Instruct (thinking mode) on IFBench Original (opens in a new tab)AccuracyReported Technical reportGoogle DeepMindSelf-reportedRecord for Gemma 4 31B Instruct on IFBench Original
IFEval OriginalReported as: thinking mode98.9%huggingface.co for Gemma 4 31B Instruct (thinking mode) on IFEval Original (opens in a new tab)AccuracyReported Model cardGoogleSelf-reportedGoogle DeepMindRecord for Gemma 4 31B Instruct on IFEval Original
MATH-Vision OriginalReported as: thinking mode85.6%huggingface.co for Gemma 4 31B Instruct (thinking mode) on MATH-Vision Original (opens in a new tab)AccuracyReported Model cardGoogleSelf-reportedGoogle DeepMindRecord for Gemma 4 31B Instruct on MATH-Vision Original
MedXpertQA MMReported as: thinking mode61.3%huggingface.co for Gemma 4 31B Instruct (thinking mode) on MedXpertQA MM (opens in a new tab)AccuracyReported Model cardGoogleSelf-reportedGoogle DeepMindRecord for Gemma 4 31B Instruct on MedXpertQA MM
MMLU-Pro OriginalReported as: thinking mode85.2%huggingface.co for Gemma 4 31B Instruct (thinking mode) on MMLU-Pro Original (opens in a new tab)AccuracyReported Model cardGoogleSelf-reportedGoogle DeepMindRecord for Gemma 4 31B Instruct on MMLU-Pro Original
MMLU-Pro Original — Mercor single-shotConfiguration: Mercor single-shot agent85.8%mercor.com for Gemma 4 31B Instruct on MMLU-Pro Original — Mercor single-shot (opens in a new tab)Archived copy (opens in a new tab)AccuracyReported Benchmark leaderboardMercorIndependentRecord for Gemma 4 31B Instruct on MMLU-Pro Original — Mercor single-shot
SciCode OriginalReported as: thinking mode43.0%arxiv.org for Gemma 4 31B Instruct (thinking mode) on SciCode Original (opens in a new tab)AccuracyReported Technical reportGoogle DeepMindSelf-reportedRecord for Gemma 4 31B Instruct on SciCode Original
SimpleQA Verified Original — Epoch AI harnessConfiguration: Epoch AI harness10.4%epoch.ai for Gemma 4 31B Instruct on SimpleQA Verified Original — Epoch AI harness (opens in a new tab)Archived copy (opens in a new tab)AccuracyReported Evaluated Benchmark leaderboardEpoch AIIndependentRecord for Gemma 4 31B Instruct on SimpleQA Verified Original — Epoch AI harness
SWE-bench Verified — Mercor mini-swe-agent, 1000 stepsConfiguration: Mercor mini-swe-agent, 1000 steps, 3 hours46.1%mercor.com for Gemma 4 31B Instruct on SWE-bench Verified — Mercor mini-swe-agent, 1000 steps (opens in a new tab)Archived copy (opens in a new tab)Resolved TasksReported Benchmark leaderboardMercorIndependentRecord for Gemma 4 31B Instruct on SWE-bench Verified — Mercor mini-swe-agent, 1000 steps
Terminal-Bench 2.0 — Vals AI Terminus 2Effort: HighConfiguration: Terminus-2 harness39.3%vals.ai for Gemma 4 31B Instruct (high) on Terminal-Bench 2.0 — Vals AI Terminus 2 (opens in a new tab)Archived copy (opens in a new tab)Terminal-Bench AccuracyReported Benchmark leaderboardVals AIIndependentRecord for Gemma 4 31B Instruct on Terminal-Bench 2.0 — Vals AI Terminus 2
Terminal-Bench 2.1 — Mercor Terminus 2Configuration: Mercor Terminus 2, 1000 steps, 3 hours46.8%mercor.com for Gemma 4 31B Instruct on Terminal-Bench 2.1 — Mercor Terminus 2 (opens in a new tab)Archived copy (opens in a new tab)Terminal-Bench AccuracyReported Benchmark leaderboardMercorIndependentRecord for Gemma 4 31B Instruct on Terminal-Bench 2.1 — Mercor Terminus 2