AutomationBench 1.0.6 Effort: Max 1.7% zapier.com for Gemma 4 31B Instruct (max) on AutomationBench 1.0.6 (opens in a new tab) Archived copy (opens in a new tab) Task pass rate Reported October 7, 2026 Benchmark leaderboard Zapier Independent Zapier AutomationBench Team Record for Gemma 4 31B Instruct on AutomationBench 1.0.6 BIG-Bench Extra Hard Original — micro average Reported as: thinking mode 74.4% huggingface.co for Gemma 4 31B Instruct (thinking mode) on BIG-Bench Extra Hard Original — micro average (opens in a new tab) Micro-average accuracy Reported July 24, 2026 Model card Google Self-reported Google DeepMind Record for Gemma 4 31B Instruct on BIG-Bench Extra Hard Original — micro average DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps Configuration: Mercor mini-swe-agent, 500 steps, 2 hours 0.0% mercor.com for Gemma 4 31B Instruct on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps (opens in a new tab) Archived copy (opens in a new tab) DeepSWE Pass@1 Reported October 7, 2026 Benchmark leaderboard Mercor Independent Record for Gemma 4 31B Instruct on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps GPQA Diamond Reported as: thinking mode 84.3% huggingface.co for Gemma 4 31B Instruct (thinking mode) on Graduate-Level Google-Proof Q&A Diamond (opens in a new tab) GPQA Diamond Accuracy Reported July 24, 2026 Model card Google Self-reported Google DeepMind Record for Gemma 4 31B Instruct on Graduate-Level Google-Proof Q&A Diamond GPQA Diamond — Mercor single-shot Configuration: Mercor single-shot agent 85.5% mercor.com for Gemma 4 31B Instruct on Graduate-Level Google-Proof Q&A Diamond — Mercor single-shot (opens in a new tab) Archived copy (opens in a new tab) GPQA Diamond Accuracy Reported October 7, 2026 Benchmark leaderboard Mercor Independent Record for Gemma 4 31B Instruct on Graduate-Level Google-Proof Q&A Diamond — Mercor single-shot Humanity's Last Exam Text only — Mercor single-shot Configuration: Mercor single-shot agent 23.2% mercor.com for Gemma 4 31B Instruct on Humanity's Last Exam Text only — Mercor single-shot (opens in a new tab) Archived copy (opens in a new tab) Humanity's Last Exam Accuracy Reported October 7, 2026 Benchmark leaderboard Mercor Independent Record for Gemma 4 31B Instruct on Humanity's Last Exam Text only — Mercor single-shot IFBench Original Reported as: thinking mode 76.0% arxiv.org for Gemma 4 31B Instruct (thinking mode) on IFBench Original (opens in a new tab) Accuracy Reported July 24, 2026 Technical report Google DeepMind Self-reported Record for Gemma 4 31B Instruct on IFBench Original IFEval Original Reported as: thinking mode 98.9% huggingface.co for Gemma 4 31B Instruct (thinking mode) on IFEval Original (opens in a new tab) Accuracy Reported July 24, 2026 Model card Google Self-reported Google DeepMind Record for Gemma 4 31B Instruct on IFEval Original MATH-Vision Original Reported as: thinking mode 85.6% huggingface.co for Gemma 4 31B Instruct (thinking mode) on MATH-Vision Original (opens in a new tab) Accuracy Reported July 24, 2026 Model card Google Self-reported Google DeepMind Record for Gemma 4 31B Instruct on MATH-Vision Original MedXpertQA MM Reported as: thinking mode 61.3% huggingface.co for Gemma 4 31B Instruct (thinking mode) on MedXpertQA MM (opens in a new tab) Accuracy Reported July 24, 2026 Model card Google Self-reported Google DeepMind Record for Gemma 4 31B Instruct on MedXpertQA MM MMLU-Pro Original Reported as: thinking mode 85.2% huggingface.co for Gemma 4 31B Instruct (thinking mode) on MMLU-Pro Original (opens in a new tab) Accuracy Reported July 24, 2026 Model card Google Self-reported Google DeepMind Record for Gemma 4 31B Instruct on MMLU-Pro Original MMLU-Pro Original — Mercor single-shot Configuration: Mercor single-shot agent 85.8% mercor.com for Gemma 4 31B Instruct on MMLU-Pro Original — Mercor single-shot (opens in a new tab) Archived copy (opens in a new tab) Accuracy Reported October 7, 2026 Benchmark leaderboard Mercor Independent Record for Gemma 4 31B Instruct on MMLU-Pro Original — Mercor single-shot SciCode Original Reported as: thinking mode 43.0% arxiv.org for Gemma 4 31B Instruct (thinking mode) on SciCode Original (opens in a new tab) Accuracy Reported July 24, 2026 Technical report Google DeepMind Self-reported Record for Gemma 4 31B Instruct on SciCode Original SimpleQA Verified Original — Epoch AI harness Configuration: Epoch AI harness 10.4% epoch.ai for Gemma 4 31B Instruct on SimpleQA Verified Original — Epoch AI harness (opens in a new tab) Archived copy (opens in a new tab) Accuracy Reported October 7, 2026 Evaluated August 27, 2026 at 7:30 PM UTC Benchmark leaderboard Epoch AI Independent Record for Gemma 4 31B Instruct on SimpleQA Verified Original — Epoch AI harness SWE-bench Verified — Mercor mini-swe-agent, 1000 steps Configuration: Mercor mini-swe-agent, 1000 steps, 3 hours 46.1% mercor.com for Gemma 4 31B Instruct on SWE-bench Verified — Mercor mini-swe-agent, 1000 steps (opens in a new tab) Archived copy (opens in a new tab) Resolved Tasks Reported October 7, 2026 Benchmark leaderboard Mercor Independent Record for Gemma 4 31B Instruct on SWE-bench Verified — Mercor mini-swe-agent, 1000 steps Terminal-Bench 2.0 — Vals AI Terminus 2 Effort: High Configuration: Terminus-2 harness 39.3% vals.ai for Gemma 4 31B Instruct (high) on Terminal-Bench 2.0 — Vals AI Terminus 2 (opens in a new tab) Archived copy (opens in a new tab) Terminal-Bench Accuracy Reported June 4, 2026 Benchmark leaderboard Vals AI Independent Record for Gemma 4 31B Instruct on Terminal-Bench 2.0 — Vals AI Terminus 2 Terminal-Bench 2.1 — Mercor Terminus 2 Configuration: Mercor Terminus 2, 1000 steps, 3 hours 46.8% mercor.com for Gemma 4 31B Instruct on Terminal-Bench 2.1 — Mercor Terminus 2 (opens in a new tab) Archived copy (opens in a new tab) Terminal-Bench Accuracy Reported October 7, 2026 Benchmark leaderboard Mercor Independent Record for Gemma 4 31B Instruct on Terminal-Bench 2.1 — Mercor Terminus 2