APEX-Agents Original Effort: High 22.7% mercor.com for NVIDIA Nemotron 3 Ultra 550B-A55B (high) on APEX-Agents Original (opens in a new tab) Archived copy (opens in a new tab) Pass rate Reported October 7, 2026 Benchmark leaderboard Mercor Independent Record for NVIDIA Nemotron 3 Ultra 550B-A55B on APEX-Agents Original BrowseComp 44.4% huggingface.co for NVIDIA Nemotron 3 Ultra 550B-A55B on BrowseComp BrowseComp (opens in a new tab) BrowseComp Accuracy Reported June 4, 2026 Model card NVIDIA Self-reported Record for NVIDIA Nemotron 3 Ultra 550B-A55B on BrowseComp BrowseComp DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps Effort: High Configuration: Mercor mini-swe-agent, 500 steps, 2 hours 0.6% mercor.com for NVIDIA Nemotron 3 Ultra 550B-A55B (high) on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps (opens in a new tab) Archived copy (opens in a new tab) DeepSWE Pass@1 Reported October 7, 2026 Benchmark leaderboard Mercor Independent Record for NVIDIA Nemotron 3 Ultra 550B-A55B on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps Finance Agent Benchmark 2 — October 2026 evaluation 37.7% vals.ai for NVIDIA Nemotron 3 Ultra 550B-A55B on Finance Agent Benchmark 2 — October 2026 evaluation (opens in a new tab) Archived copy (opens in a new tab) Finance Agent Accuracy Reported October 6, 2026 Benchmark leaderboard Vals AI Independent Record for NVIDIA Nemotron 3 Ultra 550B-A55B on Finance Agent Benchmark 2 — October 2026 evaluation GPQA Diamond Reported as: no-tools Configuration: No tools 87.0% huggingface.co for NVIDIA Nemotron 3 Ultra 550B-A55B (no-tools) on Graduate-Level Google-Proof Q&A Diamond (opens in a new tab) GPQA Diamond Accuracy Reported June 4, 2026 Model card NVIDIA Self-reported Record for NVIDIA Nemotron 3 Ultra 550B-A55B on Graduate-Level Google-Proof Q&A Diamond GPQA Diamond — Epoch AI harness Configuration: Epoch AI harness 85.4% epoch.ai for NVIDIA Nemotron 3 Ultra 550B-A55B on Graduate-Level Google-Proof Q&A Diamond — Epoch AI harness (opens in a new tab) Archived copy (opens in a new tab) GPQA Diamond Accuracy Reported October 7, 2026 Evaluated August 10, 2026 at 5:46 PM UTC Benchmark leaderboard Epoch AI Independent Record for NVIDIA Nemotron 3 Ultra 550B-A55B on Graduate-Level Google-Proof Q&A Diamond — Epoch AI harness GPQA Diamond — Mercor single-shot Effort: High Configuration: Mercor single-shot agent 87.8% mercor.com for NVIDIA Nemotron 3 Ultra 550B-A55B (high) on Graduate-Level Google-Proof Q&A Diamond — Mercor single-shot (opens in a new tab) Archived copy (opens in a new tab) GPQA Diamond Accuracy Reported October 7, 2026 Benchmark leaderboard Mercor Independent Record for NVIDIA Nemotron 3 Ultra 550B-A55B on Graduate-Level Google-Proof Q&A Diamond — Mercor single-shot GPQA Diamond — Vals AI harness Configuration: Vals AI harness: mean of zero-shot and few-shot CoT 86.1% vals.ai for NVIDIA Nemotron 3 Ultra 550B-A55B on Graduate-Level Google-Proof Q&A Diamond — Vals AI harness (opens in a new tab) Archived copy (opens in a new tab) GPQA Diamond Accuracy Reported September 1, 2026 Benchmark leaderboard Vals AI Independent Record for NVIDIA Nemotron 3 Ultra 550B-A55B on Graduate-Level Google-Proof Q&A Diamond — Vals AI harness HLAB October 2026 evaluation 0.4% vals.ai for NVIDIA Nemotron 3 Ultra 550B-A55B on Harvey's Legal Agent Benchmark October 2026 evaluation (opens in a new tab) Archived copy (opens in a new tab) HLAB Task Pass Rate Reported October 6, 2026 Benchmark leaderboard Vals AI Independent Record for NVIDIA Nemotron 3 Ultra 550B-A55B on Harvey's Legal Agent Benchmark October 2026 evaluation Humanity's Last Exam Full set — No tools Reported as: no-tools Configuration: No tools 26.7% huggingface.co for NVIDIA Nemotron 3 Ultra 550B-A55B (no-tools) on Humanity's Last Exam Full set — No tools (opens in a new tab) Humanity's Last Exam Accuracy Reported June 4, 2026 Model card NVIDIA Self-reported Record for NVIDIA Nemotron 3 Ultra 550B-A55B on Humanity's Last Exam Full set — No tools Humanity's Last Exam Text only — Mercor single-shot Effort: High Configuration: Mercor single-shot agent 28.1% mercor.com for NVIDIA Nemotron 3 Ultra 550B-A55B (high) on Humanity's Last Exam Text only — Mercor single-shot (opens in a new tab) Archived copy (opens in a new tab) Humanity's Last Exam Accuracy Reported October 7, 2026 Benchmark leaderboard Mercor Independent Record for NVIDIA Nemotron 3 Ultra 550B-A55B on Humanity's Last Exam Text only — Mercor single-shot MCP Atlas Public, April 2026 update Reported as: thinking 63.1% labs.scale.com for NVIDIA Nemotron 3 Ultra 550B-A55B (thinking) on MCP Atlas Public, April 2026 update (opens in a new tab) Pass rate Reported September 24, 2026 Benchmark leaderboard Scale AI Independent Record for NVIDIA Nemotron 3 Ultra 550B-A55B on MCP Atlas Public, April 2026 update MMLU-Pro Original — Mercor single-shot Effort: High Configuration: Mercor single-shot agent 86.0% mercor.com for NVIDIA Nemotron 3 Ultra 550B-A55B (high) on MMLU-Pro Original — Mercor single-shot (opens in a new tab) Archived copy (opens in a new tab) Accuracy Reported October 7, 2026 Benchmark leaderboard Mercor Independent Record for NVIDIA Nemotron 3 Ultra 550B-A55B on MMLU-Pro Original — Mercor single-shot MMLU-Pro Original — Vals AI harness Configuration: Vals AI harness 85.8% vals.ai for NVIDIA Nemotron 3 Ultra 550B-A55B on MMLU-Pro Original — Vals AI harness (opens in a new tab) Archived copy (opens in a new tab) Accuracy Reported September 1, 2026 Benchmark leaderboard Vals AI Independent Record for NVIDIA Nemotron 3 Ultra 550B-A55B on MMLU-Pro Original — Vals AI harness SWE-bench Multilingual 67.7% huggingface.co for NVIDIA Nemotron 3 Ultra 550B-A55B on SWE-bench Multilingual (opens in a new tab) Resolved Tasks Reported June 4, 2026 Model card NVIDIA Self-reported Record for NVIDIA Nemotron 3 Ultra 550B-A55B on SWE-bench Multilingual SWE-bench Verified 70.7% huggingface.co for NVIDIA Nemotron 3 Ultra 550B-A55B on SWE-bench Verified (opens in a new tab) Resolved Tasks Reported June 4, 2026 Model card NVIDIA Self-reported Record for NVIDIA Nemotron 3 Ultra 550B-A55B on SWE-bench Verified SWE-bench Verified — Vals AI mini-swe-agent Configuration: mini-swe-agent harness 69.0% vals.ai for NVIDIA Nemotron 3 Ultra 550B-A55B on SWE-bench Verified — Vals AI mini-swe-agent (opens in a new tab) Archived copy (opens in a new tab) Resolved Tasks Reported September 1, 2026 Benchmark leaderboard Vals AI Independent Record for NVIDIA Nemotron 3 Ultra 550B-A55B on SWE-bench Verified — Vals AI mini-swe-agent Terminal-Bench 2.1 56.4% huggingface.co for NVIDIA Nemotron 3 Ultra 550B-A55B on Terminal-Bench 2.1 (opens in a new tab) Terminal-Bench Accuracy Reported June 4, 2026 Model card NVIDIA Self-reported Record for NVIDIA Nemotron 3 Ultra 550B-A55B on Terminal-Bench 2.1 Terminal-Bench 2.1 — Mercor Terminus 2 Effort: High Configuration: Mercor Terminus 2, 1000 steps, 3 hours 49.8% mercor.com for NVIDIA Nemotron 3 Ultra 550B-A55B (high) on Terminal-Bench 2.1 — Mercor Terminus 2 (opens in a new tab) Archived copy (opens in a new tab) Terminal-Bench Accuracy Reported October 7, 2026 Benchmark leaderboard Mercor Independent Record for NVIDIA Nemotron 3 Ultra 550B-A55B on Terminal-Bench 2.1 — Mercor Terminus 2 Terminal-Bench 2.1 — Vals AI Terminus 2 Configuration: Terminus-2 harness 50.9% vals.ai for NVIDIA Nemotron 3 Ultra 550B-A55B on Terminal-Bench 2.1 — Vals AI Terminus 2 (opens in a new tab) Archived copy (opens in a new tab) Terminal-Bench Accuracy Reported September 28, 2026 Benchmark leaderboard Vals AI Independent Record for NVIDIA Nemotron 3 Ultra 550B-A55B on Terminal-Bench 2.1 — Vals AI Terminus 2 Terminal-Bench 4.0 — Mercor mini-swe-agent Effort: High Configuration: Mercor mini-swe-agent, 1000 steps, 8 hours 0.0% mercor.com for NVIDIA Nemotron 3 Ultra 550B-A55B (high) on Terminal-Bench 4.0 — Mercor mini-swe-agent (opens in a new tab) Archived copy (opens in a new tab) Terminal-Bench Accuracy Reported October 7, 2026 Benchmark leaderboard Mercor Independent Record for NVIDIA Nemotron 3 Ultra 550B-A55B on Terminal-Bench 4.0 — Mercor mini-swe-agent Toolathlon Verified 34.3% toolathlon.xyz for NVIDIA Nemotron 3 Ultra 550B-A55B on Toolathlon Verified (opens in a new tab) Archived copy (opens in a new tab) Tasks completed Reported July 15, 2026 Benchmark leaderboard Toolathlon Independent Toolathlon Authors Record for NVIDIA Nemotron 3 Ultra 550B-A55B on Toolathlon Verified Vibe Code Bench 1.1 — OpenHands, October 2026 evaluation Configuration: OpenHands harness 7.6% vals.ai for NVIDIA Nemotron 3 Ultra 550B-A55B on Vibe Code Bench 1.1 — OpenHands, October 2026 evaluation (opens in a new tab) Archived copy (opens in a new tab) Vibe Code Bench Score Reported October 6, 2026 Benchmark leaderboard Vals AI Independent Record for NVIDIA Nemotron 3 Ultra 550B-A55B on Vibe Code Bench 1.1 — OpenHands, October 2026 evaluation