Benchmarks

23 results across 13 benchmarks

Benchmark results for NVIDIA Nemotron 3 Ultra 550B-A55B
BenchmarkScoreSource
APEX-Agents OriginalEffort: High22.7%mercor.com for NVIDIA Nemotron 3 Ultra 550B-A55B (high) on APEX-Agents Original (opens in a new tab)Archived copy (opens in a new tab)Pass rateReported Benchmark leaderboardMercorIndependentRecord for NVIDIA Nemotron 3 Ultra 550B-A55B on APEX-Agents Original
BrowseComp44.4%huggingface.co for NVIDIA Nemotron 3 Ultra 550B-A55B on BrowseComp BrowseComp (opens in a new tab)BrowseComp AccuracyReported Model cardNVIDIASelf-reportedRecord for NVIDIA Nemotron 3 Ultra 550B-A55B on BrowseComp BrowseComp
DeepSWE 1.1 — Mercor mini-swe-agent, 500 stepsEffort: HighConfiguration: Mercor mini-swe-agent, 500 steps, 2 hours0.6%mercor.com for NVIDIA Nemotron 3 Ultra 550B-A55B (high) on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps (opens in a new tab)Archived copy (opens in a new tab)DeepSWE Pass@1Reported Benchmark leaderboardMercorIndependentRecord for NVIDIA Nemotron 3 Ultra 550B-A55B on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps
Finance Agent Benchmark 2 — October 2026 evaluation37.7%vals.ai for NVIDIA Nemotron 3 Ultra 550B-A55B on Finance Agent Benchmark 2 — October 2026 evaluation (opens in a new tab)Archived copy (opens in a new tab)Finance Agent AccuracyReported Benchmark leaderboardVals AIIndependentRecord for NVIDIA Nemotron 3 Ultra 550B-A55B on Finance Agent Benchmark 2 — October 2026 evaluation
GPQA DiamondReported as: no-toolsConfiguration: No tools87.0%huggingface.co for NVIDIA Nemotron 3 Ultra 550B-A55B (no-tools) on Graduate-Level Google-Proof Q&A Diamond (opens in a new tab)GPQA Diamond AccuracyReported Model cardNVIDIASelf-reportedRecord for NVIDIA Nemotron 3 Ultra 550B-A55B on Graduate-Level Google-Proof Q&A Diamond
GPQA Diamond — Epoch AI harnessConfiguration: Epoch AI harness85.4%epoch.ai for NVIDIA Nemotron 3 Ultra 550B-A55B on Graduate-Level Google-Proof Q&A Diamond — Epoch AI harness (opens in a new tab)Archived copy (opens in a new tab)GPQA Diamond AccuracyReported Evaluated Benchmark leaderboardEpoch AIIndependentRecord for NVIDIA Nemotron 3 Ultra 550B-A55B on Graduate-Level Google-Proof Q&A Diamond — Epoch AI harness
GPQA Diamond — Mercor single-shotEffort: HighConfiguration: Mercor single-shot agent87.8%mercor.com for NVIDIA Nemotron 3 Ultra 550B-A55B (high) on Graduate-Level Google-Proof Q&A Diamond — Mercor single-shot (opens in a new tab)Archived copy (opens in a new tab)GPQA Diamond AccuracyReported Benchmark leaderboardMercorIndependentRecord for NVIDIA Nemotron 3 Ultra 550B-A55B on Graduate-Level Google-Proof Q&A Diamond — Mercor single-shot
GPQA Diamond — Vals AI harnessConfiguration: Vals AI harness: mean of zero-shot and few-shot CoT86.1%vals.ai for NVIDIA Nemotron 3 Ultra 550B-A55B on Graduate-Level Google-Proof Q&A Diamond — Vals AI harness (opens in a new tab)Archived copy (opens in a new tab)GPQA Diamond AccuracyReported Benchmark leaderboardVals AIIndependentRecord for NVIDIA Nemotron 3 Ultra 550B-A55B on Graduate-Level Google-Proof Q&A Diamond — Vals AI harness
HLAB October 2026 evaluation0.4%vals.ai for NVIDIA Nemotron 3 Ultra 550B-A55B on Harvey's Legal Agent Benchmark October 2026 evaluation (opens in a new tab)Archived copy (opens in a new tab)HLAB Task Pass RateReported Benchmark leaderboardVals AIIndependentRecord for NVIDIA Nemotron 3 Ultra 550B-A55B on Harvey's Legal Agent Benchmark October 2026 evaluation
Humanity's Last Exam Full set — No toolsReported as: no-toolsConfiguration: No tools26.7%huggingface.co for NVIDIA Nemotron 3 Ultra 550B-A55B (no-tools) on Humanity's Last Exam Full set — No tools (opens in a new tab)Humanity's Last Exam AccuracyReported Model cardNVIDIASelf-reportedRecord for NVIDIA Nemotron 3 Ultra 550B-A55B on Humanity's Last Exam Full set — No tools
Humanity's Last Exam Text only — Mercor single-shotEffort: HighConfiguration: Mercor single-shot agent28.1%mercor.com for NVIDIA Nemotron 3 Ultra 550B-A55B (high) on Humanity's Last Exam Text only — Mercor single-shot (opens in a new tab)Archived copy (opens in a new tab)Humanity's Last Exam AccuracyReported Benchmark leaderboardMercorIndependentRecord for NVIDIA Nemotron 3 Ultra 550B-A55B on Humanity's Last Exam Text only — Mercor single-shot
MCP Atlas Public, April 2026 updateReported as: thinking63.1%labs.scale.com for NVIDIA Nemotron 3 Ultra 550B-A55B (thinking) on MCP Atlas Public, April 2026 update (opens in a new tab)Pass rateReported Benchmark leaderboardScale AIIndependentRecord for NVIDIA Nemotron 3 Ultra 550B-A55B on MCP Atlas Public, April 2026 update
MMLU-Pro Original — Mercor single-shotEffort: HighConfiguration: Mercor single-shot agent86.0%mercor.com for NVIDIA Nemotron 3 Ultra 550B-A55B (high) on MMLU-Pro Original — Mercor single-shot (opens in a new tab)Archived copy (opens in a new tab)AccuracyReported Benchmark leaderboardMercorIndependentRecord for NVIDIA Nemotron 3 Ultra 550B-A55B on MMLU-Pro Original — Mercor single-shot
MMLU-Pro Original — Vals AI harnessConfiguration: Vals AI harness85.8%vals.ai for NVIDIA Nemotron 3 Ultra 550B-A55B on MMLU-Pro Original — Vals AI harness (opens in a new tab)Archived copy (opens in a new tab)AccuracyReported Benchmark leaderboardVals AIIndependentRecord for NVIDIA Nemotron 3 Ultra 550B-A55B on MMLU-Pro Original — Vals AI harness
SWE-bench Multilingual67.7%huggingface.co for NVIDIA Nemotron 3 Ultra 550B-A55B on SWE-bench Multilingual (opens in a new tab)Resolved TasksReported Model cardNVIDIASelf-reportedRecord for NVIDIA Nemotron 3 Ultra 550B-A55B on SWE-bench Multilingual
SWE-bench Verified70.7%huggingface.co for NVIDIA Nemotron 3 Ultra 550B-A55B on SWE-bench Verified (opens in a new tab)Resolved TasksReported Model cardNVIDIASelf-reportedRecord for NVIDIA Nemotron 3 Ultra 550B-A55B on SWE-bench Verified
SWE-bench Verified — Vals AI mini-swe-agentConfiguration: mini-swe-agent harness69.0%vals.ai for NVIDIA Nemotron 3 Ultra 550B-A55B on SWE-bench Verified — Vals AI mini-swe-agent (opens in a new tab)Archived copy (opens in a new tab)Resolved TasksReported Benchmark leaderboardVals AIIndependentRecord for NVIDIA Nemotron 3 Ultra 550B-A55B on SWE-bench Verified — Vals AI mini-swe-agent
Terminal-Bench 2.156.4%huggingface.co for NVIDIA Nemotron 3 Ultra 550B-A55B on Terminal-Bench 2.1 (opens in a new tab)Terminal-Bench AccuracyReported Model cardNVIDIASelf-reportedRecord for NVIDIA Nemotron 3 Ultra 550B-A55B on Terminal-Bench 2.1
Terminal-Bench 2.1 — Mercor Terminus 2Effort: HighConfiguration: Mercor Terminus 2, 1000 steps, 3 hours49.8%mercor.com for NVIDIA Nemotron 3 Ultra 550B-A55B (high) on Terminal-Bench 2.1 — Mercor Terminus 2 (opens in a new tab)Archived copy (opens in a new tab)Terminal-Bench AccuracyReported Benchmark leaderboardMercorIndependentRecord for NVIDIA Nemotron 3 Ultra 550B-A55B on Terminal-Bench 2.1 — Mercor Terminus 2
Terminal-Bench 2.1 — Vals AI Terminus 2Configuration: Terminus-2 harness50.9%vals.ai for NVIDIA Nemotron 3 Ultra 550B-A55B on Terminal-Bench 2.1 — Vals AI Terminus 2 (opens in a new tab)Archived copy (opens in a new tab)Terminal-Bench AccuracyReported Benchmark leaderboardVals AIIndependentRecord for NVIDIA Nemotron 3 Ultra 550B-A55B on Terminal-Bench 2.1 — Vals AI Terminus 2
Terminal-Bench 4.0 — Mercor mini-swe-agentEffort: HighConfiguration: Mercor mini-swe-agent, 1000 steps, 8 hours0.0%mercor.com for NVIDIA Nemotron 3 Ultra 550B-A55B (high) on Terminal-Bench 4.0 — Mercor mini-swe-agent (opens in a new tab)Archived copy (opens in a new tab)Terminal-Bench AccuracyReported Benchmark leaderboardMercorIndependentRecord for NVIDIA Nemotron 3 Ultra 550B-A55B on Terminal-Bench 4.0 — Mercor mini-swe-agent
Toolathlon Verified34.3%toolathlon.xyz for NVIDIA Nemotron 3 Ultra 550B-A55B on Toolathlon Verified (opens in a new tab)Archived copy (opens in a new tab)Tasks completedReported Benchmark leaderboardToolathlonIndependentToolathlon AuthorsRecord for NVIDIA Nemotron 3 Ultra 550B-A55B on Toolathlon Verified
Vibe Code Bench 1.1 — OpenHands, October 2026 evaluationConfiguration: OpenHands harness7.6%vals.ai for NVIDIA Nemotron 3 Ultra 550B-A55B on Vibe Code Bench 1.1 — OpenHands, October 2026 evaluation (opens in a new tab)Archived copy (opens in a new tab)Vibe Code Bench ScoreReported Benchmark leaderboardVals AIIndependentRecord for NVIDIA Nemotron 3 Ultra 550B-A55B on Vibe Code Bench 1.1 — OpenHands, October 2026 evaluation