Benchmarks

35 results across 25 benchmarks

Benchmark results for Muse Spark 1.1
BenchmarkScoreSource
APEX-Agents OriginalEffort: xHigh31.8%mercor.com for Muse Spark 1.1 (xhigh) on APEX-Agents Original (opens in a new tab)Archived copy (opens in a new tab)Pass rateReported Benchmark leaderboardMercorIndependentRecord for Muse Spark 1.1 on APEX-Agents Original
BabyVision OriginalEffort: xHigh76.3%dev.meta.ai for Muse Spark 1.1 (xhigh) on BabyVision Original (opens in a new tab)AccuracyReported Model pageMetaSelf-reportedRecord for Muse Spark 1.1 on BabyVision Original
Chartography October 2026 leaderboardEffort: xHigh24.4%surgehq.ai for Muse Spark 1.1 (xHigh) on Chartography October 2026 leaderboard (opens in a new tab)Archived copy (opens in a new tab)Chartography Pass@1Reported Benchmark leaderboardSurge AIIndependentRecord for Muse Spark 1.1 on Chartography October 2026 leaderboard
CharXiv Descriptive and reasoning — Mercor single-shotEffort: xHighConfiguration: Mercor single-shot agent95.2%mercor.com for Muse Spark 1.1 (xhigh) on CharXiv Descriptive and reasoning — Mercor single-shot (opens in a new tab)Archived copy (opens in a new tab)CharXiv AccuracyReported Benchmark leaderboardMercorIndependentRecord for Muse Spark 1.1 on CharXiv Descriptive and reasoning — Mercor single-shot
CharXiv Reasoning — No toolsEffort: xHighConfiguration: No tools88.4%dev.meta.ai for Muse Spark 1.1 (xhigh) on CharXiv Reasoning — No tools (opens in a new tab)CharXiv AccuracyReported Model pageMetaSelf-reportedRecord for Muse Spark 1.1 on CharXiv Reasoning — No tools
CyberGym OriginalEffort: High59.0%ai.meta.com for Muse Spark 1.1 (high) on CyberGym Original (opens in a new tab)Vulnerability reproduction successReported Evaluation reportMetaSelf-reportedRecord for Muse Spark 1.1 on CyberGym Original
DeepSWE 1.1 — Mercor mini-swe-agent, 500 stepsEffort: xHighConfiguration: Mercor mini-swe-agent, 500 steps, 2 hours40.4%mercor.com for Muse Spark 1.1 (xhigh) on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps (opens in a new tab)Archived copy (opens in a new tab)DeepSWE Pass@1Reported Benchmark leaderboardMercorIndependentRecord for Muse Spark 1.1 on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps
DeepSWE 1.1 — mini-swe-agentEffort: xHighConfiguration: mini-swe-agent harness53.3%dev.meta.ai for Muse Spark 1.1 (xhigh) on DeepSWE 1.1 — mini-swe-agent (opens in a new tab)DeepSWE Pass@1Reported Model pageMetaSelf-reportedRecord for Muse Spark 1.1 on DeepSWE 1.1 — mini-swe-agent
Finance Agent Benchmark 2 — October 2026 evaluationEffort: xHigh57.2%vals.ai for Muse Spark 1.1 (xhigh) on Finance Agent Benchmark 2 — October 2026 evaluation (opens in a new tab)Archived copy (opens in a new tab)Finance Agent AccuracyReported Benchmark leaderboardVals AIIndependentRecord for Muse Spark 1.1 on Finance Agent Benchmark 2 — October 2026 evaluation
GPQA Diamond — Mercor single-shotEffort: xHighConfiguration: Mercor single-shot agent90.8%mercor.com for Muse Spark 1.1 (xhigh) on Graduate-Level Google-Proof Q&A Diamond — Mercor single-shot (opens in a new tab)Archived copy (opens in a new tab)GPQA Diamond AccuracyReported Benchmark leaderboardMercorIndependentRecord for Muse Spark 1.1 on Graduate-Level Google-Proof Q&A Diamond — Mercor single-shot
GPQA Diamond — Vals AI harnessEffort: xHighConfiguration: Vals AI harness: mean of zero-shot and few-shot CoT91.2%vals.ai for Muse Spark 1.1 (xhigh) on Graduate-Level Google-Proof Q&A Diamond — Vals AI harness (opens in a new tab)Archived copy (opens in a new tab)GPQA Diamond AccuracyReported Benchmark leaderboardVals AIIndependentRecord for Muse Spark 1.1 on Graduate-Level Google-Proof Q&A Diamond — Vals AI harness
HLAB October 2026 evaluationEffort: xHigh20.0%vals.ai for Muse Spark 1.1 (xhigh) on Harvey's Legal Agent Benchmark October 2026 evaluation (opens in a new tab)Archived copy (opens in a new tab)HLAB Task Pass RateReported Benchmark leaderboardVals AIIndependentRecord for Muse Spark 1.1 on Harvey's Legal Agent Benchmark October 2026 evaluation
Humanity's Last Exam Full set — With toolsEffort: xHighConfiguration: With tools62.1%dev.meta.ai for Muse Spark 1.1 (xhigh) on Humanity's Last Exam Full set — With tools (opens in a new tab)Humanity's Last Exam AccuracyReported Model pageMetaSelf-reportedRecord for Muse Spark 1.1 on Humanity's Last Exam Full set — With tools
JobBench Main 65 tasksEffort: xHigh54.7%dev.meta.ai for Muse Spark 1.1 (xhigh) on JobBench Main 65 tasks (opens in a new tab)Mean rubric scoreReported Model pageMetaSelf-reportedRecord for Muse Spark 1.1 on JobBench Main 65 tasks
MCP Atlas Public, April 2026 update88.1%labs.scale.com for Muse Spark 1.1 on MCP Atlas Public, April 2026 update (opens in a new tab)Pass rateReported Benchmark leaderboardScale AIIndependentRecord for Muse Spark 1.1 on MCP Atlas Public, April 2026 update
MedXpertQA MM — Mercor single-shotEffort: xHighConfiguration: Mercor single-shot agent77.0%mercor.com for Muse Spark 1.1 (xhigh) on MedXpertQA MM — Mercor single-shot (opens in a new tab)Archived copy (opens in a new tab)AccuracyReported Benchmark leaderboardMercorIndependentRecord for Muse Spark 1.1 on MedXpertQA MM — Mercor single-shot
MMLU-Pro Original — Mercor single-shotEffort: xHighConfiguration: Mercor single-shot agent89.9%mercor.com for Muse Spark 1.1 (xhigh) on MMLU-Pro Original — Mercor single-shot (opens in a new tab)Archived copy (opens in a new tab)AccuracyReported Benchmark leaderboardMercorIndependentRecord for Muse Spark 1.1 on MMLU-Pro Original — Mercor single-shot
MMLU-Pro Original — Vals AI harnessEffort: xHighConfiguration: Vals AI harness88.7%vals.ai for Muse Spark 1.1 (xhigh) on MMLU-Pro Original — Vals AI harness (opens in a new tab)Archived copy (opens in a new tab)AccuracyReported Benchmark leaderboardVals AIIndependentRecord for Muse Spark 1.1 on MMLU-Pro Original — Vals AI harness
MMMU-Pro 1,730 questions — Mercor single-shotEffort: xHighConfiguration: Mercor single-shot agent81.3%mercor.com for Muse Spark 1.1 (xhigh) on MMMU-Pro 1,730 questions — Mercor single-shot (opens in a new tab)Archived copy (opens in a new tab)MMMU-Pro accuracySetting: Not statedReported Benchmark leaderboardMercorIndependentRecord for Muse Spark 1.1 on MMMU-Pro 1,730 questions — Mercor single-shot
MMMU-Pro Vals AI harnessEffort: xHighConfiguration: Vals AI harness86.6%vals.ai for Muse Spark 1.1 (xhigh) on MMMU-Pro Vals AI harness (opens in a new tab)Archived copy (opens in a new tab)MMMU-Pro accuracySetting: Standard (4 options)Reported Benchmark leaderboardVals AIIndependentRecord for Muse Spark 1.1 on MMMU-Pro Vals AI harness
OSWorld VerifiedEffort: xHigh80.8%dev.meta.ai for Muse Spark 1.1 (xhigh) on OSWorld Verified (opens in a new tab)OSWorld-Verified success rateReported Model pageMetaSelf-reportedRecord for Muse Spark 1.1 on OSWorld Verified
OSWorld Verified — 100 stepsConfiguration: 100-step limit80.7%os-world.github.io for Muse Spark 1.1 on OSWorld Verified — 100 steps (opens in a new tab)Archived copy (opens in a new tab)OSWorld-Verified success rateReported Benchmark leaderboardOSWorldIndependentXLang LabRecord for Muse Spark 1.1 on OSWorld Verified — 100 steps
Riemann-bench October 2026 leaderboardEffort: xHigh20.8%surgehq.ai for Muse Spark 1.1 (xHigh) on Riemann-bench October 2026 leaderboard (opens in a new tab)Archived copy (opens in a new tab)Riemann-bench Pass@1Reported Benchmark leaderboardSurge AIIndependentRecord for Muse Spark 1.1 on Riemann-bench October 2026 leaderboard
SimpleQA Verified Original — Epoch AI harnessConfiguration: Epoch AI harness57.8%epoch.ai for Muse Spark 1.1 on SimpleQA Verified Original — Epoch AI harness (opens in a new tab)Archived copy (opens in a new tab)AccuracyReported Evaluated Benchmark leaderboardEpoch AIIndependentRecord for Muse Spark 1.1 on SimpleQA Verified Original — Epoch AI harness
SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 stepsEffort: xHighConfiguration: Mercor mini-swe-agent, 250 steps, 1 hour3.9%mercor.com for Muse Spark 1.1 (xhigh) on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps (opens in a new tab)Archived copy (opens in a new tab)Mean pass@1Reported Benchmark leaderboardMercorIndependentRecord for Muse Spark 1.1 on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps
SWE Atlas Codebase QnA — Public; mini-swe-agentEffort: xHighConfiguration: mini-swe-agent harness42.2%labs.scale.com for Muse Spark 1.1 (xHigh) on SWE Atlas Codebase QnA — Public; mini-swe-agent (opens in a new tab)Archived copy (opens in a new tab)Mean pass@1Reported Benchmark leaderboardScale AIIndependentRecord for Muse Spark 1.1 on SWE Atlas Codebase QnA — Public; mini-swe-agent
SWE-bench Verified — Vals AI mini-swe-agentEffort: xHighConfiguration: mini-swe-agent harness82.0%vals.ai for Muse Spark 1.1 (xhigh) on SWE-bench Verified — Vals AI mini-swe-agent (opens in a new tab)Archived copy (opens in a new tab)Resolved TasksReported Benchmark leaderboardVals AIIndependentRecord for Muse Spark 1.1 on SWE-bench Verified — Vals AI mini-swe-agent
SWE-bench Pro Private — Scale AI mini-swe-agentConfiguration: mini-swe-agent harness51.5%labs.scale.com for Muse Spark 1.1 on SWE-bench Pro Private — Scale AI mini-swe-agent (opens in a new tab)Archived copy (opens in a new tab)Public tasks resolvedReported Benchmark leaderboardScale AIIndependentRecord for Muse Spark 1.1 on SWE-bench Pro Private — Scale AI mini-swe-agent
SWE-bench Pro PublicEffort: xHigh61.5%dev.meta.ai for Muse Spark 1.1 (xhigh) on SWE-bench Pro Public (opens in a new tab)Public tasks resolvedReported Model pageMetaSelf-reportedRecord for Muse Spark 1.1 on SWE-bench Pro Public
SWE-bench Pro Public — Scale AI mini-swe-agentConfiguration: mini-swe-agent harness61.5%labs.scale.com for Muse Spark 1.1 on SWE-bench Pro Public — Scale AI mini-swe-agent (opens in a new tab)Archived copy (opens in a new tab)Public tasks resolvedReported Benchmark leaderboardScale AIIndependentRecord for Muse Spark 1.1 on SWE-bench Pro Public — Scale AI mini-swe-agent
Terminal-Bench 2.1 — mini-swe-agentEffort: xHighConfiguration: mini-swe-agent harness80.0%dev.meta.ai for Muse Spark 1.1 (xhigh) on Terminal-Bench 2.1 — mini-swe-agent (opens in a new tab)Terminal-Bench AccuracyReported Model pageMetaSelf-reportedRecord for Muse Spark 1.1 on Terminal-Bench 2.1 — mini-swe-agent
Terminal-Bench 2.1 — Vals AI Terminus 2Effort: xHighConfiguration: Terminus-2 harness69.3%vals.ai for Muse Spark 1.1 (xhigh) on Terminal-Bench 2.1 — Vals AI Terminus 2 (opens in a new tab)Archived copy (opens in a new tab)Terminal-Bench AccuracyReported Benchmark leaderboardVals AIIndependentRecord for Muse Spark 1.1 on Terminal-Bench 2.1 — Vals AI Terminus 2
Toolathlon VerifiedEffort: xHigh75.6%dev.meta.ai for Muse Spark 1.1 (xhigh) on Toolathlon Verified (opens in a new tab)Tasks completedReported Model pageMetaSelf-reportedRecord for Muse Spark 1.1 on Toolathlon Verified
Vibe Code Bench 1.1 — OpenHands, October 2026 evaluationEffort: xHighConfiguration: OpenHands harness72.2%vals.ai for Muse Spark 1.1 (xhigh) on Vibe Code Bench 1.1 — OpenHands, October 2026 evaluation (opens in a new tab)Archived copy (opens in a new tab)Vibe Code Bench ScoreReported Benchmark leaderboardVals AIIndependentRecord for Muse Spark 1.1 on Vibe Code Bench 1.1 — OpenHands, October 2026 evaluation
τ³-bench τ³ — Banking, AllTools retrieval, v1.0.1 gradingEffort: xHighConfiguration: AllTools retrieval40.5%sierra-tau-bench-public.s3.us-west-2.amazonaws.com for Muse Spark 1.1 (xhigh) on τ³-bench τ³ — Banking, AllTools retrieval, v1.0.1 grading (opens in a new tab)Archived copy (opens in a new tab)Task success rateReported Evaluated Benchmark leaderboardSierraIndependentSierra ResearchRecord for Muse Spark 1.1 on τ³-bench τ³ — Banking, AllTools retrieval, v1.0.1 grading