Benchmarks

24 results across 13 benchmarks

Benchmark results for Qwen3.5-397B-A17B
BenchmarkScoreSource
APEX-Agents Original24.9%mercor.com for Qwen3.5-397B-A17B on APEX-Agents Original (opens in a new tab)Archived copy (opens in a new tab)Pass rateReported Benchmark leaderboardMercorIndependentRecord for Qwen3.5-397B-A17B on APEX-Agents Original
CharXiv Descriptive and reasoning — Mercor single-shotConfiguration: Mercor single-shot agent91.6%mercor.com for Qwen3.5-397B-A17B on CharXiv Descriptive and reasoning — Mercor single-shot (opens in a new tab)Archived copy (opens in a new tab)CharXiv AccuracyReported Benchmark leaderboardMercorIndependentRecord for Qwen3.5-397B-A17B on CharXiv Descriptive and reasoning — Mercor single-shot
DeepSWE 1.1 — Mercor mini-swe-agent, 500 stepsConfiguration: Mercor mini-swe-agent, 500 steps, 2 hours0.0%mercor.com for Qwen3.5-397B-A17B on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps (opens in a new tab)Archived copy (opens in a new tab)DeepSWE Pass@1Reported Benchmark leaderboardMercorIndependentRecord for Qwen3.5-397B-A17B on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps
GPQA Diamond88.4%alibabacloud.com for Qwen3.5-397B-A17B on Graduate-Level Google-Proof Q&A Diamond (opens in a new tab)GPQA Diamond AccuracyReported Launch postAlibaba Cloud CommunitySelf-reportedQwen TeamRecord for Qwen3.5-397B-A17B on Graduate-Level Google-Proof Q&A Diamond
GPQA Diamond — Epoch AI harnessEffort: NoneConfiguration: Epoch AI harness86.4%epoch.ai for Qwen3.5-397B-A17B (none) on Graduate-Level Google-Proof Q&A Diamond — Epoch AI harness (opens in a new tab)Archived copy (opens in a new tab)GPQA Diamond AccuracyReported Evaluated Benchmark leaderboardEpoch AIIndependentRecord for Qwen3.5-397B-A17B on Graduate-Level Google-Proof Q&A Diamond — Epoch AI harness
GPQA Diamond — Epoch AI harnessConfiguration: Epoch AI harness85.9%epoch.ai for Qwen3.5-397B-A17B on Graduate-Level Google-Proof Q&A Diamond — Epoch AI harness (opens in a new tab)Archived copy (opens in a new tab)GPQA Diamond AccuracyReported Evaluated Benchmark leaderboardEpoch AIIndependentRecord for Qwen3.5-397B-A17B on Graduate-Level Google-Proof Q&A Diamond — Epoch AI harness
Humanity's Last Exam Full set — No toolsConfiguration: No tools28.7%alibabacloud.com for Qwen3.5-397B-A17B on Humanity's Last Exam Full set — No tools (opens in a new tab)Humanity's Last Exam AccuracyReported Launch postAlibaba Cloud CommunitySelf-reportedQwen TeamRecord for Qwen3.5-397B-A17B on Humanity's Last Exam Full set — No tools
Humanity's Last Exam Full set — With toolsConfiguration: With tools48.3%alibabacloud.com for Qwen3.5-397B-A17B on Humanity's Last Exam Full set — With tools (opens in a new tab)Humanity's Last Exam AccuracyReported Launch postAlibaba Cloud CommunitySelf-reportedQwen TeamRecord for Qwen3.5-397B-A17B on Humanity's Last Exam Full set — With tools
Humanity's Last Exam Text only — Mercor single-shotConfiguration: Mercor single-shot agent29.2%mercor.com for Qwen3.5-397B-A17B on Humanity's Last Exam Text only — Mercor single-shot (opens in a new tab)Archived copy (opens in a new tab)Humanity's Last Exam AccuracyReported Benchmark leaderboardMercorIndependentRecord for Qwen3.5-397B-A17B on Humanity's Last Exam Text only — Mercor single-shot
IFBench Original76.5%alibabacloud.com for Qwen3.5-397B-A17B on IFBench Original (opens in a new tab)AccuracyReported Launch postAlibaba Cloud CommunitySelf-reportedQwen TeamRecord for Qwen3.5-397B-A17B on IFBench Original
MMMU85.0%alibabacloud.com for Qwen3.5-397B-A17B on Massive Multi-discipline Multimodal Understanding MMMU (opens in a new tab)MMMU AccuracyReported Launch postAlibaba Cloud CommunitySelf-reportedQwen TeamRecord for Qwen3.5-397B-A17B on Massive Multi-discipline Multimodal Understanding MMMU
MedXpertQA MM — Mercor single-shotConfiguration: Mercor single-shot agent65.8%mercor.com for Qwen3.5-397B-A17B on MedXpertQA MM — Mercor single-shot (opens in a new tab)Archived copy (opens in a new tab)AccuracyReported Benchmark leaderboardMercorIndependentRecord for Qwen3.5-397B-A17B on MedXpertQA MM — Mercor single-shot
MMLU-Pro Original87.8%alibabacloud.com for Qwen3.5-397B-A17B on MMLU-Pro Original (opens in a new tab)AccuracyReported Launch postAlibaba Cloud CommunitySelf-reportedQwen TeamRecord for Qwen3.5-397B-A17B on MMLU-Pro Original
MMLU-Pro Original — Mercor single-shotConfiguration: Mercor single-shot agent88.0%mercor.com for Qwen3.5-397B-A17B on MMLU-Pro Original — Mercor single-shot (opens in a new tab)Archived copy (opens in a new tab)AccuracyReported Benchmark leaderboardMercorIndependentRecord for Qwen3.5-397B-A17B on MMLU-Pro Original — Mercor single-shot
SWE-bench Multilingual69.3%alibabacloud.com for Qwen3.5-397B-A17B on SWE-bench Multilingual (opens in a new tab)Resolved TasksReported Launch postAlibaba Cloud CommunitySelf-reportedQwen TeamRecord for Qwen3.5-397B-A17B on SWE-bench Multilingual
SWE-bench Multilingual — Mercor mini-swe-agent, 1000 stepsConfiguration: Mercor mini-swe-agent, 1000 steps, 3 hours61.7%mercor.com for Qwen3.5-397B-A17B on SWE-bench Multilingual — Mercor mini-swe-agent, 1000 steps (opens in a new tab)Archived copy (opens in a new tab)Resolved TasksReported Benchmark leaderboardMercorIndependentRecord for Qwen3.5-397B-A17B on SWE-bench Multilingual — Mercor mini-swe-agent, 1000 steps
SWE-bench Verified76.4%alibabacloud.com for Qwen3.5-397B-A17B on SWE-bench Verified (opens in a new tab)Resolved TasksReported Launch postAlibaba Cloud CommunitySelf-reportedQwen TeamRecord for Qwen3.5-397B-A17B on SWE-bench Verified
SWE-bench Verified — Mercor mini-swe-agent, 1000 stepsConfiguration: Mercor mini-swe-agent, 1000 steps, 3 hours72.1%mercor.com for Qwen3.5-397B-A17B on SWE-bench Verified — Mercor mini-swe-agent, 1000 steps (opens in a new tab)Archived copy (opens in a new tab)Resolved TasksReported Benchmark leaderboardMercorIndependentRecord for Qwen3.5-397B-A17B on SWE-bench Verified — Mercor mini-swe-agent, 1000 steps
Terminal-Bench 2.1 — Mercor Terminus 2Configuration: Mercor Terminus 2, 1000 steps, 3 hours53.9%mercor.com for Qwen3.5-397B-A17B on Terminal-Bench 2.1 — Mercor Terminus 2 (opens in a new tab)Archived copy (opens in a new tab)Terminal-Bench AccuracyReported Benchmark leaderboardMercorIndependentRecord for Qwen3.5-397B-A17B on Terminal-Bench 2.1 — Mercor Terminus 2
Toolathlon Verified40.7%toolathlon.xyz for Qwen3.5-397B-A17B on Toolathlon Verified (opens in a new tab)Archived copy (opens in a new tab)Tasks completedReported Benchmark leaderboardToolathlonIndependentToolathlon AuthorsRecord for Qwen3.5-397B-A17B on Toolathlon Verified
τ³-bench τ³ — AirlineReported as: enabled81.5%sierra-tau-bench-public.s3.us-west-2.amazonaws.com for Qwen3.5-397B-A17B (enabled) on τ³-bench τ³ — Airline (opens in a new tab)Archived copy (opens in a new tab)Task success rateReported Evaluated Benchmark leaderboardSierraIndependentSierra ResearchRecord for Qwen3.5-397B-A17B on τ³-bench τ³ — Airline
τ³-bench τ³ — Banking, text-embedding-3-large retrieval, v1.0.1 gradingReported as: enabledConfiguration: text-embedding-3-large retrieval9.8%sierra-tau-bench-public.s3.us-west-2.amazonaws.com for Qwen3.5-397B-A17B (enabled) on τ³-bench τ³ — Banking, text-embedding-3-large retrieval, v1.0.1 grading (opens in a new tab)Archived copy (opens in a new tab)Task success rateReported Evaluated Benchmark leaderboardSierraIndependentSierra ResearchRecord for Qwen3.5-397B-A17B on τ³-bench τ³ — Banking, text-embedding-3-large retrieval, v1.0.1 grading
τ³-bench τ³ — RetailReported as: enabled84.4%sierra-tau-bench-public.s3.us-west-2.amazonaws.com for Qwen3.5-397B-A17B (enabled) on τ³-bench τ³ — Retail (opens in a new tab)Archived copy (opens in a new tab)Task success rateReported Evaluated Benchmark leaderboardSierraIndependentSierra ResearchRecord for Qwen3.5-397B-A17B on τ³-bench τ³ — Retail
τ³-bench τ³ — TelecomReported as: enabled97.8%sierra-tau-bench-public.s3.us-west-2.amazonaws.com for Qwen3.5-397B-A17B (enabled) on τ³-bench τ³ — Telecom (opens in a new tab)Archived copy (opens in a new tab)Task success rateReported Evaluated Benchmark leaderboardSierraIndependentSierra ResearchRecord for Qwen3.5-397B-A17B on τ³-bench τ³ — Telecom