Benchmarks

41 results across 27 benchmarks

Benchmark results for Inkling
BenchmarkScoreSource
APEX-Agents OriginalEffort: High33.8%mercor.com for Inkling (high) on APEX-Agents Original (opens in a new tab)Archived copy (opens in a new tab)Pass rateReported Benchmark leaderboardMercorIndependentRecord for Inkling on APEX-Agents Original
BrowseComp 130-question subset — Mercor web research agentEffort: HighConfiguration: Mercor web research agent, 100 steps, 3 hours50.4%mercor.com for Inkling (high) on BrowseComp 130-question subset — Mercor web research agent (opens in a new tab)Archived copy (opens in a new tab)BrowseComp AccuracyReported Benchmark leaderboardMercorIndependentRecord for Inkling on BrowseComp 130-question subset — Mercor web research agent
BrowseComp With context management — Inkling evaluationReported as: effort=0.99Configuration: Context management (Inkling evaluation)77.1%thinkingmachines.ai for Inkling (effort=0.99) on BrowseComp With context management — Inkling evaluation (opens in a new tab)BrowseComp AccuracyReported Launch postThinking Machines LabSelf-reportedRecord for Inkling on BrowseComp With context management — Inkling evaluation
Chartography October 2026 leaderboardEffort: High10.0%surgehq.ai for Inkling (High) on Chartography October 2026 leaderboard (opens in a new tab)Archived copy (opens in a new tab)Chartography Pass@1Reported Benchmark leaderboardSurge AIIndependentRecord for Inkling on Chartography October 2026 leaderboard
CharXiv Descriptive and reasoning — Mercor single-shotEffort: HighConfiguration: Mercor single-shot agent88.9%mercor.com for Inkling (high) on CharXiv Descriptive and reasoning — Mercor single-shot (opens in a new tab)Archived copy (opens in a new tab)CharXiv AccuracyReported Benchmark leaderboardMercorIndependentRecord for Inkling on CharXiv Descriptive and reasoning — Mercor single-shot
CharXiv Reasoning — No toolsReported as: effort=0.99Configuration: No tools78.1%thinkingmachines.ai for Inkling (effort=0.99) on CharXiv Reasoning — No tools (opens in a new tab)CharXiv AccuracyReported Model cardThinking Machines LabSelf-reportedRecord for Inkling on CharXiv Reasoning — No tools
CharXiv Reasoning — Python enabledReported as: effort=0.99Configuration: With Python82.0%thinkingmachines.ai for Inkling (effort=0.99) on CharXiv Reasoning — Python enabled (opens in a new tab)CharXiv AccuracyReported Model cardThinking Machines LabSelf-reportedRecord for Inkling on CharXiv Reasoning — Python enabled
CWE-bench 1 — OpenCodeEffort: HighConfiguration: OpenCode harness37.0%cwe-bench.com for Inkling (high) on CWE-bench 1 — OpenCode (opens in a new tab)Archived copy (opens in a new tab)CWE-bench Programmatic Pass@1Reported Benchmark leaderboardCollinear AIIndependentRecord for Inkling on CWE-bench 1 — OpenCode
DeepSWE 1.1 — Mercor mini-swe-agent, 500 stepsEffort: HighConfiguration: Mercor mini-swe-agent, 500 steps, 2 hours5.0%mercor.com for Inkling (high) on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps (opens in a new tab)Archived copy (opens in a new tab)DeepSWE Pass@1Reported Benchmark leaderboardMercorIndependentRecord for Inkling on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps
Finance Agent Benchmark 2 — October 2026 evaluationReported as: 0.9946.6%vals.ai for Inkling (0.99) on Finance Agent Benchmark 2 — October 2026 evaluation (opens in a new tab)Archived copy (opens in a new tab)Finance Agent AccuracyReported Benchmark leaderboardVals AIIndependentRecord for Inkling on Finance Agent Benchmark 2 — October 2026 evaluation
FrontierSWE 2 — October 2026 evaluationConfiguration: Proximus harness, 5 trials, 20-hour budget4.1%frontierswe.com for Inkling on FrontierSWE 2 — October 2026 evaluation (opens in a new tab)Archived copy (opens in a new tab)FrontierSWE ScoreReported Benchmark leaderboardProximalIndependentRecord for Inkling on FrontierSWE 2 — October 2026 evaluation
GPQA DiamondReported as: effort=0.9987.2%thinkingmachines.ai for Inkling (effort=0.99) on Graduate-Level Google-Proof Q&A Diamond (opens in a new tab)GPQA Diamond AccuracyReported Model cardThinking Machines LabSelf-reportedRecord for Inkling on Graduate-Level Google-Proof Q&A Diamond
GPQA Diamond — Epoch AI harnessEffort: xHighConfiguration: Epoch AI harness88.3%epoch.ai for Inkling (xhigh) on Graduate-Level Google-Proof Q&A Diamond — Epoch AI harness (opens in a new tab)Archived copy (opens in a new tab)GPQA Diamond AccuracyReported Evaluated Benchmark leaderboardEpoch AIIndependentRecord for Inkling on Graduate-Level Google-Proof Q&A Diamond — Epoch AI harness
GPQA Diamond — Mercor single-shotEffort: HighConfiguration: Mercor single-shot agent87.2%mercor.com for Inkling (high) on Graduate-Level Google-Proof Q&A Diamond — Mercor single-shot (opens in a new tab)Archived copy (opens in a new tab)GPQA Diamond AccuracyReported Benchmark leaderboardMercorIndependentRecord for Inkling on Graduate-Level Google-Proof Q&A Diamond — Mercor single-shot
GPQA Diamond — Vals AI harnessReported as: 0.99Configuration: Vals AI harness: mean of zero-shot and few-shot CoT87.1%vals.ai for Inkling (0.99) on Graduate-Level Google-Proof Q&A Diamond — Vals AI harness (opens in a new tab)Archived copy (opens in a new tab)GPQA Diamond AccuracyReported Benchmark leaderboardVals AIIndependentRecord for Inkling on Graduate-Level Google-Proof Q&A Diamond — Vals AI harness
HLAB October 2026 evaluationReported as: 0.992.1%vals.ai for Inkling (0.99) on Harvey's Legal Agent Benchmark October 2026 evaluation (opens in a new tab)Archived copy (opens in a new tab)HLAB Task Pass RateReported Benchmark leaderboardVals AIIndependentRecord for Inkling on Harvey's Legal Agent Benchmark October 2026 evaluation
Humanity's Last Exam Text only — Mercor single-shotEffort: HighConfiguration: Mercor single-shot agent25.5%mercor.com for Inkling (high) on Humanity's Last Exam Text only — Mercor single-shot (opens in a new tab)Archived copy (opens in a new tab)Humanity's Last Exam AccuracyReported Benchmark leaderboardMercorIndependentRecord for Inkling on Humanity's Last Exam Text only — Mercor single-shot
IFBench OriginalReported as: effort=0.9979.8%thinkingmachines.ai for Inkling (effort=0.99) on IFBench Original (opens in a new tab)AccuracyReported Launch postThinking Machines LabSelf-reportedRecord for Inkling on IFBench Original
MCP Atlas Public, April 2026 updateEffort: xHigh76.0%labs.scale.com for Inkling (xHigh) on MCP Atlas Public, April 2026 update (opens in a new tab)Pass rateReported Benchmark leaderboardScale AIIndependentRecord for Inkling on MCP Atlas Public, April 2026 update
MedXpertQA MM — Mercor single-shotEffort: HighConfiguration: Mercor single-shot agent60.0%mercor.com for Inkling (high) on MedXpertQA MM — Mercor single-shot (opens in a new tab)Archived copy (opens in a new tab)AccuracyReported Benchmark leaderboardMercorIndependentRecord for Inkling on MedXpertQA MM — Mercor single-shot
MMLU-Pro Original — Mercor single-shotEffort: HighConfiguration: Mercor single-shot agent86.8%mercor.com for Inkling (high) on MMLU-Pro Original — Mercor single-shot (opens in a new tab)Archived copy (opens in a new tab)AccuracyReported Benchmark leaderboardMercorIndependentRecord for Inkling on MMLU-Pro Original — Mercor single-shot
MMLU-Pro Original — Vals AI harnessReported as: 0.99Configuration: Vals AI harness86.3%vals.ai for Inkling (0.99) on MMLU-Pro Original — Vals AI harness (opens in a new tab)Archived copy (opens in a new tab)AccuracyReported Benchmark leaderboardVals AIIndependentRecord for Inkling on MMLU-Pro Original — Vals AI harness
MMMU-Pro 1,730 questions — Mercor single-shotEffort: HighConfiguration: Mercor single-shot agent69.7%mercor.com for Inkling (high) on MMMU-Pro 1,730 questions — Mercor single-shot (opens in a new tab)Archived copy (opens in a new tab)MMMU-Pro accuracySetting: Not statedReported Benchmark leaderboardMercorIndependentRecord for Inkling on MMMU-Pro 1,730 questions — Mercor single-shot
MMMU-Pro Vals AI harnessReported as: 0.99Configuration: Vals AI harness78.5%vals.ai for Inkling (0.99) on MMMU-Pro Vals AI harness (opens in a new tab)Archived copy (opens in a new tab)MMMU-Pro accuracySetting: Standard (4 options)Reported Benchmark leaderboardVals AIIndependentRecord for Inkling on MMMU-Pro Vals AI harness
Riemann-bench October 2026 leaderboardEffort: xHigh15.2%surgehq.ai for Inkling (xHigh) on Riemann-bench October 2026 leaderboard (opens in a new tab)Archived copy (opens in a new tab)Riemann-bench Pass@1Reported Benchmark leaderboardSurge AIIndependentRecord for Inkling on Riemann-bench October 2026 leaderboard
SimpleQA Verified OriginalReported as: effort=0.9943.9%thinkingmachines.ai for Inkling (effort=0.99) on SimpleQA Verified Original (opens in a new tab)AccuracyReported Launch postThinking Machines LabSelf-reportedRecord for Inkling on SimpleQA Verified Original
SimpleQA Verified Original — Epoch AI harnessEffort: xHighConfiguration: Epoch AI harness40.3%epoch.ai for Inkling (xhigh) on SimpleQA Verified Original — Epoch AI harness (opens in a new tab)Archived copy (opens in a new tab)AccuracyReported Evaluated Benchmark leaderboardEpoch AIIndependentRecord for Inkling on SimpleQA Verified Original — Epoch AI harness
StrongREJECT OriginalReported as: effort=0.9998.6%thinkingmachines.ai for Inkling (effort=0.99) on StrongREJECT Original (opens in a new tab)Harmful-request refusal rateReported Launch postThinking Machines LabSelf-reportedRecord for Inkling on StrongREJECT Original
SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 stepsEffort: HighConfiguration: Mercor mini-swe-agent, 250 steps, 1 hour21.2%mercor.com for Inkling (high) on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps (opens in a new tab)Archived copy (opens in a new tab)Mean pass@1Reported Benchmark leaderboardMercorIndependentRecord for Inkling on SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps
SWE-bench Multilingual — Mercor mini-swe-agent, 1000 stepsEffort: HighConfiguration: Mercor mini-swe-agent, 1000 steps, 3 hours77.7%mercor.com for Inkling (high) on SWE-bench Multilingual — Mercor mini-swe-agent, 1000 steps (opens in a new tab)Archived copy (opens in a new tab)Resolved TasksReported Benchmark leaderboardMercorIndependentRecord for Inkling on SWE-bench Multilingual — Mercor mini-swe-agent, 1000 steps
SWE-bench Verified — bash-only harnessReported as: effort=0.99Configuration: bash-only harness77.6%thinkingmachines.ai for Inkling (effort=0.99) on SWE-bench Verified — bash-only harness (opens in a new tab)Resolved TasksReported Launch postThinking Machines LabSelf-reportedRecord for Inkling on SWE-bench Verified — bash-only harness
SWE-bench Verified — Vals AI mini-swe-agentReported as: 0.99Configuration: mini-swe-agent harness77.6%vals.ai for Inkling (0.99) on SWE-bench Verified — Vals AI mini-swe-agent (opens in a new tab)Archived copy (opens in a new tab)Resolved TasksReported Benchmark leaderboardVals AIIndependentRecord for Inkling on SWE-bench Verified — Vals AI mini-swe-agent
SWE-bench Pro PublicReported as: effort=0.9954.3%thinkingmachines.ai for Inkling (effort=0.99) on SWE-bench Pro Public (opens in a new tab)Public tasks resolvedReported Launch postThinking Machines LabSelf-reportedRecord for Inkling on SWE-bench Pro Public
Terminal-Bench 2.1 — Thinking Machines internal coding harnessReported as: effort=0.99Configuration: Thinking Machines internal coding harness63.8%thinkingmachines.ai for Inkling (effort=0.99) on Terminal-Bench 2.1 — Thinking Machines internal coding harness (opens in a new tab)Terminal-Bench AccuracyReported Launch postThinking Machines LabSelf-reportedRecord for Inkling on Terminal-Bench 2.1 — Thinking Machines internal coding harness
Terminal-Bench 2.1 — Vals AI Terminus 2Reported as: 0.99Configuration: Terminus-2 harness47.6%vals.ai for Inkling (0.99) on Terminal-Bench 2.1 — Vals AI Terminus 2 (opens in a new tab)Archived copy (opens in a new tab)Terminal-Bench AccuracyReported Benchmark leaderboardVals AIIndependentRecord for Inkling on Terminal-Bench 2.1 — Vals AI Terminus 2
Terminal-Bench 4.0 — Mercor mini-swe-agentEffort: HighConfiguration: Mercor mini-swe-agent, 1000 steps, 8 hours0.5%mercor.com for Inkling (high) on Terminal-Bench 4.0 — Mercor mini-swe-agent (opens in a new tab)Archived copy (opens in a new tab)Terminal-Bench AccuracyReported Benchmark leaderboardMercorIndependentRecord for Inkling on Terminal-Bench 4.0 — Mercor mini-swe-agent
Terminal-Bench 4.0 — Vals AI mini-swe-agentReported as: 0.99Configuration: mini-swe-agent harness0.5%vals.ai for Inkling (0.99) on Terminal-Bench 4.0 — Vals AI mini-swe-agent (opens in a new tab)Archived copy (opens in a new tab)Terminal-Bench AccuracyReported Benchmark leaderboardVals AIIndependentRecord for Inkling on Terminal-Bench 4.0 — Vals AI mini-swe-agent
Toolathlon VerifiedReported as: effort=0.9945.5%thinkingmachines.ai for Inkling (effort=0.99) on Toolathlon Verified (opens in a new tab)Tasks completedReported Launch postThinking Machines LabSelf-reportedRecord for Inkling on Toolathlon Verified
Vals Index 2.1Reported as: 0.9928.7%vals.ai for Inkling (0.99) on Vals Index 2.1 (opens in a new tab)Archived copy (opens in a new tab)Vals Index GDP-weighted AccuracyReported Benchmark leaderboardVals AIIndependentRecord for Inkling on Vals Index 2.1
Vibe Code Bench 1.1 — OpenHands, October 2026 evaluationReported as: 0.99Configuration: OpenHands harness19.2%vals.ai for Inkling (0.99) on Vibe Code Bench 1.1 — OpenHands, October 2026 evaluation (opens in a new tab)Archived copy (opens in a new tab)Vibe Code Bench ScoreReported Benchmark leaderboardVals AIIndependentRecord for Inkling on Vibe Code Bench 1.1 — OpenHands, October 2026 evaluation
τ³-bench τ³ — Banking, AllTools retrieval, v1.0.1 gradingEffort: MaxConfiguration: AllTools retrieval25.0%sierra-tau-bench-public.s3.us-west-2.amazonaws.com for Inkling (max) on τ³-bench τ³ — Banking, AllTools retrieval, v1.0.1 grading (opens in a new tab)Archived copy (opens in a new tab)Task success rateReported Evaluated Benchmark leaderboardSierraIndependentSierra ResearchRecord for Inkling on τ³-bench τ³ — Banking, AllTools retrieval, v1.0.1 grading