Shared benchmarks

24
Shared benchmarks for GPT-6 Sol and GPT-6.1 Sol
BenchmarkGPT-6 SolGPT-6.1 Sol
  • 54.3%
  • 60.0%
Evaluation details for APEX-Agents Original

The recorded benchmark version, metric, and evaluator sets match. Evaluation methodology is not recorded by the registry.

GPT-6 Sol

Record for GPT-6 Sol on APEX-Agents Original

GPT-6.1 Sol

Record for GPT-6.1 Sol on APEX-Agents Original
AutomationBench 1.0.6Potentially non-equivalent: Evaluator sets differ.
  • 21.2%low · 1.0.6 · OpenAI
  • 26.9%medium · 1.0.6 · OpenAI
  • 31.2%high · 1.0.6 · OpenAI
  • 33.2%xhigh · 1.0.6 · OpenAI
  • 32.0%max · 1.0.6 · OpenAI
  • 9.1%none · 1.0.6 · Zapier AutomationBench Team
  • 24.7%low · 1.0.6 · OpenAI
  • 31.7%medium · 1.0.6 · OpenAI
  • 33.2%high · 1.0.6 · OpenAI
  • 35.5%xhigh · 1.0.6 · OpenAI
  • 36.1%max · 1.0.6 · OpenAI
Potentially non-equivalent · Evaluation details for AutomationBench 1.0.6

Evaluator sets differ. Evaluation methodology is not recorded by the registry.

GPT-6 Sol

Record for GPT-6 Sol on AutomationBench 1.0.6
Record for GPT-6 Sol on AutomationBench 1.0.6
Record for GPT-6 Sol on AutomationBench 1.0.6
Record for GPT-6 Sol on AutomationBench 1.0.6
Record for GPT-6 Sol on AutomationBench 1.0.6
Record for GPT-6 Sol on AutomationBench 1.0.6

GPT-6.1 Sol

Record for GPT-6.1 Sol on AutomationBench 1.0.6
Record for GPT-6.1 Sol on AutomationBench 1.0.6
Record for GPT-6.1 Sol on AutomationBench 1.0.6
Record for GPT-6.1 Sol on AutomationBench 1.0.6
Record for GPT-6.1 Sol on AutomationBench 1.0.6
  • 89.4%
  • 95.4%
Evaluation details for BrowseComp 130-question subset — Mercor web research agent

The recorded benchmark version, metric, and evaluator sets match. Evaluation methodology is not recorded by the registry.

GPT-6 Sol

GPT-6.1 Sol

  • 95.0%
  • 96.0%
Evaluation details for CharXiv Descriptive and reasoning — Mercor single-shot

The recorded benchmark version, metric, and evaluator sets match. Evaluation methodology is not recorded by the registry.

  • 37.2%low
  • 56.6%medium
  • 65.3%high
  • 66.6%xhigh
  • 68.8%max
  • 64.4%low
  • 73.0%medium
  • 75.2%high
  • 71.9%xhigh
  • 71.9%max
Evaluation details for DeepSWE 1.1

The recorded benchmark version, metric, and evaluator sets match. Evaluation methodology is not recorded by the registry.

GPT-6 Sol

Record for GPT-6 Sol on DeepSWE 1.1
Record for GPT-6 Sol on DeepSWE 1.1
Record for GPT-6 Sol on DeepSWE 1.1
Record for GPT-6 Sol on DeepSWE 1.1
Record for GPT-6 Sol on DeepSWE 1.1

GPT-6.1 Sol

Record for GPT-6.1 Sol on DeepSWE 1.1
Record for GPT-6.1 Sol on DeepSWE 1.1
Record for GPT-6.1 Sol on DeepSWE 1.1
Record for GPT-6.1 Sol on DeepSWE 1.1
Record for GPT-6.1 Sol on DeepSWE 1.1
  • 70.5%
  • 72.3%
Evaluation details for DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps

The recorded benchmark version, metric, and evaluator sets match. Evaluation methodology is not recorded by the registry.

GPT-6 Sol

Record for GPT-6 Sol on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps

GPT-6.1 Sol

Record for GPT-6.1 Sol on DeepSWE 1.1 — Mercor mini-swe-agent, 500 steps
  • 49.1%
  • 52.0%
Evaluation details for Finance Agent Benchmark 2 — October 2026 evaluation

The recorded benchmark version, metric, and evaluator sets match. Evaluation methodology is not recorded by the registry.

  • 94.3%
  • 95.4%
Evaluation details for GPQA Diamond — Epoch AI harness

The recorded benchmark version, metric, and evaluator sets match. Evaluation methodology is not recorded by the registry.

GPT-6 Sol

Record for GPT-6 Sol on Graduate-Level Google-Proof Q&A Diamond — Epoch AI harness

GPT-6.1 Sol

Record for GPT-6.1 Sol on Graduate-Level Google-Proof Q&A Diamond — Epoch AI harness
  • 94.4%
  • 95.1%
Evaluation details for GPQA Diamond — Mercor single-shot

The recorded benchmark version, metric, and evaluator sets match. Evaluation methodology is not recorded by the registry.

GPT-6 Sol

Record for GPT-6 Sol on Graduate-Level Google-Proof Q&A Diamond — Mercor single-shot

GPT-6.1 Sol

Record for GPT-6.1 Sol on Graduate-Level Google-Proof Q&A Diamond — Mercor single-shot
  • 1.7%
  • 5.4%
Evaluation details for HLAB October 2026 evaluation

The recorded benchmark version, metric, and evaluator sets match. Evaluation methodology is not recorded by the registry.

GPT-6 Sol

Record for GPT-6 Sol on Harvey's Legal Agent Benchmark October 2026 evaluation

GPT-6.1 Sol

Record for GPT-6.1 Sol on Harvey's Legal Agent Benchmark October 2026 evaluation
  • 81.3%
  • 85.7%
Evaluation details for MedXpertQA MM — Mercor single-shot

The recorded benchmark version, metric, and evaluator sets match. Evaluation methodology is not recorded by the registry.

GPT-6 Sol

Record for GPT-6 Sol on MedXpertQA MM — Mercor single-shot

GPT-6.1 Sol

Record for GPT-6.1 Sol on MedXpertQA MM — Mercor single-shot
  • 88.9%
  • 91.0%
Evaluation details for MMLU-Pro Original — Mercor single-shot

The recorded benchmark version, metric, and evaluator sets match. Evaluation methodology is not recorded by the registry.

GPT-6 Sol

Record for GPT-6 Sol on MMLU-Pro Original — Mercor single-shot

GPT-6.1 Sol

  • 83.0%
  • 85.8%
Evaluation details for MMMU-Pro 1,730 questions — Mercor single-shot

The recorded benchmark version, metric, and evaluator sets match. Evaluation methodology is not recorded by the registry.

GPT-6 Sol

Record for GPT-6 Sol on MMMU-Pro 1,730 questions — Mercor single-shot

GPT-6.1 Sol

Record for GPT-6.1 Sol on MMMU-Pro 1,730 questions — Mercor single-shot
  • 43.9%low
  • 54.0%medium
  • 58.3%high
  • 60.5%xhigh
  • 64.4%max
  • 59.0%low
  • 66.8%medium
  • 69.6%high
  • 69.4%xhigh
  • 71.4%max
Evaluation details for OSWorld 2.0 v2026.08.08 Offline — Partial

The recorded benchmark version, metric, and evaluator sets match. Evaluation methodology is not recorded by the registry.

GPT-6 Sol

Record for GPT-6 Sol on OSWorld 2.0 v2026.08.08 Offline — Partial
Record for GPT-6 Sol on OSWorld 2.0 v2026.08.08 Offline — Partial
Record for GPT-6 Sol on OSWorld 2.0 v2026.08.08 Offline — Partial

GPT-6.1 Sol

Record for GPT-6.1 Sol on OSWorld 2.0 v2026.08.08 Offline — Partial
  • 60.7%
  • 73.9%
Evaluation details for SimpleQA Verified Original — Epoch AI harness

The recorded benchmark version, metric, and evaluator sets match. Evaluation methodology is not recorded by the registry.

  • 44.1%
  • 48.4%
Evaluation details for SWE Atlas Codebase QnA — Mercor mini-swe-agent, 250 steps

The recorded benchmark version, metric, and evaluator sets match. Evaluation methodology is not recorded by the registry.

  • 81.7%
  • 84.9%
Evaluation details for SWE-bench Multilingual — Mercor mini-swe-agent, 1000 steps

The recorded benchmark version, metric, and evaluator sets match. Evaluation methodology is not recorded by the registry.

  • 80.3%
  • 87.3%
Evaluation details for SWE-bench Verified — Mercor mini-swe-agent, 1000 steps

The recorded benchmark version, metric, and evaluator sets match. Evaluation methodology is not recorded by the registry.

GPT-6 Sol

GPT-6.1 Sol

  • 49.4%
  • 58.2%
Evaluation details for Terminal-Bench 4.0 — Codex

The recorded benchmark version, metric, and evaluator sets match. Evaluation methodology is not recorded by the registry.

GPT-6 Sol

Record for GPT-6 Sol on Terminal-Bench 4.0 — Codex

GPT-6.1 Sol

Record for GPT-6.1 Sol on Terminal-Bench 4.0 — Codex
  • 43.8%
  • 59.6%
Evaluation details for Terminal-Bench 4.0 — Mercor mini-swe-agent

The recorded benchmark version, metric, and evaluator sets match. Evaluation methodology is not recorded by the registry.

GPT-6 Sol

Record for GPT-6 Sol on Terminal-Bench 4.0 — Mercor mini-swe-agent

GPT-6.1 Sol

Record for GPT-6.1 Sol on Terminal-Bench 4.0 — Mercor mini-swe-agent
  • 44.4%
  • 55.1%
Evaluation details for Terminal-Bench 4.0 — Vals AI mini-swe-agent

The recorded benchmark version, metric, and evaluator sets match. Evaluation methodology is not recorded by the registry.

GPT-6 Sol

Record for GPT-6 Sol on Terminal-Bench 4.0 — Vals AI mini-swe-agent

GPT-6.1 Sol

Record for GPT-6.1 Sol on Terminal-Bench 4.0 — Vals AI mini-swe-agent
Terminal-BenchPotentially non-equivalent: Benchmark versions differ. Evaluator sets differ.
  • 83.1%2.1 — Vals AI Terminus 2 · Vals AI
  • 43.7%low · Science 0.1 · OpenAI
  • 47.6%medium · Science 0.1 · OpenAI
  • 51.1%high · Science 0.1 · OpenAI
  • 53.7%xhigh · Science 0.1 · OpenAI
  • 57.0%max · Science 0.1 · OpenAI
Potentially non-equivalent · Evaluation details for Terminal-Bench

Benchmark versions differ. Evaluator sets differ. Evaluation methodology is not recorded by the registry.

GPT-6 Sol

Record for GPT-6 Sol on Terminal-Bench 2.1 — Vals AI Terminus 2

GPT-6.1 Sol

Record for GPT-6.1 Sol on Terminal-Bench Science 0.1
Record for GPT-6.1 Sol on Terminal-Bench Science 0.1
Record for GPT-6.1 Sol on Terminal-Bench Science 0.1
Record for GPT-6.1 Sol on Terminal-Bench Science 0.1
Record for GPT-6.1 Sol on Terminal-Bench Science 0.1
  • 57.5%
  • 61.2%
Evaluation details for Vals Index 2.1

The recorded benchmark version, metric, and evaluator sets match. Evaluation methodology is not recorded by the registry.

GPT-6 Sol

Record for GPT-6 Sol on Vals Index 2.1

GPT-6.1 Sol

Record for GPT-6.1 Sol on Vals Index 2.1
  • 87.8%
  • 88.9%
Evaluation details for Vibe Code Bench 1.1 — OpenHands, October 2026 evaluation

The recorded benchmark version, metric, and evaluator sets match. Evaluation methodology is not recorded by the registry.