AI Tools Review Editorial TeamOn 22 September 2026, Anthropic launched Claude Opus 5.5 and, about 90 minutes later, OpenAI launched GPT-6 Sol and GPT-6 Luna. GPT-6 Astra had shipped three weeks earlier. Each launch came with a benchmark table. None of those tables was built to be read alongside the others.
This page is a benchmark matrix with one rule: a number only sits next to another number if both came from the same table. Where that is not true, we say "not comparable" rather than guess. For model-by-model detail, see our Claude Opus 5.5 review and GPT-6 Astra review.
Choosing between Claude, ChatGPT, Gemini and Grok?
Benchmarks are one input. Our free 20-page guide compares the big four on price, features and what each is actually good at, so you can match a model to your work.
Update, 02/10/2026:
Google published a new frontier comparison on 30/09/2026 for Gemini 4 Argon, including DeepSWE v1.1, AutomationBench and CWE-bench v1 results. See the full Gemini 4 Argon benchmark breakdown.
Update 29/09/2026: Claude Sonnet 5.5 and GPT-6.1 Sol Scores
Two mid-tier models landed after this page was first published. Anthropic released Claude Sonnet 5.5 on 28 September 2026, and OpenAI launched GPT-6.1 Sol at DevDay on 29 September. OpenAI also confirmed it had shelved GPT-6.1 Astra. Following our same-table rule, each vendor's new figures sit in their own table below. For the head-to-head buying decision, read Claude Sonnet 5.5 vs GPT-6.1 Sol.
Matrix D: Anthropic's Sonnet 5.5 launch table
| Benchmark | Sonnet 5.5 | Opus 5.5 | Sonnet 5 |
|---|---|---|---|
| Terminal-Bench 4.0 (agentic coding) | 70.6% | 66.4% | 10.3% |
| CursorBench 4.0 (coding) | 55.5% | 57.8% | — |
| GDPval-AA v2.1 (knowledge work, Elo) | 1844 | 1846 | — |
| Humanity's Last Exam (with tools) | 64.5% | 67.7% | — |
| OSWorld 2.1 (computer use) | 80.1% | 81.8% | — |
| SWE-Bench Pro (system card) | 81.3% | 89.9% | — |
| AutomationBench (system card) | 44.7% | 42.5% | — |
Source: Anthropic Claude Sonnet 5.5 launch page and system card, 28 September 2026. — = not reported. SWE-Bench Pro and AutomationBench figures are from the system card; the AutomationBench run is not the same as the 40.0% Opus 5.5 figure in Matrix A, so do not mix them. In the same launch table Anthropic lists GPT-6 Sol at 1487 on GDPval-AA v2.1.
The headline: Sonnet 5.5 beats Opus 5.5 on Terminal-Bench 4.0 (70.6% vs 66.4%), up from 10.3% for Sonnet 5, and edges it on AutomationBench in the system card (44.7% vs 42.5%). Opus 5.5 still leads SWE-Bench Pro by 8.6 points and Humanity's Last Exam with tools by 3.2. On GDPval-AA the two are 2 Elo apart. Anthropic prices Sonnet 5.5 at half of Opus 5.5 ($2/$10 vs $4/$20).
Matrix E: OpenAI's GPT-6.1 Sol launch figures
| Benchmark | GPT-6.1 Sol | GPT-6 Astra |
|---|---|---|
| DeepSWE v1.1 | 75.2% (high) | 74.8% |
| OSWorld 2.0 (OpenAI harness) | 71.4% | 73.5% |
| Terminal-Bench Science | 57.0% | 68.1% |
| AutomationBench | ~36.1% (max) | — |
Source: OpenAI, 29 September 2026, via OpenAI's GPT-6.1 Sol model documentation and The Next Web. All figures OpenAI-reported. — = not reported.
GPT-6.1 Sol narrowly passes Astra on DeepSWE v1.1 (75.2% vs 74.8%), up from GPT-6 Sol's 68.8% in Matrix B, but trails Astra on OSWorld 2.0 and by 11.1 points on Terminal-Bench Science. Do not compare these with Matrix D. OSWorld 2.0 in OpenAI's harness and OSWorld 2.1 in Anthropic's are different measurements, and the AutomationBench runs are from different vendor reports.
The independent view is below: on the Artificial Analysis index, Sonnet 5.5 scores 56 and GPT-6.1 Sol 52. For where the new models fit overall, see our best AI models guide and, for coding specifically, the best AI coding agents.
The Verdict by Category
| Category | Leader on available evidence | Key figures | Reported by |
|---|---|---|---|
| Agentic coding | Claude Opus 5.5 / Claude Sonnet 5.5 | Terminal-Bench 4.0: Sonnet 5.5 70.6%, Opus 5.5 66.4%. FrontierCode 54.4%, CursorBench 57.8% (Opus 5.5) | Anthropic |
| Knowledge work | Claude Opus 5.5 ≈ Claude Sonnet 5.5 | GDPval-AA v2.1 1846 vs 1844 Elo (Astra 1542) | Anthropic |
| Business automation | GPT-6 Astra ≈ Claude Opus 5.5 | AutomationBench 41.4% vs 40.0% | Anthropic table (Zapier benchmark) |
| Scientific research | GPT-6 Astra | Terminal-Bench-Science 0.1 64.6% (Opus 5.5 58.7%) | Anthropic table |
| Maths & science reasoning | GPT-6 Astra | FrontierMath Tier 4 v2 97.6, GPQA Diamond 96.0 | OpenAI (no Opus 5.5 figure) |
| Independent composite | Claude Opus 5.5 | AA Intelligence Index: Opus 5.5 58, Sonnet 5.5 56, Fable 5.1 53, Astra 53, GPT-6.1 Sol 52 | Artificial Analysis (checked 29 Sept) |
Sources: Anthropic, 22 and 28 September 2026; OpenAI via DataCamp; Artificial Analysis. Checked 23 September 2026; updated 29 September 2026.
Two things stand out. First, GPT-6 Sol does not lead any category. It is not built to: at $2/$10 per million tokens it is priced against mid-tier models, not flagships (see our AI API pricing comparison). Second, the maths row has no Claude 5.5 challenger at all, because Anthropic did not publish GPQA or FrontierMath for Opus 5.5 in its launch table. An empty cell is not a loss. It is missing data.
Matrix A: Anthropic's Opus 5.5 Launch Table
This is the most useful single table right now because it holds five models across nine benchmarks. Claude figures are at max effort with production safeguards on. The GPT figures are, in Anthropic's words, "as reported by OpenAI". Anthropic did not rerun them.
| Benchmark | Opus 5.5 | Fable 5.1 | Opus 5 | GPT-6 Astra | GPT-5.6 Sol |
|---|---|---|---|---|---|
| Terminal-Bench 4.0 (agentic coding) | 66.4% | 55.8% | 52.3% | 57.9% | 37.3% |
| FrontierCode v1.1 Main (coding) | 54.4% | 50.3% | 48.0% | 53.3% | 47.5% |
| CursorBench 4.0 (coding) | 57.8% | 51.8% | 46.6% | — | 41.7% |
| GDPval-AA v2.1 (knowledge work, Elo) | 1846 | 1735 | 1708 | 1542 | 1588 |
| AutomationBench (Zapier) | 40.0% | 31.4% | 26.9% | 41.4% | 28.8% |
| Humanity's Last Exam (with tools) | 67.7% | 65.6% | 63.6% | 57.2% | — |
| Terminal-Bench-Science 0.1 | 58.7% | 52.6% | 29.0% | 64.6% | 22.4% |
| OSWorld 2.0 (partial credit) | 81.8% | 80.7% | 74.0% | — | — |
| Chartography (with tools) | 89.0% | 88.4% | 83.4% | — | — |
Source: Anthropic, 22 September 2026. Bold marks the row leader. Claude models at max effort with safeguards on; GPT figures as reported by OpenAI. Terminal-Bench 4.0: Opus 5.5 at xhigh effort, GPT-6 Astra at high; standard error ±2.6 pts for Opus 5.5. — = not reported in this table.
Opus 5.5 leads seven of nine rows; Astra leads two. Anthropic also claims cost advantages: at default effort Opus 5.5 beats Astra on FrontierCode at about 20% of the cost per task, and matches Astra on Terminal-Bench 4.0 at about 40%. Those are Anthropic's claims, not independent measurements.
Terminal-Bench 4.0: agentic coding
Anthropic's launch table only. Opus 5.5 at xhigh, GPT-6 Astra at high. Higher is better.
- OpenAI
- Anthropic
Data table
| Model | Vendor | Value |
|---|---|---|
| Claude Opus 5.5 (xhigh, ±2.6 SE) | Anthropic | 66.4% |
| GPT-6 Astra (high, as reported by OpenAI) | OpenAI | 57.9% |
| Claude Fable 5.1 | Anthropic | 55.8% |
| Claude Opus 5 | Anthropic | 52.3% |
| GPT-5.6 Sol (as reported by OpenAI) | OpenAI | 37.3% |
Source: Anthropic Claude Opus 5.5 launch table, 22 September 2026. As of 23 September 2026.
GDPval-AA v2.1: professional knowledge work (Elo)
Anthropic's launch table only. Claude at max effort. Higher is better.
- OpenAI
- Anthropic
Data table
| Model | Vendor | Value |
|---|---|---|
| Claude Opus 5.5 | Anthropic | 1,846 |
| Claude Fable 5.1 | Anthropic | 1,735 |
| Claude Opus 5 | Anthropic | 1,708 |
| GPT-5.6 Sol (as reported by OpenAI) | OpenAI | 1,588 |
| GPT-6 Astra (as reported by OpenAI) | OpenAI | 1,542 |
Source: Anthropic Claude Opus 5.5 launch table, 22 September 2026. As of 23 September 2026.
Matrix B: OpenAI's GPT-6 Sol/Luna Launch Table
OpenAI compared Sol and Luna against Claude Opus 5 and Fable 5 / 5.1, not Opus 5.5. That is not evasion: Opus 5.5 launched about 90 minutes earlier. It does mean this table cannot tell you how Sol compares with Opus 5.5, except on AutomationBench (see the bridge). Effort levels are shown in brackets because OpenAI varied them by model.
| Benchmark | GPT-6 Sol | GPT-6 Luna | GPT-6 Astra | GPT-5.6 Sol | Fable 5.1 | Fable 5 | Opus 5 |
|---|---|---|---|---|---|---|---|
| AutomationBench 1.0.6 | 33.2% (xhigh) | — | 30.3% (low) | — | 31.4% | — | 26.9% |
| DeepSWE v1.1 | 68.8% (max) | 66.6% (max) | — | — | — | 69.9% (xhigh) | 66.0% (med) |
| OSWorld 2.0 (offline, OpenAI harness) | 60.5% (xhigh) | 58.1% (max) | 72.6% | 57.0% (med) | — | — | 60.3% (med) |
| Agents' Last Exam | 56.4% (max) | — | 59.3% | 53.6% | — | — | — |
Source: OpenAI, 22 September 2026, via VentureBeat and Vellum. — = not reported in this table. Effort level shown where OpenAI stated it.
The story OpenAI tells here is cost, not rank. On DeepSWE, Sol (68.8%) sits just behind Fable 5 (69.9%) at what OpenAI says is about 80% lower cost per task. On AutomationBench, Sol costs $0.27 per task, which OpenAI says is 8.9x cheaper than Fable 5.1 and 11.1x cheaper than Opus 5. OpenAI also reports Sol's deception rate on an adversarial coding test fell to 1.3% from 10.4% for its predecessor. For the full Sol and Luna picture, see GPT-6 Sol and Luna explained and GPT-6 Sol vs Claude Opus 5.5.
Matrix C: OpenAI's GPT-6 Astra Card
| Benchmark | GPT-6 Astra | GPT-5.6 Sol | Fable 5.1 | Opus 5 | Gemini 3.8 Flash |
|---|---|---|---|---|---|
| GPQA Diamond | 96.0 | 94.6 | 93.7 | 93.7 | 95.3 |
| FrontierMath Tier 4 v2 | 97.6 | 83.0 | 87.8 | 73.2 | — |
| ExploitBench | 100.0 | — | — | — | — |
| SRE-Bench | 88.0 | — | — | — | — |
| ScreenSpot-Pro | 92.7 | — | — | — | — |
Source: OpenAI GPT-6 Astra release, via DataCamp. All figures OpenAI-reported. — = not reported. Opus 5.5 did not exist when this card was published.
Astra's card is strongest on reasoning. FrontierMath Tier 4 at 97.6 is 9.8 points clear of Fable 5.1 and 24.4 clear of Opus 5 in the same table. GPQA Diamond is tighter: Astra 96.0, Gemini 3.8 Flash 95.3, and both Claude models 93.7. On long context, OpenAI reports 100% on MRCR v2 8-needle at 256K-512K tokens and 96.3% at 512K-1M.
ExploitBench at 100.0 is a capability flag as much as a score. OpenAI rates Astra "Critical" for cyber under its Preparedness Framework. Our Claude Opus 5.5 vs GPT-6 Astra comparison covers what that means in practice.
Benchmarks tell you who won a test. Which one fits your work?
Get our free 20-page guide comparing Claude, ChatGPT, Gemini and Grok on price, features and what each is good at. It was written before this week's launches, so pair it with the matrices on this page.
The Traps
1. OSWorld 2.0 means two different things
OSWorld 2.0 appears in both vendors' tables. Anthropic scores it with partial credit. OpenAI runs an offline version in its own harness. The proof that these are different measurements is Claude Opus 5, which appears in both: 74.0% in Anthropic's table (max effort) and 60.3% in OpenAI's (medium effort). Same model, same benchmark name, 13.7 points apart.
Now imagine the naive comparison: Opus 5.5 81.8% (Anthropic) against GPT-6 Astra 72.6% (OpenAI), and "Opus 5.5 wins computer use by 9 points". That headline would be built on two different scoring methods. The gap between harnesses for Opus 5 alone is bigger than the gap between the two models. We mark OSWorld 2.0 as not comparable across vendors.
2. ARC-AGI-3 at 99.9% needs an asterisk
GPT-6 Astra's 99.9% on ARC-AGI-3 was achieved only with an expensive stateful adapter harness. On ARC Prize's standard stateless harness it scores roughly 17% to 63% depending on reasoning tier. Quote the 99.9% without the harness and you overstate it by a wide margin.
3. Effort levels differ inside the same chart
On OpenAI's AutomationBench chart, GPT-6 Sol runs at xhigh effort (33.2%) while GPT-6 Astra runs at low (30.3%). That makes Sol look like it beats Astra. In Anthropic's table, Astra scores 41.4%. Effort settings change results by double digits, and vendors choose them.
4. Vendor-run is not independent
Every number in Matrices A to E was published by a model vendor. Anthropic's table takes its GPT figures from OpenAI's own reporting rather than rerunning them. The only independent composite here is Artificial Analysis.
5. Anthropic says margins matter less now
Anthropic itself says that at this level of capability, benchmark margins are a less reliable guide to real-world differences, and that in its own use the gap between Opus 5.5 and Fable 5.1 is narrower than its scores suggest. When a vendor tells you to discount its own lead, take it seriously.
The One Bridge: AutomationBench
Zapier's AutomationBench is the only benchmark where the two vendors' tables agree on shared anchors. Claude Opus 5 scores 26.9% and Fable 5.1 scores 31.4% in both Anthropic's and OpenAI's tables. Matching anchors mean the scores come from a consistent run, so we can place GPT-6 Sol (from OpenAI) on the same scale as Opus 5.5 (from Anthropic). That gives the only fair six-model ranking available this week.
AutomationBench: business workflow automation
Combined from Anthropic's and OpenAI's tables, which agree on Opus 5 (26.9) and Fable 5.1 (31.4). Higher is better.
- OpenAI
- Anthropic
Data table
| Model | Vendor | Value |
|---|---|---|
| GPT-6 Astra (Anthropic's table) | OpenAI | 41.4% |
| Claude Opus 5.5 (Anthropic's table) | Anthropic | 40.0% |
| GPT-6 Sol (xhigh, OpenAI's table) | OpenAI | 33.2% |
| Claude Fable 5.1 (both tables) | Anthropic | 31.4% |
| GPT-5.6 Sol (Anthropic's table) | OpenAI | 28.8% |
| Claude Opus 5 (both tables) | Anthropic | 26.9% |
Source: Anthropic Opus 5.5 launch table and OpenAI GPT-6 Sol launch (AutomationBench 1.0.6, via VentureBeat/Vellum), 22 September 2026. As of 23 September 2026.
The result is a clear tier. Astra and Opus 5.5 are 1.4 points apart at the top. Sol is about 7 points behind Opus 5.5 but, per OpenAI, runs at $0.27 per task. One caveat: our sources do not state the effort level behind Astra's 41.4%, and OpenAI's own chart shows Astra at low effort scoring 30.3%. We used 41.4% because it is the figure in the same table as Opus 5.5.
Independent: Artificial Analysis Intelligence Index
Artificial Analysis Intelligence Index
Independent composite score. Higher is better. Grok 4.7 figure from our 23 September check.
- OpenAI
- Anthropic
- xAI
Data table
| Model | Vendor | Value |
|---|---|---|
| Claude Opus 5.5 | Anthropic | 58 |
| Claude Sonnet 5.5 | Anthropic | 56 |
| Claude Fable 5.1 | Anthropic | 53 |
| GPT-6 Astra | OpenAI | 53 |
| GPT-6.1 Sol | OpenAI | 52 |
| GPT-6 Sol (max effort) | OpenAI | 48 |
| Grok 4.7 (checked 23 Sept) | xAI | 46 |
Source: Artificial Analysis Intelligence Index leaderboard. As of 29 September 2026.
This is the one ranking not produced by a vendor. Updated 29 September 2026: Artificial Analysis now indexes Claude Opus 5.5 at 58, the top score on the board, with the new Claude Sonnet 5.5 second at 56. Fable 5.1 and GPT-6 Astra tie at 53, and GPT-6.1 Sol scores 52, four points up on GPT-6 Sol's 48. On 23 September, Artificial Analysis measured GPT-6 Sol at 131.2 output tokens per second with a blended price of $1.54 per million tokens, and Grok 4.7 scored 46. The independent index therefore puts both Anthropic 5.5 models ahead of every OpenAI model, including the $10/$50 Astra.
Gemini and Grok: What We Have, and What We Don't
Gemini. Our only figure for Google's current model is from OpenAI's Astra card: Gemini 3.8 Flash scores 95.3 on GPQA Diamond, ahead of both Claude models in that table (93.7) and behind Astra (96.0). That is strong for a Flash-tier model. Google's Pro tier is stuck: Gemini 3.5 Pro was announced at I/O on 19 May 2026 and is still unreleased after missing June, July and August targets. The live Pro model is Gemini 3.1 Pro, which appears in none of these tables. Gemini 4 has confirmed pre-training but no release.
Grok. Grok 4.7 launched on 21 September 2026. Its only comparable number is 46 on the Artificial Analysis index, two points behind GPT-6 Sol. Neither Anthropic nor OpenAI included it in their tables. Independent testers describe it as slow and verbose. For how the four families compare beyond benchmarks, see Claude vs ChatGPT vs Gemini vs Grok.
Where Each Leader Loses
- Do not choose Claude Opus 5.5 on benchmarks if your work is scientific research workflows (Astra 64.6% vs 58.7% on Terminal-Bench-Science) or maths-heavy reasoning, where Anthropic published no GPQA or FrontierMath figure. For terminal-heavy agentic coding, Claude Sonnet 5.5 scores higher on Terminal-Bench 4.0 (70.6% vs 66.4%) at half the price.
- Do not choose GPT-6 Astra on benchmarks if your work is coding or professional documents. It trails Opus 5.5 on Terminal-Bench 4.0 (57.9% vs 66.4%), GDPval-AA (1542 vs 1846) and Humanity's Last Exam with tools (57.2% vs 67.7%). It also costs $10/$50 per million tokens against Opus 5.5's $4/$20 (see Claude API pricing).
- Do not choose GPT-6 Sol expecting flagship scores. It trails Opus 5.5 by about 7 points on AutomationBench and Astra on Agents' Last Exam (56.4% vs 59.3%). Its case is price, and at the same $2/$10 list price GPT-6.1 Sol now scores higher (52 vs 48 on Artificial Analysis).
- Do not choose GPT-6.1 Sol for science workloads over Astra. OpenAI's own figures put it 11.1 points behind Astra on Terminal-Bench Science (57.0% vs 68.1%).
- Do not use this matrix at all if your task is not on it. None of these benchmarks tests your data, your prompts or your latency budget. Run your own evaluation on the two or three models that lead your category.
Methodology
- Same-table rule. Numbers are only compared when they come from the same published table. Cross-table comparison is allowed only where shared anchor models score identically in both (AutomationBench: Opus 5 and Fable 5.1).
- Attribution. Every figure is labelled by who reported it: Anthropic, OpenAI, or Artificial Analysis. Competitor figures inside a vendor table are marked "as reported by" the rival.
- Effort levels. Shown where the vendor stated them. Where not stated, we say so.
- Exclusions. OSWorld 2.0 is not compared across vendors. ARC-AGI-3 is not charted because the headline figure depends on a non-standard harness. Figures from secondary sources are taken as reported by VentureBeat, Vellum and DataCamp.
- No estimates. Blank cells stay blank. We do not impute missing cross-vendor scores or a Gemini 3.5 Pro result.
- Checked: 23 September 2026; Sonnet 5.5, GPT-6.1 Sol and Artificial Analysis figures checked 29 September 2026.
Sources
- Anthropic: Claude Sonnet 5.5
- OpenAI: GPT-6.1 Sol model documentation
- The Next Web: OpenAI GPT-6.1 Sol at DevDay
- Artificial Analysis: model leaderboard
- Anthropic: Claude Opus 5.5
- OpenAI: Introducing GPT-6 Sol and Luna
- OpenAI: GPT-6 Astra
- VentureBeat: OpenAI releases GPT-6 Sol and Luna
- Vellum: GPT-6 Sol and Luna benchmarks explained
- DataCamp: GPT-6 Astra
- TechCrunch: OpenAI launches GPT-6 Sol and Luna
- Artificial Analysis: GPT-6 Sol
Last updated: 29 September 2026 (Claude Sonnet 5.5, GPT-6.1 Sol and Artificial Analysis index added). Benchmark figures are vendor-reported unless marked as Artificial Analysis.
Get the free Claude vs ChatGPT, Gemini & Grok guide
A 20-page guide comparing the big four AI families on price, features and what each is good at. Use it with this matrix to shortlist the models worth testing on your own work.
