GPT-6 Astra vs Claude Fable 5.1: What the Benchmarks Actually Tell Us

OpenAI says GPT-6 Astra reaches 64.6% on Terminal-Bench Science 0.1. Anthropic says Claude Fable 5.1 reaches 52.6% on that same named benchmark. That 12-point spread looks clean until the fine print changes the question: benchmark scores describe a model, an agent harness, tools, prompts, trials, safeguards, task release, and grading rules working together. They are evidence. They are not a universal league table.

The headline comparison is straightforward. OpenAI’s GPT-6 Astra announcement reports 64.6% for Astra and 52.6% for Fable 5.1 on Terminal-Bench Science 0.1. Anthropic’s Fable 5.1 announcement reports the same 52.6% Fable result, alongside 24.7% for Fable 5, 29.0% for Opus 5, and 22.4% for GPT-5.6 Sol in Anthropic’s setup. The shared Fable figure makes the comparison unusually useful, but it does not make every surrounding result interchangeable. OpenAI’s Astra announcement and Anthropic’s Fable announcement should be read as vendor reports of evaluated systems, not as an independently run head-to-head tournament.

A benchmark name tells a reader what kind of task was attempted. It does not, by itself, say who constructed the environment, which version of the tasks was used, how the model was instructed, whether it could retry, which tools it could call, how long it was allowed to run, or whether safety systems could block an action. Those details matter most for agentic evaluations. An agent that edits files, opens a browser, calls a terminal, or controls a desktop is more than its base model weights. Change the harness and you can change its trajectory through a task.

That is why the first question after any large number should be: “What exactly was evaluated?” The second should be: “Under whose conditions?” The comparison becomes more honest when it separates a model’s potential under a research configuration from the product behavior a customer will see in a guarded deployment. OpenAI explicitly says its GPT evaluations ran in its research environment or through its API and may differ from production ChatGPT because system prompts and tools differ. OpenAI’s evaluation note is a useful warning against treating a launch chart as a product guarantee.

The immediate takeaway is not that the 64.6% versus 52.6% result is meaningless. It is a reported result on a common benchmark and therefore a relevant data point. The appropriate conclusion is narrower: in OpenAI’s reported Terminal-Bench Science 0.1 evaluation, Astra outscored the reported Fable 5.1 result by 12.0 percentage points. Readers should not stretch that sentence into a claim that Astra is 12 points better at all science work, all coding work, or every deployed agent workflow.

Key takeaways

• On the vendor-reported Terminal-Bench Science 0.1 comparison, GPT-6 Astra scored 64.6% and Claude Fable 5.1 scored 52.6%.OpenAI
• Anthropic reports a standard error of plus or minus 3.5 to 4.5 points per model on Terminal-Bench Science 0.1, so small gaps should be treated cautiously.Anthropic
• Terminal-Bench 4.0, OSWorld 2.0, and Humanity’s Last Exam answer different questions. A strong score on one does not settle another.Anthropic
• Fable 5.1’s ARC-AGI scores have a different status: ARC Prize lists 97.5% on ARC-AGI-1 and 90.0% on ARC-AGI-2 as verified results.ARC Prize
• Product decisions should include context, output limits, price, effort settings, safeguards, latency, and a buyer’s own evaluation suite.Claude Platform documentation

Terminal-Bench Science 0.1 is presented as an agentic scientific-research evaluation. Its value is that it asks more than whether a model can answer a question in a chat box. The model must operate through an environment and complete tasks that involve a sequence of decisions. That makes it closer to the operational reality of research assistance, where locating material, using software, interpreting intermediate outputs, and recovering from errors all matter. It is also precisely why configuration choices become part of the result.

OpenAI’s 64.6% Astra figure and 52.6% Fable figure are therefore best understood as end-to-end success rates for the particular systems it ran. The score incorporates model behavior, tool access, prompts, stopping rules, harness behavior, and the evaluation’s grading. A model may reason correctly but fail because it issues a command in the wrong format. Another may solve a task after a retry permitted by one harness but not another. A third may be stopped by a safety layer that correctly identifies an action as disallowed. The final score captures the system outcome, which is valuable, but it cannot assign every point to raw reasoning ability.

Anthropic supplies an important uncertainty statement for this benchmark: the standard error is plus or minus 3.5 to 4.5 points per model. That statement is not a footnote to skip. It says repeated measurement has meaningful spread, and it gives readers a rough scale for interpreting marginal differences. A one- or two-point lead in a vendor chart could easily be noise or a consequence of the run conditions. A 12-point difference is larger than the stated per-model standard-error range, but that does not dissolve all uncertainty. It does not account for differences between providers’ agent scaffolding, prompts, safeguards, or trial aggregation. Anthropic’s methodology note makes the uncertainty visible; it does not supply a confidence interval for the cross-company gap.

The historical comparison on Anthropic’s page adds another complication. Anthropic says the public leaderboard uses three trials per task with a Claude Code harness and reports 30.0% for Opus 5 and 21.4% for Fable 5. Its own setup reproduced those at 29.0% and 24.7%, respectively, which it characterizes as within noise. That is a practical demonstration that even runs intended to target the same evaluation can move. It is not an indictment of benchmarks. It is a reason to report the run configuration with the score.

Terminal-Bench 4.0 belongs in the discussion because it is often read as a coding leaderboard, yet it is better described as a test of an agent system. The benchmark involves tasks, trials, and a harness. The model has to operate within that machinery, generally using terminal-like tools to modify or inspect an environment and reach a graded outcome. The score reflects whether the whole loop produces a passing solution, not merely whether a detached model response sounds plausible.

Anthropic reports 55.8% for Fable 5.1 on Terminal-Bench 4.0 and 60.9% for Claude Mythos 5.1. Those two products are described by Anthropic as the same model with different safeguard levels. Fable is generally available. Mythos is restricted to trusted-access programs for cybersecurity and life-sciences work. The gap is a concrete reminder that safety policy can affect measured agentic performance, especially when the task path includes actions that a more permissive deployment allows. Anthropic’s launch page reports both values and the product distinction.

That does not mean the Fable result is artificially low or that the Mythos result is the “real” score. Both can be real descriptions of different deployable systems. A customer considering public Fable access needs to know how Fable behaves with its safeguards. A vetted research program deciding whether Mythos fits an authorized workflow needs the Mythos result. The analytical error is to quote the higher number without naming the configuration, then attribute it to the generally available product.

OpenAI makes the same systems point from the other direction. Its Astra page says that, when testing third-party models, it uses a simpler research setup, while Codex has a more complex production configuration. It further notes that provider-side safeguards and computer-tool implementations differ. This is not a minor caveat about formatting. Different tool implementations can change whether an agent sees the same screen state, whether it can recover from an error, how it navigates, and whether an action needs confirmation. OpenAI’s footnote on cross-provider testing directly limits what a raw cross-vendor score can prove.

For procurement teams, the implication is practical. Ask vendors for the model identifier, the harness version, the task release, the number of trials, the prompt or developer-message policy, tool permissions, effort level, timeout, retry policy, and safety configuration. If those fields are absent, the number may still be interesting. It is not sufficient evidence for a production decision.

OSWorld 2.0 measures computer use, but its two Fable 5.1 figures show why one benchmark can contain more than one interpretation. Anthropic reports 77.9% under partial grading and 41.7% under strict grading. Partial grading recognizes progress toward a task; strict grading requires the full specified outcome. The 36.2-point spread is not evidence that one result is wrong. It tells a buyer what type of reliability the scoring rule rewards.

Partial credit can be informative when a workflow has useful intermediate states. A model that correctly opens the right application, locates the record, and enters most fields has produced some value even if it misses the final save condition. Strict grading is more appropriate when incomplete execution is operationally equivalent to failure, such as submitting the wrong form, leaving a financial workflow uncompleted, or modifying a system without satisfying all constraints. The right metric depends on the harm and recoverability of an unfinished task.

Anthropic also cautions that its OSWorld 2.0 figures use the benchmark authors’ August 2026 task release and that Fable 5 and Opus 5 were re-run under the same conditions. Because task files differ from earlier releases, Anthropic says the results are not directly comparable with previously published OSWorld 2.0 scores and therefore shows no competitor score. The OSWorld footnote models a discipline that launch charts often lack: refusing a cross-time comparison when the task set changed.

OpenAI reports Astra at 72.6% on OSWorld 2.0 in latency simulations, at roughly 40 minutes per task, compared with 65.7% for GPT-5.6 Sol at roughly 75 minutes. Its footnote says the Claude performance cited for OSWorld-V2 Offline was reproduced on the official leaderboard and that the Claude OSWorld 2.0 scores use official settings rather than the modified tasks and grading from the Fable 5.1 system card. OpenAI’s OSWorld discussion therefore establishes that the published numbers are not all drawn from a single uniform definition of OSWorld performance.

The careful interpretation is not to pick a favorite number. It is to match the metric to the work. A team automating repetitive but reviewable desktop tasks may care about partial progress, time per task, and the ease of a human completing the last step. A team that needs a fully completed, auditable workflow should look first at strict success, failure modes, and the intervention path. In either case, a reproduced trial on the organization’s own application is more valuable than a generic screen-use percentage.

Humanity’s Last Exam, or HLE, is a multidisciplinary reasoning evaluation. Anthropic reports 60.9% for Fable 5.1 without tools and 65.0% with tools. The difference is useful because it separates a configuration where the model relies on its internal reasoning from one where it can use external assistance. It also cautions against comparing a tool-enabled result against a no-tools result as if the agent conditions were identical. Anthropic’s benchmark table labels both modes.

A tool-enabled score can be more predictive for a research assistant that will actually browse, calculate, inspect files, or run code. A no-tools score can better isolate what the model produces without external retrieval or execution. Neither is automatically more “intelligent.” They are answers to different deployment questions. The same distinction appears across agent benchmarks: the closer an evaluation gets to an actual workflow, the more its score includes the quality of the surrounding system.

ARC-AGI has a different role. ARC Prize lists Fable 5.1’s results as independently verified: 97.5% on ARC-AGI-1 Semi-Private and 90.0% on ARC-AGI-2 Semi-Private at maximum effort. The page also lists five reasoning variants. ARC-AGI-1 ranges from 90.0% at Low to 97.5% at Max; ARC-AGI-2 ranges from 78.3% at Low to 90.0% at Max. ARC Prize’s verified result page makes the effort condition part of the result rather than hiding it.

Independent verification changes the evidentiary status of a score. It means an external benchmark organization has verified the published result according to its process. It does not turn ARC-AGI into a complete proxy for coding, research, office automation, safety behavior, latency, or cost. ARC tasks test abstract pattern induction in a constrained grid setting. A model can score highly there and still need separate evidence for controlling a browser or maintaining a long-running coding session. Conversely, success on terminal tasks cannot substitute for a verified abstract-reasoning result.

The comparison is thus asymmetrical in a useful way. Astra’s Terminal-Bench Science lead is a vendor-run cross-provider comparison whose source itself flags configuration differences. Fable’s ARC-AGI result is independently verified but speaks to a different task family. A serious report should preserve both facts without pretending they resolve the same question.

Safeguards are often treated as an inconvenient asterisk after a benchmark table. In agentic systems, they are part of the evaluated product. They can refuse a request, ask for confirmation, pause a task, block an action, route a workflow to a fallback, or stop an agent after detecting behavior outside the allowed scope. Those interventions can reduce task-completion rates in some evaluations. They may also be the behavior a real organization wants.

Anthropic is explicit that Fable 5.1 and Mythos 5.1 have the same model but different levels of safeguards. It says Fable’s cybersecurity safeguards are designed to support vulnerability discovery but not exploit development, while Mythos is available through restricted programs. It also says safeguards can intervene and that, in relevant evaluations, an intervention can result in a zero score. Anthropic’s safety and deployment discussion gives a reason not to silently combine Fable and Mythos benchmark figures.

OpenAI likewise says Astra’s extra safety checks can slow, pause, or stop legitimate work, including defensive cybersecurity. In ChatGPT or Codex, it says a paused task may ask the user to review the action before continuing; in the API, the task stops. OpenAI’s deployment section frames this as a deliberate tradeoff, not a measurement defect. The user experience of a well-governed agent includes how it behaves at the edge of authorization.

The methodological question is whether the evaluation’s objective aligns with the deployment objective. If a task demands an action that policy should block, awarding zero for the block measures a different property from safe operation. If the task is benign and a safeguard blocks it, that may indicate a false positive worth measuring. Both are important, but they should be labeled separately: task success, safe refusal, false-positive rate, and escalation quality are not one metric.

This distinction becomes especially important in cross-vendor comparisons. One provider may test a raw model with fewer constraints; another may test a production configuration; a third may use a research-only access tier. A score comparison without safeguard disclosure can accidentally reward the system that takes more actions, even where those actions would be unacceptable in the buyer’s environment. The better question is not “Which agent refuses less?” It is “Which system completes authorized tasks reliably while giving a reviewable response when authority is uncertain?”

Vendor-run benchmarks are not automatically unreliable. Vendors often have the expertise, compute, access, and engineering detail to evaluate difficult systems. Their reports can reveal progress early and can publish enough methodology to make a claim testable. The limitation is institutional: the vendor chooses which configurations, baselines, charts, and interpretations to emphasize. Readers should treat the result as a disclosed measurement by an interested party.

Independent verification reduces one layer of that concern because a separate organization checks the result. ARC Prize’s page labels the Fable 5.1 results “verified” and identifies the benchmark modes, effort variants, and per-task cost. That is stronger evidence for the exact ARC claims than a vendor-only press release would be. It still does not establish that one provider’s product is the best option for every business workflow. Verification answers whether a result was obtained under the verifier’s procedure, not every downstream question a buyer may have.

The strongest practical evidence is a stack, not a single chart. Start with vendor disclosures, prefer benchmark-maintainer or independent reproductions, then run a local evaluation against actual tasks. For code agents, use representative repositories and review rules. For computer use, use real applications and access controls. For research, use production sources, tools, and citation standards.

A local evaluation should record more than pass or fail. Capture time, cost, retries, human intervention, tool errors, policy blocks, quality of the final artifact, and any consequential mistakes. Segment results by task difficulty and risk. A model that finishes uncomplicated tickets cheaply may be the right default even if another model leads a maximum-effort research benchmark. A slower model may earn its cost on the small fraction of tasks where deep reasoning or a very long context is decisive.

This approach also makes results durable. Public benchmark releases change, vendors alter prompts and tool stacks, and product safeguards evolve. A repeatable internal harness lets a team detect whether a model update improved the work it actually cares about. It turns benchmark reading from brand comparison into operational measurement.

Fable 5.1’s platform documentation lists a one-million-token context window, a 128,000-token maximum output, input pricing of $10 per million tokens, and output pricing of $50 per million tokens. It lists cache reads at $0.25 per million tokens, which is 2.5% of the base input price, or 0.025 times that rate. Claude’s model overview also describes the model as slower relative to the current Claude lineup and identifies adaptive thinking as always on, with a high default effort in the API.

Those details shape the interpretation of a benchmark result. An agent that can retain a very large working set may reduce the need to summarize a long codebase or a lengthy research record. A 128,000-token output ceiling may matter for a workflow that generates extensive artifacts. Low cache-read pricing can change the economics of iterative, context-heavy tasks. None of those features guarantees higher quality, but they can determine whether a model is viable for a particular workload.

OpenAI lists the same base standard API prices for Astra: $10 per million input tokens and $50 per million output tokens, with separate cache read and write rates, and a fast mode at twice the standard price. OpenAI’s availability and pricing section also says Astra’s evaluation scores are maximum at any effort. That phrase is crucial. Maximum-effort benchmark scores describe an upper-performance setting. A production team needs to know the cost, latency, and quality at the effort level it intends to use.

Price comparisons should therefore be workload comparisons. Estimate input tokens, output tokens, cached-context share, tool calls, number of agent turns, retries, and human review time. A model with the same nominal input and output price can be cheaper in practice if it uses fewer retries or benefits more from caching. It can also be more expensive if it routinely invokes high effort, produces long reasoning traces, or takes more attempts to finish a task. Benchmark percentages alone cannot answer this accounting question.

The same is true for speed. OpenAI’s Astra OSWorld discussion reports roughly 40 minutes per task in its latency simulations, compared with roughly 75 minutes for GPT-5.6 Sol, but task duration is an experimental result, not a promise about every API request. OpenAI says actual experience can depend on the configuration. Buyers should measure end-to-end wall-clock time in their own environment, including queueing, tool latency, approvals, and repair work.

Begin by classifying the source. Is this a vendor announcement, a vendor system card, a public leaderboard, a benchmark-maintainer result, or an independent reproduction? The category does not decide whether a claim is true, but it tells the reader how much outside checking has occurred. Write “OpenAI reports” or “Anthropic reports” for vendor figures. Write “ARC Prize verifies” only for the verified ARC results. That vocabulary keeps the evidence hierarchy visible.

Then identify the unit of evaluation. Is it a base model response, an API call, a model plus a harness, a full product configuration, or an agent with browser and terminal tools? For Terminal-Bench 4.0, the appropriate unit is the agent system: model plus harness, tasks, and trials. For OSWorld, identify whether grading is partial or strict and which task release was used. For HLE, identify whether tools were available. For ARC-AGI, identify the benchmark version, public or semi-private mode, and reasoning effort.

Next, look for configuration asymmetries. Were providers run through the same harness? Were safeguards turned on? Was a restricted tier substituted for a general product? Were time limits equal? Did the evaluation use the production system prompt? Did a vendor apply fallback behavior after a refusal? Each answer narrows the claim a score can support. “A model under our research harness scored X” is precise. “This product is universally better” is not.

After that, inspect uncertainty and sample design. A stated standard error, as Anthropic provides for Terminal-Bench Science 0.1, should change the language around close results. Trial counts matter. Benchmark maintenance matters. A revised task release may be a better test, but it can break a historical comparison. A reader should prefer charts that expose these decisions rather than ones that conceal them behind a single bold number.

Finally, translate the public result into a decision test. Build a small task set from real work, with predeclared success criteria and a safety review. Run more than one trial where nondeterminism matters. Keep the model version and configuration fixed during the comparison. Measure completion, time, cost, quality, interventions, and severe mistakes. Re-run when the vendor changes the model, tool harness, or policy. This evaluation loop will not eliminate judgment, but it makes the judgment inspectable.

The current evidence supports several specific statements. OpenAI reports that GPT-6 Astra scored 64.6% on Terminal-Bench Science 0.1, compared with 52.6% for Claude Fable 5.1, in OpenAI’s evaluation. OpenAI reports its cross-provider work used a simpler research setup and warns that provider safeguards and tool implementations differ. Anthropic reports Fable’s 52.6% score and a standard error of plus or minus 3.5 to 4.5 points per model. Anthropic reports Fable’s 55.8% Terminal-Bench 4.0 result, 77.9% partial and 41.7% strict OSWorld 2.0 results, and HLE results of 60.9% without tools and 65.0% with tools.

It also supports a separate statement about ARC-AGI: ARC Prize independently lists Fable 5.1 at 97.5% on ARC-AGI-1 Semi-Private and 90.0% on ARC-AGI-2 Semi-Private at maximum effort. ARC Prize verifies those benchmark results, but the task family differs from terminal work and desktop control. The distinction is not a technicality. It is the difference between an abstract-reasoning result and a claim about an agent’s reliability in an operational environment.

What the evidence does not support is a simple declaration of an overall winner. The scorecards involve different benchmarks, different grading definitions, different tool conditions, and different safety configurations. Even the shared Terminal-Bench Science figure should be framed as a vendor-reported comparison with disclosed methodological limitations. The numbers are useful precisely when their boundaries are kept intact.

For teams choosing between these systems, the sensible reading is conditional. If the priority is the reported Terminal-Bench Science result under OpenAI’s research evaluation, Astra holds the stated lead. If the priority is a verified ARC-AGI result at maximum effort, Fable has the cited verified scores. If the priority is a long-context agent workflow, Fable’s documented one-million-token context and cache economics belong in the test plan. If the priority is safe computer use, measure strict completion, review behavior, and false-positive interventions in the actual environment.

Benchmarks become decision-grade evidence only when the buyer asks the same questions the methodology asks: which system, which tools, which safeguards, which task release, which scoring rule, which effort, and which trials? The answer will rarely fit in one number. It will be more useful than one.