Ship Agents on Verified Capability: Astra ARC-AGI-3 Check

You are deciding whether to deploy a frontier agent in production. A capability claim needs the run behind it before shipping. The Astra ARC-AGI-3 split is the warning. The Provider Adapter-harness run of GPT-6 Astra reached 99.9% at high reasoning and cost $18,817. The Standard-harness ARC-AGI-3 Semi-Private run reached 62.7% at max reasoning and cost $26,098.

The harness changed the number

A harness is the wrapper that decides what the model sees, what it can carry, and how long the conversation lasts. A runner is the code that calls the model, handles tools, and decides when to stop. Those choices can change the result more than a model label.

ARC Prize's Standard harness makes the model itself choose which notes to keep visible during play. ARC Prize's Provider Adapter harness carries the model's opaque reasoning state across requests and compacts longer conversations. A vendor can report the higher number and call it the model. An operator must ask what the runner did.

At ARC-AGI-3 launch, all frontier systems tested scored under 1%, and Opus 4.6 at max reasoning led with 0.50%. The near-zero baseline makes the harness question more important. A benchmark that starts near zero can make a later jump look dramatic. That does not make the jump less real.

During a max-effort ARC-AGI-3 run with OpenAI's context-management adapter, GPT-6 Astra beat the median human action count on 96% of finished levels, averaging 51.7% fewer moves per level. Fewer moves matter when the job is a long workflow with tool calls. Action count is a proxy for effort. In agent work, effort is money and latency. A model that solves the task in fewer moves may also finish the job faster.

In the independent Artificial Analysis ranking, GPT-6 Astra's abstract Intelligence score was 61, putting it third and five points behind Anthropic's Fable 5.1. The abstract score is a different measure. It does not replace the interactive result. It does show that a strong axis can coexist with a weaker one.

The check is the deployment gate

Keep this list on the deployment ticket. Each item is testable from a log, a run manifest, or a repeatable re-audit note. The list is short because the failure modes are short. Most vendor claims fail early. Most production incidents start there.

• Is the harness named, and does the claim say what it preserves or discards?
• Is the runner named, including context handling, tool calls, and retries?
• Is the reasoning effort stated, not implied by a model label?
• Is the cost per run shown, including tokens, retries, and failed attempts?
• Is the re-audit path shown, with logs, seeds, or a repeatable run?
• Are comparisons made within the same harness, not across different runners?
• Is there a five-job internal eval on your own jobs, not just the public benchmark?

Memory changes the score. Tooling can hide the model's real behavior. A near-perfect run can still be uneconomic. A single production result is not a signal. The checklist forces the vendor to answer those four risks directly.

Different runners can change the score. Comparisons outside one harness should block the claim. A public benchmark is a proxy. The five-job eval keeps the check honest. Your production work has its own failure modes. Model selection for agents should end with your own tasks, not a leaderboard. Benchmark verification stops at the vendor's number. Production verification starts with your own jobs.

The five-job eval is the last gate

Treat the frontier model evaluation as a production process, not a single score. Ask for the harness, the runner, the cost, and the re-audit path. Then run the same agent on your own jobs.

Request the repeatable run manifest before the demo. Ask for the failure log and cost breakdown before the pitch. If the vendor cannot name the harness, treat the claim as unverified. Missing cost makes the run uneconomic. Missing re-audit makes the result anecdotal. The benchmark gap is real, but the deployment decision is operational. Ship only when the claim survives the check.