Check benchmark, cost, and fleet math before buying a robosecretary

A robosecretary pitch usually starts with a demo. The operator's job is to turn the demo into a budget item. Before deploying a computer-use agent fleet, check the public score, the per-task price, and the fleet math. A vendor can make all three look clean. Verify each one. The numbers must survive your workload.

The score is the gate

A benchmark score tells you how often the agent finishes a task without human repair. It does not tell you whether your tasks are in that set. Simular announced general availability of its computer-use agent Sai on September 23, 2026. In a Minecraft demonstration, Sai ran for 14 hours without step-by-step human intervention. During that session, it completed 8 of 15 goals at about $0.01 per hour. That demo is not a production workload.

A long demo proves endurance. It does not prove fit. Running for hours in a game is useful. Your tasks may be office work, data entry, or support queues. Check whether the benchmark includes those actions.

The public benchmark separates a strong demo from a repeatable system. The University of Hong Kong's XLANG Lab released OSWorld 2.0 in June. In August 2026, Sai posted a 73.0% partial score on the 108-task OSWorld 2.0 benchmark. If the vendor's score is not on a public benchmark, ask for the task list, the environment, and the scoring rule.

A partial score can hide where the agent fails. An agent may finish most steps and fail at the end. Another may fail early and save cost. Request the step-level result, not just the final score.

Per-task cost is the real cost

Per-task cost is the number your finance team will actually use. It includes compute, retries, and failed attempts. Sai's OSWorld 2.0 cost was $15.70 per task, compared with $23.70 for Claude Opus 5 Max Thinking and $26.62 for GPT-5.6 Sol Max. The spread is the entry fee. If your tasks are longer, noisier, or less standard, the per-task cost will move.

Get the cost of a failed task, not just a completed task. A retry can cost more than an earlier attempt. A stuck agent can use compute while it waits for a human. Your finance model needs the worst case, not the best case.

The vendor's task definition matters. A task may end when the agent says it is done, when a human confirms it, or when a test passes. The definition changes the cost.

Fleet pricing is an assumption until reproduced

Fleet math is where a robosecretary becomes a cost center. Simular claimed its SaiFleet could run 100 computers in parallel for under $1 in total. That claim is easy to quote and hard to reproduce. It depends on idle time, task length, machine size, and how often the agent restarts.

A parallel price assumes a stable task mix. It also assumes the machines stay busy. If your work arrives in bursts, the fleet may sit idle. If your tasks fail often, the fleet may restart. These effects change the total.

An autonomous computer fleet behaves like a set of machines, each with a runtime, a failure rate, and a cost. Before you buy, run the same workload on your own infrastructure. Compare the vendor's fleet pricing against your measured spend. If it does not survive that run, treat it as a sales figure.

Keep a simple log for the test run. Note the start time, the end time, the number of machines, and the spend. That log is your baseline. It gives you a number you can defend in a budget review.

Run the pre-buy check before signing

• Check that the public benchmark score covers your task class.
• Check that the per-task cost includes retries and failures.
• Check that the fleet price states machine count, idle time, and failure rate.