Audit a Software-Factory Agent Before It Touches Your Repos

The money is not the audit

You are deciding whether an end-to-end coding agent can access production repos. Before you do, audit the vendor's stage map, control points, and rollback evidence. GitHub Copilot Autofix generated a security patch for snowflake-connector-net, but the patch introduced an exploitable shell-injection flaw; an unauthenticated attacker stole a Jira token within five days of the patch.

Factory closed a $200 million round on Sept. 15, 2026, lifting its valuation to $5 billion, more than three times its earlier value. Cognition raised $2 billion in early September 2026, at a $48 billion valuation. Factory's software factories are live at organizations including NVIDIA, EY, Adobe, Palo Alto Networks, Adyen, Blackstone, Wipro, and Comarch. Those signals show demand, not control.

The loop is the map

Factory describes its loop as moving changes through build, test, review, security checks, release, and monitoring. It routes narrow, measurable tasks to simple Droid agents or skills. Automations coordinate repeatable workflows with a shared objective and memory. Droid Computers handle persistent or local execution, while Missions split complex multi-agent work into parallel tracks that run for hours or days. Warp moves tickets through spec, implementation, review, and verification in version-controlled pipelines called Factories; Factories cover 30-35% of Warp's internal tasks. The audit follows the lifecycle and asks where the agent stops and a person takes over.

The checklist maps the loop

Use the loop, not the vendor name, to evaluate the agent. Each item is a one-minute test. A vague answer means the gate is missing.

• Autonomy level: ask what the agent may do without a human, and where a person must approve.
• Model routing: ask which model handles planning, code, tests, review, and security checks.
• Human review gates: ask where a person must approve before merge, deploy, or config change.
• Per-stage failure rates: ask for failure rates for construction, tests, review, security, release, and monitoring.
• Rollback paths: ask how a bad change is reverted, and who can trigger the revert.
• Review-gate logs: ask for logs showing who approved, what was checked, and when.
• Prompt-injection defenses: ask how repository or tool content is treated as data, not instructions.
• Sandboxing: ask what network, secrets, and deploy rights the agent has.
• Kill switch: ask how to stop all agent work in one action.
• Evidence: ask for sample pull requests, failed runs, and reverts.

Work the list in order for every vendor. A later gate cannot fix an earlier missing control.

Gates need logs, not promises

Cognition's Devin analyzes repositories, maps dependencies, plans steps, executes, self-tests, and self-reviews before opening a pull request, with human review at the pull-request checkpoint rather than continuous keystroke-level supervision. A November 2025 Cognition performance review put Devin's pull-request merge rate at 67%, up from 34% a year earlier, and claimed a fourfold speedup in problem-solving and twice the resource efficiency versus its 2024 predecessor. Devin works best on clearly defined requirements and still struggles with ambiguous or exploratory coding. A merge rate is a useful number, not a safety proof.

1Password's engineering team kept human oversight over cross-file dependencies, test-suite maintenance, and rollback logic while AI agents refactored a large monolithic codebase. The pattern is a named gate, a named owner, and a record of what happened.

A September 2026 preprint from University of Illinois researchers described attacks against named versions of 12 coding-agent products, where repository or tool content was turned into higher-authority instructions, persistent rules, or configuration changes. A review log is the only way to see whether the gate caught the problem.

Monitoring is part of the loop. A shipped change that later breaks is still a failure. Ask for the last bad change, the revert, and the time between detection and rollback. If the vendor cannot show that sequence, keep the agent out of the repo.