Ship Agent PRs Like DoltLite: Gate Tests, Track Divergences, Keep Humans on Exceptions

DoltLite's beta results suggest a gate for agent PRs. It is a SQLite fork that keeps the standard SQLite components above the B-tree layer. Its development involved about 2,000 pull requests and a team of agents. It reached Beta version 0.50.0 about five months after launch.

For operators, the reported results point to a merge policy. The policy should test behavior, measure performance, record exceptions, and keep humans accountable for architecture, security, and migration.

What DoltLite actually shipped

DoltLite passes 100% of sqllogictest, described as a suite of 5.8M complex queries. It passes 99.46% of SQLite's 892,277 TCL-based acceptance tests, with 4,809 known divergences.

A 99.46% pass rate is not the same as 100% compatibility. Treat the 4,809 divergences as the exception list. A production repository needs that list made explicit.

In a reported benchmark, in-memory DoltLite databases were 10% slower on reads and 60% slower on writes than SQLite. A hot-path regression blocks merge unless the ledger records an approved exception. The PR must show the cost. If the cost is acceptable, the ledger records why.

At Beta, DoltLite's storage format is described as stable, with future breaking changes expected to have a supported migration path. Agents may propose storage changes. A human should approve whether a breaking change is allowed, how migration is handled, and what support window applies.

The agent PR merge gate

Use a five-part gate for agent-generated code in production repositories. Enforce the gate in CI.

1. Task must be small and testable. Split agent work into narrow PRs. A PR that touches schema, query planning, error handling, and UI is too large. A PR that fixes one parser branch, adds one test, and updates one ledger entry is mergeable.
2. Pass regression suite or document divergence. The agent must not be allowed to weaken the suite. If a test fails, the PR must include a divergence record: test name, expected behavior, actual behavior, risk, owner, and review status.
3. Benchmark performance impact. Run a fixed benchmark set for the affected subsystem. Report read and write latency, throughput, memory, and error rate where relevant. A PR that improves correctness but degrades a hot path needs an explicit performance exception.
4. Update divergence ledger. The ledger is a living document. It lists known differences from the reference system, the reason, the owner, and the exit criteria. Divergences should shrink over time. If they grow, the team should stop merging and investigate.
5. Human owns architecture, security, and migration decisions. Agents can propose changes. Humans approve changes that alter data format, authentication, authorization, network exposure, upgrade path, or operational runbooks. That is the control point.

Agent workflow controls

Start with a protected branch and a merge policy. Agent PRs should be opened against a staging branch, not main. CI should run the full regression suite, the benchmark suite, and a divergence-lint check. The divergence-lint check verifies that every failing test has a ledger entry and that every ledger entry has an owner.

Code review should be split. Automated review checks style, test coverage, and policy violations. Human review checks intent, risk, and exception quality. Do not ask a human to read every line of generated code. Ask the human to verify that the PR is inside the allowed scope and that the evidence is sufficient.

Keep the agent's scope narrow. For a database engine, that might mean parser fixes, query planner hints, or test generation. For a web service, it might mean API validation, error mapping, or documentation. The narrower the scope, the easier the gate becomes. The gate should be specific to the subsystem, not generic.

Track three numbers. First, the regression pass rate. Second, the number of open divergences. Third, the benchmark delta. If the pass rate falls, stop. If divergences rise without a clear reason, stop. If the benchmark delta exceeds the agreed threshold, stop. Record the stop decision and the number that triggered it.

Merge agent PRs only when the CI gate passes, the divergence ledger is updated, and a human approves the exception. If any number moves against you, stop.