Ship a production agent by checking the harness: context, tools, memory, evals, traces, escalation

Why the agent harness matters

Evaluation should test production failure modes, not only benchmark response quality. Check the harness before launch. The harness controls context, tool access, and approval gates.

The model does not inherently know business systems. Context assembly is part of the harness. The harness must select, format, and limit the information the model receives.

Run the checklist before launch

Use this checklist as a launch gate. Do not ship an agent because it passes response benchmarks. Ship it when traces show what the agent did, why it did it, and what it was allowed to do.

1. Context assembly. Define what the agent may read. Include only the records, policies, and task state needed for the current step. Exclude unrelated data, secrets, and stale drafts. Limit the context window to the current step. Version the context template. Record which sources were included for each run.
2. Tool contracts. Every tool should define bounded tool actions, schemas, timeouts, and error states to limit unsafe or repeated actions. A tool that can change state needs a contract. The contract should specify required fields, allowed values, maximum duration, and the error the agent must receive when the call fails. Define stable error states so failures are visible. Do not rely on undefined behavior.
3. Tool permissions. Permissions should scope agent actions and reduce prompt injection risk. Separate read tools from write tools. Separate low-risk writes from high-risk writes. A read-only agent may read records and request approval. It should not perform irreversible actions without review. Scope permissions by role, tenant, and task. Revoke access when the task ends.
4. Memory policy. If the system stores state, decide what stored state it retains, where it stores it, and when it expires. Scope stored state to the user, account, or case. Do not let it leak across accounts. Store only what future decisions require. Redact sensitive fields before persistence. Provide a way to inspect and delete stored state. Record why stored state was retained.
5. Evaluation. Evaluation should test production failure modes, not only benchmark response quality. Build a test set from real incidents, edge cases, and rejected actions. Include cases where the expected action is to stop. Include cases where the tool returns an error, a record is missing, or the user asks for an out-of-scope action. Track whether the agent chose the right tool, used the right context, respected permissions, and escalated when required. Run evals when a new failure mode appears.
6. Observability. Preserve inspectable traces of agent steps. Each trace must show the prompt, context, tool calls, results, permission checks, and reason. The trace should let an operator reconstruct the run.
7. Human escalation. Human escalation or confirmation should gate write actions. Define the actions that require approval: any write action that changes state or cannot be undone. The escalation path should include the proposed action, the evidence, the risk, and the expected outcome. A human should be able to approve, reject, or modify the action. If the human is unavailable, the agent should stop.

If a step cannot be checked, block launch.

Operate the loop after launch

Review traces after every incident and when a repeated failure appears. Look for repeated tool failures, permission denials, context that exceeds the intended scope, and escalations that humans reject. Review each pattern to decide whether context, tool contracts, permissions, evals, traces, or escalation need a change.

Update the failing control: context, tool contract, permission, eval, trace, or escalation. Add the incident to evals. Treat every incident as a test case. If traces show an error, add the case to the evaluation set. If traces show an undefined tool path, tighten the tool contract. If traces show data outside the intended context, tighten context assembly. If traces show an action without approval, tighten escalation.

Ship only when every step has a trace, permission check, and escalation path.