Keep Agent State Across Restarts: Checkpoint and Test Recovery

A restart is a failure you can schedule

The hermes-agent gateway described in issue 7181 can leave a user seeing 'Still working... (iteration 43/90)' and then receiving no error, summary, or resume path after a restart. The same gateway discards the active task when it restarts during a task. In production, restarts are routine. The operator's job is to make the agent resume pending work without repeating completed steps.

LangChain's agent development lifecycle has four stages: Build, Test, Deploy, and Monitor. In that lifecycle, durable execution means an agent checkpoints progress and resumes instead of losing work when something fails. LangGraph offers a runtime for agentic systems, supporting branching, looping, pausing, resuming, and state persistence over time. AgentRewind, a runtime recovery framework, saves matched checkpoints of both agent context and the controlled environment, so an agent can roll back and continue from that point. Long tasks span tool calls, external writes, and decisions that must survive the process boundary.

Durable agent state includes the task, the context, the environment, and the side effects already committed. If any one is missing, the agent may resume with incomplete context and incorrect side effects.

Checkpoint the task, not just the transcript

• Define task state. List every field that must survive a restart: goal, current step, tool outputs, pending actions, user constraints, budget, deadline, and idempotency keys. Done looks like a schema you can serialize, diff, and restore without guessing.
• Checkpoint context and environment. Save the model context, tool state, external resources, and the exact environment needed to continue. The snapshot must be restorable without reconstructing external state. The common mistake is saving the transcript and assuming external state is saved. A tool call, a database row, a file, or a message can still be lost.
• Persist pending steps. Store unfinished actions outside process memory with status, retry rules, and a way to mark them complete. You know it is done when a queue of next actions survives a kill.
• Test resume after kill or restart. Stop the process at known checkpoints, then start it again and assert the agent resumes from the saved point. Use long tasks where partial progress matters. MettleBench is a benchmark that measures task completion and partial progress in long-horizon engineering work. The test should fail if the agent restarts from the beginning.
• Verify no duplicate or skipped work. After resume, compare side effects: no double sends, no missing files, no repeated payments, and no skipped validation. A passing check produces a recovery report that lists the checkpoint, the resumed step, and the external effects checked.

Prove recovery before you trust it

Production agent recovery is a test you can automate. Build a repeatable scenario that covers the restart you must survive: scheduled deploy, crash, memory exhaustion, config reload, and node drain. For each scenario, record the checkpoint before the stop, the process signal, the restart delay, and the first action after resume. Then compare the result against the expected task state.

Do not test only the happy path. A restart can land between a model call and a tool call, between a tool call and a write, or between a write and a confirmation. Your checkpoint must make that boundary explicit. If the agent cannot say which side of the boundary it is on, it cannot resume safely.

Use a benchmark when the task is long enough to matter, but your own failure test must use your tools, your data, and your restart trigger. It should fail clearly when the agent repeats a side effect or skips a required step.

Keep the recovery path visible. After restart, the agent should either resume, report the saved checkpoint, or ask for a safe continuation.

Make the test part of the deploy gate. A failed checkpoint test stops the deploy. A recovery report that shows duplicate work stops the release. An agent that cannot prove where it left off does not get production traffic.