Keep agent workflows alive when an LLM provider goes down

The provider is a fragile dependency

Your agent workflow is calling an LLM provider, and the provider is down. ChatGPT, Claude, Gemini and Grok suffered service outages that affected thousands of users for several minutes. Problems with four popular chatbots began in the afternoon around 15:00 Spain peninsular time. El Mundo reported that Downdetector showed the incident extended to Groq, described as Elon Musk's AI, and Amazon Web Services, and that early hypotheses blamed sudden traffic that collapsed a Cloudflare route.

OpenAI's official status page said it was experiencing problems and severe errors in ChatGPT and Codex. 77% of reported ChatGPT problems were directly related to the chatbot, 10% to OpenAI's code assistant Codex, and 7% to the application. Claude's outage was reported to affect Mythos 5.1, Fable 5.1, Opus 5, Opus 4.8 and Opus 4.6. ChatGPT was the first service to return to normal operation.

For an agent workflow, the incident is a dependency failure. Model calls can fail, time out, or return degraded output while the rest of the workflow still expects a clean answer. The operator's job is to keep the system useful, not to pretend the provider is healthy.

Prepare the fallback before the outage

Choose a secondary provider that can handle peak load. Test it with the same prompts, tools, and guardrails. Keep API keys, rate limits, billing alerts, and a named owner in the same incident channel. If the workflow depends on a particular output format or tool behavior, verify the fallback supports the same contract. Fallbacks that only work for simple chat fail in production.

Map each workflow to a minimum viable response. Agents can run with a smaller model, return a cached answer, or stop and ask a human. Decide that before the incident, not during it.

Track the fallback provider's pricing in the incident channel. Fallbacks that burn budget can start another incident. Set a spend cap and an approval path for extended use.

The LLM dependency runbook keeps the workflow alive

The runbook is four steps: detect, degrade, route, notify.

• Detect the failure before users discover it. Instrument every LLM call with latency, error rate, token count, and output validity checks. Timeouts, server errors, empty completions, and output that does not match the expected format are signals. Alert when error rate crosses a threshold, and show provider health by model, region, and workflow.
• Degrade the workflow to a safe state. Stop long-running agent loops that depend on the failed provider. Queue inbound work, reduce concurrency, and disable optional extra steps. Keep core requests answering, even with fewer sources, shorter summaries, or a manual-review flag.
• Route traffic to a fallback provider or a cached response. The fallback must be pre-approved, tested, and mapped to the same input/output contract. If no fallback exists, serve a limited mode. Do not silently switch models. Users can notice changed tone, reasoning quality, or refusal behavior. Tell them what changed. The fallback response must match the original contract and show a visible note that the model changed.
• Notify the people who need to act. Send an internal incident message with the affected provider, models, workflows, and user impact. Send a user-facing status note when the change affects their output. Leave a timestamped incident channel, a customer status update, and a clear owner for each recovery task.

Recovery is not silence

After the provider returns, do not flip traffic back all at once. Verify the fallback and primary paths with the same test suite. Check token usage, latency, and output quality before increasing load. When the incident changes user-visible behavior, say so in the post-incident note. Operators should record what failed, what degraded, what routed, and what users were told.

Keep the runbook short enough that a tired engineer can use it late at night. Print the steps. Put the alert thresholds, fallback owner, and user message template on a single page. The next provider failure will not wait for a design review.

Use the incident to improve the system. If detection was slow, add a synthetic probe that calls the provider regularly. Manual fallback routing needs automation for known failure patterns. User complaints about changed output need a visible model badge or status line. Schedule the fix now, before the next outage.