LLM Provider Routing Checklist for Agent Startups

Match the primary to the riskiest task

Your agent will call an LLM for drafting, extraction, classification, tool calls, long-document review, and user-facing replies. Before launch, choose a primary provider, a fallback, and a routing rule for each call. At the time of publication, only Google's Gemini 1.5 could handle up to 2 million tokens, and it was available only as a Google cloud service.

OpenAI released ChatGPT in November 2022. Single-vendor stacks are easy to build and hard to leave. Startups ship fast on an API, but the stack inherits that vendor's limits, pricing, and availability. Pick the primary for the task with the highest failure cost, not the loudest brand. A misclassified ticket may be cheap; a wrong legal summary may cost more.

Choose the model for the task, the data path, and the failure mode. Choose the vendor for the ops team.

Weigh availability against context size. A cloud-only path adds a dependency for every call.

Anthropic released Claude 2.1 in November 2023 with a 200,000-token context window, equivalent to roughly 150,000 words. Large context windows matter for agents that must keep long histories, retrieved documents, and tool outputs in a prompt. If the primary cannot hold the working prompt, the window becomes the bottleneck. Size the prompt to the window before the demo.

Meta's Llama 3.2 1B model has a 128K-token context window, supports multilingual text-only input, and requires about 2 GB of GPU VRAM. Reserve a small local model for low-risk, high-volume steps where privacy or edge constraints matter. Keep the primary for the rest.

Build the fallback before the outage

Name the fallback before the production incident. Each primary task needs an alternate provider, a health check, and a timeout. The alternate should run the same prompt shape, not just a similar model. It must preserve the output contract: same fields, same tool-call format, same refusal behavior, and the same level of safety review.

Do not pick the fallback for price alone. Cheaper models fail in different ways. If a task touches customer data, payment decisions, or regulated content, the alternate must meet the same data-handling standard. If it cannot, the reroute becomes a risk transfer.

Test the reroute under load. The test is done when you can reroute a failing provider without dropping the agent's state. Keep the conversation, tool results, and pending actions in your own layer. Store prompts and output contracts in version control. When the primary changes, the alternate should not require a rewrite. When the alternate changes, the primary should not need a new prompt.

Route each call with four named fields

Build one routing rule with four named fields. Every agent step gets an entry that decides which model runs, when it can degrade, and who catches the failure.

• Context need: route by context first. Short classifiers run on small models. Long-document review may need a larger window. If the prompt grows past the model's working limit, split the work or move to a larger context provider.
• Latency budget: set the latency budget next. User-facing replies need fast responses. Background jobs wait. Base the budget on the user experience, not the model's marketing page.
• Cost ceiling: cap the cost after quality is set. Avoid forcing a step that can spend more onto a cheap model just because it is cheap. Avoid burning a premium model on a step that can spend less.
• Fallback provider: name the alternate for each entry. It must run the same prompt shape, preserve the output contract, and meet the same data-handling standard. The entry names the provider that catches the failure.

Review the routing settings when the agent changes. A new tool call, a longer document, or a stricter user requirement can move a task from a cheap option to a premium option. Make that move visible in the rule. Log which field forced each fallback decision so the operator sees the reroute path.