Routing and Failover
How the AI Routing Screen Is Organized
The Routing tab lays out an LLM Gateway's configuration as a decision chain that mirrors the order a request is actually evaluated in. A summary band at the top of the screen — Request → Conditional Gate → Target Selection → Failover → Provider — is clickable: selecting a step scrolls the screen to the matching section and highlights which stage the current configuration is in.
Below the decision chain the tab carries two more groups of settings that apply to every request rather than to one branch of it: a Client "model" field card, and the Connection & Delivery Settings group. The group has a Connection card (timeouts and the connection pool), a Streaming card (the Streaming switch, the client "stream" field and the stream limits), and the SSL/mTLS, outbound proxy and anonymous access cards. See Connection and Delivery Settings. The Client "model" field card holds its two settings directly on the card — there is no Configure drawer to open, and each selection is saved the moment you make it. A user with read-only access on the proxy sees the two selected values as a summary instead of the option cards.
Wherever a step resolves to a specific provider and model — a conditional route, a load-balancing pool entry, the primary itself, or a failover chain entry — an expandable row shows the same model metadata panel: the provider's endpoint, deployment type, and live health status; the model's context window, max output tokens, and per-million-token pricing (input, output, cached); and a note when the model is switched off. This standard panel appears at every selection point, so you can check a model's limits and pricing without leaving the screen.
How a Target Is Chosen
The gateway resolves the target for each request in tiers, stopping at the first one that yields a usable provider and model:
- Conditional routes — evaluated in order; the first match wins.
- Default target group — reached when no route matched: if a load-balancing pool is configured, its algorithm picks the active member; otherwise the single default provider and model is used. The AI Gateway list card follows the same rule: it shows the pool's first member when a pool is configured, otherwise the single default target.
- Failover chain (error path) — reached when the default target group is empty or unusable: the first usable chain entry is promoted to primary for this request, and the rest of the chain stays behind it as failover.
- No suitable provider — if no tier yields a target, the request is rejected with HTTP 503 and the message "No suitable LLM provider found for this request".
This means any one tier on its own is a complete configuration. A gateway with nothing but conditional routes works; so does one with nothing but a pool, or nothing but a failover chain. You only need to define the default provider and model if you want a fallback for requests that match no route — or if your routes and pool entries deliberately leave fields blank to inherit from it.
A candidate is skipped — and the next tier is tried — when its provider has been deleted or disabled, or when neither the entry nor the proxy-level default supplies a model. A matched conditional route whose target is unusable is skipped too, and evaluation continues with the next route.
What is not skipped: a model that is switched off on the provider, a model the calling consumer is not allowed to call, or a modality mismatch. Those return an error on the selected target rather than silently rerouting you somewhere else — they are configuration or authorization problems you need to see, not conditions to route around.
You can save the Routing tab with no target at all: no conditional route, no pool row, no failover row, and an empty default provider and model. The tab shows a warning. The proxy can be saved and deployed, but every request fails with HTTP 503 (POLICY_ROUTING_AI_NO_SUITABLE_PROVIDER) until you add a target. Deploy is not blocked.
A definition that fills a target field but produces no target is still rejected on save. For example, a provider is selected but the model is empty and there is no list entry.
Conditional Routing
A Conditional Route overrides the primary provider and model for requests that match a rule, a semantic pattern, or both — evaluated before the primary or the load-balancing pool. Routes are evaluated in order and the first match wins; if a matching route leaves its provider or model blank, it inherits the primary's. If no route matches, routing falls through to the load-balancing pool or the single primary as usual.
| Field | Description |
|---|---|
| Route Name | A label for the route, shown in the routing screen's summary table (for example, "Code questions → DeepSeek"). |
| Condition | A rule built from Apinizer's condition editor. Left empty, the route always matches on the rule side. |
| Target type | Single target (default) or Target pool (load balancing). See Target Pool Routes. |
| Target Provider · Model | Shown for Single target. The provider connection and model this route sends matching requests to. Leave either blank to inherit the primary's value. |
Target Pool Routes
Select Target pool (load balancing) as the Target type when the same model runs on several servers. The route then distributes the matching requests across its targets with its own algorithm. A single route needs no second proxy in front of the servers.
Example: the model qwen3-8b runs on the servers a1, a2 and a3.
- Create a route with the rule
model = qwen3-8b. Select Target pool (load balancing) and add one target for each server. - Create a second route with the rule
model = deepseek-r1. Select Target pool (load balancing) and add its targets.
Each route has its own pool and its own balance state. Round Robin on one route does not affect another route.
| Field | Description |
|---|---|
| Algorithm | The same seven algorithms as the Default Target Group: Round Robin, Weighted, Least Latency, Random, LRU, Least Cost, and Prefix Affinity. See Load Balancing Across Equal-Priority Primaries. |
| Affinity Key Header and Bounded Load Factor | Shown for Prefix Affinity. Each route has its own values. Defaults: session_id and 1.25. |
| Display Name | The name of the target in the route list and in the pool summary. |
| LLM Provider | Required for every target. |
| Model ID | Required for every target. A target does not inherit the model of the default target. |
| Weight | Shown for Weighted only. A higher weight receives more traffic. |
Click Add Target to add a row. Saving is rejected when a target has no provider or no model.
The route list and the route summary show Target pool · algorithm · N targets for a pool route. A route that was saved before this option existed opens as Single target. No migration is needed.
Whether the other targets of the pool are tried is set by Try the other pool targets in the Error path settings. See Try the Other Pool Targets.
Semantic Matching
In addition to (or instead of) a rule, a route can match by meaning: define a list of example utterances that represent the route's intent, and Apinizer embeds the incoming prompt and compares it against them by cosine similarity.
- Similarity Threshold — the minimum cosine similarity an utterance must reach for the route to match. Default
0.75. - Embedding Provider and Model — configured once per proxy and shared by every route's utterance list; the inbound prompt is embedded once per request, not once per route.
- Rule + Semantic Combined — when a route defines both a condition and utterances, both must match (AND): the rule is checked first, and semantic evaluation only runs once the rule has already passed.
- No Route Matches — if none of the conditional routes match, the request falls through to the load-balancing pool or the single primary; this is expected behavior, not an error.
- No Embedding Provider Configured — a route cannot carry utterances unless the proxy also has a semantic embedding provider selected. Saving is rejected with "A conditional route uses semantic utterances, so the Routing tab must also have an embedding provider." Earlier versions saved such a proxy successfully and then evaluated every semantic route on its rule alone, with no signal on screen. Once a provider is selected, request-time behavior is unchanged and deliberately fail-open: if the provider is unreachable, the route is treated as "not matched" and routing continues.
Environment Variables in Routing Text
The utterances that drive semantic matching, and the header name the affinity strategies read their key from, both accept an environment variable reference in ${VARIABLE_NAME} form, so one routing configuration can carry per-environment wording or header naming.
Utterance resolution happens once, when the route's embeddings are first built and cached — not per request. That is why only ${...} environment variables are accepted here and #{...} context variables are not: a per-request value would be embedded on the first request and those same vectors would then be served to every later one.
Route Match Methods and the LLM Classifier
In the route dialog, Match Method has three options: Rule-based, Semantic, and Classifier. For Semantic and Classifier, the checkbox Also require rule conditions (AND) is shown. Select it to require the rule conditions in addition to the semantic or classifier match.
The route list shows one match type badge per route: RULE, SEMANTIC, SEMANTIC + RULE, CLASSIFIER, or CLASSIFIER + RULE.
The Hybrid button no longer exists. A route that was saved as Hybrid opens as Semantic with Also require rule conditions (AND) selected. The saved configuration does not change.
Classifier Routing
Classifier routing selects the target by how well it suits the request, not only by cost. For example, you can send coding prompts to model A, finance prompts to model B, and legal prompts to model C.
A small LLM, the classifier, reads the prompt and answers with one label. The route with that label is selected. Each classifier route has two fields:
| Field | Description |
|---|---|
| Label | A short name for the type of request. Use lowercase letters, digits, _, and -. The maximum length is 64 characters. The label must start and end with a letter or digit. The label none is reserved. |
| Description | Required. The maximum length is 500 characters. The classifier model decides by this text, so describe the kind of request that belongs to the route. ${env} environment variables are resolved; #{...} context variables are not. |
Several routes can use the same label. In this case, the first route in the list whose rule conditions also match wins. Use this to split one label by request size. For example:
- Route 1: label
code, with the condition Estimated Request Token Count greater than 2000. Target: a large model. - Route 2: label
code, without a condition. Target: a small model.
The LLM Classifier Settings
Open the Conditional Routes drawer to configure the LLM classifier box. This box is shared by all classifier routes of the proxy.
| Field | Description |
|---|---|
| Classifier Provider | The provider that runs the classifier. The list is grouped. See Which Providers Can Classify. |
| Classifier Model | The model of that provider. |
| Model behaviour | Standard (default) or Reasoning. See Model Behaviour. Not shown for decision models. |
| Reasoning budget (tokens) | Shown only for Reasoning. Range 256 to 8192. Default 1024. |
| Timeout (ms) | The maximum time to wait for the classifier. Standard: default 1000, range 100 to 10000. Reasoning: default 15000, up to 60000. |
| Max input characters | The maximum length of the user message sent to the classifier. Longer messages are truncated. Default 2000. Range 200 to 20000. |
| Additional instructions | Optional. Extra text for the classifier, for example "Questions about invoices are finance." ${env} environment variables are resolved; #{...} context variables are not. The user prompt is never resolved. |
The box also shows a preview of the label list that is sent to the classifier.
Which Providers Can Classify
The Classifier Provider list shows every provider of the project. The list has these groups:
| Group | Providers |
|---|---|
| Chat models | OpenAI-compatible chat providers and Anthropic. |
| Decision models | TypeSafe Jev. |
| Cannot be a route classifier | Providers that are listed but disabled. The reason is shown under the name. |
These providers can classify:
- OpenAI-compatible chat providers: OpenAI, Azure OpenAI, DeepSeek, Moonshot (Kimi), Zhipu, Mistral, Groq, Ollama, vLLM, xAI, the Google AI Studio OpenAI endpoint, Together, Fireworks, Cerebras, Hugging Face, OpenRouter, Perplexity, and NVIDIA NIM. A custom provider can classify when its Adapter family is
OPENAI_COMPATIBLEorVLLM. - Anthropic, through the native Messages API.
- TypeSafe Jev. See TypeSafe Jev as Classifier.
These providers cannot classify: AWS Bedrock, Google Vertex AI, Cohere, the native Gemini wire format, embedding-only providers, TEI guard classifiers, and custom providers without an adapter family. For a custom provider, set the adapter family.
The classifier call uses the credentials stored on the provider. Set the provider credentials to Provided by AI gateway. A provider set to Provided by client has no client key for the classifier call, and the provider answers with 401 or 403.
TypeSafe Jev and TEI make decisions only. You cannot use them as a routing target, a pool member, a failover target, or an embedding provider. They are not listed in those lists. Apinizer rejects a save that refers to one of them.
Model Behaviour: Standard or Reasoning
Model behaviour tells Apinizer how the classifier model answers.
| Behaviour | Use it for | What Apinizer sends |
|---|---|---|
| Standard (default) | Small instruct models that answer at once. | Temperature 0 and a small output limit. For Ollama, reasoning_effort: "none", so that thinking models such as qwen3 and deepseek-r1 answer directly. |
| Reasoning | Models that think before they answer: DeepSeek Reasoner, Kimi thinking, Claude extended thinking, OpenAI o-series, qwen3 with thinking on. | No temperature. The Reasoning budget (tokens) is the output limit for thinking and answer together. For Ollama, thinking stays on. |
A value that you set always wins over the default of the behaviour.
For Anthropic, Standard sends a normal Messages call. Reasoning turns on extended thinking. Newer models use adaptive thinking. Older models use fixed thinking with a token budget. Apinizer tries one shape, then remembers the shape that works for the provider and model.
Some models accept only their default temperature, for example Kimi and the OpenAI o-series and gpt-5 models. Apinizer retries once without the temperature and remembers this for the model.
If the model uses its whole reasoning budget before it answers, the outcome is TRUNCATED and no classifier route matches. Raise Reasoning budget (tokens) or select Standard with a non-thinking model.
The classifier runs before every request that reaches a classifier route. A reasoning model adds several seconds and its token cost to each of these requests. Use Test routing to see the latency and the thinking tokens before you deploy.
If the provider refuses to answer for safety reasons, the classifier treats this as no match.
TypeSafe Jev as Classifier
TypeSafe Jev is a structured decision model. It does not generate text. Apinizer sends the route labels and their descriptions as the choices, with a "none of these" option, and the user prompt as the state. Jev returns one choice with a confidence between 0 and 1.
- Model behaviour, Reasoning budget (tokens), and the output limit do not apply. The screen shows an information note instead.
- Test routing shows the Confidence value. The routing trace shows it too.
- Create the provider with the type TypeSafe Jev. See LLM Providers.
Choosing the Classifier Model
Use a small, fast model. The classifier must answer with one word.
- Small instruct models. Select Standard. Examples for Ollama: qwen2.5:1.5b-instruct, llama3.2:3b, and granite3.3:2b.
- Reasoning ("thinking") models. Select Reasoning and set a Reasoning budget (tokens) and a longer Timeout (ms). With Standard, a thinking model writes its reasoning into the short answer limit. The answer is empty and the outcome is
TRUNCATED(Output limit reached). With a short timeout, the same problem can show asTIMEOUT.- Ollama: In Standard, Apinizer turns thinking off. A model such as qwen3 then answers directly.
- vLLM and other OpenAI-compatible servers: For Standard, serve a non-thinking variant of the model, or turn thinking off on the server.
- OpenAI and Azure OpenAI: Apinizer sends
max_completion_tokensto OpenAI. Azure OpenAI gets it fromapi-version2024-09-01 on, andmax_tokenson older versions.
- Timeout. Use Test routing to see the classifier latency, then set Timeout (ms) to about twice that value. In our tests, local 8B to 14B models on Ollama answered in 0.25 to 0.7 seconds after warm-up. The first call after a model loads is slower.
- Ambiguous requests. If a type of request goes to the wrong route, describe it in Additional instructions. Example: "Tax filings, VAT refunds and invoices are finance, even when they mention documents or regulations." An instruction can also change the answers for other requests, especially on a small model. After you change it, test again with sample prompts for every route.
Accepted Answer Format
The classifier answer must be one of the labels, or none. Apinizer accepts these variations:
- Any letter case, surrounding quotes, backticks, bold markers, and a final period.
- A
category:orlabel:prefix. - A leading
<think>…</think>block. - A JSON object with one member,
labelorcategory. Example:{"label": "code"}. - Only the first non-empty line is read.
Apinizer does not guess a label from a sentence. An answer such as "code - the user asks about Python" gives UNKNOWN_LABEL, and no classifier route matches.
How Routes Are Evaluated
- Apinizer evaluates the routes from top to bottom.
- For each route, it checks the rule conditions first. Then it checks the semantic or classifier match. Both must match (AND) when the rule conditions are defined.
- The classifier is called at most once per request. It is called only when a classifier route is reached.
- Only the labels of classifier routes whose rule conditions are met are offered to the classifier.
- The classifier receives the last user message, truncated to Max input characters. It answers with one label or
none. In Standard behaviour, the temperature is 0 and the answer is limited to 32 output tokens. In Reasoning behaviour, the limit is the reasoning budget. - The classifier runs only in the routing phase, after authentication and PII masking. It receives the masked prompt. An unauthenticated request does not cause a classifier call.
If the classifier fails, the request is not rejected. This is fail-open. The classifier routes do not match in these cases:
- The classifier returns an error.
- The classifier times out.
- The model reaches its output limit before it answers (
TRUNCATED). - The provider refuses to answer for safety reasons.
- The classifier answers with a label that no route uses.
- No classifier provider is configured.
- The classifier model is switched off on its provider, or its ID is ambiguous. Apinizer makes no call and records the outcome
NO_PROVIDER. - The classifier failed recently. After a timeout, an error, or
TRUNCATED, Apinizer skips classifier calls for that proxy for 15 seconds. The outcome isBACKOFF. Redeploying the proxy clears the back-off.
The request then continues down the list. If no route matches, the request goes to the Default Target Group.
The proxy protection limit of the token rate limit, the semantic cache lookup, and the loop guard run in routing. They run once the provider and model are final and the model access list has let the caller through. When a classifier route wins, these rules use the model of the selected route. No rule runs with a guessed model.
The request body size limit (413) and the stream compatibility checks (400) run before the token limit (429) and before a cache answer. A very long prompt gets 413, not 429.
A provider whose Chat Completion Path is /messages streams Anthropic frames. If its provider type is not Anthropic (for example vLLM), the gateway cannot read the answer text from the stream. When the API has a response policy that must read the answer (PII Mask on the response, DLP Guard or Topic Guard in streaming buffer mode), a streaming request ("stream":true) is rejected with HTTP 400 and error code ERR-312 before any provider call. A failover entry of this kind is skipped for a streaming request. If every entry is skipped this way, the client gets the same 400. Requests with "stream":false and streams that the gateway can read do not change.
A user can try to steer the classifier choice with the text of the prompt. The choice is always limited to the routes that you defined. Quotas and the Limit Plan are not affected by the chosen route.
In the first stage of the routing screen, the panel shows two information messages: Classifier: provider · model · timeout ms and Embedding: provider · model.
Routing by Request Size
You can route by the size of the request. In the rule conditions of a route, select the context value Estimated Request Token Count. Set the data type to Numeric and select an operator, for example greater than.
The value is an estimate of the tokens in the request body. Apinizer uses the body that goes to the backend. If there is none, it uses the body from the client.
Test Routing
Use Test routing to check which route a sample prompt selects. The button is shown only on the Conditional Routes panel.
- Click Test routing.
- Select a deployed environment.
- Enter a sample prompt.
- Click Run test.
The worker evaluates the saved routing. The target model is not called. Nothing is saved.
The test calls the classifier and embedding providers. These calls can have a cost.
The result shows:
- The estimated request tokens.
- The classifier label, outcome, latency, tokens, and cost. For a reasoning model, the tokens include the thinking tokens. If the provider does not report thinking tokens (for example Ollama or vLLM), Apinizer estimates them from the thinking text the model returns. For TypeSafe Jev, the result also shows the Confidence.
- A table with one row for each route: the rule, the semantic score, the classifier answer, and the result. The result is Selected, No match, Target unusable, or Not evaluated. Hold the pointer over Selected to see the target provider and model.
- For a pool route, the result is Selected → Target pool · algorithm · N targets. The gateway picks the target when the request arrives, so the test does not name a pool target.
- The message "No route matched — the request goes to the Default Target Group" when no route matches.
Rules other than Estimated Request Token Count need a real request. The test assumes that these rules match and shows them as RUNTIME_ONLY. A NOT over a rule that cannot be evaluated without a real request is shown as INDETERMINATE.
The test never resolves a value that contains ${...} (environment variable) or #{...} (context variable). An environment variable can be secret, and the test result must not expose it. A rule with such a value is shown as RUNTIME_ONLY and assumed to match. The classifier instructions, the route descriptions and the semantic example utterances also stay as written in the test. On live traffic, ${...} is resolved as usual.
The existence and emptiness operators (exists, does not exist, empty, not empty, and exists but empty) do not read a second value. The test decides them from the sample request.
When a comparison would fail on a live request, for example a numeric comparison with a value that is not a number, the test shows EVALUATION_ERROR and treats the route as not matched. AND, OR and NOT do not turn this into a match.
The sample model field accepts at most 256 characters.
The test ignores the 15-second back-off. It always calls the classifier. If embedding fails, the dialog shows a warning with the error type only.
Observability
The traffic log has these sparse fields for the classifier:
| Field | Description |
|---|---|
aircl | The label that the classifier returned. |
aircs | The outcome: MATCHED, NONE, UNKNOWN_LABEL, TIMEOUT, ERROR, TRUNCATED (shown as Output limit reached), BACKOFF (shown as Skipped (back-off)), NO_PROVIDER, or NO_INPUT. |
aircms | The classifier latency in milliseconds. |
aircti | The classifier input tokens. |
aircto | The classifier output tokens. |
airccm | The classifier cost in micro-USD. |
The AI traffic drawer and the traffic detail dialog show a Routing Classifier block with the classifier decision (outcome and label), Decision time in milliseconds, and Classifier cost.
The trace drawer shows a Route classifier section. When AI trace is on, it also shows a Routing decision panel with the result for each route.
If you write traffic logs to a JDBC database (Oracle, MySQL, PostgreSQL, or MSSQL), run the ALTER script for your database before you upgrade. Otherwise, traffic log inserts to this database fail.
The ALTER scripts for Oracle, MySQL, PostgreSQL, and SQL Server are on the 2026.10.0 — Log Table ALTER Scripts page. They add these columns: ai_route_cls_label, ai_route_cls_outcome, ai_route_cls_latency_ms, ai_route_cls_token_in, ai_route_cls_token_out, and ai_route_cls_cost_micro.
For Elasticsearch, no action is needed. A startup migration updates the traffic index template automatically, and the index is rolled over.
Client "model" Field
Two independent settings decide what the gateway does with the model field the client sends in the request body. Whichever value they resolve to becomes the effective model for that request — the same model that rate limiting, semantic cache lookups, cost calculation, and traffic logs use, regardless of what the client originally sent.
You will find them on the Routing tab, in the Client "model" field card between the failover section and the connection settings. Selecting an option saves it immediately; if the save fails, the selection returns to its previous value. The card is not shown on an A2A agent's routing tab, where the client never sends a model field of its own.