Ana içeriğe geç

Routing and Failover

How the AI Routing Screen Is Organized​

The Routing tab lays out an LLM Gateway's configuration as a decision chain that mirrors the order a request is actually evaluated in. A summary band at the top of the screen — Request → Conditional Gate → Target Selection → Failover → Provider — is clickable: selecting a step scrolls the screen to the matching section and highlights which stage the current configuration is in.

Below the decision chain the tab carries two more groups of settings that apply to every request rather than to one branch of it: a Client "model" field card, and the Connection & Delivery Settings group. The group has a Connection card (timeouts and the connection pool), a Streaming card (the Streaming switch, the client "stream" field and the stream limits), and the SSL/mTLS, outbound proxy and anonymous access cards. See Connection and Delivery Settings. The Client "model" field card holds its two settings directly on the card — there is no Configure drawer to open, and each selection is saved the moment you make it. A user with read-only access on the proxy sees the two selected values as a summary instead of the option cards.

Wherever a step resolves to a specific provider and model — a conditional route, a load-balancing pool entry, the primary itself, or a failover chain entry — an expandable row shows the same model metadata panel: the provider's endpoint, deployment type, and live health status; the model's context window, max output tokens, and per-million-token pricing (input, output, cached); and a note when the model is switched off. This standard panel appears at every selection point, so you can check a model's limits and pricing without leaving the screen.

How a Target Is Chosen​

The gateway resolves the target for each request in tiers, stopping at the first one that yields a usable provider and model:

  1. Conditional routes — evaluated in order; the first match wins.
  2. Default target group — reached when no route matched: if a load-balancing pool is configured, its algorithm picks the active member; otherwise the single default provider and model is used. The AI Gateway list card follows the same rule: it shows the pool's first member when a pool is configured, otherwise the single default target.
  3. Failover chain (error path) — reached when the default target group is empty or unusable: the first usable chain entry is promoted to primary for this request, and the rest of the chain stays behind it as failover.
  4. No suitable provider — if no tier yields a target, the request is rejected with HTTP 503 and the message "No suitable LLM provider found for this request".

This means any one tier on its own is a complete configuration. A gateway with nothing but conditional routes works; so does one with nothing but a pool, or nothing but a failover chain. You only need to define the default provider and model if you want a fallback for requests that match no route — or if your routes and pool entries deliberately leave fields blank to inherit from it.

What counts as "unusable"

A candidate is skipped — and the next tier is tried — when its provider has been deleted or disabled, or when neither the entry nor the proxy-level default supplies a model. A matched conditional route whose target is unusable is skipped too, and evaluation continues with the next route.

What is not skipped: a model that is switched off on the provider, a model the calling consumer is not allowed to call, or a modality mismatch. Those return an error on the selected target rather than silently rerouting you somewhere else — they are configuration or authorization problems you need to see, not conditions to route around.

Routing with no target

You can save the Routing tab with no target at all: no conditional route, no pool row, no failover row, and an empty default provider and model. The tab shows a warning. The proxy can be saved and deployed, but every request fails with HTTP 503 (POLICY_ROUTING_AI_NO_SUITABLE_PROVIDER) until you add a target. Deploy is not blocked.

A definition that fills a target field but produces no target is still rejected on save. For example, a provider is selected but the model is empty and there is no list entry.

Conditional Routing​

A Conditional Route overrides the primary provider and model for requests that match a rule, a semantic pattern, or both — evaluated before the primary or the load-balancing pool. Routes are evaluated in order and the first match wins; if a matching route leaves its provider or model blank, it inherits the primary's. If no route matches, routing falls through to the load-balancing pool or the single primary as usual.

FieldDescription
Route NameA label for the route, shown in the routing screen's summary table (for example, "Code questions → DeepSeek").
ConditionA rule built from Apinizer's condition editor. Left empty, the route always matches on the rule side.
Target typeSingle target (default) or Target pool (load balancing). See Target Pool Routes.
Target Provider · ModelShown for Single target. The provider connection and model this route sends matching requests to. Leave either blank to inherit the primary's value.

Target Pool Routes​

Select Target pool (load balancing) as the Target type when the same model runs on several servers. The route then distributes the matching requests across its targets with its own algorithm. A single route needs no second proxy in front of the servers.

Example: the model qwen3-8b runs on the servers a1, a2 and a3.

  1. Create a route with the rule model = qwen3-8b. Select Target pool (load balancing) and add one target for each server.
  2. Create a second route with the rule model = deepseek-r1. Select Target pool (load balancing) and add its targets.

Each route has its own pool and its own balance state. Round Robin on one route does not affect another route.

FieldDescription
AlgorithmThe same seven algorithms as the Default Target Group: Round Robin, Weighted, Least Latency, Random, LRU, Least Cost, and Prefix Affinity. See Load Balancing Across Equal-Priority Primaries.
Affinity Key Header and Bounded Load FactorShown for Prefix Affinity. Each route has its own values. Defaults: session_id and 1.25.
Display NameThe name of the target in the route list and in the pool summary.
LLM ProviderRequired for every target.
Model IDRequired for every target. A target does not inherit the model of the default target.
WeightShown for Weighted only. A higher weight receives more traffic.

Click Add Target to add a row. Saving is rejected when a target has no provider or no model.

The route list and the route summary show Target pool · algorithm · N targets for a pool route. A route that was saved before this option existed opens as Single target. No migration is needed.

When a pool target fails

Whether the other targets of the pool are tried is set by Try the other pool targets in the Error path settings. See Try the Other Pool Targets.

Semantic Matching​

In addition to (or instead of) a rule, a route can match by meaning: define a list of example utterances that represent the route's intent, and Apinizer embeds the incoming prompt and compares it against them by cosine similarity.

  • Similarity Threshold — the minimum cosine similarity an utterance must reach for the route to match. Default 0.75.
  • Embedding Provider and Model — configured once per proxy and shared by every route's utterance list; the inbound prompt is embedded once per request, not once per route.
  • Rule + Semantic Combined — when a route defines both a condition and utterances, both must match (AND): the rule is checked first, and semantic evaluation only runs once the rule has already passed.
  • No Route Matches — if none of the conditional routes match, the request falls through to the load-balancing pool or the single primary; this is expected behavior, not an error.
  • No Embedding Provider Configured — a route cannot carry utterances unless the proxy also has a semantic embedding provider selected. Saving is rejected with "A conditional route uses semantic utterances, so the Routing tab must also have an embedding provider." Earlier versions saved such a proxy successfully and then evaluated every semantic route on its rule alone, with no signal on screen. Once a provider is selected, request-time behavior is unchanged and deliberately fail-open: if the provider is unreachable, the route is treated as "not matched" and routing continues.

Environment Variables in Routing Text​

The utterances that drive semantic matching, and the header name the affinity strategies read their key from, both accept an environment variable reference in ${VARIABLE_NAME} form, so one routing configuration can carry per-environment wording or header naming.

Note

Utterance resolution happens once, when the route's embeddings are first built and cached — not per request. That is why only ${...} environment variables are accepted here and #{...} context variables are not: a per-request value would be embedded on the first request and those same vectors would then be served to every later one.

Route Match Methods and the LLM Classifier​

In the route dialog, Match Method has three options: Rule-based, Semantic, and Classifier. For Semantic and Classifier, the checkbox Also require rule conditions (AND) is shown. Select it to require the rule conditions in addition to the semantic or classifier match.

The route list shows one match type badge per route: RULE, SEMANTIC, SEMANTIC + RULE, CLASSIFIER, or CLASSIFIER + RULE.

Routes created with the former Hybrid button

The Hybrid button no longer exists. A route that was saved as Hybrid opens as Semantic with Also require rule conditions (AND) selected. The saved configuration does not change.

Classifier Routing​

Classifier routing selects the target by how well it suits the request, not only by cost. For example, you can send coding prompts to model A, finance prompts to model B, and legal prompts to model C.

A small LLM, the classifier, reads the prompt and answers with one label. The route with that label is selected. Each classifier route has two fields:

FieldDescription
LabelA short name for the type of request. Use lowercase letters, digits, _, and -. The maximum length is 64 characters. The label must start and end with a letter or digit. The label none is reserved.
DescriptionRequired. The maximum length is 500 characters. The classifier model decides by this text, so describe the kind of request that belongs to the route. ${env} environment variables are resolved; #{...} context variables are not.

Several routes can use the same label. In this case, the first route in the list whose rule conditions also match wins. Use this to split one label by request size. For example:

  1. Route 1: label code, with the condition Estimated Request Token Count greater than 2000. Target: a large model.
  2. Route 2: label code, without a condition. Target: a small model.

The LLM Classifier Settings​

Open the Conditional Routes drawer to configure the LLM classifier box. This box is shared by all classifier routes of the proxy.

FieldDescription
Classifier ProviderThe provider that runs the classifier. The list is grouped. See Which Providers Can Classify.
Classifier ModelThe model of that provider.
Model behaviourStandard (default) or Reasoning. See Model Behaviour. Not shown for decision models.
Reasoning budget (tokens)Shown only for Reasoning. Range 256 to 8192. Default 1024.
Timeout (ms)The maximum time to wait for the classifier. Standard: default 1000, range 100 to 10000. Reasoning: default 15000, up to 60000.
Max input charactersThe maximum length of the user message sent to the classifier. Longer messages are truncated. Default 2000. Range 200 to 20000.
Additional instructionsOptional. Extra text for the classifier, for example "Questions about invoices are finance." ${env} environment variables are resolved; #{...} context variables are not. The user prompt is never resolved.

The box also shows a preview of the label list that is sent to the classifier.

Which Providers Can Classify​

The Classifier Provider list shows every provider of the project. The list has these groups:

GroupProviders
Chat modelsOpenAI-compatible chat providers and Anthropic.
Decision modelsTypeSafe Jev.
Cannot be a route classifierProviders that are listed but disabled. The reason is shown under the name.

These providers can classify:

  • OpenAI-compatible chat providers: OpenAI, Azure OpenAI, DeepSeek, Moonshot (Kimi), Zhipu, Mistral, Groq, Ollama, vLLM, xAI, the Google AI Studio OpenAI endpoint, Together, Fireworks, Cerebras, Hugging Face, OpenRouter, Perplexity, and NVIDIA NIM. A custom provider can classify when its Adapter family is OPENAI_COMPATIBLE or VLLM.
  • Anthropic, through the native Messages API.
  • TypeSafe Jev. See TypeSafe Jev as Classifier.

These providers cannot classify: AWS Bedrock, Google Vertex AI, Cohere, the native Gemini wire format, embedding-only providers, TEI guard classifiers, and custom providers without an adapter family. For a custom provider, set the adapter family.

The classifier provider must use credentials held by the AI gateway

The classifier call uses the credentials stored on the provider. Set the provider credentials to Provided by AI gateway. A provider set to Provided by client has no client key for the classifier call, and the provider answers with 401 or 403.

Decision-only providers are not routing targets

TypeSafe Jev and TEI make decisions only. You cannot use them as a routing target, a pool member, a failover target, or an embedding provider. They are not listed in those lists. Apinizer rejects a save that refers to one of them.

Model Behaviour: Standard or Reasoning​

Model behaviour tells Apinizer how the classifier model answers.

BehaviourUse it forWhat Apinizer sends
Standard (default)Small instruct models that answer at once.Temperature 0 and a small output limit. For Ollama, reasoning_effort: "none", so that thinking models such as qwen3 and deepseek-r1 answer directly.
ReasoningModels that think before they answer: DeepSeek Reasoner, Kimi thinking, Claude extended thinking, OpenAI o-series, qwen3 with thinking on.No temperature. The Reasoning budget (tokens) is the output limit for thinking and answer together. For Ollama, thinking stays on.

A value that you set always wins over the default of the behaviour.

For Anthropic, Standard sends a normal Messages call. Reasoning turns on extended thinking. Newer models use adaptive thinking. Older models use fixed thinking with a token budget. Apinizer tries one shape, then remembers the shape that works for the provider and model.

Some models accept only their default temperature, for example Kimi and the OpenAI o-series and gpt-5 models. Apinizer retries once without the temperature and remembers this for the model.

If the model uses its whole reasoning budget before it answers, the outcome is TRUNCATED and no classifier route matches. Raise Reasoning budget (tokens) or select Standard with a non-thinking model.

A reasoning classifier adds time and cost to every classified request

The classifier runs before every request that reaches a classifier route. A reasoning model adds several seconds and its token cost to each of these requests. Use Test routing to see the latency and the thinking tokens before you deploy.

If the provider refuses to answer for safety reasons, the classifier treats this as no match.

TypeSafe Jev as Classifier​

TypeSafe Jev is a structured decision model. It does not generate text. Apinizer sends the route labels and their descriptions as the choices, with a "none of these" option, and the user prompt as the state. Jev returns one choice with a confidence between 0 and 1.

  • Model behaviour, Reasoning budget (tokens), and the output limit do not apply. The screen shows an information note instead.
  • Test routing shows the Confidence value. The routing trace shows it too.
  • Create the provider with the type TypeSafe Jev. See LLM Providers.

Choosing the Classifier Model​

Use a small, fast model. The classifier must answer with one word.

  • Small instruct models. Select Standard. Examples for Ollama: qwen2.5:1.5b-instruct, llama3.2:3b, and granite3.3:2b.
  • Reasoning ("thinking") models. Select Reasoning and set a Reasoning budget (tokens) and a longer Timeout (ms). With Standard, a thinking model writes its reasoning into the short answer limit. The answer is empty and the outcome is TRUNCATED (Output limit reached). With a short timeout, the same problem can show as TIMEOUT.
    • Ollama: In Standard, Apinizer turns thinking off. A model such as qwen3 then answers directly.
    • vLLM and other OpenAI-compatible servers: For Standard, serve a non-thinking variant of the model, or turn thinking off on the server.
    • OpenAI and Azure OpenAI: Apinizer sends max_completion_tokens to OpenAI. Azure OpenAI gets it from api-version 2024-09-01 on, and max_tokens on older versions.
  • Timeout. Use Test routing to see the classifier latency, then set Timeout (ms) to about twice that value. In our tests, local 8B to 14B models on Ollama answered in 0.25 to 0.7 seconds after warm-up. The first call after a model loads is slower.
  • Ambiguous requests. If a type of request goes to the wrong route, describe it in Additional instructions. Example: "Tax filings, VAT refunds and invoices are finance, even when they mention documents or regulations." An instruction can also change the answers for other requests, especially on a small model. After you change it, test again with sample prompts for every route.

Accepted Answer Format​

The classifier answer must be one of the labels, or none. Apinizer accepts these variations:

  • Any letter case, surrounding quotes, backticks, bold markers, and a final period.
  • A category: or label: prefix.
  • A leading <think>…</think> block.
  • A JSON object with one member, label or category. Example: {"label": "code"}.
  • Only the first non-empty line is read.

Apinizer does not guess a label from a sentence. An answer such as "code - the user asks about Python" gives UNKNOWN_LABEL, and no classifier route matches.

How Routes Are Evaluated​

  1. Apinizer evaluates the routes from top to bottom.
  2. For each route, it checks the rule conditions first. Then it checks the semantic or classifier match. Both must match (AND) when the rule conditions are defined.
  3. The classifier is called at most once per request. It is called only when a classifier route is reached.
  4. Only the labels of classifier routes whose rule conditions are met are offered to the classifier.
  5. The classifier receives the last user message, truncated to Max input characters. It answers with one label or none. In Standard behaviour, the temperature is 0 and the answer is limited to 32 output tokens. In Reasoning behaviour, the limit is the reasoning budget.
  6. The classifier runs only in the routing phase, after authentication and PII masking. It receives the masked prompt. An unauthenticated request does not cause a classifier call.

If the classifier fails, the request is not rejected. This is fail-open. The classifier routes do not match in these cases:

  • The classifier returns an error.
  • The classifier times out.
  • The model reaches its output limit before it answers (TRUNCATED).
  • The provider refuses to answer for safety reasons.
  • The classifier answers with a label that no route uses.
  • No classifier provider is configured.
  • The classifier model is switched off on its provider, or its ID is ambiguous. Apinizer makes no call and records the outcome NO_PROVIDER.
  • The classifier failed recently. After a timeout, an error, or TRUNCATED, Apinizer skips classifier calls for that proxy for 15 seconds. The outcome is BACKOFF. Redeploying the proxy clears the back-off.

The request then continues down the list. If no route matches, the request goes to the Default Target Group.

Rules that depend on the model

The proxy protection limit of the token rate limit, the semantic cache lookup, and the loop guard run in routing. They run once the provider and model are final and the model access list has let the caller through. When a classifier route wins, these rules use the model of the selected route. No rule runs with a guessed model.

Status code order

The request body size limit (413) and the stream compatibility checks (400) run before the token limit (429) and before a cache answer. A very long prompt gets 413, not 429.

A provider whose Chat Completion Path is /messages streams Anthropic frames. If its provider type is not Anthropic (for example vLLM), the gateway cannot read the answer text from the stream. When the API has a response policy that must read the answer (PII Mask on the response, DLP Guard or Topic Guard in streaming buffer mode), a streaming request ("stream":true) is rejected with HTTP 400 and error code ERR-312 before any provider call. A failover entry of this kind is skipped for a streaming request. If every entry is skipped this way, the client gets the same 400. Requests with "stream":false and streams that the gateway can read do not change.

Prompt steering

A user can try to steer the classifier choice with the text of the prompt. The choice is always limited to the routes that you defined. Quotas and the Limit Plan are not affected by the chosen route.

In the first stage of the routing screen, the panel shows two information messages: Classifier: provider · model · timeout ms and Embedding: provider · model.

Routing by Request Size​

You can route by the size of the request. In the rule conditions of a route, select the context value Estimated Request Token Count. Set the data type to Numeric and select an operator, for example greater than.

The value is an estimate of the tokens in the request body. Apinizer uses the body that goes to the backend. If there is none, it uses the body from the client.

Test Routing​

Use Test routing to check which route a sample prompt selects. The button is shown only on the Conditional Routes panel.

  1. Click Test routing.
  2. Select a deployed environment.
  3. Enter a sample prompt.
  4. Click Run test.

The worker evaluates the saved routing. The target model is not called. Nothing is saved.

The classifier and embedding providers are called

The test calls the classifier and embedding providers. These calls can have a cost.

The result shows:

  • The estimated request tokens.
  • The classifier label, outcome, latency, tokens, and cost. For a reasoning model, the tokens include the thinking tokens. If the provider does not report thinking tokens (for example Ollama or vLLM), Apinizer estimates them from the thinking text the model returns. For TypeSafe Jev, the result also shows the Confidence.
  • A table with one row for each route: the rule, the semantic score, the classifier answer, and the result. The result is Selected, No match, Target unusable, or Not evaluated. Hold the pointer over Selected to see the target provider and model.
  • For a pool route, the result is Selected → Target pool · algorithm · N targets. The gateway picks the target when the request arrives, so the test does not name a pool target.
  • The message "No route matched — the request goes to the Default Target Group" when no route matches.

Rules other than Estimated Request Token Count need a real request. The test assumes that these rules match and shows them as RUNTIME_ONLY. A NOT over a rule that cannot be evaluated without a real request is shown as INDETERMINATE.

The test never resolves a value that contains ${...} (environment variable) or #{...} (context variable). An environment variable can be secret, and the test result must not expose it. A rule with such a value is shown as RUNTIME_ONLY and assumed to match. The classifier instructions, the route descriptions and the semantic example utterances also stay as written in the test. On live traffic, ${...} is resolved as usual.

The existence and emptiness operators (exists, does not exist, empty, not empty, and exists but empty) do not read a second value. The test decides them from the sample request.

When a comparison would fail on a live request, for example a numeric comparison with a value that is not a number, the test shows EVALUATION_ERROR and treats the route as not matched. AND, OR and NOT do not turn this into a match.

The sample model field accepts at most 256 characters.

The test ignores the 15-second back-off. It always calls the classifier. If embedding fails, the dialog shows a warning with the error type only.

Observability​

The traffic log has these sparse fields for the classifier:

FieldDescription
airclThe label that the classifier returned.
aircsThe outcome: MATCHED, NONE, UNKNOWN_LABEL, TIMEOUT, ERROR, TRUNCATED (shown as Output limit reached), BACKOFF (shown as Skipped (back-off)), NO_PROVIDER, or NO_INPUT.
aircmsThe classifier latency in milliseconds.
airctiThe classifier input tokens.
airctoThe classifier output tokens.
airccmThe classifier cost in micro-USD.

The AI traffic drawer and the traffic detail dialog show a Routing Classifier block with the classifier decision (outcome and label), Decision time in milliseconds, and Classifier cost.

The trace drawer shows a Route classifier section. When AI trace is on, it also shows a Routing decision panel with the result for each route.

Upgrade note: JDBC traffic log databases

If you write traffic logs to a JDBC database (Oracle, MySQL, PostgreSQL, or MSSQL), run the ALTER script for your database before you upgrade. Otherwise, traffic log inserts to this database fail.

The ALTER scripts for Oracle, MySQL, PostgreSQL, and SQL Server are on the 2026.10.0 — Log Table ALTER Scripts page. They add these columns: ai_route_cls_label, ai_route_cls_outcome, ai_route_cls_latency_ms, ai_route_cls_token_in, ai_route_cls_token_out, and ai_route_cls_cost_micro.

For Elasticsearch, no action is needed. A startup migration updates the traffic index template automatically, and the index is rolled over.

Client "model" Field​

Two independent settings decide what the gateway does with the model field the client sends in the request body. Whichever value they resolve to becomes the effective model for that request — the same model that rate limiting, semantic cache lookups, cost calculation, and traffic logs use, regardless of what the client originally sent.

You will find them on the Routing tab, in the Client "model" field card between the failover section and the connection settings. Selecting an option saves it immediately; if the save fails, the selection returns to its previous value. The card is not shown on an A2A agent's routing tab, where the client never sends a model field of its own.

If the Request Already Has a Model Field​

OptionBehavior
Use the client's modelThe gateway sends the client's model to the provider. A model on the provider's list must be enabled. A model that is not on the list is also sent when the provider does not have Only listed models on and the model ID is a plain name (see below). Otherwise the request is rejected with HTTP 400.
Overwrite with the routing model (default)The model selected by routing replaces the client's value.
Reject the requestRequests that carry a model field are rejected with HTTP 400.

Models That Are Not on the Provider's List​

With Use the client's model, the gateway sends a model that is not on the provider's list only when both conditions are true:

  • The provider does not have the Only listed models switch on.
  • The model ID is a plain name: it starts with a letter or digit, contains only letters, digits and the characters . _ : @ + -, is at most 128 characters long, and does not contain ...

If one condition is not met, the gateway rejects the request with HTTP 400. A model ID that matches more than one listed model is also rejected. Unlisted models have no price and no access control, so the USD limits do not meter them.

This rule applies to the primary target only. A failover leg keeps its own model when the client's model is not on the leg's list. It does not send the unlisted client model to its provider.

If the Request Has No Model Field​

OptionBehavior
Leave it absent (default)The body is forwarded without a model field. Providers that resolve the model from the URL or a deployment work this way.
Add the routing modelThe model selected by routing is added to the body.
Reject the requestRequests without a model field are rejected with HTTP 400.
Legacy Behavior

Unset values keep the previous behavior: overwrite the model when the client sends one, leave it absent when the client doesn't. An existing gateway keeps working exactly as it did before, with no change required.

Provider Notes
  • Gemini, Vertex, and Bedrock carry the model in the URL path — the effective model still selects that path. On Gemini, Vertex, and Anthropic models on Bedrock the native request body never gets a model field. Other Bedrock models (Titan, Llama, Mistral, Nova) are still forwarded in OpenAI shape, so on those this setting applies to the body exactly as it does on any OpenAI-compatible provider.
  • Azure OpenAI pins the model in the endpoint: a deployment endpoint such as https://<resource>.openai.azure.com/openai/deployments/<deployment> already names the deployment that will answer, so the body's model field cannot change which model is actually called. Use the client's model is therefore not applied there — see below.
  • Anthropic always writes the effective model into the native request body itself, independent of what the client sent.
Azure OpenAI: "Use the client's model" Is Not Honored

Gemini, Vertex, and Bedrock take the effective model from the gateway and write it into the URL themselves, so attribution and the model that actually answers always agree. An Azure OpenAI deployment endpoint (/openai/deployments/<name>) is the exception (an Azure v1 endpoint, /openai/v1, takes the model in the body like OpenAI and is not affected): the deployment is fixed in the endpoint the operator configured, so the body's model field cannot change which model answers — it could only change attribution, meaning which model's entry rate limits, quota, semantic cache, cost calculation, and traffic logs are booked against.

Letting the client move that attribution would let it be billed and counted at a cheaper model's rate than the deployment that actually served the call. Apinizer therefore does not apply Use the client's model on such a provider: the effective model stays the configured routing model and no model check is run against the client's value (the client's model is never called, so an unlisted value is not an error either). The request Apinizer sends to Azure carries the configured routing model in its body, exactly as it does under Overwrite with the routing model — what the client sent simply stops being a reason to reject the request. Overwrite with the routing model, Reject the request, and every option on the "no model field" axis behave exactly as described above.

Failover Legs​

A failover leg re-evaluates both settings against its own provider's models. With Use the client's model, if the client's model isn't one of that leg's enabled models, the leg falls back to its own configured model instead of failing the request — only the primary target rejects outright when the model does not match.

A leg that does not define a model of its own inherits the configured routing model, not the effective model the primary resolved for this request. That distinction matters under Use the client's model: the client's model is used on a leg only when that leg's provider has it as an enabled model, and otherwise the leg falls back to a model the provider is known to serve, rather than carrying the client's choice into a provider that never offered it.

Context Window​

The Context window card sits on the Routing tab, below the Client "model" field card. It sets what the gateway does when a request does not fit the context window of the model that routing chose. The card is not shown on an A2A agent's routing tab.

The check runs once the model is decided. It applies only to chat/completions and responses requests.

The limit is the model's context window, minus the max_completion_tokens, max_tokens or max_output_tokens value of the request, minus a safety margin of 512 tokens. If the result is 0 or less (a small context window or a large output token value), no request fits: the Reject the request and Truncate the conversation options reject the request. Apinizer counts text only. Images, audio and files are not counted.

Set the context window of a model in Models, in the model settings. A model with no context window value is not checked. The Models in this proxy table on the card shows No value, not checked for such a model.

OptionBehavior
Reject the request (default)The gateway rejects the request with HTTP 400 and error code ERR-311. It does not call the provider. Proxies that were never configured use this option.
Truncate the conversationThe gateway removes the oldest conversation turns until the request fits. If the request still does not fit, the gateway rejects it.
Leave it to the providerThe gateway does no check. The provider applies its own limit.

The default body of the rejection is:

{
"error": {
"code": 400,
"message": "Request exceeds the model's context window",
"type": "context_length_exceeded"
}
}

You can change this message on the Error Messages screen. The traffic log records the request as BLOCKED.

Truncation Rules​

  • The gateway always keeps the system messages that the client sent, the last user message, and everything after it. The only change allowed to the last user message is the shortening that Last user message sets (see below).
  • A tool call and its result are removed together.
  • Removal starts with the oldest turn. A turn is a user message and the assistant and tool messages after it.
  • Recent turns to keep sets how many of the latest turns the gateway never removes. The value is from 0 to 50. The default is 0.
  • Context added by RAG has two options. Keep (default) never removes the context that the gateway added. Shorten from the end removes the least relevant last chunks when removing turns is not enough.
  • Last user message has two options. Keep (default) never changes the last user message. Shorten from the middle removes the middle of its text when the steps above are not enough.

The gateway applies the steps in this order:

  1. It removes the oldest conversation turns.
  2. When Context added by RAG is Shorten from the end, it cuts the end of the RAG context.
  3. When Last user message is Shorten from the middle and the request still does not fit, it removes the middle of the last user message.

When the gateway shortens the last user message, it keeps the start and the end of the text. In place of the removed part it puts a note: [... N characters removed by the gateway to fit the model context window ...]. The model can see that content is missing. The gateway measures the request again with the note. If the request still does not fit, the gateway rejects it with HTTP 400.

Rules for Shorten from the middle:

  • The last user message can have several text parts (for example the text parts of the Chat content array or the input_text parts of the Responses input). The gateway cuts the largest text part first. If this is not enough, it also cuts the next parts.
  • Each text part loses at most half of its own text. Characters are counted as Unicode code points. So every part keeps its start and end, and the whole message also loses at most half of its text. Each cut part gets its own note.
  • If more is needed, the gateway rejects the request.
  • The cut moves to a line break when one is close.
  • The gateway cuts text only. It never changes images, audio, files, tool calls or tool results.
  • It works with Chat Completions messages[].content (a string or text parts) and with Responses input (a string or user input items).
  • A RAG context set to Keep is never cut. If several RAG Injection policies run on the proxy, the gateway cannot find the kept context in the message. In this case it cuts nothing and rejects the request.
Content is lost

Typical use is a whole file pasted into one message. Pasted JSON, XML or code can break in the removed part. The answer cannot take the removed content into account. To analyze a whole document, the client should split the content into several requests.

The response to a truncated request carries the X-Apinizer-AI-Truncated header. The header holds the number of removed messages. When only the last user message was shortened, the value is 0. The traffic log also marks the request.

Click See an example to open a side drawer. The drawer runs the saved rules on a sample conversation. Each message shows its turn and a badge: Always kept (system messages and the last turn), Kept, Kept (setting) (kept by Recent turns to keep), Removed, Shortened at the end (the RAG context) or Shortened in the middle (the last user message). A marker in the text shows where tokens were cut, and the token count shows before → after.

Failover and Known Limits​

  • The model that routing chose decides. During failover, the gateway skips a backup model whose context window cannot take the request.
  • Requests to A2A proxies are not checked.
  • When the Token Rate Limit policy switches to a cheaper model, the gateway does not check the request again.

For the per-request token limit of the Token Rate Limit policy, see Token Rate Limit.

Connection and Delivery Settings​

These cards apply to every request that the LLM Gateway forwards, not to one branch of the routing decision. Each value is labeled with the request type it applies to: All requests, Single-response requests, or Streaming requests. Each card has its own Configure drawer: the Connection drawer holds the timeouts, the retry count and the connection pool; the Streaming drawer holds the stream limits. A value is edited in one place only.

Connection Card​

FieldApplies toDescription
Connect Timeout (seconds)All requestsThe maximum time to open the connection (TCP and TLS) to the LLM provider. Range 1 to 300. Default 10.
Response Timeout (seconds)Single-response requestsThe maximum time to wait for the provider's answer to a request that is not streamed. Range 1 to 3600. Default 60. If you leave the field empty, the gateway uses the default of 60 seconds. Right after you save an empty field, the card shows 60 (default).
Retry CountSingle-response requestsHow many times a request is retried after a transient connection error. Range 0 to 10. Default 0. Streaming requests are not retried.
Connect timeout and response timeout are separate

The connect timeout used to bound only the streaming connection, and the single-response connection used the response timeout. Now the connect timeout bounds the connection of both request types, and the response timeout bounds only the wait for the answer.

Connection Pool​

The Connection pool group on the same card decides how the gateway reuses connections to the LLM provider. The gateway creates the pool when you deploy the LLM Gateway. It applies to streaming requests.

OptionBehavior
General (default)The LLM Gateway has its own pool. Its size and idle timeout come from the AI Settings defaults.
CustomThe LLM Gateway has its own pool with the values you enter in Connections kept open per provider (range 1 to 1000, default 8) and Close unused connections after (seconds) (range 1 to 3600, default 4).
NoneNo pooling. Every request opens a new connection, and the connection closes when the request ends. This does not hold for a provider that has a provider-specific pool setting in a routing entry.
  • Connections kept open per provider is applied to each LLM provider of the gateway separately. A busy provider does not evict the warm connections of another provider, for example the failover provider.
  • The size limits only the idle connections that are kept for reuse. It does not limit concurrent requests, and a request never waits for a connection.
  • Keep Close unused connections after (seconds) shorter than the idle timeout of the provider. For example, vLLM and other uvicorn servers close an idle connection after 5 seconds. If the gateway keeps the connection longer, it can send a request on a connection that the provider already closed. The streaming request then fails with 502. The gateway resends a streaming request only when sending it on a reused pooled connection failed (a write or flush error). A write timeout or an interrupt does not count as a send failure, so the request is not resent. If the request was sent and the failure happens while the gateway reads the response (even before any response header), the gateway does not send it again. The provider may be running the request, so the client gets 502.
  • Known limits: on HTTP/2, if the connection is torn down at the same instant, a flush error can be reported after the last frame already left. On HTTPS, a rare TLS-level case (a TLS KeyUpdate after the data was handed to the socket) can also report a send error after the data left. In these rare cases the request can reach the provider twice.
  • On the Routing tab, the Connection card shows AI Settings default for the size and the timeout when the option is General.

The General defaults are in System Settings & Limits, on the AI Gateway tab, in the Connection Pool Defaults section. This tab is shown when your license includes the AI Gateway. Only a system administrator can save it.

FieldRangeDefault
Idle Connections per Provider1 to 100064
Idle Connection Timeout (s)1 to 36004

A change in AI Settings reaches an LLM Gateway on its next deploy. The pool of a gateway that uses General is rebuilt then. A gateway that uses Custom keeps its own values.

Provider-Specific Pool Setting​

The Primary (pool member), Failover chain and Conditional route dialogs have an Connection pool for this provider (optional) panel. It sets the pool values for the provider you select.

FieldRange
Connections kept open per provider1 to 1000
Close unused connections after (seconds)1 to 3600
  • When a field is empty, the routing-level setting applies.
  • The gateway keeps one pool for each provider. If the same provider has different values in several entries, the largest value of each field is used, and a warning icon shows in the dialog.
  • When the routing-level option is None, a provider still gets a pool if an entry fills one of these fields. The other value, if left empty, takes the value stored at routing level first (the custom pool size or the idle connection timeout). If the routing has no such value, it takes the AI Settings default. The screen clears the routing-level values when None is chosen, so for a routing saved from the screen the result is the AI Settings default. A routing written with APIops can keep routing-level values under None. Then these values are used.
  • The panel applies to streaming requests only.

Streaming Card​

FieldApplies toDescription
Streaming (SSE) switchStreaming requestsTurns streaming on for the gateway. How it combines with the client's stream field depends on the Client "stream" field setting below.
Stream Idle Timeout (seconds)Streaming requestsThe gateway closes the stream when the provider sends no new chunk for this long. Range 1 to 3600. Default 120.
Max Response BytesStreaming requestsThe gateway cuts the stream when the response reaches this size. 0 means unlimited.
Max ChunksStreaming requestsThe gateway closes the stream after this many chunks. 0 means unlimited.

Client "stream" Field​

This setting decides what the gateway does with the stream value that the client sends in the request body. It is on the Streaming card. Selecting an option saves it immediately; if the save fails, the selection returns to its previous value. A user with read-only access sees the selected value as a summary. The card is not shown on an A2A agent's routing tab.

OptionBehavior
Use the client's valuestream: true opens a streaming response, and stream: false returns a single response. A request without a stream field follows the Streaming (SSE) switch.
Overwrite with the proxy settingThe Streaming (SSE) switch decides for every request. The client's value is replaced. With the switch off, a request with stream: true still gets a single response.
  • A new LLM Gateway starts with Use the client's value. An existing LLM Gateway that has no stored value works as Overwrite with the proxy setting, exactly as before.
  • The setting applies to chat completion, Responses and additional endpoint requests. Text completions (/completions) follow only the client's stream field: they stream when the client sends stream: true, whatever the proxy setting is. Embedding, audio and image requests keep the proxy setting.
  • A client can switch streaming on only when the LLM Gateway has no policy that works only on a single response. If the LLM Gateway has such a policy and the client sends stream: true, the proxy setting applies and the client gets a single response. These policies are: Groundedness Guard; MCP Call (LLM) and A2A Call (LLM) on the response side; PII Mask that restores masked values inline, or that decodes encoded content on the response side; Topic Guard and DLP Guard both in buffer mode; and a DLP Guard, Topic Guard or PII Mask policy on the response side that uses an external guardrail engine. A client can always switch streaming off.
  • The semantic cache keeps separate entries for streaming and single-response requests when the setting is Use the client's value.
Single-response responses carry a Content-Type

A single-response answer from an LLM Gateway reaches the client with a Content-Type header. Earlier versions could send the answer without it. This also holds for an answer that comes from the semantic cache. Headers that only the provider uses are not forwarded.

Certificate Trust for the LLM Provider​

The gateway checks the certificate of the LLM provider with the certificates in the Apinizer certificate store. This applies to single-response and streaming requests. It applies when the routing does not use mTLS and Skip SSL Verification is off. The gateway uses the certificates that you assign to the environment of the LLM Gateway on the Management → Secrets Management → Certificates screen.

  • A certificate that you add or remove applies to new connections right away. You do not need to redeploy the LLM Gateway. This is the same behavior as in API proxy routing.
  • If the routing uses mTLS, the trust settings of the mTLS configuration apply.
  • This does not change hostname verification.
PKIX path building failed

This error means that the certificate chain of the provider is not trusted. Add the issuing certificate authority (or the certificate) to the certificate store for the environment of the LLM Gateway. See SSL/TLS Certificate Issues.

Failover Chains​

When you configure an LLM Gateway, you can list more than one provider connection for a model — a primary and one or more backups. If a request to the primary fails (timeout, error response, or the primary is unavailable), Apinizer automatically retries it against the next connection in the chain, without the client seeing the failure.

When a Limit Is Reached​

A Limit Plan or the Token Rate Limit policy can use the overflow action Failover. The request then goes to the backup provider. A backup whose own limit is full is not called; the request moves to the next backup in the chain. If no backup can be called, Apinizer rejects the request with HTTP 429. It does not send the request to the over-budget primary.

Failover on Streaming Requests​

A streaming request moves to the next target in the same cases as a single-response request, but only before the first byte reaches the client:

  • The gateway could not open the connection to the target, or the connection was lost before the gateway sent the request.
  • The target answered with an HTTP error that starts a failover, before it sent any content. By default these are 5xx, 408 and 429; the Trigger status codes field changes the list.

The gateway does not move to another target in these cases:

  • The target already sent the request content and then timed out or dropped the connection. The target may have processed the request, so the gateway does not send it again.
  • The stream has started. The status of the answer cannot change after the first byte. If the provider breaks the stream after it started, the client receives one error event in its own format (the Anthropic format for a Messages client).

Other rules for a streaming request:

  • The order of targets, the failover strategy, the open circuits, the Limit Plan budgets, the context window and the cost cap work as they do for a single-response request. With Priority Strict, the gateway moves to the next target only when it cannot reach the current one.
  • The gateway tries each target once. A stream does not retry on the same target.
  • Each attempt has its own row in the traffic log. See Traffic Log of an AI Request.
  • The policies that run when a stream ends run once, after the last attempt.
  • When the last target fails before it starts, the client receives the real HTTP status of that target. A single-response request in the same case receives the exhausted-targets answer (HTTP 502 by default). A known provider error keeps its own text in both cases. See Errors From the LLM Provider.

Circuit Breaker​

The circuit breaker of the AI routing is off by default. You turn it on with the Management API. See Update AI Routing. The breaker is separate for each provider record and endpoint address.

  • The breaker counts connection errors and HTTP statuses that start a failover. A client error such as 400 does not open a circuit. A client that sends bad requests cannot open the circuit of a healthy server.
  • The breaker works for single-response and streaming requests. It skips a target whose circuit is open. For a stream, it records the connection and the HTTP status once. It does not record what happens after the first byte.

Response Headers When Every Target Fails​

When the primary and all backup targets fail, the client gets an error response. The response status and body follow your Error Message settings. If the last answer of a provider is a known provider error, that error's own text is used instead. See Errors From the LLM Provider. Apinizer also adds these response headers:

HeaderValueWhen it is sent
X-Apinizer-AI-ExhaustedtrueThe chain is exhausted for a non-streaming request, or a streaming request has no target left to try because the circuit breaker is open on every remaining target or the cost cap leaves every remaining target out. Other reasons for leaving every target out have their own answers (see What the Client Receives).
X-Apinizer-AI-Exhausted-ReasonOne reason code (see below)Together with X-Apinizer-AI-Exhausted, when Apinizer could classify the last failed attempt. If it could not, only X-Apinizer-AI-Exhausted is sent.

The reason code describes only the last failed attempt:

Reason codeMeaning
connect_timeoutThe connection to the provider did not open in time.
read_timeoutThe provider did not answer in time.
dns_errorThe provider host name could not be resolved.
tls_errorThe TLS handshake with the provider failed.
connection_refusedThe provider refused the connection.
transport_errorAnother network failure.
upstream_statusThe provider answered with an error status.
circuit_openEvery remaining target of a streaming request was skipped because its circuit is open.

The headers carry no provider message, address or body. A streaming request whose last target fails after a request was sent does not get these headers. It gets the real status of that target.

If every attempt to the provider (primary, retries and failover) fails on TLS, the traffic log records the error type POLICY_ROUTING_SSLHANDSHAKE. It records POLICY_ROUTING_SSLEXPIRED when all attempts failed because the certificate is expired or not yet valid. Apinizer does not record a TLS connection that is cut during the handshake as an SSL error. If the failures are mixed (for example TLS and timeout), the error type stays POLICY_ROUTING_AI_PROVIDER_EXHAUSTED. The client response, including the tls_error reason header, does not change.

Errors From the LLM Provider​

What the Client Receives​

SituationThe client receives
A single-response request, and the provider answers with an error that does not start a failover (by default a 4xx other than 408 and 429, such as 400, 401 or 404)The text of a known provider error if Apinizer recognizes the answer. Otherwise the status and the body of the provider, as the provider sent them
A single-response request, and every target failedThe text of a known provider error if Apinizer recognizes the last answer. Otherwise the exhausted-targets text (HTTP 502 by default, ERR-250)
A streaming request, and the last target answers with an HTTP error before its first chunkThe text of a known provider error if Apinizer recognizes the answer. Otherwise the provider-error text (ERR-251) with the real status of the provider
A streaming request, and no target is left to try because the circuit breaker is open on every remaining target, or the cost cap (or a mix of the two) leaves every remaining target outThe exhausted-targets text (HTTP 502 by default)
A streaming request, and every target is left out for another reasonThe answer of that reason: the context window gives HTTP 400 with ERR-311, a stream policy gives HTTP 400 with ERR-312, and the budget of a Limit Plan gives HTTP 429

The first row has two details. In Standard mode, if the client and the provider speak different API families (for example a Messages client and an OpenAI-format server), the body of an error that Apinizer does not recognize is put in the envelope of the client's family. The text stays the same. In Pass-through mode and for an additional endpoint, the answer is never changed.

Known Provider Errors​

Apinizer recognizes these provider errors on every path except Pass-through mode and additional endpoints. It reads the HTTP status first and the message of the provider second.

Provider errorStatusError codeError type in the body
The request does not fit the context window of the model400ERR-311context_length_exceeded
The model server does not accept tool calls (tool calling is off)400ERR-316tools_not_enabled
The model is not found on the provider404ERR-317model_not_found
The provider refused the key that the gateway holds (401 or 403)502ERR-318provider_auth_failed
The provider rate limit was reached (429)429ERR-319rate_limit_exceeded
  • ERR-311 also covers a context overflow that the provider reports. Before, only the check of the gateway itself used it. See Context Window.
  • A provider 401 or 403 becomes ERR-318 only when the gateway holds the provider key. If the client brings its own key, the 401 or 403 of the provider stays as it is.
  • On a 429, Apinizer sends the Retry-After header of the provider to the client, so that the client SDK waits and retries.
  • An error that matches none of these keeps the text of ERR-251 or ERR-250.
  • You can change the text, the status code and the error code of each type on the Error Messages screen. The default text never contains the message of the provider.
If you changed ERR-250 or ERR-251

The new types come as new records. A text that you wrote for ERR-250 or ERR-251 does not apply to them. Write the same text for ERR-316, ERR-317, ERR-318 and ERR-319 if you want it for the recognized errors too. ERR-311 now also covers the context overflow that the provider reports.

Error Format of the Client​

The default error text is written in the format of the API that the client uses.

ClientFormat
Anthropic Messages (for example Claude Code){"type":"error","error":{"type":"…","message":"…"}}
Chat completions and Responses{"error":{"code":…,"message":"…","type":"…"}}
  • For a Messages client, the text of a context overflow starts with prompt is too long:. Claude Code recognizes this text.
  • The code value in the default text is the status that the answer is sent with. For example, a request to a method that the gateway does not serve gets 405 in code.
  • A text that you wrote on the Error Messages screen is sent as written. Apinizer does not change its format.
  • When a stream reaches the limit of Max Response Bytes or Max Chunks, the stream ends with an error event. A Messages client receives it in the Anthropic format.

Show the Answer of the Provider​

The default error text hides the message of the provider. If you want the client to see it, use two template variables in the text of an AI error on the Error Messages screen:

VariableValue
#{aiProviderErrorBody}The body of the provider's last error answer
#{aiProviderStatus}The HTTP status of that answer

Both are empty when the provider did not answer, for example after a connection error.

To send the answer of the provider as it is:

  1. Open the error type on the Error Messages screen. Use the provider-error type (ERR-251), the exhausted-targets type (ERR-250) or one of the known types above.
  2. In Error Response Rules, click Add Rule.
  3. Leave the Status Code of the rule empty. An empty status means the real status of the provider.
  4. Write #{aiProviderErrorBody} as the template.
  5. Save the rule.

The provider-error type (ERR-251) is also used for errors that do not come from the provider, for example a gateway configuration error (HTTP 400) or a failed model discovery. The variables are empty for these errors. A rule on ERR-251 with no status and only #{aiProviderErrorBody} as the template would therefore send an empty body for them. Give the rule a condition on the status, or use the variable inside a fuller template, for example a text that also names the status.

The answer of the provider can leak information

The message of a provider can contain parts of the prompt, internal host names or version details. With these variables, this information reaches the client. The Error Messages screen shows the same warning on every AI error. Use the variables only for a provider that you trust.

Tokens of an Error Answer​

An error answer books only the usage that the provider reported. If the provider reported none, the booked usage is 0, and the token reservation is released. The estimate of the gateway is not booked for an error.

Traffic Log of an AI Request​

The detail drawer of a record in the AI traffic view shows each hop of the request. See Reports and Analytics.

TabWhat it shows
Request from ClientThe request as the client sent it, with the key of the client.
Request to Target APIThe request that the gateway really sent to the provider: the address parameters, the headers and the body. A binary body shows the download marker.
Response from Target APIThe answer of the provider as it came: the status, the headers and the body, before the gateway translated it. For a stream that fails before its first chunk, this is the error answer of the provider.
Response to ClientThe answer that the client received.
  • The key of the provider is always masked in Request to Target API. This covers the header and the query parameter that carry the key of each target, and a user name and password in the address. A target with a different key header, for example in a failover chain, is masked by its own header name.
  • The key that the client sends is visible in Request from Client. This is expected.
  • Each attempt of a failover has its own row, with its own request and its own answer. A connection error does not show the answer of the earlier attempt.
  • Response from Target API shows the answer of the provider even when the client received a different text, for example the default text of ERR-251. The provider answer is therefore available in the log without the template variables in Show the Answer of the Provider.
  • The size of the log body follows the log settings of the connector.
  • The response size of a streaming request is the number of bytes written to the client. The size is also counted when the client disconnects.

Load Balancing Across Equal-Priority Primaries​

Instead of a single primary connection, you can define a pool of equal-priority primaries and let Apinizer choose which one handles each request. This choice is made once, before the request is attempted — a different, earlier stage than the failover chain below, which only activates once the chosen primary has already failed. A pool member that isn't selected still becomes a failover sibling, tried ahead of the explicit Provider Failover Chain if the selected member fails.

Configure the pool from Target Selection → Load Balancing on the routing screen. Seven algorithms are available. The notes after the cards describe how each one behaves.

Every pool entry must name a provider. Its model is optional and inherits the default target's model when left blank — the same rule applies to failover chain entries, and an entry with no model of its own is rejected when there is no default model to inherit. An entry with no provider is skipped by the gateway at request time, so saving one is rejected rather than left on screen as configuration that never runs.

These checks run when you save the routing configuration itself — from the Routing tab, the APIops ai-routing endpoint, a promotion, or an import. Unrelated operations on the same proxy (deploying it, adding a policy, editing permissions) do not re-run them, so a proxy that predates a rule never becomes impossible to open and fix.

Round Robin

Distributes requests evenly across the pool, in turn.

Weighted

Distributes requests proportionally to a configured weight per pool member — a higher weight receives more traffic. The members receive their requests in runs, not alternately.

Random

Picks a pool member at random for each request. The weight of a member does not count.

Least Recently Used (LRU)

Sends the request to whichever member has been selected the fewest times. It behaves like Round Robin.

Least Cost

Picks the member whose model has the lowest price, weighted as input price + output price × the configurable output ratio (default 0.25, reflecting that a typical chat workload generates far more input tokens than output). A member whose model has no price is excluded from the comparison; if none of the pool has a known price, the pool falls back to Round Robin. Members with the same lowest price share the traffic in turn.

Least Latency

Picks the member that is expected to answer fastest, from the measurements of real traffic. A streaming request uses the time to the first token. A single-response request uses the time per output token. Each member has its own value for each kind of request. The choice also counts the requests that are running on a member now. The details are in Least Latency. With no measurement for any member, Apinizer uses the periodic health-check result, and then Round Robin.

Prefix Affinity (Sticky)

Routes the same conversation to the same pool member every time, to keep that member's prompt/KV cache warm on a self-hosted LLM farm (vLLM, SGLang, Ollama). The affinity key is the value of a configurable request header (default session_id) when present, otherwise a digest of the whole system message plus the first user message; if neither is available, falls back to Round Robin.

Do not combine Prefix Affinity with a cache-aware upstream router

If the pool members sit behind a layer that already makes its own KV-cache routing decision — such as llm-d or a GKE Inference Gateway — do not select Prefix Affinity. Pinning requests from Apinizer's side conflicts with that layer's own decision.

Least Cost and Least Latency only take effect with 2 or more pool members; with fewer, the pool silently uses Round Robin instead (the routing screen shows a warning when this applies to the current configuration).

A pool member is skipped when its provider is deleted or disabled, or when it has no resolvable model — the pool simply load-balances across what remains. If nothing remains, routing falls through to the single default target and then to the failover chain.

When you save the Default Target Group drawer from the screen, the screen clears the older proxy-level single default provider and model (the one shown with the Single default badge). The pool is then the only default target, so removing the last member leaves the group empty and routing goes to the failover chain. If a conditional route, pool member, or failover entry has an empty provider or model and used the cleared single default, the save is rejected: give that route or entry its own model. A routing written with APIops can still keep the single default. Then it is used as described above.

Bounded-load protection: for Prefix Affinity, if the target member's current load exceeds the pool average by more than the configurable Bounded Load Factor (default 1.25), the request shifts to the next member instead — this keeps one popular conversation from overloading a single pool member. The load is the number of requests that are running on the member now.

Least Latency​

  • Apinizer measures each attempt from the moment it starts. The time that the request spent before the attempt, for example in policies or in an earlier attempt, does not count.
  • A member has a separate value for streaming and single-response requests. A request uses the value of its own kind. A value that is older than 10 minutes counts as no measurement.
  • A new measurement that comes after a long gap is blended with the old value, half and half. After a gap longer than 10 minutes, the value starts from scratch.
  • A member that has not been measured for more than 30 seconds is tried now and then, about once in 20 requests, so that its value stays current.
  • Only a successful attempt gives a measurement. When a member fails to connect or times out, its value rises to the double of the current value, with a lower bound of 60 seconds for a stream and 1 second per token for a single-response request. This penalty ends after 60 seconds. A client that cancels its request does not cause a penalty.
  • The cost of a member is its value multiplied by the number of requests that are running on it, plus one. A fast member with many running requests can cost more than a slower idle member.
  • A member without a measurement gets the best value among the members that have no penalty. The penalty of a failed member is not carried over to a member without a measurement. It receives requests, and so it gets its own measurement.
  • Members within 10 percent of the fastest member's cost are equal. Apinizer picks one of them at random.

Pool Members and Failover Order​

With the Sequential and Conditional strategies, the members of the pool that were not selected are tried as failover targets before the explicit failover chain. Their order is shuffled for each request, and the weight of a member makes it more likely to come first. The order of the explicit chain does not change. With Priority Strict, a pool member is tried only like a backup: when the selected target cannot be reached. An error answer such as 503 is returned to the client.

Try the Other Pool Targets​

The Try the other pool targets switch is in the Error path (Failover) drawer. It decides what happens when the selected pool target fails. It applies to the Default Target Group and to every conditional route pool.

SettingOrder of attempts
On (default)The selected target, then retries on the same target (if any), then the other pool targets, then the failover chain.
OffThe selected target, then retries on the same target (if any), then the failover chain directly. If the chain is empty, the request returns an error.

If the switch was never saved, it is on.

Apinizer moves to the next step in these cases: a connection error, a timeout, a failover status code, or by default a 5xx, 408 or 429 response. This list does not change. A model access denial or a 4xx client error does not move to the next step. A streaming request follows the same walk until its first byte, with the limits in Failover on Streaming Requests.

Retries on the same target are a separate setting.

Routing Strategies​

These strategies control how Apinizer iterates the Provider Failover Chain once the active primary — whether a single primary or one selected from the pool above — has failed:

Sequential

Always try connections in the order listed; move to the next one only on failure.

Round Robin

Rotate the failover attempts across the connections in the chain, in turn (by the weight of each entry).

Priority Strict

The primary handles all traffic. Backups are used only when the primary cannot be reached (connection failure or open circuit); an error response from the primary is returned as is.

Conditional

Also fails over when a successful response is slower than the latency threshold. If no other connection answers faster, the slow response is returned. This strategy can send a second, billed request: the slow but successful answer is kept, and the request goes to the next connection. The client waits for both calls. Choose Sequential to return a slow answer at once.

Looking for cost- or latency-based selection?

That's Least Cost and Least Latency — two of the seven algorithms available for load balancing across a pool of equal-priority primaries above, not a Provider Failover Chain strategy. The chain above only runs after a primary has already failed.

Failover Chain Fields​

The failover chain is defined row by row on the Routing tab. These settings apply to the whole chain:

FieldDescription
Failover StrategyHow the chain is walked: Sequential, Conditional (latency/error), Priority Strict or Round Robin (load balance)
Trigger status codesThe HTTP status codes that start a failover. When empty, 5xx / 408 / 429 are used
Retries per stepThe number of attempts on each failover entry (default 1)
Conditional Latency Threshold (ms)Read only by the Conditional strategy

For each entry you pick the provider and model, and can set a timeout, a maximum cost (see Per-Leg Cost Cap) and, for Round Robin, a weight.

These four strategies are not the seven load-balancing algorithms above. The algorithms choose among equal-priority primaries before the request is sent; the strategies decide how the failover chain is walked after a failure.

The failover chain is part of the proxy's configuration, so it moves with the proxy on export and import and can also be managed with the APIops REST API.

Per-Leg Cost Cap​

You can set a maximum cost for an individual failover leg. Before forwarding a request to a connection, Apinizer estimates its cost from the input tokens and the requested maximum output tokens, using the model prices of the provider. If the projected cost exceeds that leg's cap, the leg is skipped and the next one in the chain is tried. A connection whose model has no price is not skipped by this check — the request goes through rather than being blocked by an unknown price.

Reliable Retries and Billing​

Failover retries are billed correctly: a failed attempt against one connection is never counted against your usage or budget — only the connection that actually served the response is billed and logged.

Tool-Call Loop (Agentic Requests)​

When a model's response includes a tool call, Apinizer can dispatch it to the tools configured for the proxy and feed the result back to the model automatically, continuing the exchange until the model returns a final answer or a configurable turn limit is reached (5 turns by default). Usage and cost are tracked per turn, so multi-step tool use shows up in reports the same way a single request would.

An MCP or A2A Call policy placed inside a Policy Group on the response side takes part in the loop, the same as one placed directly on the proxy. Apinizer enters the group when the group is active and its condition is met. Tool results are masked with the proxy's Personal Data Masking rules; see Tool Results in the Tool-Call Loop.

Anthropic Messages Clients​

An AI API proxy accepts clients that speak the Anthropic Messages API, such as Claude Code, next to chat clients. One proxy serves all models. No separate proxy or route is needed.

RequestPurpose
POST <proxy>/v1/messagesA Messages request.
POST <proxy>/v1/messages/count_tokensA token count for a Messages request.
HEAD <proxy>/api/helloThe reachability check that the client sends.

Route conditions on the model field of the request body work in the same way as for chat clients. See Conditional Routing.

Which Providers Serve It​

The gateway sends the request to the /messages path under the endpoint of the provider in these cases:

  • The provider type is Anthropic, vLLM or Ollama.
  • The Wire Protocol of the provider is Anthropic.

For any other provider, the Chat Completion Path of the provider decides. A path that ends in /messages states that the endpoint serves the Messages API, and the gateway uses it. A provider that has such a path keeps working for Messages clients.

After you upgrade, remove the /messages value from the Chat Completion Path of vLLM and Ollama providers. While the value stays, chat requests to that provider also go to this path, and OpenAI clients of that provider fail. Keep the value on a provider of any other type that serves the Messages API.

A provider that cannot serve the Messages API gets no call. The client receives HTTP 400 with the error type messages_unsupported and error code ERR-313. The same answer comes for a unary request and for a streaming request. A failover entry of this kind is skipped. You can change the text, status code and error code of ERR-313 on the Error Messages screen.

Client Setup​

Set the address of the proxy as the base URL of the client:

export ANTHROPIC_BASE_URL=https://<gateway-address>/<proxy-path>

Claude Code asks for a large max_tokens value by default. If the --max-model-len of the model on your vLLM server is smaller than that value, the server rejects the request. Set CLAUDE_CODE_MAX_OUTPUT_TOKENS to a value that fits the model, or increase --max-model-len:

export CLAUDE_CODE_MAX_OUTPUT_TOKENS=8192

Limits on Messages Requests​

  • PII masking and DLP Guard on a Messages stream mask the text, the tool call input and the thinking text. See Tool Calls and Reasoning in Streams.
  • A chat client that reaches a provider whose Chat Completion Path is /messages, with a PII masking policy and "stream":true, is refused with the error type stream_policy_unsupported (ERR-312).
  • The context window action does not run on Messages requests.
  • Semantic cache is off for Messages requests.

Request Handling​

The Request Handling card on the Routing tab decides how the gateway treats the request and the answer.

OptionBehavior
StandardThe gateway converts the request from the client format to the provider format, and the answer back to the client format. Use it when clients and providers speak different formats.
Pass-throughThe gateway sends the request and the answer as they are. Authentication, model access control, policies, logging and cost still apply.

In Pass-through mode:

  • The client model field and stream field are sent as they are. The client model settings and the Streaming (SSE) switch are not applied.
  • The path of the request selects the endpoint, as in Standard mode. See Which Paths the Gateway Serves.
  • Failover uses only providers with the same wire format.
  • A streaming chat request works in this mode only on providers that use the OpenAI format.

A new proxy and an existing proxy use Standard.

Which Paths the Gateway Serves​

The path after the address of the proxy selects the endpoint. The rule is the same in Standard and Pass-through mode. The gateway makes no call to a provider for a request that it does not serve.

RequestResult
A built-in endpoint with its method: POST to /chat/completions, /completions, /messages, /messages/count_tokens, /embeddings, /audio/transcriptions, /audio/speech, /images/generations, /images/edits, /rerank, /systemone or /responses; GET to /modelsThe gateway serves it.
The calls on a stored response: GET or DELETE on /responses/{id}, GET on /responses/{id}/input_items, POST to /responses/{id}/cancel, /responses/compact or /responses/input_tokens; GET on /models/{id}The gateway serves it.
A path and method in the Additional Endpoints table of the providerThe gateway serves it. See Additional Endpoints.
Another path with the POST methodThe request goes to chat completions.
A built-in endpoint with another method, for example GET on /chat/completionsHTTP 405 with an Allow header
Another path with another method, for example GET or DELETEHTTP 404
A known API path that the gateway does not serve: /files, /batches, /moderations, /uploads, /vector_stores, /conversations, /fine_tuning, /assistants, /threads, /realtime, /containers, /evals, /videos, /chatkit, /audio/translations, /images/variations, /chat/completions/{id}HTTP 404, also for POST

A row in the Additional Endpoints table of the provider opens a path of the last row. The 404 and 405 answers use the error type with code ERR-315. You can change its text on the Error Messages screen.

A POST to the address of the proxy itself goes to chat completions, as before. Another method gets 404.

A stored response can be read again with GET /responses/{id}:

  • With ?stream=true, the gateway relays the stored response as SSE, as the provider sends it. It books no token usage, because the original request already booked it.
  • If the proxy has an active response policy, for example PII Mask, a request with ?stream=true is refused with HTTP 400.
  • Without stream, the answer is one JSON answer and goes through the response policies.
  • If the stream breaks after it started, the client receives an SSE error event.

Answers of Additional Endpoints​

The gateway sends the body, the query string and the method of the client to the provider and returns the answer as it is. It does not translate the answer.

  • Streaming: The decision is the same as for chat. In Standard mode the Client "stream" field setting and the Streaming (SSE) switch decide. In Pass-through mode the stream field in the body decides. When the decision is streaming, the gateway relays the answer as it arrives, with the Content-Type of the provider. When the decision is single response, the gateway reads the whole answer first, even if the provider streams it.
  • Response policies: A proxy with any active response policy, or with a request-side Personal Data Masking policy that restores the masked values in the answer, cannot relay the answer piece by piece, because the format of the answer is not known. The gateway reads the whole answer, and the policies work on it. This applies to text answers such as application/json, text/event-stream, application/x-ndjson, application/xml and text/*. An answer of a binary type (image, audio, video, application/pdf, application/zip and application/octet-stream) goes to the client unchanged.
  • Accept header: The gateway sends the Accept header of the client to the provider. Without one, it sends application/json.
  • Limits: The Max Response Bytes limit and the Stream Idle Timeout of the Streaming card apply, as for a chat stream.
  • Errors: An error answer of the provider goes to the client as it came, in a streaming request too.
  • Usage: If the answer contains the usage of the provider, the gateway books it. If a complete answer contains no usage, the gateway books 0, as for a single-response request. If the answer is cut because the client or the provider disconnected, or because a limit was reached, the gateway books an estimate from the request and the bytes it read.
  • Redirects: A streaming request follows HTTP redirects, as a chat stream does. A header that carries the provider key under a custom name is not removed when the request follows a redirect. Use a provider address that does not redirect.
A file upload is refused when a content policy is active

The gateway refuses a multipart upload with HTTP 400 (error type content_policy_uninspectable, error code ERR-314) when the proxy has an active request content policy: PII Mask, DLP Guard, Topic Guard or Prompt Guard. The policy cannot read the file. See Text Completions, Rerank, Image Edit and System One.

Endpoints Panel​

Below the decision-chain sections, the Routing tab lists an Endpoints panel: one row for every environment configured in the installation, showing the data-plane address to call in that environment, with a copy icon next to it.

Rows are not filtered by deployment status

The panel lists every environment defined in the installation, not just the ones the proxy has actually been deployed to. Running Test against a row for an environment the proxy hasn't been deployed to returns an error from the gateway — the proxy isn't running there to answer the request.

Each row also has a Test button that opens the Test Console pre-filled with a ready-to-send POST request against that row's address. The request fits the Model Type of the routed model:

Model typePathBody
Chat or empty/v1/chat/completionsThe model, a single ping user message and max_tokens: 16
Embedding/v1/embeddingsThe model and a sample text
Rerank/v1/rerankThe model, a sample query and two sample documents
Image/v1/images/generationsThe model and a sample image prompt
Audio, text to speech/v1/audio/speechThe model, a sample text and the voice alloy
Audio, speech to text/v1/audio/transcriptionsA multipart form with the model field. Add the audio file in the file field

For an audio model, a model id that contains tts or speech gets the text-to-speech request. Another audio model gets the speech-to-text request.

This test call reaches your real LLM provider

Sending the pre-filled request from a row's Test button goes to the actual configured LLM provider, the same way a real client request would — and consumes cost/quota accordingly. max_tokens is kept at 16 to keep a test call cheap, but it is not free, and a cost/quota warning is shown both on the panel and in the test dialog.

The Develop tab of an LLM Gateway has a similar shortcut. Its address list shows one row per environment, with a Test Endpoint button next to the address of every environment the proxy is deployed to. The button always opens the chat request (/v1/chat/completions with ping and max_tokens: 16) in a Test Endpoint dialog and is available to users who may run tests. The dialog carries the same warning: the call goes to the actual LLM provider and consumes cost and quota, and the minimal request (max_tokens: 16) is pre-filled to keep it low.

The panel is hidden when the proxy is embedded in an A2A agent (its traffic doesn't go through /v1/chat/completions), and the Test button is disabled unless you hold manage permission on the proxy.

Testing a streaming response and a token/cost summary for the test call are not covered by this panel — use a full API client for those.

Next Steps​