# Gateway & Multi-Protocol Routing Notes (as of 2026-10-25)

*Scope: protocol surfaces and translation fidelity relevant to coding agents and multi-provider routing. Compiled from vendor docs, GitHub release notes, and source code on 2026-10-25 (Asia/Shanghai). "Latest" versions observed: LiteLLM Proxy v1.106.3, vLLM v0.32.0, SGLang v0.5.22, Ollama v0.40.11, llama.cpp b11870. OpenRouter, Bedrock, Google Cloud Agent Platform, and Azure Foundry are hosted services without pinnable versions. Where a claim is inferred from source code or absent from docs rather than stated explicitly, it is marked **[inferred]** or **[unverified]**.*

---

## (a) Executive summary

By late 2026 the "OpenAI-compatible" chat-completions shape is still the lingua franca of model serving, but the agent ecosystem has split into three de facto dialects that a gateway must bridge: OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages — with Google's Interactions API now the recommended Gemini surface and legacy `generateContent` still supported. Every serious serving stack exposes at least one dialect natively (vLLM, SGLang, Ollama, LM Studio, and llama.cpp all ship Anthropic `/v1/messages`; most also ship OpenAI `/v1/chat/completions`; several now ship `/v1/responses`), and every cloud exposes provider-native and normalized surfaces (Bedrock InvokeModel vs Converse vs its new OpenAI-compatible endpoints; Vertex `rawPredict` for Anthropic vs OpenAI-compat `/endpoints/openapi` vs Gemini `generateContent`/Interactions; Azure's v1 `/openai/v1` surface vs legacy deployment-scoped URLs). Gateways add value through routing, fallback, cost tracking, and a single credential plane, but they are lossy exactly where coding agents care most: reasoning items/thinking signatures, `cache_control` breakpoints and TTLs, server-side built-in tools, stateful `previous_response_id`/`previous_interaction_id`, mid-conversation `system` messages, beta header/body-field pairs, and semantically typed streaming events. The practical rule is: translate as little as possible, pin and forward reasoning state byte-for-byte, treat statelessness and fallback as explicit routing constraints rather than invisible implementation details, and treat streaming/proxy configuration as a first-class compatibility requirement rather than an ops afterthought.

---

## (b) Gateway / provider matrix

Legend: **Ingress** = formats clients can speak to the server; **Egress** = upstream protocols it can reach; **State** = server-side conversation state; **Native Anthropic** = serves Anthropic Messages without translation.

| Gateway / server | Ingress formats (2026-10) | Egress / translation | State | Notes |
|---|---|---|---|---|
| **LiteLLM Proxy** (v1.106.3) | OpenAI Chat Completions, OpenAI Responses, Anthropic Messages (`/v1/messages`), Gemini `generateContent` (`/v1beta/models/{m}:generateContent`), Gemini Interactions (`/v1beta/interactions`), Bedrock Converse (`/converse`), `/v1/decisions`, provider pass-through endpoints (`/anthropic/…`, `/vertex_ai/…`, etc.) | 100+ providers; per-provider translations; native passthrough opt-in via `model_info.supported_endpoints` (`/v1/messages` since v1.92.0, `/v1/responses` since v1.101.0) | Yes, proxy-side response store + `store_prompts_in_spend_logs` for bridged `previous_response_id`; passthrough delegates to upstream | Config: `model_list` → `router_settings` → `litellm_settings` → `general_settings`. By default raises on unsupported params unless `drop_params: true` |
| **OpenRouter** | OpenAI Chat Completions (`/api/v1/chat/completions`), OpenAI Responses (`/api/v1/responses`), Anthropic Messages (`/api/v1/messages`; `ANTHROPIC_BASE_URL=https://openrouter.ai/api`), embeddings, image/TTS/STT/rerank/decisions | 400+ models across many providers; routing/fallback inside OpenRouter | Responses API **stateless only**; Anthropic Messages stateless; no Gemini `generateContent`/Interactions ingress documented | Request-level provider routing object (`order`, `only`, `ignore`, `allow_fallbacks`, `require_parameters`, `data_collection`, `zdr`, `sort`, `max_price`), plus `models`/`fallbacks` arrays; in-region hosts `eu.openrouter.ai`, `us.openrouter.ai` |
| **vLLM** (v0.32.0) | OpenAI Completions, Chat Completions (+batch), Responses (`/v1/responses`, get/cancel), Embeddings, transcriptions/translations, **Anthropic Messages** (`/v1/messages`, `/v1/messages/count_tokens`), `/v1/decisions`; renderer/derender APIs | Self-hosted models only; no egress translation | Responses API has store/cancel semantics; Anthropic endpoint **[inferred]** stateless | `extra_body` for vLLM-specific params; `user` ignored; Anthropic support is smaller than Anthropic's spec (no `cache_control`/`anthropic-beta` handling in protocol/serving code **[inferred]**) |
| **SGLang** (v0.5.22) | OpenAI Chat Completions, Completions, Embeddings, vision; **Anthropic Messages** (`/v1/messages`, `count_tokens`); **Ollama-compatible** (`/api/chat`, `/api/generate`, `/api/tags`); native SGLang APIs | Self-hosted only | Anthropic/Ollama endpoints stateless; SGLang session/radix caches are server-side optimizations, not API state | Anthropic route registered by default, model-agnostic; tool calls need `--tool-call-parser`, reasoning needs `--reasoning-parser`; **no documented Responses API** |
| **Ollama** (v0.40.11) | Native `/api/chat`, `/api/generate`, `/api/tags`; OpenAI-compatible `/v1/chat/completions` + stateless `/v1/responses`; **Anthropic-compatible `/v1/messages`** (local and cloud) | Local models + Ollama Cloud; no external provider routing | Hosted Responses stateful features explicitly unsupported; local `/v1/responses` stateless | Anthropic subset: no `count_tokens`, `tool_choice`, `metadata`, prompt caching, batches, citations, PDF `document`; base64 images only; thinking budget accepted but not enforced; local server ignores API keys, cloud requires `Authorization: Bearer` |
| **LM Studio** (0.4.x) | OpenAI `/v1/models`, `/v1/responses` (stateful `previous_response_id`), `/v1/chat/completions`, `/v1/completions`, `/v1/embeddings`, `/v1/decisions`; **Anthropic `/v1/messages`**; native `/api/v1/chat` (stateful), model load/unload/download | Local models only | `/v1/responses` and native `/api/v1/chat` are stateful; chat-completions/messages are not | Remote MCP tools supported on Responses and native chat; Responses streaming uses standard `response.*` events |
| **llama.cpp server** (b11870) | OpenAI `/v1/models`, `/v1/completions`, `/v1/chat/completions` (+control), `/v1/responses` (bridged to chat completions, `/v1/responses/input_tokens`), `/v1/embeddings`, **Anthropic `/v1/messages` + `count_tokens`**, TypeSafe-compatible; native `/completion`, `/infill`, `/props`, `/slots`, `/health`, `/metrics` | Local models only | No server-side response store; `previous_response_id` not supported **[inferred]** | `--jinja` required for tools; Anthropic docs say "no strong claims of compatibility"; `extra_params` escape hatch |
| **Amazon Bedrock** | Native InvokeModel (Anthropic Messages JSON at `/model/{model}/invoke[-with-response-stream]`, `anthropic_version` + `anthropic_beta` in body); **Converse/ConverseStream** normalized API; OpenAI Chat Completions on `bedrock-runtime` (`/openai/v1/chat/completions`) and `bedrock-mantle` (`/v1/chat/completions`); OpenAI Responses on both; Claude Code Bedrock format `/model/{model}/invoke*` + `count-tokens` | Models on Bedrock | Responses stateful: 30-day retention by default, `store:false` disables `previous_response_id`; Converse itself stateless | `bedrock-runtime` is recommended: synchronous only, no server-side tools, default Project only, no OpenAI `GET /models`; `bedrock-mantle`: async/background inference, server-side tools, `/v1/models`, legacy path quirks (`/v1` vs `/openai/v1` per model) |
| **Google Cloud Agent Platform** (former Vertex AI) | Anthropic `rawPredict`/`streamRawPredict` (publisher path, `anthropic_version: "vertex-2023-10-16"`, beta in headers), OpenAI-compatible endpoint `…/locations/{loc}/endpoints/openapi` → `/chat/completions`, Gemini `generateContent`/`streamGenerateContent` (legacy but supported), Gemini Interactions API (GA 2026-06, recommended) | Models on Agent Platform | Interactions: stateful by default (`store=false` opt-out), `previous_interaction_id`, `background=true`, execution steps | Claude Code has a dedicated Agent Platform format (`CLAUDE_CODE_USE_VERTEX`); `count-tokens:rawPredict` optional |
| **Azure OpenAI / Foundry** | v1 API `https://{resource}.openai.azure.com/openai/v1/` → `/chat/completions`, `/responses`; legacy deployment-scoped paths with dated `api-version`; Responses recommended; cross-provider models (DeepSeek, Grok) via chat-completions syntax | Models sold by Azure/Foundry | Responses: server-side storage, 30-day retention default; retrieve/delete by id | v1 API removes `api-version`; two host forms `.openai.azure.com` and `.services.ai.azure.com`; deployment-name vs model-name differences remain the classic client bug |

**Reading the matrix.** Two different kinds of "gateway" appear here: *routing gateways* (LiteLLM, OpenRouter, the cloud managed surfaces) that translate between dialects and own policy, and *servers* (vLLM, SGLang, Ollama, LM Studio, llama.cpp) that expose dialects over local weights. Self-hosted servers increasingly avoid translation by serving Anthropic and OpenAI shapes natively — which is exactly why gateways should prefer native passthrough when both ends support it.

---

## (c) Translation fidelity table

Columns describe the *route* (client dialect → upstream surface). Cell values: ✅ preserved; ⚠️ lossy; ❌ unsupported/dropped; — not applicable.

| Feature | Native Anthropic passthrough (vLLM/SGLang/Ollama/LM Studio/llama.cpp, Vertex rawPredict, Bedrock InvokeModel) | Anthropic → OpenAI Responses (LiteLLM `openai/` egress) | Anthropic → Chat Completions (generic OpenAI-compat egress) | Responses → Chat Completions bridge (LiteLLM default for `custom_openai/`; llama.cpp; Ollama) | Bedrock Converse (normalized) | Gemini Interactions ↔ generateContent ↔ OpenAI-compat |
|---|---|---|---|---|---|---|
| Text / system / images | ✅ | ✅ system list→`instructions`; non-text system blocks ❌ | ⚠️ provider-dependent | ✅ images via `input_image` | ⚠️ image bytes/source forms normalized | ✅ but image/PDF block schemas differ; URL vs inline differences |
| PDF / document blocks | ⚠️ only where implemented (Ollama ❌, vLLM/SGLang model-dependent) | ⚠️ mapped to Responses file/input file where supported | ❌/⚠️ | ❌/⚠️ | ⚠️ Converse document block, model-dependent | ⚠️ native `file`/inline data differs from OpenAI `input_file` |
| Tool definitions incl. strict schemas | ⚠️ schemas accepted; strict enforcement depends on server/parser; local servers need tool parser flags | ✅ structural mapping; `strict` translated, defaults false | ⚠️ `strict` often ignored | ⚠️ strict/`defer_loading`/`tool_reference` generally dropped | ⚠️ tool spec normalized; model-native params need `additionalModelRequestFields` | ⚠️ `functionDeclarations` shape differs; strict schema support varies |
| `tool_choice` (auto/any/tool) | ✅ most servers; Ollama ❌ | ✅ `auto→auto`, `any→required`, `tool→function` | ⚠️ "required" semantics vary | ⚠️ | ✅ `auto/any/tool` normalized | ⚠️ `AUTO/ANY/NONE` + `allowedFunctionNames` |
| Tool-call IDs / correlation | ✅ Anthropic `toolu_*` IDs | ⚠️ `tool_use.id`→`call_id`; `tool_result` hoisted to top-level `function_call_output` | ⚠️ `tool_call_id` is provider-generated; correlation preserved but IDs change | ⚠️ IDs regenerated | ⚠️ `toolUseId` round-trip required; mismatches → 400 | ⚠️ IDs/`functionCall`/`functionResponse` correlation differs |
| Plain reasoning text | ✅ if parser configured (vLLM/SGLang need `--reasoning-parser`; Ollama supports thinking blocks) | ⚠️ reasoning → summary/`output_text`-adjacent; summary always `detailed` | ⚠️ provider-specific `reasoning_content` | ❌ most bridges drop reasoning | ⚠️ reasoningContent events, model-specific | ⚠️ thought summaries/steps exposed differently |
| **Reasoning signatures / encrypted reasoning** | ⚠️ only servers that actually sign (see caveat) | ❌ Anthropic thinking signatures cannot be reproduced; returned "thinking" has no signature | ❌ | ❌ | ⚠️ reasoningText signature is a conversation hash; **must replay all prior messages** or Converse errors | ❌ encrypted/opaque reasoning typically dropped |
| Preserved-thinking / thinking binding | ⚠️ depends on upstream; gateway must keep prefix byte-identical | ❌ replaying history through a different provider breaks binding (`bound to a different conversation` 400) | ❌ | ❌ | ⚠️ signature binding to prior messages | ❌ |
| `cache_control` breakpoints + TTL | ⚠️ forwarded by LiteLLM native passthrough after reducing to `{"type":"ephemeral"}` (TTL dropped) unless `cache_control_ttl: true`; vLLM/Ollama ❌/ignored | ❌ dropped by LiteLLM translation | ❌/field ignored | ❌ | ⚠️ LiteLLM translates to Bedrock `cachePoint`; simplified caching (no top-level field) | ⚠️ LiteLLM/OpenRouter can translate breakpoints; TTL is **not** translated across dialects (OpenRouter explicitly drops TTL) |
| Server-side built-in tools (web search, code execution, etc.) | ⚠️ only provider-native ones (Anthropic server tools on Anthropic; Bedrock server tools only on Mantle) | ⚠️ Anthropic `web_search` maps to `web_search_preview`; other built-ins approximated | ❌ | ❌ | ⚠️ guardrails supported; server-side tools not on runtime Converse | ⚠️ Gemini native tools (search/URL context) don't exist on other routes; LiteLLM may intercept and execute some websearch |
| Stateful `previous_response_id` / `previous_interaction_id` | — (Messages API is stateless) | Depends on egress: OpenAI/Azure/Bedrock/LM Studio yes; OpenRouter/Most OpenAI-compat no | ❌ | ❌ unless gateway stores prompts (`store_prompts_in_spend_logs`) or upstream supports it | — (Converse stateless) | Interactions stateful by default; bridging to `/responses` (LiteLLM) loses interaction-step semantics |
| Streaming event semantics | ✅ native SSE (`message_start`…`message_stop`) where implemented; Bedrock InvokeModel uses binary event stream | ⚠️ Responses semantic events re-encoded as Anthropic events; `ping` may need synthesis | ⚠️ chunk deltas have no semantic types; tool/thinking deltas inferred | ⚠️ re-chunked; `response.output_text.delta` synthesized from deltas | ⚠️ `ConverseStream` typed events; binary event stream, not SSE | ⚠️ Interactions "steps" vs generateContent chunks vs OpenAI deltas — three incompatible event models |
| Mid-conversation `system` messages | ✅ sent as-is | ✅ where target supports; LiteLLM places them stably for prefix binding | ⚠️ often hoisted to top-level system (breaks prefix) | ⚠️ | ❌ Bedrock rejects `role:"system"` inside `messages` on every model | ⚠️ |
| Beta headers + paired body fields | ✅ forward `anthropic-beta`/`anthropic-version` verbatim; local servers generally ignore them | ❌ header/body pairing broken → 400 | ❌ | ❌ | ⚠️ `anthropic_beta` lives in body for InvokeModel; Converse has no beta mechanism | ❌ Gemini/OpenAI betas differ entirely |
| Usage / cache accounting | ⚠️ Anthropic fields only if upstream emits them; local servers often approximate tokens | ✅ normalized to `prompt_tokens`/`completion_tokens`; `cache_creation_input_tokens` Anthropic-only | ⚠️ cache fields missing | ⚠️ | ⚠️ normalized `usage` (input/output only) | ⚠️ token/cache fields differ; reasoning-token accounting inconsistent |
| `stop_sequences` / `top_k` / `speed` | ✅ mostly; Ollama ✅ stop/top_k; `speed` Anthropic-only | ❌ dropped silently by LiteLLM Responses mapping | ⚠️ `stop` mapped; `top_k` non-standard | ⚠️ | ⚠️ `stopSequences`, extra params via `additionalModelRequestFields` | ⚠️ `stopSequences`/`topK` exist in Gemini; not portable |
| Structured output strictness | ⚠️ model/server dependent (vLLM structured outputs, SGLang, llama.cpp grammar) | ⚠️ `output_config.format` → `text.format` JSON schema; `strict` defaults false | ⚠️ `response_format` support varies | ⚠️ | ⚠️ model-specific `additionalModelRequestFields` | ⚠️ `responseSchema` vs OpenAI `json_schema` vs Anthropic `output_format` |

**Documented, quotable losses:**

- LiteLLM's Anthropic→Responses mapping drops `stop_sequences`, `top_k`, and `speed` silently; maps `thinking.budget_tokens` to coarse effort buckets (`>=10000 high`, `>=5000 medium`, `>=2000 low`, `<2000 minimal`), always sets reasoning summary to `detailed`, replays Anthropic thinking blocks as `output_text` (signature lost), and joins system text blocks with `\n` ([LiteLLM mapping doc](https://docs.litellm.ai/docs/anthropic_unified/messages_to_responses_mapping)).
- LiteLLM's fallback cannot decrypt reasoning produced by another deployment/provider: it drops Responses reasoning items except readable summaries and drops Anthropic thinking blocks whole, so the target reasons fresh; this applies to every fallback type and to mid-stream failover, with `encrypted_content_affinity` recommended for ordered cross-provider groups ([LiteLLM reliability docs](https://docs.litellm.ai/docs/proxy/reliability)).
- LiteLLM's native `/v1/messages` passthrough reduces every `cache_control` to `{"type":"ephemeral"}` to survive strict Anthropic implementations (`cache_control.ttl: 1h is not supported`), unless `cache_control_ttl: true` is set ([LiteLLM native passthrough](https://docs.litellm.ai/docs/anthropic_unified/native_passthrough)).
- OpenRouter's Responses API rejects `store: true` and non-null `previous_response_id` with `400`; cross-format cache markers are translated but TTLs are not (`cache_control.ttl` dropped toward OpenAI, `prompt_cache_breakpoint.ttl` dropped toward Anthropic/Google) ([OpenRouter Responses](https://openrouter.ai/docs/api_reference/responses/overview), [caching](https://openrouter.ai/docs/guides/best-practices/prompt-caching)).
- Bedrock Converse returns a normalized envelope and drops most model-native fields by default (request them via `additionalModelResponseFieldPaths`); reasoning `signature` is a conversation hash that must be replayed with **all** previous messages or the request errors ([Bedrock Converse](https://docs.aws.amazon.com/bedrock/latest/userguide/conversation-inference.html)).
- Ollama's Anthropic compatibility lacks `count_tokens`, `tool_choice`, `metadata`, prompt caching (`cache_control`), batches, citations, and PDF documents; `thinking.budget_tokens` is advisory ([Ollama Anthropic compatibility](https://docs.ollama.com/api/anthropic-compatibility)).

---

### D1. 400-class translation failures (most frequent, least obvious)

1. **Unpaired beta header/body field.** A gateway that forwards `context_management`, `output_config`, `strict`/`defer_loading` tool fields, or `thinking.type:"adaptive"` without the matching `anthropic-beta` — or forwards an Anthropic-format body to Bedrock/Agent Platform — gets `400 Extra inputs are not permitted` or an "unexpected field" error. Claude Code's gateway guide lists the exact pairs and symptoms, and notes the client auto-retries without the pair only when it can match the upstream error wording ([Claude Code gateway protocol](https://code.claude.com/docs/en/llm-gateway-protocol)).
2. **Mid-conversation `system` role.** `Unexpected role "system"` on Bedrock Converse (rejects it on every model); Claude Code retries with the text in user messages only if the error is forwarded verbatim ([same doc](https://code.claude.com/docs/en/llm-gateway-protocol)). LiteLLM has explicit, prefix-stable placement logic for `supports_mid_conversation_system` models ([LiteLLM preserved thinking](https://docs.litellm.ai/docs/providers/anthropic_preserved_thinking)).
3. **Preserved-thinking binding failures.** `bound to a different conversation` when `system`, `tools`, or earlier messages changed between turns — exactly what happens when a gateway rewrites prefixes or reorders `system` blocks, or when a conversation is replayed through a different provider after a fallback. The Claude Code guide documents its own recovery (drop earlier thinking blocks and retry).
4. **Stateless upstream meets stateful client.** `store:true` / `previous_response_id` on OpenRouter Responses → 400; `previous_response_id` on an OpenAI-compatible server that doesn't persist responses → 400 (LiteLLM's native `/v1/responses` passthrough returns the upstream error unchanged) ([LiteLLM Responses passthrough](https://docs.litellm.ai/docs/response_api_native_passthrough)).
5. **Strict `cache_control` TTL.** Anthropic-compatible servers that don't implement `ttl` reject `{"type":"ephemeral","ttl":"1h"}`; this is why LiteLLM's passthrough strips TTL by default.
6. **Model-ID / path mismatches.** Bedrock: foundation model ID vs system inference profile (`global.`/`us.` prefixes); Mantle per-model `/v1` vs `/openai/v1` prefix; Azure: deployment name vs model name; Vertex: publisher model path vs endpoint ID; LM Studio/Ollama: arbitrary model IDs. Symptom: 400/404 that looks like protocol failure but is naming.
7. **Auth dialect confusion.** Anthropic SDK sends `x-api-key` (+ `anthropic-version`); OpenAI SDK sends `Authorization: Bearer`; Bedrock uses SigV4 or Bedrock API keys; Ollama Cloud requires Bearer and rejects pre-`2023-06-01` versions; Claude Code with claude.ai login needs the OAuth `anthropic-beta` value forwarded or requests fail 401. Gateways that normalize auth must also normalize the beta/OAuth headers.
8. **`drop_params` surprises.** LiteLLM raises by default on unsupported OpenAI params; OpenAI-compatible clients that rely on silent drops will break at the gateway, not at the provider ([LiteLLM drop_params](https://docs.litellm.ai/docs/completion/drop_params)).

### D2. Streaming / proxy failures

- **SSE buffering.** Any reverse proxy (nginx `proxy_buffering on`, compression, buffering load balancers) that buffers `text/event-stream` makes the agent appear to stall. Claude Code treats a body that ends cleanly before final `message_delta`/`message_stop` as a dropped connection and shows an incomplete-response error.
- **Missing keep-alive pings.** Claude Code aborts a stream when no bytes arrive within its streaming idle watchdog; during long thinking pauses the only bytes may be upstream `ping` events. Gateways that strip/buffer pings — or translate from a non-SSE upstream such as Bedrock's binary event stream without synthesizing pings — trip the timeout. Serving stacks are aware of this: vLLM v0.29.0 added SSE keep-alive comments for idle streams, and clients should verify that the chosen gateway forwards them.
- **Content-Type rewrites.** Bedrock InvokeModel streaming is `application/vnd.amazon.eventstream`; converting it to SSE or rewriting `Content-Type` breaks clients (e.g., Claude Code in Bedrock format). Conversely Anthropic-format gateways must return `text/event-stream`.
- **Transport details.** Compression can interact badly with flushing; missing `Transfer-Encoding: chunked`/wrong `Content-Length`, HTTP/2 ALPN negotiation, and intermediate proxies that re-buffer can all produce stalls that reproduce only in production. LiteLLM notes its outbound calls default to HTTP/1.1 (aiohttp) and that `litellm_settings.http2: true` switches to httpx for h2 negotiation ([LiteLLM server tuning](https://docs.litellm.ai/docs/proxy/server_tuning)); inbound HTTP/2 requires Hypercorn.
- **Timeouts.** LiteLLM's default request timeout is 600s; reasoning models with 1M-context turns and long thinking frequently exceed client default timeouts (SGLang's Claude Code guide recommends `API_TIMEOUT_MS=3000000`). Gateway timeouts must exceed the longest single-stream duration, and Kubernetes `terminationGracePeriodSeconds` must exceed the proxy timeout.
- **Idempotency/retry duplication.** Mid-stream failover can replay a request whose visible deltas were already emitted; clients must treat a retried stream as a new attempt, and gateways should not silently stitch two streams.

### D3. State and routing failures

- **Fallback changes the reasoning context.** Any fallback across models/providers/API keys loses encrypted reasoning or thinking signatures; the target reasons fresh. This is documented LiteLLM behavior, not a bug, and it is the most under-appreciated multi-provider routing cost for coding agents.
- **Cache-affinity loss.** Load-balancing across providers (including multiple keys for one model) destroys provider-side prompt caches; OpenRouter mitigates with provider sticky routing keyed on `cache_control`/`prompt_cache_key`, LiteLLM with deployment/session affinity modes.
- **Semantic streaming downgrades.** Bridging Responses→Chat Completions re-chunks typed events (`response.output_text.delta`, reasoning-summary parts) into plain deltas, so tool-progress and reasoning-part boundaries disappear.
- **Data residency drift.** Bedrock global inference profiles may store Responses state in any commercial region they route to (use geographic profiles); OpenRouter in-region routing requires the `eu.`/`us.` host; Azure and Bedrock retain responses 30 days by default unless `store:false` or account retention policy says otherwise.

---

## (e) Selection criteria and recommendations

Selection criteria for a gateway (in priority order for coding agents): (1) native passthrough for the client dialect you standardize on (Anthropic Messages for Claude Code; Responses for Codex-style agents); (2) explicit, configurable statefulness; (3) faithful reasoning/thinking/cache handling (byte-preserving, not "close enough"); (4) streaming correctness (no buffering, pings, correct content types, timeouts); (5) routing/fallback with predictable semantics and observability of the decision; (6) cost/token accounting including cache reads/writes and reasoning tokens; (7) auth/tenancy/data-residency; (8) deployability (self-hosted vs SaaS), version cadence, and error fidelity (upstream error bodies must reach the client unmodified for client-side recovery logic).

**Ten practical recommendations**

1. **Pick one client dialect and prefer native upstream support.** If your agent speaks Anthropic Messages, route to `/v1/messages` wherever it exists natively (Anthropic, vLLM, SGLang, Ollama, LM Studio, llama.cpp, Vertex `rawPredict`, Bedrock InvokeModel) and enable LiteLLM's `model_info.supported_endpoints: ["/v1/messages"]` passthrough for OpenAI-compatible servers that serve it. Avoid double translation (Anthropic→OpenAI→Anthropic).
2. **Standardize on Responses, not Chat Completions, for new OpenAI-shaped agents.** Enable native `/v1/responses` passthrough (`supported_endpoints: ["/v1/responses"]`, LiteLLM ≥1.101.0) when the upstream serves it; accept the known bridge losses only for models with no Responses surface.
3. **Treat reasoning state as opaque bytes.** Never re-serialize thinking blocks, signatures, or `reasoning.encrypted_content`; replay exactly what the model returned, append-only. Pin `system`, `tools`, prompt-template versions, and MCP tool lists for the lifetime of a conversation — prefix changes cause binding drops or 400s and destroy cache hits.
4. **Forward beta header/body pairs verbatim, and don't rewrite request bodies.** Inspect but do not modify; a gateway that redacts/reorders bodies breaks both beta pairing and thinking binding. When the upstream lags, set `CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS=1` on the client rather than stripping fields in the proxy.
5. **Engineer streaming explicitly.** Disable proxy buffering and compression on SSE paths, ensure event ordering through `message_delta`/`message_stop`, synthesize pings for translated streams, keep Bedrock `application/vnd.amazon.eventstream` intact for Bedrock-format clients, set idle/read timeouts well above worst-case thinking time, and test over the same HTTP version and load balancer as production.
6. **Make state a routing constraint.** Decide per deployment whether `store`/`previous_response_id`/`previous_interaction_id` is supported; never silently ignore a `store` request. For stateless-only upstreams (OpenRouter Responses), either send full history or use a gateway-side store (LiteLLM `store_prompts_in_spend_logs`).
7. **Budget for fallback loss.** Prefer same-protocol, same-provider fallbacks first (optionally with `encrypted_content_affinity`); expect a fresh reasoning turn after any cross-provider fallback and design retry/cost budgets accordingly. Keep upstream error bodies unmodified so clients can run their own capability-rejection retries.
8. **Normalize observability, not semantics.** Emit request IDs, chosen deployment, translation decisions, cache read/write tokens, and reasoning-token usage at the gateway; forward upstream usage fields too (Anthropic `cache_creation_input_tokens`/`cache_read_input_tokens`, Responses `cached_tokens`, reasoning details). Don't let cost dashboards become the only record of which protocol actually ran.
9. **Handle cache markers per route.** `cache_control` is portable only where the adapter implements it: LiteLLM reduces TTL to ephemeral (enable `cache_control_ttl` when supported) and translates markers to Bedrock `cachePoint`/Gemini `cachedContents`/OpenAI breakpoints; OpenRouter translates block markers across dialects but explicitly drops TTL. TTL portability is therefore dialect- and adapter-dependent — verify it rather than assuming it. Keep breakpoints stable and aligned with prefixes.
10. **Pin versions and test the fidelity matrix.** Behavior is version-dependent (vLLM's Responses surface changed materially across v0.29–v0.32; LiteLLM passthrough opt-ins landed in v1.92.0/v1.101.0; SGLang/Ollama Anthropic support is new). Maintain golden requests per feature per route and assert expected preserved/lossy/unsupported outcomes as part of gateway upgrades.

---

## (f) Version-dependent claims, uncertainty, and source index

**Uncertain / inferred (flagged):**
- vLLM's Anthropic protocol has no `cache_control` field and its serving layer does not reference `anthropic-beta`; therefore cache markers and beta features are most likely accepted-and-ignored rather than honored. **[inferred from `vllm/entrypoints/anthropic/protocol.py` and `serving.py` on main, 2026-10-25]**
- SGLang v0.5.22 documents OpenAI Chat Completions, Anthropic Messages, and Ollama-compatible APIs but no Responses API; absence from docs is not proof of absence in code. **[inferred]**
- LM Studio's exact current version was not determined; the /v1/responses + stateful chat capability set is documented for the 0.4.x line. **[unverified version]**
- OpenRouter does not document Gemini `generateContent` or Interactions ingress; treat as unsupported. **[inferred from docs index]**
- Gemini Interactions is documented GA (2026-06) on the Gemini Developer API; availability on Google Cloud Agent Platform was not independently confirmed in this pass. **[unverified]**
- llama.cpp's `/v1/responses` "works by converting Responses request into Chat Completions request"; statefulness beyond that is not claimed. **[inferred]**

**Source index (all accessed 2026-10-25; raw copies under `research/sources/gateway/`):**

LiteLLM (v1.106.3, docs last_updated 2026-10-24):
- https://docs.litellm.ai/llms.txt (documentation index) · https://docs.litellm.ai/docs/proxy/configs · https://docs.litellm.ai/docs/proxy/reliability
- https://docs.litellm.ai/docs/response_api · https://docs.litellm.ai/docs/response_api_native_passthrough.md
- https://docs.litellm.ai/docs/anthropic_unified/index.md · https://docs.litellm.ai/docs/anthropic_unified/messages_to_responses_mapping.md · https://docs.litellm.ai/docs/anthropic_unified/native_passthrough.md
- https://docs.litellm.ai/docs/generateContent.md · https://docs.litellm.ai/docs/interactions.md · https://docs.litellm.ai/docs/bedrock_converse.md
- https://docs.litellm.ai/docs/reasoning_content · https://docs.litellm.ai/docs/completion/drop_params.md · https://docs.litellm.ai/docs/completion/prompt_caching.md
- https://docs.litellm.ai/docs/providers/anthropic_preserved_thinking.md · https://docs.litellm.ai/docs/proxy/server_tuning.md · https://docs.litellm.ai/docs/pass_through/anthropic_completion

OpenRouter (docs live, no version):
- https://openrouter.ai/docs/llms.txt · https://openrouter.ai/docs/quickstart.md · https://openrouter.ai/docs/guides/overview/models.md
- https://openrouter.ai/docs/api_reference/responses/overview.md · https://openrouter.ai/docs/api_reference/responses/reasoning.md · https://openrouter.ai/docs/api_reference/responses/tool-calling.md
- https://openrouter.ai/docs/api/api-reference/anthropic-messages/create-a-message.md · https://openrouter.ai/docs/guides/community/anthropic-agent-sdk.md
- https://openrouter.ai/docs/guides/routing/provider-selection.md · https://openrouter.ai/docs/guides/routing/model-fallbacks.md
- https://openrouter.ai/docs/guides/best-practices/reasoning-tokens.md · https://openrouter.ai/docs/guides/best-practices/prompt-caching.md

vLLM (v0.32.0, 2026-10-18; docs "latest" = dev preview):
- https://docs.vllm.ai/en/latest/serving/online_serving/ · https://docs.vllm.ai/en/latest/serving/openai_compatible_server/ (redirects)
- https://github.com/vllm-project/vllm/blob/main/docs/serving/online_serving/openai_compatible_server.md · https://github.com/vllm-project/vllm/blob/main/docs/serving/online_serving/README.md
- https://github.com/vllm-project/vllm/blob/main/vllm/entrypoints/anthropic/protocol.py · .../serving.py
- https://docs.vllm.ai/en/latest/api/vllm/entrypoints/anthropic/api_router/ · https://api.github.com/repos/vllm-project/vllm/releases

SGLang (v0.5.22, 2026-10-19):
- https://docs.sglang.io/llms.txt · https://docs.sglang.io/docs/basic_usage/anthropic_api · https://docs.sglang.io/docs/basic_usage/openai_api_completions · https://docs.sglang.io/docs/basic_usage/ollama_api · https://docs.sglang.io/docs/basic_usage/overview

Ollama (v0.40.11, 2026-10-24):
- https://docs.ollama.com/api/openai-compatibility.md · https://docs.ollama.com/api/anthropic-compatibility.md · https://docs.ollama.com/api/introduction.md · https://api.github.com/repos/ollama/ollama/releases

LM Studio (0.4.x):
- https://lmstudio.ai/docs/developer/rest.md · https://lmstudio.ai/docs/developer/openai-compat.md · https://lmstudio.ai/docs/developer/openai-compat/responses.md · https://lmstudio.ai/docs/developer/openai-compat/chat-completions.md · https://lmstudio.ai/docs/developer/anthropic-compat/messages.md · https://lmstudio.ai/docs/developer/rest/stateful-chats.md · https://lmstudio.ai/blog/openresponses

llama.cpp (b11870, 2026-10-25):
- https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md

Cloud surfaces:
- https://docs.aws.amazon.com/bedrock/latest/userguide/conversation-inference.html (Converse) · https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters-anthropic-claude-messages.html (InvokeModel Anthropic) · https://docs.aws.amazon.com/bedrock/latest/userguide/bedrock-mantle.html (Responses API, runtime vs mantle) · https://docs.aws.amazon.com/bedrock/latest/userguide/inference-chat-completions.html (OpenAI Chat Completions)
- https://cloud.google.com/vertex-ai/generative-ai/docs/partner-models/claude/use-claude (rawPredict, `vertex-2023-10-16`) · https://cloud.google.com/vertex-ai/generative-ai/docs/start/openai (`/endpoints/openapi`) · https://ai.google.dev/gemini-api/docs/interactions-overview (Interactions GA 2026-06) · https://ai.google.dev/gemini-api/docs/generate-content/text-generation (legacy generateContent)
- https://learn.microsoft.com/en-us/azure/ai-foundry/openai/api-version-lifecycle (v1 API) · https://learn.microsoft.com/en-us/azure/ai-foundry/openai/how-to/responses (Responses, 30-day retention)

Client-side compatibility requirements:
- https://code.claude.com/docs/en/llm-gateway-protocol (Claude Code gateway compatibility guide: endpoints, headers, streaming, beta pass-through, preserved thinking, model discovery) · https://code.claude.com/docs/en/llm-gateway-connect · https://code.claude.com/docs/en/env-vars (`CLAUDE_CODE_ATTRIBUTION_HEADER`, timeouts, capability overrides)