How It Works

The request path

Each request is served against one immutable snapshot of the gateway's runtime configuration, and follows this order:

1. Read and bound the body. Normalize the surface and resolve identity, model, installation, issued key and billing regime. Keyed requests routed through a connector pass the router stage here, resolving the routing profile and producing a routing decision record.
2. Apply the governance gate -- on the decided model, before any cache is consulted.
3. Check the caches. The exact-match cache, and the semantic cache where it is permitted. Coding-client traffic can bypass caches.
4. Apply the spend gates. On a provider-bound request, the rate, token and spend ceilings, and resolve the forwarding credential.
5. Apply transformations. Deterministic lossless transforms always; content-dropping transforms only where they have been promoted and their rollout stage permits. Retrievable originals may be preserved for eligible transforms.
6. Forward to the provider and parse either the complete response or the streaming metadata.
7. Account and record. Accrue actual usage and cost, write eligible cache entries, and enqueue telemetry without blocking the response.

What the control plane decides

The gateway asks the control plane four kinds of question, and caches each answer behind a freshness window:

Every one of these degrades rather than fails. A control-plane blip is served from the cached answer; a stale answer still routes; an answer that cannot be established at all is refused explicitly rather than defaulted, wherever defaulting would silently change where your traffic goes or who pays for it.

What comes back on the response

Keyed inference responses carry two join keys:

X-Cache is stamped MISS before the cache is consulted and overwritten by whichever hit path serves, so the value is exhaustive -- you never have to read an absent header as a miss.

Caching

Caches are keyed on the decided model. A request served by auto mode has no decided model until the pick is made, so delegated turns bypass the exact and semantic caches -- auto costs one uncached turn per session, not an uncached conversation, provided the client supplies a session id and the pin can be read. Without either, every turn delegates and the whole conversation is uncached.

Failure behaviour

Keyed traffic egresses through one connector upstream, so its outage has no alternative path. The gateway therefore carries an explicit failure posture:

• A bounded retry budget and a circuit breaker, installed as the proxy's transport so retries are structurally confined to the window before any body is streamed.
• Retries fire only for connect failures, the upstream's own request timeout, and the upstream's own 5xx. Throttles, provider errors and caller-side rejections are never retried.
• There is no queueing and no emergency direct-provider egress.

Every failure is classified into a provider-neutral taxonomy that records where the failure happened -- upstream unreachable, the upstream's own error, a provider behind it, a mid-stream break, an upstream cap, or the gateway's own gate. Callers receive a stable provider-neutral error envelope with the upstream's detail attached under dz_upstream, never the upstream's own message.

A mid-stream failure cannot change a status that has already been committed, so it is reported in-band as a terminal SSE error frame rather than as a silent truncation.

Health and readiness

The gateway publishes two separate probe endpoints, and the difference matters if you are operating it yourself:

Both are also served on a probe-only listener that is shut down last, after the drain and the telemetry flush.