Self-Hosting
What the Helm chart brings, the settings that matter, the state store, the replica rule, optional ingress and the upgrade operator.
Self-Hosting
Three of the four deployment methods run the gateway in your own environment, and they are the production default for one reason: prompts and retrievable originals never leave your network.
Take the install command from the install flow -- it comes pre-filled with your team id, auth token, control-plane endpoint and installation id. This page explains what that command brings with it and which settings you may want to change.
Required settings
| Setting | Environment variable | Meaning |
|---|---|---|
config.teamID | TEAM_ID | Your team. Required. |
config.backendEndpoint | BACKEND_ENDPOINT | The control-plane endpoint the gateway reports to. Required. |
config.authToken | AUTH_TOKEN | The installation's token. The chart creates a Secret for it. |
config.installationId | INSTALLATION_ID | The stable id everything about this deployment is filed under. |
config.installationName | INSTALLATION_NAME | A human-readable label, for example prod-us-east. |
The installation id must not change. It is the key the control plane files this deployment's health, its dashboard-managed configuration, its operator state, its version pin and every telemetry row under.
Leave it empty and the chart derives one from your team id, the release namespace and the release name -- inputs that are identical across replicas and survive restarts and upgrades. Anything you supply wins over the derivation. Because the derivation cannot see the cluster, set it explicitly if you run the same release name in the same namespace in two clusters.
The AI Gateway product switch
Keyed traffic -- the part of the product that reaches the product surfaces -- is off until three settings agree.
| Setting | Environment variable | Role |
|---|---|---|
governance.issuedKeys | GOVERNANCE_ISSUED_KEYS | The AI Gateway product's on-switch. With it off, sk-dz- keys are not detected at all, so there is no keyed traffic to route. |
governance.byokForward | GOVERNANCE_BYOK_FORWARD | Required by issued keys. Lets the gateway resolve and forward a credential. |
governance.mode | GOVERNANCE_MODE | shadow or enforce. Required by both of the above. |
The dependency runs one way: issued keys require a governance mode and
BYOK forwarding; BYOK forwarding requires a governance mode. shadow mode
evaluates every decision and records what it would have done without denying
anything, which is the safe way to introduce governance to live traffic.
Ports
| Port | Purpose | Environment variable |
|---|---|---|
8080 | OpenAI-compatible surface, the main proxy, and the management API | LISTEN_ADDR |
8081 | Native Anthropic surface | ANTHROPIC_LISTEN_ADDR |
8082 | Native Gemini surface | GEMINI_LISTEN_ADDR |
8083 | Native OpenAI / Codex surface | OPENAI_LISTEN_ADDR |
8090 | Probes and the Prometheus exporter: /health, /ready and /metrics | PROBE_LISTEN_ADDR |
The chart's service publishes 8080, 8081, 8082 and 8083. The probe port
is deliberately not a service port -- it exists so readiness can keep
answering during a drain, and Prometheus autodiscovery scrapes /metrics on it
by pod IP. Emptying it also turns the exporter off: /health and /ready are
mounted on the request listeners too, but /metrics has no second home.
Profiling is bound to loopback on purpose, and that is a security boundary rather than a convenience: a heap profile of this process contains prompts.
The state store
The chart ships two Redis instances by default, and they are separate because their eviction requirements are opposite.
| Instance | Holds | Eviction policy |
|---|---|---|
| Cost-bearing state | Governance counters, the routing pin, stickiness markers and representation leases | noeviction |
| Bodies | The exact-match response cache and retrievable originals | allkeys-lru |
On one instance a burst of response bodies fills memory and every subsequent governance increment fails -- body volume knocking over the money-bearing counters.
Replicas
The chart's floor is two replicas, with a disruption budget and a hostname spread. Those three are one decision -- each is unsafe without the others.
More than one replica requires the shared cost-bearing store, and the requirement is enforced twice: the chart refuses to render above one replica with no address configured, and the gateway refuses to boot when it is told it has more than one replica and no live instance was established.
This is the one state path that does not fail open. Per-replica counters, routing pins and leases are not a degraded version of shared ones: each replica would enforce its own share of a spend budget, and one conversation could be handed a different model on every turn.
The disruption budget is expressed as at most one unavailable, never as a minimum available. That form is load-bearing: a minimum-available budget of one against a single replica permanently allows zero evictions, so a node drain hangs and nothing in the error names the gateway. One replica stays a supported configuration for single-node clusters and development installs.
The topology spread defaults to a soft constraint on hostname. A hard constraint, and a second zone-level one, are documented as hardening and are not rendered -- under a hard constraint a cluster whose nodes are all in one zone cannot satisfy a zone spread at all.
Optional ingress
Off by default: self-hosted installs have always handled their own ingress. When you enable it, the chart renders either a standard Ingress or Gateway API routes, and derives four hostnames from the one base host you give it:
| Hostname | Surface | Port |
|---|---|---|
<host> | OpenAI-compatible | 8080 |
anthropic-<host> | Native Anthropic | 8081 |
gemini-<host> | Native Gemini | 8082 |
openai-<host> | Native OpenAI / Codex | 8083 |
Four hostnames rather than one host with four paths, because the OpenAI-compatible surface and the OpenAI/Codex native surface serve the same paths to different upstreams. Nothing downstream can tell them apart by path, so hostname or port is the only discriminator.
Wildcard TLS is a trap here. For a base host of llm.example.com the four
names are siblings, not children: *.example.com covers all four and
*.llm.example.com covers none of them.
Without ingress, clients address the four listener ports directly. In-cluster, that is the gateway's own service address on ports 8080 to 8083.
Draining and probes
| Endpoint | Behaviour |
|---|---|
/health | Liveness. Always answers 200, with a body of {"status":"ok"}. Keyed upstream health is published additively on the same response. |
/ready | Readiness and the only route that reports the drain. {"status":"ready"} with 200 normally, {"status":"draining"} with 503 from the instant the process observes SIGTERM. |
Liveness can never fail on upstream state, because failing it during a provider outage would have Kubernetes restart gateways that are working. Folding a degraded upstream into the probe would turn a provider outage into a DevZero outage.
The termination budget is 60 seconds, split as a 5-second pre-stop delay covering asynchronous endpoint removal, then the server drain, then the telemetry flush. The contract is not that nothing is ever lost: the measured 95th-percentile request is protected, the long tail is knowingly cut, and the client is always told.
WebSocket relays are drained separately, because a hijacked connection is outside the HTTP server's in-flight accounting. A long-lived connection is closed promptly with the RFC 6455 Going Away code, which well-behaved clients read as "reconnect"; a turn already streaming finishes inside the same drain budget.
Upgrades
The chart includes an in-cluster upgrade operator, on by default. It polls the control plane and performs verified, rollback-capable upgrades of the gateway release.
The control plane's release catalog and its stable channel are authoritative
for which version a gateway should run. Global admins move the channel; a
rollback requires explicit confirmation. Preproduction and production keep
independent catalogs and channel pointers, and registering a release does not
itself promote it.
The optional ML sidecar
A local sidecar for compression and embeddings, off by default. Its client fails open to verbatim content on timeout or error, so a sidecar problem degrades optimization rather than failing requests.
Known limitations worth reading first
The gateway ships a provider-limitations document describing, per provider, what does not work over an OpenAI-compatible wire and what is unreliable -- including the cross-provider problem of upstreams silently ignoring parameters they do not support. Read it before assuming a vendor-specific parameter survives a translation.