Self-Hosting
What the Helm chart brings, the settings that matter, the state store, the replica rule, optional ingress and the upgrade operator.
Self-Hosting
Two of the three deployment methods -- Helm and Docker Compose -- run the gateway in your own environment, and they are the production default for one reason: prompts never leave your network.
Take the install command from the install flow -- it comes pre-filled with your team id, auth token, backend endpoint and installation id. This page explains what that command brings with it and which settings you may want to change.
Required settings
| Setting | Environment variable | Meaning |
|---|---|---|
config.teamID | TEAM_ID | Your team. Required. |
config.backendEndpoint | BACKEND_ENDPOINT | The backend endpoint the gateway reports to. Required. |
config.authToken | AUTH_TOKEN | The installation's token. The chart creates a Secret for it. |
config.installationId | INSTALLATION_ID | The stable id everything about this deployment is filed under. |
config.installationName | INSTALLATION_NAME | A human-readable label, for example prod-us-east. |
The installation id must not change. It is the key the backend files this deployment's health, its dashboard-managed configuration, its operator state, its version pin and every telemetry row under.
Leave it empty and the chart derives one from your team id, the release namespace and the release name -- inputs that are identical across replicas and survive restarts and upgrades. Anything you supply wins over the derivation. Because the derivation cannot see the cluster, set it explicitly if you run the same release name in the same namespace in two clusters.
Governance
Every installation boots governed: governance enforcing, credential
forwarding on and issued-key (sk-dz-) detection on, which is what lets keyed
traffic reach the product surfaces. The chart exposes
that posture as one switch plus a few siblings.
| Setting | Default | Role |
|---|---|---|
governance.enabled | true | Renders GOVERNANCE_MODE=enforce, GOVERNANCE_BYOK_FORWARD=true and GOVERNANCE_ISSUED_KEYS=true together. false renders an ungoverned gateway: none of the three, passthrough surfaces only. |
governance.sharedDecisionCache | true | Share authorize decisions across replicas through the governance Redis, so a fleet costs the backend one round trip per key, model and provider per cache TTL. |
governance.staleSubstitute | false | Let a stale-served allow substitute the team credential for an issued key during a backend outage, bounded by the last revoked-key sync. |
governance.insecureBackendOK | false | Permit a non-https config.backendEndpoint to carry credential material. Only for a local or in-cluster plaintext backend. |
governance.timeout | "" (200ms) | Bounds each authorize call to the backend. It is inline on every governed request that misses the decision cache, so raising it raises real-traffic latency. |
The retired per-knob values governance.mode, governance.issuedKeys and
governance.byokForward fail the chart render with a message naming
governance.enabled; remove them from any values file you carry forward. A
governed gateway also needs a reachable Redis for its cost-bearing counters,
so governance.enabled=true refuses to render without the bundled store or an
external address.
The gateway binary also has a GOVERNANCE_MODE=shadow, which evaluates every
decision and records what it would have done without denying anything. The
chart cannot set it: governance.enabled renders enforce and nothing else.
A Compose install reaches shadow by editing the GOVERNANCE_MODE environment
variable in the generated service; a DevZero-hosted gateway runs whatever mode
the backend is configured with (llm_gateway_governance_mode, enforce when
empty), which DevZero sets, not the team. In shadow mode a would-deny is
forwarded only for traffic that carries its own credential; an issued-key
(sk-dz-) would-deny reaches the upstream with no credential and comes back to
the caller as a 502 upstream_auth whose message names shadow mode and the
deny reason.
Ports
| Port | Purpose | Environment variable |
|---|---|---|
8080 | OpenAI-compatible surface, the main proxy, and the management API | LISTEN_ADDR |
8081 | Native Anthropic surface | ANTHROPIC_LISTEN_ADDR |
8082 | Native Gemini surface | GEMINI_LISTEN_ADDR |
8083 | Native OpenAI / Codex surface | OPENAI_LISTEN_ADDR |
8090 | Probes and the Prometheus exporter: /health, /ready and /metrics | PROBE_LISTEN_ADDR |
The chart's service publishes 8080, 8081, 8082 and 8083. The probe port
is deliberately not a service port -- it exists so readiness can keep
answering during a drain, and Prometheus autodiscovery scrapes /metrics on it
by pod IP. Emptying it also turns the exporter off: /health and /ready are
mounted on the request listeners too, but /metrics has no second home.
Profiling is bound to loopback on purpose, and that is a security boundary rather than a convenience: a heap profile of this process contains prompts.
The state store
The chart ships two Redis instances by default, and they are separate because their eviction requirements are opposite.
| Instance | Holds | Eviction policy |
|---|---|---|
| Cost-bearing state | Governance counters, the routing pin, stickiness markers and representation leases | noeviction |
| Bodies | The exact-match response cache and retrievable originals | allkeys-lru |
On one instance a burst of response bodies fills memory and every subsequent governance increment fails -- body volume knocking over the money-bearing counters.
Replicas
The chart's floor is two replicas, with a disruption budget and a hostname spread. Those three are one decision -- each is unsafe without the others.
More than one replica requires the shared cost-bearing store, and the requirement is enforced twice: the chart refuses to render above one replica with no address configured, and the gateway refuses to boot when it is told it has more than one replica and no live instance was established.
This is the one state path that does not fail open. Per-replica counters, routing pins and leases are not a degraded version of shared ones: each replica would enforce its own share of a spend budget, and one conversation could be handed a different model on every turn.
The disruption budget is expressed as at most one unavailable, never as a minimum available. That form is load-bearing: a minimum-available budget of one against a single replica permanently allows zero evictions, so a node drain hangs and nothing in the error names the gateway. One replica stays a supported configuration for single-node clusters and development installs.
The topology spread defaults to a soft constraint on hostname. A hard constraint, and a second zone-level one, are documented as hardening and are not rendered -- under a hard constraint a cluster whose nodes are all in one zone cannot satisfy a zone spread at all.
Optional ingress
Off by default: self-hosted installs have always handled their own ingress. When you enable it, the chart renders either a standard Ingress or Gateway API routes, and derives four hostnames from the one base host you give it:
| Hostname | Surface | Port |
|---|---|---|
<host> | OpenAI-compatible | 8080 |
anthropic-<host> | Native Anthropic | 8081 |
gemini-<host> | Native Gemini | 8082 |
openai-<host> | Native OpenAI / Codex | 8083 |
Four hostnames rather than one host with four paths, because the OpenAI-compatible surface and the OpenAI/Codex native surface serve the same paths to different upstreams. Nothing downstream can tell them apart by path, so hostname or port is the only discriminator.
Wildcard TLS is a trap here. For a base host of llm.example.com the four
names are siblings, not children: *.example.com covers all four and
*.llm.example.com covers none of them.
Without ingress, clients address the four listener ports directly. In-cluster, that is the gateway's own service address on ports 8080 to 8083.
Draining and probes
| Endpoint | Behaviour |
|---|---|
/health | Liveness. Always answers 200, with a body of {"status":"ok"}. Keyed upstream health is published additively on the same response. |
/ready | Readiness and the only route that reports the drain. {"status":"ready"} with 200 normally, {"status":"draining"} with 503 from the instant the process observes SIGTERM. |
Liveness can never fail on upstream state, because failing it during a provider outage would have Kubernetes restart gateways that are working. Folding a degraded upstream into the probe would turn a provider outage into a DevZero outage.
The termination budget is 60 seconds, split as a 5-second pre-stop delay covering asynchronous endpoint removal, then the server drain, then the telemetry flush. The contract is not that nothing is ever lost: the measured 95th-percentile request is protected, the long tail is knowingly cut, and the client is always told.
WebSocket relays are drained separately, because a hijacked connection is outside the HTTP server's in-flight accounting. A long-lived connection is closed promptly with the RFC 6455 Going Away code, which well-behaved clients read as "reconnect"; a turn already streaming finishes inside the same drain budget.
Upgrades
The chart includes an in-cluster upgrade operator, on by default. It polls the backend and performs verified, rollback-capable upgrades of the gateway release.
The backend's release catalog and its stable channel are authoritative
for which version a gateway should run. Global admins move the channel; a
rollback requires explicit confirmation. Preproduction and production keep
independent catalogs and channel pointers, and registering a release does not
itself promote it.
Known limitations worth reading first
The gateway ships a provider-limitations document describing, per provider, what does not work over an OpenAI-compatible wire and what is unreliable -- including the cross-provider problem of upstreams silently ignoring parameters they do not support. Read it before assuming a vendor-specific parameter survives a translation.