Nautili
Documentation

Self-Hosting

What the Helm chart brings, the settings that matter, the state store, the replica rule, optional ingress and the upgrade operator.

Self-Hosting

Two of the three deployment methods -- Helm and Docker Compose -- run the gateway in your own environment, and they are the production default for one reason: prompts never leave your network.

Take the install command from the install flow -- it comes pre-filled with your team id, auth token, backend endpoint and installation id. This page explains what that command brings with it and which settings you may want to change.

Required settings

SettingEnvironment variableMeaning
config.teamIDTEAM_IDYour team. Required.
config.backendEndpointBACKEND_ENDPOINTThe backend endpoint the gateway reports to. Required.
config.authTokenAUTH_TOKENThe installation's token. The chart creates a Secret for it.
config.installationIdINSTALLATION_IDThe stable id everything about this deployment is filed under.
config.installationNameINSTALLATION_NAMEA human-readable label, for example prod-us-east.

The installation id must not change. It is the key the backend files this deployment's health, its dashboard-managed configuration, its operator state, its version pin and every telemetry row under.

Leave it empty and the chart derives one from your team id, the release namespace and the release name -- inputs that are identical across replicas and survive restarts and upgrades. Anything you supply wins over the derivation. Because the derivation cannot see the cluster, set it explicitly if you run the same release name in the same namespace in two clusters.

Governance

Every installation boots governed: governance enforcing, credential forwarding on and issued-key (sk-dz-) detection on, which is what lets keyed traffic reach the product surfaces. The chart exposes that posture as one switch plus a few siblings.

SettingDefaultRole
governance.enabledtrueRenders GOVERNANCE_MODE=enforce, GOVERNANCE_BYOK_FORWARD=true and GOVERNANCE_ISSUED_KEYS=true together. false renders an ungoverned gateway: none of the three, passthrough surfaces only.
governance.sharedDecisionCachetrueShare authorize decisions across replicas through the governance Redis, so a fleet costs the backend one round trip per key, model and provider per cache TTL.
governance.staleSubstitutefalseLet a stale-served allow substitute the team credential for an issued key during a backend outage, bounded by the last revoked-key sync.
governance.insecureBackendOKfalsePermit a non-https config.backendEndpoint to carry credential material. Only for a local or in-cluster plaintext backend.
governance.timeout"" (200ms)Bounds each authorize call to the backend. It is inline on every governed request that misses the decision cache, so raising it raises real-traffic latency.

The retired per-knob values governance.mode, governance.issuedKeys and governance.byokForward fail the chart render with a message naming governance.enabled; remove them from any values file you carry forward. A governed gateway also needs a reachable Redis for its cost-bearing counters, so governance.enabled=true refuses to render without the bundled store or an external address.

The gateway binary also has a GOVERNANCE_MODE=shadow, which evaluates every decision and records what it would have done without denying anything. The chart cannot set it: governance.enabled renders enforce and nothing else. A Compose install reaches shadow by editing the GOVERNANCE_MODE environment variable in the generated service; a DevZero-hosted gateway runs whatever mode the backend is configured with (llm_gateway_governance_mode, enforce when empty), which DevZero sets, not the team. In shadow mode a would-deny is forwarded only for traffic that carries its own credential; an issued-key (sk-dz-) would-deny reaches the upstream with no credential and comes back to the caller as a 502 upstream_auth whose message names shadow mode and the deny reason.

Ports

PortPurposeEnvironment variable
8080OpenAI-compatible surface, the main proxy, and the management APILISTEN_ADDR
8081Native Anthropic surfaceANTHROPIC_LISTEN_ADDR
8082Native Gemini surfaceGEMINI_LISTEN_ADDR
8083Native OpenAI / Codex surfaceOPENAI_LISTEN_ADDR
8090Probes and the Prometheus exporter: /health, /ready and /metricsPROBE_LISTEN_ADDR

The chart's service publishes 8080, 8081, 8082 and 8083. The probe port is deliberately not a service port -- it exists so readiness can keep answering during a drain, and Prometheus autodiscovery scrapes /metrics on it by pod IP. Emptying it also turns the exporter off: /health and /ready are mounted on the request listeners too, but /metrics has no second home.

Profiling is bound to loopback on purpose, and that is a security boundary rather than a convenience: a heap profile of this process contains prompts.

The state store

The chart ships two Redis instances by default, and they are separate because their eviction requirements are opposite.

InstanceHoldsEviction policy
Cost-bearing stateGovernance counters, the routing pin, stickiness markers and representation leasesnoeviction
BodiesThe exact-match response cache and retrievable originalsallkeys-lru

On one instance a burst of response bodies fills memory and every subsequent governance increment fails -- body volume knocking over the money-bearing counters.

Replicas

The chart's floor is two replicas, with a disruption budget and a hostname spread. Those three are one decision -- each is unsafe without the others.

More than one replica requires the shared cost-bearing store, and the requirement is enforced twice: the chart refuses to render above one replica with no address configured, and the gateway refuses to boot when it is told it has more than one replica and no live instance was established.

This is the one state path that does not fail open. Per-replica counters, routing pins and leases are not a degraded version of shared ones: each replica would enforce its own share of a spend budget, and one conversation could be handed a different model on every turn.

The disruption budget is expressed as at most one unavailable, never as a minimum available. That form is load-bearing: a minimum-available budget of one against a single replica permanently allows zero evictions, so a node drain hangs and nothing in the error names the gateway. One replica stays a supported configuration for single-node clusters and development installs.

The topology spread defaults to a soft constraint on hostname. A hard constraint, and a second zone-level one, are documented as hardening and are not rendered -- under a hard constraint a cluster whose nodes are all in one zone cannot satisfy a zone spread at all.

Optional ingress

Off by default: self-hosted installs have always handled their own ingress. When you enable it, the chart renders either a standard Ingress or Gateway API routes, and derives four hostnames from the one base host you give it:

HostnameSurfacePort
<host>OpenAI-compatible8080
anthropic-<host>Native Anthropic8081
gemini-<host>Native Gemini8082
openai-<host>Native OpenAI / Codex8083

Four hostnames rather than one host with four paths, because the OpenAI-compatible surface and the OpenAI/Codex native surface serve the same paths to different upstreams. Nothing downstream can tell them apart by path, so hostname or port is the only discriminator.

Wildcard TLS is a trap here. For a base host of llm.example.com the four names are siblings, not children: *.example.com covers all four and *.llm.example.com covers none of them.

Without ingress, clients address the four listener ports directly. In-cluster, that is the gateway's own service address on ports 8080 to 8083.

Draining and probes

EndpointBehaviour
/healthLiveness. Always answers 200, with a body of {"status":"ok"}. Keyed upstream health is published additively on the same response.
/readyReadiness and the only route that reports the drain. {"status":"ready"} with 200 normally, {"status":"draining"} with 503 from the instant the process observes SIGTERM.

Liveness can never fail on upstream state, because failing it during a provider outage would have Kubernetes restart gateways that are working. Folding a degraded upstream into the probe would turn a provider outage into a DevZero outage.

The termination budget is 60 seconds, split as a 5-second pre-stop delay covering asynchronous endpoint removal, then the server drain, then the telemetry flush. The contract is not that nothing is ever lost: the measured 95th-percentile request is protected, the long tail is knowingly cut, and the client is always told.

WebSocket relays are drained separately, because a hijacked connection is outside the HTTP server's in-flight accounting. A long-lived connection is closed promptly with the RFC 6455 Going Away code, which well-behaved clients read as "reconnect"; a turn already streaming finishes inside the same drain budget.

Upgrades

The chart includes an in-cluster upgrade operator, on by default. It polls the backend and performs verified, rollback-capable upgrades of the gateway release.

The backend's release catalog and its stable channel are authoritative for which version a gateway should run. Global admins move the channel; a rollback requires explicit confirmation. Preproduction and production keep independent catalogs and channel pointers, and registering a release does not itself promote it.

Known limitations worth reading first

The gateway ships a provider-limitations document describing, per provider, what does not work over an OpenAI-compatible wire and what is unreliable -- including the cross-provider problem of upstreams silently ignoring parameters they do not support. Read it before assuming a vendor-specific parameter survives a translation.

On this page