Engineering
Best Open Source LLM Gateways in 2026: 5 Options Compared, Plus Where Nautili Fits
Compare five open source LLM gateways in 2026 (LiteLLM, Portkey, Bifrost, Agent Router, Kong) and see where closed-source, self-hostable Nautili fits.

TL;DR
- An open-source LLM gateway is a proxy you run yourself that gives your apps one endpoint for many model providers, with the source code public. This guide reviews five projects (LiteLLM, Portkey, Bifrost, Agent Router and Kong) and then compares Nautili, an LLM gateway that routes, meters and governs each call, on the same criteria, noting that Nautili is self-hostable but not open source.
- Licenses are permissive: LiteLLM (apart from its enterprise directory) and Portkey are MIT, while Bifrost, Agent Router and Kong's open core are Apache 2.0, so you can modify and embed the open code freely, though Kong's AI Proxy Advanced, semantic and PII Sanitizer plugins need an AI license.
- The free tier decides the real cost: LiteLLM keeps SSO beyond five users, SCIM and audit logs behind a license key, and Bifrost keeps SSO, role-based access control and audit logs in its enterprise build.
- Each project has a clear limit: Agent Router leans on Envoy and Kubernetes, Kong's quickstart needs a Konnect token, Portkey lists observability as a hosted feature, and Bifrost keeps guardrails in its enterprise build.
- Self-hosting moves key storage, upgrades, and uptime onto your team, and Nautili is the gateway in this list that routes, governs, and attributes keyed model calls in one place and tracks the rest, because governance applies to keyed traffic only.
Suppose we have a platform team of four at a company with a dozen services calling three model providers. Each service holds its own provider key, nobody can say which team spent what last month, and one provider outage has already taken two products down.
The team decides every model call should go through one place, and they can either write a proxy themselves or adopt an open-source one. This guide gives each project a full entry with its pros, cons, and limitations, so the team can choose without finding a surprise license key six months later.
What Does an Open Source LLM Gateway Do?
An LLM gateway sits between your application and model providers such as OpenAI and Anthropic. The application calls the gateway with one address and one kind of key, and the gateway forwards the request to the right provider. Because every call passes through it, the gateway becomes the single place for keys, logging, limits, and fallbacks.
"Open source" adds one more property: you can read the code and run it on your own infrastructure. That suits teams that cannot send prompts through a third-party service, and teams that want to avoid lock-in to a hosted router.
How to Judge an Open Source AI Gateway

Eight criteria separate these projects: license, language and runtime, deployment effort, provider coverage, keys and budgets, observability, guardrails, and what stays paid. Each map to a question a platform team will be asked later, and the tables below score every project on all eight.
Open Source LLM Gateways Compared Across All Eight Criteria
Two tables cover the five open source projects, with Nautili added so you can read it against the rest. A dash means the project's own page does not state that point.
The first table covers license, language, and how you run each project:
| Project | License | Language | How you run it |
|---|---|---|---|
| LiteLLM | LicenseMIT | LanguagePython | How you run itSelf-hosted proxy plus SDK (Rust core in beta, Python SDK) |
| Portkey | LicenseMIT | LanguageTypeScript | How you run itnpx, Docker, Node or Cloudflare Workers |
| Bifrost | LicenseApache 2.0 | LanguageGo | How you run itnpx or a single Docker container |
| Agent Router | LicenseApache 2.0 | LanguageGo | How you run itaigw run standalone, or Kubernetes with Envoy Gateway |
| Kong | LicenseApache 2.0 | Language- | How you run itSelf-hosted data planes or Kubernetes, managed through Konnect or decK |
| Nautili | LicenseNot published as open source | Language- | How you run itHelm on Kubernetes, Docker Compose, or Hosted Gateway |
The second table covers providers, budgets, observability, guardrails, and what stays paid:
| Project | Providers | Budgets | Observability | Guardrails | What stays paid |
|---|---|---|---|---|---|
| LiteLLM | Providers100+ | BudgetsBudgets and rate limits | ObservabilitySpend tracking, logging callbacks | GuardrailsFramework with Presidio masking | What stays paidSSO beyond 5 users, SCIM, audit logs, role-based access |
| Portkey | Providers45+ | BudgetsNot listed | ObservabilityMarked as a hosted feature | Guardrails40+ pre-built | What stays paidPII redaction, access control, key RBAC |
| Bifrost | Providers1000+ models | BudgetsHierarchical budgets | ObservabilityBuilt in | GuardrailsEnterprise build | What stays paidSSO, RBAC, clustering, audit logs, guardrails |
| Agent Router | Providers22 | BudgetsQuotas | ObservabilityUsage attribution | Guardrails- | What stays paidNot described |
| Kong | ProvidersSeveral named | BudgetsAI Consumer Groups (licensed) | ObservabilityLLM metrics, OpenTelemetry | GuardrailsPrompt Guard; Sanitizer (licensed) | What stays paidMost advanced AI plugins are Enterprise or Konnect |
| Nautili | Providers500+ models across 80+ providers, with direct connectors for OpenAI, Anthropic, Mistral, Bedrock and OpenRouter | BudgetsGuardrail spend ceiling | ObservabilityTraces, sessions, attribution | GuardrailsDetectors with Flag only, Redact, Block | What stays paid4% platform fee on pay-as-you-go |
The Five Open Source LLM Gateways Reviewed in Full
Each entry follows the same order: who it suits, what it does, its pros, its cons and limitations, what it costs, and a verdict.
1. LiteLLM: Best for Python Teams That Want the Widest Provider List

Best for: Python teams that want virtual keys, budgets, and a long provider list in one proxy.
LiteLLM describes itself as "a self-hosted AI Gateway (proxy)" and an open-source Python SDK, released under MIT except for the code in its enterprise directory. It routes to more than 100 providers through one OpenAI-compatible API, and its repository now describes the project as a Rust core with a Python SDK, though the Rust AI Gateway is in beta and opt-in, with Python still handling auth, routing and callbacks.
Key functionalities:
- Virtual keys, budgets and rate limits, spend tracking and an admin UI are in the open-source tier.
- The same page lists a guardrails framework with custom guardrails and Presidio for PII masking, plus an MCP gateway.
Pros:
- The open tier is broad: keys, budgets, and spend tracking work without a license key.
- A Python service is easy for most teams to read, extend, and debug.
- One OpenAI-compatible API reaches 100+ providers.
Cons and limitations:
- SSO is free for up to five users; larger teams need an enterprise license.
- SCIM, audit logs, role-based access, key rotation, JWT authentication, and secret manager integrations also need a license key, as do tag-based budgets and per-team guardrails.
Pricing: the gateway is free under MIT. The enterprise license price is not published on its docs page, and a 30-day enterprise trial is offered.
Verdict: a strong default for Python teams, as long as you check your SSO and audit needs against the license key list first.
2. Portkey: Easy to Try, but Dormant Since the Acquisition

Best for: teams that want to try a gateway in minutes or at the edge and accept that the open source repo has gone quiet since the acquisition.
Portkey open sourced its gateway on Jan 9, 2024, written in TypeScript, and its repository carries an MIT license with a quick start of npx @portkey-ai/gateway. Palo Alto Networks closed its acquisition of Portkey on May 29, 2026, and the product now appears as Prisma AIRS AI Gateway, so the repository still carries the MIT license, but its last commit landed on 25 May 2026 and its latest release is v1.15.2 from January, so treat it as dormant since the acquisition.
Key functionalities:
- The README lists fallbacks, retries, load balancing, conditional routing, and request timeouts.
- Simple and semantic caching, virtual keys, and an MCP gateway are listed alongside them.
Pros:
- A single npx command starts it, and Docker, Node, Cloudflare Workers and an AWS EC2 template are listed as other ways to deploy.
- Guardrails are included in the open gateway, and you can bring your own.
- The README still announces Gateway 2.0 as a pre-release.
Cons and limitations:
- The repository page does not list budgets.
- Observability and prompt template management are marked as hosted-app features, so a self-hosted setup needs another tool for traces.
- The enterprise version adds PII redaction, access control and professional support, and the page gives several different provider counts, so check that yours are listed.
Pricing: the gateway is free under MIT. Hosted and enterprise pricing is not published on the repository page.
Verdict: a fast way to try a gateway rather than a pick for new production work, with the caveats that the open source repo is dormant since the acquisition, that the project now sits under Palo Alto's Prisma AIRS brand and that traces and budgets need a second look.
3. Bifrost: Best for Teams Wanting a Go Gateway With Budgets Built In

Best for: teams that want a compiled gateway and per-team budgets without paying for SSO.
Bifrost is built by Maxim, runs on Go, and states Apache 2.0. Its maker's own article lists virtual keys, hierarchical budgets and rate limits, semantic caching and a basic MCP client and server as open source, while the governed MCP gateway is enterprise.
Key functionalities:
- An OpenAI-compatible API reaches 1000+ models, according to Maxim.
- It starts with
npx -y @maximhq/bifrostor a single Docker container.
Pros:
- Hierarchical budgets are in the open tier, so a per-team spend limit needs no license.
- Maxim says Go keeps per-request cost low under high concurrency and reports 11 microseconds of overhead per request at 5,000 requests per second in its own benchmark on a t3.xlarge instance.
Cons and limitations:
- The enterprise build adds SSO, role-based access control, adaptive load balancing, cluster mode, audit logs, vault integration, VPC deployment and guardrails, so a team that needs SSO or an audit trail does not get them in the open tier.
- Nearly every public source on Bifrost comes from Maxim itself, and the speed figure comes from its own benchmark, so test it yourself.
Pricing: the open tier is free under Apache 2.0, and Maxim offers a 14-day trial of Bifrost Enterprise. The enterprise price is not published.
Verdict: the best fit when per-team budgets are the main need, and SSO is not.
4. Agent Router: Best for Kubernetes Teams Already Running Envoy

Best for: platform teams that want AI traffic to follow the same policies as the rest of their Envoy traffic.
Agent Router, formerly Envoy AI Gateway, is an open-source project built on Envoy Gateway that handles request traffic from application clients to generative AI services. It keeps the Apache 2.0 license, and its home page lists routing to 22 providers. The project moved to its own site and joined the Agentic AI Foundation in September 2026, with the same maintainers.
Key functionalities:
- One OpenAI-compatible API covers hosted providers, self-hosted inference, and MCP servers.
- Platform teams manage credentials, routing, quotas, failover, and usage attribution through Envoy and Envoy Gateway.
Pros:
- It reuses the Envoy policies your platform team already operates.
- The
aigw runcommand runs it standalone on a laptop with a provider key, so you can try it before touching Kubernetes.
Cons and limitations:
- The full deployment needs Kubernetes with Envoy Gateway, so teams without that experience will find other options faster to start.
- The README does not mention a dashboard.
- The rename and site move mean older tutorials and links may still use the Envoy AI Gateway name.
Pricing: free under Apache 2.0, with no paid tier from the project itself, while Tetrate sells a hosted Agent Router Service and Agent Router Enterprise, an enterprise build on top of it.
Verdict: a natural choice for teams that already run Envoy, and a heavy one for teams that do not.
5. Kong AI Gateway: Best for Teams Already Running Kong for APIs

Best for: teams that operate Kong already and want LLM traffic to follow the same gateway.
Kong's gateway is licensed under Apache 2.0, and its AI Gateway page lists LLM, MCP and A2A traffic through one endpoint with routing, load balancing, and failover.
Key functionalities:
- With a license (Enterprise or Konnect), AI Consumer Groups cover team or tier budgets, alongside AI Rate Limiting Advanced and model cost management.
- The open source guardrail is AI Prompt Guard, while AI Semantic Prompt Guard, AI Sanitizer for PII redaction, and cloud guardrails from AWS, Azure, and Google Cloud need a license.
- Observability lists audit logs, LLM metrics, and OpenTelemetry attributes.
Pros:
- AI traffic follows the same plugin and policy model as your other APIs.
- Data planes can run self-hosted or on Kubernetes.
Cons and limitations:
- Only some AI plugins are free to use: AI Proxy, AI Prompt Guard, AI Prompt Decorator, AI Prompt Template and the request and response transformers carry no license label, while AI Proxy Advanced, AI Rate Limiting Advanced, the semantic cache and semantic guard plugins, the PII Sanitizer, the cloud guardrail plugins and the MCP and A2A plugins are marked "AI License Required". Kong's plugin hub lists a license label on each plugin page, so check it on every plugin you plan to use.
- The quickstart requires a Konnect access token.
- Kong AI Gateway 2.0 is a Konnect control plane with its own entity model, while the plugins still run on self-hosted Kong 3.x, and 2.0 dropped ai-proxy from its bundle in favour of the Enterprise ai-proxy-advanced, so read Kong's upgrade notes first.
Pricing: the gateway is free under Apache 2.0, and Konnect pricing is not published on the AI Gateway page.
Verdict: the strongest fit for teams that already run Kong, with a licensing check before you rely on a specific control.
Other Open Source Gateways Worth Knowing About
Apache APISIX is the one to add to a shortlist: it is an Apache Software Foundation project whose AI plugins (ai-proxy, ai-rate-limiting and ai-prompt-guard) are Apache 2.0 with no enterprise tier, so it suits teams that already run APISIX as their API gateway.
Two other names no longer make good new picks: TensorZero was archived on June 12, 2026, and Helicone was acquired by Mintlify on March 3, 2026 and is in maintenance mode.
What Self-Hosting an LLM Gateway Really Costs You
The license is free, but running the gateway is work. You own uptime because the gateway sits in the path of every call, key storage because provider keys now live in your infrastructure, upgrades as providers change request formats, and the SSO, SCIM and audit logs that may sit behind a paid license.
Teams with strict data rules often accept that work, because self-hosting keeps traffic inside their own infrastructure.
How Nautili Compares With the Open Source Gateways
Nautili is an LLM gateway you point an app or coding tool at by changing one base URL and one key. It picks the model for each request, records latency and cost per call, applies the limits you set, and puts routing, keys, guardrails and attribution behind that one endpoint. It is not published as open source, so it sits outside the five entries above, but it competes with them on the self-hosted side.
Best for: teams that want routing, keys, spend ceilings, guardrails and per-team attribution in one self-hostable gateway.
Key functionalities:
- One base URL and one key move an app or coding tool onto Nautili, and it works with the OpenAI, Anthropic and Gemini SDKs, the Vercel AI SDK, Claude Code, Codex, Cursor and Continue.dev. Direct connectors cover OpenAI, Anthropic, Mistral, Bedrock and OpenRouter, with 500+ models across 80+ providers.
- Routing profiles set the default model, up to four fallback models and a provider ranking, with an opt-in auto mode that pins its first pick for the session.
- Traces list every request with latency, cost, tokens and cache outcome, and trials test a candidate model on live traffic before anything is promoted.
- Issued
sk-dz-keys put traffic under governance, with key custody set to managed or bring-your-own provider key. - A guardrail sets a spend ceiling, request and token limits, and content detectors.
- Attribution reports spend by person, team, department, service and seat.
Pros:
- Requests that name no model follow the profile's default and fallbacks, so swapping a model needs no code change.
- Once accrued spend reaches a guardrail's limit, the gateway refuses further requests with a 429
budget_exhaustederror (400 on the Anthropic surface) until the window resets. - Detectors can run in Flag only mode, which records a match without changing the request, so you can test a rule before it blocks.
Cons and limitations:
- Governance applies to keyed traffic, meaning requests that carry an issued key. Requests that use your own provider key, a Claude seat token or a ChatGPT login are forwarded as-is, with no routing profile or budget enforcement. The gateway still strips the
X-Dz-*headers, addsanthropic-versionwhen it is missing and records the request, and a seat guardrail can still apply a model allowlist. Content rules cover keyed traffic only and inspect requests, not model output, by design. - Coding tools usually land on passthrough, so for them you get telemetry and per-developer attribution but no budgets, rate limits or routing profiles.
- A routing profile never replaces a model the request names, and auto mode works only on profiles that enable it, pinning its pick only when the request carries a session id.
- Nautili is closed source, so you cannot read or fork its code.
Pricing: a Demo tier gives a new account a free starter credit to try Nautili on the Shared Gateway, with a 7-day activity log but no budgets or per-key attribution, while pay-as-you-go, the plan the account moves to once it adds credits and the one that includes self-hosting, adds a 4% platform fee with tokens passed through at provider list price (the first $50K a month of bring-your-own-key traffic carries no fee). Enterprise pricing is custom, and the Hosted Gateway option is arranged with the Nautili team.
Verdict: the pick for teams that want routing, spend control and attribution in one self-hostable gateway, and a second tool beside a code-level gateway for teams that need an open license.

How Routing Profiles Choose the Model for Each Request
A routing profile is a named bundle that sets a default model, up to four fallback models and a provider ranking by cost, latency, or throughput. Its default model applies only when a request names no model, while the provider ranking, allow and deny lists and fallbacks still apply to named models, and a model the request names is never swapped for another.
Profiles bind to a team, a person or an API key, and the per-request override in the X-Dz-Router header sits on top of them, merged field by field. Auto mode is opt-in per profile, so an override that asks for auto on a profile that has not enabled it is refused with routing_override_not_permitted instead of a silent switch.
The LLM routing profiles guide covers the full ladder.
How an Issued Key Puts Traffic Under Governance
An issued key is the line between governed traffic and passthrough. When you create a key, you set a name, an expiration, a credit limit with a reset interval, and custody: managed means a provider credential Nautili supplies for the team, and BYOK means your own provider credential.


How a Guardrail Sets a Spend Ceiling
A guardrail is created in a five-step wizard: identity and custody, limits, access, content filters, and review. The limits step sets a spend ceiling in USD, in total or per period, along with request and token limits per minute.

How Flag Only Mode Tests a Guardrail Before It Blocks
Each content detector in the content filters step has four settings: Off, Flag only, Redact, and Block. Flag only records a match without changing the request, so you can watch what a rule would catch on real traffic before you let it block.

How Attribution Shows Who Spent What by Team
Attribution answers the question that started this guide, which is who spent what. The People page lists every member with team, department, requests, and spend.

The Teams page rolls the same numbers up per team, with sessions and the share of spend each team holds:

The Overview page adds the totals: requests, active users, total tokens, retry rate, and spend by provider for the selected window.

How Nautili Installs Next to Your Own Infrastructure
The install wizard recommends the Shared Gateway that every team already has, and labels the self-hosted option (Helm on Kubernetes or Docker Compose) as advanced. A Hosted Gateway that Nautili runs for your team is a per-team proof-of-concept entitlement, and self-hosting starts on pay-as-you-go.

The LLM governance guide explains how keys, guardrails, and usage fields work in detail. For a hosted-router comparison, see Nautili vs OpenRouter.
Which Open Source Gateway Fits Which Team
| If your team | Start with | Why |
|---|---|---|
| Wants the longest provider list in Python | Start withLiteLLM | WhyMore than 100 providers behind one OpenAI-compatible API |
| Wants a one-command trial or edge deployment | Start withPortkey | WhyOne npx command, plus Cloudflare Workers, but dormant since the acquisition |
| Needs per-team budgets and no enterprise SSO | Start withBifrost | WhyHierarchical budgets in the open tier |
| Runs Kubernetes and Envoy already | Start withAgent Router | WhyAI traffic follows the same policies as the rest |
| Already runs Kong for APIs | Start withKong | WhyBudgets and guardrails sit beside the gateway you operate |
| Wants routing, spend ceilings and attribution in one gateway | Start withNautili | WhyRouting profiles, keys, guardrails, and attribution in one gateway |
How to Pick an Open Source Gateway for Your Team
Start from the control you cannot do without, such as per-team budgets, SSO or audit logs, because each project draws that line differently. Then run a two-week pilot with one service and one provider before you migrate the rest, and check the paid-tier list of your top two projects against your security checklist.
If the control you cannot do without is routing, spend ceilings and per-team attribution in one place, Nautili covers all three behind one base URL and one key. Its Shared Gateway is already set up for every team, so an issued key is enough to send a first request without installing anything.
FAQs
What is an open source LLM gateway?
It is a proxy you can run yourself that gives applications one endpoint for several model providers, and whose source code is public. It usually adds keys, budgets, fallbacks and logging.
Which open source LLM gateway is best?
It depends on your runtime and your controls. LiteLLM suits Python teams that want the widest provider list, Portkey is easy to try but dormant since the acquisition, so it is a trial pick rather than a production pick, and Agent Router suits Kubernetes platform teams.
Is LiteLLM free?
The gateway is open source under MIT apart from its enterprise directory, per its docs. SSO is free for up to five users, and larger teams plus features such as SCIM and audit logs need a license key, and a 30-day enterprise trial is available.
How does an open source AI gateway differ from OpenRouter?
An open source gateway runs in your own infrastructure, and you manage the keys, while OpenRouter is a hosted router you call over the internet. The trade-off is control and data location against the operational work you take on.
Can Nautili be self-hosted?
Yes, Nautili runs on Helm or Docker Compose in your own infrastructure, and the install wizard labels that option advanced. The gateway caches control-plane verdicts for 30 seconds and serves a cached verdict for up to 5 minutes if the control plane is unreachable. On a cache miss, the gateway asks the control plane before forwarding the request, and prompts never pass through the control plane. It is not published as open source, so the Nautili team sets the license.
Written by
Nautili team


