What an AI Gateway Does, and Why We Built Our Own

What an AI Gateway Does, and Why We Built Our Own 

Every team that adds AI to a product starts the same way. A developer gets access to a model provider, puts the credentials in a config file, and ships. It works. Then a second team does the same, and a third, and a year later nobody can say how many credentials exist, who controls them, what they cost, or which model touched which data. 

An AI gateway fixes that. This article covers what it does, the options on the market, why we built our own, and what ours does today. 

Where a gateway sits 

An AI gateway is a proxy between your applications and the model providers. Every call from an app, an agent, or a developer's coding assistant goes to the gateway first, and the gateway forwards it to a provider. 

Some things have to be true of every call, whichever app made it: who may make it, which credentials it uses, how much it may spend, and what gets recorded. Without a gateway, each application implements those rules itself or doesn't. With one, they live in a single place. 

 

Without a gateway, apps call providers directly. With a gateway, every call passes through one controlled path. 

Without a gateway, every app connects to every provider on its own. With one, every call takes the same path. 

What a gateway does 

A gateway ties each call to a person, team, or project, and holds provider credentials centrally instead of letting them spread across applications and laptops. It routes calls to several providers through one endpoint, and handles load balancing, retries, and fallback when a provider has problems. 

It tracks spend per call and enforces budgets, rate limits, and token quotas. Some gateways also cache similar requests to cut costs. Every call is logged, so usage and cost are something you can query instead of guess. 

Some add guardrails, such as content filters and personal-data masking. Newer ones also mediate access to tools through the Model Context Protocol (MCP), so agents can operate under the same controls as other workloads. 

A gateway doesn't replace testing your prompts, evaluating output quality, or reviewing the code AI writes. It controls and records the traffic. It doesn't judge it. 

The options 

There are several common routes to consider. 
 

Route	Example	Fits when
Open-source proxy you host	LiteLLM	You want control and are willing to run it
Cloud platform gateway	Azure API Management's AI gateway	Your AI workloads already run on that cloud
Edge or hosted service	Cloudflare AI Gateway	You already use Cloudflare and want routing and caching with little setup
Subscription-login proxy	CLIProxyAPI	One developer or a small group wants CLI subscriptions exposed as an API. A compatibility layer, not a governance layer.

 

The table is the short version. Here is how each route works in practice, based on the vendors' documentation. 

How each route works 

LiteLLM is the open-source proxy many teams encounter first. Each user, team, or project can get a virtual key with its own allowed models, budgets, and rate limits. It tracks spend by key, team, user, and tag; handles fallbacks and guardrails; ships logs to tools like Langfuse, Datadog, OpenTelemetry, and Prometheus; and provides an admin UI. It also supports MCP and agent access. 

SSO beyond five users, audit logs, role-based access, and key rotation require the Enterprise license. Because you host it, you also patch it, scale it, and own its uptime. 

Azure API Management's AI gateway is the natural pick when your workloads already run on Azure. It can authenticate with managed identities instead of relying on application-managed credentials, limit tokens per minute by subscription, and emit token metrics to Azure Monitor and Application Insights. It also supports semantic caching backed by an external Redis instance, load balancing with a circuit breaker across model backends, content safety checks, and MCP server support. You configure these capabilities through policies, so expect configuration work rather than a finished AI product. 

Cloudflare AI Gateway is hosted at the edge. You host nothing, and it provides caching, retries, model fallback, and analytics. Per-user tracking can be implemented by attaching custom metadata to each request, and spend limits can be keyed to that metadata. This makes it a practical route to visibility and cost control without operating your own proxy. 

CLIProxyAPI is a different kind of tool, and people often group it with the others. It's an MIT-licensed proxy that exposes OpenAI-, Gemini-, and Claude-compatible endpoints on top of CLI tools. It supports authentication through existing subscription accounts, including ChatGPT Codex, Claude Code, Grok Build, and Gemini through Antigravity, rather than relying exclusively on conventional API credentials. It can also round-robin across several accounts. 

That helps an individual developer expose supported CLI-based workflows through an API. But CLIProxyAPI is built for individual use, not for team governance. It does not provide the same built-in per-user keys, budgets, and access-policy controls expected from a governance-oriented gateway. Its built-in usage statistics were removed in v6.10.0 in favour of third-party add-ons. 

How to choose 

Four questions narrow it down quickly. 

Where do your workloads run? If everything is on Azure, start with API Management. If you already operate another API gateway, assess whether it can cover your AI requirements before adding another service. 

Who will operate it? Self-hosting an open-source proxy can be inexpensive in licensing and expensive in attention. If nobody owns it, a hosted option may win. 

What do you need to attribute spend and usage to? A user is relatively easy to track. A team or project needs appropriate identities, keys, or metadata. A client engagement, with its own approved models, security rules, and budget, requires more deliberate configuration—and not every product supports that model equally well. 

Where must the logs live, and who may read them? Enterprise clients will ask. A hosted gateway puts traffic metadata, and potentially prompts, on infrastructure operated by another party. Understand exactly what is collected, stored, and accessible. 

My view: for a team under about ten people that wants spend control this month, LiteLLM or Cloudflare can get you there quickly. For a company already deep in Azure with a platform team, API Management is the default. The choice gets harder when governance has to follow client contracts, and that was our situation. 

This market changes every quarter. Check each product's current documentation against your requirements before you decide. 

Why we built our own 

Most companies shouldn't. If you run one product on one cloud, start with your cloud's gateway or an open-source proxy. It will cover credentials, budgets, routing, and logging for far less effort than writing your own. 

Our governance model has three layers: 

1. Governed AI policy: client-approved usage, approved tools and providers, usage standards, periodic audits, and AI champions in each team. 

1. Shared AI engineering capacity: shared skills and commands, structured workflows across the software development lifecycle (SDLC), security guardrails, and reusable components. 

1. A centralized AI gateway: a control point that enforces the first two layers and keeps the record. 

The gateway had to sit inside that model and carry it, not next to it. Our own products—AI DevKit, QAKit, Presale Kit, Code Review Agent, and EIS Claw—run on it from the start. It uses the company identity system and role structure we already operate. And the record it keeps is the one we want to be able to show a client, not a general-purpose log adapted after the fact. 

The cost is plain. We own it: patching, uptime, new providers, and new features. A vendor with a bigger team will ship some things faster than we do. We accepted that because fit mattered more to us than speed. Your answer may differ. 

What ours does today 

Identity and access 

People sign in through company SSO (Microsoft Entra ID) and get a built-in role—User, Admin, or Super Admin—or a custom role. Applications use their own service keys. 

Access to models follows identity and policy, not simply whoever happens to hold a credential. 

Provider credentials stay server-side 

Provider credentials stay on the server. We manage them centrally, so they don't need to reach an engineer's machine or a code repository. Central management reduces the number of places where credentials can be exposed and makes access and rotation easier to control. 

One endpoint, multiple providers 

One endpoint reaches Anthropic or OpenAI, with guardrails enforced at the gateway. The products on top don't need to know which provider sits behind them, so adding or swapping a provider can be handled centrally rather than changing every product. 

Guardrails at the gateway 

Routing and logging are only part of the job. The gateway also applies security controls to requests and responses, with each guardrail enabled or disabled independently for each pool. 

A pool is a group of provider accounts and models assigned to a team or client, with its own rules. Different pools can therefore have different guardrails and access policies to reflect the requirements of their workloads. 

The guardrails include: 

• Credential redaction. Detects and removes secrets such as API keys, access tokens, and private keys before a prompt leaves for a provider. Coverage includes credentials associated with AWS, GitHub, GitLab, OpenAI, Anthropic, Slack, Stripe, and more. 

• PII masking. Identifies and replaces sensitive personal information—including email addresses, phone numbers, payment-card details, and national ID numbers—with typed placeholders. It supports Vietnamese formats as well. Instead of exposing an actual email address, for example, the model receives [REDACTED_EMAIL], preserving the sentence's structure while masking the value. Masking can also be applied to the model's response on its way back. 

• Prompt-injection detection. Identifies requests that attempt to override instructions or manipulate the model's behaviour. Depending on the pool's policy, the gateway can warn or block. 

• Tool-call firewall. Blocks tool use on pools that should not have access to tools. 

• Custom sensitive-term redaction. Each pool can have its own list of sensitive terms, such as client names or internal code names. These lists are stored encrypted, and matching terms are redacted before the request is sent to the provider. 

• Context compression. Compresses context to reduce unnecessary token usage while retaining information needed by the task. 

Each guardrail can also run in shadow mode. It inspects real traffic and records what it would have done without changing the request or response. Teams can observe the impact and review potential false positives before enforcing a rule. 

Guardrail decisions are logged without storing the prompt text in those event records. Policy changes are versioned, and the event log can be exported in signed form to a security team's SIEM (Security Information and Event Management system). 

The point is not to turn every control on for every workload. It's to make the controls configurable, observable, and enforceable at the point where traffic passes through. 

Usage, cost, and capacity 

Every call is recorded with its timestamp, user, model, token count, cost, and pool. That's one line per call, written automatically, with cost analytics that break usage down by user and pool. 

The gateway also monitors quota across provider accounts and alerts on quota risk, gateway health, and failure rates. Teams can see capacity constraints, accounts approaching their limits, and relevant availability or quota information before a workload is disrupted. 

It sits underneath every tool, so teams can switch tools while the control point stays in place. Governance that depends on every tool behaving is a hope. Governance below the tools is a mechanism. 

A gateway with no policy enforces nothing in particular. A policy with no gateway is a document. You need both. 

What it doesn't do 

Ours records and routes traffic and enforces configured controls. It doesn't judge whether AI output is correct, and it doesn't prove that a human reviewed anything. 

We keep human review on everything that ships, and that remains a people process. 

It also only sees traffic that goes through it. If a vendor says a gateway "guarantees safe AI," ask what it checks, what it blocks, and what it records. Traceability isn't quality assurance, and guardrails are not a substitute for human judgement. 

Five questions for any gateway, ours or someone else's 

Can you list every model provider used on my project this month? If the list comes from memory, it's a guess. 

Can you show usage by person, not just by team? Team totals hide who did what. 

Where is the record kept, and who can edit it? A log its owner can rewrite proves little. 

What happens when someone uses a tool or provider I haven't approved? A policy says don't. A control determines what happens. 

Can I audit this without asking you to prepare a report? A prepared report is a story about the record. 

If the answers need a meeting to assemble, the control isn't real yet. 

What's coming next 

The record can get finer. Today it tells us who called which model and what it cost. We're planning to work at the prompt level, so the gateway can also understand what each call is for. 

That opens two possibilities. 

The first is observability by task: which work burns the most tokens, which prompts fail or get retried, and where a team pays twice for the same job. 

The second is token optimization. The gateway could select a model based on the task—a smaller one for a rename or a summary, and a stronger one for a difficult debugging session—so engineers don't have to choose by habit. 

Both are planned, not live. 

This also changes what the gateway is. Today it handles metadata. At the prompt level, it would process content, including a client's code and data. The security questions get harder, and any vendor planning this should answer them in writing before the feature ships. 

Our questions are: 

• Does the gateway store metadata only, or prompt content? 

• If it stores content, for how long, in which region, and who can read it? 

• Can a client restrict its project to metadata only? 

• What happens to secrets and personal data that appear in a prompt? 

• Does automatic model selection stay within the models the client has approved? 

• Can a client see why a model was chosen and override it? 

We'll publish the answers before any of this goes live. If they can't go into a contract in plain terms, the feature isn't ready. 

If you're choosing a gateway or working through these questions for your own engineering contracts, we'll walk you through ours and a sample log.