DM Explains · No. 033 · Orchestration
The AI gateway: one front door for every model your company calls
As teams wire apps and agents to dozens of model providers, the sprawl of point-to-point connections becomes the real risk. An AI gateway turns that sprawl into one governed chokepoint for cost, security and routing.
· 13 min read
Read me first
An AI gateway genuinely does stop five teams from wiring five separate messes to five different model providers, but it only works if every one of those teams actually routes through it instead of quietly keeping a direct connection on the side.
Illustrative: five apps (support bot, billing agent, coding copilot, search tool, chat assistant) once called providers directly; rerouted through one gateway — 1 each integrates once → 2 the billing agent's call is checked at the door → 3 routed to whichever model answers best, with failover ready → 4 spend logged against a shared ledger, not five bills → 5 prompts and responses screened both ways → 6 the exchange is logged — if it wasn't logged, it didn't happen
A company's five internal teams had each wired their own app straight to their own favourite model provider: a support bot here, a billing agent there, a coding copilot, a search tool, an internal chat assistant, each with its own API keys, its own retry logic, and no shared view of what any of it cost or logged. Nobody could say, on demand, who had sent what to which model last Tuesday.
Without a shared checkpoint, spend spikes go unnoticed until the invoice arrives, one provider's outage takes down every app that called it directly, sensitive data leaks into prompts unfiltered, and when compliance eventually asks for an audit trail, there isn't one.
Everyday picture. The house used to have five separate doors to the street, one per room, each family member coming and going as they pleased. Now there is one front door, and Papa stands there checking who comes in, Meera keeps the shared shopping ledger so nobody overspends the budget unnoticed, and Arjun picks which gate to use for his walk depending on which park is open that day.
Where the picture breaks. A front door only checks who enters and logs comings and goings; it does not also decide which errand each visitor should run next or inspect what they are carrying, which is work a gateway's routing and content-filtering layers do separately.
Why it exists now
The problem is not new, but the scale of it is. Enterprises now call many different model providers from many different applications and agents, which produces sprawling point-to-point integrations, unpredictable costs, inconsistent security controls, and no unified audit trail (CData). Gartner's October 2025 Market Guide for AI Gateways projects that 70% of software engineering teams building multimodel applications will use AI gateways by 2028, up from roughly 25% in 2025 (Gartner) — a jump that tracks how fast organisations are moving from a single pilot project to many production apps hitting many providers at once.
Agentic systems make this worse, not better. An agent making its own tool calls and model calls generates outbound traffic nobody designed to be observed; InfoQ describes this as a "missing layer" problem, where agent and large language model runtime traffic lacks a traffic interceptor and policy engine, leaving rate limits, header injection and unsafe-prompt rejection unenforced (InfoQ). What changed to make a gateway the obvious fix now is simply that there are too many calls, to too many providers, from too many semi-autonomous callers, for anyone to police by hand.
How it works
Illustrative: support bot calls flow in as app traffic → 1 one integration point → 2 service account authenticated → 3 routed to GPT-4o → 4 counted against $500/day budget → 5 guardrail allows benign query → 6 logged with full audit trail → response returned.
Single door
One integration beats five fragile ones
What it is. Apps talk to the gateway once instead of to every provider separately.
Why it exists. An AI gateway is a centralised control layer sitting between AI consumers — applications, agents, copilots — and the models, tools and data systems they use (CData). Every application integrates once with the gateway rather than separately with each provider and data source, so swapping or adding a provider means changing the gateway, not every app.
Where it breaks. If the gateway itself goes down or is misconfigured, every app behind it loses access at once; it concentrates risk even as it reduces complexity.
How that's handled. The gateway is run and monitored like the critical piece of infrastructure it has become, with its own availability targets, not bolted on as an afterthought.
In technical terms
In technical terms: the gateway presents a normalised interface to calling applications while holding the actual provider connections behind it, so API differences between vendors are absorbed in one place.
Who's allowed?
Checked at the door, not in each room
What it is. The gateway decides which user, service or agent may send a request before it goes anywhere.
Why it exists. A production AI gateway governs control domains including authentication and access control alongside routing, rate limiting, data filtering and logging (CData). Centralising this means no calling application has to reimplement its own identity checks per provider.
Where it breaks. This only governs who can ask; it says nothing about whether what they ask is safe or sensible.
How that's handled. Access control is paired with the content-screening layer further along the chain, not treated as sufficient on its own.
In technical terms
In technical terms: the gateway checks caller identity and permissions against policy at the point of entry, before a request is forwarded to any provider or tool.
Pick a route
Same request, whichever model answers best
What it is. Requests are sent to a specific model based on rules like cost, latency or task type, with failover if one provider fails.
Why it exists. The narrower model-routing component inside an AI gateway, often called an LLM gateway, routes requests to one or more model providers through a single endpoint and handles multi-model routing, failover, caching and cost tracking (CData). It supports configurable routing strategies based on criteria like token usage and prompt complexity, with automatic fallback to keep the service available during an outage (AWS Labs).
Where it breaks. Routing logic is only as good as the rules written into it; bad rules route traffic to the wrong model just as confidently as good ones route it to the right one.
How that's handled. Routing rules are reviewed and tested the way any production configuration would be, rather than set once and forgotten.
In technical terms
In technical terms: the routing layer evaluates signals such as token count, prompt complexity and provider health against configured strategies, and swaps to a backup provider automatically when the primary one is unavailable.
Cap the spend
A shared ledger, not five surprise bills
What it is. Usage and spend are capped and attributed per team, project or application.
Why it exists. Gartner's Market Guide notes that AI gateways reduce the risk of unexpected costs from AI providers (Gartner), and gateway-level rate limiting and cost attribution give centralised control over usage limits and detailed cost tracking across teams, projects and applications (AWS Labs).
Where it breaks. Caps prevent overspend but do not evaluate whether the spend is producing useful output; a well-capped but low-value workload still wastes budget, just more slowly.
How that's handled. Cost dashboards are reviewed by the team that owns the budget, not just watched by whoever runs the gateway.
In technical terms
In technical terms: the gateway meters requests and tokens per caller, enforces configured rate limits, and attributes cost to the originating team or project in a shared ledger.
Screen the content
What goes out and what comes back, inspected
What it is. Prompts and responses are filtered for sensitive data, adversarial content and unsafe replies.
Why it exists. Gartner states that AI gateways protect private data in API traffic (Gartner); data access and permission filtering restrict what information can reach a model, while prompt inspection and guardrails screen prompts and responses against policy before they pass through (CData).
Where it breaks. Filtering catches what it is configured to catch; novel attacks or unanticipated data types can slip past rules written for yesterday's risks.
How that's handled. Guardrail rules get updated as new attack patterns surface, the same way any security filter does.
In technical terms
In technical terms: a policy engine evaluates each request and response against rules such as rejecting unsafe prompts or blocking header injection, in addition to enforcing rate limits (InfoQ).
Keep the record
If it wasn't logged, it didn't happen
What it is. Every request, response, cost and policy decision is recorded for later inspection.
Why it exists. Observability and audit logging are listed among the control domains a production AI gateway governs (CData); a traffic interceptor captures all outbound traffic from agents or model runtimes so it can be reconstructed later (InfoQ).
Where it breaks. Logs document what occurred but do not themselves prevent bad outcomes; they are forensic, not protective, unless paired with the earlier controls.
How that's handled. Logs are treated as evidence for compliance and debugging, reviewed routinely rather than only pulled out after an incident.
In technical terms
In technical terms: the gateway persists structured records of each routed call — caller identity, model used, cost, policy outcome — in one place queryable by engineering and compliance teams alike.
Compared with what came before
| Point-to-point integration | AI gateway | |
|---|---|---|
| Who connects to providers | Every app, separately | The gateway, once |
| Cost visibility | Per app, if tracked at all | Centralised, attributed by team |
| Outage handling | App-specific retry code, or none | Shared failover rules |
| Security checks | Reimplemented per app | Enforced centrally before any call |
| Audit trail | Scattered or absent | One queryable record |
| What you give up | Flexibility to bypass shared rules | Nothing structural, but a new dependency on the gateway's own uptime |
The judgement is not that point-to-point calling was foolish; for one app calling one provider it is simpler and faster to ship. The judgement is that it stops scaling the moment a second app, a second provider, or a second compliance requirement shows up, and most organisations now have all three.
Where it breaks
Not a smarter model
A gateway does not improve the underlying quality, reasoning ability or accuracy of a model's outputs, and it does not replace the model evaluation, fine-tuning or prompt engineering work done upstream of it. It is plumbing, not intelligence.
The gateway becomes a single point of failure in exchange for removing many smaller points of failure; if it is misconfigured or overloaded, everything behind it stops. It also governs who can ask and what passes through, not whether the request made business sense in the first place — a well-authenticated, well-logged, badly-conceived query is still a badly-conceived query. And filtering rules, like any security control, are only as current as the last update; a gateway configured against last year's risks is not automatically safe against this year's.
For a single application making occasional calls to one model provider, the overhead of deploying and operating a gateway can outweigh the benefit it provides.
Numbers to carry
| Figure | What it means |
|---|---|
| 70% by 2028, up from roughly 25% in 2025 | Gartner's projected share of multimodel engineering teams using AI gateways (Gartner), as of October 2025 |
| Three deployment approaches | Cloud-managed gateways, universal API gateways, and on-premises solutions are the typical options organisations choose from (AWS Labs) |
| Control domains, six of them | Authentication, routing, rate limiting, data filtering, prompt inspection, and observability, as listed by CData's breakdown (CData) |
Myths worth killing
| Commonly repeated | What is actually true |
|---|---|
| "An AI gateway and an LLM gateway are the same thing" | An LLM gateway is the narrower model-routing layer inside a broader AI gateway, which also governs access control, data permissions and audit logging across tools and data, not just models |
| "A gateway makes the model better" | It does not change reasoning ability or output quality; it routes, caps, filters and logs traffic to and from whatever model is already there |
| "Logging is the same as security" | Logging documents what happened after the fact; it does not itself stop a bad request, which is why it is paired with access control and content filtering, not a substitute for them |
| "Any API gateway will do" | A generic API gateway manages arbitrary backend traffic; an AI gateway is purpose-built for AI-specific concerns like token-based rate limiting, prompt inspection and model failover |
Worked example
- Papa, running engineering, notices the five apps — support bot, billing agent, coding copilot, search tool, internal chat assistant — each hold their own provider keys and nobody can answer what last month's AI spend actually was.
- The team puts a gateway in front of all five, so each app now integrates once with the gateway instead of five times with five providers.
- Meera's billing agent gets a spend cap tied to its project budget; when it approaches the limit, the gateway flags it rather than letting the bill run unseen.
- The coding copilot's requests are routed to whichever provider is fastest for code-shaped prompts that hour, with automatic failover configured for the day a provider has an outage.
- The support bot's prompts, which sometimes contain customer account details, pass through the content-filtering layer before reaching any model.
- Illustrative only: if each of the five apps previously made, say, 10,000 calls a month directly to providers with no shared cap, and the gateway now caps combined usage at a fixed monthly ceiling per team, the company goes from five unmonitored bills to one reviewable ledger — the arithmetic here is illustrative, not measured.
- Six months later, compliance asks who sent what to which model on a specific date; Papa pulls the answer from the gateway's log in minutes instead of chasing five teams for screenshots.
Should you use it?
Worth it when an organisation calls multiple AI model providers across many applications and needs a single place to manage cost, security and routing; when centralised cost attribution and rate limiting are needed to prevent unexpected bills; when compliance or data-sovereignty rules demand audit trails for every AI interaction; or when agentic systems are making enough outbound tool and model calls that policy enforcement in real time becomes necessary. Not worth it for a single application making occasional calls to one provider, where deploying and running a gateway costs more than the sprawl it would prevent.
The cheapest first experiment: route one existing app's provider calls through a gateway configuration without changing the app's code, and compare the cost and log visibility it gains against the apps still calling providers directly.
Say it in the room
"Put a gateway in front of every model call before adding a sixth app, not after the first surprise bill."
Lines to carry
- "One door beats five fragile ones."
- "Checked at the door, not in each room."
- "A shared ledger stops surprise bills."
- "If it wasn't logged, it didn't happen."
- "Plumbing, not a smarter model."
- "Logs are forensic, not protective, on their own."
FAQ
Is an AI gateway the same as an LLM gateway?
No. An LLM gateway is the narrower piece that routes requests to large language model providers, with features like failover and caching. An AI gateway is the broader layer that also handles access control, data permissions and audit logging across models, tools and data sources, with the LLM gateway sitting inside it.
Does a gateway slow down every request?
The dossier does not give a latency figure, so this is worth measuring directly before deciding, since the answer depends on routing configuration and where the gateway is deployed relative to the apps and providers.
Can a gateway stop a model from giving a wrong answer?
No. It does not change reasoning ability or output quality. It can filter unsafe or non-compliant content, cap cost and log the exchange, but the correctness of what a model says is a separate problem solved upstream, through evaluation and prompt work.
What happens if the gateway itself fails?
Every app behind it loses access at once, since all AI traffic is meant to route through it. That is the trade-off for consolidating five fragile integrations into one: it has to be run and monitored like critical infrastructure.
Do small teams need one?
Generally not yet. A single application making occasional calls to one provider usually gets more value from staying simple than from deploying and operating a gateway.
Glossary
- AI gateway — a centralised control layer between applications/agents and the AI models, tools and data they use, handling routing, security and governance.
- LLM gateway — a narrower component inside an AI gateway that routes requests specifically to large language model providers, with features like failover and caching.
- Model routing — the process of deciding which AI model or provider should handle a given request.
- Failover — automatically switching to a backup model or provider if the primary one is unavailable.
- Rate limiting — restricting how many requests or how much usage a user or application can make in a given time.
- Cost attribution — tracking AI usage costs back to the specific team, project or application that generated them.
- Semantic caching — storing and reusing previous model responses for semantically similar requests to save cost and time.
- Guardrails — rules or filters that screen AI prompts and responses for unsafe, adversarial or non-compliant content.
- Observability — the ability to monitor and inspect what is happening inside a system, such as logging AI requests and responses.
- Agentic traffic — network requests generated by autonomous AI agents as they call tools, models and data sources on their own.
- API normalisation — presenting a single, consistent interface to applications even though different AI providers have different underlying APIs.
- Control plane — the part of a system responsible for managing and directing how traffic and operations flow, as opposed to doing the actual processing itself.
Sources
- What Is an AI Gateway, and Why It Matters Now — CData, 2026
- Market Guide for AI Gateways — Gartner, 2025
- How AI Gateways Enable Scalable AI Connectivity — Kong Inc.
- The Missing Layer in AI Infrastructure: Aggregating Agentic Traffic — InfoQ
- LLM Gateway - Generative AI Atlas — AWS Labs