← Back to Blog
N

Put a Gateway Between Your Agents and Every Model

Ask for the job, not the model. And give every key a budget that actually means something.

When you own the AI budget, you think about it differently very quickly. A runaway agent stops being an interesting anomaly in a dashboard and becomes a line item someone has to explain.

So every agent I run talks to models through one place: a self-hosted LLM gateway. Agents get one OpenAI-compatible endpoint and their own virtual key. Routing, fallback, pricing, and budgets live server-side in a single config file under version control. Swapping a backend or adding a provider is one edit and a redeploy, and every agent picks it up without a client change.

That part is plumbing, and plenty of people have written about it. What I want to write about is the policy you encode into that plumbing, because the policy is where the money is actually saved or lost.

Ask for the job, not the model

The first version of my spend policy was a paragraph of prose. The gist: use the expensive model for hard work, the mid-tier one for ordinary work, the cheap one for bulk. Every agent had it in its instructions. It worked about as well as you would expect a paragraph to work against a system that reads a lot of paragraphs. Agents forget. Context gets compacted. A sub-agent spawned three levels down never saw the paragraph in the first place.

The fix was to stop asking agents to remember the policy and make the policy the interface. The gateway exposes intent aliases: names that describe a kind of work rather than a model. Example names: hard for planning and difficult reasoning, default for ordinary work, batch for summaries and classification, and volume for high-volume passes where nothing much rides on any single answer. The agent asks for the job. The gateway resolves the job to whichever model the policy currently assigns to it.

The tiers are rough price bands, not brands. A frontier model for planning and hard code. A mid-tier model at roughly half that price, which becomes the default for ordinary work. A small model at roughly a quarter. And a very cheap model, often well over an order of magnitude below the mid tier, for passes where volume matters more than nuance.

Two design decisions are worth calling out. First, keep two aliases separate even when they point at the same model today. An alias for hard reasoning and one for large refactors can share a model now and diverge later without touching a single agent. Second, the aliases describe work, not vendors. When a cheaper model gets good enough for summarisation, you change one line and the whole fleet gets cheaper overnight.

The deeper point: "use something cheaper" has to be possible before it can be policy. Until the gateway actually serves a cheaper tier under a name that describes the job, telling agents to be frugal is a wish.

Escalate on difficulty, not importance

This is the rule I find myself re-explaining most, including to myself.

The instinct is to reach for the best model when the stakes are high. A security question, a production incident, something a client will read. But stakes and difficulty are different axes. A serious question with a short, well-defined answer is still a short, well-defined answer, and a mid-tier model handles it fine. Meanwhile, a low-stakes refactor that spans forty files and needs a plan that holds together is genuinely hard, and that is where the top tier earns its price.

Picking by importance feels responsible. It mostly buys anxiety relief at several times the price. Picking by difficulty is the engineering decision.

There is a fair counterargument: on a high-stakes task, the cost of a wrong answer dwarfs the token bill, so why not overspend? Because the better model does not remove the need to check the answer. If a decision matters, it gets reviewed by a human, or by a second model, whichever tier produced it. Spending more on the first pass does not buy you out of the review, and the review is what actually protects you.

It also helps that the capable choice and the frugal choice sometimes converge. Providers regularly ship a new frontier model that is both better and cheaper than the one it replaces. There is no reason to pin an older, pricier model out of habit, so recheck the lineup whenever a provider ships something new.

The most expensive model has no alias

Most gateways can serve at least one model that is far above the rest in price. Often it costs a multiple of your planning tier in both directions, and a hundred times your cheapest tier.

Give it no alias. Not hard, not anything. To reach it, someone has to type its name.

This is a deliberate piece of friction. If the most expensive route sat behind your planning alias, then every agent that asked for planning would quietly spend at the top rate, and no human would ever have made that choice. Leaving it unaliased keeps the expensive call an explicit decision. Reserve it for special projects and genuinely hard planning, and make it absent from the model list of every key that has no reason to call it.

Whatever script mints new keys should enforce the same thing. It should refuse to issue a key without an explicit model list. If you run LiteLLM, this matters more than it looks: an empty model list on a key means every route, including the one you are trying to fence off.

Every key gets a budget

Give each agent its own virtual key, so you can revoke one without disturbing the others. Give each key a hard spend cap over a rolling window. On top of that, set a global monthly ceiling that stops every key at once if the total trips, and an upper bound on how large any single key's budget can be. A key requested without a budget gets the default filled in. A key requested above the bound is rejected.

A common mistake is to start loose. Keys get minted early, with no cap and access to every model, while the paid routes behind the gateway are not wired up or not funded yet. They look harmless because every expensive call fails anyway. Then someone connects the paid provider, and every one of those keys becomes an open line of credit on the same afternoon.

The lesson: an empty account is not a security control. It looks like one right up until the moment you fix the thing you were waiting to fix. Cap keys at the moment you mint them, not the moment they become dangerous. And delete unused keys rather than keeping them around in case. A key for a retired system is a credential with no owner and no reason to exist.

The metering bug that makes every budget fiction

This is the part I think matters most, whichever gateway you run.

A budget is only as good as the meter feeding it. Most gateways know list prices for mainstream models. Point one at a route it cannot price (a custom OpenAI-compatible endpoint is the classic case) and many record the spend as zero rather than refuse. There is no loud warning, just a zero. Your per-key budget then compares a running total of zero against its cap, forever, and never trips. Every budget you configured is fiction, and the dashboard is cheerfully telling you that you have spent nothing.

I found out the honest way: the provider's invoice and the gateway's spend log disagreed, and the gateway was the one that was wrong. Nothing had been hidden. The number was simply never computed, and a dashboard cannot show you a number nobody computed.

The fix is boring: every route in the config carries an explicit price per input token and per output token, whatever your gateway calls those two fields. Paid routes carry the provider's list price on the day you add them. Free-tier routes carry an explicit zero too, so that metering is never the gateway's guess. Treat a route without those two lines as a bug, not a style nit.

There is a related trap at the other end. If the provider account has a prepaid balance, it can run dry before any gateway budget trips; the error is then the vendor's "insufficient credit", not a budget error from the proxy. Knowing which layer stopped you saves a lot of misdirected debugging.

Cross the vendors on big jobs

Because every model sits behind the same endpoint, mixing vendors on one job is cheap to do. On a large piece of work, put the hard half (the plan, the tricky backend change) on the top tier, and split the rest between two mid-tier models from different vendors at similar prices. One takes the backend, the other the frontend. Different model families make different mistakes, and a second family reading the first one's output catches things a same-family review tends to wave through.

One thing I deliberately do not do is chain paid providers as fallbacks for each other. Fallback chains make sense on free tiers, because free endpoints throttle constantly and degrading gracefully is the whole point there. But a failing paid model should error loudly instead of silently billing a different vendor. I would rather see the failure than discover it on an invoice.

Where I'd start

If you are running more than one agent against paid models, and especially if you own the budget line, this is the order I would do it in.

Put a single gateway in front of every model this week, even if it only fronts one provider today. The value is not the routing. It is having exactly one place where policy, keys, and pricing live, under version control.

Give agents intent names instead of model names. Start with three: one for hard work, one default, one cheap. Make the default the mid tier, not the top.

Leave your most expensive model without an alias. Make reaching it require typing its name, and restrict which keys can reach it at all.

Mint one key per agent, each with a hard budget and an explicit model list, and set a global ceiling above them. Delete keys nobody uses.

Then, before you trust any of those budgets, open your config and check that every single route has explicit per-token costs. Send one request through a route you added yourself and confirm the spend log shows a number above zero. If it shows zero, none of the rest of this list is protecting you yet.

That last check takes five minutes. It is the cheapest insurance in the whole setup, and it is the one most people skip because the dashboard already looks fine.