Your own LLM or someone else's API: what AI automation costs to run

The first question about an AI feature is whether it will work. The second, always, is what it will cost to run once it does, and that is the one most proposals answer badly.

There are three ways to run a language model in production. Each has a different cost shape, and the shape matters more than the headline number.

Path one: a hosted API

You call Anthropic, OpenAI or Google, and pay per token. No infrastructure, no GPUs, the newest models the day they ship.

The thing teams miss is how wide the price range is inside one provider. These are Anthropic's published rates per million tokens as of October 2026:

Model	Input	Output
Claude Haiku 5.5	$0.10	$0.50
Claude Sonnet 5.5	$2.00	$10.00
Claude Opus 5.5	$4.00	$20.00
Claude Fable 5.1	$10.00	$50.00

A hundredfold spread on input, and every row is the same API with the same code. Which model you pick changes your bill by two orders of magnitude. Which provider you pick usually changes it by tens of percent. Teams spend weeks on the second question and minutes on the first.

Two more things cut a hosted bill before any architectural decision:

Prompt caching. If every request resends the same long instructions or the same document, you are paying full price for identical tokens each time. Cached reads cost a fraction of that. On a system with a large stable prompt, this alone is often the difference between viable and not.

Batching. Work that does not need an answer this second can go through a batch endpoint at around half price. Overnight classification, enrichment, summarising yesterday's tickets: all of it qualifies, and almost nobody uses it.

Get those two right and the hosted bill frequently stops being the problem you were trying to solve.

Path two: your own model

Open-weights models, from the Llama, Qwen, Mistral and DeepSeek families among others, can be downloaded and run on hardware you control. We have built and deployed these.

You will see this described as "zero token cost". That is literally true and slightly misleading, so here is the honest version.

What goes away: the per-token bill. It does not matter whether you process a thousand requests this month or ten million; nobody is metering you. For a business whose volume is growing, that is the whole argument, because a per-token bill grows exactly in proportion to success.

What replaces it: a GPU bill. A single modern data-centre GPU costs on the order of a dollar or two per hour to rent, so running one continuously is on the order of a thousand dollars a month before a single request arrives. Buy rather than rent and you have swapped it for capital and a machine to look after.

What else replaces it, and this is the part proposals leave out: the engineering. A serving stack, batching and quantisation so the GPU is actually busy, monitoring, a rollback path, somebody to call when it stops at 3am, and an evaluation set so you can tell whether the model you upgraded to is better or just newer. None of that is exotic. All of it is work that the API provider was doing for you.

So the trade is not free against expensive. It is variable against fixed.

Where the break-even sits

The arithmetic is simpler than it looks.

Take your expected monthly tokens, multiply by the hosted rate for the model you would actually use, and compare it with roughly a thousand dollars per GPU per month plus the engineering to run it.

Three things decide the answer.

Volume, and specifically its direction. Steady and low favours the API. High, or growing fast, favours your own. The crossover is a real number you can compute in an afternoon rather than a matter of opinion.

Which model the job needs. This is the one that flips results. A classification or extraction task that runs fine on a small open model is a completely different calculation from one that genuinely needs a frontier model, because the open model may need a fraction of a GPU while the frontier model has no self-hosted equivalent at all.

Whether the data can leave. For some companies this ends the discussion before cost is mentioned, and it is the most common reason our clients self-host. Personal data under GDPR, client confidentiality, a regulator, a contract, or a board that will not approve sending customer records to a third party. Weights on your own hardware in your own jurisdiction is a different conversation with your DPO than a processing agreement.

Three smaller factors that still matter: latency, because a local model on a local network has no internet round trip; rate limits, which your own hardware does not have; and predictability, because a fixed monthly infrastructure line is easier to defend in a budget than a variable one that spikes with a good month.

And one honest counterweight. The best open-weights models have closed much of the gap and are genuinely excellent for most production tasks, but the frontier is still ahead of them. If your task needs the very top of the range, that is an argument for the API, and we will tell you so.

"Can you train it on our database?"

Everyone asks this, and the answer is usually not the one expected.

Fine-tuning teaches a model form: a tone, a format, a classification scheme, a way of structuring an answer. It is excellent at that.

It does not reliably teach facts. If the goal is "answer questions about our documents, our products, our policies", fine-tuning is the expensive way to get a worse result, because the moment a document changes the model is wrong and you are retraining. Retrieval is the right tool: the model looks the answer up at the moment of asking, from a source you control and can update, and it can cite where it came from.

We wrote the full version of this up separately in RAG vs fine-tuning: which one you actually need. The short form: most systems want retrieval, some want fine-tuning on top of it, and knowing which is most of the value an experienced team adds in the first week.

Path three: both, which is where most systems land

In practice the interesting architectures are not pure.

A small self-hosted model handles the high-volume, low-judgement work: classification, routing, extraction, tagging, the millions of calls where the per-token bill would hurt. A hosted frontier model handles the small number of genuinely hard requests where quality decides whether the feature is worth having. Sensitive data is processed locally; everything else takes the cheaper path.

That design is more work to build than either pure option, and for a system of any volume it is usually the one that survives contact with the invoice.

What to do first

Measure before you architect. Count the tokens your workload actually needs. Most estimates are wrong by a factor of ten in one direction or the other, and the whole decision turns on that number.

Take the free wins before the structural ones. Right-size the model, turn on caching, move what can wait to batch. If the bill is still a problem after that, you have a real infrastructure question. Often it is not.

Build an evaluation set early. Without one you cannot tell whether the cheaper model is good enough, which means you cannot safely take any of the savings above. This is the step teams skip and the reason they end up paying frontier prices for classification.

Decide the data question honestly and first. If the data cannot leave your infrastructure, the cost comparison is interesting but not decisive, and the project should be designed around that from day one rather than discovering it at the security review.

---

This is the work we do: AI automation that holds up in production, built by people who will tell you when the cheaper answer is the right one. We start with a fixed-price AI process audit that counts the repetitive work and the tokens, and says which processes justify automating and which do not.

If you have a process you suspect could be automated, tell us what it is and we will run the numbers with you.