RAG, fine-tuning, or just a better prompt?

Almost every AI automation request we get arrives with the architecture already chosen. "We need a RAG system." "We want to fine-tune a model on our data." Rarely: "we want to stop answering the same forty questions by hand."

The last one is the actual requirement. The first two are guesses at how to meet it, and they are usually more expensive than the problem deserves.

Here is the order we work through, cheapest first. It is deliberately boring.

Start by assuming you need none of it

A surprising share of "AI projects" are answered by a well-constructed prompt with the relevant material pasted into it.

Modern models take very large inputs. If the knowledge your task depends on is a handful of documents - a price list, a returns policy, an onboarding checklist, a product spec - you can put the whole thing in the prompt and be done. No vector database, no embedding pipeline, no retrieval quality to tune, no infrastructure to keep alive at three in the morning.

The honest test is a boring one:

Can you paste everything the task needs into a single request, and does it still answer correctly?

If yes, ship that. You can always add retrieval later; you cannot easily remove it once a team has built around it.

This is not a trick to avoid work. It is the same instinct as not adding a cache before you have a performance problem: infrastructure you did not need is not free, it is a permanent maintenance cost paid for a benefit you never measured.

Add retrieval when the corpus stops fitting - or stops holding still

RAG earns its place under two conditions, and they are both about scale of a particular kind.

The corpus is too large to send. Thousands of support tickets, years of contracts, an entire documentation site. You cannot include it all, so something has to select the relevant slice per question. That selection is what retrieval is.

The corpus changes faster than you can redeploy. Pricing that shifts weekly, inventory, policies edited by people who do not ship code. Retrieval reads from a source of truth at request time, so an edit takes effect immediately.

That second point is the one teams underrate. It is not really about model capability at all - it is a content freshness problem, and retrieval is the plumbing that solves it.

What RAG actually costs, and what people forget to budget:

• Ingestion. Getting documents out of wherever they live, in a form worth indexing. PDFs with two-column layouts and scanned tables will consume more of the project than the model work does.
• Chunking. How you split documents decides what can be found. Split a contract mid-clause and the retrieved fragment answers the wrong question with total confidence.
• Evaluation. Without a test set of real questions and correct answers, you cannot tell an improvement from a regression. Most stalled RAG projects stalled here.
• Keeping it in sync. A source document changes; the index has to notice.

None of that is exotic. But it is a system, not a feature, and it should be chosen deliberately rather than by default.

Fine-tune for form, not for facts

Fine-tuning is the option that gets picked for the wrong reason most often. The intuition is: I want the model to know my business, so I will train it on my business.

That is not what fine-tuning is good at. It reliably teaches a model how to respond - house tone, a rigid output schema, a domain's phrasing conventions, a classification scheme specific to your operation. It teaches what is true right now poorly and expensively, because facts baked into weights are frozen at training time and every correction means another training run.

So the split is roughly:

Need	Reach for
The model must know current facts	Retrieval
The model must answer in a fixed shape or voice	Fine-tuning
The task is narrow and the material is small	A good prompt
Consistent structured output	Schema-constrained generation, usually not fine-tuning

That last row matters: teams often fine-tune to get reliable JSON when the API can simply be told to produce a given structure. Try the cheap mechanism first.

Three questions that settle it

When we scope an AI automation, these three decide the architecture. They take an afternoon, not a discovery phase.

1. What does the task need to know, and where does it live? If the answer is "these four documents", you are in prompt territory. If it is "whatever is in the ticket system today", you need retrieval. If it is "general knowledge about our industry", you probably need neither - the model has that.

2. How often does that knowledge change, and who changes it? Weekly edits by a non-technical team is the strongest argument for retrieval there is. Annual edits by an engineer is an argument against.

3. What does being wrong cost? An internal draft assistant can be wrong sometimes; a human reads every output. Something quoting prices to customers cannot. High cost of error does not automatically mean more architecture - often it means narrower scope, a confidence threshold, and a human in the loop. That is a process decision, and it is cheaper than any of the above.

Why we start with an audit rather than a build

The pattern we keep seeing: a team spends a quarter building retrieval infrastructure for a workflow that ran fourteen times last month. The system works. It is also the most expensive way anyone has ever answered fourteen questions.

So the first thing worth doing is not building. It is counting: which repetitive work actually happens, how often, and what each instance costs in someone's time. Most lists come back with a dozen candidates of which two or three are worth automating, and the winners are frequently not the ones that felt most impressive at the start.

That is the shape of our AI automation work - a scoped audit first, then a fixed-price pilot on the one or two processes that justify it, with the architecture chosen after we know the answer rather than before.

If you are staring at a proposal that specifies a vector database before anyone has counted the tickets, that is worth pausing on.