Fine-tune, retrieve or just prompt?

Start with prompting. Add RAG when the model needs facts it does not have. Fine-tune only when you need a behavior, such as a strict format, a house style or a narrow skill, that a well-written prompt with examples still cannot deliver reliably.

Fine-tuning vs RAG is the wrong framing, because the three are not rivals. Prompting changes what you ask, RAG (retrieval-augmented generation) changes what the model can see, and fine-tuning changes the model itself. In 2026 the balance tipped toward prompting: million-token context windows became standard, caching made long prompts cheap, and self-serve fine-tuning of closed models began to disappear.

01Fine-tuning vs RAG vs prompting: what each one changes

A handy way to remember it: prompting is a briefing, RAG is an open-book exam, and fine-tuning is training a new habit. Briefings are cheap and flexible. Open books keep answers current and checkable. Habits are powerful, but slow to form and slow to change.

02Start with prompting (it goes further than you think)

Most “the model can’t do this” problems are really briefing problems. A clear goal, the relevant context, a few examples of good output and an explicit format fix more than people expect. Our prompt engineering guide covers the method.

Examples do most of the heavy lifting. Before you consider training anything, try a few-shot prompt like this one:

Sort each support ticket into exactly one category: billing, bug, feature_request or account.
Reply with the category only.

Ticket: "I was charged twice this month."
Category: billing

Ticket: "The export button does nothing on Safari."
Category: bug

Ticket: "Can you add dark mode?"
Category: feature_request

Ticket: "{new ticket}"
Category:

Prompting can carry knowledge too. Paste the relevant document straight into the prompt and the model answers from it, which is often all a small project needs. The next sections show when that stops being enough.

03When RAG wins

RAG searches your documents first and hands the model only the passages that matter. It wins when your knowledge is:

• Too big for the window. Company wikis, ticket histories and product catalogs often outgrow even a million tokens.

• Changing. Prices, policies and stock levels change weekly. Re-indexing one document takes seconds; retraining does not.

• In need of citations. Retrieved passages come with their sources, so readers can check the answer.

• Permissioned. Search can filter by who is asking, so people only get answers from documents they may read.

Research backs the instinct that retrieval beats training for facts. A 2023 study compared RAG with unsupervised fine-tuning (further training on raw text) as ways to add knowledge to a model. RAG consistently won, both for facts the model had seen before and for entirely new ones. The models struggled to learn new facts through fine-tuning, though seeing each fact phrased many ways helped.

For the machinery, see our guides to RAG and to the embeddings that power its search.

04When fine-tuning wins

Fine-tuning teaches behavior, not knowledge. Reach for it when a problem survives a good prompt and you have plenty of examples. Typical wins:

• Strict, consistent output, such as extracting the same 30 fields into the same schema a million times a month.

• A voice or style that is hard to describe but easy to show in hundreds of samples.

• Shorter prompts at high volume. Behavior trained into the weights no longer needs pages of instructions on every request.

• Making a small model do a big model’s job. You train a small, cheap model on a large model’s outputs, a technique called distillation.

Tuning can even make a small model beat a much larger one. In OpenAI’s 2022 InstructGPT study, people preferred answers from a 1.3 billion parameter model fine-tuned with human feedback over answers from the 175 billion parameter GPT-3.

The real cost of fine-tuning is rarely the training bill. It is collecting and cleaning examples, building a test set that proves the tuned model is better, and repeating that work each time the base model is upgraded.

05How long context windows changed the answer in 2026

Three shifts in 2026 moved the line between these approaches.

Huge windows became standard. Claude Opus 5.5, Sonnet 5 and Fable 5.1 accept 1 million tokens at the standard per-token price, roughly 555,000 words by Anthropic’s estimate. OpenAI’s GPT-6 models take up to 922,000 input tokens, though the input price doubles past 272,000. Gemini 3.8 Flash accepts 1,048,576.

Repeated context got cheap. Prompt caching bills repeated input at a fraction of the normal price. On Claude Sonnet 5, a cached token costs $0.20 per million instead of $2. Here is what that means per question:

Writing to the cache costs a little extra the first time: 1.25 times the normal input price for a five-minute cache. Even so, for about 100,000 tokens of material, pasting it all in and caching it costs only about a cent more per question than RAG, and takes an afternoon instead of a week. Near the million-token limit, RAG is still about 18 times cheaper per question.

Bigger windows are not perfect windows, though. Anthropic’s engineers describe context rot: the more tokens in the context, the harder it is for a model to recall any one detail accurately. Past a certain size, sending less but better-chosen text is both cheaper and more accurate.

Closed-model fine-tuning shrank. OpenAI closed its self-serve fine-tuning to newcomers on May 7, 2026, and even active customers lose the ability to start new jobs on January 6, 2027. Google’s Gemini API has had no tunable model since May 2025, though Google’s enterprise cloud platform still offers tuning. Today, open-weight models such as Gemma 4 and gpt-oss, which anyone can download and fine-tune, are the most accessible route.

06Combining them

The best systems usually mix approaches. Some common pairings:

07A four-question decision path

1. 01

   Can a clear prompt with a few examples do it?

   Write the brief, add three to five examples and test on 20 real inputs. If the results are good, stop here.

2. 02

   Does the model need facts it does not have?

   If they fit comfortably in the window and rarely change, paste them in and cache them. If they are large, changing or permissioned, build RAG.

3. 03

   Is the remaining problem behavior rather than knowledge?

   Format drift, off-brand tone and a narrow skill the model keeps missing are fine-tuning problems. Missing facts are not.

4. 04

   Can you prove the change helped?

   Keep a fixed test set and score every version against it. Our guide to testing AI on your own tasks shows how.

08FAQ

Next, see how to cut your AI costs once your approach works, or learn when a small model beats a giant one.