Your AI feature gives wrong answers, ignores your format or sounds nothing like your company. You have three ways to fix that. You can write better instructions (prompting). You can give the model the right documents at question time (retrieval-augmented generation, RAG). Or you can train the model further on your own examples (fine-tuning). Fine-tuning often sounds like the thorough solution. In practice, it takes the most effort to set up and to change — and it often fixes the wrong problem.
This is part 4 of Level 3 (Advanced). You will learn what each method changes, what data and upkeep it needs and how the three work together. A decision flow and a table of examples help you choose. The part builds on prompt design for developers from Level 1 and RAG step by step from Level 2.
Three levers on the same request
Every answer of a language model depends on two things: what the model receives with the request and what it learned in training. The three methods work on different parts of this:
- Prompting changes the instructions: system prompt, rules, examples and output format. The change works from the next request.
- RAG changes the knowledge in the request. Your app searches your documents and adds the most relevant passages to the prompt. The model itself stays the same.
- Fine-tuning changes the model. You continue training it on pairs of input and desired output, and its weights — the numbers it learned in training — change.
This difference explains most decisions. Instructions and knowledge travel with each request, so you can change them at any time. What a model learns through fine-tuning is built into it. It costs no extra tokens per request, but changing it takes a new training run.
Fine-tuning teaches behavior, not facts
A common mistake is to fine-tune a model so that it “knows” your documents. Research shows why this rarely works. A Microsoft study compared two ways of adding knowledge to a model: RAG and fine-tuning on the documents themselves. RAG did better than fine-tuning, both for facts the model had seen before and for entirely new ones (Source: Ovadia et al., “Fine-Tuning or Retrieval?”, EMNLP 2024). A second study found that models learn new facts slowly through fine-tuning — and once they have learned them, they tend to hallucinate more (Source: Gekhman et al., “Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?”, EMNLP 2024).
Fine-tuning is strong where prompts reach their limits:
- A fixed format or style that must hold across thousands of requests, for example a ticket summary that always has the same five fields.
- A narrow, repetitive task, such as sorting texts into categories or extracting fields. Here, a small fine-tuned model can replace a large general one.
- Shorter prompts: behavior that needed long instructions and many examples moves into the model, so each request uses fewer tokens.
- Behavior that is hard to describe but easy to show, such as the way your support team declines a request politely.
You do not need huge amounts of data, but you need good data. In the LIMA study, researchers at Meta fine-tuned a large model on only 1,000 carefully chosen examples and got strong results (Source: Zhou et al., “LIMA: Less Is More for Alignment”, NeurIPS 2023). Clean, consistent examples matter more than a large number of messy ones.
What each method needs and costs
The three methods differ in data, effort and upkeep:
| Prompting | RAG | Fine-tuning | |
|---|---|---|---|
| What changes | Instructions in each request | Knowledge in each request | The model’s weights |
| Data you need | A clear task and a few examples | Your documents, kept up to date | Many consistent pairs of input and desired output |
| Effort to start | Low | Medium: index and search pipeline | High: collect data, train, evaluate |
| Running cost | Long prompts use more tokens | Search plus longer prompts | Training runs; hosting for open-weight models |
| When things change | Edit the prompt | Update the index | Train again |
| Can cite sources | Only from text in the prompt | Yes, from the retrieved passages | No |
| Typical failure | Long prompts with conflicting rules | Wrong or missing passages | Learns errors from the data; loses general skills |
The last row needs a word of explanation. A model that is trained hard on one narrow task can get worse at others. This is called catastrophic forgetting. LoRA, the method that trains only small adapter matrices, keeps more of the base model’s general skills than full fine-tuning — but it also learns less of the new task (Source: Biderman et al., “LoRA Learns Less and Forgets Less”, TMLR 2024).
The effort also differs over time. Prompting stays cheap to change. RAG needs steady care for the index. Fine-tuning needs the most work at the start, and the work comes back whenever your data or the base model changes:
For RAG, you need a vector database; for fine-tuning an open-weight model, you need GPUs. In our comparison, Pinecone is a vector database for search and retrieval, and Runpod rents GPUs for training and running models, according to the vendors. Devs.ai is a platform where teams build AI apps and agents on their own data:
Short descriptions from our tool comparison, based on the vendors' information — not tested by ABXK.
Start with prompting, then diagnose
The order matters more than the method. Begin with the cheapest lever, and move on only when your tests show a specific problem. Before you change anything, build a small eval set: typical inputs with the answers you expect, as described in test AI features with evals. Without it, you cannot tell whether a change helped.
Then collect answers that went wrong and sort them by cause. Each cause points to one lever:
- Instructions ignored or misunderstood? Improve the prompt: clearer rules, examples, an output schema. Many APIs can also enforce a JSON format, as shown in structured output and function calling.
- Facts missing, outdated or private? Add RAG. The model needs the information at question time.
- Format or style drifts across many requests, although the prompt is clear? Now fine-tuning is worth a test.
- Good answers, but too slow or too expensive? Fine-tune a smaller model for the task, so it no longer needs the long prompt.
- Wrong reasoning even with good information? The task may be too hard for the model. Compare stronger models in our LLM database.
Fine-tuning in practice: closed or open models
There are two ways to fine-tune. Some providers offer fine-tuning of selected closed models as a service: you upload examples, and the provider trains and hosts your version. With an open-weight model, you train it yourself — usually with LoRA on rented GPUs — and you host the result. Part 3 of this level shows how to size and rent GPUs for that.
Smaller open-weight models are common starting points, because they train quickly and fit on a single rented GPU with QLoRA. The number with the “B” in each name is roughly the size in billions of parameters. A few examples from our LLM database:
| Model | Released | Weights | ECI | Rank |
|---|---|---|---|---|
| Llama 3.1-8B Meta AI | Jul 23, 2024 | open | 116.6 | 215 |
| Qwen3.5-9B Alibaba | Feb 24, 2026 | open | 139.4 | 124 |
| gpt-oss-20b OpenAI | Aug 5, 2025 | open | 137.8 | 132 |
| Qwen 3.8 27B Alibaba | Aug 14, 2026 | open | 151.6 | 52 |
| Gemma 4 31B IT Google DeepMind | Apr 2, 2026 | open | 142.9 | 102 |
ECI = Epoch Capabilities Index, one overall score across many benchmarks; rank among all models with an ECI. Data: Epoch AI, retrieved on October 17, 2026, licensed CC BY 4.0. All models: LLM database.
Check two things before you start. The license decides whether you may fine-tune the model and use the result commercially. And the base model’s own test results set the ceiling: fine-tuning shapes how a model uses what it already learned. It does not turn a weak model into a strong one.
A special case is distillation: you train a small model on the answers of a large one, so the small model takes over a narrow task at lower cost. Check the terms of the large model’s provider first, because some do not allow using their outputs to train other models.
Combine them: one assistant, three layers
The methods are not rivals. Many production systems use all three, each for the part it does best. A useful rule: put what changes often into the layers that are easy to change.
Fine-tuning can even make RAG better. Researchers at UC Berkeley trained models on questions together with retrieved documents, including irrelevant ones. The models learned to quote the right passage and to ignore the distractors (Source: Zhang et al., “RAFT”, UC Berkeley, 2024). This is an advanced step. Take it only when a working RAG system still misuses the passages it receives.
Decision table with examples
Use this table as a starting point; your eval set has the final say.
| Situation | First choice | Why |
|---|---|---|
| A support bot must answer from current return and shipping rules | RAG | The facts change, and answers should cite the rule |
| Replies must always be valid JSON with fixed fields | Prompting with a schema | Structured output enforces the format without training |
| 50,000 tickets per day must be sorted into 12 categories | Fine-tune a small model | Narrow task, high volume, lower cost per request |
| Product texts should sound like your brand | Prompting with examples | Fine-tune only if the voice drifts across many texts |
| Questions about a 10-page manual that rarely changes | Prompting | The whole manual fits into the context window |
| An internal assistant needs your house style and current documents | Fine-tuning plus RAG | Behavior from training, facts from retrieval |
| A model gets legal or medical facts wrong | RAG with expert review | Fine-tuning would not make the facts reliable |
| A long prompt with many examples makes every request slow and expensive | Fine-tune a smaller model | The examples move into the weights |
Whatever you choose, keep the prompts, the documents behind your index and the training data under version control. Then you can trace every change in quality back to the lever that caused it.