{"author":{"@type":"Person","name":"Alexander Bock","url":"https://abxk.ai/en/autor/alexander-bock/"},"categories":["AI Development"],"content":{"markdown":"Your AI feature gives wrong answers, ignores your format or sounds nothing like your company. You have three ways to fix that. You can write better instructions (**prompting**). You can give the model the right documents at question time (**retrieval-augmented generation**, RAG). Or you can train the model further on your own examples (**fine-tuning**). Fine-tuning often sounds like the thorough solution. In practice, it takes the most effort to set up and to change — and it often fixes the wrong problem.\n\nThis is part 4 of Level 3 (Advanced). You will learn what each method changes, what data and upkeep it needs and how the three work together. A decision flow and a table of examples help you choose. The part builds on [prompt design for developers](/en/ai-development/learn/prompt-design-for-developers/) from Level 1 and [RAG step by step](/en/ai-development/learn/retrieval-augmented-generation-step-by-step/) from Level 2.\n\n## Three levers on the same request\n\nEvery answer of a language model depends on two things: what the model receives with the request and what it learned in training. The three methods work on different parts of this:\n\n- **Prompting** changes the instructions: system prompt, rules, examples and output format. The change works from the next request.\n- **RAG** changes the knowledge in the request. Your app searches your documents and adds the most relevant passages to the prompt. The model itself stays the same.\n- **Fine-tuning** changes the model. You continue training it on pairs of input and desired output, and its **weights** — the numbers it learned in training — change.\n\n![Diagram of one request: the prompt contains instructions, passages from your documents and the user's question; it goes to the model, which writes the answer; prompting acts on the instructions, RAG on the passages and fine-tuning on the model](/images/learn/en/fine-tuning-vs-rag-vs-prompting-levers.webp \"Three levers on the same request: prompting and RAG change what the model receives, fine-tuning changes the model.\")\n\nThis difference explains most decisions. Instructions and knowledge travel with each request, so you can change them at any time. What a model learns through fine-tuning is built into it. It costs no extra tokens per request, but changing it takes a new training run.\n\n## Fine-tuning teaches behavior, not facts\n\nA common mistake is to fine-tune a model so that it \"knows\" your documents. Research shows why this rarely works. A Microsoft study compared two ways of adding knowledge to a model: RAG and fine-tuning on the documents themselves. RAG did better than fine-tuning, both for facts the model had seen before and for entirely new ones (Source: Ovadia et al., \"Fine-Tuning or Retrieval?\", EMNLP 2024). A second study found that models learn new facts slowly through fine-tuning — and once they have learned them, they tend to hallucinate more (Source: Gekhman et al., \"Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?\", EMNLP 2024).\n\nFine-tuning is strong where prompts reach their limits:\n\n- **A fixed format or style** that must hold across thousands of requests, for example a ticket summary that always has the same five fields.\n- **A narrow, repetitive task**, such as sorting texts into categories or extracting fields. Here, a small fine-tuned model can replace a large general one.\n- **Shorter prompts:** behavior that needed long instructions and many examples moves into the model, so each request uses fewer tokens.\n- **Behavior that is hard to describe** but easy to show, such as the way your support team declines a request politely.\n\nYou do not need huge amounts of data, but you need good data. In the LIMA study, researchers at Meta fine-tuned a large model on only 1,000 carefully chosen examples and got strong results (Source: Zhou et al., \"LIMA: Less Is More for Alignment\", NeurIPS 2023). Clean, consistent examples matter more than a large number of messy ones.\n\n## What each method needs and costs\n\nThe three methods differ in data, effort and upkeep:\n\n| | Prompting | RAG | Fine-tuning |\n|---|---|---|---|\n| What changes | Instructions in each request | Knowledge in each request | The model's weights |\n| Data you need | A clear task and a few examples | Your documents, kept up to date | Many consistent pairs of input and desired output |\n| Effort to start | Low | Medium: index and search pipeline | High: collect data, train, evaluate |\n| Running cost | Long prompts use more tokens | Search plus longer prompts | Training runs; hosting for open-weight models |\n| When things change | Edit the prompt | Update the index | Train again |\n| Can cite sources | Only from text in the prompt | Yes, from the retrieved passages | No |\n| Typical failure | Long prompts with conflicting rules | Wrong or missing passages | Learns errors from the data; loses general skills |\n\nThe last row needs a word of explanation. A model that is trained hard on one narrow task can get worse at others. This is called **catastrophic forgetting**. LoRA, the method that trains only small adapter matrices, keeps more of the base model's general skills than full fine-tuning — but it also learns less of the new task (Source: Biderman et al., \"LoRA Learns Less and Forgets Less\", TMLR 2024).\n\nThe effort also differs over time. Prompting stays cheap to change. RAG needs steady care for the index. Fine-tuning needs the most work at the start, and the work comes back whenever your data or the base model changes:\n\n![Line chart without numbers: the total work for your team over time; prompting rises slowly, RAG starts higher and grows steadily, fine-tuning starts highest and jumps again with new training data and a new base model](/images/learn/en/fine-tuning-vs-rag-vs-prompting-effort.webp \"Setup and upkeep over time. Illustration of a typical pattern, not measured data.\")\n\nFor RAG, you need a vector database; for fine-tuning an open-weight model, you need GPUs. In our comparison, Pinecone is a vector database for search and retrieval, and Runpod rents GPUs for training and running models, according to the vendors. Devs.ai is a platform where teams build AI apps and agents on their own data:\n\n{{\u003c tools \"pinecone runpod devs-ai\" \u003e}}\n\n## Start with prompting, then diagnose\n\nThe order matters more than the method. Begin with the cheapest lever, and move on only when your tests show a specific problem. Before you change anything, build a small **eval set**: typical inputs with the answers you expect, as described in [test AI features with evals](/en/ai-development/learn/test-ai-features-with-evals/). Without it, you cannot tell whether a change helped.\n\nThen collect answers that went wrong and sort them by cause. Each cause points to one lever:\n\n![Decision flow: collect 20 to 50 wrong answers and sort them by cause; ignored instructions lead to a better prompt, missing or outdated facts to RAG, drifting format or style at high volume to fine-tuning, and slow or costly answers to a smaller fine-tuned model; if none applies, the task may need a stronger model](/images/learn/en/fine-tuning-vs-rag-vs-prompting-decide.webp \"Diagnose first: the cause of the errors decides which lever helps.\")\n\n- **Instructions ignored or misunderstood?** Improve the prompt: clearer rules, examples, an output schema. Many APIs can also enforce a JSON format, as shown in [structured output and function calling](/en/ai-development/learn/structured-output-and-function-calling/).\n- **Facts missing, outdated or private?** Add RAG. The model needs the information at question time.\n- **Format or style drifts across many requests, although the prompt is clear?** Now fine-tuning is worth a test.\n- **Good answers, but too slow or too expensive?** Fine-tune a smaller model for the task, so it no longer needs the long prompt.\n- **Wrong reasoning even with good information?** The task may be too hard for the model. Compare stronger models in our [LLM database](/en/llm/).\n\n{{\u003c callout type=\"try\" title=\"Sort 20 failures before you decide\" \u003e}}\nTake 20 wrong answers from your logs or tests. Label each one with a single cause: instruction, knowledge, style, speed or cost, or reasoning. Count the labels. The largest group tells you which lever to try first.\n{{\u003c /callout \u003e}}\n\n## Fine-tuning in practice: closed or open models\n\nThere are two ways to fine-tune. Some providers offer fine-tuning of selected closed models as a service: you upload examples, and the provider trains and hosts your version. With an open-weight model, you train it yourself — usually with LoRA on rented GPUs — and you host the result. Part 3 of this level shows how to [size and rent GPUs](/en/ai-development/learn/gpu-infrastructure-for-ai/) for that.\n\nSmaller open-weight models are common starting points, because they train quickly and fit on a single rented GPU with QLoRA. The number with the \"B\" in each name is roughly the size in billions of parameters. A few examples from our LLM database:\n\n{{\u003c llm-models \"llama-3-1-8b qwen3-5-9b gpt-oss-20b qwen-3-8-27b gemma-4-31b-it\" \u003e}}\n\nCheck two things before you start. The license decides whether you may fine-tune the model and use the result commercially. And the base model's own test results set the ceiling: fine-tuning shapes how a model uses what it already learned. It does not turn a weak model into a strong one.\n\nA special case is **distillation**: you train a small model on the answers of a large one, so the small model takes over a narrow task at lower cost. Check the terms of the large model's provider first, because some do not allow using their outputs to train other models.\n\n{{\u003c callout type=\"warning\" title=\"Training data can come back out\" \u003e}}\nA model can memorize parts of its training data and repeat them word for word. Researchers extracted names, contact details and code from a language model this way (Source: Carlini et al., \"Extracting Training Data from Large Language Models\", USENIX Security 2021). Remove personal data and secrets from your examples before you fine-tune, and use only texts you have the rights to.\n{{\u003c /callout \u003e}}\n\n## Combine them: one assistant, three layers\n\nThe methods are not rivals. Many production systems use all three, each for the part it does best. A useful rule: put what changes often into the layers that are easy to change.\n\n![Three layers of a support assistant: the prompt holds task and rules and changes any time; RAG supplies current policies from the help center and changes with every document update; the fine-tuned model holds the house style and ticket fields and changes rarely, with a new training run](/images/learn/en/fine-tuning-vs-rag-vs-prompting-combine.webp \"One assistant, three layers: what changes often belongs in the layers that are easy to change.\")\n\nFine-tuning can even make RAG better. Researchers at UC Berkeley trained models on questions together with retrieved documents, including irrelevant ones. The models learned to quote the right passage and to ignore the distractors (Source: Zhang et al., \"RAFT\", UC Berkeley, 2024). This is an advanced step. Take it only when a working RAG system still misuses the passages it receives.\n\n## Decision table with examples\n\nUse this table as a starting point; your eval set has the final say.\n\n| Situation | First choice | Why |\n|---|---|---|\n| A support bot must answer from current return and shipping rules | RAG | The facts change, and answers should cite the rule |\n| Replies must always be valid JSON with fixed fields | Prompting with a schema | Structured output enforces the format without training |\n| 50,000 tickets per day must be sorted into 12 categories | Fine-tune a small model | Narrow task, high volume, lower cost per request |\n| Product texts should sound like your brand | Prompting with examples | Fine-tune only if the voice drifts across many texts |\n| Questions about a 10-page manual that rarely changes | Prompting | The whole manual fits into the context window |\n| An internal assistant needs your house style and current documents | Fine-tuning plus RAG | Behavior from training, facts from retrieval |\n| A model gets legal or medical facts wrong | RAG with expert review | Fine-tuning would not make the facts reliable |\n| A long prompt with many examples makes every request slow and expensive | Fine-tune a smaller model | The examples move into the weights |\n\nWhatever you choose, keep the prompts, the documents behind your index and the training data under version control. Then you can trace every change in quality back to the lever that caused it.","text":"Your AI feature gives wrong answers, ignores your format or sounds nothing like your company. You have three ways to fix that. You can write better instructions (prompting). You can give the model the right documents at question time (retrieval-augmented generation, RAG). Or you can train the model further on your own examples (fine-tuning). Fine-tuning often sounds like the thorough solution. In practice, it takes the most effort to set up and to change — and it often fixes the wrong problem.\nThis is part 4 of Level 3 (Advanced). You will learn what each method changes, what data and upkeep it needs and how the three work together. A decision flow and a table of examples help you choose. The part builds on prompt design for developers from Level 1 and RAG step by step from Level 2.\nThree levers on the same request Every answer of a language model depends on two things: what the model receives with the request and what it learned in training. The three methods work on different parts of this:\nPrompting changes the instructions: system prompt, rules, examples and output format. The change works from the next request. RAG changes the knowledge in the request. Your app searches your documents and adds the most relevant passages to the prompt. The model itself stays the same. Fine-tuning changes the model. You continue training it on pairs of input and desired output, and its weights — the numbers it learned in training — change. Three levers on the same request: prompting and RAG change what the model receives, fine-tuning changes the model. This difference explains most decisions. Instructions and knowledge travel with each request, so you can change them at any time. What a model learns through fine-tuning is built into it. It costs no extra tokens per request, but changing it takes a new training run.\nFine-tuning teaches behavior, not facts A common mistake is to fine-tune a model so that it \u0026ldquo;knows\u0026rdquo; your documents. Research shows why this rarely works. A Microsoft study compared two ways of adding knowledge to a model: RAG and fine-tuning on the documents themselves. RAG did better than fine-tuning, both for facts the model had seen before and for entirely new ones (Source: Ovadia et al., \u0026ldquo;Fine-Tuning or Retrieval?\u0026rdquo;, EMNLP 2024). A second study found that models learn new facts slowly through fine-tuning — and once they have learned them, they tend to hallucinate more (Source: Gekhman et al., \u0026ldquo;Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?\u0026rdquo;, EMNLP 2024).\nFine-tuning is strong where prompts reach their limits:\nA fixed format or style that must hold across thousands of requests, for example a ticket summary that always has the same five fields. A narrow, repetitive task, such as sorting texts into categories or extracting fields. Here, a small fine-tuned model can replace a large general one. Shorter prompts: behavior that needed long instructions and many examples moves into the model, so each request uses fewer tokens. Behavior that is hard to describe but easy to show, such as the way your support team declines a request politely. You do not need huge amounts of data, but you need good data. In the LIMA study, researchers at Meta fine-tuned a large model on only 1,000 carefully chosen examples and got strong results (Source: Zhou et al., \u0026ldquo;LIMA: Less Is More for Alignment\u0026rdquo;, NeurIPS 2023). Clean, consistent examples matter more than a large number of messy ones.\nWhat each method needs and costs The three methods differ in data, effort and upkeep:\nFine-Tuning vs. RAG vs. Prompting: A Decision Guide Prompting RAG Fine-tuning What changes Instructions in each request Knowledge in each request The model\u0026rsquo;s weights Data you need A clear task and a few examples Your documents, kept up to date Many consistent pairs of input and desired output Effort to start Low Medium: index and search pipeline High: collect data, train, evaluate Running cost Long prompts use more tokens Search plus longer prompts Training runs; hosting for open-weight models When things change Edit the prompt Update the index Train again Can cite sources Only from text in the prompt Yes, from the retrieved passages No Typical failure Long prompts with conflicting rules Wrong or missing passages Learns errors from the data; loses general skills The last row needs a word of explanation. A model that is trained hard on one narrow task can get worse at others. This is called catastrophic forgetting. LoRA, the method that trains only small adapter matrices, keeps more of the base model\u0026rsquo;s general skills than full fine-tuning — but it also learns less of the new task (Source: Biderman et al., \u0026ldquo;LoRA Learns Less and Forgets Less\u0026rdquo;, TMLR 2024).\nThe effort also differs over time. Prompting stays cheap to change. RAG needs steady care for the index. Fine-tuning needs the most work at the start, and the work comes back whenever your data or the base model changes:\nSetup and upkeep over time. Illustration of a typical pattern, not measured data. For RAG, you need a vector database; for fine-tuning an open-weight model, you need GPUs. In our comparison, Pinecone is a vector database for search and retrieval, and Runpod rents GPUs for training and running models, according to the vendors. Devs.ai is a platform where teams build AI apps and agents on their own data:\nPineconeInfrastructure \u0026amp; Hosting · AI at the coreVector database for AI applications such as search and retrieval. RunpodInfrastructure \u0026amp; Hosting · AI at the coreGPU cloud for training and running AI models, billed by usage. Devs.aiApp Building \u0026amp; Testing · AI at the corePlatform where teams build and run AI applications and custom agents on their own data. Short descriptions from our tool comparison, based on the vendors' information — not tested by ABXK.\nStart with prompting, then diagnose The order matters more than the method. Begin with the cheapest lever, and move on only when your tests show a specific problem. Before you change anything, build a small eval set: typical inputs with the answers you expect, as described in test AI features with evals. Without it, you cannot tell whether a change helped.\nThen collect answers that went wrong and sort them by cause. Each cause points to one lever:\nDiagnose first: the cause of the errors decides which lever helps. Instructions ignored or misunderstood? Improve the prompt: clearer rules, examples, an output schema. Many APIs can also enforce a JSON format, as shown in structured output and function calling. Facts missing, outdated or private? Add RAG. The model needs the information at question time. Format or style drifts across many requests, although the prompt is clear? Now fine-tuning is worth a test. Good answers, but too slow or too expensive? Fine-tune a smaller model for the task, so it no longer needs the long prompt. Wrong reasoning even with good information? The task may be too hard for the model. Compare stronger models in our LLM database. ▸Sort 20 failures before you decide\nTake 20 wrong answers from your logs or tests. Label each one with a single cause: instruction, knowledge, style, speed or cost, or reasoning. Count the labels. The largest group tells you which lever to try first.\nFine-tuning in practice: closed or open models There are two ways to fine-tune. Some providers offer fine-tuning of selected closed models as a service: you upload examples, and the provider trains and hosts your version. With an open-weight model, you train it yourself — usually with LoRA on rented GPUs — and you host the result. Part 3 of this level shows how to size and rent GPUs for that.\nSmaller open-weight models are common starting points, because they train quickly and fit on a single rented GPU with QLoRA. The number with the \u0026ldquo;B\u0026rdquo; in each name is roughly the size in billions of parameters. A few examples from our LLM database:\nModels from our LLM database ModelReleasedWeightsECIRank Llama 3.1-8B Meta AIJul 23, 2024open116.6215 Qwen3.5-9B AlibabaFeb 24, 2026open139.4124 gpt-oss-20b OpenAIAug 5, 2025open137.8132 Qwen 3.8 27B AlibabaAug 14, 2026open151.652 Gemma 4 31B IT Google DeepMindApr 2, 2026open142.9102 ECI = Epoch Capabilities Index, one overall score across many benchmarks; rank among all models with an ECI. Data: Epoch AI, retrieved on October 17, 2026, licensed CC BY 4.0. All models: LLM database.\nCheck two things before you start. The license decides whether you may fine-tune the model and use the result commercially. And the base model\u0026rsquo;s own test results set the ceiling: fine-tuning shapes how a model uses what it already learned. It does not turn a weak model into a strong one.\nA special case is distillation: you train a small model on the answers of a large one, so the small model takes over a narrow task at lower cost. Check the terms of the large model\u0026rsquo;s provider first, because some do not allow using their outputs to train other models.\n!Training data can come back out\nA model can memorize parts of its training data and repeat them word for word. Researchers extracted names, contact details and code from a language model this way (Source: Carlini et al., \u0026ldquo;Extracting Training Data from Large Language Models\u0026rdquo;, USENIX Security 2021). Remove personal data and secrets from your examples before you fine-tune, and use only texts you have the rights to.\nCombine them: one assistant, three layers The methods are not rivals. Many production systems use all three, each for the part it does best. A useful rule: put what changes often into the layers that are easy to change.\nOne assistant, three layers: what changes often belongs in the layers that are easy to change. Fine-tuning can even make RAG better. Researchers at UC Berkeley trained models on questions together with retrieved documents, including irrelevant ones. The models learned to quote the right passage and to ignore the distractors (Source: Zhang et al., \u0026ldquo;RAFT\u0026rdquo;, UC Berkeley, 2024). This is an advanced step. Take it only when a working RAG system still misuses the passages it receives.\nDecision table with examples Use this table as a starting point; your eval set has the final say.\nFine-Tuning vs. RAG vs. Prompting: A Decision Guide (2) Situation First choice Why A support bot must answer from current return and shipping rules RAG The facts change, and answers should cite the rule Replies must always be valid JSON with fixed fields Prompting with a schema Structured output enforces the format without training 50,000 tickets per day must be sorted into 12 categories Fine-tune a small model Narrow task, high volume, lower cost per request Product texts should sound like your brand Prompting with examples Fine-tune only if the voice drifts across many texts Questions about a 10-page manual that rarely changes Prompting The whole manual fits into the context window An internal assistant needs your house style and current documents Fine-tuning plus RAG Behavior from training, facts from retrieval A model gets legal or medical facts wrong RAG with expert review Fine-tuning would not make the facts reliable A long prompt with many examples makes every request slow and expensive Fine-tune a smaller model The examples move into the weights Whatever you choose, keep the prompts, the documents behind your index and the training data under version control. Then you can trace every change in quality back to the lever that caused it.\n","words":1894},"date":"2026-10-14","description":"What prompting, RAG and fine-tuning each change, what data and upkeep they need, how to combine them — and a decision table with practical examples.","image":"https://abxk.ai/images/cover/fine-tuning-vs-rag-vs-prompting.webp","json":"https://abxk.ai/en/ai-development/learn/fine-tuning-vs-rag-vs-prompting/index.json","kind":"page","language":"en","learning_path":{"category":"ai-development","level":3,"level_name":"Advanced"},"modified":"2026-10-14","related_tools":["Pinecone","Runpod","Devs.ai"],"section":"ai-development","tags":["fine-tuning","rag","prompting","llm-apps"],"title":"Fine-Tuning vs. RAG vs. Prompting: A Decision Guide","translations":[{"json":"https://abxk.ai/de/ai-development/learn/fine-tuning-vs-rag-vs-prompting/index.json","language":"de","url":"https://abxk.ai/de/ai-development/learn/fine-tuning-vs-rag-vs-prompting/"}],"url":"https://abxk.ai/en/ai-development/learn/fine-tuning-vs-rag-vs-prompting/"}