Language models are impressive, but they do not know your private documents, and their knowledge stops at a cut-off date. Retrieval-augmented generation, or RAG, solves both problems. Before the model answers, the system searches your own data for relevant passages and gives them to the model as context. The answer is then based on real sources instead of memory alone.
The idea in one paragraph
A user asks a question. The system converts the question into an embedding, finds the most similar chunks of your documents, adds those chunks to the prompt, and asks the language model to answer using only that material. Because the supporting text is right there, the model is less likely to invent facts, and you can show the sources.
The RAG pipeline step by step
1. Collect and clean documents
Gather PDFs, web pages, policies, manuals, FAQs or database records. Extract the text and remove noise such as repeated headers, page numbers and navigation menus. Poor extraction produces poor answers.
2. Split into chunks
Long documents are cut into smaller pieces, commonly a few hundred words each, often with slight overlap so that sentences at the boundary are not lost. Splitting along natural structure, like headings and paragraphs, works better than cutting at random lengths.
3. Create embeddings
Each chunk is converted into a vector using an embedding model. These vectors capture meaning, so similar ideas are close together.
4. Store in an index
Vectors and their text (plus metadata like title and page) are saved in a vector store, which can be a dedicated database, a database extension or a simple file for small projects.
5. Retrieve at question time
The question is embedded with the same model, and the closest chunks are fetched, usually the top 3 to 8. Many systems combine this with keyword search (hybrid retrieval) or re-rank the results with a more precise model.
6. Build the prompt
The retrieved chunks and the question are placed into a prompt that tells the model how to behave, for example:
Answer the question using only the context below.
If the answer is not in the context, say "I could not find this in the documents."
Cite the source titles you used.
Context:
[1] Return policy: Items can be returned within 7 days with the receipt...
[2] Shipping: Delivery inside Kathmandu valley takes 1 to 2 days...
Question: Can I return a product after ten days?7. Generate and show sources
The model writes the answer. Displaying the source documents lets users verify it.
Why use RAG instead of fine-tuning?
| RAG | Fine-tuning | |
|---|---|---|
| Updating knowledge | Replace documents, instantly | Retrain the model |
| Shows sources | Yes | No |
| Cost to maintain | Lower | Higher |
| Teaches style or format | Weak | Strong |
| Private data stays separate | Yes, in your store | Absorbed in weights |
They are not exclusive. Fine-tuning can shape tone and behaviour, while RAG supplies facts.
Common problems and fixes
- Irrelevant retrieval: improve chunking, use better embeddings, add keyword search or re-ranking.
- Missing context: the answer spans several chunks; retrieve more or use larger chunks.
- Hallucinated answers: strengthen the instruction to answer only from context and to say when it cannot find the answer.
- Outdated content: re-index when documents change.
- Wrong access: filter retrieval by user permissions to prevent leaks.
- Tables and scanned PDFs: need special parsing or OCR.
- Language issues: use a multilingual embedding model for Nepali or mixed content.
Testing a RAG system
Build a small evaluation set of 30 to 100 real questions with the expected source and answer. Measure:
- Retrieval quality: did the correct chunk appear in the top results?
- Answer quality: is the answer correct and faithful to the sources?
- Refusals: does it say "not found" when the answer is not in the data?
Re-run these tests after every change to chunking, models or prompts.
When RAG is a good fit
- Company knowledge bases and internal help desks
- Customer support over product manuals and policies
- Legal, HR or compliance document search
- Education: question answering over course material
- Research assistants that cite papers
It is less suitable when answers require complex calculation across data (use SQL or tools) or when the information is tiny enough to paste directly into the prompt.
Frequently Asked Questions
Does RAG remove hallucinations completely?
No. It reduces them by giving the model reliable context, but the model can still misread or ignore it, so evaluation and citations remain important.
How many chunks should I retrieve?
Start with three to five and adjust based on testing. Too few miss information; too many add noise and cost.
Do I need a vector database?
For small collections, no. A simple in-memory index is enough. Use a database when the collection or traffic grows.
Can RAG work with Nepali documents?
Yes, if you use embedding and language models that support Nepali well. Test with real examples from your domain.
Conclusion
RAG connects a language model to your own knowledge: split, embed, retrieve, then answer from the retrieved text. It is a practical and maintainable way to build trustworthy assistants, provided you invest in good chunking, retrieval and testing.