How to Build a Simple RAG Chatbot Over Your Own Documents

In this tutorial you will build a small but real retrieval-augmented chatbot in Python. It reads text files from a folder, splits them into chunks, creates embeddings, finds the chunks most relevant to a question and asks a language model to answer using only those chunks. It uses plain Python and NumPy, so you can see every step before moving to a vector database.

What you will need

Python 3.10 or newer
A few .txt files in a folder called docs, such as FAQs or policies
An API key for a language model provider, set as an environment variable
Packages installed with pip

pip install sentence-transformers numpy anthropic

Step 1: load and chunk documents

We split each file into overlapping chunks of about 120 words and keep the file name for citations.

from pathlib import Path

def load_chunks(folder="docs", size=120, overlap=30):
    chunks = []
    for path in Path(folder).glob("*.txt"):
        words = path.read_text(encoding="utf-8").split()
        step = size - overlap
        for start in range(0, max(len(words), 1), step):
            piece = " ".join(words[start:start + size])
            if piece:
                chunks.append({"source": path.name, "text": piece})
    return chunks

Overlap prevents an important sentence from being cut in half at a chunk boundary.

Step 2: embed the chunks

We use a small open-source model that runs on a normal laptop. Normalising the vectors lets us use a dot product as cosine similarity.

import numpy as np
from sentence_transformers import SentenceTransformer

embedder = SentenceTransformer("all-MiniLM-L6-v2")

def build_index(chunks):
    texts = [c["text"] for c in chunks]
    vectors = embedder.encode(texts, normalize_embeddings=True)
    return np.asarray(vectors, dtype="float32")

For Nepali or other languages, replace the model with a multilingual one such as paraphrase-multilingual-MiniLM-L12-v2 and test it on your content.

Step 3: retrieve relevant chunks

def retrieve(question, chunks, vectors, k=4):
    q = embedder.encode([question], normalize_embeddings=True)[0]
    scores = vectors @ q                     # cosine similarity
    top = np.argsort(scores)[::-1][:k]
    return [(float(scores[i]), chunks[i]) for i in top]

Step 4: ask the language model

The prompt tells the model to use only the supplied context, to admit when the answer is missing and to cite sources.

import anthropic

client = anthropic.Anthropic()   # needs ANTHROPIC_API_KEY in the environment

def answer(question, hits):
    context = "\n\n".join(
        f"[{i+1}] ({chunk['source']}) {chunk['text']}"
        for i, (score, chunk) in enumerate(hits)
    )
    prompt = (
        "Answer the question using ONLY the context below. "
        "If the context does not contain the answer, say: "
        "'I could not find this in the documents.' "
        "Mention the source numbers you used, like [1].\n\n"
        f"Context:\n{context}\n\nQuestion: {question}"
    )
    reply = client.messages.create(
        model="claude-sonnet-5-5",
        max_tokens=400,
        messages=[{"role": "user", "content": prompt}],
    )
    return reply.content[0].text

Step 5: put it together

if __name__ == "__main__":
    chunks = load_chunks("docs")
    vectors = build_index(chunks)
    print(f"Indexed {len(chunks)} chunks. Type 'quit' to exit.")

    while True:
        q = input("\nYou: ").strip()
        if q.lower() in {"quit", "exit"}:
            break
        hits = retrieve(q, chunks, vectors)
        if hits[0][0] < 0.25:            # weak match: do not force an answer
            print("Bot: I could not find this in the documents.")
            continue
        print("Bot:", answer(q, hits))
        print("Sources:", ", ".join(sorted({c['source'] for _, c in hits})))

The similarity threshold of 0.25 is only a starting point. Print the scores for your own questions and choose a value that separates relevant from irrelevant results.

Improving it

Cache the index. Save vectors with np.save and chunks as JSON so you do not re-embed on every start.
Better chunking. Split on headings and paragraphs rather than fixed word counts.
Hybrid search. Combine keyword matching (BM25) with embeddings for names and codes.
Re-ranking. Retrieve 20 candidates and re-rank them with a cross-encoder model.
Chat memory. Include a short history, and rewrite follow-up questions like "what about the price?" into standalone questions before retrieval.
Vector database. Move to FAISS, pgvector or a dedicated database when you outgrow memory.
Web interface. Wrap retrieve and answer in a Django or FastAPI endpoint and call it from a web page or a Flutter app.

Safety and quality checklist

Do not index confidential documents unless the access rules are enforced at retrieval time.
Keep a test list of questions and expected answers, and rerun it after changes.
Show sources so people can verify answers.
Log failed questions to learn what content is missing.
Remember that the prompt text and chunks are sent to your model provider, so read their data policy.

Frequently Asked Questions

Why not paste all documents into the prompt?

Small collections can fit, but large ones exceed the context window and cost more. Retrieval selects only the relevant parts.

Which embedding model should I use?

Start with a small, popular model for prototyping and compare alternatives on your own questions. Use a multilingual model if your content is not English.

How do I handle PDFs?

Extract text with a library such as pypdf or pdfplumber, and use OCR for scanned pages before chunking.

Can I run it without an external API?

Yes. Replace the answer step with a local open-source model, accepting a trade-off in speed and quality depending on your hardware.

Conclusion

A RAG chatbot is just four moves: chunk, embed, retrieve and answer from the context. With under 100 lines of Python you have a working prototype, and the improvements above show how to grow it into a dependable tool for your own documents.