Build a Local RAG Chatbot on Your Own Documents (Full Code)
Build a Local RAG Chatbot on Your Own Documents (Full Code)
A local RAG chatbot is the best first project in AI. It runs on your laptop, costs zero, and answers questions about your own files. No API keys, no subscriptions, no data leaving your machine.
I built one for my meeting notes and PDFs in an afternoon. Here is the full working version, plus the three mistakes that wasted my time so you skip them.
What you need
- Python 3.10 or newer
- Ollama, the local model runner
- A ChromaDB install for storage
- A folder of text or markdown files
Hardware: an 8GB RAM laptop is enough. The model we use, llama3.2, runs fine on CPU. You will not need a GPU for this.
Install everything
# Python packages pip install ollama chromadb # The model runner curl -fsSL https://ollama.com/install.sh | sh # Pull a small embedding model and a chat model ollama pull nomic-embed-text ollama pull llama3.2
Two models, one job each. nomic-embed-text turns text into vectors for search. llama3.2 writes the answers. Splitting them matters: mixing embedding and chat models is the most common setup error in RAG.
Step 1: Load and chunk your documents
RAG works by splitting documents into chunks, embedding each chunk, and storing the vectors. At query time it finds the chunks closest to your question and hands them to the chat model.
Chunking is where quality lives. Too big and retrieval gets fuzzy. Too small and the context loses meaning. Two to four paragraphs is the sweet spot for most documents.
import os
CHUNK_SIZE = 800
OVERLAP = 100
def load_and_chunk(folder):
chunks = []
for fname in sorted(os.listdir(folder)):
path = os.path.join(folder, fname)
if not fname.endswith((".md", ".txt")):
continue
with open(path, encoding="utf-8", errors="ignore") as f:
text = f.read()
# split on paragraph boundaries, then merge into chunks
paras = [p.strip() for p in text.split("\n\n") if p.strip()]
buf = ""
for p in paras:
if len(buf) + len(p) < CHUNK_SIZE:
buf += "\n\n" + p
else:
chunks.append(buf)
# overlap keeps context across the cut
buf = buf[-OVERLAP:] + "\n\n" + p
if buf:
chunks.append(buf)
return chunks
chunks = load_and_chunk("notes")
print(f"{len(chunks)} chunks loaded")
The overlap parameter is the quiet hero. Without it, sentences get severed mid-thought and the chatbot answers with missing context. With 100 characters of overlap, the model can see what came before the cut.
Step 2: Embed and store
ChromaDB stores the vectors and handles the search. One collection, add the embeddings, done.
import chromadb
import ollama
client = chromadb.PersistentClient(path="./chroma_db")
collection = client.get_or_create_collection("notes")
# embed every chunk
ids = [str(i) for i in range(len(chunks))]
embeddings = [
ollama.embed(model="nomic-embed-text", input=c)["embeddings"][0]
for c in chunks
]
collection.upsert(ids=ids, embeddings=embeddings, documents=chunks)
print("Embeddings stored")
PersistentClient matters. The default in-memory client wipes your database on restart, and rebuilding embeddings takes time. The ./chroma_db folder keeps everything on disk.
Step 3: Query with retrieval
The query loop does two things: pull the closest chunks, then ask the chat model to answer using only those chunks.
def ask(question):
# retrieve the 4 closest chunks
res = collection.query(
query_embeddings=ollama.embed(
model="nomic-embed-text", input=question
)["embeddings"],
n_results=4,
)
context = "\n\n".join(res["documents"][0])
prompt = f"""Answer the question using only the context below.
If the context does not contain the answer, say so.
Context:
{context}
Question: {question}"""
out = ollama.chat(
model="llama3.2",
messages=[{"role": "user", "content": prompt}],
)
return out["message"]["content"]
while True:
q = input("Ask (or 'quit'): ")
if q.lower() == "quit":
break
print(ask(q))
The ground rule in the prompt is not decoration. It stops the model from inventing answers when your documents do not cover the question. Hallucination control is a prompt problem, not a model problem.
The three mistakes I made
First, I embedded the whole document as one chunk. Retrieval matched on the document level, so every answer dragged in unrelated sections. Chunking fixed it.
Second, I used the chat model for embeddings too. It worked, it was slow, and the vector quality was mediocre. Dedicated embedding models are smaller and better at this. That is why nomic-embed-text exists.
Third, I skipped the overlap and got answers that read like they were missing a page. Sentences start mid-thought when the cut lands inside a paragraph. One hundred characters of overlap eliminated it.
Adding a relevance filter
The base version returns the four closest chunks no matter what. If your question is not in the documents, it still returns something, and the chat model strains to answer. A distance filter fixes it.
res = collection.query(
query_embeddings=ollama.embed(
model="nomic-embed-text", input=question
)["embeddings"],
n_results=6,
)
# keep only reasonably close chunks
filtered = [
(doc, dist) for doc, dist in zip(
res["documents"][0], res["distances"][0]
) if dist < 1.0
]
if not filtered:
print("Nothing relevant in the documents.")
continue
context = "\n\n".join(d for d, _ in filtered)
The distance threshold needs tuning per embedding model. Start at 1.0, watch which questions get rejected, and adjust. A filter that is too tight says "no answer" to questions you have documents for. Too loose and you are back to hallucination town.
Why this beats cloud RAG for most people
A hosted RAG service costs money per query, sends your documents to a third party, and locks the pipeline to one vendor. The local version costs a laptop and keeps everything on disk. For personal knowledge bases, meeting notes, and internal docs, local wins on every axis except raw model quality.
The quality gap matters less than people think. Retrieval quality comes from chunking and embedding, not from the chat model. The chat model in this setup is doing constrained generation over a narrow context. That is a simple job. A frontier model shines on open-ended reasoning, which is the opposite of what RAG does.
If you hit the ceiling, the fix is not a bigger cloud subscription. It is a better chunking strategy or a larger local model. Both are free.
Handling PDFs and Word files
Text files are the easy case. Your real documents live in PDFs. Add two dependencies and a few lines to the loader.
pip install pypdf python-docx
from pypdf import PdfReader
from docx import Document
def load_file(path):
if path.endswith(".pdf"):
reader = PdfReader(path)
return "\n\n".join(p.extract_text() or "" for p in reader.pages)
if path.endswith(".docx"):
doc = Document(path)
return "\n\n".join(p.text for p in doc.paragraphs)
with open(path, encoding="utf-8", errors="ignore") as f:
return f.read()
One warning: PDF text extraction is inconsistent. Scanned PDFs have no text layer at all, and the chunks come out as garbage. If your documents are scans, run OCR first (tesseract works) before you feed them to the pipeline. This is the hidden trap in every RAG project.
Where to take it next
This base version handles text and markdown. Extend it in three directions:
- PDFs and Office files: add
pypdfordocxparsing in the loader - Better retrieval:
n_results=6with a relevance filter on the distance score - Web UI: Open WebUI speaks the Ollama API natively, so your collection shows up there for free
Total cost of this project: your electricity. The models run locally, the storage is a folder, and the code is about forty lines. That is the point. RAG is not a cloud feature you rent, it is a pattern you can run on a laptop while the wifi is off.