I Built a Chatbot for My Own Documents in One Evening (Free, Local, No API Key)

The stack, and why

Related: GPT-5 Is Here: Everything You Need to Know About OpenAI's Most Powerful Model Yet →

Related: Ollama vs LM Studio vs Jan: I Used All Three for a Month →

Setup

Related: Will AI Coding Agents Replace Developers? We Asked 100 Engineers →

Related: How to Run a Local LLM on 8GB of RAM (What Actually Works in 2026) →

ollama pull qwen3:8b
ollama pull nomic-embed-text
pip install chromadb pypdf ollama

The ingest script

Related: The 27 Best AI Tools in 2026 (Tested for 90 Days) →

Related: كيف تبني وكيل ذكاء اصطناعي يعمل على جهازك بدون إنترنت (Ollama + MCP) — جرّبته 3 أسابيع (2026) →

import os, pypdf, chromadb, ollama

client = chromadb.PersistentClient(path="./store") col = client.get_or_create_collection("docs")

def chunk(text, size=900, overlap=150): out, i = [], 0 while i < len(text): out.append(text[i:i+size]) i += size - overlap return out

for fn in os.listdir("docs"): if not fn.endswith(".pdf"): continue reader = pypdf.PdfReader(f"docs/{fn}") for page_no, page in enumerate(reader.pages): text = (page.extract_text() or "").strip() if len(text) < 50: continue for j, c in enumerate(chunk(text)): emb = ollama.embeddings(model="nomic-embed-text", prompt=c)["embedding"] col.add(ids=[f"{fn}-{page_no}-{j}"], embeddings=[emb], documents=[c], metadatas=[{"source": fn, "page": page_no + 1}]) print("indexed", col.count(), "chunks")

The query script

Related: ChatGPT vs Claude 4: Which AI Should You Actually Pay For in 2026? →

Related: كيف تؤتمت عملك بالذكاء الاصطناعي باستخدام n8n مجاناً (دليل عملي خطوة بخطوة 2026) →

import chromadb, ollama

col = chromadb.PersistentClient(path="./store").get_collection("docs")

def ask(q, k=5): emb = ollama.embeddings(model="nomic-embed-text", prompt=q)["embedding"] res = col.query(query_embeddings=[emb], n_results=k) ctx = "\n\n---\n\n".join( f"[{m['source']} p.{m['page']}]\n{d}" for d, m in zip(res["documents"][0], res["metadatas"][0])) prompt = ( "Answer using ONLY the context below. Cite the source and page for each claim. " "If the answer is not in the context, say so plainly.\n\n" f"CONTEXT:\n{ctx}\n\nQUESTION: {q}") return ollama.chat(model="qwen3:8b", messages=[{"role": "user", "content": prompt}])["message"]["content"]

print(ask("What is the refund window in the contract?"))

That is the whole thing. Run ingest once, then query as often as you like.

Experience log: the two mistakes that wasted my evening

Related: Google Gemini 3 Ultra Review: Has Google Finally Caught Up? →

Related: 12 Best AI Tools for Coding in 2026 (Tested for Solo Developers & Freelancers) →

Mistake 1 — chunks that were too small. I started at 300 characters with no overlap because a tutorial said so. Answers were confidently wrong: the retriever pulled a sentence containing "30 days" from an unrelated clause. Moving to 900 characters with 150 overlap fixed roughly 70% of the bad answers immediately. Small chunks retrieve precisely and answer badly, because the model loses the surrounding meaning.

Mistake 2 — no page filter on empty text. Scanned pages returned empty strings, which still got embedded, and empty vectors sat near everything in the index — so garbage kept appearing in results. The len(text) < 50 guard removed it. If your PDFs are scans, you need OCR (ocrmypdf) before any of this works at all.

Making the answers better

Related: Midjourney vs DALL-E 4 vs Flux 1.1: The Definitive AI Image Generator Comparison →

Related: How to Run an AI Model Locally on Your Own Computer (Step-by-Step 2026 Guide) →

• Fetch more, then trim. Retrieve 10 chunks, ask the model to discard irrelevant ones, then answer. Slower, clearly more accurate.
• Keep the filename meaningful. 2026-supplier-contract.pdf gives the model context that scan001.pdf does not.
• Re-index when documents change. Chroma will not notice on its own; delete by id or rebuild the collection.
• Cap what you feed in. Five 900-character chunks is roughly 1,200 tokens. Twenty chunks will overflow a small model's context and quality collapses.

When this approach is not the right one

Related: 9 Free AI Coding Tools Every Developer Should Try in 2026 →

If your question needs to consider the entire document rather than a passage — "summarise the whole report", "how many times is X mentioned" — retrieval is the wrong tool. Feed the full document to a long-context model instead, or count with plain code.

FAQ

Related: Sora 2 Review: OpenAI's Video Model Is Finally Useful for Real Work →

How many documents can this handle? Chroma is comfortable into the low hundreds of thousands of chunks on a laptop. Past that, look at pgvector or Qdrant.

Can I use this on Word documents and web pages? Yes — swap pypdf for python-docx or trafilatura. The rest is unchanged.

Does it work in other languages? nomic-embed-text handles multilingual content reasonably; for heavy non-English use, bge-m3 performed better in my tests.

Is this private? Completely, as long as both models are local. Nothing leaves the machine — verify with ss -tunap | grep ollama if you want to be sure.