A chatbot that doesn't hallucinate: grounding answers
A bot doesn't lie because it's malicious. It lies because you never gave it permission to say "I don't know."
We once launched a FAQ bot for a support team. In the demo it answered beautifully. Two days later the client forwarded a chat where the bot confidently explained a refund policy the company had never had. Invented from scratch. Polite, fluent, precise down to the percentage. And completely false.
A language model doesn't know what it doesn't know. Ask it about your refund policy and it returns the most plausible-sounding text, not the true one. Those are different things. The fix isn't a smarter model — it's forcing the model to answer only from your documents. That's what RAG is.
How it works, step by step
First the documents get cut into pieces — chunks. Not thousand-word blocks, not single sentences. I usually go with 300–500 tokens and a small overlap, so a thought doesn't get sliced clean in half at the boundary. Each chunk runs through an embedding model and becomes a vector — coordinates of meaning. All of it sits in a vector database.
When a question comes in, it becomes a vector too, and the database returns the few nearest chunks. Those pieces — and only those — go into the prompt as context. The model answers from them, not from what it "remembers" off the internet. And then comes the part that actually matters.
The right to say "I don't know"
Most bots hallucinate not because retrieval failed but because the prompt was sloppy. If you just hand the model some context and a question, it will always answer — even when the context is empty. You have to explicitly let it give up. Here's the system prompt I set by default.
You answer ONLY from the provided context.
If the answer is not in the context, reply exactly:
"I don't have that in our docs — let me connect you to a human."
Never use outside knowledge. Never guess.
After each answer, cite the source: [source: <doc>, <section>].
---
Context:
{{ retrieved_chunks }}
Question: {{ user_question }}If retrieval returns zero chunks above your similarity threshold — don't call the model at all. Return "I don't know" outright and hand off to a human. Empty context plus a language model is a recipe for confident fiction.
And always show the sources. Not for decoration — it's your lie detector. When an answer carries "section 4.2 of the refund policy," the user can verify it, and during debugging you see exactly which chunk the bot leaned on. If there's an answer but no citation, the model is making things up again, and now you can see it at a glance.
"I don't know, but I'll get a human" isn't the bot failing. It's the only honest answer when the answer isn't in the docs.
A bot that honestly says "I don't know" 30% of the time is more useful than one that answers everything and lies one time in ten. You trust the first. The second you have to double-check, and then what's the point. Grounding isn't about making the model know more. It's about making it stay quiet when it doesn't.
Want this in your own project?
Let's look at your process and where it's safe to automate.