How to stop a chatbot from making things up

Why chatbots invent answers in the first place

The model's training objective is next-word prediction: given the text so far, output the most likely continuation. Nothing in that objective points at truth — only at which words tend to follow which. When the context does not contain the answer, the model still completes the sentence, and it completes it fluently, because fluency is what it was optimised for. Confidence in the output is a style property, not a knowledge property. Seeing this clearly removes the temptation to blame "bad prompting" for structural behaviour: the model is doing exactly what it was built to do, and your job is to build the conditions under which doing it is safe.

Grounding: answer only from provided material

The most effective measure by a wide margin is to stop asking the model what it knows and hand it what you know: retrieve the relevant documents at question time and instruct it to answer only from that material. This is the RAG pattern, and combined with a hard refusal rule — if the material does not contain the answer, say you do not know — it turns a generalist that guesses into a specialist that abstains. The rule must be tested, not assumed: models drift into general knowledge under pressure unless a fixed eval set proves they do not. Grounding also relocates the problem, from "is the model right" to "is the material right", which is a problem you can audit page by page.

Refusal rules, low temperature and format constraints

Three smaller dials tighten a grounded system. Refusal instructions: define the out-of-scope case explicitly and script the exact sentence the bot should produce, so abstaining is a designed behaviour rather than a hope. Low temperature: sampling at or near zero makes output more deterministic and removes a source of drift between test and production. Format constraints: if the answer feeds another system — a booking engine, a calculator, a database — require structured output and validate it in code before trusting it; never let model prose cross a system boundary unexamined. None of these fixes a missing-knowledge problem. They make a grounded system consistent enough to ship — and to keep shipping as it changes.

A fixed eval set, rerun after every change

The discipline that holds everything together is a fixed set of fifty to a hundred questions with known answers — including ten the material deliberately cannot answer — rerun automatically after every change to prompt, model or retrieval. Without it, every edit is a coin flip nobody inspects: the tweak that fixes Tuesday's complaint silently breaks Wednesday's fact. With it, hallucination rate becomes a number that goes down on purpose. The unanswerable questions matter most. A system that always answers looks excellent on answerable questions while failing precisely the scenario your customers hit at the edges — and edges are where chatbots live or die.

Frequently asked questions

Do longer or more polite prompts reduce hallucination?

No. Adding "please be accurate" or several paragraphs of instructions changes nothing structural — the model still predicts plausible text, now with a preamble. What works is changing what it can see (grounding), what it may do (refusal rules) and whether you measure it (an eval set). Prompt polish is not on that list.

Can a chatbot ever be hallucination-free?

Not in the absolute sense: models can misread correct material, retrieve the wrong chunk, or fail on rare phrasings. What you can reach is a measured, low rate with honest refusals on uncovered questions — usually good enough to ship, provided the eval set keeps watching after launch and after every change.

Where is human review mandatory?

Where a wrong answer causes harm no disclaimer absorbs: medical, legal and financial guidance, anything affecting safety, and any output that becomes a contractual or official document. In those domains the model may draft and retrieve, but a qualified human approves before the answer reaches the person it affects.