You stop an AI chatbot making up answers by working out which of five causes you have, then fixing that one. Nothing grounds the answer in your documents. Retrieval hands the model the wrong evidence. The prompt rewards a guess. The system has no permission to refuse. Or nobody measures any of it.
Each cause has its own fix, so diagnose before you change anything. And none of the fixes takes made-up answers, or hallucinations, to zero. The realistic goal is a rate you measure on real questions and reduce release over release.
In February 2024, British Columbia's Civil Resolution Tribunal ruled on an airline's website chatbot. It had suggested a customer could apply for a bereavement fare retroactively. The airline did not allow that. The chatbot's answer even linked to the airline's own "Bereavement travel" page, which said the policy does not apply once travel is complete.
The airline argued it could not be held liable for information its chatbot provided. The tribunal called this "a remarkable submission" and found negligent misrepresentation. In its words: "It makes no difference whether the information comes from a static page or a chatbot" (Moffatt v. Air Canada, 2024 BCCRT 149).
The award was small: $812.02 (Canadian) including interest and fees. The reasoning is what matters. The tribunal treated the chatbot's answer as the company's own statement. The decision does not say what technology the chatbot used, so it tells you nothing about any model. What it shows is a tribunal holding the company, not the software, responsible for a wrong answer.
| Cause | What you see | The fix |
|---|---|---|
| No grounding | Fluent, plausible answers with nothing from your documents behind them | Retrieve passages from your documents and answer only from them |
| Wrong evidence | A faithful summary of the wrong passage | Fix retrieval, and measure it apart from answer quality |
| A prompt that rewards guessing | Specific answers where the passages are vague or silent | Make "not in our documents" an acceptable answer |
| No permission to refuse | An answer to every question, however far out of scope | Design refusal and a handoff to a person |
| No measurement | Each fix seems to work, then something else breaks | A golden set, scored on every release |
A language model on its own knows only what was in its training data. Ask it about your refund policy and it will write something that sounds like a refund policy. Fluent, plausible, and not yours.
The fix is retrieval-augmented generation (RAG): fetch passages from your own documents and have the model answer from those. The 2020 paper that introduced RAG models tested this on generation tasks. The RAG models wrote language that was "more specific, diverse and factual" than a baseline relying on its training alone (Lewis et al.).
Fine-tuning is the weaker tool for this job. It changes how a model behaves, its style and format, but it is not the best way to give a model new facts. We cover that choice in RAG vs fine-tuning.
Grounding also makes citations possible. A citation is a claim to check, not proof. NIST calls hallucination "confabulation", and warns that outputs can include "confabulated logic or citations that purport to justify or explain the system's answer" (NIST AI 600-1).
The airline's chatbot linked to the very page that contradicted it. The tribunal noted the airline did not explain "why customers should have to double-check information found in one part of its website on another part of its website." Your users will not check the citation. Your system has to.
Grounding only helps if retrieval finds the right passage. When it does not, the model faithfully summarises the wrong paragraphs, and the answer sounds as confident as a correct one. In our experience this is more common than the model misreading a passage it was given, and it is the failure teams misdiagnose as the model making things up.
For twenty wrong answers the test takes an afternoon. For each, check whether the correct passage was in the retrieved set at all. If it was not, no prompt change will help. We use hybrid retrieval (keyword matching with BM25, embeddings and reranking) tuned to the questions people actually ask. The full diagnosis is in Your RAG pipeline is retrieving the wrong chunks.
A system prompt can quietly tell the model that a gap is worse than a guess. "Always give the customer an answer." "Never say you don't know." Each line is reasonable on its own. Together they make a plausible invention the most obedient thing the model can produce.
The layout of the evidence matters too. In 2023, Liu and colleagues found performance is often highest when the relevant information sits "at the beginning or end of the input context". It drops when that information is in the middle of a long context (Lost in the Middle). Twenty loosely relevant passages with the right one buried in the middle make that passage harder to use.
The fix is in the instructions:
- State that the supplied passages are the only source of facts.
- Say that "this is not covered in our documents" is a correct answer when it is true, and give the exact wording.
- Ask for every claim to name the passage it came from.
- Pass fewer, better passages, with the strongest first.
Refusal has to be designed, not hoped for. Decide three things: the reply when the documents do not cover a question, the reply when a question is out of scope, and where the person goes next.
Something like "I can't find that in our policy documents, so I've passed your question to the support team" beats an invented answer.
We treat citation as a hard gate in production. An answer that cannot cite is sent back to retrieval or refused outright.
Every fix above can make something else worse. A stricter prompt raises refusals. A retrieval change helps one kind of question and breaks another. Without a fixed test set you will see neither.
Build a golden set: real questions with checked answers, pulled from tickets and logs rather than invented. Thirty to fifty is enough to start. Include questions your documents cannot answer, because those test refusal.
Score each answer as correct, wrong, correctly declined or wrongly declined. Count an answer as correct only when the passages it cites support every claim, not when a citation is merely present.
Do not score "I don't know" as simply wrong. Kalai and colleagues argue that language models hallucinate because "the training and evaluation procedures reward guessing over acknowledging uncertainty" (Why Language Models Hallucinate, 2025). In their words, "guessing when uncertain improves test performance." An internal eval that scores a refusal like an error repeats that mistake inside your own release process.
Run the set on every release, including prompt edits, and block the release if quality regresses. That is how we work; the AI Lab sets out the gate.
NIST describes confabulation as "a natural result of the way generative models are designed." Our position, from the generative AI service page, is that hallucinations do not go to zero, but we measure and reduce them release over release.
Putting an assistant in front of customers?
Our generative AI service builds RAG assistants with citations, refusal logic and PII redaction from day one, and runs continuous evals against your real questions.
See the generative AI servicePull twenty answers your assistant got wrong. For each, ask two questions. Was the right passage retrieved? Did the answer stay inside what that passage says?
If the passage was missing, look at retrieval, or at grounding if there is no retrieval at all. If the passage was there and the answer went beyond it, look at the prompt and at refusal. If you cannot answer either question for a given case, that is the fifth cause, and it is the one to fix first.
Fix the biggest pile, then rerun the same twenty.