An AI chatbot “hallucinates” when it produces an answer that sounds confident but is false or not supported by any source: an invented statistic, a fake citation, a wrong date. It happens because these systems generate text by predicting likely words, not by looking up verified facts, and because they have often been trained in ways that reward a confident guess over “I don’t know”. You cannot switch it off, but you can learn where it is likely and how to catch it.
What “hallucination” means
The US National Institute of Standards and Technology (NIST) prefers the term confabulation. In its Generative AI risk profile (July 2024), it defines this as when AI systems “generate and confidently present erroneous or false content in response to prompts”, noting that it is “colloquially also referred to as ‘hallucinations’ or ‘fabrications'”. NIST’s definition also covers answers that drift from what you asked or contradict something the model said earlier in the same conversation.
The confidence is the problem. A chatbot’s tone is the same whether it is right or wrong, so the usual cues we use to judge a human source do not work.
Why it happens
1. The model predicts, it does not look up. A large language model generates text one token at a time, choosing likely continuations (our guide to how large language models work explains this in detail). NIST says confabulations are “a natural result of the way generative models are designed”: they produce outputs that “approximate the statistical distribution of their training data”. A plausible-sounding answer and a true answer are not the same thing.
2. Gaps in training data. Models learn from their training text. For well-documented facts, the likely answer and the true answer usually coincide. For rare facts, such as an obscure person’s birthday, a specific court case or a niche statistic, the model has little to go on, but it can still produce something fluent.
3. Pressure to answer. A September 2025 research paper published by OpenAI, “Why Language Models Hallucinate” by Kalai and colleagues, argues that training and testing reward guessing. Most benchmarks score answers simply as right or wrong, so “I don’t know” earns nothing while a guess sometimes scores. In the authors’ words: “Under binary grading, abstaining is strictly sub-optimal.” Like a student facing an exam with no negative marking, the model learns to guess.
The paper gives a telling example. Asked for one of the author’s birthdays, and told to reply only if it knew, one widely used open model, DeepSeek-V3, gave three different wrong dates across three attempts, according to the authors. The authors argue hallucinations “need not be mysterious” and are not inevitable for post-trained systems if evaluation stops penalising honest uncertainty.
The main types
Researchers classify hallucinations in several ways. A widely cited survey by Ji and colleagues distinguishes intrinsic hallucinations, where output “contradicts the source content”, from extrinsic ones, where output “can neither be supported nor contradicted by the source”. In everyday use, they show up in recognisable forms:
| Type | What it looks like | Where it is common |
|---|---|---|
| Invented facts | A wrong date, figure or name stated confidently | Rare or recent facts |
| Fake citations | Real-looking references, case names or URLs that do not exist | Legal, academic and medical questions |
| Contradicting the source | A summary that says the opposite of the document you gave it | Summaries of long documents |
| Unsupported additions | Plausible details that appear nowhere in your source | Summaries, reports, translations |
| False reasoning | Convincing step-by-step logic leading to a wrong answer | Maths, analysis, “explain why” questions |
The last type is easy to miss. NIST warns that models may produce “confabulated logic or citations that purport to justify or explain the system’s answer”, which can lead people to trust a wrong answer more.
A real case: fake cases in a US court
The most cited real-world example is Mata v. Avianca, a suit in the US District Court for the Southern District of New York by a passenger who said he was injured when a metal serving cart struck his knee on an Avianca flight. Lawyers for the plaintiff filed a brief citing court decisions that did not exist. In his Opinion and Order on Sanctions of 22 June 2023, Judge P. Kevin Castel found that the lawyers “submitted non-existent judicial opinions with fake quotes and citations created by the artificial intelligence tool ChatGPT, then continued to stand by the fake opinions after judicial orders called their existence into question”. He described the legal analysis in one invented decision as “gibberish” and imposed a $5,000 penalty on the two lawyers and their firm.
The judge was explicit that using AI was not itself the problem: “there is nothing inherently improper about using a reliable artificial intelligence tool for assistance”. The failure was not checking.
What developers do about it
Retrieval (grounding). Instead of relying only on what the model memorised, the system first searches a trusted source, such as the web, a company’s documents or a legal database, and gives the relevant passages to the model to answer from. This approach, called retrieval-augmented generation (RAG), was described in a 2020 paper by Lewis and colleagues, who found it produced “more specific, diverse and factual language” than a model working from memory alone.
Retrieval reduces errors but does not remove them. A 2024 study of AI legal research tools that use retrieval found they still hallucinated “between 17% and 33% of the time” on the researchers’ test questions.
Citations. Many tools now show links to the passages they used. That helps only if you open them: a citation can be real but not actually say what the answer claims.
Rewarding uncertainty. The Kalai paper proposes changing how benchmarks score models, so that an honest “I’m not sure” is not punished more than a wrong guess.
Limiting what errors can do. When a model can take actions, not just write text, a hallucination becomes a wrong action. That is one reason AI agents need tight permissions and human approval; see what is an AI agent?
A checklist for verifying AI answers
Use this before you rely on a chatbot’s answer for anything that matters: money, health, law, work you sign your name to, or something you will publish.
- Ask: is this a rare or specific fact? Names, dates, figures, quotes, citations and recent events are the highest-risk categories.
- Open every source. Check that the link exists and that the page actually says what the answer claims.
- Check citations in the official place. For a law, use the official gazette or the government website. For a court case, use the court’s records. For a paper, find it on the journal’s site.
- Compare with the original document. If the AI summarised something you gave it, spot-check key claims against the text.
- Re-ask in a fresh chat. If you get a different answer to the same factual question, treat both as unverified.
- Ask it to show uncertainty. Prompts such as “say if you are not sure” or “quote the exact passage you relied on” can help, though they are no guarantee.
- Be extra careful with confident reasoning. A clean step-by-step explanation is not proof. Recheck numbers yourself.
- Keep a human sign-off. For anything consequential, the person who uses the answer is responsible for it, as the Mata v. Avianca lawyers learned.
The point: Chatbots hallucinate because they generate the most plausible text, not verified truth, and because training often rewards a confident guess. Retrieval and citations reduce the problem but do not end it. Treat specific facts, figures and citations from AI as leads to check, not answers to trust.
Sources
- Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (NIST AI 600-1), NIST, July 2024
- Why Language Models Hallucinate, Kalai, Nachum, Vempala and Zhang, arXiv, September 2025
- Survey of Hallucination in Natural Language Generation, Ji et al., arXiv, 2022 (revised 2024)
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, Lewis et al., NeurIPS 2020
- Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools, Magesh et al., arXiv, May 2024
- Mata v. Avianca, Inc., No. 22-cv-1461 (PKC), Opinion and Order on Sanctions, US District Court, Southern District of New York, 22 June 2023 (via CourtListener)