Why Do AI Chatbots Sometimes Make Things Up? The Truth About AI Hallucinations
Ask a chatbot a question it doesn't know the answer to, and it will rarely say "I don't know." Instead, it may invent a fact, a quotation, a date — even a whole court case — and deliver it with complete confidence. This isn't a rare glitch. It's a fundamental feature of how these systems work.
Researchers call it hallucination. Some prefer the word confabulation. Either way, understanding why it happens is the key to using AI safely — because the chatbot itself doesn't know it's making things up.
What "AI Hallucination" Actually Means
In AI research, a hallucination is when a language model generates an answer that is false, misleading, or nonsensical — while sounding completely plausible and confident.
A few important things to know:
- It's not lying. Lying requires knowing the truth and choosing to hide it. A chatbot has no concept of truth; it only produces text that looks likely.
- Confidence means nothing. A chatbot sounds equally sure about real facts and invented ones. Confidence is a feature of its writing style, not a measure of accuracy.
- It happens to all of them. ChatGPT, Gemini, Claude, Grok, Copilot — every large language model can hallucinate. The details differ, but the underlying cause is the same.
A Better Word: "Confabulation"
Some researchers argue that "hallucination" is the wrong word — and that "confabulation" describes it more accurately.
In psychology, a hallucination is a false sensory experience, like seeing something that isn't there. AI models have no senses and no experiences. Confabulation, by contrast, is what happens when someone fills gaps in their memory with plausible-sounding details, without intending to deceive — they genuinely believe the false memory.
That's much closer to what a language model does: it has a gap in its knowledge, and it fills that gap with the most statistically plausible text. NIST's AI Risk Management Framework deliberately chose the term "confabulation" for exactly this reason. "Hallucination" remains the mainstream term, but knowing about confabulation helps you see the problem clearly: it's a memory-and-prediction failure, not a perception failure.
Why It Happens: The Next-Word Guessing Machine
A chatbot isn't a search engine and it isn't a database. It's a next-token prediction machine — a system trained to predict which word (or piece of a word, called a token) is most likely to come next in a sentence.
Here's the chain that leads to made-up answers:
- It's trained to write plausible text, not true text. During training, the model reads enormous amounts of text and learns patterns: how sentences are structured, how facts are usually presented, how experts sound. It learns the shape of a correct answer, not a verified list of facts.
- It has no built-in fact checker. The model doesn't pause to verify anything before answering. It generates one word at a time, always picking what sounds most likely next. A fabricated court case sounds just as "likely" as a real one if the model has seen many real case citations.
- It can't say "I don't know" naturally. Its training rewards complete, helpful-sounding answers. When knowledge runs out, the gap gets filled with plausible invention rather than an admission of ignorance.
- Its knowledge has a cutoff. Training data ends at some point in time. Ask about anything newer — or anything obscure that barely appeared in training — and the model guesses instead of knowing.
- "Temperature" controls creativity. Models have a setting called temperature that controls randomness. Higher temperature means more varied, creative output — and more made-up facts. Even at low temperature, though, hallucinations aren't eliminated.
For a deeper look at how these models learn, see how large language models work.
Real Cases: When Made-Up Answers Caused Real Damage
This isn't theoretical. Fabricated AI answers have caused court sanctions, legal liability, and lawsuits.
The lawyers who filed fake court cases
In 2023, two New York lawyers — Steven Schwartz and Peter LoDuca — filed a legal brief in the case Mata v. Avianca, Inc. that cited six judicial opinions invented entirely by ChatGPT. The fake cases came with fabricated quotations and citations. When asked whether the cases were real, ChatGPT even assured the lawyer they could be found in legal databases like LexisNexis and Westlaw.
On 22 June 2023, Judge P. Kevin Castel of the US District Court for the Southern District of New York fined the lawyers $5,000 and found they had acted with "subjective bad faith" under Rule 11, noting that one of the invented case summaries was "gibberish." The lesson the judge's order drove home: asking the chatbot isn't verification — it's just asking for a second hallucination.
Since then, courts have recorded over a thousand more cases of AI-fabricated material in filings (researcher Damien Charlotin's public database passed 2,000 global cases by September 2026), with fines, bar referrals, and even attorney suspensions — including a Colorado lawyer suspended in November 2023 over ChatGPT-fabricated citations.
The airline that had to honour its chatbot's made-up policy
In November 2022, Jake Moffatt asked Air Canada's website chatbot about bereavement fares after his grandmother died. The chatbot invented a policy: buy a full-price ticket now and claim a reduced bereavement fare within 90 days. The real policy said the opposite — no retroactive claims.
When Air Canada refused the refund, Moffatt took the airline to British Columbia's Civil Resolution Tribunal. Air Canada argued it couldn't be held liable for its own chatbot — calling it effectively a separate legal entity. On 14 February 2024, tribunal member Christopher Rivers rejected that argument as "remarkable," ruled the chatbot was simply part of the airline's website, and held Air Canada liable for negligent misrepresentation, awarding Moffatt about CA$812 plus interest and fees.
The takeaway for businesses is blunt: your bot's answers are your answers.
What Makes Hallucinations More or Less Likely
Not all questions are equally risky. Some situations practically invite the model to invent things:
| Makes made-up answers more likely | Makes made-up answers less likely |
|---|---|
| Very specific details (dates, quotes, citations, URLs) | Broad summaries of well-known topics |
| Questions about events after the model's knowledge cutoff | Questions with answers repeated widely in training data |
| Asking for niche facts, rare names, or obscure statistics | Asking for explanations and reasoning, not facts |
| High "temperature" (creative mode) | Low "temperature" (focused mode) |
| The model being given no sources to work from | The model being given documents to read (RAG) |
| Prompts that demand certainty ("give me exactly 10 cases") | Prompts that allow uncertainty ("say when you're unsure") |
What Actually Reduces Hallucinations
Researchers and engineers use several real techniques to make models more truthful:
- Retrieval-Augmented Generation (RAG). Instead of answering from memory alone, the model first retrieves relevant documents from a trusted source, then writes its answer based on those documents. Think of it as an open-book exam instead of a closed-book one. RAG is widely regarded as the single most effective technique for reducing factual hallucinations.
- Citations and source attribution. Models forced to point at the source of each claim hallucinate less — and users can check the source themselves.
- Lower temperature. Reducing randomness makes outputs more conservative, cutting down on wild inventions (though it can't eliminate them).
- Better prompting. Explicitly telling the model "say 'I don't know' if you're unsure" genuinely helps. Models follow instructions — including honest ones.
- Post-generation verification. Some systems run a second check on the answer against trusted sources before showing it.
None of these is a cure. RAG can retrieve bad sources; low temperature can't fix a model that confidently "remembers" wrong. The honest scientific position is that current architectures cannot guarantee truth — they can only reduce error.
Takeaway checklist
Whether you're using a chatbot for homework, work, or a court filing, these habits matter:
- Never trust specifics on first sight. Dates, names, quotations, citations, statistics, and URLs are the highest-risk outputs. Verify each one independently.
- Ask for sources — then actually check them. A model that can show its work is easier to audit. But remember the Mata case: asking the model whether its sources are real is not verification.
- Be careful with high-stakes answers. Legal research, medical information, financial decisions, and anything you'll sign your name to deserve human verification.
- Encourage honesty in your prompts. Adding "only answer from what you know, and say when you're unsure" measurably improves results.
- Cross-check with a search engine. If a claim matters, spend 60 seconds confirming it. Chatbots are brilliant for drafting and explaining — they're dangerous as primary sources of fact.
- Watch out for the confidence trap. A well-written, detailed, authoritative-sounding answer is exactly what a hallucination looks like. Confidence is a writing style, not evidence.
AI chatbots are extraordinary tools for thinking, writing, and explaining — as long as you remember one thing: they predict plausible words, not true ones. The truth-checking is your job. And that's fine — you just need to know the job exists.
This matters more every year as businesses hand these systems real responsibilities — see how AI agents are changing work.
