Safety and the AI Act

AI hallucinations at work: how to check answers

AI can confidently give you a made-up law, number or source. Why it happens and how to check answers before they reach a client. Four habits that work.

Bartosz CzerniawskiFounder of dajtoAI
Published: 6 min read
Two colleagues at a table in a bright office checking printed pages and marking a line with a highlighter
Contents

An AI hallucination is an answer that sounds confident and fluent but is false: an invented law, a source that doesn’t exist, a number from thin air. The model isn’t lying on purpose. It builds a plausible text, and plausible doesn’t mean true.

Below I explain where this comes from, what to watch most closely and how to build simple checking habits at work. Position as of 14 November 2026.

Where do hallucinations come from?

A language model learns which words tend to follow each other. With things that show up often and consistently in the data (grammar, syntax, widely known facts) it does very well. It does worse with rare, arbitrary facts, such as the birthday of a little-known person or the section number of an obscure statute. There’s no pattern to work it out from.

OpenAI researchers described another reason in September 2025: the tests used to evaluate models often reward guessing. “I don’t know” scores zero, while a guess has some chance of being right, so the model learns to guess. That explains why hallucinations won’t vanish with one update, even if newer models are usually quicker to admit they don’t know.

For a business the lesson is simple. A good tool reduces the problem but doesn’t remove it. You need to build the checking on your side.

What gets invented most often?

The riskiest things are concrete ones, because they look true.

Type of content Typical error How to check
Numbers and statistics A realistic-looking figure with nothing behind it Find the primary source and compare
Quotes A sentence attributed to someone who never said it Search for the exact quote in the original
Links and sources An address or publication that doesn’t exist Open every link
Laws and regulations An article number from another act, or outdated wording Check the official text (EUR-Lex, national legal database)
Names and company details Mixed-up products, people or features The company’s own site, registers
Recent developments A description of the state of things months ago Turn on web search and check the dates

The rule of thumb: the more concrete something is, and the easier it is to paste on, the more carefully you check it.

What happens when nobody checks?

In 2023 two American lawyers filed a court brief citing cases that ChatGPT had suggested. The cases didn’t exist. A New York court fined the lawyers and their firm $5,000, and the judge found bad faith, because they kept defending the citations even after the court raised doubts. Technology wasn’t the main problem there. The missing check was.

The same logic shows up in the Canadian Air Canada case, which I describe in the post on an AI assistant built on company knowledge: a company answers for what its chatbot says, just as it does for what’s on its website.

Habit 1: Give it source material

The most effective approach is not to ask a model from memory, but to hand it a document and tell it to work from that. “Summarise this policy”, “answer based on this offer”, “pull the deadlines out of this contract”. The model then isn’t guessing. It’s reading.

That’s also how well-built assistants on company knowledge work: they answer from your documents and show which passage they used. The rules for pasting data into tools are in the post on what not to paste into ChatGPT.

Habit 2: Allow “I don’t know”

Say it right in the prompt: “if you’re not sure or the data is missing, say so and don’t guess”. Add: “mark what’s a fact from the document and what’s your inference”. This doesn’t guarantee correctness, but it cuts down on confident inventions and makes checking easier, since you can see at once which sentences are conclusions.

A simple trick helps too: ask for sources, then open every one. A model can invent the source itself, so a link is never to be taken on trust.

Habit 3: Check whatever has consequences

You can’t check every sentence. You can check whatever money, law or reputation depends on. A rule I’d write into your company AI policy:

  • numbers, prices and deadlines are checked at the source,
  • regulations are read in the original, not in an AI summary,
  • quotes and names are always verified,
  • texts about people and companies are read by someone who knows them.

A draft email to a colleague or a few headline ideas can go after a light read. A reply to a customer about a refund can’t.

Habit 4: A second pair of eyes before sending

When content goes outside (an offer, a newsletter, a reply to a complaint, a blog post), someone other than the author should read it. Authors see what they meant to write, not what the AI wrote. A short checklist helps, three to five questions: do the numbers have a source, do the links work, are the names right, does anything look suspiciously smooth.

For automations that send messages on their own, the second pair of eyes is an approval step before sending. Where a mistake costs something, nothing goes out without a human.

What about the law?

Article 4 of the AI Act requires companies to ensure the AI literacy of the people who use it. Recognising hallucinations and knowing when to check are exactly that kind of skill. I cover the details in the guide to Article 4. It isn’t a separate duty to “check content”, just good practice that the provision reinforces.

Do it yourself or get help?

You can introduce the four habits on your own in an afternoon: write the rules, add a checklist, name a reviewer. If you want your team to learn this on its own examples and tools, we do it in AI training, from PLN 4,900 net (+23% VAT).

Summary

Hallucinations aren’t a flaw of one tool. They’re a feature of how language models work. You can’t eliminate them entirely, but you can catch them. Give source material, allow “I don’t know”, check what has consequences, and don’t send anything out without a second pair of eyes.

One thing to do today: take the last text AI prepared for you to send, open every link and check every number. You’ll see how it looks in practice.

Sources

  1. OpenAI, Why language models hallucinate, 5 September 2025.
  2. Bloomberg Law, Bogus ChatGPT citations lead to $5,000 fines (Mata v. Avianca, S.D.N.Y., 22 June 2023).
  3. Moffatt v. Air Canada, 2024 BCCRT 149, Civil Resolution Tribunal (British Columbia), 14 February 2024.
  4. Regulation (EU) 2024/1689 (AI Act), Article 4, EUR-Lex.
  • hallucinations
  • fact-checking
  • safety
  • ChatGPT

Written with the help of AI, edited and checked by a human.

Bartosz CzerniawskiFor 10 years he made films, campaigns and content for businesses. Today he builds AI tools and processes.More about me
Call · 30 min · free

Facing a similar problem?

Book a 30-minute call.

Book a free 30-minute call(opens in a new tab)
All posts
Newsletter

AI in business, straight to the point

What works, what it costs and what to avoid. Straight to your inbox, no spam. The newsletter is in Polish.

The data controller is Moon Lion Media Bartosz Czerniawski. Sign-ups are handled by Kit. More in the privacy policy.

Manage cookies

Essential · always on

They remember your choice and keep the site secure. The site won't work without them.