The other day I got an email from the content director at a marketing agency. They had an urgent request with three bright red exclamation points. It was a five-alarm-fire situation. All hands on deck. They had been hired to produce a 20-page summary of a 263-page regulatory filing. A newer member of their team used an AI tool to condense it, which was a reasonable approach for a document that size. Unfortunately, it wasn’t fact-checked before they sent the draft to their client. It came back with redlines and comments on every page. Half the summary had been hallucinated, and their team hadn’t caught the mistakes before they hit send.

They were in danger of losing one of their biggest clients over the mixup, and they asked me to go through the comments and the original document and rewrite it accurately … while an account manager smoothed over the relationship.

As I reviewed the summary, I found the problems were significant. The section headings in the second half of the summary didn’t correspond to anything in the original document. Sure, it made sense in the context of the summary. If you hadn’t read the original document, you wouldn’t think anything of it. But it didn’t match anything in the original filing.

The most important graph had also been misinterpreted, and the AI reported the opposite of the actual findings. Perhaps worst of all, the data points cited on page 2 of the summary contradicted figures on page 20 of the same summary.

These weren’t minor formatting issues. They were a series of inaccuracies in a public-facing document that would have created real business problems.

This is what large language model (LLM) hallucinations look like in practice.

What is an AI-generated hallucination?

A hallucination is when a large language model generates content that appears correct but is factually wrong, totally made up, or inconsistent with the source material. The term comes from the way these errors seem plausible, but actually aren’t true.

Most people refer to LLMs as “AI.” They’re tools like ChatGPT, Claude, Gemini, Grok, and similar. We’ve all gotten used to calling them “AI,” but they’re not really “intelligent.” They’re basically an advanced version of autocorrect. Let me explain:

LLMs are based on tokens, not words. These models aren’t analyzing your text. They’re predicting the most statistically likely tokens that answer your request. If you type in “Mary had a little lamb,” it will reply, “whose fleece was white as snow.”

Why? Because that is the most statistically likely series of tokens to finish that sentence.

LLMs don’t verify outputs against truth. Yes, they can search the web. They can scan documents, but they have memory limitations with longer documents (which is why the AI tool hallucinated the second half of that summary).

They don’t independently analyze text. LLMs generate the most likely tokens that satisfy your query.

For business owners and executives using AI to produce reports, summaries, or client-facing content, understanding this distinction is critical.

LLMs are great at structuring sentences, copying a defined writing tone, and formatting information correctly. However, they can also confidently make up information.

Hallucinations aren’t glitches. They’re not something you can easily prompt around. They’re a structural reality of how LLMs work.

What are the most common types of hallucinations?

There are five main types of AI-generated hallucinations to watch out for:

• Fabricated data — statistics, percentages, or figures that don’t exist
• Structural misrepresentation — invented headings, sections, or document hierarchies when summarizing long documents
• Visual misinterpretation — graphs or charts described with reversed or inaccurate findings
• Internal inconsistency — contradictory claims within the same document
• Source conflation — information from one source attributed to another

Why does it do this? Because LLMs were trained on internet forums and existing documents. It will scan its training data and pick out the most statistically likely tokens, not the ones which are most correct.

Why Business Documents Are High-Risk

AI tools perform well on short drafts and general-purpose writing. No one needs to reinvent the wheel when writing a blog post on “10 ways to save water.” The risk increases when the output comes specific documents or subjects where precision matters. In our example, a regulatory filing was misinterpreted. But there are also risks in summarizing financial reports, legal briefs, and academic research.

In these cases, a hallucination can have serious consequences, including fines and opening your organization to the risk of legal action.

The more technical, dense, or data-heavy the source material, the higher the hallucination risk. This is especially true for documents containing tables, charts, and numerical data, which current models can misread or misinterpret.

Can You Prompt to Reduce Hallucinations?

How you instruct the model affects your output accuracy. So, yes, you can prompt to help reduce hallucinations. However, in my experience, you’ll still get them with longer documents due to memory limitations.

Here’s a sample prompt you can use to help reduce LLM hallucinations (disclaimer: you should still fact check the output).

Prompt template:

You are summarising the following document. Your task is to accurately represent only the information contained in this document. Do not infer, extrapolate, or add context not explicitly stated in the source.

Follow these rules strictly:

1. Use only section headings and titles that appear verbatim in the original document.
2. When citing data, statistics, or findings, include the page number or section reference from the source.
3. If a chart or graph is referenced, describe only what is explicitly labelled — do not interpret trends unless the document states them.
4. If you are uncertain about any figure or finding, flag it with [VERIFY] rather than guessing.
5. Do not introduce information from outside this document.

Document: [paste or attach source and give the file name here]

Output: A [X]-page summary organized by the document’s original section structure.

Note: This prompt won’t eliminate hallucinations entirely, but it creates structured constraints to reduce the model’s tendency to fill in gaps. The [VERIFY] flag should alert you to uncertainty and shaky data.

Fact-Checking Checklist for AI-Generated Content

Once you have your output, always fact check. Use this checklist before any LLM-generated content leaves your desk.

Structure

• Every section heading in the AI output corresponds to a section that exists in the source
• The document structure follows the original (not an invented hierarchy)

Data and statistics

• Every numerical figure has been verified against the source document
• Data points are consistent — cross-check figures that appear more than once
• Percentages, totals, and comparisons have been manually confirmed

Charts and visual data

• Each graph or chart reference has been checked against the original
• Directional findings (increases, decreases, peaks) match what the chart shows
• The AI has not invented visual data that does not exist

Internal consistency

• Claims on later pages do not contradict claims on earlier pages
• Conclusions align with the data presented in the body

Source fidelity

• No information appears that isn’t traceable to the source document
• Quoted language has been verified verbatim against the original
• Any [VERIFY] flags have been resolved

The Reality of AI-Generated Hallucinations

AI tools speed up our work. There’s no denying that. A 263-page document summary is exactly the kind of task where we want help. But being fast and being accurate aren’t the same thing. AI output needs human review.

The people involved in the example at the start of this post weren’t trying to be careless. They were under time pressure, so they used a widely available tool to speed up the work. That tool produced something that looked professional but was filled with errors.

Companies need a formal fact-check process when generating content.

Building a verification step into your AI workflow can mean the difference between AI as your productivity partner and AI as a liability.

If you’re producing AI-assisted content for clients, stakeholders, or public audiences, we can take the editing off your plate and do the fact-checks for you. Contact Jocelyn for a no-obligation quote.