The other day I got an email from the content director at a marketing agency. They had an urgent request with three bright red exclamation points. It was a five-alarm-fire situation. All hands on deck. They had been hired to produce a 20-page summary of a 263-page regulatory filing. A newer member of their team used an AI tool to condense it, which was a reasonable approach for a document that size. Unfortunately, it wasn’t fact-checked before they sent the draft to their client. It came back with redlines and comments on every page. Half the summary had been hallucinated, and their team hadn’t caught the mistakes before they hit send.
They were in danger of losing one of their biggest clients over the mixup, and they asked me to go through the comments and the original document and rewrite it accurately … while an account manager smoothed over the relationship. Mistakes happen, and we were going to get it fixed.
As I reviewed the summary, I found the problems were significant. The section headings in the second half of the summary didn’t correspond to anything in the original document. Sure, it made sense in the context of the summary. If you hadn’t read the original document, you wouldn’t think anything of it. But it didn’t match anything in the original filing.
The most important graph had also been misinterpreted, and the AI reported the opposite of the actual findings. Perhaps worst of all, the data points cited on page 2 of the summary contradicted figures on page 20 of the same summary.
These weren’t minor formatting issues. They were a series of inaccuracies in a public-facing document that would have created real business problems.
This is what large language model (LLM) hallucinations look like in practice.
What is an AI-generated hallucination?
A hallucination is when a large language model generates content that appears correct but is factually wrong, totally made up, or inconsistent with the source material. The term comes from the way these errors seem plausible, but actually aren’t true.
Most people refer to LLMs as “AI.” They’re tools like ChatGPT, Claude, Gemini, Grok, and similar. We’ve all gotten used to calling them “AI,” but they’re not really “intelligent.” They’re basically an advanced version of autocorrect. Let me explain:
LLMs are based on tokens, not words. These models aren’t analyzing your text. They’re predicting the most statistically likely tokens that answer your request. If you type in “Mary had a little lamb,” it will reply, “whose fleece was white as snow.”
Why? Because that is the most statistically likely series of tokens to finish that sentence.
LLMs don’t verify outputs against truth. Yes, they can search the web. They can scan documents, but they have memory limitations with longer documents (which is why the AI tool hallucinated the second half of that summary).
They don’t independently analyze text. LLMs generate the most likely tokens that satisfy your query.
For business owners and executives using AI to produce reports, summaries, or client-facing content, understanding this distinction is critical.
LLMs are great at structuring sentences, copying a defined writing tone, and formatting information correctly. However, they can also confidently make up information.
Hallucinations aren’t glitches. They’re not something you can easily prompt around. They’re a structural reality of how LLMs work. In their own terms of service, generative AI companies tell users to check outputs.
This is why human editing should always be the final step in an AI content workflow. (There’s an AI editing checklist below.)
What are the most common types of hallucinations?
There are five main types of AI-generated hallucinations to watch out for:
- Fabricated data — statistics, percentages, or figures that don’t exist
- Structural misrepresentation — invented headings, sections, or document hierarchies when summarizing long documents
- Visual misinterpretation — graphs or charts described with reversed or inaccurate findings
- Internal inconsistency — contradictory claims within the same document
- Source conflation — information from one source attributed to another
Why does it do this? Because LLMs were trained on internet forums and existing documents. It will scan its training data and pick out the most statistically likely tokens, not the ones which are most correct.
Why Business Documents Are High-Risk
AI tools perform well on short drafts and general-purpose writing. No one needs to reinvent the wheel when writing a blog post on “10 ways to save water.” The risk increases when the output summarizes specific documents or is asked to summarize large amounts of information. In the earlier example, a long regulatory filing was misinterpreted. There are also risks in summarizing financial reports, legal briefs, and academic research.
In business and academic cases, a hallucination can have serious consequences, including fines or opening your organization to the risk of legal action.
The more technical, dense, or data-heavy the source material, the more it needs a fact check. This is especially true for documents containing tables, charts, and large amounts of numerical data, which current public models can sometimes misread or misinterpret.
Even enterprise LLMs (the ones used by businesses for hundreds of dollars per month) need to be fact checked.
Can You Prompt to Reduce Hallucinations?
How you instruct the model affects your output. So, yes, you can prompt to help reduce hallucinations. However, in my experience, you’ll still get them now and then, especially with longer documents.
Here’s a sample prompt you can use to help reduce LLM hallucinations (disclaimer: you should still fact check the output).
Prompt template:
You are summarizing the following document. Your task is to accurately represent only the information contained in this document. Do not infer, extrapolate, or add context not explicitly stated in the source.
Follow these rules strictly:
- Use only section headings and titles that appear verbatim in the original document.
- When citing data, statistics, or findings, include the page number or section reference from the source.
- If a chart or graph is referenced, describe only what is explicitly labelled. Do not interpret trends unless the document states them.
- If you are uncertain about any figure or finding, flag it with [VERIFY] rather than guessing.
- Do not introduce information from outside this document.
Document: [paste or attach source and give the file name here]
Output: A [X]-page summary organized by the document’s original section structure.
Note: This prompt won’t eliminate hallucinations entirely, but it creates structured constraints to reduce a model’s tendency to fill in gaps. The [VERIFY] flag should alert you to uncertainty or shaky data.
AI Editing Checklist for Generated Summaries
Once you have your output, always fact check. Use this checklist before any AI-generated content leaves your desk.
Structure
- Every section heading in the AI output corresponds to a section that exists in the source
- The document structure follows the original (it’s not an invented hierarchy)
Data and statistics
- Every numerical figure has been verified against the source document
- Data points are consistent — cross-check figures that appear more than once
- Percentages, totals, and comparisons have been manually confirmed
Charts and visual data
- Each graph or chart reference has been checked against the original
- Directional findings (increases, decreases, peaks) match what the chart shows
- The AI has not invented visual data that does not exist
Internal consistency
- Claims on later pages do not contradict claims on earlier pages
- Conclusions align with the data presented in the original or verifiable sources
Source check
- No information appears that isn’t traceable to the source document
- Quoted language has been verified verbatim against the original
- Any [VERIFY] flags have been resolved
Speed vs Accuracy: AI-Generated Hallucinations
AI tools speed up our work. There’s no denying that. A 263-page document summary is exactly the kind of task where we want help. But being fast and being accurate aren’t the same thing. AI still needs human review.
The people involved in the example at the start of this post weren’t trying to be careless. They were under time pressure, so they used a tool to speed up the work. It was perfectly understandable, and businesses do it every day.
However, AI tools sometimes produce output that looks and sounds professional but can be filled with errors.
Companies need a formal fact-check process when generating content.
Building a verification step into your AI workflow can mean the difference between AI as your productivity partner and AI as a liability.
If you’re producing AI-assisted content for clients, stakeholders, or public audiences, a professional editor can smooth out phrasing and double check the facts. Contact Jocelyn for a no-obligation quote for AI editing services.