AI as a search engine: when ChatGPT and Claude make things up and how to check

Guide · AI and verification · Check it yourself Chat bubbles with the question How do I check this and a confident answer, next to a magnifying glass with a question mark and a checkmark: verifying AI answers
⏱ approx. 18 min read 📚 Category: Blog 🕓 Updated: October 22, 2026

Short answer: ChatGPT, Claude, Gemini and other chatbots don’t “lie” in the human sense; they generate plausible-sounding text that is sometimes false (so-called hallucinations). They are most often wrong about rare facts, recent events, quotes, links and numbers, and they say it in a confident tone. Research shows that even tools with search can give wrong sources. So treat AI as a fast assistant for finding leads, not as the final source: ask for sources, open them, check the date and confirm important things with a second, independent source.

This article complements our guides to zero-click searches and meta-prompting. There we wrote about how AI answers for you and how to write prompts; here we focus on when not to believe it and how to check quickly. This applies to all tools, including Claude.

Why AI makes things up: “hallucinations” in plain English

According to OpenAI, hallucinations are confidently stated answers that sound plausible but are false. A model learns to predict the next word from a huge amount of text that has no “true” and “false” labels. Spelling mistakes disappear with scale because they follow patterns, but rare facts, like a specific person’s birthday, cannot be predicted from patterns, so the model guesses them. The authors give an example in which a popular chatbot, asked for the title of one of the study authors’ PhD dissertation, gave three different wrong answers, and for his birthday three different wrong dates.

The second cause is how models are evaluated. OpenAI argues that standard tests reward guessing rather than admitting uncertainty. In the SimpleQA test, the older o4-mini model abstained in 1% of cases, was correct 24% of the time and wrong as much as 75% of the time, while the newer gpt-5-thinking-mini abstained in 52%, was correct in 22% and wrong in 26%. Guessing yields slightly more hits at the cost of many more errors.

Practical takeaway: a model’s error is not an exception but a consequence of how it works. The more specific and rare your question (a name, a date, a number, a quote), the greater the risk of fabrication.

What research says: how often AI gets it wrong

Selected studies on the accuracy of AI assistants and AI search engines
StudyWhat was testedResult
EBU/BBC (published 2025)Over 3,000 answers from ChatGPT, Copilot, Gemini and Perplexity; 22 public service media, 18 countries, 14 languages45% of answers with a significant issue, 31% with serious sourcing problems, 20% with major factual errors; Gemini performed worst (76% with a significant issue)
Tow Center / Columbia Journalism Review (March 2025)8 AI search engines, matching quotes to sourcesOver 60% of answers without correct information; from 37% errors (Perplexity) to 94% (Grok-3 Search); answers often confident despite being wrong
Demos, “Election hallucinations” (2026, via Demagog)Questions about elections put to chatbotsFactual errors: Replika 56.4%, ChatGPT 46.2%, Grok 9%; nearly 47% of answers without sources or links
AI Security Institute (2025, via Demagog)Comparison of GPT-4o, Claude and Mistral with Google Search for acquiring knowledgeChatbots improved knowledge by 0.13 points (7-point scale), Google by 0.08; working with AI was 6–10% faster; 87% of participants considered chatbots reliable
Reuters Institute, Digital News Report 2025Whether and what people use chatbots for7% had used a chatbot for news in the past week (15% of under-25s); to check suspicious information, 9% would turn to a chatbot, 14% to social media, 38% to a trusted news source
How to read these numbers: the studies differ in method, language and year, and models change every few months, so don’t compare percentages between studies and don’t assume a given figure applies to today’s version. The shared conclusion stays the same, though: a confident tone is not proof of correctness, and sources given by AI need checking.

When AI is most often wrong: 10 situations

High-risk situations and how to respond to them
#SituationWhy it’s riskyWhat to do
1Rare facts: dates, people, titlesNo pattern to predict from (OpenAI)Check in the primary source or an encyclopedia
2Recent eventsA model’s knowledge has a cutoff date. Demagog described how ChatGPT (GPT-3.5) claimed to be up to date although its data ended in 2021, and wrongly stated that Benedict XVI was aliveUse a mode with search and check the source’s date
3Quotes and linksTow Center: many tools give wrong citations; Gemini and Grok fabricated links more often than they gave correct onesOpen every link and find the quote in the text
4Case law and legislationA model can invent non-existent cases (the Mata v. Avianca case)Check in the official database of laws or judgments
5Numbers and statisticsThe model “rounds” and mixes data from different yearsFind the table in the source (national statistics office, Eurostat, the report)
6Elections and politicsDemos: wrong dates, constituencies and candidates; it’s easy to confuse fact with opinion (Demagog)Use official election commission websites and fact-checkers
7Health, medicines, dosageA mistake can cost your health; the model doesn’t know your historyCheck the leaflet and consult a doctor or pharmacist
8Local and niche topicsFew texts in the training data, so more guessingCheck the website of the authority or company, or local media
9Questions with a false premiseThe model tends to agree with the user (research on sycophancy)Ask the question neutrally, without suggesting the answer
10Absurd or joking sources onlineIn 2024, Google’s AI Overviews gave, among other things, advice about glue on pizza, which journalists linked to forum jokes and satireCheck whether the source is serious and not a meme

Case study: four cases that cost money

1. Lawyers and non-existent cases (Mata v. Avianca, 2023)

Lawyers filed a brief in a New York court citing non-existent cases generated by ChatGPT. When the other side raised the problem, they continued to claim the cases were real. In a ruling of June 22, 2023, the court fined the two lawyers and their firm $5,000, pointing in the documents to obvious irregularities, such as mismatched judges’ names and sentences cut off mid-way. According to reports, ChatGPT didn’t confirm the cases existed; it just generated them. Lesson: asking the same model “is this real?” is not verification.

2. An airline chatbot (Moffatt v. Air Canada, 2024)

Air Canada’s chatbot suggested to a passenger that he could buy a regular ticket and apply for a bereavement fare refund within 90 days, which contradicted the airline’s policy. A tribunal in British Columbia ruled that the airline is responsible for information on its website, including from a chatbot, and awarded a partial refund of CAD 650.88 plus interest and fees. The tribunal also noted that customers cannot be expected to verify information across different parts of a website. Lesson: for important terms (refunds, contracts), save the answer and confirm it in the terms and conditions.

3. An AUD 440,000 report (Deloitte, 2025)

A report for the Australian government contained fabricated footnotes, including a made-up quote from a federal court judgment and references to non-existent academic papers. The corrected version of October 3, 2025 states that a generative AI system (Azure OpenAI) was used, and Deloitte agreed to refund part of its fee. Lesson: even professionals don’t check every footnote, and it’s footnotes that fail most often.

4. Glue on pizza (Google AI Overviews, 2024)

In May 2024, AI Overviews displayed advice such as adding “non-toxic” glue to pizza sauce and eating one small rock a day. According to The Register, this was influenced by Reddit jokes and a satirical article from The Onion, among others. Google admitted that some answers were odd or inaccurate, and announced better detection of satire and less reliance on user-generated content, e.g. from forums. Lesson: AI can repeat an internet joke with a completely straight face.

Model flattery: when AI tells you what you want to hear

In a 2023 study, Anthropic described sycophancy: a model tailors its answer to the user’s beliefs rather than to the truth. The authors link this to training on human ratings, because answers that agree with the user’s views are more likely to win approval, and five leading assistants tested showed sycophancy across four tasks. The study doesn’t give a single figure for how often it happens.

  • Don’t suggest the answer: instead of “is it true that X is harmful?” ask “what does research say about the effect of X on health, and how strong is the evidence?”.
  • Ask for counterarguments: “give me the strongest arguments against my thesis”. This complements what we described in our article on echo chambers.
  • Don’t push with your tone: if the model answered correctly and you write “are you sure?”, a changed answer is not proof that the original was wrong.

How to check AI answers: 9 steps

This is our own compilation, based on the research and cases described above. It requires no specialist tools, and the first check takes a few minutes.

1

Assess the stakes of the question

Before you start, decide: does a mistake cost something (health, legal issues, money, publication)? The higher the stakes, the more verification. A brainstorm needs less than a quote in an article.

2

Ask for sources, but don’t take them on trust

A list of sources can give a false sense of credibility. Demagog described a test in which ChatGPT gave non-existent sources, and the Tow Center showed that tools with search also give wrong citations.

3

Open every link

Click to see whether the page exists at all (a 404 error is a red flag), whether it’s the right organisation and whether the text is on topic. Fabricated links were most common in Gemini and Grok in the Tow Center study.

4

Find the specific sentence in the source

Copy the key claim from the AI’s answer and search for it in the source text (Ctrl+F). Chatbots may cite pages that don’t support their claim.

5

Check the date

See when the source was created and whether the matter has been updated. A model’s knowledge has a cutoff date, and Demagog showed that a model can insist it is up to date even though its data is older.

6

Confirm with a second, independent source

Search for the same fact on Google, a trusted website or a database. A second chatbot is not an independent source, because it can repeat the same mistake. We describe the techniques in our guide to step-by-step fact-checking.

7

Check numbers and quotes in the original

Take percentages, dates and legal wording from the table, the legal act or the report, not from a summary. Our advice: if the model gives a number, ask for a link to the table and find it in that table.

8

Test the model with a question you know the answer to

First ask a question from your own field or about something verifiable. If it gets it wrong, treat the rest of the conversation with more caution.

9

For health and law, go to a human

In matters that affect life decisions, AI can help you prepare questions, but consult the decision with a doctor, lawyer or public authority. We do not give medical or legal advice.

10 prompts that reduce the risk of fabrication

A good prompt won’t eliminate hallucinations, but it helps you catch them. These are our suggestions (ready to copy and adapt). You’ll find more on building prompts in our guide to meta-prompting.

Prompts for searching and verifying with AI (our proposal)
#PromptWhy
1If you’re not sure, say “I don’t know” instead of guessing.Limits the guessing that OpenAI writes about
2Separate confirmed facts from assumptions and mark the level of certainty.Shows what needs checking
3Give sources with working links and publication dates.Makes checking easier, but you still have to open the links
4Quote verbatim the passage of the source on which you base this claim.Lets you search for the quote in the text
5What is your knowledge cutoff date, and can you check this online?Reminds about recency
6Give the strongest arguments against this thesis.Limits flattery and confirmation bias
7Which parts of your answer are most likely to be wrong?Points to places to verify
8Ask me clarifying questions before you answer.Less guessing of context
9Suggest 5 phrases and operators to search for this on Google and in source databases.AI as a search aid, not a judge (see our 15 Google operators)
10List exactly what I need to check manually, and where.Gives you a checklist
Note: the model can also fabricate a verbatim quote or link, so always combine prompts 3 and 4 with steps 3 and 4 from the list above.

When AI really helps with searching

The AI Security Institute study (via Demagog) found that chatbots can be as effective as a search engine, even slightly faster, but still require verification. They work well for tasks where a mistake is easy to spot or cheap:

  • Brainstorming keywords: synonyms, technical names and foreign-language equivalents of phrases you’ll type into Google. See 15 Google operators.
  • Summarising a text you pasted yourself: the risk of error is lower because the model works on your material, though it may still leave something out.
  • Explaining concepts and planning research: ask for a map of the topic and a list of questions to check.
  • Drafting arguments and counterarguments: for further verification.
  • Translating and reading foreign-language sources: as a first pass; check the important passages.

A simple rule (ours): AI for finding leads, search engines and sources for confirming. We’ve also written about how you increasingly get an answer without visiting a website, which makes it all the more important to check where it comes from (zero-click searches).

Frequently asked questions (FAQ)

Not in the human sense, because it has no intention to deceive. It generates text that sounds plausible but may be false. Such errors are called hallucinations.

According to OpenAI, they are confidently stated answers that look plausible but are false. They arise partly because the model predicts the next words, and rare facts cannot be predicted from patterns.

Yes. Every current AI assistant, including Claude, ChatGPT, Gemini and Perplexity, can give wrong information, invent a source or use outdated data. So important information needs to be verified regardless of the tool.

It helps with recency but doesn’t guarantee correctness. In March 2025, the Tow Center showed that AI search engines failed to give correct information in over 60% of the cases tested, and results varied widely between tools.

Open the link, check the domain, date and author, then find the sentence the AI refers to in the text. If the page doesn’t exist or doesn’t support the claim, the answer is unreliable.

You can use it as an extra signal, but it’s not independent verification, because models can repeat the same mistakes. It’s better to check with a search engine and the primary source.

Models generate fluent text with no built-in “I’m not sure” signal. The Tow Center noted that many tools answer in a confident tone despite errors, and OpenAI points out that tests reward guessing.

Not without checking. In Mata v. Avianca, ChatGPT generated non-existent cases, and Deloitte’s report for the Australian government contained fabricated footnotes.

It can help with planning: suggest phrases to search for, point to types of sources and explain concepts. The judgement of what is true should rest on primary sources and fact-checkers.

Ask neutrally, ask for a confidence level, sources and a date, and for counterarguments. Don’t suggest the answer, because models tend to agree with the user (sycophancy).

It can help you understand concepts and prepare questions, but it doesn’t replace a doctor or lawyer. Wrong dosage or invented case law can have serious consequences.

In Moffatt v. Air Canada, the tribunal ruled that the airline is responsible for information given by its chatbot. However, that is a Canadian ruling, and laws in your country may differ, so in a dispute consult a lawyer.

There’s no single answer. Results depend on the tool, the version and the topic, and studies go out of date quickly. Choose a tool that shows its sources, and always check them.

Yes. Demagog, a Polish fact-checker, described chatbot errors, including on political matters and when citing sources. For any local context, it’s important to check in local, official sources.

More from our blog

Sources: OpenAI: Why Language Models Hallucinate; arXiv: Why Language Models Hallucinate; Journalism.co.uk: EBU/BBC report; Nieman Lab: Tow Center study; Demagog: how chatbots get political topics wrong; Demagog: how chatbots work and what to watch for with sources; Press Gazette: Digital News Report 2025; Seyfarth: the Mata v. Avianca case; Manatt: Moffatt v. Air Canada; Fast Company: Deloitte’s report for the Australian government; The Register: Google AI Overviews; Anthropic: Towards Understanding Sycophancy in Language Models. Study results refer to specific tool versions and dates, and models change quickly. This text is not medical or legal advice.

Scroll to Top