“Generative AI models consistently produce factual inaccuracies in their outputs.”

The preliminary results of FACTS send a clear signal to the industry: despite models becoming increasingly intelligent, they are far from perfect. All tested models, including Gemini3Pro, GPT-5, and Claude4.5Opus, failed to achieve an overall accuracy rate exceeding 70%. The FACTS (Factual Consistency and Truthfulness Score) team from Google has jointly released the FACTS Benchmark Suite with the data science platform Kaggle today.

Large language models (LLMs) still have a habit of making things up—what researchers call hallucinations. These outputs can look perfectly plausible yet be factually wrong or unfaithful to their source. The “30% rule” for AI is a media shorthand suggesting that roughly 30% of AI outputs may contain errors or hallucinations, though actual rates vary widely by model, domain, language, and benchmark design.

Hallucination in Large Language Models (LLMs) refers to outputs that appear fluent and coherent but are factually incorrect, logically inconsistent, or entirely fabricated. As LLMs are increasingly deployed in education, healthcare, law, and scientific research, understanding and mitigating hallucinations has become critical.

We tested 19 different AI models, each of which had an average similarity score in the range between 0.9065 and 0.7896. xAI, Google, and Anthropic produced the four most factually consistent models (xAI Grok-3, Google Gemini-Flash-1.5, Anthropic Claude-3.5-Haiku, xAI Grok-4), whereas OpenAI's models all performed worse. Grok-3 was the only model to score above the benchmark for all 15 topics.

The original leaderboard, released two years ago, became a key benchmark for measuring hallucination rates in generative AI. Some of the latest LLMs now hallucinate at rates between 1 and 3 percent, according to analysis by Vectara.

The hallucination rates for conventional chatbots were approximately 40%. For questions on information that is not issued by CIS, the hallucination rates for Google-based chatbots were 19% for GPT-4 and 35% for GPT-3.5.

A study conducted by researchers at CCC, which is based at the MIT Media Lab, found that state-of-the-art AI chatbots — including OpenAI's GPT-4, Anthropic's Claude 3 Opus, and Meta's Llama 3 — sometimes provide less-accurate and less-truthful responses to users who have lower English proficiency, less formal education, or who originate from outside the United States.

Large Language Models (LLMs) have made significant strides in generating human-like text, but their tendency to hallucinate—producing factually incorrect or fabricated information—remains a pressing issue. This tutorial provides a comprehensive exploration of hallucinations in LLMs, introducing participants to the key concepts and challenges in this domain.

Accuracy and reasoning issues remained. There were cases of misinformation even in advanced systems. OECD.AI warned of operational risks from incorrect responses and reinforced the need for frameworks to report risks. Stanford's AI Index 2025 confirmed that, although there were advances, challenges in security and reasoning persisted, slowing down critical applications without additional controls.

On apples-to-apples benchmarks, such as Vectara's summarization leaderboard, performance improved across the board. Several top models dropped below 1%, including Google’s Gemini-2.0-Flash at roughly 0.7%, with OpenAI and Gemini variants clustering around 0.8–1.5% (Vectara, 2025). However, hallucinations remain high in complex reasoning and open-domain factual recall, where rates can exceed 33%.

Hallucination in LLMs is a phenomenon where models produce fluent yet factually unsupported outputs in open-world settings. Hallucination is increasingly recognized as a structural feature of deep learning models—particularly under the open-world assumption, where models confront an unbounded, ever-evolving environment and must generalize far beyond finite training data.

However, a known issue of LLMs is their tendency to generate falsehoods or hallucinate contents, posing a significant hurdle to broader applications. Even state-of-the-art LLMs such as ChatGPT are susceptible to this issue as shown in Borji (2023); Zhuo et al. (2023); Min et al. (2023), which raises concerns about the practical utility of these models.

AI does not “think” like humans. It does not know and retrieve facts like a database, neither does it understand meaning or have emotions. Instead, AI generates predictions based on patterns. Large Language Models (LLMs), a subset of AI, has been trained on huge amounts of data and uses patterns to predict the words most likely to follow. These create three recurring risks: Hallucinations (confident but false answers), Bias (reproduction of unfair patterns), Ecological impact (high computational and energy usage).

The top-performing model (Gemini 3 Pro) achieves just 68.8% overall accuracy. That means even the best available AI gets facts wrong roughly one-third of the time across these tasks. All models perform significantly better when they can search the web (60-84% accuracy) versus relying on internal knowledge alone.

A recent study highlighted that generative AI tools could agree with false statements up to a quarter of the time, depending on the statement category. Such inaccuracies can undermine efforts to provide clear and reliable information in critical areas. Microsoft's Bing AI, for example, was found to produce inaccurate sources nearly one in ten times when tasked with answering complex questions.

Gemini-2.0-Flash-001 recorded the lowest AI hallucination rate at 0.7% as of April 2025 (Vectara Leaderboard). Four AI models now have sub-1% hallucination rates on summarization benchmarks. OpenAI’s o3 reasoning model hallucinated 33% of the time on PersonQA, double the rate of its predecessor o1.

Some of the latest LLMs now hallucinate at rates between 1 and 3 percent, according to analysis by Vectara. Over the past two years, the overall trend is that hallucination rates in many AI models have fallen. According to data collected by the AI company Hugging Face, the hallucination rate of LLMs has so far decreased by around 3 percentage points each year.

Combining symbolic logic with deep learning, neuro-symbolic AI is gaining traction in fields requiring reasoning and factual accuracy. In 2026, these hybrid systems are revolutionising legal AI, scientific research and education. This approach mitigates the problem of hallucinations common in pure neural models, leading to more reliable outputs.

Generative AI systems frequently produce incorrect information with remarkable confidence, creating a fundamental trust problem for enterprises. These "hallucinations" occur in AI outputs at rates between 3% and 27%, posing significant challenges for organizations that require factual accuracy.

In a benchmark test, OpenAI's o3 model was found to hallucinate 33% of the time, more than twice the rate of the older o1 model. The newer o4-mini model hallucinated 48% of the time. The article notes that hallucination rates are also rising for competing reasoning models such as those from Google and DeepSeek.

Vectara's Hallucination Leaderboard is a widely recognized industry benchmark for evaluating hallucination rates in summarization tasks using grounded documents. Top models in 2025 consistently achieve rates under 3%, with leaders below 1%, demonstrating substantial improvements over prior years.