Verify any claim · lenz.io
“Artificial intelligence systems produce incorrect answers in 70% of evaluated cases.”
The conclusion
Available evidence does not establish a general 70% error rate for artificial intelligence systems. Reported rates vary dramatically by model, task, benchmark, and definition of error. A result near 70% appears in a narrow medical evaluation, but presenting it as broadly representative of evaluated AI answers is unsupported.
Caveats
- The 70% figure appears to come from a narrow evaluation and cannot be generalized to all AI systems.
- Error and hallucination rates depend heavily on the task, model, benchmark, and scoring method.
- Some apparent corroboration comes from secondary headlines lacking enough methodological context to verify equivalence.
Fact-check inside the tools you already use
Connect Lenz to ChatGPT, Claude, or WhatsApp and check a claim mid-conversation.
Or create a free account to bookmark this verification and run your own checks.
Sources
Sources used in the analysis
Out of the 150 cases, ChatGPT provided correct answers for 74/150 (49%) of cases (Fig 1).
Gemini 2.5 Pro demonstrated the highest performance, correcƟng the physician's error in 55.0% of cases (n=110/200), followed by Claude Sonnet 3.5 (48.5%) and Sonnet 4 (47.0%). … In contrast, DeepSeek V3 corrected only 20.0% of cases.
On MMLU-Pro (10-choice; primary-model accuracy 0.16–0.29 on the full sample), every primary instruct model overstates its confidence verbally. … Averaged over variants passing the V P R ≥ 0.80 inclusion threshold, mean verbal confidence is 60–78 percentage points above accuracy and 25–60 points above mean token-probability confidence on the same rows.
LLM-as-a-Judge has become the dominant evaluation paradigm for language models, but judge validation in practice relies on exact-match agreement, a metric that does not correct for chance and systematically overstates discriminative ability. … First, kappa deflation: raw agreement overstates chance-corrected discrimination by 33–41pp in all 21 evaluated models.
A study published in BMJ Open suggests that half of answers provided by five publicly available artificial intelligence (AI)–driven chatbots in response to medically related questions are inaccurate and incomplete. … About half (49.6%) of responses were problematic, with 30% considered somewhat problematic and 19.6% deemed highly problematic.
Collectively, they provided incorrect answers to more than 60 percent of queries.
Using its newly introduced FACTS Benchmark Suite, the company found that even the best AI models struggle to break past a 70% factual accuracy rate. … The top performer, Gemini 3 Pro, reached 69% overall accuracy, while other leading systems from OpenAI, Anthropic, and xAI scored even lower.
We show that current state-of-the-art LLMs do not accurately diagnose patients across all pathologies (performing significantly worse than physicians), follow neither diagnostic nor treatment guidelines, and cannot interpret laboratory results, thus posing a serious risk to the health of patients.
Management safety performance was markedly lower. Safe corticosteroid recommendations (e.g. not recommending immediately with evidence of active infection) occurred for as few as 7.2% (Gemini 3 Flash Preview; 95% CI 5.6-8.8) and up to only 23.5% (GPT-5 mini; 95% CI 20.8-26.1) cases.
In a new accuracy benchmark, hallucination rates across 26 top models range from 22% to 94%.
Physiologically impossible errors (E2) showed the lowest performance: 46.2% (CT) and 33.7% (MRI) for GPT-4.1, compared with 32.7% and 25.0% for Llama 3.3, respectively. … Mislabeling errors (E1) were detected in 49.0% by GPT‑4.1 and 33.7% by Llama 3.3 for MRI.
We evaluate a wide range of models on these platinum benchmarks and find that, indeed, frontier LLMs still exhibit failures on simple tasks such as elementary-level math word problems.
When similarity rates were evaluated, the highest similarity was observed in 'Prescribed medications' (98.3%), followed by 'Dosage information of prescribed medications' (97.5%), while the lowest similarity was found in 'In-depth inquiry of complaints, signs, and symptoms' (0.0%), followed by 'Potential side effects of prescribed medications' (13.3%).
We study the error rate of LLMs on tasks like arithmetic that require a deterministic output, and repetitive processing of tokens drawn from a small set of alternatives. … Our empirical tests span 8 different tasks and 3 different models with 0.2 million total test prompts. Formula (1) provides a surprisingly good description of the accuracy in almost all cases.
Today the BBC and EBU (European Broadcasting Union) published a detailed study which shows that around 45% of AI news queries to ChatGPT, MS Copilot, Gemini, and Perplexity produce errors.
|Model|Hallucination Rate|Factual Consistency Rate|Answer Rate|Average Summary Length (Words)| … |antgroup/finix_s1_32b|1.8 %|98.2 %|99.5 %|172.4| … |microsoft/Phi-4-mini-instruct|23.5 %|76.5 %|92.5 %|420.2|
The tech giant used its new FACTS Benchmark Suite to find that even the best-performing AI models fail to achieve a factual accuracy rate of over 70%. … In simple terms, today's chatbots give an incorrect answer about one-third of the time, even if they sound completely confident.
On Vectara's Hughes Hallucination Evaluation Model (HHEM) summarisation benchmark, frontier models in 2026 hallucinate on approximately 1.0-2.5 percent of summaries, down from 3-8 percent in 2023. … On RAG-faithfulness benchmarks (RAGTruth), frontier-model hallucination rates are 4-9 percent, materially higher than summarisation, because RAG must integrate external context with internal knowledge. … On closed-book factuality benchmarks (TruthfulQA, FACTS Grounding), frontier models score 80-90 percent accuracy in 2026, up from 50-65 percent in 2023; long-tail facts remain the dominant failure mode.
According to research published in the European Journal of Pathology (the official journal of the European Society of Pathology), artificial intelligence models provided erroneous answers in nearly 70% of diagnostic cases and fabricated or inaccurate references in over 30%. … Useful answers were provided in 62.2% of cases, but only 32.1% were entirely error-free.
If a benchmark reports 90% accuracy, expect 70-80% in production when accounting for consistency and faults.
There is no single hallucination rate, and anyone quoting one number without naming the benchmark is telling you less than they appear to. … Reading 94% as "AI is wrong 94% of the time" The AA-Omniscience figures count guesses on questions the model could not answer, questions written to be hard. On everyday, well-documented topics, frontier models are right far more often than not.
Published hallucination rates range from a couple of per cent to well over half, and the studies are not contradicting each other. … “Models hallucinate 20 per cent of the time” is not a statement about anything. “On a short fact-seeking QA benchmark published in 2024, this named model snapshot gave an incorrect answer to N per cent of questions it attempted, declining on M per cent” is a statement somebody can check.
Retrieval failures — not reasoning failures — drove more than 70% of errors.
study-shows-ai-systems-fail-accuracy-tests-about-70-percent-of-the-time
Continue your research
Verify a related claim next.
Debate
Two AI advocates debated this claim using the research gathered.
Argument for
Research published in the European Journal of Pathology demonstrates that AI models provide erroneous answers in nearly 70% of diagnostic cases, highlighting severe reliability issues in complex evaluations (Source 20). Although performance varies across different benchmarks, this high failure rate is strongly corroborated by broader studies showing AI systems failing accuracy tests about 70% of the time (Source 29) and exhibiting hallucination rates that can reach up to 94% (Source 10).
The Proponent generalizes from Source 20's low-standing secondary report of a single diagnostic setting to all AI systems and evaluated cases, an ecological fallacy that the source itself cannot support. Source 29 supplies only a headline rather than methods or results, while Source 10 reports a 22%–94% range; citing its maximum as corroboration of a universal 70% rate is cherry-picking, contradicted by the low-error result in Source 16.
Argument against
The claim invents a universal 70% error rate despite the brief showing that performance is benchmark-, task-, and model-specific: Stanford HAI (Source 10) reports hallucination rates spanning 22% to 94%, while PLOS ONE (Source 1) found 49% correct in one clinical-diagnosis evaluation—not a 70% failure result. Direct counterexamples make the categorical figure untenable: Vectara's leaderboard (Source 16) reports a 1.8% hallucination rate for one model, and the FACTS report summarized in Source 18 places the best systems at roughly one-third incorrect, so no evidence establishes that AI systems generally produce incorrect answers in 70% of evaluated cases.
The Opponent cherry-picks narrow summarization metrics from github.com (Source 16) to obscure the reality that in rigorous, high-stakes evaluations, the 70% failure rate is explicitly documented. By ignoring both wsu.edu (Source 29), which directly confirms AI systems fail accuracy tests approximately 70% of the time, and ai2med.eu (Source 20), which corroborates this exact 70% error rate in complex diagnostic cases, the Opponent fails to refute the specific evaluations where this claim is demonstrably true.
Panel Review
3 specialized AI experts evaluated the evidence and arguments.
Reviewer 1 — The Logic Examiner
Source 20 reports a nearly 70% error rate only in a particular diagnostic-case evaluation, whereas Sources 1, 10, 16, and 18 show materially different results across models, tasks, and benchmarks, so that result cannot validly establish a general 70% rate for AI systems. The claim is Mostly False because the evidence demonstrates benchmark-dependent error rates rather than the asserted uniform rate, although a narrowly defined evaluation may indeed yield a result near 70%.
Reviewer 2 — The Source Auditor
High-quality sources, including Stanford HAI (Source 10) and PLOS ONE (Source 1), demonstrate that AI error rates vary widely depending on the model and specific task, ranging from under 2% to over 90%, rather than a universal 70% failure rate. The claim falsely generalizes a specific error rate from isolated studies to all AI systems and evaluated cases.
Reviewer 3 — The Precision Analyst
The claim asserts a single universal figure—that AI systems produce incorrect answers in 70% of evaluated cases—yet the evidence shows highly task-, model-, and benchmark-dependent rates ranging from ~1.8% (Source 16) and ~31% incorrect for top models (Sources 7, 18) through ~45–51% (Sources 1, 5, 15) to nearly 70% only in one secondary medical-diagnosis report (Source 20) and a thin headline (Source 29), with Source 10 spanning 22–94%. As worded, the unqualified 70% therefore overstates and universalizes a narrow or cherry-picked result, rendering the claim false at its stated strength.
Panel summary
Source analysis finds no representative evidence establishing a 70% error rate across artificial intelligence systems. Results instead range from very low error rates to above 90%, depending on the model, task, benchmark, and scoring method. Logically, extending a near-70% result from a narrow medical evaluation to AI systems generally is a hasty generalization, reinforced by cherry-picking. Quantitatively, the unqualified figure conflicts with numerous reported rates, including approximately 31%, 45%, and 51%. Although a specific benchmark may produce a result near 70%, that narrow kernel does not support the broad statement as written.