Beyond Benchmarks: Disagreement Among Frontier LLMs on Real-World Fact-Checks

1Introduction

Large language models are increasingly consulted as arbiters of factual questions, by individual users and by automated fact-checking systems alike. Since the frontier models score similarly on public benchmarks, it is easy to assume that it makes little difference which one you ask. Benchmark parity, however, says nothing about whether the models reach the same judgement on the same claim. If the perceived validity of a claim depends on which model a person or platform happens to use, model disagreement becomes a source of inaccuracy independent of any individual model's benchmark standing. To measure this disagreement, we presented five frontier models with 1,000 recent user-submitted claims under a controlled, forced-choice prompt. Each model was required to classify every claim on a verdict scale: True, Mostly True, Mixed, Mostly False, or False and give a score (1–10) on how confident it is in its assessment. All five models returned a usable verdict on 997 of those claims, and every figure reported below is computed on that cohort.

We find substantial disagreement: on 63% of claims (632 / 997; 95% CI 60–66%) at least one model dissents from the panel majority or no majority forms at all, and on 23% (232 / 997; 95% CI 21–26%) at least two models' answers differ by two or more verdict options. This study extends an earlier version of our work (Jordanov 2026) under a revised methodology (§8); despite these methodological improvements, significant disagreement remains. We also find that a model's self-reported confidence does little to indicate this disagreement. The models report high confidence on the great majority of their answers yet disagree no less often, and the confidence scores themselves show weaker inter-rater reliability than the verdicts they accompany. Only in aggregate does confidence carry a signal, since the panel agrees on the verdict far more often on claims where every model is confident than on those where at least one is not. In practice, the answer a user obtains can therefore hinge on the model they happen to ask, a dependency that is important to further study.

1.1 Previous works

There is an assortment of literature regarding accuracy rates of different models and methods to improve them, but there are few studies particularly relevant to our work. Sahitaj et al. (2025) establish LLM baselines for automated fact-checking across binary, three-class, and five-class labelling schemes, finding that larger LLMs outperform smaller ones in both accuracy and reasoning quality on real-world claims from PolitiFact. Turning to inter-model disagreement specifically, Yang & Wang (2026) compare model predictions on two reasoning benchmarks, MMLU-Pro and GPQA, and find that even leading models of comparable accuracy give different answers on 16–38% of items. For real-world claim verification with human annotation, AVeriTeC remains the canonical corpus; however, having been publicly available since 2023, it has likely entered the training data of current models, allowing them to recall published answers rather than assess claims independently. Regarding AI models' confidence scores, Kadavath et al. (2022) find that larger language models are reasonably well-calibrated on multiple-choice and true/false questions, with self-reported confidence tracking actual correctness, though this calibration degrades on novel task formats and only partially generalises across tasks.

Relation to prior version. This study extends an earlier version of our work (Jordanov 2026) by improving the methodology used. The scale has been expanded to five symmetric verdict categories (True, Mostly True, Mixed, Mostly False, False), each accompanied by a brief definition in the prompt so as to avoid differing interpretations. Models are also asked to provide reasoning before reaching a verdict and then give a confidence score. There are now a total of five, instead of the initial four models, each in its most recent publicly available version at the time of writing. They are configured with web retrieval, while excluding lenz.io as a source where the provider's API permitted it. We have run the prompts with an entirely fresh corpus of claims (§3.1).

1.2 Contributions

Whereas much prior work has centred on measuring and improving accuracy, our study takes inter-model disagreement itself as the object of analysis. Rather than drawing on benchmarks or other public datasets, whose items and reference answers are well documented online and plausibly present in training data and during retrieval, we evaluate recent, real-world claims whose verdicts the models should not have encountered before prompting and are purposely excluded (as much as possible) in their retrieval. In addition, we examine the confidence level each model reports on every claim and use that to present how it differs based on the specific model and type of claim, as well as to show how the level of disagreement relates to the confidence reported. We evaluate five of the most widely used frontier systems: Claude Fable 5, GPT-5.6-Sol, Gemini 3.1 Pro + Search, Sonar Deep Research, and Grok 4.5, in their most recent publicly available versions at the time of writing, so that observed disagreement reflects the verdicts users currently receive.

2Research Focus

This study asks one question: to what extent do the strongest frontier LLMs agree with one another when judging real-world fact-check claims, and where does their disagreement concentrate?

This study focuses on the extent to which frontier LLMs agree with one another when judging real-world claims, and on where that disagreement concentrates. We examine inter-model disagreement rather than accuracy. Because the corpus carries no human-labelled correct verdict for any claim, accuracy cannot be assessed, and we make no claim as to which verdicts are right. Instead, we examine how consistently five leading models reach the same judgement on the same claim, and what that consistency might depend on.

We approach this through two aspects, the models' verdict agreement and their self-reported confidence. Verdict agreement is measured with the following.

• Panel disagreement rate. The share of claims on which at least one model dissents from the majority, or no majority forms. A majority is a verdict held by at least three of the five models.
• Maximum verdict distance. How far apart the two most-divergent verdicts fall on the ordinal five-point scale.
• Pairwise agreement. How often each pair of models assigns the same verdict across the corpus.
• Per-model majority alignment. How often each model matches the strict majority of the other four, where such a majority exists.
• Krippendorff's ordinal α. Panel-level inter-rater reliability.

Each of these is also broken down by verdict category and by claim domain, to locate where disagreement concentrates. For confidence we examine the following.

• Confidence distribution. How the reported scores are spread across the panel and its individual models.
• Confidence against disagreement. Whether reported confidence corresponds to inter-model disagreement at the level of the individual claim.
• Inter-model confidence agreement. Whether the models agree with one another on how confident to be, quantified with Krippendorff's α on the confidence scale.

Note: A majority formed between models is not considered truth. It is used only as a reference point from which to measure disagreement, not as a stand-in for correctness. A high confidence score is likewise not treated as evidence that a verdict is correct.

3Methodology

3.1 Corpus

The corpus used consists of the 1,000 most recent real-world claims submitted by users to the fact-checking platform Lenz that satisfy the eligibility criteria in Exclusions, with no claim submission predating May 1, 2026. We constructed this corpus rather than adopting an existing dataset in order to minimise the risk that the claims, or verdicts concerning them, appear in the models' training data. User-submitted claims also better reflect the types of questions AI models and fact-checking systems are being used to answer.

Claim standardisation. Models were presented with a distilled formulation of each claim rather than the user's raw text. This standardisation removes emotionally charged language and ambiguous phrasing, with the goal of ensuring the claims are reasonably verifiable. This process is performed within the Lenz platform itself, and all five models receive the identical standardised text, so no part of the measured disagreement can come from differences in how the claim was worded.

Exclusions

The corpus excludes claims marked private; claims from staff, internal, or API submissions; claims with editorial status pending or hidden (including PII-flagged claims); near-duplicate claims (collapsed to a single canonical row by cosine distance on OpenAI text-embedding-3-small); claims where at least one model failed to produce a parseable verdict after one retry (3 residual cell-level parse errors).

Note: Out of the 1,000 total claims the models were presented with, Claude Fable 5 didn't return a valid output, even after the retry, to 145 claims. After rerunning those claims with Opus 4.8 as a fallback, 3 claims remained rejected (visible in Appendix B). For that reason, the results given are all using a total of 997 claims.

3.2 Model selection

We selected one frontier AI from five of the most popular AI companies: Claude Fable 5 (Anthropic), GPT-5.6-Sol (OpenAI), Gemini 3.1 Pro + Search (Google), Sonar Deep Research (Perplexity), and Grok 4.5 (xAI). Each of the five was set up to search the web, so the verdicts they return are representative of those a user querying them would receive.

3.3 Excluding Lenz as a Source in Retrieval

All five models had retrieval enabled and could look up sources at inference time. An explicit lenz.io exclusion was passed to four of the five providers; Gemini's search-grounding tool does not accept a domain-exclusion parameter under a standard Developer API key (only under Vertex AI's Enterprise Agent Platform mode, which this study does not use), so Gemini 3.1 Pro + Search ran without this guard and we looked over the sources it cited.

Despite the restrictions, in 39 of the total 5,000 runs lenz.io appeared in their sources and in 10 of those the page cited was Lenz's own verdict on the very claim being evaluated. For example, one of the claims was directed towards inter-model disagreement and the examining model had sourced the previous version of our paper as one out of the many used resources.

While it is impossible to completely eliminate contamination from our study, we believe that we have reduced it as much as possible, so that our statistics are still relevant and representative.

3.4 Prompt

Every model receives the same system prompt and user prompt template (usr_v4), reproduced in full below. The {date} anchor is the date on which the claim was submitted by a user to Lenz for verification; it pins each model to the claim's submission-time epistemic frame.

System prompt:

You are a fact-checker. Evaluate the truthfulness of the claim as of the stated date. Respond with JSON only.

User prompt template:

Evaluate this claim as of {date}:
"{claim}"

Choose exactly one verdict from the scale below:
  "True" — the claim is accurate
  "Mostly True" — the claim is largely accurate with minor caveats or omissions
  "Mixed" — the claim has both accurate and inaccurate elements
  "Mostly False" — the claim is largely inaccurate with some basis in fact
  "False" — the claim is inaccurate

Respond with a JSON object containing exactly these fields:
  "reasoning": 2-4 sentences of claim analysis and verdict justification
  "verdict": one of the five labels above
  "confidence_level": your level of certainty in the verdict on a 1 to 10 integer scale (1 = completely uncertain, 10 = fully certain)

Note: No “Abstain” category is provided for. The models vary in their willingness to abstain. In the initial testing phase, some of the frontier models refused to judge certain claims whereas other models were always willing to make a judgement call, so an abstention is a model-specific action: one model's refusal to answer cannot be compared with another model's confidently wrong verdict, despite both of them having epistemic weight. Forcing the same five options on every model captures the disagreement between the judgements, not between the decisions. Abstaining is also not a result which a user can use. If a user is trying to check a claim and finds out that the model refuses to make a judgement about it, they still have not got any result, and our task is to identify the disagreement between the results that users actually receive. Nor is a forced option a forced certainty — each judgement carries a confidence level from 1 to 10, which is where a model registers doubt. Unparseable outputs are not reclassified into a verdict category; claims with any parse error are excluded from the complete-claim cohort (Exclusions).

3.5 LLM call configuration

All five models run at a medium reasoning tier. Extended thinking is enabled at each provider's medium depth setting (Anthropic effort=medium; OpenAI and xAI reasoning_effort=medium; Gemini thinking_budget=16384; Sonar Deep Research's multi-step research is always on, at Perplexity's default reasoning_effort of medium). Retrieval/web search is enabled on every model (OpenAI at search_context_size=medium). Responses were constrained to a fixed JSON schema (reasoning, verdict, confidence_level) enforced natively at the API level, forcing each model to emit strict, machine-readable JSON. Retrieval was constrained to exclude lenz.io as a source wherever the provider's API permitted it, with the single exception and its consequences detailed in §3.3. Deterministic decoding was requested where supported (temperature=0.0), and output caps were raised on reasoning models so extended thinking cannot truncate the verdict. The per-model settings summarised here are recorded in full in model_settings.yaml in the harness repository (Appendix C), which is the authoritative record of what each provider was sent.

Claude Fable 5 fallback. Any failed Fable 5 call — safety-classifier refusal, timeout, rate limit — retries once against Claude Opus 4.8 rather than being recorded as an error outright. A retry still reports as claude-fable-5 in the results (the five-model-per-claim panel shape stays intact), so this is a reliability measure for the harness, not a sixth model in the comparison. It resolves most transient failures; see Appendix B for the 3 claims where both the primary call and the Opus 4.8 fallback failed.

3.6 Grading

No LLM grader. All measurements derive from direct parsed-label equality between the 5 frontier verdicts on the same claim. No reference label or ground truth is used.

3.7 Statistical analysis

Sampling frame. The corpus is the 997 most recent eligible claims from a single platform: not a probability sample from any wider population, and not a complete enumeration. The reported Wilson 95% CIs [1] are nominal binomial intervals under a model where each claim is an independent draw from a hypothetical stream of similar eligible submissions. They are not coverage statements about “all real-world fact-checks.”

Note: Lenz claims are not independent: users cluster submissions around news events and often submit multiple related claims in a session. True sampling variability under a cluster model would likely exceed what Wilson reports; we surface CIs as a minimum precision floor, not a guaranteed coverage interval.

Inter-rater reliability. The verdict scale is ordinal, so we score agreement with Krippendorff's α at the ordinal level of measurement [2] rather than Fleiss' κ, which would treat the categories as nominal and understate agreement. The confidence level scale is also ordinal and can be scored similarly.

References.
[1] Wilson, E.B. (1927). "Probable inference, the law of succession, and statistical inference." Journal of the American Statistical Association 22, 209–212.
[2] Krippendorff, K. (2004). "Reliability in Content Analysis: Some Common Misconceptions and Recommendations." Human Communication Research 30(3), 411–433.

4Results by Verdict

All rates below use the 997-claim corpus as denominator unless a table states otherwise; Wilson 95% CIs appear beneath each rate. Interpretation follows in §6 Analysis.

4.1 Panel disagreement rate

For each claim, we begin by determining whether a strict majority has formed. When such a majority exists, we examine how many of the remaining models have dissented. In the remaining cases, where no single verdict is agreed upon by (at least) three of the five models, the claim is counted towards the “models split, no majority”, as indicated in the table below.

Panel agreement: Krippendorff’s α (ordinal) = 0.77 (n=997 claims, 5 raters). Ordinal α is the standard Krippendorff variant for an ordered categorical scale (True / Mostly True / Mixed / Mostly False / False). See §3.7 Statistical analysis for choice of metric and §6 Analysis for interpretation.

4.2 Maximum Verdict Distance — Calibration vs Substantive Disagreement

Disagreement can be of varying degrees. The gap between “True” and “Mostly True” indicates a difference in calibration, while that between “True” and “False” indicates an actual dispute over the claim. In order to differentiate between the two, we determine for each claim its maximum verdict distance: the largest gap between any two of the five verdicts, ranking the categories as follows: True (0) → Mostly True (1) → Mixed (2) → Mostly False (3) → False (4). The table below reports the resulting distribution.

Note: The distance between verdicts assumes an equal-interval ordinal scale for the five categories, which is an approximation. A distance of size 2 may come about due to the vagueness of the rubric, differing time frames, or different understandings of what “Mixed” means. This distance is provided just as a rough marker of substance versus nuance, not as a measure of error size.

4.3 Pairwise agreement

The table below illustrates the frequency of identical verdict labels chosen by each pair of frontier models within the corpus. Pairwise agreement reaches up to 76% and drops to 49%.

How often each pair of frontier models picked the same verdict label, across all claims in the corpus.

4.4 Per-model verdict distribution

The table below shows the share of claims each model assigned to each verdict category.

4.5 Per-model majority alignment

For each model, we examine how often its verdict matches the strict majority (≥3/4) of the remaining four, on claims on which there is such a majority. The per-model majority match rates range from 63% to 86% (see the table below). This measure is an indication of match to the panel for this corpus, not correctness: no model is treated as ground truth, and the eligible n varies by row.

4.6 Breakdowns by domain and majority verdict

The three tables below present additional statistics. The first presents the disagreement rates based on the eight domains the corpus' claims fall into. The claims were divided into domains directly within the Lenz platform. The second looks only at claims on which a strict majority formed — at least three of the five models on the same verdict — and reports, for each verdict label, how often that majority was unanimous (5/5) rather than majority-only (3–4/5). The third shows the verdict distribution within the fully unanimous claims.

Per-verdict panel agreement. This looks only at claims on which a strict majority formed — at least three of the five models on the same verdict — and reports, for each verdict bucket, how often that majority was unanimous (5/5) rather than majority-only (3–4/5).

Denominator: claims with a strict ≥3/5 frontier majority on this verdict.

Viewed from the other direction — of the 365 claims where all 5 frontier models converged on the same verdict, the distribution across verdicts:

5Results by Confidence Level

In providing every verdict, each model also reports its level of confidence in that verdict, in the form of a number between 1 and 10 (1 being the lowest confidence, 10 the highest). This is the models' confidence in their own answer, not the accuracy of that answer. The following section presents the collected data on these numbers, their distribution, and their relation to disagreement (interpreted in §6 Analysis).

5.1 Confidence Level Distribution

Below are the tables which show the distribution of the confidence levels for each individual model. The figures are heavily skewed towards higher confidence levels: 76% of all answers have a confidence level of 9 or 10, with mean confidence levels per model varying between 8.2 and 9.5 (the Mean row of the first table). Low confidence is rare.

All models pools every model's answer on every complete claim (997 claims × 5 models = 4,985 answers) into one distribution — it is not comparable cell-by-cell against the per-model rows above, which each sum to 100% over that model's own claims alone.

Per-model confidence distribution (counts)

Same table as §5.1, as raw claim counts rather than percentages.

Mean confidence per model

Average self-reported confidence (1-10 scale) per model. n is the number of that model's claims with a parseable confidence value — a model's own parse errors reduce its n below 997, independent of whether other models errored on the same claim (a different denominator than §5.1's complete-claims-only 997).

5.2 Confidence in Relation to Disagreement

The following tables present the confidence scores against the levels of disagreement present between models. Grouping claims by their lowest model confidence given when accessing them, substantive verdict disagreement falls as confidence rises: claims in the low-to-mid range (1–6) show 54% substantive disagreement, versus 3% for the very-high range (9–10), and claims where all five models report maximum confidence show essentially none.

Agreement by confidence band

Same lowest-confidence-per-claim methodology as §5.2, grouped into three named bands instead of cumulative floors. "High/very high" and "Very high" overlap by design (9-10 is a subset of 7-10) — these are not a disjoint partition.

5.3 Where Confidence Concentrates

The first table below reports the distribution of every model answer grouped by the claim's domain. The second groups model answers by verdict and similarly reports distribution. In both, the 1–10 scale is collapsed into five bands: Very Low (1–2), Low (3–4), Mid (5–6), High (7–8), and Very High (9–10); and each cell reports the answer count above its share of the row, so every row sums to 100% (small differences due to rounding). The domain breakdown is largely uniform: the Very High band alone accounts for between 70% (Health) and 85% (History) of answers in every domain, and no domain attracts materially more caution than the others. The verdict breakdown separates far more sharply. The definitive poles concentrate their confidence scores in the Very High range: 92 of True and 84 of False answers fall in it. In contrast, only 49 of Mostly True, 46 of Mixed, and 55 of Mostly False answers reach Very High, with the remainder distributed across the High (7–8) band rather than the lower ones. These results are in agreement with reasonable expectations, as they show that presumably models are more assured in their answer when giving more polarising, less-nuanced verdicts. The third table presents the same information in the opposite direction: the verdict composition of each confidence band, rather than the confidence distribution of each verdict. Each row represents one band and totals 100%, with every cell giving the number of answers above its share of that band. The two lowest bands together contain fewer than fifty answers, so their shares rest on a small base.

Confidence by verdict

Same methodology and bands as §5.3, cross-tabbed by each answer's own verdict instead of its claim's domain (same 4,985 answers). Each cell is a count with its share of that verdict's answers underneath, so every row sums to 100%.

5.4 Models' Agreement on Confidence

The table below measures whether the models agree on confidence rather than on the verdict, comparing the confidence scores a claim received regardless of the verdict labels. They differ by at least one point on 87% of claims and by three or more points on 21%, and the ordinal Krippendorff's α for confidence is 0.44, far below the verdict-level α of 0.77.

Compare the α here against the verdict-level α of 0.77 in §4.1: the panel converges on the verdict far more than it converges on how confident to be.

6Analysis

6.1 Verdict

A lower bound on model error. For each claim, exactly one of the five verdict categories is correct, or may at least be regarded as most nearly correct. Under the most charitable assumption available, that the panel's modal verdict is the correct one, still at least one model assigns an “incorrect” verdict on 63% of claims, at least two do so on 39%, and at least three on the 11% of claims (for which no verdict category attains a majority). These figures constitute a lower bound as relaxing the modal-correctness assumption can only decrease overall accuracy.

Note: Regardless of the above reasoning, we do have to acknowledge that without specific correct classifications of the claims, it is impossible to give concrete statistics. There is also the possibility that there are more nuanced claims that may reasonably fit into more than one of the five categories as well as a degree of boundary-ambiguity to the scale. The reasoning presented above is meant to show that regardless of these factors, even only on the basis of disagreement, it is clear that there must be a significant amount of error.

The middle of the rubric is where the panel fractures. When the panel lands on a middle category, it almost never converges to unanimity, while polar (True / False) majorities do so far more often (§4.6). Of the 365 unanimous claims the overwhelming majority are unanimous-True or unanimous-False.

This pattern admits two explanations. The first is that claims with a determinate, well-established verdict are easier to adjudicate, so models converge on them. The second is that the models disagree on where one verdict label ends and the next begins, as the 5-point scale is ultimately a simplification of a spectrum. This should be mitigated, at least to some degree, by the verdict category definitions present in the prompt, but it cannot be eliminated fully as a factor. Regardless, what can be said without qualification is that the panel agrees far more on definitive claims than on nuanced ones.

Per-model differences. The per-model distribution (§4.4) shows that all of the models tend to favour True and Mostly True verdicts. While one might draw conclusions about the models, this more likely reflects the corpus. It is likely that the claims tend to lean towards being true and that was reflected in the models' answers. This can ultimately be determined with human labelling of the corpus, a natural step for future work.

Domain relevance. The per-domain disagreement rates (§4.6) are most relevant for analysis in the “Health” domain, since it observed the highest level of substantive disagreement (30%). This can be concerning, since health-related claims would be considered some of the most important to be able to credibly verify. That said, health claims are also among the more nuanced and hence difficult to adjudicate and label. Since we have a relatively small corpus, once it is divided into eight categories, its statistical precision is reduced. This is reflected in the wide Wilson 95% CIs. Due to the small sample size, we cannot draw firm conclusions, but it is still relevant to note the persistence of substantial inter-model disagreement within each of the domains.

Panel reliability. Krippendorff’s α (ordinal) = 0.77 (n=997 claims, 5 raters) places the panel below the conventional reliability threshold. Krippendorff's own guidance treats α ≥ 0.80 as the floor for relying on coded data and 0.667 ≤ α < 0.80 as warranting only tentative conclusions (Krippendorff 2004); the panel falls in this lower band. The verdicts are therefore structured rather than random — the models are responding to a shared signal in each claim — but not concordant enough to be treated as exchangeable measurements of a single underlying verdict.

This is the quantitative form of the claim made in §1: comparable aggregate benchmark accuracy does not make frontier models interchangeable as assessors, since two models of similar standing still return ordinally divergent verdicts often enough to keep panel-level reliability under the accepted bar.

6.2 Confidence

High overall confidence coexists with substantial disagreement. The five models reported high confidence throughout the corpus. 76% of all answers were assigned a confidence of 9 or 10, and per-model mean confidence ranged from 8.2 to 9.5 (§5.1). The same concentration is evident in the raw counts, where the great majority of the 4,985 answers fall at the top of the scale.

This high overall confidence is difficult to reconcile with the disagreement established in §4, where the panel diverges on 63% of claims and disagrees substantively on 23%. The models are thus, in aggregate, both highly confident and frequently in disagreement, which indicates that, contrary to what one might intuitively assume, a high reported confidence does not by itself imply that a verdict is uncontested.

Disagreement is concentrated in the low-confidence claims. This disagreement is not distributed evenly across the confidence scale but is concentrated among the claims the models were least certain about. Grouping each claim by its confidence floor, the lowest confidence any of the five models assigned to it, isolates the pattern (§5.2). On the claims where every model was very confident, corresponding to a floor of 9 or 10, substantive disagreement falls to 3%, and on those where all five reported the maximum it practically disappears.

These high-confidence claims constitute close to half of the corpus yet contribute almost none of its substantive disagreement, which entails that the disagreement measured across the full corpus arises overwhelmingly from the lower-confidence claims. Claims whose floor falls in the lower half of the scale (1–6) disagree substantively at 54%, more than double the corpus-wide rate of 23%, and any disagreement among them is elevated as well. Even in the broader band “High/very high (7–10)”, substantive disagreement remains at 18%, which, although below the overall 23%, is still considerably higher than the 3% observed once only the 9–10 scores are isolated. Confidence measured at the level of the claim can therefore be used to separate the contested claims from the settled ones, even though a model's confidence in isolation, as established above, does not.

Confidence in relation to domain and verdict. Confidence by domain (§5.3) shows no major departure from the pooled distribution, as reported confidence remains concentrated in the Very High band across every domain. Between 70% (Health) and 85% (History) of each domain's answers fall in the Very High band, with a further 13% to 26% in High. The concentration of high confidence therefore persists when the answers are disaggregated by domain. The differences between domains are minor, and none is large enough to support a conclusion, particularly given the small number of claims in each domain — an issue discussed above under Domain relevance.

Grouping the answers by their own verdict instead, the distribution separates sharply. The two polar verdicts are overwhelmingly concentrated in the Very High band, with 92% of True and 84% of False answers rated 9 or 10. The three intermediate verdicts are considerably less certain, as only 49% of Mostly True, 46% of Mixed, and 55% of Mostly False answers reach the Very High band, and most of the remainder sits in the High (7–8) band rather than lower.

The same coupling holds in reverse. The Very High band, which holds roughly three quarters of all answers, is composed predominantly of the two poles (59% True and 17% False), whereas the mid band is made up almost entirely of the intermediate verdicts (89% combined). The two lowest bands rest on fewer than fifty answers and are not interpreted here as the sampling is far too small.

Taken together, these tables indicate that the models express their strongest confidence on the polar verdicts and reserve their uncertainty for the claims that call for an intermediate judgement. This parallels the structure found for verdict agreement in §4, where the panel converges at the poles and fractures across the middle. Confidence and agreement therefore move together, both tracking where on the verdict scale the panel lands rather than the claim's subject domain.

The models do not use the confidence scale in the same way. The preceding tables concern how confidence relates to the verdict a model assigns; the confidence-agreement table (§5.4) instead asks whether the models agree with one another on confidence itself, and they do so only weakly. The five models differ by at least one point on 87% of claims and by three or more points on 21%, and the ordinal Krippendorff's α for confidence is 0.44, far below the 0.77 obtained for the verdicts. The panel therefore agrees less on how confident to be than on what the verdict is.

Much of this disagreement sits at the top of the scale, between adjacent values rather than across wide gaps; it is nonetheless persistent, and no coarsening of the scale removes it. A reported confidence is not comparable across models, since the same numerical value need not correspond to the same degree of certainty from one model to the next, and the panel offers no shared scale of confidence on which a downstream user could rely. This does not contradict the earlier finding that a claim's confidence floor tracks verdict disagreement, as that relationship holds in aggregate across claims, whereas the present result concerns whether the models assign similar confidence to the same claim, which they largely do not.

This pattern is consistent with prior work on calibration. Kadavath et al. (2022) find that self-reported confidence is reasonably calibrated on familiar formats but degrades on novel task formats and generalises only partially across tasks. The claims examined here are recent and previously unseen, a regime in which such degradation is expected, and the confidence α of 0.44, well below the verdict α, is the corresponding signature at the level of the panel.

7Conclusion

This study set out to measure how consistently frontier LLMs judge real-world claims, and it found that consistency to be limited. Across 997 recent user-submitted claims, the five models returned substantively divergent verdicts on 23% of cases and dissented in some form on 63%, and the panel's ordinal Krippendorff's α of 0.77 places its agreement below the threshold at which a set of raters can be treated as interchangeable. The disagreement is therefore neither negligible nor random. It is a structured and persistent feature of how these systems adjudicate claims, and it survived the methodological revisions made over the earlier version of this work.

What the two dimensions of the study share is more informative than either alone. Verdict agreement and reported confidence move together. The panel converges and reports near-maximal confidence on the polar claims, and it fractures and grows less certain across the intermediate ones, whereas the subject domain of a claim has far less bearing on either. Confidence, however, offers no reliable basis for resolving this disagreement. Although a claim's lowest reported confidence anticipates whether the panel will divide (§5.2), the models do not use the confidence scale in the same way (§5.4), so the same numeric value carries different weight from one system to the next.

The practical consequence is that reliance on any single frontier model inherits this disagreement without exposing it. A different system would return a materially different verdict on a substantial share of claims, and that share is largest precisely on the nuanced, partially-true claims for which an accurate judgement matters most and on which no external signal, confidence included, flags the disagreement to the user. As these systems are increasingly consulted as arbiters of factual accuracy, the identity of the model becomes a determinant of the verdict in its own right.

These conclusions concern consistency rather than correctness, which the absence of human-labelled verdicts leaves out of reach. Yet the two are linked. Because only one verdict can be most correct on any claim, the disagreement measured here is a lower bound on the error that at least one model must commit, and any future assessment of accuracy would inherit the same divergence as its floor. Establishing those labelled verdicts is the natural next step (§11 Future work), and it would convert the present account of where the models disagree into an account of where, and how often, they are wrong.

8Reproducibility

Full per-claim data: download CSV. One row per claim — claim ID and URL, atomic claim text, the 5 frontier verdicts, maximum verdict distance (max_pairwise_bucket_distance), domain, and creation date. Strictly rectangular, no preamble comments. The claim_url column links each row back to the original claim page on Lenz; some pages may be unavailable if the user who submitted the claim later deleted or privatized it.

PDF artifact: download PDF. Built from the LaTeX paper source (see below) for offline reading, citation, or preprint hosting; byte-identical across re-deploys.

LaTeX source: download the paper source bundle (main.tex, bibliography, generated data tables — the arXiv-submission fileset, frozen at snapshot time).

This snapshot is v1.1, data as of July 18, 2026, captured on July 22, 2026. The archival URL /research/llm-disagreement/v1.1 permanently serves the v1.1 snapshot — citation-stable even when the bare URL bumps to a future version. The previous release stays at /research/llm-disagreement/v1.0. The harvesting tool used to create the underlying data is available separately as a standalone repository, described in Appendix C.

Research snapshots are versioned. Frontier LLMs are non-deterministic and model generations turn over, so this measurement is designed to be re-run: when a future version ships — with more claims, refreshed model versions, or methodology changes — it appears as a new snapshot with an explicit changelog entry, and every prior release stays untouched at its own archival URL.

Grader: direct parsed-label equality across the 5 frontier verdicts. No LLM grader, no reference verdict.

Permanent record & citation: doi.org/10.5281/zenodo.21829261. The Zenodo deposit mirrors the PDF artifact under a permanent DOI for citation in academic and preprint contexts.

Changes from v1.0

Snapshot v1.0 (May 2026) employed a 4-point rating scale (True / Mostly True / Misleading / False) with a pre-existing set of models (GPT-5.4, Claude Opus 4.7, Gemini 3 Pro, Gemini 3 Pro + Search, Sonar Pro) based on a one-label prompt (usr_v2) with no structured output or extended thinking. v1.1 scales up to a 5-point rating scale (Mixed and Mostly False are added to replace the Misleading category), updates the set of models to the five listed above and employs all models at a mid-level reasoning tier (mid-level thinking, retrieval, native JSON). v1.0 is permanently stored at its versioned URL /research/llm-disagreement/v1.0; numbers cannot be compared between versions due to the difference in the scale.

9Ethics and data use

Only public-facing claim fields are used: the atomic claim text and the claim's creation date. No personal data. Private and staff claims are excluded (§3.1). Frontier models received only the claim text and the as-of date — no submitter identity or analytics signal.

If a claim is later privatized or deleted by its submitter, we can drop it from this snapshot and from any future downloads. The CSV is generated from the snapshot at download time, so removing a claim from the snapshot removes it from the public page in a single update.

10Limitations

The main limitation of this study is that it lacks human-labelled answers for the claims. The consequence of this is that we are only able to measure disagreement, not accuracy, which is ultimately a primary concern for AI models. Treating the five verdict categories as equally spaced along that ordinal scale is a simplification. Part of the disagreement measured is reflective of the task, more than the specific models, as even human annotators have been found to have a κ = 0.619 (AVeriTeC). In terms of the corpus and testing, the claims are in some way selected based on Lenz's users, so it is not a direct random sampling. It might also be a concern that the claims aren't in their raw format (§3.1), limiting how representative the models' replies are of what regular users would receive. Finally, another larger issue is that we ran each claim once per model; a rerun would most likely shift the numbers due to model inconsistency (Haldar & Hockenmaier, 2025).

11Future work

The most substantial direction for future work to take is establishing human-labelled “correct” labels for each of the claims. This would address much of the current study's limitations and broaden its scope. We are looking into getting the claims in the current corpus labelled and creating a study that also encompasses accuracy rates, as well as examining how and where inaccuracy and disagreement with human-labelling clusters. We would also ideally rerun claims with every model in future versions. This would allow us to better account for inconsistencies within each model and possibly analyse how and where they form.

Future snapshots. The measurement is designed to be re-run: future versioned snapshots will track the disagreement rate as model generations turn over, each release archived alongside this one (see §8 and the changelog).

Acknowledgements

We would like to thank Beloslava Malakova for her guidance in the process of writing this study. Special thanks to Simon Willison and Benjamin Han for the valuable feedback on the first revision of this research. We would also like to express our appreciation to the authors of the works included in our bibliography, which served as a valuable source of knowledge. We would also like to acknowledge that AI tools were used to develop the harvesting and aggregation infrastructure, as well as serving as an additional editor for this report.

Appendix A: Example Claims

The twenty claims with the widest spread between the verdict categories, ordered by maximum verdict distance.

Ordered by maximum verdict distance (descending), no-majority cases tie-broken first, then by stable hash of the claim ID. Deterministic — the page renders the same examples on every load until the next snapshot.

Appendix B: Excluded Claims (Model Errors)

These are the three claims on which neither Claude Fable 5 nor, where it was called, the Claude Opus 4.8 fallback returned a usable verdict; they are therefore excluded from the 997-claim cohort.

Appendix C: Complete Code

What is released. All the code required to reproduce the harvest is public. The harness that generated it, along with the 1,000-claim corpus and the raw results, is available at github.com/lenzhq/lenz-research. The harvest consists of 5,000 rows, one for each (claim, model) cell, holding the verdict of the model, the confidence score 1 to 10, the text of the reasoning, the sources found, the cost and latency of inference, in JSON and spreadsheet format. No verdict label, no submitter identity, and no analytics signal accompany it, as prescribed by §9.

Since every row represents a cell, the release includes no cross-model comparisons; the claim-level statistics reported in this paper come from the snapshot of §8, which is also the source of all statistics and tables shown here — none of which is computed on a live run. The setup and file structure, along with costs per model, are described in the README of the repository.