AI Consensus Is a Signal, Not a Scoreboard

Key Findings

Four results carry the study: how often unanimous answers were right, how often they were wrong, how fast accuracy fell when models split, and how the majority compared with the best single model.

• Measured: the more the models agreed, the more often the majority answer matched the known answer, from 94.0% (unanimous) down to 33.5% (weak agreement).
• Measured: on the 2,303 questions that had a majority, the majority scored 77.1% and Claude alone 77.5%. The majority did not clearly beat the strongest single model.
• Interpretation: the value of several models is less a higher score than a visible signal: how much they agree, where they split, and what to verify.

Agreement Tracks Accuracy, but Does Not Guarantee It

When all models gave the same answer, it was correct 94.0% of the time. Majority accuracy then fell to 70.5%, 56.2% and 33.5% as agreement weakened. This is an association measured on multiple-choice questions, not proof that agreement causes accuracy.

73 Times, Every Model Was Wrong Together

A false consensus is a question where every answering model gave the same wrong answer. It happened 73 times in 2,500 questions: 44 in MMLU-Pro, 15 in TruthfulQA, 9 in PubMedQA and 5 in GPQA Diamond. Unanimity raises confidence. It does not remove shared blind spots.

Model Results: Scope Matters More Than Rank

Single-model accuracy ranged from 63.6% to 74.7%, but the models did not all answer the same questions, and some scores include second-pass answers. Perplexity answered a separate 533-question sample, so it must not be read against the others on 2,500 questions. This is not a leaderboard.

Why second-pass answers matter. 654 answers came from a second pass for replies that had no usable letter. They were right less often than first-pass answers: Claude 46.9% on 475 recovered answers against 81.2% on 2,025 first-pass answers. The first pass failed mostly on hard questions, so Claude's overall score depends on this mechanism. We report both views.

Majority Vote vs. the Best Single Model

Not clearly. On the 2,303 questions where a majority existed, the majority answer was right 77.1% of the time and Claude alone 77.5%. Counting the 197 ties as failures, the majority drops to 71% on all 2,500 questions.

On this benchmark, majority voting performed essentially on par with the strongest individual model rather than clearly surpassing it. The case for comparing several models is therefore not a higher raw score. It is that one model does not tell you when it is wrong, and agreement among several does tell you how much to check.

Not Every Question Had Six Voters

The number of models answering varied: five on 1,965 questions, six on 533 and four on 2. A unanimity of five is easier to reach than a unanimity of six, so we also report a constant jury of five and a constant jury of six. The conclusion holds in both.

Why not six everywhere? Perplexity ran on a fixed sample of 533 questions instead of all 2,500, to limit cost. Two further questions had only four valid answers, so they are counted in the totals but not shown here.

Our headline results use the full protocol, including format-recovery passes. We also report a protocol-only robustness check; it leads to the same substantive conclusion (93.1% of unanimous answers correct without recovery, 94.0% with). Of the 1,221 unanimous questions, 71 include a second-pass answer, and so do 5 of the 73 false consensuses. Satcove reports agreement by jury size rather than treating every question as a six-model vote.

Source and Topic Results: Read Them Together

Agreement and accuracy varied sharply by source: majority accuracy was 55.2% on GPQA Diamond, 74.4% on MMLU-Pro, 84.2% on PubMedQA and 89.2% on TruthfulQA. Topics are shown only inside their source, because the same topic name mixes sets of different difficulty.

What This Benchmark Does Not Prove

It does not show that agreement proves truth, that Satcove's verdicts are more accurate than a single model's, or why models fail together. It measures how often a majority matched a known answer on 2,500 multiple-choice questions, and how that changed with agreement.

• Agreement is not proof.

  73 questions were answered identically by every model and all 73 were wrong.

• Multiple choice, one letter.

  The test measures option selection. It does not measure free-text reasoning or the way Satcove synthesizes an answer.

• Truncated, not random.

  The 2,500 questions are the first 2,500 of a 2,997-question file. MMLU-Pro supplies 1,597 of them (63.9%), so it weighs heavily on every overall figure.

• Variable jury.

  Five models answered 1,965 questions, six answered 533 and four answered 2. Perplexity answered a fixed 533-question sample.

• Versions and recovery passes shape the scores.

  654 answers came from a second pass with a different protocol per model. Four of the six model identifiers are undated aliases.

• Topic results mix sources.

  Physics, chemistry and biology combine GPQA Diamond and MMLU-Pro, and several cells hold fewer than 30 questions.

• No reading of position bias.

  Questions have between 2 and 10 options and each source distributes correct answers differently, so letter frequencies cannot be read as a model bias.

• One answer per model per question.

  The test cannot measure how stable an answer is across repeated runs.

• No independence or contamination test.

  We do not know why models agree. The public question sets may also appear in training data. Neither point was tested.

From Disagreement to Decision

A single AI answer does not tell you when to doubt it. Satcove sends a question to several models, each answering separately, measures how much they align, names where they differ and suggests what to verify before you act. The benchmark supports reading agreement as a signal; it does not test Satcove's free-text verdicts.

Related: does AI consensus prove truth, how often AI models agree, our 75-question disagreement report, AI answer reliability, Satcove Open, false consensus, agreement score and when not to use multi-AI consensus.

Methodology

Each model received every question once, with the same instruction to reply with a single option letter, in a fresh conversation. We compared the letters with the known answer and measured agreement as the share of the jury giving the majority letter. Full definitions and scripts are published.

Download the Data and Scripts

Every figure on this page can be recomputed from the answers file. The files below include the claims list, the second-pass cells, the analysis script and checksums. No question text, prompt, key or user data is included.

Frequently Asked Questions

Short answers on what agreement between AI models can and cannot tell you, based on this benchmark.