# AI Consensus Is a Signal, Not a Scoreboard

Satcove Research. Benchmark of 22 October 2026. Exploratory, not peer reviewed. Canonical page: https://satcove.com/research/ai-consensus-benchmark

## Short answer

AI agreement is a useful signal, not proof. We asked up to six AI models 2,500 questions with known answers. When they all gave the same answer, it was right 94.0% of the time (1,148 of 1,221). Yet 73 of those unanimous answers were wrong. When they split and agreement fell below 60%, the majority was right only 33.5% of the time. So Satcove reads agreement as a signal, and disagreement as a reason to verify.

## Key findings

- 2,500 multiple-choice questions with known answers (MMLU-Pro 1,597; TruthfulQA 503; PubMedQA 202; GPQA Diamond 198).
- Unanimous answers: 1,148 of 1,221 correct (94.0%, 95% CI 92.5-95.2). 73 unanimous answers were wrong (6.0%).
- Majority accuracy: 94.0% unanimous, 70.5% strong agreement (80% or more), 56.2% clear majority (60-79%), 33.5% weak agreement (under 60%; 194 questions with a majority, 197 exact ties excluded).
- On the 2,303 questions with a majority, the majority scored 77.1% and Claude alone 77.5%. Counting 197 ties as failures, the majority scores 71.0% on 2,500. The majority did not clearly beat the strongest single model.

## Majority accuracy by agreement level

| Agreement level | Questions | With a majority | Correct | Accuracy | 95% interval |
|---|---:|---:|---:|---:|---:|
| Unanimous | 1,221 | 1,221 | 1,148 | 94.0% | 92.5-95.2% |
| Strong agreement (80% or more) | 440 | 440 | 310 | 70.5% | 66.0-74.5% |
| Clear majority (60-79%) | 448 | 448 | 252 | 56.2% | 51.6-60.8% |
| Weak agreement or tie (under 60%) | 391 | 194 | 65 | 33.5% | 27.2-40.4% |

Final data, including second-pass answers. Jury of 4 to 6 models per question.

## False consensus

A false consensus is a question where every answering model gave the same wrong answer: 73 of 2,500 questions (44 MMLU-Pro, 15 TruthfulQA, 9 PubMedQA, 5 GPQA Diamond). The full list is in the answers file.

| Source | Unanimous | Wrong | Share wrong |
|---|---:|---:|---:|
| GPQA Diamond | 43 | 5 | 11.6% |
| MMLU-Pro | 696 | 44 | 6.3% |
| PubMedQA | 127 | 9 | 7.1% |
| TruthfulQA (mc1) | 355 | 15 | 4.2% |

## Model results (each on its own question set)

| Model | Version tested | Questions answered | Accuracy | Notes |
|---|---|---:|---:|---|
| Claude | claude-sonnet-4-6 | 2,500 | 74.7% | Includes 475 recovered answers |
| Gemini | gemini-3.5-flash-lite | 2,499 | 69.9% | Includes 85 recovered answers |
| Mistral | mistral-large-latest | 2,499 | 67.5% | Includes 7 recovered answers |
| GPT | gpt-4o-2024-08-06 | 2,500 | 63.8% | No recovered answers |
| Grok | grok-4.20-0309-non-reasoning | 2,500 | 63.6% | Includes 1 recovered answer |
| Perplexity | sonar | 533 | 70.0% | Separate 533-question sample; includes 86 recovered answers |

Not a leaderboard: versions tested are not necessarily Satcove's current panel; four of six identifiers are undated aliases.

Same 533 questions for all six models: Claude 76.7%, Gemini 71.7%, Perplexity 70.0%, Mistral 68.7%, Grok 66.2%, GPT 64.0%.

## Jury size

Five models answered 1,965 questions, six answered 533, four answered 2. Constant jury of 5 (2,498 questions): majority 77.2%, 80 of 1,235 unanimous answers wrong (6.5%). Constant jury of 6 (533 questions): majority 77.6%, 13 of 259 unanimous answers wrong (5.0%). First-pass answers only (2,460 questions): 1,154 of 1,240 unanimous answers correct (93.1%).

## What this benchmark does not prove

- **Agreement is not proof.** 73 questions were answered identically by every model and all 73 were wrong.
- **Multiple choice, one letter.** The test measures option selection. It does not measure free-text reasoning or the way Satcove synthesizes an answer.
- **Truncated, not random.** The 2,500 questions are the first 2,500 of a 2,997-question file. MMLU-Pro supplies 1,597 of them (63.9%), so it weighs heavily on every overall figure.
- **Variable jury.** Five models answered 1,965 questions, six answered 533 and four answered 2. Perplexity answered a fixed 533-question sample.
- **Versions and recovery passes shape the scores.** 654 answers came from a second pass with a different protocol per model. Four of the six model identifiers are undated aliases.
- **Topic results mix sources.** Physics, chemistry and biology combine GPQA Diamond and MMLU-Pro, and several cells hold fewer than 30 questions.
- **No reading of position bias.** Questions have between 2 and 10 options and each source distributes correct answers differently, so letter frequencies cannot be read as a model bias.
- **One answer per model per question.** The test cannot measure how stable an answer is across repeated runs.
- **No independence or contamination test.** We do not know why models agree. The public question sets may also appear in training data. Neither point was tested.

## Methodology

Each model received every question once, with the instruction to reply with one option letter, in a fresh conversation, on 22 October 2026. Versions: Claude claude-sonnet-4-6, GPT gpt-4o-2024-08-06, Gemini gemini-3.5-flash-lite, Mistral mistral-large-latest, Perplexity sonar, Grok grok-4.20-0309-non-reasoning. 654 answers come from a second pass (Claude 475, Perplexity 86, Gemini 85, Mistral 7, Grok 1) with a different protocol per model; results are also reported on first-pass answers only. The 2,500 questions are the first 2,500 of a 2,997-question file, not a random draw. Wilson 95% intervals. Data (all rights reserved; verification copy): https://satcove.com/research/artifacts/ai-consensus-benchmark-2026-10-22-answers.csv , claims: https://satcove.com/research/artifacts/claims.csv , checksums: https://satcove.com/research/artifacts/SHA256SUMS.txt. All rights reserved. Aggregate results and analysis artifacts are provided for verification; question texts are not redistributed.

## FAQ

### Does agreement between AI models mean the answer is correct?

No. In our benchmark, unanimous answers were correct 94.0% of the time (1,148 of 1,221 questions), so agreement is a strong signal, but 73 unanimous answers were wrong. Agreement shows that models converged, not that anyone checked the facts. Models can share the same blind spot. Treat agreement as a reason for more confidence, and keep verifying anything with real consequences, such as a medical, legal or financial decision.

### How often are unanimous AI answers wrong?

In our test, 73 of 1,221 unanimous questions were unanimously wrong, or 6.0%. The rate differed by source: 4.2% for TruthfulQA, 6.3% for MMLU-Pro, 7.1% for PubMedQA and 11.6% for GPQA Diamond, although GPQA rests on only 43 unanimous questions. This is the rate on these multiple-choice questions, with up to six models answering (five on most questions). It is not a general error rate for AI.

### What does low agreement between AI models mean?

It means the models split. When agreement fell below 60%, the majority answer was correct only 33.5% of the time (194 questions with a majority; 197 more ended in an exact tie). On those questions, no single answer was a reliable guide. Low agreement does not say which model is right. It says the question is contested, hard or ambiguous, and that the answer needs checking against a source before you rely on it.

### Can multi-AI consensus replace fact-checking?

No. Consensus tells you where to look, not what is true. In our benchmark the majority answer was correct 77.1% of the time on the 2,303 questions where one existed, and 71.0% across all 2,500 questions when ties count as failures. Claude alone scored 77.5% on the same 2,303 questions. Agreement is useful for triage: high agreement lowers the priority of a check and disagreement raises it. Facts that matter still need a primary source.

### Why can ChatGPT, Claude and Gemini make the same mistake?

We did not test the cause, so this is a hypothesis, not a finding. Plausible reasons include overlapping training data, similar training methods, misconceptions common in the text the models learn from, and question wording that pulls toward the same wrong option. Our data only shows that it happens: on 73 questions every answering model chose the same wrong answer. We tested GPT-4o, Claude Sonnet 4.6 and Gemini 3.5 Flash-Lite through their APIs, not the consumer apps.

### When should you verify an AI answer?

Verify when models disagree, when the stakes are high, and when the answer rests on one claim you cannot easily judge, such as a date, a dose, a legal citation or a price. In our test, majority accuracy fell to 56.2% for clear majorities (60-79% agreement) and 33.5% below 60%. Even unanimous answers were wrong 73 times. Checking a primary source costs little compared with acting on a shared error.

### What is a false consensus in multi-AI systems?

A false consensus is a question where every model that answered gave the same answer and that answer was wrong. We found 73 among 2,500 questions: 44 in MMLU-Pro, 15 in TruthfulQA, 9 in PubMedQA and 5 in GPQA Diamond. They matter because agreement looks like confirmation. A system that reports only a majority hides these cases. The safeguard is to read the agreement level together with the stakes, and to check what matters against an external source.

### How does Satcove use disagreement between AI models?

Satcove sends one question to several models, each answering separately, then shows an agreement score and names the points where the models differ, with a next step to verify. When models split, the verdict says so instead of averaging the differences away. This benchmark tested simple letter agreement on multiple-choice questions, not Satcove's free-text synthesis. It supports reading agreement as a signal; it does not measure the accuracy of a Satcove verdict.