Yes. In our test, when all the AI models that answered gave the same answer, that answer was right 94.0% of the time (1,148 of 1,221 questions), which means 73 unanimous answers were wrong. Agreement is a useful signal about where to look, not proof that an answer is correct.
What did we test?
We sent 2,500 multiple-choice questions with known answers to six AI models, one fresh conversation per question, answer as a single letter. The questions come from four public sets: GPQA Diamond (198), MMLU-Pro (1,597), TruthfulQA (503) and PubMedQA (202). Perplexity answered a fixed sample of about 20% of the questions; the others answered all of them. We ran the test on 22 October 2026.
How accurate is the answer when the models agree?
The more the models disagree, the less often the majority is right. A single model gives you no such signal: it sounds equally sure when it is right and when it is wrong.
Is a majority of six models better than one model?
Not clearly, in this test. The majority answer was right on 77.1% of the 2,303 questions where it could be computed (ties and questions with too few answers are excluded). The best single model, Claude, was right on 74.7% of all 2,500 questions. The two figures are not on exactly the same set of questions, and the gap is small, so we do not claim that six models are more accurate than one.
Where did all the models get it wrong together?
On 73 questions, across all four question sets: 5 of 43 unanimous GPQA answers (11.6%), 44 of 696 on MMLU-Pro (6.3%), 9 of 127 on PubMedQA (7.1%) and 15 of 355 on TruthfulQA (4.2%). The models' mistakes overlap: they tend to pick the same wrong letter, so asking more of them does not remove the risk. Some "wrong" unanimous answers may also reflect questionable labels in the public sets; we have not audited them one by one.
What does this mean if I use several AI models?
Use the level of agreement to decide where to spend your checking time. Unanimous answers deserve a light check; split answers deserve a source. For consequential claims, check a primary source even when every model agrees. See how to cross-check AI answers, the AI verification feature and how to verify AI answers.
Limits
- This is not Satcove's free-form consensus: here each model answered with a single letter.
- The models were older or lower-cost versions (exact identifiers available on request), not the current models in Satcove. Results for current models may differ.
- Claude often answered in prose on hard questions; 475 of its answers came from a second pass that forced a letter. Without that pass its accuracy looks higher, but it is biased toward easier questions.
- Grok chose "A" in 24.4% of its answers, against 19.6% of correct answers being "A".
- One run, no repetitions. Multiple-choice accuracy is not how people use these tools.
- Some questions in the public sets may be mislabelled or contested.
Sources
- Satcove benchmark workbook, generated 22 October 2026: method, per-model results, calls and costs. Available on request.
- GPQA Diamond, MMLU-Pro, TruthfulQA (mc1) and PubMedQA (labeled) public question sets; we publish no question text.
- Models called through their providers' APIs, one fresh conversation per question. Exact version identifiers are in the benchmark workbook, available on request.
Related reading: Cross-check AI answers · Consensus engine · Governance tool vs review step · Multi-AI platform · AI truth-finding · Best AI for fact checking · Satcove for Enterprise