Satcove asked up to six AI models the same 2,500 questions, each with a known correct answer, and compared how often they agreed with how often they were right. This is our own research, not an analysis of outside reports.
What we found
When every model gave the same answer, that answer was right 94.0% of the time (1,148 of 1,221 cases). In 73 cases, all models gave the same wrong answer. When agreement fell below 60%, the majority answer was right only 33.5% of the time.
Agreement is therefore a useful signal, not proof. A split is a good reason to verify; unanimity is a good sign, not a guarantee.
Majority vote versus the best model
On the 2,303 questions where the comparison applies, majority vote scored 77.1%, against 77.5% for the best single model, Claude. Majority vote did not beat the best model in this test.
Limits
The questions came from MMLU-Pro (1,597), TruthfulQA (503), PubMedQA (202) and GPQA (198). Answers were letter-only multiple choice, so the results may not carry over to open-ended questions. Perplexity ran on a fixed sample of 533 questions instead of all 2,500, to limit cost. Two further questions had only four valid answers. The study is exploratory.
Read the data
The full method, the 73 unanimous-but-wrong cases and the downloadable files are on the AI Consensus Benchmark page. To see how several models answer your own question, try Satcove Open.