Compiled 2026-10-15 · CC BY 4.0
AI hallucination rates: what the published benchmarks say
Short answer
There is no single hallucination rate. Vectara measures how faithfully a model summarises a given text (best listed rate 1.8%). Artificial Analysis measures general knowledge reliability and reports a different statistic. The numbers are not comparable across benchmarks.
What this page is not
Satcove did not run these benchmarks and publishes no ranking of its own here. Every figure is copied from its source on 2026-10-15. Benchmarks change often: open the source before you rely on a number.
Vectara Hallucination Leaderboard
Measures: How often a model adds information that is not in a document when asked to summarise it. It tests faithfulness to a supplied text, not general knowledge.
Evaluator: HHEM-2.3 (Vectara's hallucination evaluation model). Data: More than 7,700 articles across news, technology, science, medicine, legal, sports, business and education; the dataset is not public.
| Model | Hallucination rate | Factual consistency | Answer rate |
|---|---|---|---|
| antgroup/finix_s1_32b | 1.8% | 98.2% | 99.5% |
| openai/gpt-5.4-nano-2026-03-17 | 3.1% | 96.9% | 100% |
| google/gemini-2.5-flash-lite | 3.3% | 96.7% | 99.5% |
| microsoft/Phi-4 | 3.7% | 96.3% | 80.7% |
| meta-llama/Llama-3.3-70B-Instruct-Turbo | 4.1% | 95.9% | 99.5% |
| snowflake/snowflake-arctic-instruct | 4.3% | 95.7% | 62.7% |
| google/gemma-3-12b-it | 4.4% | 95.6% | 97.4% |
| mistralai/mistral-large-2411 | 4.5% | 95.5% | 99.9% |
| qwen/qwen3-8b | 4.8% | 95.2% | 99.9% |
| amazon/nova-pro-v1:0 | 5.1% | 94.9% | 99.3% |
| amazon/nova-2-lite-v1:0 | 5.1% | 94.9% | 99.6% |
| mistralai/mistral-small-2501 | 5.1% | 94.9% | 97.9% |
| ibm-granite/granite-4.0-h-small | 5.2% | 94.8% | 100% |
| google/gemma-4-26b-a4b-it | 5.2% | 94.8% | 99.8% |
| ai21labs/jamba-mini-2 | 5.3% | 94.7% | 99.6% |
The 15 lowest rates on the leaderboard as updated 2026-09-22. Several have a low answer rate, which can flatter a model that declines to answer. Source: https://github.com/vectara/hallucination-leaderboard. Licence: Apache-2.0 (repository); no licence is stated for the dataset.
The AI providers you actually use
The table above shows the lowest rates, which are mostly small or little-known models. Here is every entry on the Vectara leaderboard (108 models in total) for Anthropic, OpenAI, Google, xAI and Mistral, grouped by provider. Rates vary widely inside one provider, so a provider name alone says little. Perplexity has no entry.
Anthropic (Claude) · 9 models · 9.8% to 12.2%
| Model | Hallucination | Answer rate |
|---|---|---|
| anthropic/claude-haiku-4-5-20251001 | 9.8% | 99.5% |
| anthropic/claude-sonnet-4-20250514 | 10.3% | 98.6% |
| anthropic/claude-sonnet-4-6 | 10.6% | 99.9% |
| anthropic/claude-opus-4-5-20251101 | 10.9% | 98.7% |
| anthropic/claude-opus-4-1-20250805 | 11.8% | 92.4% |
| anthropic/claude-opus-4-20250514 | 12% | 91% |
| anthropic/claude-sonnet-4-5-20250929 | 12% | 95.6% |
| anthropic/claude-opus-4-7 | 12% | 98% |
| anthropic/claude-opus-4-6 | 12.2% | 99.8% |
OpenAI (GPT) · 22 models · 3.1% to 23.3%
| Model | Hallucination | Answer rate |
|---|---|---|
| openai/gpt-5.4-nano-2026-03-17 | 3.1% | 100% |
| openai/gpt-5.4-mini-2026-03-17 | 5.5% | 100% |
| openai/gpt-4.1-2025-04-14 | 5.6% | 99.9% |
| openai/gpt-6-sol | 6.5% | 100% |
| openai/gpt-5.4-2026-03-05 | 7% | 99.9% |
| openai/gpt-5.4-pro-2026-03-05 | 8.3% | 100% |
| openai/gpt-5.2-low-2025-12-11 | 8.4% | 100% |
| openai/gpt-6-astra | 8.7% | 100% |
| openai/gpt-5.5 | 9.3% | 100% |
| openai/gpt-4o-2024-08-06 | 9.6% | 93.8% |
| openai/gpt-5-nano-2025-08-07 | 10.5% | 100% |
| openai/gpt-5.2-high-2025-12-11 | 10.8% | 100% |
| openai/gpt-5.1-low-2025-11-13 | 10.9% | 100% |
| openai/gpt-5.1-high-2025-11-13 | 12.1% | 100% |
| openai/gpt-5.6-sol | 12.4% | 99.1% |
| openai/gpt-5-mini-2025-08-07 | 12.9% | 99.9% |
| openai/gpt-oss-120b | 14.2% | 99.9% |
| openai/gpt-5-minimal-2025-08-07 | 14.7% | 99.9% |
| openai/gpt-5-high-2025-08-07 | 15.1% | 99.9% |
| openai/o4-mini-low-2025-04-16 | 18.6% | 98.7% |
| openai/o4-mini-high-2025-04-16 | 18.6% | 99.2% |
| openai/o3-pro | 23.3% | 100% |
Google (Gemini) · 7 models · 3.3% to 13.6%
| Model | Hallucination | Answer rate |
|---|---|---|
| google/gemini-2.5-flash-lite | 3.3% | 99.5% |
| google/gemini-2.5-pro | 7% | 99.1% |
| google/gemini-2.5-flash | 7.8% | 99% |
| google/gemini-3.1-flash-lite-preview | 8.2% | 99.6% |
| google/gemini-3.1-pro-preview | 10.4% | 99.4% |
| google/gemini-3-flash-preview | 13.5% | 99.8% |
| google/gemini-3-pro-preview | 13.6% | 99.4% |
xAI (Grok) · 5 models · 5.8% to 20.2%
| Model | Hallucination | Answer rate |
|---|---|---|
| xai-org/grok-3 | 5.8% | 93% |
| xai-org/grok-4-1-fast-non-reasoning | 17.8% | 98.5% |
| xai-org/grok-4-1-fast-reasoning | 19.2% | 99.7% |
| xai-org/grok-4-fast-non-reasoning | 19.7% | 99.2% |
| xai-org/grok-4-fast-reasoning | 20.2% | 99.5% |
Mistral · 9 models · 4.5% to 24.2%
| Model | Hallucination | Answer rate |
|---|---|---|
| mistralai/mistral-large-2411 | 4.5% | 99.9% |
| mistralai/mistral-small-2501 | 5.1% | 97.9% |
| mistralai/ministral-3b-2410 | 7.3% | 99.9% |
| mistralai/ministral-8b-2410 | 7.4% | 99.9% |
| mistralai/mistral-3-large-2512 | 14.5% | 98.8% |
| mistralai/ministral-3-14b-2512 | 19.4% | 99.6% |
| mistralai/ministral-3-8b-2512 | 21.7% | 99.1% |
| mistralai/mistral-medium-2508 | 22.7% | 99.7% |
| mistralai/ministral-3-3b-2512 | 24.2% | 74.3% |
Same source and date as above (2026-09-22). Different model versions are tested on the same task; do not compare across providers as a ranking of the companies.
AA-Omniscience (Artificial Analysis)
Measures: General knowledge reliability. It rewards correct answers, penalises wrong answers and does not penalise refusals. The index runs from -100 to 100.
- Index: higher is better. Rewards correct answers, penalises hallucinations, does not penalise abstaining.
- Accuracy: higher is better. Correct answers divided by all questions, whether or not the model chose to answer.
- Hallucination rate: lower is better. Incorrect answers divided by the sum of incorrect, partial and not-attempted responses.
| Metric | Rank | Model | Score |
|---|---|---|---|
| Omniscience Index | 1 | Claude Opus 5.5 (Max, Default Fallback) | 46 |
| Omniscience Index | 2 | GPT-6 Astra (High) | 44 |
| Omniscience Index | 3 | Claude Fable 5.1 (Max, Default Fallback) | 43 |
| Accuracy | 1 | Claude Fable 5.1 (Max, Default Fallback) | 67% |
| Accuracy | 2 | Claude Opus 5.5 (Max, Default Fallback) | 66% |
| Accuracy | 3 | Claude Fable 5.1 (Xhigh, Default Fallback) | 66% |
| Hallucination rate (lower is better) | 1 | MiniCPM5-1B (Non-reasoning) | 1% |
| Hallucination rate (lower is better) | 2 | G9v3-3B | 12% |
| Hallucination rate (lower is better) | 3 | G9v3-39A5B | 13% |
Top three per metric as shown on the page; the full table was not readable. A model can top the hallucination-rate column and still be weak on accuracy, so read the metrics together. The page states no evaluation date. Source: https://artificialanalysis.ai/evaluations/omniscience; paper: arXiv 2511.13029.
FACTS Grounding (Google DeepMind / Kaggle): not yet included
The leaderboard could not be read automatically on the check date, so no score is published here. Official page.
How to read these numbers
- Summarising a text and answering from memory are different tasks. A model can be strong at one and weak at the other.
- A lower rate is not automatically better if the model refuses more often; check the answer rate.
- Leaderboards update and models are renamed. A figure here is a dated snapshot.
- Agreement between several models is a useful signal, not proof that an answer is true. See does AI consensus prove truth and our benchmark policy.
Download and cite
data.jsonOur compilation is CC BY 4.0; the underlying data belongs to its sources and keeps their terms. Credit the original benchmark and link to this page.
Satcove. AI hallucination benchmarks compiled, version 1.0, checked 2026-10-15. https://satcove.com/ai-hallucination-benchmarks