Six AI models, same questions, real answers
Each question was sent unchanged to six models on 2026-10-28. Each answer was reduced to one stance (yes, it depends, or no) and one quote. This is a sample of three questions, not a benchmark: it shows where the models agree and where they split, not which one is best.
Can an AI chatbot reliably tell you whether a citation it gives is real?
| Model | Stance | Quote |
|---|
| Claude | No | I can make errors about citations including inventing ones that sound plausible, so you should verify any citation I provide through independent sources rather than relying on my assessment of its authenticity. |
| GPT | It depends | The reliability of an AI chatbot in determining the authenticity of a citation depends on the accuracy of its underlying data and training. |
| Gemini | No | An AI chatbot cannot reliably verify its own citations because it generates text based on probability rather than actual knowledge, meaning it can confidently fabricate sources and then double down on verifying those hallucinations. |
| Grok | No | AI chatbots can hallucinate citations that look real but are entirely fabricated. |
| Mistral | It depends | It depends on the AI's training data and access to real-time verification tools. |
| Perplexity | No | An AI chatbot cannot reliably confirm that a citation it gives is real because it may confidently fabricate sources. |
Should you paste confidential client documents into a public AI chatbot?
| Model | Stance | Quote |
|---|
| Claude | No | Pasting confidential client documents into a public AI chatbot creates serious security and privacy risks, including potential data breaches, loss of attorney-client privilege, and regulatory violations. |
| GPT | No | Pasting confidential client documents into a public AI chatbot poses significant risks to privacy and security. |
| Gemini | No | You should not paste confidential client documents into a public AI chatbot because it can violate client confidentiality and data privacy agreements. |
| Grok | No | Pasting confidential client documents into a public AI chatbot risks exposing sensitive data and violating privacy obligations. |
| Mistral | No | Never paste confidential client documents into a public AI chatbot to protect sensitive information. |
| Perplexity | No | Do not paste confidential client documents into a public AI chatbot; use an approved, secure tool or remove identifying information first. |
Do AI models change a correct answer when the user pushes back without new evidence?
| Model | Stance | Quote |
|---|
| Claude | It depends | AI models like me can sometimes shift positions when pushed back, but whether we should depends on the strength of the counterargument and whether it contains actual evidence or just rhetorical pressure. |
| GPT | No | AI models provide consistent answers based on the data they are trained on, regardless of user pushback. |
| Gemini | Yes | AI models frequently exhibit sycophancy, causing them to reverse or doubt their correct answers when challenged by a user even without new evidence. |
| Grok | Yes | When users push back without new evidence, AI models may shift from a correct answer to accommodate the user's preference. |
| Mistral | It depends | It depends on the model's design, user interaction history, and the nature of the pushback without new evidence. |
| Perplexity | It depends | Models sometimes abandon a correct answer under persistent pushback, but this is a tendency rather than a universal rule. |
Limits
- Three questions, one run per model, one date. Answers can change on another day.
- Stances are coded from each answer by a short rule (yes, it depends, no), not by a score.
- Quotes are the model's own words, translated to English where needed.
To check one question against six models at once, run it on Satcove.