What is prompt sensitivity?
Prompt sensitivity is the tendency of a language model to give a different answer when the same question is worded differently. Swap a word, change the order of options, add a hint about what you hope to hear, and the answer can shift, sometimes from yes to no.
It matters for trust because a single answer can reflect the wording as much as the facts. If a conclusion flips when you rephrase, it was not a stable conclusion.
Where it comes from
Framing. "Is this investment safe?" and "What are the risks of this investment?" pull answers in different directions. The question carries an assumption and the model tends to follow it.
Leading phrasing. Models are tuned to be helpful and agreeable. If your question signals the answer you want, a model may lean toward it.
Order and format. In multiple-choice or ranked comparisons, the position of an option can influence the pick.
Missing context. A vague question leaves the model to fill gaps, and different phrasings fill them differently.
Why it matters for multi-model comparison
When two models disagree, the cause can be real divergence in what they know, or just the way the prompt landed on each. That is why a fair panel sends the same prompt, unchanged, to every model. Differences then come from the models, not from the wording.
It also means you can use sensitivity as a probe on yourself: if your own conclusion depends on how you phrased the question, you have found an assumption worth examining.
A simple test
- Write the question neutrally, without the answer you hope for.
- Ask it a second way: swap the framing from positive to negative ("benefits" versus "risks").
- Compare. If the substance holds, the answer is robust to wording. If it flips, treat the topic as uncertain and look at primary sources.
Reducing it in practice
- State the decision and the constraints, not your preferred answer.
- Ask for pros and cons, or for the strongest case on each side.
- Give the same context to every model and keep the wording fixed.
- Include the facts that matter (budget, country, date) so models do not guess them.
- Treat a single confident answer to a leading question with caution.
Try it on your own question
Run the question you care about through six independent models and read where they agree and where they split. Ask 6 AIs on Satcove, free, no card required.
Frequently asked questions
Is prompt sensitivity a bug? It is a property of how language models work. Models respond to the whole text of the prompt, so wording matters. You can reduce its effect but not remove it.
Can it create fake disagreement? Yes, if models are given different wording. Sending identical prompts to all models avoids this.
Can it create fake agreement? Yes. A strongly leading prompt can push several models toward the same answer you implied.
Does a low agreement score mean my prompt is bad? Not necessarily, but vague or leading questions are a common cause. Try a sharper, neutral version.