When you query multiple AI models and see "87% consensus," what does that number actually mean? How is it calculated, and what should you do with it? This guide explains how consensus scoring works and how to interpret the results.
What Is a Consensus Score?
A consensus score measures how much agreement exists among multiple AI models when answering the same question. It's not a simple average of confidence scores—it's a measure of inter-model agreement.
How Consensus Is Calculated
Query multiple models
Same question sent to GPT-4, Claude, Gemini, Grok, etc.
Extract key claims
Each response is broken into discrete factual claims
Cross-validate claims
Each model rates the accuracy of other models' claims
Calculate agreement
Percentage of claims that most models agree on
Interpreting Consensus Levels
90%+ — Strong Consensus
Most models agree on most claims. While not proof of accuracy, this represents independent confirmation across different training data and architectures. Light verification is typically sufficient.
70-89% — Moderate Consensus
General agreement with some divergence on specifics. The core claims are likely reliable, but details should be verified. Pay attention to where models disagree.
50-69% — Low Consensus
Significant disagreement exists. This often indicates the question touches on areas where AI knowledge is uncertain, the topic is genuinely controversial, or the question is ambiguous. Human investigation is required.
<50% — No Consensus
Models actively disagree. Don't use AI-generated information here without primary source verification. This is valuable data—it tells you where AI can't help and human expertise is essential.
Why Consensus Matters More Than Confidence
Individual AI models often display high confidence even when wrong. A single model saying "I'm 95% confident" tells you about the model's internal pattern matching, not about real-world accuracy. Consensus is different.
Single-Model Confidence
- •Based on internal pattern matching
- •Doesn't correlate well with accuracy
- •Can be high for hallucinations
- •One training dataset, one perspective
Multi-Model Consensus
- •Based on independent agreement
- •Calibrated to cross-validation
- •Hallucinations typically don't replicate
- •Multiple datasets, multiple perspectives
What Consensus Doesn't Tell You
High consensus is not proof of truth. All models could share the same blind spot or error from overlapping training data. Consensus tells you where you can have calibrated confidence, not certainty.
For high-stakes decisions, consensus scores help you allocate verification effort: less time on high-consensus claims, more time on low-consensus ones.
Understanding consensus scores transforms how you use AI research. Instead of wondering "Can I trust this answer?", you can ask "How much verification does this specific claim need?" That's a much more actionable question.
Frequently Asked Questions
What is a consensus score?
A measure of inter-model agreement — how much multiple AI models agree with each other — not a single model’s self-reported confidence.
How do you interpret consensus levels?
The post’s bands: 90%+ is strong, 70–89% moderate, 50–69% low (investigate), and below 50% means verify independently.
What does a consensus score not tell you?
It doesn’t prove the agreed answer is true — the post says to use scores to allocate verification effort, not as proof of correctness.