You ask ChatGPT for the section number in a contract, or the year a regulation came into force, and it gives you a confident answer. It sounds looked up, not guessed. Then you check. Wrong year. Wrong section.
That confidence is not a personality quirk. A hallucination, in this context, is a fluent wrong answer delivered with the same certainty as a correct one, because the model was never taught to sound different when it is guessing. Adam Kalai and Ofir Nachum, OpenAI researchers, found the cause in a 2025 paper: the benchmarks used to grade these models score a wrong guess and an honest “I don’t know” identically, at zero. Guessing costs nothing. So models learn to guess. Occasionally the confident wrong answer is a citation that does not exist at all, quoted as if it were sitting open on the desk.
This is the same gap that Agni, an app that measures how people actually work with AI day to day, was built to make visible: not whether you can operate a chatbot, but whether you catch it when it is wrong.
Where does that confidence actually come from?
AI models are trained, in part, against a scoreboard, and the scoreboard is the problem. Kalai and Nachum, in “Why Language Models Hallucinate” (arXiv:2509.04664), looked at widely used evaluations including GPQA, MMLU-Pro and SWE-bench and found the grading itself rewards guessing: right or wrong is scored, but there is no partial credit for hedging and no penalty worse than a flat zero for being wrong. An honest “I don’t know” scores the same zero as a confident falsehood.
Run that arithmetic forward. If a model has even a small, genuine belief about the answer, guessing beats abstaining every time, because guessing has a chance of scoring and abstaining never does. Kalai and Nachum prove this formally: under binary grading, any nonzero confidence makes guessing the mathematically better move. A model that behaved honestly, hedging when unsure, would simply score worse on the leaderboards that labs use to compare and sell their models. So the training pressure runs one way.
What did OpenAI’s hallucination paper actually find?
Kalai and Nachum’s paper proposes a corrected scoring rule: give partial credit for “I don’t know,” penalise wrong answers more than a flat zero, and have the model answer only when its internal confidence clears a set threshold. Applied against existing benchmarks, their analysis suggests models would abstain and answer “I don’t know” on roughly 30% of queries, instead of guessing through all of them.
| Binary grading (current practice) | Corrected grading (Kalai and Nachum’s proposal) | |
|---|---|---|
| Right answer | Full credit | Full credit |
| Wrong answer | Zero | Penalised below zero |
| ”I don’t know” | Zero, same as wrong | Partial credit |
| Model’s best strategy | Always guess | Answer only above a confidence threshold |
That is not a small adjustment. Roughly three in ten answers a model currently states outright, it would flag as uncertain under fairer scoring. Today, none of that uncertainty reaches you. The interface shows a clean sentence, not a probability.
Does this mean you can’t trust anything AI tells you?
No, and Kalai and Nachum’s paper gives a sharper answer than blanket suspicion. The risk concentrates in one place: a fact that appears rarely, or only once, anywhere in the model’s training data. Kalai and Nachum illustrate this with a real example from their own testing: asked for one author’s birthday, a chatbot returned three different confident, wrong dates across three attempts, because that fact was too sparse in the training data for the model to have ever really learned it. No amount of extra scale fixes a fact that was only stated once to begin with.
That gives you a working test. A fact repeated everywhere online, a well-known date, is comparatively safe ground. A precise figure that only turns up once, an obscure citation, a niche regulation, is where the incentive to guess confidently is strongest, and where your own checking matters most.
How do you build the habit of checking confident answers?
Treat fluency and accuracy as two separate questions, because the model itself was never trained to keep them separate. A wrong answer and a right one come out of the same process, phrased with the same assurance, so the writing quality tells you nothing about whether the fact underneath is real. Before repeating an exact number, date or citation a model gives you, find it in a second, independent source. If it is not there, do not treat the first answer as settled. That habit, checking before you act rather than counting how often you open the tool, is itself a separate skill from fluency with the interface; see How to tell if you’re using AI tools well, not just often.
Agni’s AI Literacy Score quiz, fourteen questions, about three minutes, scores exactly this habit under its “Use it” dimension: not whether you know how to prompt a chatbot, but whether you check a confident-sounding answer before you act on it. It returns a score out of 100 and a percentile against people in your job and country, turning “sounds right” versus “is right” into something you can measure rather than guess at. It sits at couragehorizon.com/agni.
Common questions
Is a confident wrong answer the same as the model lying? No. Lying requires knowing the truth and choosing to hide it. A language model has no fact-check step running underneath its sentences; it produces the most statistically likely continuation of the text, and binary grading trains that continuation to sound decisive whether or not it is accurate.
Will this get fixed as models get bigger? Not on its own. Kalai and Nachum’s argument targets scoring, not scale: a bigger model graded on the same binary benchmarks still faces the same incentive to guess. Their proposed fix is to add confidence thresholds to the benchmarks themselves; as of their paper, most major leaderboards had not adopted it.
How do I find out where my own verification habits stand? There is no universal test for this, but Agni’s AI Literacy Score quiz gives you one measurable version of it, scored across four dimensions rather than a single grade. It is a self-assessment tool, not a certification, and the app itself is still in development for iPhone, not yet available to download. If the wider question nagging at you is whether you’re generally behind on AI rather than just anxious about it, that is a related but separate question; see How do I know if I’m behind on AI, or just anxious about it?
Next time a chatbot answers instantly and precisely, remember what that precision actually costs it to produce: nothing. Uncertainty is the only answer it was ever trained to avoid.
Sources
- Kalai, A. T., Nachum, O., Vempala, S. S. and Zhang, E. “Why Language Models Hallucinate.” arXiv:2509.04664, September 2025. arxiv.org/abs/2509.04664
- OpenAI. “Why language models hallucinate.” OpenAI, September 2025. openai.com/index/why-language-models-hallucinate