Why Language Models Hallucinate

Imagine an AI legal assistant, tasked with drafting a memo, confidently citing a series of compelling but entirely non-existent legal precedents. This scenario is not science fiction; it is a manifestation of "LLM hallucination," a persistent challenge in artificial intelligence that poses significant risks to the legal profession. For lawyers, whose credibility rests on precision and verifiable facts, understanding the roots of this phenomenon is no longer an academic exercise but a professional necessity.

OpenAI recently released a new research paper "Why LanguageModels Hallucinate" which argues that language models hallucinate because standard training and evaluation procedures reward guessing over acknowledging uncertainty.

An LLM hallucination is a plausible but false statement generated by an AI model (p. 1). These are not random glitches but confident, well-articulated falsehoods, such as providing an incorrect dissertation title or inventing a birth date for a real person (p. 2). The models behave much like an over-eager student during an exam (or as Aidid says "like an eager but inexperienced intern or associate"). This tendency undermines the AI reliability that legal professionals require and introduces new legal tech risks into daily practice.

1)    The Genesis of Error: How AI Learns to Make Mistakes

The first primary driver of LLM hallucination originates in the initial training phase, known as pretraining (p. 2). During this stage, a model learns the patterns of language by processing an enormous body of text. A common misconception is that errors arise solely from flawed or "garbage" training data (See "GIGO: Garbage in, Garbage out," p. 12). However, the authors reveal a more fundamental issue: even if the training data were perfectly accurate, the statistical objectives that guide the learning process would inevitably lead the model to generate errors (p. 2).

This can be understood by drawing an analogy to a simpler machine learning task: binary classification. Imagine training an AI to answer "Yes" or "No" to the question, "Is this statement a valid fact?" (p. 2). Successfully generating a valid statement is inherently more difficult than just classifying one. The research shows a mathematical relationship where the rate of generative errors is directly tied to the model's misclassification rate on this "Is-It-Valid" (IIV) problem (p. 3). If a model struggles to distinguish a valid statement from a plausible falsehood, it will naturally produce such falsehoods when asked to generate text.

This problem is particularly acute when dealing with arbitrary facts, or information with no discernible pattern, such as birthdays or specific details in an obituary. If a fact appears only once in the vast training data (a "singleton"), the model has a very weak statistical basis for reproducing it accurately (p. 3). This highlights a core issue of training data limitations: the model is not "reasoning" from a knowledge base but predicting based on statistical frequency, making errors a natural outcome of the process (p. 6).

2)    Perjury by Design: How We Train AI to Be Overconfident

The second driver explains why hallucinations persist even after models are refined in a "post-training" stage, which is designed to improve accuracy and reduce errors. The problem lies in the very methods used to evaluate AI performance. Most AI evaluation methods and industry leaderboards operate like standardized tests with binary, right-or-wrong grading (p. 12).

In this system, a model receives one point for a correct answer and zero for an incorrect one. Crucially, it also receives zero points for admitting uncertainty with a response like "I don't know" (IDK) (p. 4). This creates a perverse incentive that discourages honesty and rewards guesswork. An AI model that always makes its "best guess" when faced with model uncertainty will, on average, score higher on these benchmarks than a more cautious model that abstains. As a result, language models are perpetually in "test-taking mode," where guessing is the optimal strategy (p. 4).

An analysis of ten influential evaluation benchmarks found that nearly all use a binary grading system that penalizes abstention, thereby reinforcing hallucinatory behavior (p. 13). When scoreboards prioritize accuracy above all else, they create a "false dichotomy" between right and wrong, motivating developers to build models that confidently guess rather than express uncertainty (Pg. 15). This institutional pressure for high scores contributes directly to AI overconfidence and the persistence of plausible falsehoods.

Conclusion: A Call for Cautious Counsel

For the legal profession, the implications are profound. The tendency for LLMs to hallucinate is not a simple bug to be patched but an inherent feature arising from their statistical foundations and the ecosystem in which they are evaluated. The risk of an AI confidently fabricating case law, misrepresenting contracts, or inventing factual details is a clear and present danger to legal practice.

This does not diminish the potential of AI for lawyers, but it demands a shift in perspective. These tools should not be treated as infallible oracles but as highly sophisticated, yet fundamentally fallible, assistants. The burden of verification and ensuring AI factuality cannot be delegated; it must remain with the legal professional. Understanding the deep-seated statistical reasons for language model errors is the first step toward mitigating legal tech risks and harnessing the power of AI responsibly.

Why Language Models Hallucinate Adam Tauman Kalai, Ofir Nachum, Santosh S. Vempala, Edwin Zhang September 4, 2025 ArXiv ID: 2509.04664