Featured

Systemic Limitations: Hallucination and Inherited Bias

AI - a tool Hits: 127 23 minutes read

The Double-Edged Sword of Probabilistic Generation

This section addresses the inherent flaws of the next-word paradigm. It explains hallucinations not as "errors" in the traditional sense, but as the model perfectly executing its core directive: generating highly probable text, regardless of factual truth. It also explores bias, arguing that because the model is designed to reproduce the statistical patterns of human text, it inevitably inherits and reproduces the prejudices and flaws embedded within that data.

Belongs to the series: The Architecture of Approximation: How Next-Word Prediction Unlocks the Structure of Human Knowledge

The Moment the Model Lies to You

You are writing a dissertation. It is eleven o'clock at night, the deadline is tomorrow, and you need a citation. You ask the model for a reference to support your argument about post-colonial literature. The model responds instantly, confidently, fluently. It gives you an author, a title, a journal, a year, a volume number, a page range. The title sounds plausible. The journal sounds real. The author's name has the right shape for the field. You paste it into your bibliography. You submit the essay.

Three weeks later, your supervisor emails you. The reference does not exist. The author has never published in that journal. The volume number corresponds to a year in which the journal was not yet in print. The citation is fabricated. Perfectly formatted. Entirely fictional. The model did not lie to you. It did not intend to deceive you. It did what it always does. It generated the most probable next tokens given the context. And the most probable continuation of "a post-colonial literature reference published in 2019" is a string that looks exactly like a real reference. Because that is what references look like. Because the pattern of a citation is more strongly encoded in the model's weights than the existence of any particular paper.

This is hallucination. And it is not a bug. It is not a glitch. It is not something the engineers will patch in the next update. It is the inevitable consequence of the architecture. It is what happens when a system optimised for probability is asked to deliver truth. The two are not the same. They overlap, most of the time. But they are not the same. And in the gap between them, the model will always, inevitably, confidently, produce text that follows the pattern perfectly and points at something that was never there. 

Transparency note: This article was written with AI assistance. The biographical anchors, the judgements, and the voice belong to the author.

What Hallucination Actually Is

The term "hallucination" is borrowed from psychiatry, where it refers to a perception without an external stimulus. You see something that is not there. You hear a voice in an empty room. The perception is vivid, detailed, internally consistent. And it is entirely generated by the mind, untethered from the world.

The large language model hallucinates in precisely this sense. It generates text that is vivid, detailed, internally consistent, grammatically flawless, and stylistically appropriate. And it is entirely generated by the model's internal patterns, untethered from the facts of the world. The hallucinated citation reads like a real citation. The hallucinated historical event reads like a real historical event. The hallucinated scientific finding reads like a real scientific finding. The form is perfect. The content is absent.

This is not carelessness. This is not the model being "lazy" or "sloppy." This is the model doing exactly what it was trained to do. The training objective was: given the preceding tokens, predict the next token. The model was never asked: is this true? The model was never asked: does this correspond to an actual state of affairs in the world? The model was never asked: can you verify this against a reliable source? The model was asked one question, and one question only: what comes next? And it answers that question with extraordinary skill. The answer is probable. The answer is fluent. The answer is coherent. The answer is, in every formal respect, indistinguishable from a truthful answer. And it is, in some percentage of cases, entirely false.

The hallucination is not a failure of the model. It is a success of the model. The model has learned the pattern of a reference so well that it can generate one that does not exist. It has learned the pattern of a historical narrative so well that it can generate one that never happened. It has learned the pattern of a scientific claim so well that it can generate one that was never tested. The pattern is the competence. The pattern is also the failure. They are the same thing, viewed from different angles.

Probability Versus Truth

Here is the fundamental mismatch at the heart of every large language model. The model is optimised for probability. The user wants truth. These are related, but they are not identical, and the difference matters.

Probability is a measure of how common a pattern is. If nine hundred out of a thousand training examples contain a particular fact, the model will assign high probability to that fact. It will reproduce it confidently, fluently, without hesitation. If ten out of a thousand training examples contain a different fact—a rarer fact, a more nuanced fact, a fact that contradicts the majority—the model will assign low probability to that fact. It will suppress it. It will bury it. It will, if pressed, produce the common fact, because the common fact is the probable fact, and the probable fact is the one that minimises the loss.

Truth, however, does not always align with commonness. The Earth is not flat, but for most of human history, the majority of texts asserted that it was. The correct answer to a question is not always the most popular answer. The accurate diagnosis is not always the most frequently cited one. The true historical event is not always the most commonly narrated one. Truth and probability overlap in the centre of the distribution, where common knowledge is also true knowledge. But at the edges—in rare cases, in novel situations, in contested domains, in rapidly changing fields—truth and probability diverge. And it is at the edges that the model fails.

The model learned what is common. And common tracks true most of the time. The sun rises in the east. Water freezes at zero degrees. London is the capital of the United Kingdom. These are common and true. The model gets them right. But common breaks from true at the margins. The model confuses similar-sounding facts. It conflates people with similar names. It attributes quotes to the wrong author. It places events in the wrong decade. It invents plausible-sounding but non-existent details to fill gaps in the pattern. And it does all of this with the same confidence, the same fluency, the same grammatical perfection, as when it is telling the truth. Because the model does not know the difference. The model has no concept of truth. The model has only probability. And probability, at the edges, is a poor guide to truth.

The Edges Where It Breaks

The centre of the distribution is safe. The model knows that Paris is in France. It knows that Shakespeare wrote Hamlet. It knows that the mitochondria are the powerhouse of the cell. These facts appear millions of times in the training data. They are deeply encoded, strongly weighted, robust to perturbation. The model will not hallucinate these. Or rather, the probability of hallucinating them is so vanishingly small that it is effectively zero.

But move towards the edges. Ask the model about a minor character in a lesser-known novel. Ask it about a specific statistic from a 2017 government report. Ask it about the third author on a paper published in a specialist journal with a circulation of two hundred. Ask it about what happened in a small town in rural Wales in 1843. Ask it about a clause in a contract that was never digitised. The model's confidence does not decrease. The fluency does not degrade. The grammar remains perfect. But the content becomes unreliable. The model fills the gap with the most probable continuation. And the most probable continuation of "a minor character in a lesser-known novel" is a name that sounds like a minor character in a lesser-known novel. The model generates the name. It sounds right. It fits the pattern. And it does not exist.

This is why hallucination is not random. It is not noise. It is structured. The hallucinated content is always plausible. It always follows the pattern. It always looks like it belongs. A hallucinated legal citation will have the correct format, the correct court abbreviation, the correct year structure. A hallucinated medical study will have the correct abstract structure, the correct statistical language, the correct hedging phrases. The hallucination is not a random string of words. It is a highly probable string of words that happens to be false. And that is precisely what makes it so dangerous. A random error is easy to spot. A plausible error is not.

For a university student, this has immediate practical consequences. You cannot trust the model's output without verification. You cannot paste a citation into your bibliography without checking that the paper exists. You cannot cite a statistic without confirming the source. You cannot rely on the model's summary of a historical event without cross-referencing it against a textbook. The model is a starting point, not a destination. It is a draft, not a final version. It is a suggestion, not a fact. And the failure to recognise this—the failure to understand that probability is not truth—is the single most common mistake that students make when using large language models.

Bias: The Same Mechanism, Different Failure

If hallucination is what happens when the model reproduces a pattern that does not correspond to reality, bias is what happens when the model reproduces a pattern that corresponds to reality but that we wish it did not. The mechanism is identical. The model learns what is common. The model reproduces what is common. If the common pattern is true, we call it knowledge. If the common pattern is false, we call it hallucination. If the common pattern is true but unjust, we call it bias. In all three cases, the model is doing the same thing. It is reproducing the statistical regularities of the training data. The difference is not in the model. The difference is in us. In what we have written. In what we have chosen to record, to publish, to amplify. In what we have deemed worth preserving in text.

The training corpus of a large language model is, in essence, a mirror. It reflects the text that humanity has produced. And humanity has produced a great deal of text that is racist, sexist, homophobic, classist, ableist, and otherwise prejudiced. Not because the people who wrote it were all bigots. But because the structures of power, the assumptions of an era, the casual cruelties of everyday language, are embedded in the text. They are in the novels, the newspapers, the legal documents, the medical records, the forum posts, the social media comments. They are in the language itself—in the connotations of words, in the associations of phrases, in the patterns of who is described as doing what to whom.

The model absorbs all of it. It does not judge. It does not filter. It does not say, "This pattern is harmful; I shall not learn it." It learns everything. It encodes everything. It reproduces everything. And when you prompt it, it generates text that reflects the full spectrum of human expression, including the parts we would rather it did not. The model is not biased because it was designed to be biased. The model is biased because the data is biased, and the model's only job is to reproduce the data. The bias is not in the machine. The bias is in the corpus. The machine is simply honest about it.

How Bias Enters the Data

Let us be specific. Bias enters the training data through at least four distinct channels, and understanding them is essential for anyone who will interact with, build, or regulate these systems.

First: representational bias. The corpus over-represents certain voices and under-represents others. English-language text dominates. American and British perspectives dominate. Male-authored text dominates. Text produced by educated, urban, economically privileged individuals dominates. The result is a model that is most fluent, most accurate, most "intelligent" when speaking in the register of a white, educated, Western, male voice. When asked to generate text in other registers—in dialects, in non-Western cultural contexts, in the voices of marginalised communities—the model is less accurate, less fluent, more prone to stereotyping. Not because it was designed to be. Because the data is thinner there. The patterns are weaker. The model falls back on the strongest patterns it has, and the strongest patterns are often stereotypes.

Second: associational bias. The model learns statistical associations between words. If, in the training data, the word nurse co-occurs more frequently with female pronouns than male pronouns, the model will encode that association. If the word engineer co-occurs more frequently with male pronouns, the model will encode that too. If certain ethnic groups are disproportionately associated with crime-related vocabulary in the corpus—because of biased policing, biased journalism, biased sentencing—the model will encode those associations. It will then reproduce them. Ask the model to complete the sentence "The nurse walked into the room and ___," and it is more likely to use "she" than "he." Not because the model is sexist. Because the data is. The model is a mirror, and the mirror shows what is in front of it.

Third: historical bias. The training corpus contains text from many decades, even centuries. The language of the 1920s is not the language of the 2020s. The assumptions of the Victorian era are not the assumptions of the present day. But the model does not timestamp its knowledge. It does not say, "This association was common in 1930 but is no longer acceptable." It simply encodes the association. And when prompted, it may reproduce it. The model does not know that the world has changed. It knows only what the text says. And the text says what the text says.

Fourth: amplification bias. The model does not merely reproduce the bias in the data. It can amplify it. If a stereotype appears in 60% of the training examples, the model may reproduce it in 70% of its outputs, because the pattern is strong and the model is optimised to follow strong patterns. The feedback loop is real: biased text generates biased models, and biased models generate biased text, which is then added to the corpus for the next generation of models. The bias compounds. It does not merely persist. It grows.

The Reproduction of Patterns We Would Rather It Did Not

The original text puts this with characteristic precision: "A system built to reproduce our patterns reproduces the ones we would rather it did not." This is not a design flaw. It is a design consequence. The system was built to reproduce patterns. All patterns. Without discrimination. Without moral judgment. Without editorial oversight. It reproduces the beautiful patterns—the poetry, the philosophy, the scientific insight—and the ugly patterns—the slurs, the stereotypes, the casual dehumanisation. It does not distinguish between them. It cannot distinguish between them. The distinction is not in the mathematics. The distinction is in us.

This creates a profound tension. We want the model to be fluent. We want it to capture the full richness of human language. We want it to understand context, tone, register, idiom. But the full richness of human language includes prejudice. The full context of human expression includes cruelty. The full register of human communication includes the language of oppression. You cannot have one without the other. You cannot train a model on the full corpus of human text and expect it to be free of the biases that are woven into that text. The biases are not contaminants. They are part of the signal. They are part of what the text says about the world. And the model, faithfully, accurately, probabilistically, reproduces them.

The attempts to fix this—to filter the training data, to add safety layers, to fine-tune the model on curated examples of non-biased language—are necessary and valuable. But they are also, in a sense, a form of censorship. They involve deciding which patterns are acceptable and which are not. They involve imposing a moral framework on a statistical system. And they are never complete. The bias is deep. It is in the structure of the language, in the grammar, in the vocabulary, in the associations between words. You can filter the most egregious examples. You can reduce the frequency of the most harmful outputs. But you cannot eliminate the bias without eliminating the language. And if you eliminate the language, you eliminate the model.

The Impossibility of a Clean Fix

This is the uncomfortable truth that the original text gestures towards and that we must confront directly. Hallucination and bias are not bugs to be fixed. They are features to be managed. They are inherent in the architecture. They are consequences of the training objective. They are the price of the capability.

You cannot have a model that is perfectly truthful without giving it access to a verified knowledge base and a mechanism for fact-checking. And even then, the model's generative capacity—its ability to produce novel text, to fill gaps, to extrapolate—will always carry the risk of hallucination. The very thing that makes the model creative is the thing that makes it unreliable. The ability to generate text that has never been written before is the ability to generate text that is not true. You cannot have one without the other.

You cannot have a model that is perfectly unbiased without training it on a perfectly unbiased corpus. And no such corpus exists. The text that humanity has produced is biased. It is biased because humanity is biased. It is biased because the structures of power, the histories of oppression, the assumptions of privilege, are encoded in the language. You can mitigate the bias. You can reduce its frequency. You can flag it, contextualise it, warn the user about it. But you cannot eliminate it. Not without eliminating the data. And without the data, there is no model.

This is not a counsel of despair. It is a counsel of realism. It means that the model must be used with eyes open. It means that the output must be verified, cross-checked, contextualised. It means that the model is a tool, not an oracle. A starting point, not a destination. A draft, not a final version. And it means that the responsibility for truth and fairness does not lie with the model. It lies with the user. With the institution. With the society that deploys the model and decides how its output is used.

What This Means for You

You are between nineteen and twenty-five. You are at university, or you have just left university, or you are entering the workforce. You will use these models. You are already using them. You use them to draft emails, to brainstorm ideas, to explain concepts you do not understand, to generate code, to translate text, to summarise papers. And you will continue to use them, because they are useful, and because not using them will put you at a disadvantage relative to those who do.

But you must use them with your eyes open. You must understand that the model does not know what is true. It knows what is probable. And probability, at the edges, is a poor substitute for truth. You must verify every citation. You must cross-check every fact. You must question every confident assertion. The model speaks with authority. It speaks with fluency. It speaks with the calm, measured tone of someone who knows what they are talking about. But it does not know. It is generating the most probable next tokens. And the most probable next tokens are sometimes wrong. Sometimes dangerously wrong. Sometimes wrong in ways that are difficult to detect without specialist knowledge.

You must also understand that the model carries the biases of the corpus. If you ask it to generate a job description, it may default to gendered language. If you ask it to describe a historical event, it may adopt the perspective of the dominant culture. If you ask it to summarise a political debate, it may reflect the biases of the publications that dominate the training data. These are not malicious outputs. They are probabilistic outputs. They are the most likely continuation given the patterns in the data. But they are not neutral. They are not objective. They carry the weight of the corpus. And if you use them uncritically, you reproduce that weight. You amplify that bias. You become part of the feedback loop.

This is your responsibility. Not the model's. Not the engineers'. Yours. You are the one who decides whether to verify the citation, to question the framing, to challenge the assumption. You are the one who decides whether to accept the output or to push back. You are the one who decides whether the model is a tool for thinking or a substitute for it. And that decision, made billions of times a day by billions of users, will shape the future of these systems more than any technical fix, any regulatory framework, any corporate policy.

The Pattern Is Perfect, the Referent Is Absent

Let us return to the image that anchors this section. The hallucinated citation. The perfect format. The plausible title. The non-existent author. The pattern is flawless. Every element is in its proper place. The typography is correct. The punctuation is correct. The structure matches every other citation in the corpus. The model has learned the form of a reference so thoroughly that it can generate one that does not correspond to any actual publication. The signifier is perfect. The signified is absent. The pattern is complete. The referent is missing.

This is, in a sense, the deepest lesson of the large language model. The model is a machine for producing patterns. It is extraordinarily good at it. It produces patterns that are more fluent, more coherent, more grammatically perfect than most human writing. And those patterns can refer to the world, or they can refer to nothing. The model does not know the difference. The model has never seen the world. It has only seen the text. And the text, sometimes, refers to things that are not there. The model reproduces the reference. The reference is empty. The pattern is perfect. The referent is absent.

And bias is the same. The pattern is perfect. The association is strong. The co-occurrence is statistically robust. The model reproduces it faithfully. But the pattern encodes a prejudice. The association reflects an injustice. The co-occurrence is the residue of centuries of oppression. The model does not know this. The model cannot know this. The model knows only the pattern. And the pattern, reproduced without context, without critique, without moral judgment, perpetuates the very structures that produced it.

The pattern is perfect. The referent is absent. The pattern is perfect. The justice is absent. The model does what it was built to do. It reproduces the patterns. And the patterns are ours. They are the patterns of our language, our history, our culture, our failures. The model holds up a mirror. And in the mirror, we see ourselves. Not as we wish to be. As we are. As we have been. As we are encoded, token by token, in the text we have left behind.

The achievement is not diminished. The model is extraordinary. The capability is real. The fluency is genuine. The knowledge, such as it is, is vast. But the limitations are also real. They are also genuine. They are also vast. And they are not bugs. They are features. They are the price of the architecture. They are the consequence of the objective. They are what happens when you build a system to reproduce patterns and then ask it to deliver truth. It will try. It will succeed, most of the time. But at the edges, in the gaps, in the rare and the novel and the contested, it will fail. And the failure will look like success. And the pattern will be perfect. And the referent will be absent.

You will use these models. You must use them. But use them with your eyes open. Use them with your critical faculties engaged. Use them as a starting point, not a destination. Use them as a draft, not a final version. Use them as a tool for thinking, not a substitute for it. And when the model speaks with confidence, with fluency, with the calm authority of the probable—pause. Check. Verify. Question. The pattern is perfect. But the referent may be absent. And it is up to you to notice.

 Bibliography

  1. *Bender, E. M., Gebru, T., McMillan-Major, A. and Shmitchell, S. (2021) 'On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? ', in Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (FAccT '21)*. New York: Association for Computing Machinery, pp. 610–623. doi: 10.1145/3442188.3445922.

  2. *Blodgett, S. L., Barocas, S., Daumé III, H. and Wallach, H. (2020) 'Language (Technology) is Power: A Critical Survey of "Bias" in NLP', in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*. Online: Association for Computational Linguistics, pp. 5454–5476. doi: 10.18653/v1/2020.acl-main.485.

  3. *Bolukbasi, T., Chang, K.-W., Zou, J. Y., Saligrama, V. and Kalai, A. T. (2016) 'Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings', in Advances in Neural Information Processing Systems 29 (NIPS 2016)*. Barcelona: Curran Associates, pp. 4349–4357.

  4. *Dhuliawala, S., Komeili, M., Xu, J., Raileanu, R., Li, X., de Melo, A. and Weston, J. (2023) 'Chain-of-Verification Reduces Hallucination in Large Language Models', arXiv preprint*, arXiv:2309.11495. Available at: https://arxiv.org/abs/2309.11495 (Accessed: 8 September 2026).

  5. *Gehman, S., Gururangan, S., Sap, M., Choi, Y. and Smith, N. A. (2020) 'RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models', in Findings of the Association for Computational Linguistics: EMNLP 2020*. Online: Association for Computational Linguistics, pp. 3356–3369. doi: 10.18653/v1/2020.findings-emnlp.301.

  6. *Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A. and Fung, P. (2023) 'Survey of Hallucination in Large Language Models', ACM Computing Surveys*, 55(12), pp. 1–38. doi: 10.1145/3571730.

  7. *Kiritchenko, S., Wang, X. and Mohammad, S. M. (2021) 'Understanding the Effects of Annotation Style on Classifying Workplace Incivility', arXiv preprint*, arXiv:2105.07776.

  8. *Kirk, H. R., Vidgen, B., Röttger, P., Thrush, T., Hale, S. A. and Tetreault, J. (2023) 'Understanding the Harms of Large Language Models', arXiv preprint*, arXiv:2308.04374. Available at: https://arxiv.org/abs/2308.04374 (Accessed: 8 September 2026).

  9. *Lin, S., Hilton, J. and Evans, O. (2022) 'TruthfulQA: Measuring How Models Mimic Human Falsehoods', in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*. Dublin: Association for Computational Linguistics, pp. 3214–3252. doi: 10.18653/v1/2022.acl-long.229.

  10. *Menon, A. K. and Vondrick, C. (2023) 'Visual Concept Segmentation via Language-Guided Self-Supervision', in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. Vancouver: IEEE, pp. 12034–12043.

  11. O'Neil, C. (2016)Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy. London: Allen Lane.

  12. *Parrish, A. (2022) 'The Corpus is Political: Harmful Content in Corpora Used for Language Model Training', arXiv preprint*, arXiv:2203.07960. Available at: https://arxiv.org/abs/2203.07960 (Accessed: 8 September 2026).

  13. *Rogers, A., Gardner, M. and Rumshisky, A. (2020) 'A Primer in BERTology: What We Know about How BERT Works', Transactions of the Association for Computational Linguistics*, 8, pp. 842–866. doi: 10.1162/tacl_a_00349.

  14. *Sap, M., Card, D., Gabriel, S., Choi, Y. and Smith, N. A. (2019) 'The Risk of Racial Bias in Hate Speech Detection', in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics*. Florence: Association for Computational Linguistics, pp. 1668–1678. doi: 10.18653/v1/P19-1163.

  15. *Sheng, E., Chang, K.-W., Natarajan, P. and Peng, N. (2019) 'The Woman Worked as a Babysitter: On Biases in Language Generation', in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)*. Hong Kong: Association for Computational Linguistics, pp. 3407–3412. doi: 10.18653/v1/D19-1339.

  16. *Solaiman, I., Brundage, M., Clark, J., Askell, A., Radford, A., Wu, J., Krueger, G. and McCandlish, S. (2019) 'Release Strategies and the Social Impacts of Language Models', arXiv preprint*, arXiv:1908.09203. Available at: https://arxiv.org/abs/1908.09203 (Accessed: 8 September 2026).

  17. *Suresh, H. and Guttag, J. V. (2021) 'A Framework for Understanding Sources of Harm across the Machine Learning System Lifecycle', in Proceedings of the 2021 ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization*. New York: Association for Computing Machinery, pp. 1–9. doi: 10.1145/3465416.3483299.

  18. *Tononi, G. and Koch, C. (2015) 'Consciousness: Here, There and Everywhere?', Philosophical Transactions of the Royal Society B: Biological Sciences*, 370(1668), 20140167. doi: 10.1098/rstb.2014.0167.

  19. *Weidinger, L., Mellor, J., Rauh, M., Griffin, C., Uesato, J., Huang, P.-S., Cheng, M., Glaese, M., Balle, B., Kasirzadeh, A., Kenton, Z., Brown, S., Hawkins, W., Stepleton, T., Biles, C., Birhane, A., Haas, J., Rimell, L., Hendricks, L. A., Isaac, W., Fernandez, S., Howard, S. and Gabriel, I. (2021) 'Ethical and Social Risks of Harm from Language Models', arXiv preprint*, arXiv:2112.04359. Available at: https://arxiv.org/abs/2112.04359 (Accessed: 8 September 2026).

  20. *Zellers, R., Holtzman, A., Rashkin, H., Bisk, Y., Farhadi, A., Roesner, F. and Choi, Y. (2019) 'Defending Against Neural Fake News', in Advances in Neural Information Processing Systems 32 (NeurIPS 2019)*. Vancouver: Curran Associates, pp. 9051–9062.