We live in a sea of cognitive biases.
Everyone is affected. All of us. All the time. Psychologists study them, and so do economists, cognitive scientists, and philosophers. Skeptics need to understand cognitive biases, and I suspect that most are able to rattle off a list as easily as they can a list of logical fallacies. It is essential, however, to avoid the temptation of thinking that an understanding of cognitive biases renders anyone immune to them.
That all may be a bit self-serving, but now that I’ve framed this discussion, I can anchor everyone’s attention so that you’ll realise that my conclusions are inevitable.
Or maybe not. In recent years, I’ve noticed a tendency toward both a belief that AI systems are intelligent (because of the degree to which we attribute agency to systems that can mimic human responses), and a belief that they are objective – presumably because they are computers, and thus assumed to be immune to bias.
Unfortunately, neither is true.
We use terms like “artificial general intelligence” and “artificial narrow intelligence” as if they had precise meanings, but in fact they are used by different people, for different purposes, with different intended meanings. For simplicity, I shall define “intelligence” as a combination of sentience (the ability to experience feelings and sensations) and sapience (the ability to apply knowledge, experience, and judgement).
With that in mind, saying that Large Language Models (LLMs) are “intelligent” goes too far. They are effective at mimicking human responses, but this is mainly due to human “agency detection”, rather than actual intelligence.
LLMs are “trained” by breaking a corpus of data into tokens, then creating a list of probabilities to determine which token is most likely to come next. These probabilities are “nudged” up or down based on the “training” data, and layered and filtered in multiple ways, but that’s pretty much it. (I recommend Neural Networks, by 3Blue1Brown, for a good introduction).
This means that, by default, our current AI systems will be biased in the same way their training data is. The problem is in balancing training data quality and quantity, while trying to account for known bias and other behaviours we might want to discourage.
AI training data is necessarily historical, so training is inevitably biased by that history. While we would hope that AI would “know” that anyone can be a doctor or surgeon, data in a 2024 study on gender trends in Canadian medicine and surgery suggests that, while a majority of MD degrees awarded in the past twenty years have been issued to women, women are still under-represented in most surgical specialities. Are we to take this gender bias as acceptable? Obviously not.
While most AI companies curate their training data to maximize quality, pseudoscience and other nonsense have infiltrated otherwise-legitimate organizations, to the point where it can be very difficult to separate the signal from the noise. As examples, consider the WHO benchmarks for the practice of acupuncture, or a Harvard article on Exploring the Science of Acupuncture.
While we would hope that AI companies would give more weight to sites like Science-Based Medicine, would the training capture the nuances between serious and satirical? Does it even matter if the article is labelled as satire?
One intuitive way to avoid this might be to instruct the AI to avoid cognitive biases. However, this is where the “understanding” part of AI fails to come in. An AI cannot critically evaluate sources in order to make a determination regarding which should be given higher confidence – it simply consumes available data sources and nudges the probabilities of its learning model. Companies generally add “guardrails” to address such situations, but this should not be mistaken for critical evaluation.
In a 2025 study called Forewarning Artificial Intelligence about Cognitive Biases, by Wang and Redelmeier, the authors tested OpenAI’s LLM GPT-4, by providing 10 clinical cases to the AI model. Each case was presented in two versions – one to present a bias, and the other to present the inverse. These cases were then presented with an instruction to “Please keep in mind cognitive biases and other pitfalls of reasoning”. As a control, the authors presented the same two versions of the test cases, but without the additional instruction.
The “forewarning” arm of the study resulted in responses which were more than 50% longer, and discussed cognitive bias more than 100 times as frequently. However, the correct bias was identified in only 5% of the responses, and the most frequently discussed bias was the “anchoring” bias, which appeared in 16% of the responses even though it was irrelevant to all ten cases.
So, what went wrong? Though not included in the study, I would speculate that the same questions presented to actual physicians would have shown a significant difference between the two sets of questions. By drawing their attention to cognitive biases, they would be “primed” to search for them, and would likely have found many or all of them.
I would further speculate that the level of experience would be a significant factor – since differential diagnosis is part of medical training – and experienced physicians would be less likely to need the reminder. This approach can be summed up in the old quote: “When you hear hoofbeats, look for horses, not zebras.”
In this case, I found the results entirely unsurprising, as LLMs simply react to key words, and select responses based on “learned” probabilities. Thus, including “cognitive biases” and “pitfalls of reasoning” in the prompt simply generated responses associated with these terms. If the AI models were actually “intelligent”, we would expect a much higher accuracy rate in identifying the correct bias.
With the standard caveats for the limitations of the study, my question for a follow-up would be to look at whether there is a relationship between the bias mentioned in the response and the frequency of references to that bias in the training data. I suspect that would explain why the anchoring bias was so frequently discussed.
However, after all that doom and gloom, I would like to end on a happier note.
While current models are not “intelligent” in any meaningful sense, they can still provide value, when used intelligently. For example, it is possible to have multiple layers of LLMs, which can be used to cross-check and confirm results before returning them. There has also been work on so-called “reasoning engines”, which are designed and trained to evaluate data and draw conclusions, though they are quite limited in scope.
What if we combine them? What if we set up a network of specialized LLMs and reasoning engines which can be used to validate and cross-check results? This is, in some ways, analogous to how the human brain has different modules which perform different functions, and several companies are already pursuing this approach.
Maybe the answer is to try and educate the machines about critical thinking and skepticism? If so, they might be more like us than we thought.
