Dominic Cummings, one of the architects of Brexit, once told the BBC that the reason he used slogans like “we send £350 million a week to the EU” was because it’s an easy claim to make, but difficult to refute.
“They couldn’t just say, this is nonsense, this is not the case,” he told Laura Kuenssberg in July 2021, and he wasn’t wrong. The claim can’t be debunked because it’s true. It is also misleading, because that truth comes with a boat load of caveats and context that take a long time to explain and understand. These explanations lose many listeners once they hear, ‘That’s technically true, but…’
Readers may be familiar with my views on the placebo effect. I’ve spent many years on my podcast Skeptics with a K, and within the pages of The Skeptic, trying to explain why claims like ‘the placebo effect is real and powerful’ don’t hold up. But it is also difficult to explain, because of the required nuance and context. Which ‘placebo effect’? For what condition? What does ‘real’ mean? These distinctions need to be unpacked and understood.
So let’s go back to basics.
Testing treatments
If we have a new drug we want to test, a simple approach might be: give it to a patient and see if they get better. This is flawed, of course, because they may have gotten better for some other reason. So instead of recruiting just one person for the test, maybe we recruit a few. We also don’t give all of them the new drug, just half, so we can compare how many in each half get better.
If we only involve very small numbers of people, maybe just a few in each group, we are still vulnerable to one or two patients recovering by fluke and making the new drug look effective when it is not. So, ideally, we would have lots of people in each group, the idea being that if we have enough people, fluke recoveries will be distributed between the groups and average out.
If we can choose which patients go into which group, then we might be tempted to put particular people into the test group, because we’re convinced the drug will help them. But if we put just very ill people in one group and not so ill people in the other, then again it might look like the drug works better or worse than it actually does. So we should assign patients to the groups randomly.
And if the patients and researchers know which group is getting the real drug, that can change the results too. Researchers might be more or less likely to report changes because they know what should be happening. Patients might stick more or less rigidly to the treatment plan.
And so we should also ensure that no one knows which group is which until after the study is finished. One way we can do this is with placebos, fake pills which look like the real thing, but which have no known effect on the disease or condition we are studying.
In the ideal case then, we would have many patients, a control group, patients randomly assigned to the groups, and no one knows which group is which. This is the randomised placebo-controlled trial, or RCT, widely considered the gold standard of medical research. We work hard to ensure both groups do the same things, except for that one thing we’re trying to measure, because then we can be confident that any differences between the groups are caused by the new treatment.
What’s fascinating, and superficially unexpected, is that patients in both groups will improve. Patients who do not get the real drug really do get better. And this can result in some observers forgetting that the purpose of the control group is to give us a standard for comparison. Instead they come to believe that maybe something about the very idea of a treatment makes patients better.
So let’s take a moment to understand the reasons why someone in the control group, someone who does not get any real medicine, might record an improvement in their condition regardless.
Why do symptoms change?
How the disease works
In the first case, we have the natural history of the disease. Many conditions are self-limiting and clear up by themselves eventually. If we were to design a study to test the effects of Caramac bars on a cold, and we did a clinical trial, after two weeks, everyone in the Caramac group would have recovered. Not because of anything to do with the Caramacs, just because colds get better after a week or so. So, if we imagine a pie chart where the total area represents the improvement observed in the control group, a slice of that pie is going to be the natural history of the disease.
It’s not about placebos, or belief, or expectation. It’s just how some diseases work.
Conditions even out over time
Then we have regression toward the mean. Some of the improvement observed will be the result of the natural fluctuation of symptoms, especially in chronic conditions. Patients will typically have bad days and good days. The more bad days you have in a row, the greater the chance that you’re going to get a good day soon. The more good days you have in a row, the greater the chance you’re going to get a bad day soon. Over a long enough timeline, the condition always regresses to the average or mean state.
Crucially, patients will most commonly seek medical care during a flare-up, when their symptoms are at their peak. Which means that, in all likelihood, the very next thing to happen will be an improvement in their symptoms, simply due to regression. Again, this has nothing to do with placebos or the expectations of the patient, so it is another slice we can take from our pie chart.
People just get better
There is spontaneous improvement: sometimes patients just get better. The patient’s own immune system is a popular candidate for why, but sometimes this even happens in conditions where we don’t expect it to. Another slice to take out of the pie chart.
Simultaneous treatments
Then we have parallel interventions. One famous study, Beard 2018, investigated the effects of the removal of bone spurs and soft tissue, commonly promoted as a treatment for shoulder pain. Beard found the procedure was no more effective than placebo – but remarkably, the placebo (in this case, a fake operation) was superior to no treatment at all. This turns out to be because even the patients getting the fake operation were also given physiotherapy to help them recover from surgery, which could have improved their shoulder pain.
There are also unrecorded parallel interventions, which are far more insidious. This is where patients get some other treatment, but either don’t report it to the doctors running their trial, or if they do no one writes it down. It is much harder to know when this happens because, by definition, it is unrecorded – but one example might be Daniel Moerman’s 1983 paper on gastric ulcers.
Moerman reported that some studies showed a 90% cure rate for ulcers after the administration of a placebo, but other studies showed just a 10% cure rate. One possible cause for this wide variability might be because patients were inadvertently curing their ulcers with antibiotics they were taking for some unrelated reason. At the time, it was not understood that almost all gastric ulcers were caused by H. pylori bacteria, so patients’ antibiotic use was not recorded. Recorded and unrecorded parallel interventions are another slice we can take from the placebo effect pie.
We’re all biased
Then there are psychological effects. There are observer-expectancy effects, where desires and opinions of the researchers influence the recorded outcomes. Researchers are more likely to record the effects they expect to see and less likely to record effects they do not. Similarly, subject-expectancy effects are the same thing from the other side. Patients are more likely to notice and report effects they expect to see and are less likely to record effects they do not.
One example here is Blackwell 1972, where medical students were given pink or blue placebo pills and a list of effects they could expect to experience after taking them. So, of course, some reported those same effects! But would they have said anything if they hadn’t been told what to look for?
Crucially, these effects only change how the data is recorded, they do not change the actual condition of the patient. Another two slices to take out of the placebo pie.
Human behaviour is complex
Even just knowing they are taking part in a study is enough to change a patient’s behaviour. This is sometimes known as the Hawthorne Effect. Participants are aware that the things they do are being studied and recorded, so they may comply more tightly with standard care than they were before. They may eat a little better, or take a little more exercise, because they know that’s what they should be doing and the researchers are now checking up on them. One can easily imagine, for example, a patient in a study on asthma who is supposed to take a steroid-based inhaler daily, but is a little lax in taking it. Upon starting an asthma trial, they start taking it reliably because they know they should, and this can change their health outcomes. Another slice of the placebo pie gone.
Reporting biases mean that patients might be selective in what information they give to clinicians, perhaps through social pressure or embarrassment. A patient who regularly smokes ten cigarettes per day might report smoking only two or three because they are embarrassed by the true number. Over the course of the study, they cut down their smoking to match what they had initially reported. Patients may not regard this as a big deal, but any improvements resulting from that change in behaviour would appear in the data to be apparently spontaneous. The fact they changed their smoking habits during the study is not recorded anywhere, and if that patient happens to be in the placebo wing of a trial, this becomes data supporting a powerful placebo narrative. In fact, it was just a patient who was ashamed of their smoking. Another slice gone.
Some patients may exaggerate their symptoms at the start of the study, even unconsciously. Perhaps they’re at the end of their tether and trying to impress upon the doctors how difficult they’re finding things. There are also patients who will under-report their symptoms at the end of the study, because they think they should be better. They’ve taken the medicine, they don’t want to upset or annoy their doctor, so they report what they think they should.
That can even be in a small, subtle, way like reporting a five on a pain scale instead of a six. They may not even realise they’re doing it. But it’s another effect that makes it appear like a fake treatment is improving the patient’s condition.
Our memories are bad
Following on from this, there is a bias of recall. Patients who mis-remember how bad they felt at the start of the study, may report an exaggerated improvement at the end of the study. The patient may remember they had scored themselves as a six on the pain scale at the start, and under the belief they now feel better, report a five at the end. Except they actually don’t feel better, they’re just misremembering how they felt before. So now they’re recorded in the data as an improvement, when actually nothing has changed.
Along similar lines is enrollment bias, where doctors themselves exaggerate or misrepresent a patient’s condition to ensure they qualify for a study. In one trial investigating benign prostatic hyperplasia, researchers determined that a minimum prostate volume of 30cc was required for enrollment. However, patients recorded by their doctor as having 30cc prostates were discovered to have volumes of 27-29cc when the first follow-up measurements were taken. This created an apparent placebo effect, as patients without the active treatment seemed to immediately improve at the start of the trial, before starting to slowly deteriorate. The improvement was entirely artificial, created by the difference between exaggerated baseline measurements and accurate follow-up.
In later studies, researchers introduced a one-month lead-in period for all patients before taking baseline measurements. This eliminated the enrollment bias, and this apparent placebo effect vanished. Patients never were getting better in the placebo group, the baseline data was simply wrong. But without understanding this bias the improvements appear to be evidence of a powerful placebo.
So where is the placebo effect?
None of these effects are actually clinically relevant. None of them represent a patient actually getting any better. They are unreliable, non-specific, and often illusory effects which we should be carefully controlling for, not celebrating as ‘the amazing power of placebo’. These effects fool with our data and interpreting them improperly will lead us to inaccurate conclusions.
If biases, measurement errors, conditioning, and other confounding effects were able to be independently controlled for, if we were able to measure and subtract all of them from our placebo pie chart… how much pie is left? That’s the million-dollar question. How much effect is there after we have taken out the boilerplate and biases and cruft?
How large is this ‘true’ placebo effect, the seemingly magical effect that results from your body reacting to your mind’s insistence that you have taken some medicine.
My guess is that the answer is zero. My guess is there’s no placebo pie left and those positing a true powerful placebo effect are effectively making a ‘placebo of the gaps’ argument. So when I say ‘the placebo effect is not real’, that is what I mean and I’ve yet to see any data that persuades me otherwise.
I would just find it hard to fit all that on the side of a bus.
