Investigators who run clinical trials testing the effectiveness of medical or surgical interventions are usually pleased when they manage to generate a positive result. Such findings are more satisfying, more productive, easier to publish and often a good basis for attracting further research funds.
Yet, clinical trials cannot be expected to always produce a positive result. The reason is simple: not all treatments that are being tested are effective; many are not and will thus be cast aside.
This is true for any type of medicine, but it is particularly true for therapies that lack a solidly rational basis, such as those abundantly found in the realm of so-called alternative medicine. One would therefore expect that investigators in this area are particularly often disappointed by generating clinical trials with negative results.
As obvious and logical this assumption might seem, it is not supported by the actual evidence. Researchers in CAM are not disappointed, because they rarely produce negative findings. I am often struck by how many researchers of CAM manage to publish one positive study of their favourite treatment after another. This phenomenon has fascinated me for many years, therefore I created, some time ago, something that I satirically named the ‘Alternative Medicine Hall of Fame’ (AMHF).
The AMHF is an illustrious group of CAM researchers who have managed to go through their entire career publishing nothing but positive results related to their favourite CAM. So far, this group consists of 27 mostly well-known people (name of individual, pet therapy, country):
- Miek C Jong (homeopathy, Holland/Norway)
- Josef M. Schmid (homeopathy, Germany)
- Meinhard Simon (homeopathy, Germany)
- Richard C. Niemtzow (acupuncture, US)
- Helmut Kiene (anthroposophical medicine, Germany)
- Helge Franke (osteopathy, Germany)
- Tery Oleson (acupressure , US)
- Jorge Vas (acupuncture, Spain)
- Wane Jonas (homeopathy, US)
- Harald Walach (various CAMs, Germany)
- Andreas Michalsen ( various CAMs, Germany)
- Jennifer Jacobs (homeopath, US)
- Jenise Pellow (homeopath, South Africa)
- Adrian White (acupuncturist, UK)
- Michael Frass (homeopath, Austria)
- Jens Behnke (research officer, Germany)
- John Weeks (editor of JCAM, US)
- Deepak Chopra (entrepreneur, US)
- Cheryl Hawk (chiropractor, US)
- David Peters (osteopathy, homeopathy, UK)
- Nicola Robinson (TCM, UK)
- Peter Fisher (homeopathy, UK)
- Simon Mills (herbal medicine, UK)
- Gustav Dobos (various CAMs, Germany)
- Claudia Witt (homeopathy, Germany/Switzerland)
- George Lewith (acupuncture, UK)
- John Licciardone (osteopathy, US)
What these people have in common is that they have published plenty of papers on their favourite CAM without ever producing a negative finding. For a researcher in homeopathy, for example, this means that, in his or her hands homeopathy works for whatever disease or symptom it is being tested.
It also means that our alarm bells should start ringing, and we might ask ourselves: how trustworthy are the results of the research in question?
In an attempt to quantify this research trustworthiness, I suggested already in 2012 to calculate the amount of a researcher’s papers arriving at positive conclusions and divide it by the amount of his or her papers drawing negative conclusions. This index would be low for strongly trustworthy research, and high for those whose conclusions are worthy of more scrutiny.
A realistic example might be the case of a clinical scientist who has published a total of 100 original articles. If 50 had positive and 50 negative conclusions about the efficacy of the therapy tested, his or her ‘research trustworthy index’ (RTI) would be one; if 90 had positive and 10 negative findings, their RTI would be as high as nine.
Depending on the area of clinical medicine in question, an RTI between one and four might be acceptable. Particularly for CAM, an RTI higher than four should probably invite further examination.
And this is where the members of my AMHF are so very special. Their RTI would invariably consist of a finite number (the number of positive studies) divided by zero (the number of negative studies). That means their RTI is infinite which in turn suggests the trustworthiness of the conclusions of the research would amount to zero.
Has the RTI been a success? Has it been widely adopted in CAM?
No.
I wonder why.
