Who's afraid of zombies? Watch out for those 'hollow shells' in clinical trials

The field of medicine is always hungry for new insights from clinical trials. But a significant percentage—at least 25% in some fields—of studies published in medical journals are based on either fake or flawed data, according to a recent feature in Nature. That’s worrisome, especially given that physicians often refer to these studies when making treatment decisions. “Faked or unreliable (randomized clinical trials) are a particularly dangerous threat,” read an article in Nature. “They not only are about medical interventions, but also can be laundered into respectability by being included in meta-analyses and systematic reviews, which thoroughly comb the literature to assess evidence for clinical treatments. Medical guidelines often cite such assessments, and physicians look to them when deciding how to treat patients.” Anesthetist John Carlisle, who works for England’s National Health Service and edits the journal Anaesthesia, has been investigating this problem for years and has coined a term to describe fake or flawed studies: zombie trials. According to Carlisle, zombie trials appear to be real research but turn out to be hollow shells when inspected closely. In October 2020, after three years of scrutinizing more than 500 manuscripts that reported a randomized controlled trial, Carlisle found that close to half of about 150 trials with accessible raw data contained flawed data such as impossible statistics, incorrect calculations or duplicated numbers or figures. More than a quarter of these papers were “impossible to trust” either because the authors were incompetent or because they had faked the data. “I anticipated maybe one in 10” of the studies would have problems, said Carlisle. “I think journals should assume that all submitted papers are potentially flawed and editors should review individual patient data before publishing randomized controlled trials.” Read: Recent resignations suggest industry has too much sway with Health Canada: NDP Carlisle isn’t the only one investigating problematic trials. Other scientists, physicians and data sleuths have insisted for years that fake or unreliable trails are frighteningly widespread and cut across virtually all fields. Some even argue that the problem has been underestimated. “If you search for all randomized trials on a topic, about a third of the trials will be fabricated,” said Ian Roberts, an epidemiologist at the London School of Hygiene and Tropical Medicine in London, U.K. Ben Mol, an obstetrics and gynecology specialist at Monash University in Melbourne, Australia, said he believes as much as 30% of randomized controlled trials included in systematic reviews in women’s health are suspect. Along with a group of colleagues, Mol has called out problems in more than 800 published medical research papers and successfully pushed for more than 80 retractions and 50 expressions of concerns. Many of the questionable trials have come from countries in the Middle East—particularly Egypt—and Mol said numerous authors from these countries have refused his requests for raw data. As an example of how problematic trials might have influenced clinical decisions, Mol cited a 2018 review, published in the Cochrane Database of Systemic Reviews, on the practice of administering steroids to pregnant patients scheduled for a Caesarian section, as a way to help reduce breathing problems in their babies. The review looked at four randomized controlled trials—a British study from 2005 with more than 940 participants and three Egyptian trials from 2015 and 2018 with 3,000 participants—and concluded that the steroids may reduce rates of breathing problems among infants. The review was subsequently cited in more than 200 documents and some clinical guidelines, although the World Health Organization recommends against steroids ahead of a C-section. When Mol and other researchers looked more closely at the papers, they spotted some discrepancies. The largest study from Egypt, for example, was based on a thesis for which trial end dates differed from the dates cited in the paper. At the same time, the reported ratio of 40% male babies to 60% female babies was an impossible statistic, Mol also analyzed the other papers in the review and reached out to the authors. Their replies were not satisfactory, with one author saying he lost the research data during a move from one residence to another. The review was updated three years later with a new screening protocol and without the suspect studies from Egypt. The outcome this time around? With only the British trial to draw from, the reviewers found there was “insufficient data” to support a conclusion. About six months after this updated review, one of the Egyptian trials—the largest one, with 1,300 participants—was retracted, with the journal’s editors stating that “if the data is unreliable, women and babies are being harmed.” The other two trials are under investigation by publisher Taylor & Francis as part of a larger review of several papers. Data flaws in randomized controlled trials for ivermectin as a treatment for COVID-19 have also come to light, with an updated Cochrane review concluding that more than 40% of these trials were not trustworthy. Read: Groups want timely, accessible data on cancer to help improve outcomes “Asking for raw data would be a good policy” for medical journals, said Jack Wilkinson, a health researcher at the University of Manchester in the United Kingdom and one of a growing group of researchers working to establish checks and parameters for clinical trials. Wilkinson leads a project that’s applying 76 trustworthiness checks—such as detailed examination of trial data, statistics, study methods and authors’ publication records as well as funding grants and trial registration—to all trials cited in 50 published Cochrane reviews. The objective of the project is to see how many randomized controlled trials fail the checks and how the reviews’ conclusions might be shaped by removal of these flawed trials. Wilkinson’s ultimate goal, however, is to produce a general trustworthiness screening tool along with a tool for inspecting participant data. “The default position has just been to trust the study, but we’ve been operating from quite a naive position,” he said. In recent years, some review authors have started adopting a Cochrane screening protocol—developed by a Cochrane specialist group on pregnancy and childbirth—with a detailed list of criteria that includes whether a trial is prospectively registered or if the study of free of unusual statistics such as implausibly narrow or wide distribution of mean values in certain participant characteristics. Based on this protocol, reviewers are supposed to contact the original study authors if the trials fail the checks and to exclude the study if the authors’ responses are not adequate. In one review of drugs designed to prevent pre-term labour, the application of this screening protocol resulted in the exclusion of 44 studies, or one-quarter of the 122 trials in the literature. Lisa Bero, a senior research integrity editor at Cochrane and a bioethicist at the University of Colorado Anschutz Medical Campus in Aurora, said a big challenge is the lack of consensus on what should be checked to determine if research is untrustworthy. “Misclassification of a genuine study as problematic could result in erroneous review conclusions,” Bero and two other researchers wrote in an editorial. “Misclassification could also lead to reputational damage to authors, legal consequences, and ethical issues associated with participants having taken part in research, only for it to be discounted.” Among the multiple screening protocols currently in play, some are more labour-intensive than others. “We need to convince systematic reviewers that this is worth their time,” said Bero. The International Committee of Medical Journal Editors had proposed mandatory data-sharing from randomized controlled trials. But concerns over perceived risks to the privacy of trial participants and the availability of resources for archiving the data led the association to water down its recommendations to merely encouraging data sharing. But at some journals—including Anaesthesia, where Carlisle is editor—sharing raw data is a requirement. Most authors comply, said Carlisle. Read: A ‘game-changing moment’: using real-world evidence to approve drugs

Who's afraid of zombies? Watch out for those 'hollow shells' in clinical trials

Register for free access or login.

Related Topics