Systematic Reviews in Science: Navigating the Minefield of Research Integrity

In the latest Lund Biomedicine ReproducibiliTea Journal Club, Vera Casslén, PhD led a discussion on the challenge that face systematic reviews in scientific research, based on Holly Else's "Fake papers compromise research syntheses" (1). The session focused on growing concerns about the effect of untrustworthy studies on the reliability of conclusions that can be drawn from scientific literature.

Key Takeaways on Systematic Reviews

The discussion revealed five key findings that challenge our understanding of scientific research synthesis:

1. Different types of reviews

There are 2 types of reviews that synthesize data, each serving a different purpose.

Systematic reviews are an important tool for evidence-based medical practice, seeking to answer a precise, specific question. They are frequently clinical, but some may also examine pre- or nonclinical research. These reviews must be exhaustive and accurate since their findings directly impact patients, for example by shaping treatment guidelines or informing further medical research.

Literature reviews are a tool for scientific synthesis. They seek to answer a broad research question and aim to provide a literature review with an ultimate goal of generating new disruptive ideas and concepts in science. Evolution theory is one example of a disruptive concept generated by means of scientific synthesis.

Literature reviews are increasingly being called “systematic literature reviews” where the term “systematic” recognizes the value of being methodical and minimizing bias in the provided summary.

2. Unreliable Studies proliferate

Both types of reviews suffer from compromised and untrustworthy studies – some of which contain completely fake data. The proportion of published junk papers is currently high enough to significantly skew the literature. The inevitable consequence of this phenomenon the erosion of public trust in science.

3. Unstandardized terminology complicates things further

Despite the importance of identifying untrustworthy studies, there is a lack of assessment criteria for what is considered a fake study. No clear definitions exist, and the terminology remains unstandardized, with studies often being described with different terms, including junk, flawed, fake, falsified, fabricated, manipulated, untrustworthy, or having hallmarks of fabrication.

An interesting suggestion emerged from our discussion: Perhaps a “Risk of Fakeness” Assessment, with clearly defined criteria for what constitutes a study at high risk of being fake, should be introduced. This practice would be similar to the Risk of Bias assessments that already form a key part of the systematic review process.

4. No strategy exists to tackle published junk studies

All current efforts aim to prevent research malpractice from happening in the future. The scientific community has no strategy for how to handle junk studies that are already published!

5. A Potential Solution: Curated Research Libraries

The Cochrane Library comprises pre-filtered, carefully and critically selected references. It is an accessible source of credible medical information. Perhaps the Cochrane Library could inspire the creation of a new library containing credible research references for clinical and pre- or nonclinical research.

Join the Conversation

Our upcoming journal club meetings will continue exploring critical research questions. If you are a researcher, student, or science enthusiast, join us to engage in meaningful dialogue about improving scientific methods and research integrity.

Register your interest to receive updates about upcoming ReproducibiliTea Journal Club meetings, including our upcoming session on good research practice (March 19th).

References

1. Else H. Fake papers compromise research syntheses. Science. 2024 Nov 29;386(6725):955. doi: 10.1126/science.adu8281.

   
   

Speaker and guest author

Vera Casslén holds a PhD in translational medicine with expertise in discovery, in vitro preclinical research, and early drug development. Vera is passionate about discovery, in vitro preclinical research, early preclinical and nonclinical development, regulatory affairs and clinical trials, and competence training in science.

Editor

Rebeca Cardoso, PhD, is a Research Consultant at AdvanSci Research Solutions. Committed to making science accessible, Rebeca strives to bring cutting-edge research to a broader audience through her writing and outreach initiatives.