The methodological quality of COVID-19 systematic reviews is low, except for Cochrane reviews: a meta-epidemiological study

Objectives

The objective of this study was to investigate the methodological quality of COVID-19 systematic reviews (SRs) indexed in medRxiv and PubMed, compared with Cochrane COVID Reviews.

Study Design and Setting

This is a cross-sectional meta-epidemiological study. We searched medRxiv, PubMed, and Cochrane Database of Systematic Reviews for SRs of COVID-19. We evaluated the methodological quality using A MeaSurement Tool to Assess systematic Reviews (AMSTAR) checklists. The maximum AMSTAR score is 11, and minimum is 0. Higher score means better quality.

Results

We included 9 Cochrane reviews as well as randomly selected 100 non-Cochrane reviews in medRxiv and PubMed. Compared with Cochrane reviews (mean 9.33, standard deviation 1.32), the mean AMSTAR scores of the articles in medRxiv were lower (mean difference -2.85, 95%confidence intervals (CI): -0.96 to -4.74) and those in PubMed was also lower (mean difference -3.28, 95% CI: -1.40 to -5.15), with no difference between the latter two.

Conclusions

It should be noted that AMSTAR is not a perfect tool of assessing quality SRs other than intervention. Readers should pay attention to the potentially low methodological quality of COVID-19 SRs in both PubMed and medRxiv but less so in Cochrane COVID reviews.

PROTOCOL AND REGISTRATION

We developed the protocol before conducting this study (Kataoka Y, Oide S, Ariie T, Tsujimoto Y, Furukawa TA. Quality of COVID-19 research in preprints: a meta-epidemiological study protocol. Protocols.io 2020. <ext-link xmlns:xlink="http://www.w3.org/1999/xlink" ext-link-type="uri" xlink:href="https://doi.org/10.17504/protocols.io.bhm8j49w">https://doi.org/10.17504/protocols.io.bhm8j49w</ext-link> .).

What is new?

Key findings

• <label>-</label>

  The methodological quality of COVID-19 systematic reviews (SRs) in medRxiv and PubMed were lower than Cochrane COVID reviews.

• <label>-</label>

  The methodological quality of reviews in medRxiv and PubMed did not differ.

What this study adds to what was known

• <label>-</label>

  Expert opinions and a preliminary review suggested the low quality of COVID-19 SRs but this hypothesis has not been examined empirically.

• <label>-</label>

  We evaluated the methodological quality of COVID-19 SRs using comprehensive search and confirmed that the quality was low except for Cochrane reviews.

What is the implication and what should change now?

Readers should pay attention to the potentially low methodological quality of COVID-19 SRs in both PubMed and medRxiv but less so in Cochrane COVID reviews.

The methodological quality of COVID-19 SRs except for Cochrane COVID reviews needed to be improved.