‘Conditional Acceptance’ (additional experiments required): A scoping review of recent evidence on key aspects of Open Peer Review

Diverse efforts are underway to reform the journal peer review system. Combined with growing interest in Open Science practices, Open Peer Review (OPR) has become of central concern to the scholarly community. However, what Open Peer Review is understood to encompass and how effective some of its elements are in meeting the expectations of the peer review system, are uncertain. This scoping review updates previous efforts to summarise research on OPR to date. Following the PRISMA methodological framework, it addresses the question: “What evidence has been reported in the scientific literature from 2017 to date regarding uptake, attitudes, and efficacy of two key aspects of Open Peer Review (Open Identities and Open Reports)?” The review identifies, analyses and synthesises 52 studies matching inclusion criteria, finding that OPR is growing, but still far from common practice. Our findings indicate positive attitudes towards Open Reports and more sceptical approaches to Open Identities. Changes in reviewer behaviour seem limited. and no evidence for lower acceptance rates of review invitations or slower turnaround times is reported. Concerns about power dynamics and potential backfiring on critical reviews are in need of further experimentation. We conclude that elements of OPR seem to be gaining acceptance, but more experimentation is needed. Evidence still mainly consists of either survey data or case studies of individual or few journals, not allowing for generalisability across fields and journals, and revealing no studies which compare the quality of review under Open Identities or Open Reports versus other modes of peer review.