Ensuring an Accurate Scientific Record in an Era of Pre-print Servers

Background: Pre-prints have become an increasing part of the biomedical landscape. For example, during the first month of operation, July 2019, medRxiv received 176 submissions, one year later, in June 2020, including the first few months of COVID-19, it received 1866 submissions. The current relevant question is how to ensure an accurate scientific record given that there may be important differences between a pre-print and the peer-reviewed publication.

Methods: Based upon the experience of the authors, conversations with editors, and a focused selective review of the literature, including the recommendations of some professional groups, a limited number of practical recommendations were formulated.

Results: Peer-reviewed journals should request that authors indicate if the submitted manuscript has been posted on a pre-print server; ensure this is noted in the article if it is published by including the digital object identifier (DOI); and detail any major differences in the conclusions between the pre-print and the article. Pre-print servers should ensure that all content is marked as not peer-reviewed and be prepared to retract any pre-print that is fundamentally flawed within days that could influence clinical or public health recommendations that have therapeutic implications.

Conclusions: Authors, those responsible for pre-print servers, and editors of peer-reviewed journals are responsible for ensuring an accurate scientific record.

Pre-print servers are more than 3 decades old, although they are relatively new in biomedical research (1, 2). arXIV, which includes pre-prints in math, physics, and computer science, was founded in 1991 (3). In contrast, bioRxiv, which focuses on the biological sciences, was founded in 2013, and medRxiv which focuses on medicine and related health sciences, was founded in 2019 (4, 5). The growth of pre-print servers, particularly during the COVID-19 pandemic has been substantial. For example, over a one-year period the number of monthly submissions to medRxiv increased from 176 (July 2019) to 1866 (June 2020) (6). Estimates suggest that currently there are more than 40 preprint platforms that host biomedical research (7). Although some biomedical journals were reluctant to publish articles that were first hosted on a pre-print server, that has changed. Now most will consider manuscripts that have been posted on a pre-print server, and indeed some journals encourage the use of them. Similarly, several funders (e.g., National Institutes of Health) now support the inclusion of pre-print citations in grant applications. In many regards, pre-prints are part of the Open Science movement, which also includes open access, sharing of data, and transparent peer-review.

The potential of pre-print servers

Although there is continuing debate about the potential of pre-print servers to do harm because of inaccurate information quickly reaching the public prior to peer-review, many investigators now believe that the benefits outweigh the risks. In addition, they are better than a press release from a pharmaceutical company, research consortium, or hospital. In a recent survey of over 3500 researchers from various disciplines, a majority felt favorable toward preprints, including those investigators in medicine (8). And there is continuing concern that peer review may not be effective to screen out questionable research methods and reporting. Although pre-print servers initially served the academic community, allowing a form of open-review, rapid communication of results, and the claim of provenance of certain research findings, during the COVID-19 pandemic, the major media outlets have also reported on studies posted on pre-print servers to inform the public of the rapidly changing clinical and scientific landscape. In addition, in an era of open science, pre-prints servers offer more equitable and inclusive access to the medical literature. Regardless of any lingering concerns about pre-prints, they will remain part of the publishing landscape. The question that should be asked is – how can the entire biomedical publishing enterprise ensure an accurate scientific record?

Practical recommendations on pre-prints

Below are a limited number of practical recommendations, some of which extend the recommendations already suggested by groups such as the International Committee of Medical Journal Editors (9). These recommendations are practical and should not overburden authors, editors, journals, or those responsible for pre-print servers. The focus of these recommendations is on manuscripts reporting original research in the biomedical sciences that are expected to be submitted to a peer-reviewed biomedical journal, and not on pre-prints that are meant as working papers, conference proceedings, or are expected to undergo numerous revisions, as is often the case in some fields, such as economics, mathematics, or physics.

• Journals should request at the time of submission of a manuscript whether it has previously been posted on a pre-print server and, if so, that authors provide a copy of the preprint and cite the digital object identifier (DOI) in the manuscript. This would not be necessary for journals that have their own pre-print platform, or one linked to their journals, but would still be easier if authors provided this information. While this identifier is likely best placed in the Methods section of the article, journals differ widely in their formatting requirements. If the journal publishes the article, the pre-print DOI could be included alongside the published article’s DOI, although clearly marked as distinct from the DOI of the manuscript, and in addition included in the reference list.
• If the results and/or conclusions in the published peer-review manuscript are substantially different than in the pre-print, for example, statistical or clinical significance differs between the two manuscripts, or the pre-print has a different primary outcome, this should be acknowledged and clarified in the discussion section of the manuscript. Substantial differences between the pre-print and the submitted manuscript could be noted by either asking the peer-reviewers to review the pre-print, the editors review the pre-print, or most importantly, authors are asked to acknowledge any major differences at the time of submission. The editors and peer-reviewers could compare the pre-print to the accepted manuscript, focusing on the abstract, and on the primary outcome(s), results, and conclusions. Other major changes, such as in authorship or conflict of interest statements should also be explained.
• Pre-print servers should include a visible list of their policies, for example does the pre-print server review manuscripts prior to posting, as medRxiv does, and if so, what type of review is it (e.g., scientific, formatting) and are these criteria posted on the pre-print platform? Does the server review submissions for a statement regarding ethical conduct if the study involves human subjects or animals? Does the server require authors to include a statement about conflict of interests in the pre-print? Given the importance of conflict of interest, pre-print servers should require statements from each author.
• All pre-print servers should make it clear on every page (screen or downloaded document) that the preprint has not been peer reviewed. This could be easily included as a ‘header’ on each page of the download. Most pre-print servers already do this, and in fact, many traditional news organizations already state when quoting from a pre-print that the information has not been peer-reviewed. However, there is little consistency across social media outlets. In 2022 social media may have a stronger dissemination effect than traditional media. As such, social media should develop standards, for example simply indicating that this communication (“post:’) is based upon medical information that has not been peer-reviewed.
• Once a pre-print is officially posted on the server, this provides any individual with the opportunity to provide a meaningful critique. This is a very equitable practice and not limited to a small number of so-called ‘experts’ providing feedback. Such feedback sometimes results in revisions to the pre-print prior to submission to a peer-reviewed journal. In the rare instance when a pre-print is assessed to be fundamentally flawed it should be immediately retracted with a statement as to why it was retracted. Whereas retraction of manuscripts published in peer-reviewed journals can take months to years, because pre-prints are meant to speed the sharing of knowledge with scientists, clinicians and the public, this process must occur rapidly when these flaws emerge.

The last recommendation could be a challenge for individuals who are responsible for pre-print servers. But it is crucial, particularly for manuscripts that could influence therapeutic or diagnostic decisions or public health recommendations that if a pre-print is fundamentally flawed, as noted by a reader, and confirmed by individuals responsible for the server, it should be retracted quickly. How often this occurs is unknown. Pre-print servers could have a portal in which readers could contact them if they believe a manuscript contains misleading or incorrect information. At a time when belief in science appears to be waning, and the proliferation of misinformation appears to be increasing, this is more important than ever and is the responsibility of pre-print servers.

Long-term availability of pre-prints

There have been some suggestions that pre-prints be taken down after some period. Although the data are constantly evolving, it appears that about a third of preprints are not published within two to three years in peer-review journals (10). It is likely that a substantial percentage of these will never by published in a peer-reviewed journal. Authors of pre-prints should seek publication in peer-reviewed journals. Regardless, as open scholarship becomes a normative practice, pre-prints should not be purged and should have a permanent identifier. Twenty years ago, conducting a systematic review would typically involve searching only the published literature (e.g., Medline). Today, such a search might be considered inadequate. A search of pre-print servers is now likely to be important. Since high-quality systematic reviews include a risk of bias assessment of research reports, even if the pre-print has not been published in a peer-reviewed journal, an assessment of the study would be done.

Some have also suggested that if a pre-print is ultimately published, that should be noted on the pre-print server (e.g., including the DOI of the published article). Although this is admirable suggestion, the challenge would be in operationalizing this approach without overburdening pre-print servers. Authors could decide on their own, if a manuscript is ultimately published in a peer-review journal, to withdraw the manuscript from the pre-print server, but this would defeat the purpose of citing the pre-print in the published peer-reviewed manuscript.

Possible approaches for improvement

While pre-print servers do not offer formal peer review there are some steps they could take to help improve the pre-print product. For example, they could require authors of reports of completed randomized clinical trials to provide documentation that the CONSORT checklist was used, or for other study types, such as meta-analyses, that may influence clinical care or public health recommendations, that the appropriate reporting checklist was followed. The FAST principles are another example of providing feedback on preprints (11).

Conclusion

The discussion around pre-prints has changed. They have become part of the publishing landscape. We believe these practical recommendations do not over-burden authors, journals, journals, or those responsible for pre-print servers and will help ensure an accurate scientific record, which is the responsibility of all individuals and entities involved in the biomedical publishing enterprise.

Conflict of interest statement
The authors declare that no conflict of interests exists.

Manuscript received on 3 June 2022; revised version accepted on 12 July 2022

Corresponding author
Howard Bauchner, MD

Boston University School of Medicine

02118 Boston, USA

Cite this as:
Bauchner H, Moher D: Ensuring an accurate scientific record in an era of pre-print servers. Dtsch Arztebl Int 2022; 119: 675–8.
DOI: 10.3238/arztebl.m2022.0293

References

1. ASAPbio: What is a preprint? www.asapbio.org/preprint-info (last accessed on 25 February 2022).

2. Flanagin A, Fontanarosa PB, Bauchner H: Preprints involving medical research—Do the benefits outweigh the challenges? JAMA 2020; 324: 1840–3

3. Cornell University: arxiv. www.arxiv.org/ (last accessed on 25 February 2022).

4. Cold Spring Harbor Laboratory: bioRxiv. www.biorxiv.org/ (last accessed on 25 February 2022).

5. Cold Spring Harbor Laboratory: medRxiv. www.medrxiv.org/ (last accessed on 25 February 2022).

6. Krumholtz HM, Bloom T, Sever R, Rawlinson C, Inglis JR, Ross JS: Submissions and downloads of preprints in the first year of medRxiv. JAMA 2020; 324: 1903–5

7. Kirkham JJ, Penfold NC, Murphy F, et al.: Systematic examination of preprint platforms for use in the medical and biomedical sciences setting. BMJ Open 2020; 10: e041849

8. Soderberg CK, Errington TM, Nosek BA: Credibility of preprints: an interdisciplinary survey of researchers. R Soc Open Sci 2020; 7: 201520

9. ICMJE: Overlapping Publications. www.icmje.org/recommendations/browse/publishing-and-editorial-issues/overlapping-publications.html (last accessed on 25 May 2022).

10. Abdill RJ, Blekhman R: Tracking the popularity and outcomes of all bioRxiv preprints. Elife 2019; 8: e45133

11. Iborra SF, Polka J, Puebla I: Promoting constructive feedback on preprints with the FAST principles. Elife 2022; 11: e78424