Today I wanted to discuss how we see the responsibility of reviewers in science. Peer-review is an integral component of the scientific community. The ability to evaluate and critique the work of others is vitally important in science. I’ve sort of touched on this before when talking about skepticism in science. You can think of a scientific article as an argument that someone is presenting. They need to present the argument as a hypothesis and to defend that hypothesis with their data. At the same time, those reviewing the argument and the data need to be qualified enough to give an informative critique; they need to be peers. I’ll describe what I see as the “modern” style of peer review and then move on to what I see as the “new” style.
The “modern” style of peer review as I would call it has been in use since at least the mid-20th century. Scientific articles had been publish before this time, with editors themselves taking charge of reviewing manuscripts. In the 1950’s and 1960’s academia had grown to such a point where there was both a demand that colleagues be able to review papers in their field before publication and enough qualified people to efficiently outsource the review process (1). In this modern review process, the editor sends out the manuscript to individuals in the field. How these individuals are selected is still somewhat of a mystery to the public at large and, in my opinion, to most of the scientific community as well. Ideally, the reviewers are other scientists who have worked on the same topic or in the same scientific field. The reason for this could be twofold. For one you don’t need to provide as much background information to people already familiar with terms in the article under review, and for two you are much more likely to hold the attention and interests of people who can make connections to their own work.
There is, however, much of the criticism and concern surrounding modern peer-review. In modern peer-review the article authors are not supposed to know their reviewers. This sets up the perverse incentive for someone publishing a similar article to attack competing work unfairly or thwart it’s publication for their own interests. Alternatively, a collaborator might provide very kind review if the paper’s publication might lead to publications for themselves down the line. Most journals have mechanisms to prevent both of these from happening. When submitting an article you are allowed to list a few scientists who “you would not like to be considered for review”. The journal will also mention that you can submit “suggested reviewers” but they cannot be coauthors nor can they be individuals with whom you had professional dealing within the past 4-5 years (e.g. you didn’t write a grant together, you didn’t publish a paper together, you don’t co-advise students etc). These, of course, are just suggestions. The editor gets the final say on picking the reviewers, and they could send it to someone who will give a poor review no matter what. You will not know who the reviewers are during the process and you might never know who reviewed your article. I feel that this modern peer review style is becoming obsolete and that a new peer-review style is emerging.
There is a new style of peer-review emerging since the beginning of the 21st century: Open Review. This type of review has been gaining more mainstream acceptance with the rise of Open-access journals and sites like Publons, which track reviews and reviewer records (2). Under this type of review process everything is out in the open and accessible. All communications between the reviewers and the authors are known and even the identities of the reviewers are known. The content of the reviews will be available along with the article if it is accepted for publication. Additionally there are “post-publication” reviewer, which critique the article after it has been accepted. The goal of this type of review is to demystify the entire process and to facility better discussion about the scholarly merits of an article. The downsides are obvious: there is the threat of collusion, but some studies have shown that there is very little difference in the quality of reviews or the outcome when using an open peer review process as opposed to the more traditional process (3). By opening up the review process we can make the scientific merit of an article stand out even stronger and provide more confidence in the ability of the scientific community to promote good science.
The open peer-review process achieves what I believe should be the main goal of any reviewer: to help improve the scientific merit of an article. The traditional or modern way of thinking about publishing is that it is a battle. In this battle you have been “defeated” if you let a poor article be published so why not err on the side of caution and reject anything that seems even remotely suspect? While I see the value in being a “gate-keeper” as a reviewer I personally feel that stopping there is lazy. The reviewer is responsible for working with an author to improve their article and to strengthen the conclusion they have made about their hypothesis. It would be better to put yourself on the side of the author, to have high standards for yourself, and to ask what your experience and expertise can add to this article to make it a valuable contribution to the journal. With this in mind, open peer-review has to be the best method of fostering the author-reviewer relationship. As a society we seem to be moving ever more and more towards a need for openness and transparency in institutions as well. Having a clear, open review allows gives us confidence in the peer-review process and allows us to sniff out collusion or scientific misconduct. I would encourage everyone to take a look at some articles that have been reviewed under “open peer-review” and to decide for themselves what makes the most sense.
Thank you all for reading!
Van Rooyen S. et al. “Effect of open peer review on quality of reviews and on reviewers’ recommendations: a randomized trial” BMJ 1999;318:23