Four Peer Review Problems and a Solution
I’m mainly writing this because I don’t quite have the time to implement the solution I propose, although I’ve done some preliminary work in that direction and welcome would-be contributors.
Here are three problems with peer review:
(1) It’s a (mild) hassle to submit a paper, requiring the navigation of (mostly) poorly designed online systems, the writing of otiose cover letters, and so on. Since most papers don’t get accepted at the first place they’re sent, this mild hassle is multiplied several times.
(2) Once the editor has received a paper, they are deprived of a very important piece of information about the paper, namely whether and by whom it’s been reviewed before and, if so, what the reviewers thought and how the author responded.
(3) A referee’s work is often in vain: people will submit, a referee will work hard to point out problems with the paper, the paper will get rejected, and the author will submit again elsewhere relatively confident that the referee will not review it again.
(4) It is hard to find referees.
I think a system aimed at solving (1), the least serious problem, could help with the other three problems: in brief, the proposal is a journal-neutral submission system (call it JNSS), a single front-end from which authors can submit and resubmit a given paper to different journals without having to navigate individual sites or redundantly enter the same information numerous times. Onto this front-end an editorial backside could be added, that contains information about the paper’s history and is accessible to only and all journal editors, enabling them to see who has already reviewed the paper and what those reviewers thought of it.
Here’s how it would work in more detail. You register with the system: name, email address, affiliation, etc. You upload a paper, including the vital paper data (name, area, abstract, key words, etc.) and then you seek to submit it. You select a journal from a list the JNSS provides (say in a pull-down menu). You are then asked to enter the journal specific information: most of this you can simply copy from the vital data you entered, but some you might have to enter manually. Having done this, with the help of some web-scraping library, the JNSS submits the paper automatically for you.
You wait. If the paper gets accepted or r&r, great. If not, you can use the system to submit it elsewhere, again taking advantage of the fact that most of the important information is already recorded in the system.
That’s how it works from the author’s side. It would be, I think, a mildly useful tool to have a singular submission portal, but no more than mildly useful.
The real interestingness of this system arises from a number of helpful changes it could effect. Recall problem (4): it’s hard to find referees. Anecdotally, some people get asked a lot and some don’t get asked at all. And the problem is just that the tools available to find referees (bibliographies, philpapers, google scholar, acquaintances), while useful, are limited. We could make it a requirement that, before resubmitting a rejected paper, one is automatically put forward to review a paper. One gets added to a pool of referees (for a set period of time, in case one isn’t picked) which editors looking for referees can consult. This would ensure a equitable participation in the peer review process: if you submit, you also (offer yourself to) review. I think this would be helpful.
The real benefit of this proposal, though, would come from its ability to deal with (2) and (3). Here’s how it would work. Associated with a paper is a unique editorial login key. This lets someone registered as an editor (which could be done manually by getting lists of editors and registering them) add information to, and consult information about, the paper.
For example, if a paper is rejected, the editors use the key to upload the referee reports and the names of referees (if this sounds like it would be too onerous, it could be more or less automated via email: some journals (at least those in the taylor and francis family) tell referees the eventual verdict of the paper. One could simply add a new recipient to that email associated with the JNSS and a piece of code that would read these emails, extract the relevant information, and add it to the editorial backside for the paper in question). Then any subsequent editors would be able to access that information with the editorial login key (which the author would provide every time they submit the paper), so that they could know what previous referees thought about the paper, as well as their identity.
This goes some way towards fixing problems (2) and (3). A referee’s labour won’t be in vain. It will be in the very best interest of the author to respond to any previous referee report, because that report will be part of the information that a current editor can use when assessing the paper. So the problem of referees working in vain will be helped. Moreover, because referees know who previously reviewed the paper, they know who not to ask to review it again, and this will save time, which, in conjunction with the fact that there will be a referee pool for them to select from, should lead to referees being found much quicker.
There is, I’m sure, much to complain about this system, and I would like to hear those complaints. But the current system uses resources — authors’ time, editors’ time, referees’ labour, referees’ reports — very poorly, and a centralised journal neutral system might go some way to fixing that problem, leading to quicker decisions for authors, less wasted referee time, and a streamlined editorial workflow.
I reiterate what I said at the beginning that I’m up for people — especially people with a greater computer science background than me — to help out. And I welcome any comments about bad features of this proposal, or sociological or psychological or institutional reasons why its implementation is infeasible, as well as suggestions as to how it could be improved.
