Putting the peer back in peer review

Here’s a very familiar story. Someone submits a paper to a journal about topic x. They are, by any reasonable standard, extremely well-educated: they have been in higher education for at least a decade, as an undergrad, then a grad student, then a postdoc or adjunct or maybe even in a tenured position. Moreover, they are, by any reasonable standard, an expert in x: they most likely did a PhD in it, and they will very often have published other articles about it.

The paper gets sent out for review. Often, the reviewers will have the same credentials as the author: years of education, years of studying x. Let’s say that one suggests major revisions (if you’re not in academia, this means the reviewer thinks the paper is publishable provided extensive are changes are made to it) while the other suggests rejection. This, and I speak from experience, is very common.

Based on that, the editor will often reject the paper: after all, nobody really liked it. One said no and the other maybe.

But something troubles me about this. It’s not true that nobody liked the paper. The author surely liked the paper. And the author, by hypothesis, is an expert in the topic. Indeed, were they not the author, it could well have been sent to them to review. If the author didn’t like the paper, they wouldn’t have gone to the hassle of researching, writing, and submitting it. It strikes me that the mere fact that a paper was submitted can often provide defeasible evidence in favour of that paper (this requires the assumption that the majority of submitters are experts, which conforms to my three years experience working for a journal, but could be false). Authors should be treated as peers.

But if that is so, our calculation is changed. Now we have: one person liked the paper, one person disliked it, and a third was on the fence. In such a case, rejection is not so clear a verdict.

That’s basically the first speculative suggestion of this post: a paper’s being submitted is evidence in favour of it. This supports a much more inclusive acceptance policy than is commonly practiced.

There’s at least one objection to this view, which will lead to my second point. Call a paper already-submitted if it is submitted to a journal having already been rejected by a previous one (I would use ‘resubmit’ but obviously I can’t). Anecdotes would suggest that perhaps most papers are already-submitted (indeed, we could maybe bolster this into an argument if we assume there are many more non-elite than elite journals, and that most philosophers attempt to publish their work in elite journals before non-elite ones. Then most papers in non-elite journals will be already-submitted, which means most papers overall will be already-submitted).

Already-submissions change the equation again. In our case, we don’t have one person liked it, one disliked, and a third was on the fence. Rather, we need to factor in the original rejections — if we assume that that verdict involved at least one rejection, then we have one person liking, two disliking, and one or two on the fence. And then rejection seems more apt a verdict.

And if most papers are already-submissions, then an editor at a non-elite journal can perhaps, even if they adopted the perspective suggested above, discount the evidence in favour of the paper supplied by its being submitted in the first place by the presumed previous negative reviews (the attentive reader will realise the calculations get messed up with multi-authored submissions, but we can safely ignore them as they are at least somewhat uncommon. Anyway, the point would only substantially change with massively multi-authored submissions, which never happens in philosophy).

So where are we? I suggested initially that submission is evidence in favour of a paper’s quality; then I suggested, actually, it wasn’t, because that evidence is outweighed, at non-elite, i.e. most, journals, by the fact that the paper has probably already been submitted and rejected. Is there a way to keep the first suggestion in light of the second?

Yes, and I think the way points towards the direction peer review needs to go. The problem caused by already-submission is a problem of slight marginal costs: once one has written a paper and it’s got rejected, it takes very little effort to submit it somewhere new. A natural thought then is that if we were to impose effort — cost — on already-submission, we might be able to clear the paper market of already-submissions, and come to use the fact of a paper’s being submitted by an expert as slight evidence in its favour.

Here’s how to do it. One is only allowed to already-submit a set number of times per year. As an author, when you face a rejection, you have to decide: do I still, in the light of the criticisms, have faith in the paper? If you do, you put your money where your mouth is and use one of your limited already-submission tokens (what are these ‘tokens’ and how are they kept track of? Errr … (but seriously, where there’s a will there’s a way)).

Here’s a way to think of it. By using one of these tokens you’re essentially reaffirming faith in your paper, and adding another vote to it. So even if the editor can safely assume it’s already been rejected, that needn’t stop them taking its new submission as evidence in favour of it, because were the author to have been convinced by the previous criticisms, they would have been unwise to use one of their limited already-submission tokens on the paper.

There’s much, I’m sure, to disagree with. But I hope there’s two big picture lessons that can nevertheless be drawn: as editors, we should place some trust in our authors. We should treat them as genuine peers, whose attitude towards their paper should count in our assessment of it. As authors, we should realise that we need to incur costs of some sort by already-submitting papers.