The Problem of Peer Review Is The Most Important Philosophical Problem
(Note: a version of this post is forthcoming (as a paper) in Metaphilosophy. Stars correspond to footnotes.)
As philosophers one of our aims should be to produce as much philosophical knowledge as possible. A lot of potential philosophical knowledge is lost because of the flaws of the peer review system, and so a lot of philosophical knowledge would be gained were the system improved. Accordingly, as authors we should write papers about how to fix peer review, and as editors accept such papers.
The View
Call the (philosophical) epistemic return the amount of (philosophical) knowledge that results from a given action. Then I think the following is plausible:
(EPISTEMIC RETURN MAXIMISATION) One should as authors write, and as editors accept, those papers with the greatest expected epistemic return.
This is kind of a principle of effective philosophizing: you should spend your time on work that will add most to the stock of philosophical knowledge, just as the effective altruists (e.g. (MacAskill 2015)) tell us you should spend your time on work that will lead to the most lives saved. This leads them to the counterintuitive conclusion that working on Wall Street might help people better than working for a charity (because you can donate some of your massive salary to buying mosquito nets, which might save more lives than a lifetime working for Barnardo’s); below I’ll suggest my own counterintuitive analogue.
This principle obviously requires much spelling out. It’s unclear what, if any, knowledge philosophy produces. It could be that philosophy doesn’t aim at knowledge, and so the principle becomes useless (as phrased; presumably philosophy aims at something, and whatever it is, that’s what we should maximise). And it’s unclear in general how to aggregate knowledge so that we can speak of an amount of knowledge in a way that will track something we care about.
In this short paper (and even without it) I can’t do this spelling out. I hope the principle strikes you as plausible, as reflecting, to some extent, some of the decisions you make as author, editor, and reviewer. It’s why we prefer to read, review, and publish papers which make bigger contributions than those which merely clarify an ambiguity in some argument.
Granting this principle, the purpose of this paper is to suggest a plausible and arguably surprising application of it. The application turns on the following claim:
(PEER REVIEW) Papers about how to fix peer review are among those with the greatest expected epistemic return.
A consequence is that one should as writers spend time writing papers about how to fix peer review, and as editors we should — if they’re good — accept them.
The argument for this is as so: the flaws of the peer review system lower epistemic returns. Good papers get rejected, bad papers accepted, time that could have been spent adding new knowledge is wasted redrafting for reviewers or seeking reviewers for or rereviewing papers that have already been rejected, and so on. If you write a paper on how to fix peer review that gets published in a good journal, its suggestions are more likely to get implemented — especially if your paper gives instructions on how to implement its recommendations (this is an important step, to be expanded on below). If your suggestion does get implemented, it will permanently increase the epistemic returns on the journal system as a whole (over a given time period). Just to use some numbers for illustration: say you improve the system in such a way that, each year, two brilliant papers that would otherwise have not been added to the stock of knowledge are added. Then in five years, thanks to your intervention, you will have been indirectly responsible for the creation of ten brilliant papers. Without you, the papers wouldn’t have been added to the stock of philosophical knowledge. Over 40 years, we’re talking eighty brilliant papers.
That arguably puts you among the greatest contributors to philosophy ever (and my numbers are plausible extremely conservative). If that’s not a judgement you’re inclined to agree with — and it probably isn’t — it’s a judgement you should agree with. If you’re in philosophy for the love of wisdom, you should seek to maximise the amount of wisdom you’re responsible (in part) for creating, even if that involves ways more indirect than normal (ways more indirect than, for example, publishing a pathbreaking new paper in Mind on the irreflexivity of grounding, or teaching a seminar, or giving your colleague helpful comments on their new paper, each of which produce epistemic returns).
That these contributions be published in academic journals is also essential. But that’s non-obvious. Why isn’t it sufficient that these new ideas for peer review be disseminated on blogs? That is, after all, where most of the discussion about peer review seems to happen these days.*
(*Google site:dailynous.com, site:leiterreports.typepad.com, site:philosopherscocoon.com followed by peer review to confirm this.)
The reason is incentives. Our profession incentivises the publication of journal articles in a way that it doesn’t incentivise blogposts. Blogposts are things one does in one’s spare time, and that one reads in one’s spare time, when one is off the clock. But that’s no good — the problem of peer review is complicated enough that we need to harness on-the-clock mental energy and the antagonist arena of journal publishing, whereby papers are subjected to tough review, and then presented to the community who are incentivised to find problems with it (because they might get a publication out of it). Think of it like this: we have a means at our disposal of directing the mental energy of a lot of smart people in such a way that their efforts can permanently improve philosophy. So we should so direct it.
That’s basically all there is to my proposal. A good way to understand a view is by considering objections to it, so that’s what we’ll do. But first let me quickly run through some hopefully familiar material about the problems of peer review.
The Problems Of Peer Review
Peer review is used as an accreditation system in academic philosophy, and publications are used to assess academics for jobs, promotions, invitations to conferences, and so on. It thus plays an extremely important institutional role, and it is in our best interests that it be as good as possible. But, as is familiar, peer review has lots of problems. Here is an inexhaustive list of complaints about peer review. I write as both an author and as someone with experience behind the scenes in an editorial capacity.
Lack Of Standard Practices
(1) Unstandard reviewing processes. Some operate single blind, some double blind, some triple blind. This means that comparing a publication in say Semantics and Pragmatics (a linguistics journal which philosophers of language publish in, and which doesn’t require anonymisation) with Analysis (which is triple-blind) is not a like-for-like comparison.
(2) Unstandard reviewing styles. Some people are harsh, some lax. Some more inclined to accept, some never accept. Some are quick, some are slow. Some recuse themselves if they recognise the author, others don’t.
(3) Unstandard editorial practices. Some journals do a lot of desk-rejecting (Analysis, in its older incarnation, for example). Some don’t. Some always pass on referee reports, some don’t. Some always require at least two reviews, even if that means a much greater wait time for authors. Some have obscure editorial procedures (Ethics and Philosophical Review, for example).
Volume
(4) There are so many submissions that finding referees is extremely difficult in light of the means of finding referees available and the time available for editors to do so.
(5) Anecdotally, referees complain about getting having to do too many reviews. Anecdotally, some possible referees complain about being asked too seldom, if ever.
This latter fact is reflective of a more fundamental issue: to submit a paper, one needs no qualification, while to referee one, typically one needs to have some publications in the area. This asymmetry is plausibly a big part of the reason why finding referees is so difficult. Even taking into account that philosophers from previous generations are still around to review papers now, it’s plausible that at any time, the number of papers submitted for review is considerably greater than the number of reviewers available.
Money and Institutions
(6) Most journals are owned by large publishers whose aim is to wring every last bit of profit out of their business before academia collectively wakes up and moves to an open access model. Their cost cutting, at the minimum, wastes authors’ time when they need to recorrect ineptly corrected proofs, for example, and editors’ time when they need to fix problems they cause. The subscription model means that many are unable to access the results of research produced, which surely decreases epistemic return.
(7) Journals operate independently, with little to no information sharing. Means and conventions for sharing information — about, say, papers that have already been rejected elsewhere — would massively increase efficiency; their lack lowers epistemic returns below what they could be.
(8) Reviewing offers no incentive: it’s at best a small line on one’s cv.
(9) Editing offers greater incentive, but in a lot of cases it’s still not sufficient relative to the time expended.
(10) If reviewing and editing offer no incentive, equally problematic is that submissions impose no costs. This leads to the philosophical equivalent of spam, where authors resubmit papers even in the face of cogent negative reviews, in the hope of getting lucky.
(11) Although things are massively improving thanks to things like philpapers, the SEP, the ubiquity of academics’ websites, as a community we could share much more information than we currently do. It is hard, even with these resources, to find everyone who works on a given topic. (12) Journals (and authors) aren’t transparent with their statistics, and so we don’t have a good grasp on the numbers involved — -more transparency would enable us to tailor solutions to the field as it actually is.
To do with time
(13) The average time to acceptance for a paper must surely be well over five months. Arguably, even an extremely thorough review could be completed in a day (~7 hours). There must be a way of making these numbers less ridiculously different.
This is an inexhaustive list, and part of any project to improve peer review would involve adding to, and perhaps deleting and assessing the relative importance of the items on, this list. Hopefully it has reminded you, if you need reminded, that there is indeed a problem here. And surely, one must think, if some of these problems are fixed, then the epistemic return from the activities of the journal system over a given time period will be better than what they would have been. And so, if you’re interested in maximising epistemic return, you should work to fix it. In what follows, I consider some objections to my view which will hopefully also serve to get clear about its nature.
Objection 1: (EPISTEMIC RETURN MAXIMISATION) is False: it Overgenerates
The first objection goes as so: my principle says you should spend your time as a philosopher on papers with the greatest epistemic returns. But you shouldn’t. That would, if consistently followed, lead philosophers to try their hands at all sorts of wild stuff.
Thus consider this fact. Donald Knuth’s typesetting software LaTex has plausibly increased the epistemic returns of philosophy by making it much less time-consuming to write logic-heavy papers. Now imagine as he was writing it he had encountered some problems that he couldn’t solve — should he then have tried to publish some articles about it in The Journal Of Philosophy?
Obviously not. Problems about typesetting aren’t philosophical problems, and shouldn’t be worked on by philosophers. Similarly, one might think, problems about the institution of peer review aren’t philosophical problems and shouldn’t be worked on by philosophers.
More generally, merely being causally upstream of a philosophical contribution isn’t sufficient to make a contribution to philosophy. My principle might seem to imply that it is sufficient, so my principle is no good.
Now, there’s a fix that this suggests. You should maximise epistemic returns while working on a problem in philosophy.* Knuth wasn’t. Peer review, I’d need to claim, is. But is there any reason to think this — any reason to think it is a philosophical problem? This leads to objection two.
(* This fact gets over the analogues of criticisms put to effective altruism to the effect that it suggests working on Wall Street is morally what one should be at. The analogue would go: you should spend your time sucking up to a billionaire in the hope that they fund a chair in philosophy at your alma mater. If they did, that would yield great epistemic returns. The reason this isn’t a worry is that to suck up to a billionaire isn’t to solve a philosophical problem.)
Objection 2: The Problem Of Peer Review Isn’t A Philosophical Problem
Coming up with a peer review system that avoids as many of the above problems as possible (and some more that I haven’t thought of) isn’t recognisably a philosophical problem. So philosophers shouldn’t be working on it.
Well, note that there’s almost certainly no necessary and sufficient conditions for a problem to count as philosophical. If it resembles philosophical problems sufficiently, that should count. And here are three ways in which the problem of peer review resembles other philosophical problems.
Firstly, it’s essentially a question of creating a formal system with a range of desirable properties; of technical modelling. Many philosophers love such exercises in model building: logicians, semanticists, metaphysicians, formal ethicists, and so on happily spend their time devising abstract models of phenomena. If you have a taste for such problems, and view them as philosophical, you should view coming up with a new peer review system as philosophical. But more particularly, it seems that group epistemology and agency, game theory, certain areas of ethics and social choice theory are all going to be relevant. Since we think of work in these areas as philosophical, so we should think of fixing peer review as philosophical.
Secondly, it’s a general problem wont to fall through the cracks of other disciplines. Part of the reason that we’re still in the dark ages (or at least the early 20th century) as concerns peer review is that it’s no one’s job to work on peer review. It’s one of those sort of general problems that no one discipline has in its remit. But philosophy has those sort of general problems in its remit, so philosophy is arguably a suitable discipline to work on it.
Thirdly, it’s philosophical in the sense that it’s arguable one requires philosophical expertise to treat it. Solving the peer review problem in philosophy is different from solving it for physics or for English, requiring specialised knowledge about the discipline of academic philosophy, what good philosophy is, what incentivises philosophers, and so on.
Overall, then, and bearing in mind how loose the definition of philosophical problem is, there’s a good case to be made that peer review is a philosophical problem.
But even if it you’re not convinced, it should be so classed. And the reason for this is because of certain normative benefits it will have: it will lead to people working on it in the antagonistic atmosphere that is conducive to weeding out bad ideas. Since we should all agree that peer review needs fixed, we should all be willing to treat it as a philosophical problem to achieve these good effects. That is, I’m suggesting that we view this as an ameliorative project in Haslanger’s sense: where we treat something as being a certain way because it should be that way.*
(* For more on ameliorative projects, see the papers collected in (Haslanger 2012))
This is obviously a line that needs to be treated with care. We probably don’t want to say that in all cases where an antagonistic bunch of philosophers’ attention would help solve a problem we should consider that problem as a philosophical one and publish about it in good journals (I have some sympathy with the thought that we should, maybe, make wider use of the incentive structure the institutions of philosophy offers to herd mental energy in a productive direction, but for the point of this paper I don’t need to put forward a view about this). All I’m saying is that in this particular case, an ameliorative strategy seems good.
Objection 3: The Problem Of Peer Review Is Not Intellectual But Logistical
Now let me turn to the third and final objection which I think this is arguably the most serious. According to it, I’ve completely misunderstood what the problem is and accordingly what its solution needs to look like. It’s not primarily an intellectual problem, it’s a practical one, one having to do with all too imperfect human motives, power structures, and straightforward logistical problems. Thus consider these facts (and note again this is inexhaustive, and part of the program I’m suggesting would be to add and/or remove members from this list):
i) The people most motivated to effect change, primarily junior scholars, are those least likely to be able to do so. The realist might suggest this means change won’t happen.
ii) The people most able to effect change are those who have been treated well by the system. The cynic might suggest this means change won’t happen.
iii) Some form of centralization, or at least improved communication, between editors, will be necessary. It’s not clear the big publishers will like this, and anyway would require new technical infrastructure and ways of working that current systems and already overburdened editors will struggle to transition to.
In light of this, you might think that I’ve misunderstood the situation. It’s not the case that it’s just an intellectual problem, but it is a practical problem that no amount of ingenious formal models will change. In support of this, one could note that pieces about changing peer review are replete with many good suggestions, but hasn’t received any uptake (for example (Davies and Felappi 2017, (Katzav and Vaesen 2017))
At the risk of being an annoying philosopher, my response to this is that these are just things are need to fit into our model — they make the problem, if anything, more interesting. iii) for example, means that we need to try to make the transition as time- and infrastructurarily- cheap as possible. i) and ii) mean we need to factor in power imbalances. It might suggest, for example, that the majority of people who seek to effect change should go it alone without the support of any people who aren’t interested in changing the system, even if those are the elites who wield most power.
But that’s no reason to despair. One of the most remarkable recent developments in the past decade or so, in my opinion, is the rise of decentralised information recording and validating systems, to which anyone can contribute, anonymously, and the reliability of which is guaranteed via a clever combination of computer science, game theory, and good old folk psychology. I mean blockchains, the most famous — if not for all the right reasons — of which is bitcoin.
If you had suggested fifteen years ago that we could make payments over the internet without having recourse to powerful third-party vested interests like banks and credit card processors, people would have been scepticial. But we can (ish). I think scepticism about the possibility of implementing an improved peer review system in light of the above hazards shouldn’t be too worrying. Where there’s a will, and a bunch of clever people, there’s a way.
That’s obviously highly speculative, but a more general point stands: our formal models of group decision are based on actors who are in some sense unideal and uncooperative, and so we just need to think about how to get over the unidealness and uncooperativeness. Accordingly, a very important part of the research program I’m suggesting is that we think about how to overcome these real world practical logistical problems such that a technically beautiful solution that won’t get implemented doesn’t count as a good contribution.
But a second point: if we do come to publish these proposals in journals, it will, by publicising them, give people an incentive to work on their implementation. If Kit Fine takes a break from truthmaker semantics and gives us a nice formal model of peer review, then some grad student can probably be relied upon to code it up, to impress Kit Fine and the rest of the philosophical community. Even if there will remain implementational issues, by giving the topic more prestigious coverage it’s plausible people will be more willing to deal with these issues.
Conclusion
We should write and publish papers on how to fix peer review because doing so will increase the amount of knowledge philosophy produces and as philosophers that should be one of our main goals. The ideas in this paper are perhaps eccentric and from left field, but to the extent that taking them seriously could drastically improve our profession for the better, they should be taken seriously.
Acknowledgements
I thank Herman Cappelen and Poppy Mankowitz for help with a version of this paper.
References
Davies, Benjamin & Felappi, Giulia (2017). Publish or Perish. Metaphilosophy 48 (5):745–761.
Katzav, J. & Vaesen, K. (2017). Pluralism and Peer Review in Philosophy. Philosophers’ Imprint 17.
Haslanger, Sally (2012). Resisting Reality: Social Construction and Social Critique. Oxford University Press.
MacAskill, William (2015). Doing good better : how effective altruism can help you make a difference. New York, USA: Gotham Books.
