Like the Loch Ness monster, the subject of peer review surfaces periodically in this blog and elsewhere. Peer review isn’t going to go away (I hope), but it will undoubtedly evolve, if for no other reason than the power of the Internet to add its own weight to the judgments exercised by journals and reviewers. It’s a process limited by the imperfections of human fallibility and bias, and so for me, the relevant questions relate to how specifically to improve it rather than how to turn humans into paragons of omniscience. At this point, I remain unconvinced of the value of any proposed alternatives to modify the basic process, although some of the supplementary forms of post-publication review mentioned above seem to have merit. That’s a tentative judgment, and I hope to remain open-minded.
My thoughts have some personal basis, because I’ve had decades of experience with peer review. Like Joni Mitchell, I’ve seen it from both sides now, or more accurately all sides now. I’ve been a reviewer who has often spent hours on a difficult and sometimes undeserving manuscript. This has convinced me that even if I had been paid at one thousand times the going rate, it would still have been less than I deserved. (Translation – reviewers work for free out of a sense of obligation. It’s surprising and gratifying that so many take the time to do a good job, even if others do not. One certainty is that any change that makes reviewing more onerous will be a step backward by discouraging good people from participating in the process. This includes making too many new demands on them, or making their efforts seem superfluous).
I have been a reviewee. Some of my papers have been published in Science, Nature, and respected specialty journals, which confirmed for me that the peer review process is an efficient mechanism for identifying and communicating important scientific content. Other papers of mine have been rejected by these same journals, demonstrating that peer review is a worthless exercise run by incompetents whose work could be better done by a panel of trained monkeys. Perhaps more important, though, most papers I’ve ever written have been improved by the process, and I’m grudgingly grateful even when I’ve thought the reviewers were too stupid to recognize the obvious value of what I submitted. Subjectivity in science comes with the territory.
I have been a journal editorial board member asked to solicit reviewers from among a group of busy scientists who have their own work to worry about. This experience convinced me that I should have gone into dentistry, because pulling teeth would be easy by comparison – but again, I’ve been grateful to those who have volunteered their time. Also in my capacity as both a board member and a reviewer, I have seen not only what gets into print in a given journal, but what doesn’t. That comparison is essential, in my view, for understanding the relevance of peer review to scientific advance. In particular, even the most deserving manuscripts usually end up improved, sometimes strikingly so, as a result of the process. Anyone who wants to know what a world without peer review might look like should spend time looking at pre-review manuscripts. A glimpse of some of this comes from reading blogosphere articles on climate that never get published. Wait a minute – didn’t even the Skydragons make it into print?
Most of all I’ve been a reader grateful that I have enough time to sleep, eat, and do other nice things because there are only 24 hours in a day and reading everything would probably take more than 240 of those. Peer review doesn’t stop articles from being published (which is probably a good thing),but it does accomplish two important tasks. First, it identifies articles that have passed the standards of high impact journals that can afford to be discriminating, and that acts as a filtering mechanism when there isn’t time to read everything. Second, it guarantees that individuals with expertise have had a chance to suggest improvements. Let’s take it as self-evident that neither of these processes works to perfection, but by the same token, no reader is forced to forego the opportunity to read low impact journals or to second guess the conclusions in papers that passed peer review in high impact journals.
In my view, concerns about the very real imperfections of peer review are exaggerated. Science doesn’t advance because journals always make the right decisions, but because genuinely important work can be replicated and spurious advances can’t. My experience with journals and their reviewers is that most try to do a creditable job most of the time. The process has improved over the years – for example, most journals no longer use an adverse review as automatic grounds for rejection but encourage a dialog among authors, reviewers, and editors. It’s also my expectation that current high impact journals will be reluctant to abandon a process that has gained them their reputation. It is in their self interest to publish work that is both significant and replicable, and ultimately that self-interest should prevent them from steering too far off course. If it doesn’t, their lustre will diminish and will shine from other journals that do better. My guess is that this Darwinian process will play a large role in determining the future shape of peer review.