It’s all very well to muse on the extensive blog reviews of a few papers. But they are few indeed; the productivity is very low. The IPCC 2007 report referred to about 18500 papers. Who is going to review those?

Open review ideas are fine too. But will the reviewers come to the party? This is the resource limit in any scheme. I noted, for example, the recent open review paper of Anna Makarieva, reviewed by Judith. For a long time no other reviewer could be found, despite the publicity the paper received. And the process just seems to have fizzled out.

Traditional peer review has many faults. Many alternatives are theorized about. But where will the reviewers come from?

The editors comply with the pal reviewers. If they don’t they get into trouble.

Those of you defending the pal-peer review system against blog review are making the assumption that someone has suggested that it be replaced by blog review. Try to understand the topic, before you shoot your mouths off.

Peer review primarily serves the interests of the journals, not the authors or the readers. The process is far too cursory to pick up all but egregious errors or fundamental misunderstandings; beyond this basic sanity check most journals are really only interested in the assessment of “importance”, which it itself driven as much by current fads and fashions as by anything objective.

“The editors comply with the pal reviewers. If they don’t they get into trouble.”

Ah the old conspiracy theory defence. I never knew climate scientists were so uniquely powerful.

“Those of you defending the pal-peer review system”

I’m asking people that support an open system to address the problems with such a system, something you’re clearly unable to do.

“are making the assumption that someone has suggested that it be replaced by blog review.”

You’re posting on a blog which explicitly references blog review. Blogs are simply a manifestation of an open publishing system. Maybe the “other people” understand the topic just fine and you don’t.

There are so many adverse rewards in the system of scientific research as practiced that the minor complaints about the inconvenience of peer review seems like registering an issue with the temperature of the bathwater on the Titanic.

Researchers continue to hoard data toward publication or for other less admirable reasons (such as keeping it out of the hands of rivals, keeping contrary evidence from seeing the light of day, sloppiness, sloth, vanity, to extort or trade future favors or prestige, etc.) as there is such pressure to publish new research — meaning new data, at least ‘new’ to the world.

This plunderous attitude toward raw observations is an immediate detrimental result of inverted values, and in my opinion far more damaging than the prolonged torture of waiting for peer review.

Worse, we frequently research the wrong thing or fail to research the right thing — fail to examine areas of study we can absolutely say for certain are valuable or crucial — for frivolous reasons. This is as true in atmospheric sciences, where we have known for over a century at least that we ought be measuring gas and aerosol levels carefully to develop some profile of our air (and yet failed even while weather stations track — badly — temperature), as it is in medicine, where it’s well known that the popular cancers and diseases get research funding, while many true scourges of substantially greater rate of mortality and economic cost or treatability go unfunded.

Internet prestige ought obliterate the very precept of journal prestige.

Online distribution of ideas, with a full open data from cradle to grave philosophy, ought run data hoarders out of town on a rail.

Topical international panels with some rational foundation for shining a light on the lapses and lacks in research, while at the same time promoting the most important and interesting results, ought become the general rule in science while science bodies with lackadaisical what-interests-our-membership or catering-to-corporate-agenda approaches ought slink away in shame.

Are you saying that I over simplified beliefs of deniers? If so, I plead guilty. Sorry.

After reading the above, I am perplexed why most still accept AGW as established science ready for policy.

Two big and related problems with peer-review are that people are short of time and that reviewers are sloppy adn often poorly qualified. I have received peer-reviews, which i) were obviously written by graduate students or perhaps even undergrads, ii) contained mistaken assertions and egregious errors of statistical reasoning, and iii) were riddled with spelling and grammatical errors! I suspect that harried professors, confronted by too many demands on their time, farm out papers to their students to cut their teeth on.

Although I am sure that some students are excellent reviewers, peer-review can only work if the reviewers are suitably qualified to assess the material. Having correctly qualified reviewers can not guarantee their objectivity or lack of bias, but I suspect that no review system could do so.

Like the Loch Ness monster, the subject of peer review surfaces periodically in this blog and elsewhere. Peer review isn’t going to go away (I hope), but it will undoubtedly evolve, if for no other reason than the power of the Internet to add its own weight to the judgments exercised by journals and reviewers. It’s a process limited by the imperfections of human fallibility and bias, and so for me, the relevant questions relate to how specifically to improve it rather than how to turn humans into paragons of omniscience. At this point, I remain unconvinced of the value of any proposed alternatives to modify the basic process, although some of the supplementary forms of post-publication review mentioned above seem to have merit. That’s a tentative judgment, and I hope to remain open-minded.

My thoughts have some personal basis, because I’ve had decades of experience with peer review. Like Joni Mitchell, I’ve seen it from both sides now, or more accurately all sides now. I’ve been a reviewer who has often spent hours on a difficult and sometimes undeserving manuscript. This has convinced me that even if I had been paid at one thousand times the going rate, it would still have been less than I deserved. (Translation – reviewers work for free out of a sense of obligation. It’s surprising and gratifying that so many take the time to do a good job, even if others do not. One certainty is that any change that makes reviewing more onerous will be a step backward by discouraging good people from participating in the process. This includes making too many new demands on them, or making their efforts seem superfluous).

I have been a reviewee. Some of my papers have been published in Science, Nature, and respected specialty journals, which confirmed for me that the peer review process is an efficient mechanism for identifying and communicating important scientific content. Other papers of mine have been rejected by these same journals, demonstrating that peer review is a worthless exercise run by incompetents whose work could be better done by a panel of trained monkeys. Perhaps more important, though, most papers I’ve ever written have been improved by the process, and I’m grudgingly grateful even when I’ve thought the reviewers were too stupid to recognize the obvious value of what I submitted. Subjectivity in science comes with the territory.

I have been a journal editorial board member asked to solicit reviewers from among a group of busy scientists who have their own work to worry about. This experience convinced me that I should have gone into dentistry, because pulling teeth would be easy by comparison – but again, I’ve been grateful to those who have volunteered their time. Also in my capacity as both a board member and a reviewer, I have seen not only what gets into print in a given journal, but what doesn’t. That comparison is essential, in my view, for understanding the relevance of peer review to scientific advance. In particular, even the most deserving manuscripts usually end up improved, sometimes strikingly so, as a result of the process. Anyone who wants to know what a world without peer review might look like should spend time looking at pre-review manuscripts. A glimpse of some of this comes from reading blogosphere articles on climate that never get published. Wait a minute – didn’t even the Skydragons make it into print?

Most of all I’ve been a reader grateful that I have enough time to sleep, eat, and do other nice things because there are only 24 hours in a day and reading everything would probably take more than 240 of those. Peer review doesn’t stop articles from being published (which is probably a good thing),but it does accomplish two important tasks. First, it identifies articles that have passed the standards of high impact journals that can afford to be discriminating, and that acts as a filtering mechanism when there isn’t time to read everything. Second, it guarantees that individuals with expertise have had a chance to suggest improvements. Let’s take it as self-evident that neither of these processes works to perfection, but by the same token, no reader is forced to forego the opportunity to read low impact journals or to second guess the conclusions in papers that passed peer review in high impact journals.

In my view, concerns about the very real imperfections of peer review are exaggerated. Science doesn’t advance because journals always make the right decisions, but because genuinely important work can be replicated and spurious advances can’t. My experience with journals and their reviewers is that most try to do a creditable job most of the time. The process has improved over the years – for example, most journals no longer use an adverse review as automatic grounds for rejection but encourage a dialog among authors, reviewers, and editors. It’s also my expectation that current high impact journals will be reluctant to abandon a process that has gained them their reputation. It is in their self interest to publish work that is both significant and replicable, and ultimately that self-interest should prevent them from steering too far off course. If it doesn’t, their lustre will diminish and will shine from other journals that do better. My guess is that this Darwinian process will play a large role in determining the future shape of peer review.

The abrupt move of the Rook, left his Queen in check. The next move will be all about the Queen. No one cares about the rook anymore.

Journals are businesses. They compete. They have to find a system which authors will submit papers, reviewers will provide their time, and readers will pay money. All three are necessary. The current system is what has evolved to balance these.

“We are at the point now where is nothing can be accomplished until we get government out of the classroom.”

Because, news is not objective, and peer reviewed isn’t science?
Something everything should know, but they don’t?
Because have public baby siting instead of public education?

But we don’t need to the correct news, nor the best peer review, nor even
get rid public education, but we need more competition.
And as it happens, this is all occurring. And main driver has been the internet. So that’s good news [in world that is spinning out control:)].

As for our peer review, it’s basically avenue to get a job- it provides status, promotes your work. Like a form baboon hierarchy. Which is fine.
A question is what is wanted something which also about status- perhaps a much higher status.
I am not a scientist but seems Nature etc are rather insignificant, it seems there tons of scientific association which allow papers to be published- they have conferences all the time. Not political conferences- like Durban, but conferences which involve scientists relating to scientists. These are intended to be a service to scientists. Whereas Nature, etc is more of public relation type operation- or a non gadget oriented Popular Science- science theory rather than technology magazine.
It seems if one wants status, one needs a board with bunch scientists with a diversity of all the disciplines [not limited to a small board- could grow to hundreds]. I would imagine the most important aspect of such a board is to have this board activity waste the least amount of their time.
So one need to lowest “cost” in terms of the administration part- meaning having a system and/or have good administrators.

So I would say what needed is talented hard working executives. It’s more of management issue rather than a science issue.

There can be no “objective” peer-review in a politically intensive topic like climate science. There is just too much to loose by climate scientists; talk about seeing one’s life flash before one’s eyes. This baby is going down. If Penn State made a series of conscious mistakes to preserve the University’s reputation re its football scandal, such mechanisms were in place to save Michael Mann’s bacon as well. If climate science is perceived as distorting the peer-review process to “keep on message”, it is unlikely that any assessment of research, grant process is safe. As politics has escalated the science validity and process, so the stakes the people who contribute to such escalation place themselves are at risk. Hansen, Mann, Schmidt & others risk the ruination of their reputations, livelihood, and their old age benefits. The political game is an all or nothing game. You are in or out after election day. You are in or out after a game changing scandal. High stakes.
The answer of course is to avoid elevating one’s science into the political arena. At the current state, with so many billions/trillions at stake there is no way the current group of visible climate scientists can do anything other than continue to manipulate peer-review, trumpet minuscule positives, debase people announcing glaring negatives, and hoping that they will outlast the naysayers or a conference (like Durban) or some other conference can “score a home run.” Their hope springs eternal. Ain’t going to happen. The blogosphere has peer-reviewed the peer-reviewers, and found them wanting. No one will want their name attached to a review that will subsequently be demonstrate their ineptitude and conformational bias. There are many in the climate science industry, academia, who would suffer greatly if they were found to display misconduct. What does one do with a degree in climate science when your reputation is trashed. Teach in a community college?
I believe in the peer review process except when it becomes a high stakes game. Slow and steady ultimately wins most races to tenure, grantsmanship, and productive, believable research. Don’t engage reporters. They are seductive, splashy, flashy, and fast out the door looking for the next “Watergate” expose. An academic researcher is best soft peddling their findings to whomever they encounter. And for God’s sake, don’t testify to Congress, it distorts your own self image, which in turn distorts how one reviews manuscripts, which in turn distorts the emphasis one places upon a piece of research which in turn…… When the weather is stormy, retreat to your lab and studying what is most likely.

FM;
As I understand it, the Hypothes.is project is an attempt to accelerate the “Darwinian” processes you speak of. Some aspects of how they intend to accomplish what they claim are their goals are obscure to me, but weighting of referee comments by the community and some kind of internal linking and ranking logic that filters trivia and nonsense seems to be implied.

We’ll see. What there won’t be is discretionary gate-keeping.

First sentence last paragraph should read “Oh, and OPEC does [] raise its prices….”

“My guess is that this Darwinian process will play a large role in determining the future shape of peer review.”

Maybe we can start a Darwin Awards Honorable Mention list

The thing is that the climategate emails have several emails from CRU where they state that the medieval warm period was probably as warm as the present day and that they didn’t really believe Mann’s work. At some point they agreed to back it but were still unhappy at overegging the pudding.

However while Mann was largely fooled by his lack of knowledge about his data mining algorithm and remains too proud to admit it, Briffa seemed to be quite careful to avoid including the Yamal data that would have invalidated his own hockey stick. They clearly needed the hockey stick and were collectively involved – it wasn’t just Mann – his paper should have failed review or been debunked soon after if there hadn’t been a dishonest cabal of silence about its shortcomings.

I’m sure it brought in lots of funding for both CRU and Manns employers so money was actually made from it, which might put it into the fraud category. It needs to be tested in court. If someone wealthy would back Tim Ball then it might be.

I’d like the fossil fuel companies to grow some balls and actually do what they are accused of doing, Despite the disinformation, we do have an unfair fight here with oodles of cash available for pro agw and nothing for skeptics. Unless this actually goes to court, the scientists can lie as much as they want because there is no mechanism to stop them.

JC comments: During the past few weeks, we have seen two interesting examples of peer review: the pre-publication extended peer review of the BEST papers, …

Absolutely not. The BEST papers were pre-published WITHOUT CODE OR DATA. Without code and data, we cannot peer review them in any fashion.

How on earth you have twisted a release without code or data into an “interesting example of peer review” escapes me. You’ll have to explain to me how I could peer review the BEST papers without code or data.

It’s a serious question, Judith. I’m calling BS on your claim about BEST. Either uphold it, or correct it in your head post. You can’t just pass out untruths like that, even if (or especially because) you work for the BEST project.

Consider this to be peer review of your paper. It is unpublishable in its current form, because it contains two unsupported and untrue claims, that the BEST papers were a) capable of being peer reviewed without any accompanying code or data, and b) actually were peer reviewed without code or data.

w.

I think that’s true for relatively inconsequential reports, but any paper that claims to be a significant advance will almost invariably inspire attempts to replicate the results. Most spurious claims of significant advances have been refuted promptly, and in almost all other cases, doubt has been cast upon them promptly even if complete refutation had to wait a while.