Peer Review Discussion

by Kelle on April 23, 2012

Taking on yet another angle of related to our past discussions about posting to the arXiv before acceptance, citation etiquette, let’s talk about the peer review process.

Here are some articles describing the current peer review process:

What’s your refereeing philosophy? Do you think there should be more guidelines to try to get more consistency in the reports? Or do you think the whole system should be more open?
I personally would like it be acceptable to post an early draft to a Facebook group (or a FB-like site) in order to solicit comments and discussion before submission. Please also share in the comments any links to resources you have found useful for guiding your refereeing philosophy.

{ 29 comments… read them below or add one }

1 Phil Massey April 23, 2012 at 10:58 am

There has been few things better written about this than the piece by Parker, available on the AAS website: http://aas.org/career/ArtofSciPub.php

A version of what Paul Hodge used to have on the AJ site back when he was editor has survived and made it into IOP: http://iopscience.iop.org/1538-3881/page/Information%20for%20referees

Reply

2 Ilse van Bemmel April 24, 2012 at 2:05 pm

I’ve often wondered if arXiv couldn’t adopt a facebook model with a ‘Like’ button and options for comments on a paper. There could be groups on different topics. Or even more advanced: a rating system on papers. Instead of the number of citations or an H-index, we could actually start rating the *quality* of people’s work.
Admittedly, it is a more complex issue than just ‘Like’, after posting this comment on facebook I was thinking it could be a process similar to moderating usenet newsgroups (remember those?). The key is that more people get to assess a paper, rather than just one.

Reply

3 Ross Collins April 24, 2012 at 4:01 pm

Isn’t that what YouAstro is attempting to do? http://www.youastro.org

Admittedly that’s kinda post-peer-review for now, but the principle should be the same.

4 Kelle April 24, 2012 at 4:43 pm

wow, i’d never heard of YouAstro before!! I remember ADS had comments awhile back, but then that got turned off! Is this an ADS service or independent?

5 August Muench April 24, 2012 at 8:47 pm

independent. completely and totally independent.

6 Tanya April 25, 2012 at 5:18 am

Ross, that is an amazing interface, been playing with it for the last 2 hours! Here’s to people using the comments section more…

7 August Muench April 25, 2012 at 8:56 am

hi tanya, that is an interesting reaction bc many of us feel like the youastro ui is exactly its problem.

8 Peter Melchior April 25, 2012 at 11:39 am

Hi Ilse,

what you suggested described exactly what I created http://paperrater.org for. It is based on the submission to arXiv (all of its categories), but also links to ADS to get the information for the published papers.

You can rate and comment each paper, pre- or post-publication. Anonymous comments are allowed, LaTeX is too. On the paper page, you get the details (with links to the PDF and BibTeX), the rating distribution, and the entire discussion thread. The website will also show, which papers draw a lot of attention during the last couple of days. For convenience, you can also set an arbitrary combination of arXiv categories to follow, e.g. I’m reading the daily submissions in astro-ph.CO, astro-ph.IM, and physics.data-an.

The collaboration features are not yet as evolved as I would want them, but I’m working on it. This involves the capability of creating virtual journal clubs, which get their own space for discussion.

I would love to get feedback on the website and how to make it really useful.

9 Brooke April 24, 2012 at 2:06 pm

Recently, I had a chat with a senior person about open refereeing. This person was all in favor of it until I said that I had considered revealing myself in a recent report. The response was an unequivocal, “no, you’re too junior. Revealing yourself can only hurt you.”

I don’t know that I exactly have a vision for improving the process, but I know that I like the idea of a more open process, yet also agree with what this person (and others) said to me. I think there could be serious downsides for junior people.

Reply

10 Aleks April 24, 2012 at 6:56 pm

The problem with youastro is of course the lack of incentives. There is no good reason to comment there – with the result that practically nobody comments there. Right now the incentive to write a referee report is the direct email from the editor, some feeling that someone has to do it, and maybe a line in the CV. Not much, but apparently sufficient. An open peer-review system needs to be better than that. It’s not impossible, but requires a bit more fantasy than a simple comment thread per paper. (Admittedly, the comment function doesn’t seem to be main thing the youastro people had in mind.)

Reply

11 Kelle April 24, 2012 at 7:38 pm

hm, there’s no incentive to comment or participate in discussions on blogs or Facebook and yet that seems to be something that has caught on…

12 Ross Collins April 25, 2012 at 5:05 am

I think the only difference is that this blog and Facebook are both established enough that you know you’ll have a sufficient audience to make it worthwhile and are not talking to the wind. YouAstro may never become established, but I don’t think that’s because the idea of providing comments on papers is a bad one…

13 Aleks April 24, 2012 at 7:56 pm

Well, do I really need to explain what the incentive is for being on facebook? It’s not ‘I need to evaluate the quality of the arguments in the latest paper by X’.

Reply

14 nick April 25, 2012 at 2:20 am

unmoderated comment streams on papers doesn’t sound obviously better than the system we have now. when the comments inevitably get dominated by a couple of particularly strident people, or teams of orthodoxy police, there’s not going to be that much more meaningful content for a reader who’s not already in a position to use reputations and past history of the commenters as a filter.

the comparison to blogs and facebook isn’t really apt- there is currently close to zero career impact for commenting on blogs, and so there’s no reason to get nasty. not being total jerk might be the only meaningful motivation when you’re posting something. when the reception of your paper (for better or for worse a potentially valuable piece of scientific currency) is at stake, the atmosphere could be (I suspect will be) totally different and not that pleasant.

Reply

15 Aleks April 25, 2012 at 6:43 am

One problem with the reviewing process: The only thing that can be reviewed is the entire paper. Apart from the fact that some papers need multiple reviewers simply because one alone doesn’t understand all parts, reviewing an entire paper is a lot of work, the threshold is high, and as a result the quality suffers. Once we get past the idea that the Paper is the only currency and implement ways in which people can ‘post’ their work in a variety of formats (data table, figure, code, proposal, note, comment, paper, and, yes, peer-review), high-quality peer-review will be much easier to achieve, because it becomes an organic part of collaborative work.

Reply

16 Peter Melchior April 25, 2012 at 5:51 pm

On http://figshare.com you can upload data, plots, lab notes and such. Key feature: Uploads can be cited later on.

I like the general notion of making smaller-scale results public, but the risk is that they become incoherent. I prefer the idea of open notebooks / lab books because it not only shows results, but documents the process. I’ve seen a couple of examples, Wikipedia has a good list of them: http://en.wikipedia.org/wiki/Open_notebook_science

17 Aleks April 25, 2012 at 7:22 pm

Peter, I agree, figshare is a good idea, but I would still prefer an integrated system. I’m currently looking at paperrater.org, and might be in touch soon.

18 Sarah April 25, 2012 at 9:14 am

I agree with several other commenters that incentive is a big problem for getting this sort of open commenting to work. Anyone’s who’s reviewed journal articles knows that to be able to provide a real assessment of the work’s quality requires very careful reading, perhaps looking up other references, scrutinising plots. It takes time! If there’s no reward for doing this and writing a reasoned critique, who will bother? This has been the experience with all such experiments that I’m aware of (e.g. Nature tried open peer review for a while – it didn’t work).

And Ilse – I really hope my professional reputation doesn’t get reduced to the number of “Like” clicks I get on astro-ph. I’m no fan of citation counting but I’d prefer it to that any day.

Neuroscientists Kravitz & Baker published an interesting paper on this issue some months ago, calcualting the true cost of peer review, and proposing an alternative system that is “open”-ish. There are some cultural differences between this field and astronomy, but it’s an interesting read (further discussion also here).

Reply

19 August Muench April 25, 2012 at 9:34 am

@sarah: I’ll make two points we all already know but are worth adding after you comment:

1. There is no incentive (quantifiable) at all for peer reviewing anything. Its entirely for “free”, entirely without attribution or personal value. I would bet that a quality (detailed) open review of a paper on FB would provide incrementally more CV power than any, “I reviewed for Nature” section would.

2. Nature’s attempt at open peer review ended on 30 Sep 2006, which was just four days after Facebook went fully open.

20 Peter Melchior April 25, 2012 at 6:11 pm

Sarah and August,

the lack of incentive is the key problem, I guess we all agree on that. But then why do we accept the task to referee for a journal for free. I guess because we want to help the authors straightening out some weaknesses and thus provide the entire community with solid, relevant research.

Wouldn’t it be better if this process was open? And by that I do not necessarily mean that the referee’s name needs to be revealed. But if the referee decides to show his name, he would at least get the public recognition for the effort. Authors would not depend on a single referee, with whose opinion and criticism they might disagree. And as a community we could only learn from this process, and over time weed out sloppy refereeing. I can’t see a drawback.

21 Aleks April 25, 2012 at 10:28 am

Here another example of post-pub review: http://f1000.com/

(Of course once something like this is in place and works well, there is no real reason to maintain the pre-publication review and to limit it to papers.)

Reply

22 Phil Massey April 25, 2012 at 12:31 pm

The way scientific publishing was first explained to me (I guess) is the following. There are large groups of scientists who have come together to form organizations. In the case of U.S. astronomers, such a group is the American Astronomical Society. The Astronomical Society of the Pacific is another case. In the UK, it’s the Royal Astronomical Society. These societies publish journals of astronomical research, both by their members and by non-members. The AAS publishes ApJ and AJ; the ASP publishes PASP, and the Royal Society publishes MNRAS. The publication decisions for these journals are made by editors, who have been appointed by search committees, which have been formed by elected officials of the society. These editors often seek outside, expert advise about a paper; these experts are the referees. The referee reports are of course purely advisory to the editors, but it’s a rare occasion when a referee report is so off the wall that the editor chooses to ignore it (although it does happen).

This structure seems to be at complete variance with many of the assumptions of the underlying complaints above. Sure, anyone is free to “publish” anything on-line or elsewhere. But, a refereed paper has been vetted, and undergone a serious review—not a bunch of FB “likes”. If you don’t like the process, run for AAS president and appoint anarchists to the publication board. Good luck with that!

Reply

23 August Muench April 25, 2012 at 12:42 pm

Anarchists? Really?

24 Ben April 26, 2012 at 1:02 am

That’s the way it’s supposed to work. Sometimes it does. I think it’s legitimate to admit that it has problems. Occasionally an abusive referee can really delay a paper, and even thorough referees can’t guarantee all papers are correct, important, or cite previous work fairly. Peer review reminds me of the old joke about democracy: it’s a terrible system, but all the alternatives are much worse.

One reason I think we need peer review is not just that the referee is supposed to catch all the errors of the authors – the referee can’t possibly catch all errors. Rather, knowing that the paper will get a detailed going-over, the authors (usually) have to try to clean up the paper, eliminate mistakes, add references, and tone down claims that might be exaggerated, to pre-empt things that risk setting off the referee.

I have a lot of concerns about open commenting and reviewing of papers because I suspect it will be dominated by the most vocal people and/or the people who don’t have to worry about whether they’re irritating anyone. You’ll get fewer public opinions from people with strong internal filters, junior scientists, and so on. Consider the fact that if you have a Facebook feed, probably 90% of it is occupied by 10% of your friends; and if you call into collaboration telecons, probably 80% of the talking is done by 20% of the callers. (Maybe it’s 90/10 on telecons too, but that’s too depressing to think about.)

25 Eilat April 25, 2012 at 2:10 pm

It is my understanding the scientific editors do not get paid. Perhaps this is at the root of many of the problems: lack of incentive and the overworked editors. I am purely brainstorming here, but perhaps the journals could adopt something like the rotating officers in the NSF where someone is “borrowed” from their institution for a stint to devote a significant amount of time as editor.

Reply

26 TJ April 25, 2012 at 2:59 pm

I’ve long thought that a Slashdot “karma” model would be interesting to try. The key point is the karma accumulated by users for their contributions. It’s an imperfect metric, but it’s better than what we’ve got now. The idea would be to rate both posts (e.g. papers) and comments (discussions or follow-up work), and a given user’s karma reflects what people think of the quality of that individual’s contributions in both places (together or separately, moderated or open, TBD). That provides an incentive to make intelligent comments that others would find useful to the discussion, thus improving your own “karma”. Then instead of looking up a person’s pub number and citations, you look up their karma that also takes into account the quality in addition to quantity and includes their input on others’ work.

Reply

27 Joe April 27, 2012 at 5:36 pm

For some reason, your mention of the karma model in this context made me think of ‘whuffie’ from Cory Doctorow’s Down & Out in the Magic Kingdom ( http://craphound.com/down/ ).

Although, in that case, it was the currency for their society, and people could give it to others for whatever they thought helped to improve the society. (so in this case … would you give karma / whuffie to those whose papers you cited, or the PIs from the observing campaign whose data you used?)

28 Jason Wright April 27, 2012 at 10:13 am

A few thoughts on refereeing:

1) Motivation: Being a referee is akin to serving on jury duty. It’s unpaid, but it’s part of your job (duty) as a professional astronomer (citizen) to support the infrastructure you need to publish (get a fair trial). In a sense, any employer of astronomers is paying them to perform, among other things, professional service like this. I get ‘credit’ for refereeing every year when I fill out my Faculty Activity Report and tell Penn State that I refereed X papers.

1a) …as a follow up, I don’t see the reason I should referee Nature papers. Nature is a for-profit magazine unaffiliated with my professional society. I have nothing invested in them, and none of the sense of solidarity I have with, say, the Royal Society (so, I will referee for MNRAS). If Nature wants to make a profit selling published refereed papers, shouldn’t the refereeing be a cost of doing business?

2) Purpose: I think there are three purposes of refereeing: to be a “gatekeeper” and keep the crap out of the journals, and to give the author and editor an unbiased sense of how the paper will be received, and to improve the paper in ways the author may appreciate.

I think the first is inappropriate: the journal’s standards are the editor’s to enforce, and so the editor is the gatekeeper, acting on the referee’s perspective. In my opinion referees do not “reject” papers, editors do. Referees give the editors advice. Therefore I think the second is the most import job of the referee.

The third aspect is where the serious reading comes in. Especially if the author is a younger astronomer, the referee can really help by offering anonymous, constructive advice. It’s easy to criticize and tear down a paper; some of my best papers were made that way by thoughtful referees who brought a fresh perspective and pointed out holes and fuzzy bits that were actually much easier patched than I had anticipated.

3) Open refereeing: Great! I think it’s wonderful to let people see what you’re working on and let them make suggestions. But this is in addition to the refereeing process above: journals serve several useful purposes (archiving data and figures, editing text and tables, providing rigorous peer review) that we should embrace and protect. These services are not exclusive of a parallel, open refereeing process. Many of us in large collaborations essentially do a “limited” version of open refereeing when all of the co-authors chime in and contribute suggestions and thoughts. Extending this to broader communities should be very useful, but at some point a document should become “official”, if only to force the authors to finalize their assertions and allow them to be properly cited.

Reply

29 Jeremy April 28, 2012 at 2:44 pm

I actually think that double-blinding would be, at least in principle, useful… which is less “open” than many things being discussed.

Reply

Leave a Comment

Previous post:

Next post: