Guidelines for Refereeing Journal Articles [Wiki]

by Kelle on February 8, 2011

In collaboration with a few colleagues, I have sketched out a wiki page with some guidelines for refereeing submitted journal articles.

Nearly everyone has wildly different expectations of the goals, duties, and time expenditures expected of a referee. At times this leads to combative encounters, complaints to editors, editorial decisions thought to be bizarre by both the author and referee of the same paper, and incorrect results being accepted. While every paper and process is somewhat unique, many of these problems likely stem from a lack of any guidance and the lack of a consistent set of expectations within the community. A relatively large collection of postdocs/young faculty have no idea what the expectations are for the referee process, with the only guidance being the very wide range of referee reports we have received over the years. Below, we attempt to provide some of this lacking guidance.

ROLES OF THE REFEREE:

  • decide the “reasonableness” of the paper
  • point out obvious errors, if any
  • evaluate major claims of paper for reasonable-ness
  • suggesting changes that would improve the readability and accessibility of the paper

THINGS REFEREE IS NOT RESPONSIBLE FOR:

  • redo all calculations
  • identify and fix grammatical or typesetting errors
  • suggesting major additions. resist the urge to try to turn the paper into one that you would’ve written. Be a referee and evaluator, not a co-author

THINGS WE WISH REFEREES DID NOT DO:

  • Contrive ways to add references to their own papers
  • Take three months to return a cursory report that makes it obvious they barely read the paper
  • Be mean and condescending to authors they don’t know personally. Instead, consider trying to write in the same tone you would use if the author was standing right in front of you.

Do you agree with what we have there so far? What’s missing? Do you have any concrete advice on what you look for in a good referee report? What guidelines have you set for yourself as a referee?

UPDATE – In your comments, please just give the guideline you’re suggesting and refrain as much as possible from recounting your nightmare referee stories.

{ 14 comments… read them below or add one }

1 Tigran Khanzadyan February 8, 2011 at 7:42 am

I would add a point in the section “THINGS WE WISH REFEREES DID NOT DO:” – Agree on refereeing the manuscript when there is a conflict of interests.

Reply

2 Jeremy Sanders February 8, 2011 at 9:06 am

A referee should also make sure that authors adequately reference existing work in the area. There are too many papers which do something similar to work that was done some time ago, but don’t acknowledge that.

A referee should also make sure that what is being submitted is not a duplicate or near duplicate of a previous paper in a refereed journal (whether the submitter’s own or someone else’s work).

Reply

3 Gus February 10, 2011 at 10:42 am

i think this has a lot to do with the “tone” tests in the guidelines for refereeing. strictly speaking citation has less to do with what has been done (conceptual attribution) than to do with what has been sourced. even if you do subscribe to the idea that citation should capture the entirety of conceptual attribution then the authors may well have overlooked a series of papers on the subject. either way the *tone* when in making reference suggestions should reflect ones lack of knowledge about what the authors have or haven’t used/read/been made aware of.

Honestly, I think one wrench in the system is a referee has/appears to have/assumes s\he has anything more than a peer relationship to the authors. And, yes, this comes from someone who knows he has failed such a test as a referee.

4 Ann Onymous February 8, 2011 at 9:44 am

What I would like to see is editors not asking people who have not published anything in the last 20 years. [story snipped by editor. the point: referees should not be out of touch with relatively recent results.]

Regarding this point “suggesting major additions. resist the urge to try to turn the paper into one that you would’ve written.” I think it is really a question of balance and how impactful the addition may be. I have already suggested major additions because I thought they would significantly enhance the impact of the paper. Although, if the authors answer that they do not want to do that, it is fine with me.

Finally “Contrive ways to add references to their own papers”. Well, if you are the referee it is generally that you happen to have a few relevant papers in the field. Of course asking for citations of papers that are not relevant is not acceptable.

Reply

5 Jane Rigby February 8, 2011 at 10:37 am

The ApJ used to have a nice webpage summarizing the editorial process — what a scientific editor does, how s/he selects a referee, what the expected times are for a response, and something about the duties of a referee. Very useful for novice authors, and a good reference for old hands when problems arose. I recently tried and failed to find that page; it appears to have been lost in the migration to IOP. I asked the editorial staff at the AAS, and was told they’d try to recover it.

Reply

6 Gus February 10, 2011 at 11:16 am

i think it was suppose to be here as the topics exist but are not linked to anything.

http://authors.aas.org/

7 Alan Stockton February 8, 2011 at 5:09 pm

One way I have found that helps me to follow these sorts of guidelines (especially “trying to write in the same tone you would use if the author was standing right in front of you”) is always to let my name be known to the authors (well, almost always–there have been one or two cases where I felt it was better not to do so, for specific reasons). I was led to do this after a conversation many years ago with George Abell (yes, of Abell cluster fame), who gave the reasons why he always signed his reports. If your name is attached, you are not likely to do a slipshod job, nor to insist on including your own references unless they are strictly relevant, nor indulge in vindictiveness. I am not aware of any major repercussions to my career from this practice, and often one gets a gracious mention by name in the acknowledgements.

Reply

8 fxt February 11, 2011 at 9:47 am

for the apj, it is the scientific editor’s decision whether or not to disclose the referee’s identify to the authors should a referee indicate that are willing to waive anonymity. this is probably true for other journals as well.

9 Mordecai-Mark Mac Low February 9, 2011 at 4:32 pm

Referees are also supposed to judge the significance of the paper, and make a recommendation on whether it is suitable for the journal it was submitted to. This aspect tends to be downplayed in the major astronomical journals, but is emphasized for journals like Nature or Phys. Rev. Letters.

Reply

10 Gus February 10, 2011 at 10:30 am

something like 80% of Nature submissions never go to referee to a great degree because of a significance test by editors (instead of peers). so i disagree that is has that much to do with “refereeing.”

so i think that “significance” is poorly defined at best or illogically defined at worst. At best you would be hard pressed to quantitatively grade the significance of a paper and at worst the use of significance by “exclusive” journals is means to recursively define that journal’s value. neither have anything IMHO to do with refereeing.

11 Nicolas February 9, 2011 at 5:42 pm

There are some interesting thoughts about peer-reviewing in A&A by the Editor at http://www.aanda.org/articles/aa/abs/2004/24/aa420300f/aa420300f.html
As a referee (in addition to the above suggestions made in this post) I usually check that all references in the text can be found in the bibliography. I also try to read the most cited reference in the paper if I don’t know it already for various reasons going from getting to know better a particular area to seeing what’s been done before etc.
I agree that it’s not good to redo all the calculations, but some effort should be spent to make sure that the research presented in the paper can be reproduced by others.

Reply

12 Anon February 9, 2011 at 11:58 pm

My guess would be that checking reference/bibliography agreement is an editing-level job that the referee does not have to do (as it doesn’t require any scientific knowledge), but I don’t know. I did once have a referee once catch a citation typo of mine (before I used bibtex), where I had the right volume/page, but wrong journal. I’m pretty sure this means the referee was one of the lead authors of the cited article, but maybe they were just really, really fastidious.

13 Gus February 10, 2011 at 11:03 am

i think that the admonition about redoing calculations is misguided. Redoing calculations is more often then not impossible because astronomy’s publishing model does not aide/enforce the inclusion of meaningful supporting data to a publication. So I would posit that more often then not a referee’s recalculations are a red herring because they don’t actually have the relevant data to test.

so I think a role for a referee is to evaluate which data is critical for the reproducibility (of analysis/conclusions) of that paper’s results and to test that as much of the *relevant* supporting data is included in the paper or referenced by link or supplementary material.

I can’t see this as being so hard: a paper with SED plots and fits should tabulate the SEDs data points (as plotted). I know the journals are looking into supporting such tabulations behind figures but even so a peer referee is in the optimal position (as they has the domain knowledge presupposed by being a peer reviewer) necessary to evaluate how well the analysis and conclusions of the paper are supported by the data contents of the paper.

Reply

14 fxt February 11, 2011 at 2:32 am

eugene parker’s “the martial art of scientific publication” is worth a read, or re-read.
http://acml.gsnu.ac.kr/recomm_links/parker.pdf
or
http://aas.org/career/ArtofSciPub.php

Reply

{ 2 trackbacks }

Leave a Comment

Previous post:

Next post: