The Inside Scoop on NSF Review Panels

This is a guest post by an anonymous contributor.

There has been a lot of talk recently about the effect that sequester cuts have had on the funding situation in astronomy (and science in general).  Our field depends greatly on federal grant money (jobs! job! jobs!) and the pot is shrinking.  This means that every grant proposal you submit has a smaller chance of being approved and, unless you have been on a review panel, you may think the whole process of having your grant proposal reviewed is mysterious and random.

Over the past few years I have served on three NSF review panels.  Each time was for a different sub-topic with a different program officer (PO), so I have had a broad enough feel that I can summarize what they had in common and be able to generally comment on the process.

The process

The first thing to consider is what happens to your proposal once you submit it and it gets assigned to a review panel.  A panel typically has somewhere between six and eight reviewers and has ~20 proposals assigned to it.  Therefore each panelist is asked to read that many proposals, each ~40-60 pages long (1 page summary, 15 page science justification, budget and justification, data management plan, mentoring plan, letters of support).  Of those, a panelist is asked to write a review (which gets to be read by the PI) for about six to eight.  Those eight proposals will usually be read very carefully by the reviewer, who will write a lengthy commentary on the strengths and weaknesses of the intellectual merit and broader impacts. A given proposal will typically receive two such detailed reviews.  In addition, each proposal will have a reviewer that is assigned as scribe, which means writing down all that is said during the discussion of that proposal.  Despite best intentions, reviewers will typically read their assigned proposals carefully, and the others, well… fatigue sets in, and they don’t get read as carefully.

So, when writing your proposal, you must approach the process assuming that it won’t necessarily be read very carefully.  Be crystal clear and explicit and be sure not to bury important information in places where it may be missed.  If a reviewer missed something, it is because you were not clear enough.  On the other hand, you must also make sure you are consistent and coherent throughout, because a careful reviewer will still pick up on those finer issues.

More advice: A picture is worth a thousand words! Reviewers appreciate figures and descriptive captions.  Your figure caption should not just explain the axes and symbols (or at all if it is obvious), but should also tell the reader why the figure is important, summarizing the main point. Tell a story.  Check your spelling.  Proof read.

During the panel review, on the first day, each proposal is discussed briefly (for ~10-15 minutes, usually).  The discussion is usually led by the primary reviewers but is then opened up to the rest of the panel and the scribe works to capture all the ideas expressed in the discussion in order to write a panel summary that truly reflects the panel discussion and consensus (or, occasionally, lack thereof).

It turns out that this panel summary is one of the most important parts of the process and the second day of the panel is spent reading and editing each-others’ summaries and signing off on them once all reviewers agree that they reflect accurately what was said.  PIs can learn a lot about how their proposal was perceived by carefully reading the panel summary. The summaries are also important for the program officers when they recommend for funding the top ranked proposals to their higher-ups.

The sad reality is that there are 2-3 times more worthy proposals than can be funded.  And frequently the panel discussions aim to offer advice and suggestions for improvement, assuming that the PI will re-submit. So if you didn’t get funded don’t despair.  Come back again having hopefully considered the comments.

Broader Impacts

I found the broader impacts (BI) components to be the hardest aspect of a proposal to evaluate objectively. While excellent BIs are usually obvious to everyone, different reviewers have different standards/criteria/definitions of what consists of a strong BI statement (e.g., some want the BI to be relevant to the science proposal, others appreciate creative efforts to engage people/society regardless of the relevance to the science proposal).  Still, weak, boilerplate, or poorly thought out broader impacts do get penalized.  In particular, reviewers are not impressed with professors listing their teaching duties as a BI.  In the end, the science case is still the determining factor but outstanding broader impacts will get a boost.  And given the limited number of proposals that can be approved, often a strong BI gets a proposal over the edge.

The Money Part

Ironically, although the entire point of the proposal is to ask for funding, how much you ask for doesn’t seem to be heavily scrutinized in the early stages.  It’s all about the science. Focus on writing a highly competitive proposal and afterwards the budget will be examined and probably trimmed of excesses.  So, ask for what you need to get the work done, and you should be fine.

My Impression

The thing that impressed me the most about the NSF proposal review process was that although it seemed chaotic at first, consensus is usually reached.  My confidence was boosted by the examples of a few proposals that straddled categories and were reviewed by more than one panel, yet received similar rankings and comments.

The bottom line is that while the process is not perfect, proposers should be confident that it is one of the highest integrity.  But, as with anything, it could be improved and my observations below include suggestions on how we can all participate to make it better.

• One of my biggest frustrations with the NSF review process is that very few astronomers in senior positions seem to sit on panels.  That means that if you are a junior faculty, your proposal is being reviewed by postdocs.  Too many postdocs and not enough faculty means that the reviewers cannot assess whether a project is feasible for faculty members, especially at RUIs (Research in Undergraduate Institutions) where teaching loads are high.  ADVICE: If you are faculty and are asked to be on a review panel, say YES.

• On a similar note (although this may only have been true based on my limited experience) I found that the panels were heavily tilted toward observers and there were very few theorists.  This means that technical details (coding details, algorithm choice, moving mesh vs. SPH vs. MHD vs. N-body vs. whatever, I have no idea!) of theoretical proposals are difficult to understand and evaluate.  ADVICE (again): If you are a theorist, either volunteer or just say “yes” when asked.

• On another similar note, I used to think that it was an honor to be asked to serve on a review panel, because it meant that someone recognized my contributions to my field and thinks that I’m an expert.  Since it arrives as an invitation, it may not occur to junior people that program officers are desperate to fill panels. ADVICE: Don’t wait around to be asked. Volunteer!  Seriously, just send an email to program officers saying you are available.

Finally, while I was serving on these panels it came up a few times that the program officers are often subjected to irate complaining phone calls from unfunded PIs.  I found this shocking and feel the need to wag my finger a bit at my colleagues for this kind of behavior.  The budget situation is such that only ~10% (or less) of all proposals will be funded. That means many very good proposals don’t even get considered for funding.  There are many factors that you cannot control (like the quality of your competition!).  It takes a special kind of arrogance to think that you are above the system by which we all play.  It isn’t a perfect system, and it is terribly hamstrung by budget and congressional constraints, but it is a fair system.  Of that I can be sure.  A lot of people work very hard to ensure that it is a fair process (almost to an absurd level of care and diligence).  And so it is okay to be disappointed, but it is not okay to be angry at the PO if your proposal was not funded.

You can ask for constructive criticism and feedback, which is usually already available through the panel summary.  Consider the criticism, get feedback from colleagues, etc., resubmit and good luck!

If you’ve sat on a review panel, share some of your insight! Does your experience differ from this one? For those applying for grants, what have you learned about the process? Talk about it  in the comments!