This is a guest post by Emily Levesque, Rachel Bezanson, and Grant Tremblay. Emily is an Assistant Professor at the University of Washington, Rachel is a Hubble Fellow at the University of Arizona, and Grant is an Einstein Fellow at Yale. Their post below references this paper.
Last week, AAS President Meg Urry issued an open letter regarding use of the Physics GRE, a required element of applications to the vast majority of astronomy graduate degree programs in the United States.
Meg’s letter questions whether or not the General and Physics GREs are useful predictive tools for future success in graduate school and beyond. The problems surrounding standardized tests and their apparently inherent bias (with strong correlations between scores and the gender, race, and socioeconomic status of test-takers) are well-established (check out Helms 2009, Steele & Aronson 1995, and Miller & Stassun 2014, among others). Standardized exams also preferentially exclude students with disabilities. Finally, the General and Physics GREs are extremely expensive; between registration and score report fees for these two exams, an applicant applying to ten astronomy graduate programs in the U.S. must spend over $500 just on test-taking, in addition to the potential cost of traveling to a testing center. This financial burden is profoundly exclusionary, and likely imposes a socioeconomic pre-selection on applicant pools across the board.
These established equity shortcomings alone should give committees pause when considering the use of the General and Physics GREs in admissions decisions. However, most U.S. astronomy graduate programs still require these exams as part of a complete application for admission, and many will apply either formal or informal “cutoff” scores when considering applications. Applicants below such a cutoff are either eliminated entirely or expected to present exceptional credentials in other areas to “compensate” for a poor Physics GRE score (what’s more, even the rumor of a cutoff may convince otherwise promising students with low Physics GRE scores to opt out, not investing the time and financial burden of applying to a program that they believe won’t even look at their application.) For the Physics GRE in particular, this continued practice is based on the belief that the test is a useful gauge of whether applicants will be “successful”, itself a nebulous and hard-to-define concept. Perhaps because of this, and because of the unique niche of using the Physics GRE in astronomy graduate admissions, there has never been any study, formal or informal, of whether the Physics GRE is truly an effective predictor of “success” in astronomy.
To that end, we circulated a short, unscientific, and informal questionnaire to 271 people who have held either a Hubble, Einstein, Sagan, Jansky, or NSF prize postdoctoral fellowship between 2010-2015. We asked poll respondents to report their Physics GRE scores. We have now presented an informal, non-refereed summary of the results, which you can find here. Our goal with this work was not to editorialize. We do hope, however, that this non-scientific study might generate some discussion in our community.
We want to make absolutely clear that sending this poll only to “prize” Fellows must not in any way be interpreted as a belief that receiving such a fellowship is the best or only metric of “success” in astronomy. Holding a prize fellowship is an narrow, incomplete, and non-inclusive definition of “success”. However, the expected selection effects imposed by such a limited definition of “success” are interesting to consider. For example, people accepted into competitive graduate research programs, known for traditionally utilizing metrics such as a Physics GRE cutoff in admissions standards, will be overrepresented in our sample. If the Physics GRE is not a strong predictor of “success” even in this already-biased sample, we can reasonably expect trends to be even weaker for a more inclusively defined group of successful astronomers.
It is therefore an interesting experiment: how did recent prize fellowship winners perform on the Physics GRE?
The number of future prize fellowship winners that would be eliminated from your applicant pool, if your graduate committee institutes a hard cutoff threshold in Physics GRE performance. From Levesque, Bezanson, & Tremblay (2015)
We found no statistically sound evidence that the Physics GRE can be used as a reliable predictor of future “success”, or as an effective filter when trying to narrow down an applicant pool. Of the 149 prize fellows that responded to our questionnaire and reported scores, 43.6% (65 people) would have been rejected from a graduate school applying a strict 60th-percentile cutoff to its applicants. There is also a disproportionate gender impact; this same cutoff would reject 60.4% of all female fellows (32 people). We were unfortunately not able to collect additional demographic data from our respondents such as race or ethnicity since this would risk compromising the anonymity of some respondents (itself an extremely problematic fact); however, Prof. Casey Miller illustrated in his talk at the APS 2015 Physics Department Chairs Conference that the Physics GRE shows similar gender and race correlations to other standardized tests. Any percentile cutoff at all would result in the rejection of multiple prize fellows.
The Physics GRE also showed no correlation with the number of first author papers written either during graduate school or upon exiting graduate school. Again, while first author papers are a narrow and incomplete gauge of research productivity, one would reasonably expect at least some correlation between first author papers and PGRE score if the PGRE was an effective predictor of research performance. Instead, in a histogram sorting prize fellows by the number of papers written during or exiting grad school, the general shape of the histogram and the average number of publications are both insensitive to PGRE cutoff score.
Number of first-author papers written upon finishing graduate school. The general shape of the histogram and the average number of publications is insensitive to PGRE Score. From Levesque, Bezanson, & Tremblay (2015)
Given both these data and past studies on the use of standardized tests, we join the AAS Council in recommending that graduate programs eliminate the GRE and PGRE as a required metric of evaluation for graduate applicants.