I had the good fortune to be in the right place at the right time when Sara Goldrick-Rab, Sociologist and UW-Madison professor, took her first look at a provocative Brookings report on school vouchers. The study concluded that an experiment conducted among participants in the New York School Choice Scholarships Foundation Program had no overall impacts…except in the case of African American students. The report said African American students who participated in the school voucher program were more likely to go to college, more like to go full-time, and more likely to attend selective colleges.
Goldrick-Rab (full disclosure: Sara is a mentor and friend although its not a professional consideration for either of us; the work is the work) is a serious methodologist. Her engagement with the authors of that Brookings report unfolded, real-time on Twitter. In an era of debate about open access research, this “back talk” discussion among researchers about the finer points of analysis in an public, accessible space was quite exciting. So much so I took to this blog to write about the way we cultural construct quantitative data and maths as objective and neutral.
To the point of the inherently subjective decisions all researchers make in the dialectical process that is research, Sara has written a detailed response to the Brookings report. She was kind enough to share it with me ahead of its publication.
In her response, Sara expounds on her issues with Chingos and Peterson overstating the importance of the effect of vouchers on African-Americans. Sara finds:
The study identifies no overall impacts of the voucher offer, but the authors report and emphasize large positive impacts for African American students, including increases in college attendance, full-time enrollment, and attendance at private, selective institutions of higher education. This strong focus on positive impacts for a single subgroup of students is not warranted. There are no statistically significant differences in the estimated impact for African Americans as compared to other students, there is important but unmentioned measurement error in the dependent variables (college attendance outcomes) affecting the precision of those estimates and likely moving at least some of them out of the realm of statistical significance, the authors fail to demonstrate any estimated negative effects that could help explain the average null results, and there are previously existing differences between the African American treatment and control groups on factors known to matter for college attendance (e.g., parental education). Therefore, contrary to the report’s claim, the evidence presented suggests that in this New York City program, school vouchers did not improve college enrollment rates among all students or even among a selected subgroup of students.
I am going to do a quick technical aside. This can be skipped. As a graduate student I was surprised to discover that one of the main points of contention was the failure to reject null for interactions effects. I could be wrong but I suspect that is because the positive finding was unexpected and not the one being tested for. But, since it was the only positive finding it was reported as if it was the initial hypothesis? If that had been the case rejecting null would have been in the initial steps? I’d love more egghead dialogue on that with my fellow egghead readers.
The finer points of Sara’s critique may be of little consequence to most readers, but I encourage you to skim the article for the gist. I do so because in it is a cautionary tale for all researchers, certainly, but just as importantly there is a cautionary tale for all those whose lives are shaped by the polices enacted because of research. In short, the story of the making of research affects us all.
Education research has been plagued with charges of weak methodology and ideologically driven research. I do not claim, at all, that this is the case with the Brookings report. In the greater field of research, however, there is something to be said for our ideologically driven hunger for research that confirms what we suspect or desire to be true. In fancy terms, that is called confirmation bias. If you are lucky and you do this work someone along the way sits you down and explains the difference between social science and advocacy. The two are not incompatible — Marx, DuBois, Dewey are all theoretical giants who ventured into policy suggestions — but the two should not be conflated.
You probably remember the scientific method from middle school science class. It still guides all levels of research: you formulate a good guess about what you think is going on based on available theories (a hypothesis), you test the hypothesis, you confirm or deny the validity of the hypothesis, you do it all over again.
That is a long, messy process for the world of social policy where expediency is valued. Education is under siege and many of us, all along the spectrum of political orientation, feel a need to provide a rationale for our prescriptions to “fix” public education in this country. In that mad dash towards data that supports the solution we can sell to the public, to politicians, and to ourselves, we must be careful not to short circuit the scientific method.
We should also charge ourselves with becoming better consumers of data. Ours is being called the era of big data. Others say data, and how its aggregated and used to shape opportunities and experiences differentially, is the civil rights issue of our day. Education research is an example of an area where the consequences of this can be disastrous if all that data is misunderstood, rushed, or improperly applied. That’s how we get 8th graders who have had a different curriculum every semester for the past five years. It’s why the “new math” becomes the old math before anyone figures out if children actually learned any math. It’s why kids wake up at 5 AM to catch buses to participate in a voucher lottery.
There are real implications for our research in education that is not true of other fields. The urgency of our situation means we should take more care with the work we produce and consume, not less.
Research can always be refined and debated. I take less issue with that than I do the way the report was positioned as irrefutable evidence for a policy position on vouchers and minority students. That is not entirely the fault of the researchers but it is a consequence of how research is reported by think tanks and regurgitated by the press. As we know from many high profile examples, it is difficult to dislodge a “fact” once it is embedded in the popular imagination. Good, bad, or indifferent these findings were widely reported. I saw them rehashed, absent any analysis or informed critique, by mainstream news outlets, advocacy groups, even a few civil rights groups.
Experience also suggests that the informed critique of those findings are less likely to be reported. And that is the cautionary tale.
Finally, I’ll say if you ever have the chance to take a methods class with Sara Goldrick-Rab, do it.
Or, follow her on Twitter. Class is always in session.