Making the rounds in the world of education policy is a study from three academic economists that is (I believe) the first to find an adverse effect for students who use school choice programs to attend private schools.  An initial look at the analysts, the program analyzed, and the methodology for reaching a conclusion suggest that it would be presumptuous take its results at face value, particularly inasmuch as its negative finding is an outlier compared with other studies.

The analysts: Two of the three economists represent half of the team members of the Institute for Innovation in Public School Choice (IIPSC).  Although the organization does not say so, explicitly, its statements and work strongly suggest that it is mainly concerned with policies allowing school choice within the public school system.  In other words, they’re of the “fix the system” school of education reform, which is arguably hostile to broad school choice that changes the emphasis from a government-branded education system to government’s simply facilitating education, no matter who performs it.  And, indeed, New Orleans is the first city they list in their FAQs as a place in which the institute has “broad partnerships,” so the economists’ study of Louisiana is directly relevant to their relationships.

The program analyzed: The Louisiana voucher system has some complicating dynamics.  A student doesn’t simply pick a school that has an open seat and apply for some of the funding that would otherwise go to his or her public school.  Rather, students list some private schools that they’d like to attend, and given the availability of both seats and vouchers, they are entered into a lottery.  Then, the voucher students, and only the voucher students, at the private schools have to take the standardized tests that public-school students have to take.  This is the basis for the comparison.

These factors might affect the results on both the school side and the student side.  On the school side, the vouchers aren’t just another form of payment; they come with strings and require the private schools to apply to be participants.  On the student side, the tests may not be fully aligned with their classroom studies, and in any event, more than half of their classmates don’t even have to take them, which surely affects their scores.

Moreover, the scores that the economists analyzed were for tests taken in the spring of 2013 by students who were new to the private schools starting in fall 2012.  There’s little room for adjustment in that amount of time.

The methodology:  Nowhere, that I spotted, does the report present the overall results for students in the voucher program versus the public schools from which students were drawn.  Rather, the conclusion that vouchers harm students’ achievement on the standardized test comes from situations in which the analysts were able to find comparable students who did and didn’t receive vouchers, which (if I’m reading the numbers correctly) narrows the range of voucher students who are even considered to around 15% of the total.  That is, the scores of 85% of participating students aren’t even considered.

This could skew the results in ways that the report doesn’t capture.  For example, any students who had applied in the previous year, but who didn’t win the lottery that year, were not included.  Consider two highly motivated students who receive the exact same scores on all of the tests, one of whom applied for vouchers for two years and got one the second year and the other of whom just applied this year and didn’t win the lottery.  The student who didn’t win the lottery, this time around, will count toward the scores of public school students, but the student who did win won’t count for the private schools because he or she wasn’t a first-time applicant.

This is just one easy-to-conceptualize example.  The filter that the authors of the study apply to the voucher-receiving population may affect the results in other ways requiring deeper analysis.  When comparing students, for example, they looked at their first-choice schools.  One would have to dig into the data almost to the level of the individual student to see whether entry into first choice schools versus less-desired schools had an effect on both those winning the lottery and those losing it.

Ultimately, the whole argument for broad school choice is that families know what’s best for their children at any particular moment and for the long run.  By this measure, the report acknowledges that voucher students’ parents report high levels of satisfaction with the program.  That satisfaction may translate into better scores on this particular standardized test after another year of school-and-student adjustment, or it may take longer to prove itself in other statistics, like graduation and more-universal standardized tests.