Adversity and SATs Makes for an Interesting Scatterplot

WSJ-SATvAdversity-featured

The plans of the College Board to begin assigning SAT takers “adversity scores” so college admissions boards could take “diversity” into account in a seemingly more-objective way met with public outcry.

For one thing, there’s something unseemly about the implicit statement that a high score is not an achievement in the absence of adversity.  For another thing, adversity is so broad and individualized as to not be appropriate.  That is, a student who is advantaged by the College Board’s definition may be overcoming severe adversity in something else that the College Board knows nothing about.

Put these two considerations together, and you can see the injustice of telling a white student from Barrington High School that his 1240 SAT score is worth less than the 928 score of a student at Central High in Providence.  A teenager from Barrington can have a challenging home life, physical difficulties, psychological hurdles, and so on, while a teenager from Providence can have huge advantages that don’t register on a demographic survey.  We do grievous harm to our society if we start encouraging either to promote their disadvantages as a surreptitious way to maximize their advantages.

All that said, the Wall Street Journal has a neat interactive tool plotting schools’ median SAT scores on the y-axis and their adversity scores on the x-axis.  Applied to whole schools, and utilized by people with additional information about the communities, this could actually be a useful device for families as consumers of their children’s schools.

So, for example, my town of Tiverton is just shy of the national median SAT score, at 1030, and comes in right at the middle, with a 51 adversity score.  Westport, just over the Massachusetts line, does slightly better than median on the SAT, with 1060, despite being slightly more “adverse,” so to speak, at 55.  More dramatically, Barrington High School’s 1240 median SAT is almost exactly the average for schools that share its low adversity score of 1.  This compares with Providence Central High’s 800 SAT, which is much worse than the average for schools that share its 97 diversity.

The tool allows a leap to private schools, too.  Near Tiverton and Westport is a relatively low-cost Catholic high school, Bishop Stang.  There, an 1170 median SAT matches up to an adversity score of 61.  Such schools, in other words, offer both more diversity and relatively high scores.  La Salle Academy in Providence, by contrast, also has significantly better SAT results than nearby public schools, at 1120, but on adversity, it scores 39.  This looks more like an option for families interested in escaping some of the adversity of their city if they can afford it.

Readers should bring more skepticism to another feature of the interactive tool, although it is still intriguing from the perspective of a consumer.  The Wall Street Journal worked with Georgetown professor Jeff Strohl to weight the SAT scores to show how far schools are from the median among their adversity peers.  In other words, if we equalize things such that the median SAT score at 1-adversity schools counted the same as a the median SAT score at 99-adversity schools, how much does a particular school add to or fall short of the mark?

By this measure, Barrington is dramatically under-serving its students.  As mentioned above, its median SAT is exactly in the middle for schools similarly lacking adversity, but according to Strohl, that is entirely a function of its demographics.  With average adversity, the school’s SAT would be a substandard 1029.

When its SAT score is weighted, Central High School in Providence improves significantly, from 800 to 928, but this is only about half of the distance to the median.  In other words, not only do students at Central suffer from their adversity, but their school system is failing them above and beyond that challenge.

Again, to stress the point, these generalizations make too many assumptions and sweep aside all of the individualism that comes with educating teenagers.  Still, that sort of objection applies most sharply when the intended use of the tool is either the redistribution of wealth through government or the discounting of unfashionable applicants by an institution of higher education.  If we start to think of families as active consumers in an environment of educational freedom (as we should), many of these objections evaporate, and we just have another useful datapoint as we figure out where to send our children to be educated.



  • Joe Smith

    Ah, except Justin the RI high schools *mandate* every student take the SAT while that is not the case with schools in MA or private schools. However, MASS does require passing the MCAS Grade 10 (well with some exceptions) so the students who *do* end up taking the SAT have had a “it means something” standardized test prior to the SAT.

    In fairness, you could argue RI also mandates the PSAT – the only difference is it has no impact on students’ graduation.

    But I find it a bit confusing how you rightly cite the problem with the adversity index (that even College Board states should not be used for any kind of weighting or comparison) and then want to say but it has value as a consumer choice tool.

    Even worse is pointing to the Georgetown Center doing exactly what the College Board says researchers shouldn’t do. Not that the Georgetown Center would note that – their last work on 40 year ROI is fraught with methodological issues and they bury the “fine print” on page 22 of the appendix of that report.

    “If we start to think of families as active consumers in an environment of educational freedom (as we should), many of these objections evaporate, and we just have another useful datapoint as we figure out where to send our children to be educated”

    The fundamental problem is treating the ‘we’ as if all families value the same thing when it comes to education and in the same swath expecting taxpayers to pick up the tab for those variations without any say by the taxpayers. We point warning labels on cigarettes, educate on the dangers, and even allow market forces (health care premiums/insurance if allowed) to put some cost on that behavior yet people still do it.

    Should taxpayers subsidize ‘choice’ when the choice might not align with the larger taxpayers’ goal for an education (and even that is problematic to define clearly and measure accurately what the outcomes of publicly financed education is)? I know at least two families that send their kids away from one of the best public high schools in the state because they think their kids will have a better shot at an athletic scholarship at the private high school that has the perennial state champions in their kids’ sport.

    If they are paying the bill, that’s their call. If they want taxpayers to pick up some that bill, I’m not so sure on picking up the tab so their kids have a better shot at a college scholarship or to justify the $$ they have spent on their kids’ youth sports activities.

    • Justin Katz

      I find that a great deal of public policy argument put forth in order to protect or advance an area of government dominance takes the possibility of degrees out of the discussion. To wit, we shouldn’t allow educational freedom because then we’ll have to give all kinds of money to people sending their children to places that obviously shouldn’t be eligible.

      Well, that’s all in how you design the program. We can adjust the amount of money that families receive. We can define certain requirements for inclusion of providers. And so on.

      The next layer of argument to protect government programs has to do with the implicit “we know better.” I’m not sure I’m willing to declare, as a blanket statement, that my reason for prioritizing a school should be the same for everybody. Maybe a set of parents, knowing their children, is right to place a high value on the opportunity for athletic scholarships… to be able to get into and afford college… to further a career in athletics… or some other reason.

      You might say that is not what we intend to fund as taxpayers, but what do we intend to fund? One of my central reasons for choosing private schools for my children is that I strongly and fundamentally do not agree with much of the content we fund as taxpayers. I guess other people taking my money to pay for programs they can get their representatives to support is how the system works, but I don’t think creating a money trap for families to which to subject their children is just. This brings us back to adjusting the program; maybe there’s some line that a well-designed policy will find, so it doesn’t have to be all or nothing.

      As to your objection to the methodology of the index, as I said, seeing it as a tool for individual customer decision maker evaporates many of the problems. Individual consumers can adjust for the fact that an index doesn’t compare well across state lines and other factors. Big government control of our lives cannot fine tune so well.

      • Joe Smith

        Individual consumers can adjust for the fact that an index doesn’t compare well across state lines and other factors.

        That of course assumes the consumer has *any* clue on the limitations of the tool. It’s the same complaint about the RI “accountability” or Federal college scorecards.

        The problem is policy makers look for the most simplified manner to *help* (being generous as to motivation) “consumers” with some very complicated and contextual decisions using often the least amount of effort in terms of collecting and analyzing the data – often in response to issues they (or their predecessors) had a hand in making.

        College debt is an example.

        “takes the possibility of degrees out of the discussion” – no argument with you there. What I find troubling on both the public (usually union led) and “choice” sides is the all or nothing attitude.

        I also find it puzzling we would never have a discussion of “let’s build a county DPW organization or a county Sheriff department that only serves 1% of the county – who will be chosen through an opt-in lottery ideally to help the households who have the worst roads or most in need of law enforcement” Yet we have no problem building a parallel public school system with very little oversight that rarely leads to either shutting down its counterpart, leading to improving its counterpart, or shutting down itself because it has not proven to be better/more cost effective.

        • Justin Katz

          I kind of like the DPW comparison. Imagine if we had something like 17% of neighborhoods banding together to privately pave their own roads (without receiving any tax breaks). And suppose certain neighborhoods were particularly poorly served by their DPWs, but the Teamsters locals were so intransigent that there was no hope of improvement, so the best the voters could do was arrange for a public version of the private paving arrangements.

          • Joe Smith

            Right – so why don’t we push for “road paving” choice? Let’s start with some metrics on paving quality, cost per mile, etc. and the really low performing areas we would start by introducing competition by allowing a lottery for certain neighborhoods to pick a private paving company (or another town’s DPW crew) and the local town would pay the tab?

            My point is simply if your for choice when it comes to public provision of arguably non public goods (in the traditional economic sense, which very few goods are truly public goods), why stop at schools?

          • Justin Katz

            Sounds reasonable to me. If a road is going to be paved anyway and an alternative company or department can do it better and cheaper, that’s an obvious thing to do, isn’t it?

      • Joe Smith

        “You might say that is not what we intend to fund as taxpayers, but what do we intend to fund? ”

        Good question – this is a finding from a large scale review of voucher program in LA (one of the largest I think)

        “But one takeaway is that we shouldn’t expect parents to make choices that improve student academic achievement.”

        http://blogs.edweek.org/edweek/charterschoice/2017/11/precious_little_evidence_vouchers_improve_academic_achievement_research_finds.html

        I think it’s a larger question of what do we expect from an education. Conceptually the argument is a “basic” education is a positive externality so the market would “underproduce”, necessitating a combination of subsidies (public school option as a default) or choice subsidies and mandates (must attend school through age X or grade Y).

        i get your point on parents “valuing” other aspects of an education, but shouldn’t academic achievement be up there from the *public’s perspective” if public funds are involved?

        This brings us back to adjusting the program; maybe there’s some line that a well-designed policy will find, Again, no argument there.

        The problem is the policy folks fail to use objective data and analysis that often in designing policy nor do they want to face the harsh realities that schools in general can’t fix what may be wrong in the 80% of the time the student is NOT in school despite throwing more money and more mandates on the schools.

        Instead we want to think competing at the school level somehow fixes these societal/household issues. Again, studies of choice shows mixed results at best and that’s with the virtual analytical difficulty of disentangling the mechanisms of choice on the objective outcomes.

        and then there is this sobering study –

        https://www.wbur.org/hereandnow/2018/08/27/public-private-school-family-income-study

        • Justin Katz

          I can’t speak for all “policy folks,” but my vision of educational freedom is that it will help schools differentiate to stop devoting resources to fruitless endeavors and maybe devote some of them toward the actual problems that children have. I think there is consensus that schools should provide basic education, so that would be designed in as a baseline requirement.

Quantcast