AJE Feature | Can Student Growth Data Change the Way Americans Choose Their Schools? by David M. Houston & Jeffrey R. Henig
The full-length American Journal of Education article “The Effects of Student Growth Data on School District Choice: Evidence from a Survey Experiment” by David M. Houston & Jeffrey R. Henig can be accessed here.
Our study is motivated by two basic observations. First, average student growth is arguably a better way to measure schools’ and districts’ contributions to student learning than average student achievement at a single point in time. Second, the relationship between student demographics and growth is much weaker than the relationship between student demographics and achievement. These two observations prompt our primary research question: Would the distribution of academic performance data based on student growth influence individuals’ school district preferences in ways that run counter to the conventional wisdom that the “best” districts are almost always the whitest and most affluent districts?
To answer this question, we conducted an online survey experiment to assess whether the provision of different types of academic performance information affects participants’ choices between a set of school districts. As part of the survey, participants engage in a simulation in which they are asked to imagine themselves as parents moving to a new metropolitan area. Participants then indicate their preferred choice between the five largest school districts in that area. To guide this decision, all participants receive information on the demographic characteristics of the districts. In addition, some participants are randomly assigned to receive some form of academic performance information: either average student achievement, average student growth, both, or neither. This process is repeated for the metro areas of the nation’s five largest cities (according to the U.S. Census Bureau’s 2017 estimates): New York, Los Angeles, Chicago, Houston, and Phoenix.
Average student growth (i.e., the rate of improvement in students’ academic performance over time) offers two chief advantages over more traditional measures of average student achievement (i.e., student academic performance at a single point in time). First, student growth arguably provides a more accurate and useful—if still imperfect—indicator of schools’ and districts’ contributions to student learning. Second, compared to student achievement, student growth is less tied to the racial/ethnic and economic composition of the student body. Average student growth data allow educators and the public to revisit the conventional thinking that the best school districts and the most affluent school districts are one and the same. They make it possible to recognize high performing districts that serve a broad range of students, from the severely disadvantaged to the exceptionally privileged. Perhaps most importantly, a growth-based understanding of educational performance could potentially alter how families choose schools and districts for their children in ways that reduce the racial/ethnic and economic segregation that shapes American education systems.
Taking advantage of the comparatively weak relationship between average student growth and districts’ racial/ethnic and economic compositions, we explore whether individuals who are given average growth data are more likely to choose less white and less affluent school districts than their peers who receive either no academic performance data or only average achievement data. On one hand, existing research indicates that various perceptual filters can make people impervious to new information. The effects of providing academic performance information—of any kind—may pale in comparison to the effects of providing information about the demographic composition of the student body. It could also be the case that people are simply more interested in student achievement than student growth when making choices between districts, rendering the provision of the latter ineffectual. On the other hand, there is also a growing literature on the ability to influence individuals’ choices by altering the content or presentation of relevant information. In most of the metropolitan areas that we consider, there are relatively high growth districts that serve a disproportionate number of disadvantaged students. By emphasizing student growth rather than (or in addition to) student achievement, it may be possible to encourage some people to consider districts that they would otherwise rule out.
Our results suggest that the provision of average student growth data can cause individuals to choose less white and less affluent school districts from a set of real options in a series of metropolitan areas. Furthermore, the provision of both average achievement data and average growth data can cause individuals to choose less white and less affluent districts than the provision of achievement data alone. The magnitudes of these effects are modest to moderate: typically between one-tenth and two-tenths of a standard deviation. Considering the simplified, abstract nature of the survey experiment, these estimates may overstate the effect sizes that we would observe in more realistic settings. However, given the durability of district-level racial/ethnic and economic segregation in the U.S., even small effects could be worth pursuing. This may be especially true if the intervention is relatively inexpensive and easy to implement—such as adjusting the kinds of academic performance information that districts, states, and non-governmental organizations emphasize when they communicate with the public.
These results appear to be dependent on one important condition: the presence of a relatively high growth district that serves a relatively disadvantaged student body. In the absence of such a district (e.g., among the five largest districts in the Phoenix area), we do not observe a clear effect of the provision of growth data on the racial/ethnic and economic compositions of participants’ district choices. Moreover, the magnitude of the effect appears to be dependent on the overall level of demographic variation between districts. Among the five largest districts in the New York area, the highest achievement district is overwhelmingly white while the highest growth district educates mostly students of color, resulting in a large average difference in the racial compositions of the district choices made by participants in the achievement and growth groups. By contrast, there is less district-level demographic variation in Houston, resulting in the same general pattern of results but much smaller average effect sizes.
There are some obvious limitations to our study. Chiefly, we examine school district choice in the context of an online survey experiment. The act of choosing a district for one’s child is both more constrained (by resources, mobility, employment, discrimination, etc.) and more complex (featuring a far greater variety of information and much higher stakes) than presented in our highly stylized experimental environment. While our experiment indicates that the provision of growth data causes individuals to prefer less white and less affluent districts on average from a set of options, the precise magnitudes of these effects are unique to our specific experimental design. However, we do think our study provides useful insight to researchers examining the effects of distributing student growth data in more realistic settings as well as educational leaders making decisions about how to measure and report information on academic performance. As districts and states respond to federal requirements to create and distribute public report cards on state, district, and school performance and progress, it will be useful to consider how choices of measurement and emphasis can influence families’ behavior. The same is true for districts developing and employing universal enrollment systems in which families use a common application to rank school preferences. We also believe that our study could inform the work of non-governmental organizations, such as GreatSchools.org, that provide data about academic performance to the community.
This experiment also contributes to our broader understanding of the role of information in residential and school choice. No single metric can capture all relevant aspects of a complex, multi-dimensional concept like educational quality. When making choices about which measurements to employ, which results to distribute, and which elements of those results to emphasize, educational institutions need to be aware of how these metrics can be shaped not only by variation in the relevant construct at the institutional level—schools’ and districts’ effects on student outcomes—but also by systematic variation in the advantages and disadvantages experienced by the individuals who comprise those institutions. When we inaccurately attribute differences in educational quality to school districts because of the students they serve rather than their effectiveness in serving those students, we shortchange both district and student. When that misattribution maps closely to racial/ethnic and economic lines, we exacerbate long-standing inequalities.
