Notes/Explanation

They Say: “Funding Not Key” – Prefer Baker

Prefer Baker, not Hanushek.

Spielberg 15 — Ben Spielberg, Research Assistant at the Full Employment Project at the Center on Budget and Policy Priorities, holds a B.S. in Mathematical and Computational Sciences from Stanford University, 2015 (“The Truth About School Funding,” 34justice—a scholarly blog, October 20^th, Available Online at https://34justice.com/2015/10/20/the-truth-about-school-funding/, Accessed 07-04-2017)

1) Mehlhorn devotes a lot of space to attacking Bruce Baker for editorializing. Baker certainly does have strong opinions, but I actually think it’s nice that he’s transparent about his perspective – all researchers have biases, and it’s in many ways preferable to know about them upfront. Baker’s work is strong and consistent with other recent research. The research Mehlhorn relies on – from Eric Hanushek, a member of the Right-wing Hoover Institution (note that Mehlhorn does not once mention Hanushek’s affiliation and biases) – is typically much older and a clear outlier (as I explained above).

They Say: “JJP Wrong – Hanushek”

Hanushek’s critique of JJP is wrong.

Baker 17 — Bruce D. Baker, Professor in the Department of Educational Theory, Policy, and Administration in the Graduate School of Education at Rutgers, The State University of New Jersey, former Associate Professor of Teaching and Leadership at the University of Kansas, holds an Ed.D. in Organization and Leadership from the Teachers College of Columbia University, 2017 (“Does Money Matter in Education? Second Edition,” Albert Shanker Institute, Available Online at http://www.shankerinstitute.org/sites/shanker/files/moneymatters_edition2.pdf, Accessed 06-14-2017, p. 10-11)

Additional compelling studies have been published since my review in 2011. In 2014 and 2015, Kirabo Jackson, Rucker Johnson and Claudia Persico (JJP) released a series of NBER working papers and articles summarizing their analyses of a uniquely constructed national data set in which they evaluate the long-term effects of selective, substantial infusions of funding to local public school districts, which occurred primarily in the 1970s and 1980s, on high school graduation rates and eventual adult income. Virtues of the JJP analysis include that the analysis provides clearer linkages than many prior studies between the mere presence of “school finance reform,” the extent to which school finance reform substantively changed the distribution of spending and other resources across schools and children, and the outcome effects of those changes. The authors also go beyond the usual short-run connections between changes in the level and distribution of funding, and changes in the level and distribution of test scores, to evaluate changes in the level and distribution of educational attainment, high school completion, adult wages, adult family income and the incidence of adult poverty.

To do so, the authors use data from the Panel Study of Income Dynamics on “roughly 15,000 PSID sample members born between 1955 and 1985, who have been followed into adulthood through 2011.” The authors’ analysis rests on the assumption that these individuals, and specific individuals among them, were differentially affected by the infusions of resources resulting from school finance reforms that occurred during their years in K-12 schooling. One methodological shortcoming of this long-term analysis is the imperfect connection between the treatment and the population that received that treatment.76 The authors matched childhood address data to school district boundaries to identify whether a child attended a district likely subject to additional funding as a result of court-mandated school finance reform. While imperfect, this approach creates a tighter link between the treatment and the treated than exists in many prior national, longitudinal and even state-specific school finance analyses.77

Regarding the effects of school finance reforms on long-term outcomes, the authors summarize their findings as follows:

“Thus, the estimated effect of a 22 percent increase in per-pupil spending throughout all 12 school-age years for low-income children is large enough to eliminate the education gap between children from low-income and nonpoor families. In relation to current spending levels (the average for 2012 was $12,600 per pupil), this would correspond to increasing per-pupil spending permanently by roughly $2,863 per student.

“Specifically, increasing per-pupil spending by 10 percent in all 12 school-age years increases the probability of high school graduation by 7 percentage points for all students, by roughly 10 percentage points for low-income children, and by 2.5 percentage points for nonpoor children.

“For children from low-income families, increasing perpupil spending by 10 percent in all 12 school-age years boosts adult hourly wages by $2.07 in 2000 dollars, or 13 percent.”78

The findings of this study have been met with some criticism. Specifically, Eric Hanushek has asserted that these findings of strong, positive longitudinal effects of school finance reforms on student outcomes, running between 1972 and 2011, are entirely inconsistent with his characterization of long-term trends in school spending and national average test scores. According to Hanushek, if the effects JJP describe are real, then the massive infusions of funding to public education over time would have mitigated achievement gaps and substantially driven up national averages. Hanushek explains:

“Their analysis covers schooling experiences for the period 1970-2010. Thus, it is useful to connect these estimates to actual funding patterns over the period. Between 1970 and 1990, real expenditure per pupil increased not by 10 percent but by over 84 percent. By 2000, this comparison with 1970 topped 100 percent, and it reached almost 150 percent by 2010. No amount of adjustment for special education, LEP, or what have you will make these extraordinary increases in school funding go away.

“If a ten percent increase yields the results calculated by Jackson, Johnson, and Persico, shouldn’t we have found all gaps gone (and even reversed) by now due to the actual funding increases? And, even with small effects on the non-poor, shouldn’t we have seen fairly dramatic improvements in overall educational and labor market outcomes? In reality, in the face of dramatic past increases in school funding, the gaps in attainment, high school graduation, and family poverty have remained significant, largely resisting any major improvement. And, the stagnating labor market performance for broad swaths of the population has captured considerable recent public and scholarly attention.”

Perhaps the most illogical assertion of Hanushek is that applying the effect of funding increases estimated by JJP to the actual long-term growth in national average per-pupil [end page 10] spending would lead to the elimination or reversal of achievement gaps. As such, since gaps have not been eliminated or reversed, JJP’s estimates must be wrong. Neither JJP’s nor Hanushek’s national average spending data indicate that all spending increases from 1970 to 2010 were targeted to all high-poverty districts nationwide. If Hanushek’s average spending increases were driven as much by increases in wealthy (low poverty/minority) districts as they were by increases in poorer districts, then gaps would likely remain constant, all else being equal.79 The identification of substantial gains in lifelong outcomes for children in districts that did experience increased funding indicates that greater gains perhaps could have been achieved for children in lower-wealth, higher-poverty communities, if funding increases had been more systematically targeted to those communities, nationwide, throughout the time period.

Thus, the critical reviewer must ask whether the data, methods and analytic approach applied by JJP are sufficiently more rigorous, and thus provide more compelling evidence, than the long-term trend exposition of Hanushek. The simple answer to that is yes.

A secondary critique offered by Hanushek is that the funding increases evaluated by JJP occurred largely in the 1970s and early 1980s, when overall spending was much lower, thus making marginal gains potentially more important than now, when spending has reached and stabilized at inefficiently high levels across the board. Notably, however, JJP’s analyses span a longer period than merely the early 1970s and also span multiple contexts of higher and lower spending over the period. While subsequent replications of JJP’s findings and further exploration of their data will provide additional insights, the current studies provide compelling evidence that school finance reforms can be leveraged to equalize educational and long-term economic opportunity.

Hanushek’s critique of JJP doesn’t disprove their results.

JJP 15 — C. Kirabo Jackson, Associate Professor of Human Development and Social Policy at Northwestern University, Faculty Fellow at the Institute for Policy Research, Faculty Research Fellow at the National Bureau of Economic Research, holds a Ph.D. in Economics from Harvard University, et al., with Rucker C. Johnson, Associate Professor at the Goldman School of Public Policy at the University of California-Berkeley, Faculty Research Fellow at the National Bureau of Economic Research, Faculty Research Fellow at the W.E.B. Du Bois Institute at Harvard University, Research Affiliate at the National Poverty Center at the University of Michigan, Research Affiliate at the Institute for Poverty Research at the University of Wisconsin, holds a Ph.D. in Economics from the University of Michigan, and Claudia Persico, Assistant Professor of the Economics of Education in the Department of Educational Leadership and Policy Analysis and Faculty Affiliate of the La Follette School of Public Affairs at the University of Wisconsin-Madison, Research Affiliate at Northwestern University, holds a Ph.D. in Human Development and Social Policy from the School of Education and Social Policy at Northwestern University, 2015 (“Money Does Matter After All,” Education Next, July 17^th, Available Online at http://educationnext.org/money-matter/, Accessed 07-04-2017)

We would like to thank Eric Hanushek for his comments and interest in our work. We appreciate the opportunity to offer a brief response. Hanushek provides an accurate description of our study and is correct that the methodological details matter. His critique, however, is not an objection to any of our methodological choices; he instead disputes our results. He states “while these [questions about measurement and how spending reactions to court decision is measured…] are important methodological issues, it is more useful to focus on the substance of their findings.” We take this as clear evidence that Hanushek finds our methodology sound. When the methods are sound, the results must be taken seriously. We appreciate that Hanushek has done so in this case. His single, important critique of our key results is the “time trend” argument. Following the summary of our findings below, we present the “time trend” argument and highlight its flaws. We then discuss how we overcome the problems of the previous studies on which Hanushek bases his opinions. Finally, we discuss how our results differ from previous literature because (a) existing studies suffered from biases, and (b) the spending increases analyzed in our analysis were spent on more productive inputs than the spending increases examined in other studies.

Overview of our findings:

In most states, prior to the 1970s, most resources spent on K–12 schooling were raised at the local level, through local property taxes (Howell and Miller 1997; Hoxby 1996). Because the local property tax base is typically higher in areas with higher home values, and there are persistently high levels of residential segregation by socioeconomic status, heavy reliance on local financing contributed to affluent districts’ ability to spend more per student. In response to large within-state differences in per-pupil spending across wealthy/high-income and poor districts, state supreme courts overturned school finance systems in 28 states between 1971 and 2010, and many states implemented legislative reforms that spawned important changes in public education funding. The goal of these school finance reforms (SFRs) was to increase spending levels in low-spending districts, and in many cases to reduce the differences in per-pupil school-spending levels across districts. By design, some districts experienced increases in per-pupil spending while others may have experienced decreases (Murray, Evans, and Schwab 1998; Card and Payne 2002; Hoxby 2001). Our key finding is that increased per-pupil spending, induced by court-ordered SFRs, increased high school graduation rates, educational attainment, earnings, and family incomes for children who attended school after these reforms were implemented in affected districts. We find larger effects for low-income children, such that these reforms narrowed adult socioeconomic attainment differences between those raised in low- vs. high-income families.

Hanushek’s “time trend” argument is illogical.

JJP 15 — C. Kirabo Jackson, Associate Professor of Human Development and Social Policy at Northwestern University, Faculty Fellow at the Institute for Policy Research, Faculty Research Fellow at the National Bureau of Economic Research, holds a Ph.D. in Economics from Harvard University, et al., with Rucker C. Johnson, Associate Professor at the Goldman School of Public Policy at the University of California-Berkeley, Faculty Research Fellow at the National Bureau of Economic Research, Faculty Research Fellow at the W.E.B. Du Bois Institute at Harvard University, Research Affiliate at the National Poverty Center at the University of Michigan, Research Affiliate at the Institute for Poverty Research at the University of Wisconsin, holds a Ph.D. in Economics from the University of Michigan, and Claudia Persico, Assistant Professor of the Economics of Education in the Department of Educational Leadership and Policy Analysis and Faculty Affiliate of the La Follette School of Public Affairs at the University of Wisconsin-Madison, Research Affiliate at Northwestern University, holds a Ph.D. in Human Development and Social Policy from the School of Education and Social Policy at Northwestern University, 2015 (“Money Does Matter After All,” Education Next, July 17^th, Available Online at http://educationnext.org/money-matter/, Accessed 07-04-2017)

The Problem with Hanushek’s “Time Trend” Critique:

Now that the reader should have a clear sense of our paper and its implications, we now describe the Hanushek “time trend” argument. Hanushek points out that school spending in the United States has increased substantially between 1970 and present day. As such, he argues that, if our results are correct and school spending really does improve student outcomes (with larger effects for low-income children), outcomes should have improved over time and achievement gaps by income should have been eliminated over this time period. He then argues that any improvements between 1970 and today have been small so that it is unlikely that our conclusion that school spending improves student outcomes is correct.

While this “time trend” argument is intuitive, it is flawed for two reasons. The first reason is that it relies on the same flawed understanding of our results outlined above (i.e., that eliminating differences across two broad income groups implies eliminating all differences by socioeconomic status). The second problem with this “time trend” argument is that it is a facile argument based on fuzzy (albeit intuitive) logic. We highlight the problems of his logic below.

To see the problems of Hanushek’s logic, consider the following true statistics: between 1960 and 2000 the rate of cigarette smoking for females decreased by more than 30 percent while the rate of deaths by lung cancer increased by more than 50 percent over the same time period.[1] An analysis of these time trends might lead one to infer that smoking reduces lung cancer. However, most informed readers can point out numerous flaws in looking at this time trend evidence and concluding that “if smoking causes lung cancer, then there should have been a large corresponding reduction in cancer rates so that there can be no link between smoking and lung cancer.” However, this is exactly the facile logic invoked by Hanushek regarding the effect of school spending on student achievement.

While there are several problems with this simplistic argument, to avoid going too deeply into the weeds we focus on the most important flaw in this “time trend” argument. Simply put, the “time series” argument will hold only if nothing else has changed between 1970 and present day. It is important to bear in mind that these spending increases occurred against the backdrop of countervailing influences, such as the rise in single-parent families, more highly concentrated poverty, deterioration of neighborhood conditions for low-income families, the exodus of the middle class to the suburbs, mass incarceration, the crack epidemic, changes in migration patterns, and others. Consider just one countervailing factor: the significant rise in segregation by income between neighborhoods over the past four decades. This increased residential segregation was driven mostly by families with school-age children (Owens 2015), a simple reflection that quality of local schooling options is a key driver of segregation. This significant increase in residential sorting by income among families with school-age children would have likely led to far greater disparities in school resources by community socioeconomic status had SFRs not been an effective leveling tool.

In short, 1970 and 2010 is not an “apples-to-apples” comparison, so there is no reason to expect that the correlation between aggregate spending and aggregate outcomes over such a long time span will yield anything resembling a “causal” relationship. In fact, the observation that using simple correlations over time is unlikely to yield the true “causal” relationship is exactly what motivated us to follow a different methodological approach. Our methodological approach allows for an “apples-to-apples” comparison and allows us to disentangle the effects of school spending from that of all these other countervailing forces. Though Hanushek has chosen not to discuss the methodological advances in our work, they are important, and methods matter.

How We Overcome These Problems to Facilitate “Apples-to-Apples” Comparisons:

We make several decisions in order to facilitate more of an apples-to-apples comparison. First, we use fine-grained data on individual students, rather than comparing the entire United States in 1970 to the entire United States in 2010. With these finer-grained data we are able to account for a variety of other factors that may have changed over time such as family structure, childhood poverty, and neighborhood factors. Using these finer grained data, our main approach is to compare the outcomes of individuals with similar background characteristics born in the same school district but who attended public schools during different years (when per-pupil spending levels may have been different) — i.e., an apples-to-apples comparison. However, this is not all that we do to ensure that our results yield real causal relationships.

In our paper, we point out that even if one can carefully account for several observable factors (as we do), correlating all actual changes in school spending with changes in student outcomes is unlikely to yield causal relationships. We point out that some spending changes are unrelated to other factors that may obscure the real effect on outcomes (i.e., clean spending changes), while other kinds of spending changes would clearly yield erroneous results (i.e., confounded spending changes). We point out that many of the spending changes analyzed in previous studies may have been of the confounded variety. To give an example of such confounded spending changes, consider the following example. The federal Elementary and Secondary Education Act allocates additional funding to school districts with a high percentage of low-income students, who are more likely to have poor educational outcomes for reasons unrelated to school spending. As such, school districts serving declining neighborhoods are also those that are most likely to receive additional per-pupil spending over time. Such compensatory policies generate a negative relationship between changes in school spending and student outcomes that obscure the true relationship between school spending and student outcomes. We avoid this kind of problem by focusing only on clean spending changes. Specifically, we focus on the relationship between external “shocks” to school spending and long-run adult outcomes. The “shocks” we use are the sudden unanticipated increases in school spending experienced by predominantly low-spending districts soon after passage of court-mandated SFR.

As discussed above, by design, very soon after a court-ordered SFR in a state, some districts experienced sudden unanticipated increases in per-pupil spending (i.e., shocks) while others may have experienced decreases. Our analytic approach compares the outcomes of individuals who attended school before these spending shocks to those of similar individuals from the school district after these spending shocks. The validity of our design relies on the idea that districts that experienced sudden increases in school spending right after the passage of a court-ordered SFR were not already improving in other ways in exactly those same years. For this reason, we spend much time in our work showing that the timing of these spending shocks has nothing to do with underlying neighborhood changes or changes in family characteristics, so that changes in outcomes due to these shocks are likely to reflect a causal relationship. We encourage interested readers to consult the full paper for further detail.

Reconciling our results with the Older Literature:

Even though we outline the faulty assumptions in Hanushek’s “time trend” argument, in the interest of good social science it is helpful for us to try to reconcile our findings with the simple time-series evidence. As we explain above, our results do not imply that a 22.7 percent increase will eliminate all differences by parental socioeconomic status. However, they do suggest the much more realistic prediction that one might observe some convergence across groups over time as school spending has increased. Indeed this has been the case. For example, Krueger (1998) uses data from the NAEP and documents test score increases over time, with large improvements for disadvantaged children from poor urban areas; the Current Population Survey shows declining dropout rates since 1975 for those from the lowest income quartile (Digest of Education Statistics, NCES 2012). Murnane (2013) finds that high school completion rates have been increasing since 1970 with larger increases for black and Hispanic students; Baum, Ma and Pavea (2013) find that postsecondary enrollment rates have been increasing since the 1980s, particularly for those from poor families. Contrary to Hanushek’s assertions, outcomes have improved. Importantly, these improvements are consistent with increase in school spending playing a key role.

Finally, Hanushek proposes three reasons why our estimates (if true) may not track the national time trends very well. His ideas are not novel — we considered, tested, and addressed them ourselves in the paper and herein. First, he says there may be diminishing marginal returns to schools spending. Indeed we find that this is the case in our study. Areas with the lowest initial spending levels were also those for which increased spending had the most pronounced positive effect. The second reason he cites is that spending induced by the courts might have large effects while spending not related to judicial rulings have small effects. Indeed we find evidence of this also. Specifically, spending increases associated with court-mandated reform are much more strongly related to improvement in measured school inputs (e.g., student-to-teacher ratios, length of the school year) than ordinary spending increases. There are a few explanations for this that we explore in our study. Finally, he proposes that our estimates are wrong. We propose an alternative: the time series evidence Hanushek relies on does not reflect a causal relationship. Indeed in our larger study, we show that simple correlations are obscured by a variety of other factors that also influence student outcomes. We also present numerous pieces of analysis in our larger study that support a causal interpretation of our results.

To be clear, we do not think that our study is the final word on the question of whether increasing school spending will improve student outcomes in all contexts. As Hanushek himself concedes “none of this discussion suggests that money never matters. Or that money cannot matter.” Here we will make a similar concession; none of what we show suggests that money always matters. We show that money did matter and that it mattered quite a lot. What our study does is dispels the notion that school spending does not matter, so that one must look only at how it is spent. We find that money does matter and how it is spent matters. Contrary to Hanushek’s claims, our findings do not let policymakers off the hook. Our findings suggest that it is extremely important that money is allocated effectively and also that it is allocated equitably so that all schools have the resources necessary to help all children succeed.