Working Paper: NBER ID: w18038
Authors: Elizabeth U. Cascio; Douglas O. Staiger
Abstract: Educational interventions are often evaluated and compared on the basis of their impacts on test scores. Decades of research have produced two empirical regularities: interventions in later grades tend to have smaller effects than the same interventions in earlier grades, and the test score impacts of early educational interventions almost universally "fade out" over time. This paper explores whether these empirical regularities are an artifact of the common practice of rescaling test scores in terms of a student's position in a widening distribution of knowledge. If a standard deviation in test scores in later grades translates into a larger difference in knowledge, an intervention's effect on normalized test scores may fall even as its effect on knowledge does not. We evaluate this hypothesis by fitting a model of education production to correlations in test scores across grades and with college-going using both administrative and survey data. Our results imply that the variance in knowledge does indeed rise as children progress through school, but not enough for test score normalization to fully explain these empirical regularities.
Keywords: Educational interventions; Test scores; Knowledge accumulation; Statistical artifacts
JEL Codes: I20; I21; I28; J24
Edges that are evidenced by causal inference methods are in orange, and the rest are in light blue.
| Cause | Effect |
|---|---|
| Interventions in later grades (I24) | Smaller effects on normalized test scores (C29) |
| Variance in knowledge (D80) | Smaller effects on normalized test scores (C29) |
| Fadeout of early educational interventions (I21) | Test score impacts (I24) |
| Normalization of test scores (C46) | Misrepresentation of persistence of knowledge gains (D29) |
| Knowledge gains (D80) | Test score impacts (I24) |