Working Paper: NBER ID: w21748
Authors: Joshua Angrist; Peter Hull; Parag A. Pathak; Christopher Walters
Abstract: Conventional value-added models (VAMs) compare average test scores across schools after regression-adjusting for students’ demographic characteristics and previous scores. This paper tests for VAM bias using a procedure that asks whether VAM estimates accurately predict the achievement consequences of random assignment to specific schools. Test results from admissions lotteries in Boston suggest conventional VAM estimates are biased, which motivates the development of a hierarchical model describing the joint distribution of school value-added, bias, and lottery compliance. We use this model to assess the substantive importance of bias in conventional VAM estimates and to construct hybrid value-added estimates that optimally combine ordinary least squares and lottery-based instrumental variables estimates of VAM parameters. The hybrid estimation strategy provides a general recipe for combining non-experimental and quasi-experimental estimates. While still biased, hybrid school value-added estimates have lower mean squared error than conventional VAM estimates. Simulations calibrated to the Boston data show that, bias notwithstanding, policy decisions based on conventional VAMs are likely to generate substantial achievement gains. Hybrid estimates that incorporate lotteries yield further gains.
Keywords: Value-Added Models; School Effectiveness; Lottery-Based Estimation; Educational Policy
JEL Codes: I20; J24
Edges that are evidenced by causal inference methods are in orange, and the rest are in light blue.
| Cause | Effect |
|---|---|
| conventional VAM estimates are biased (C51) | lottery-based test reveals moderate but statistically significant bias in these estimates (H27) |
| conventional VAM estimates do not accurately predict the effects of randomized admissions at schools with lotteries (C52) | bias present in these estimates (C51) |
| hybrid estimation strategy reduces mean squared error (MSE) compared to conventional VAM estimates (C36) | improves policy targeting (J68) |
| VAM estimates can guide policy decisions effectively (C54) | informed by hybrid estimates that incorporate lottery data (H27) |