Working Paper: NBER ID: w24017
Authors: Parag A. Pathak; Peng Shi
Abstract: Discrete choice demand models are widely used for counterfactual policy simulations, yet their out-of-sample performance is rarely assessed. This paper uses a large-scale policy change in Boston to investigate the performance of discrete choice models of school demand. In 2013, Boston Public Schools considered several new choice plans that differ in where applicants can apply. At the request of the mayor and district, we forecast the alternatives' effects by estimating discrete choice models. This work led to the adoption of a plan which significantly altered choice sets for thousands of applicants. Pathak and Shi (2014) update forecasts prior to the policy change and describe prediction targets involving access, travel, and unassigned students. Here, we assess how well these ex ante counterfactual predictions compare to actual outcome under the new choice sets. We find that a simple ad hoc model performs as well as the more complicated structural choice models for one of the two grades we examine. However, the structural models' inconsistent performance is largely due to prediction errors in applicant characteristics, which are auxiliary inputs. Once we condition on the actual applicant characteristics, the structural choice models outperform the ad hoc alternative in predicting both choice patterns and policy relevant outcomes. Moreover, refitting the models using the new choice data does not significantly improve their prediction accuracy, suggesting that the choice models are indeed “structural.” Our findings show that structural demand models can effectively predict counterfactual outcomes, as long there are accurate forecasts about auxiliary input variables.
Keywords: Structural Demand Models; Counterfactual Predictions; School Choice; Discrete Choice Models
JEL Codes: C10; C78; D12; I20
Edges that are evidenced by causal inference methods are in orange, and the rest are in light blue.
| Cause | Effect |
|---|---|
| structural choice models (C35) | predict equilibrium outcomes (D50) |
| accurate forecasts about auxiliary input variables (C53) | predict equilibrium outcomes (D50) |
| structural choice models (C35) | outperform ad hoc models (C52) |
| inaccuracies in auxiliary input variables (C20) | prediction errors (C52) |
| structural models (E10) | robust performance despite changes in choice sets (D11) |
| structural demand models (R22) | value in counterfactual analysis (D46) |