Working Paper: NBER ID: w29550
Authors: Joakim A. Weill; Matthieu Stigler; Olivier Deschenes; Michael R. Springborn
Abstract: The COVID-19 pandemic brought unprecedented policy responses and a large literature evaluating their impacts. This paper re-examines this literature and investigates the role of researchers' degrees-of-flexibility on the estimated effects of mobility-reducing policies on social-distancing behavior. We find that two-way fixed effects estimates are not robust to minor changes in usually-unexplored dimensions of the degree-of-flexibility space. While standard robustness tests based on the sequential addition of covariates are very stable, small changes in the outcome variable and its transformation lead to large and sometimes contradictory changes in the estimates, where the same policy can be found to significantly increase or decrease mobility. Yet, due to the large number of degrees-of-flexibility, one can focus on a set of results that appears stable, while ignoring problematic ones. We show that recently developed heterogeneity-robust difference-in-differences estimators only partially mitigate these issues, and discuss how a strategy of identifying the point at which a sequence of ever more-stringent robustness tests eventually fail could increase the credibility of policy evaluations.
Keywords: COVID-19; mobility-reducing policies; difference-in-differences; social distancing
JEL Codes: C18; C23; H75; I12; I18
Edges that are evidenced by causal inference methods are in orange, and the rest are in light blue.
| Cause | Effect |
|---|---|
| mobility-reducing policies (MRPs) (J62) | reductions in measured mobility (J62) |
| model specifications (C52) | contradictory results of MRPs (C59) |
| transformation of outcome variable (C29) | switch sign of estimated effect of MRPs (C51) |
| sequential addition of covariates (C32) | inadequate signaling of specification sensitivity (L15) |
| existing tools for DiD analysis (C22) | insufficient conclusions about causal impact of MRPs (C59) |
| pretrends in data (C22) | complicate causal interpretation of policy effects (D78) |
| heterogeneity-robust estimators (C51) | not fully address issues of causal inference (C90) |