Working Paper: CEPR ID: DP14710
Authors: Davidson Heath; Matthew Ringgenberg; Mehrdad Samadi; Ingrid M. Werner
Abstract: After a natural experiment is first used, other researchers often reuse the setting, examining different outcome variables. We examine the consequences of reusing an experimental setting using two extensively studied natural experiments, business combination laws and the Regulation SHO pilot. We apply multiple hypothesis corrections and our findings suggest many results in the existing literature are false positives. We provide guidelines for inference when an experiment is reused using simulation evidence for several popular empirical settings including difference-in-differences regressions, instrumental variables regressions, and regression discontinuity designs.
Keywords: false positive; identification; multiple hypothesis testing; natural experiments
JEL Codes: G1; G10
Edges that are evidenced by causal inference methods are in orange, and the rest are in light blue.
| Cause | Effect |
|---|---|
| Reusing natural experiments (C93) | Increases likelihood of false positives (C52) |
| Application of Romano-Wolf correction (C59) | Identifies false positives (C52) |
| Findings from business combination laws (K22) | Significant results emerge (C52) |
| Findings from regulation SHO (G18) | Significant results emerge (C52) |
| Adjusted p-values (C12) | Observed variations likely due to random chance (C90) |
| New hypotheses (C12) | Achieve credible inference (C20) |