Working Paper: NBER ID: w29843
Authors: Michael L. Anderson; Jeremy Magruder
Abstract: Formal analysis plans limit false discoveries by registering and multiplicity adjusting statistical tests. As each registered test reduces power on other tests, researchers prune hypotheses based on prior knowledge, often by combining related indicators into evenly-weighted indices. We propose two improvements to maximize learning within these types of analysis plans. First, we develop data-driven optimized indices that can yield more powerful tests than evenly-weighted indices. Second, we discuss organizing the logical structure of an analysis plan into a gated tree that directs type I error towards these high-powered tests. In simulations we show that researchers may prefer these "optimus gates" across a wide range of data-generating processes. We then assess our strategy using the community-driven development (CDD) application from Casey et al. (2012) and the Oregon Health Insurance Experiment from Finkelstein et al. (2012). We find substantial power gains in both applications, meaningfully changing the conclusions of Casey et al. (2012).
Keywords: statistical power; pre-analysis plans; field experiments; false discoveries
JEL Codes: C12; C55; C81; C93; O1
Edges that are evidenced by causal inference methods are in orange, and the rest are in light blue.
| Cause | Effect |
|---|---|
| methodologies (B41) | substantial power gains in applications (L94) |
| data-driven optimized indices (C43) | increased statistical power (C12) |
| gated tree structure (C69) | directed Type I error towards high-powered tests (C12) |
| structure of analysis plans (C38) | likelihood of obtaining significant results (C52) |