Working Paper: NBER ID: w23544
Authors: Michael L. Anderson; Jeremy Magruder
Abstract: Preanalysis plans (PAPs) have become an important tool for limiting false discoveries in field experiments. We evaluate the properties of an alternate approach which splits the data into two samples: An exploratory sample and a confirmation sample. When hypotheses are homogeneous, we describe an improved split-sample approach that achieves 90% of the rejections of the optimal PAP without requiring preregistration or constraints on specification search in the exploratory sample. When hypotheses are heterogeneous in priors or intrinsic interest, we find that a hybrid approach which prespecifies hypotheses with high weights and priors and uses a split-sample approach to test additional hypotheses can have power gains over any pure PAP. We assess this approach using the community-driven development (CDD) application from Casey et al. (2012) and find that the use of a hybrid split-sample approach would have generated qualitatively different conclusions.
Keywords: false discoveries; preanalysis plans; field experiments; splitsample methods; hybrid approach
JEL Codes: C12; C81; C9; C93; O1
Edges that are evidenced by causal inference methods are in orange, and the rest are in light blue.
| Cause | Effect |
|---|---|
| well-designed splitsample approach (C90) | reduction of false positives (C52) |
| hybrid approach (B50) | power gains (L94) |
| optimal splitsample approach (C52) | match power of full-sample PAP (C59) |
| homogeneous hypotheses (C12) | maximization of valid rejections (C52) |
| splitsample methods (C90) | enhance flexibility (Y80) |
| splitsample methods (C90) | power losses relative to full-sample approaches (C51) |