Working Paper: CEPR ID: DP11077
Authors: Marcel Fafchamps; Julien Labonne
Abstract: We discuss a method aimed at reducing the risk that spurious results are published. Researchers send their datasets to an independent third party who randomly generates training and testing samples. Researchers perform their analysis on the former and once the paper is accepted for publication the method is applied to the latter and it is those results that are published. Simulations indicate that, under empirically relevant settings, the proposed method significantly reduces type I error and delivers adequate power. The method ? that can be combined with pre-analysis plans ? reduces the risk that relevant hypotheses are left untested.
Keywords: Bonferroni correction; Data mining; Pre-analysis plan; Publication bias
JEL Codes: C12; C18
Edges that are evidenced by causal inference methods are in orange, and the rest are in light blue.
| Cause | Effect |
|---|---|
| split sample method (C83) | reduction in Type I errors (C52) |
| split sample method (C83) | enhancement of reliability of empirical findings (C90) |
| training dataset (C55) | testing dataset leads to reduction in Type I errors (C52) |
| split sample method (C83) | identification of null hypotheses that should be rejected (C12) |
| sample sizes > 3000 (C55) | maintenance of adequate statistical power (C90) |
| split sample method allows hypothesis refinement (C90) | more credible results (C90) |