Working Paper: NBER ID: w31666
Authors: Kevin Lang
Abstract: When economists analyze a well-conducted RCT or natural experiment and find a statistically significant effect, they conclude the null of no effect is unlikely to be true. But how frequently is this conclusion warranted? The answer depends on the proportion of tested nulls that are true and the power of the tests. I model the distribution of t-statistics in leading economics journals. Using my preferred model, 65% of narrowly rejected null hypotheses and 41% of all rejected null hypotheses with |t|<10 are likely to be false rejections. For the null to have only a .05 probability of being true requires a t of 5.48.
Keywords: No keywords provided
JEL Codes: A10; C12
Edges that are evidenced by causal inference methods are in orange, and the rest are in light blue.
| Cause | Effect |
|---|---|
| proportion of true nulls and the power of the tests used (C12) | conclusion that the null hypothesis of no effect is unlikely to be true (C12) |
| modeling of t-statistics (C29) | likelihood of false rejections (C52) |
| 65% of narrowly rejected null hypotheses (C12) | likely false rejections (C52) |
| 41% of all rejected null hypotheses with t > 10 (C12) | likely false rejections (C52) |
| t statistic of 5.48 (C46) | only a 5% probability of null hypothesis being true (C12) |
| common practice of stating power against a point alternative (D74) | overstates the study's power (C90) |
| probability of a false discovery increases (C12) | considering a range of values for the alternative hypothesis (C12) |
| statistical significance dramatically exceeds conventional levels (C12) | caution in applying findings from single studies (C90) |
| caution in applying findings from single studies (C90) | relevance in policy contexts (F68) |
| statistically significant effect in a well-conducted RCT or natural experiment (C90) | null hypothesis of no effect is unlikely to be true (C12) |