Working Paper: NBER ID: w25568
Authors: Bernard Black; Alex Hollingsworth; Leticia Nunes; Kosali Simon
Abstract: Power is an important factor in assessing the likely validity of a statistical estimate. An analysis with low power is unlikely to produce convincing evidence of a treatment effect even when one exists. Of greater concern, a statistically significant estimate from a low-powered analysis is likely to overstate the magnitude of the true effect size, often finding estimates of the wrong sign or that are several times too large. Yet statistical power is rarely reported in published economics work. This is in part because modern research designs are complex enough that power cannot always be easily ascertained using simple formulae. Power can also be difficult to estimate in observational settings where researchers may not know—and have no ability to manipulate—the true treatment effect or other parameters of interest. Using an ap-plied example—the link between gaining health insurance and mortality—we conduct a simulated power analysis to outline the importance of power and ways to estimate power in complex research settings. We find that standard difference-in-differences and triple differences analyses of Medicaid expansions using county or state mortality data would need to induce reductions in population mortality of at least 2% to be well powered. While there is no single, correct method for conducting a simulated power analysis, our manuscript outlines decisions relevant for applied researchers interested in conducting simulations appropriate to other settings.
Keywords: Statistical Power; Medicaid Expansion; Health Insurance; Mortality
JEL Codes: I1
Edges that are evidenced by causal inference methods are in orange, and the rest are in light blue.
| Cause | Effect |
|---|---|
| Medicaid expansion (I18) | statistically significant relationship with mortality (I12) |
| low-powered analyses (D79) | misleading estimates (C51) |
| subpopulations more affected by treatment (C21) | statistical power (C12) |
| Medicaid expansion (I18) | mortality (I12) |
| Medicaid expansion (I18) | healthcare amenable mortality (I12) |