Working Paper: NBER ID: w22595
Authors: Angus Deaton; Nancy Cartwright
Abstract: RCTs would be more useful if there were more realistic expectations of them and if their pitfalls were better recognized. For example, and contrary to many claims in the applied literature, randomization does not equalize everything but the treatment across treatments and controls, it does not automatically deliver a precise estimate of the average treatment effect (ATE), and it does not relieve us of the need to think about (observed or unobserved) confounders. Estimates apply to the trial sample only, sometimes a convenience sample, and usually selected; justification is required to extend them to other groups, including any population to which the trial sample belongs. Demanding “external validity” is unhelpful because it expects too much of an RCT while undervaluing its contribution. Statistical inference on ATEs involves hazards that are not always recognized. RCTs do indeed require minimal assumptions and can operate with little prior knowledge. This is an advantage when persuading distrustful audiences, but it is a disadvantage for cumulative scientific progress, where prior knowledge should be built upon and not discarded. RCTs can play a role in building scientific knowledge and useful predictions but they can only do so as part of a cumulative program, combining with other methods, including conceptual and theoretical development, to discover not “what works,” but “why things work”.
Keywords: No keywords provided
JEL Codes: C10; C26; C93; O22
Edges that are evidenced by causal inference methods are in orange, and the rest are in light blue.
| Cause | Effect |
|---|---|
| Randomization does not equate to balance in treatment and control groups (C90) | Unbiased estimates of the average treatment effect (ATE) (C22) |
| Randomization aims to eliminate bias (C90) | Treatment is orthogonal to unobserved confounders (C90) |
| Treatment is orthogonal to unobserved confounders (C90) | Potential bias in the estimated ATE (C51) |
| Average treatment effect (ATE) derived from RCTs applies only to the trial sample (C90) | Caution against generalizing these results (C91) |
| Perceived precision of RCTs can be misleading (C90) | Misunderstandings about balance and the nature of causal relationships (C32) |
| RCTs are not a panacea for establishing causal relationships (C90) | Effectiveness is contingent upon the design and context of the trial (C90) |
| External validity is often overstated (C90) | Applicability of RCT results to broader populations requires careful consideration (C90) |
| Potential for spurious significance in RCT results (C90) | Issues like sample size and the distribution of treatment effects complicate interpretation (C90) |