Working Paper: NBER ID: w26993
Authors: Abhijit Banerjee; Esther Duflo; Amy Finkelstein; Lawrence F. Katz; Benjamin A. Olken; Anja Sautmann
Abstract: Pre-Analysis Plans (PAPs) for randomized evaluations are becoming increasingly common in Economics, but their definition remains unclear and their practical applications therefore vary widely. Based on our collective experiences as researchers and editors, we articulate a set of principles for the ex-ante scope and ex-post use of PAPs. We argue that the key benefits of a PAP can usually be realized by completing the registration fields in the AEA RCT Registry. Specific cases where more detail may be warranted include when subgroup analysis is expected to be particularly important, or a party to the study has a vested interest. However, a strong norm for more detailed pre-specification can be detrimental to knowledge creation when implementing field experiments in the real world. An ex-post requirement of strict adherence to pre-specified plans, or the discounting of non-pre-specified work, may mean that some experiments do not take place, or that interesting observations and new theories are not explored and reported. Rather, we recommend that the final research paper be written and judged as a distinct object from the “results of the PAP”; to emphasize this distinction, researchers could consider producing a short, publicly available report (the “populated PAP”) that populates the PAP to the extent possible and briefly discusses any barriers to doing so.
Keywords: pre-analysis plans; randomized control trials; RCTs; economics; research transparency
JEL Codes: A0
Edges that are evidenced by causal inference methods are in orange, and the rest are in light blue.
| Cause | Effect |
|---|---|
| utilizing a pre-analysis plan (PAP) (C90) | reduced publication bias (C46) |
| utilizing a pre-analysis plan (PAP) (C90) | increased transparency in research findings (O36) |
| lack of pre-analysis plans (PAPs) (C90) | selective reporting of results (C52) |
| strict adherence to pre-analysis plans (PAPs) (C90) | deter researchers from conducting valuable experiments (C90) |
| deviations from pre-analysis plans (PAPs) (C90) | stifle scientific discovery (O30) |
| implementation of pre-analysis plans (PAPs) (C90) | robustness of findings (C90) |