Working Paper: NBER ID: w27250
Authors: Eliot Abrams; Jonathan Libgober; John A. List
Abstract: The past few decades have ushered in an experimental revolution in economics whereby scholars are now much more likely to generate their own data. While there are virtues associated with this movement, there are concomitant difficulties. Several scientific disciplines, including economics, have launched research registries in an effort to attenuate key inferential issues. This study assesses registries both empirically and theoretically, with a special focus on the AEA registry. We find that over 90% of randomized control trials (RCTs) in economics do not register, only 50% of the RCTs that register do so before the intervention begins, and the majority of these preregistrations are not detailed enough to significantly aid inference. Our empirical analysis further shows that using other scientific registries as aspirational examples is misguided, as their perceived success in tackling the main issues is largely a myth. In light of these facts, we advance a simple economic model to explore potential improvements. A key insight from the model is that removal of the (current) option to register completed RCTs could increase the fraction of trials that register. We also argue that linking IRB applications to registrations could further increase registry effectiveness.
Keywords: Research Registries; Experimental Economics; RCT; File Drawer Problem; Phacking
JEL Codes: B41; C9; C91; C92; C93
Edges that are evidenced by causal inference methods are in orange, and the rest are in light blue.
| Cause | Effect |
|---|---|
| the lack of registration (K16) | the persistence of the file drawer problem (D80) |
| the quality of registration (L15) | the ability to mitigate phacking concerns (K24) |
| the social norm surrounding registration (Z13) | the low rate of compliance among researchers (K40) |
| banning late registration (K16) | the fraction of trials that register (C25) |