Working Paper: NBER ID: w25848
Authors: Omar Alubaydli; John A. List; Dana Suskind
Abstract: Policymakers are increasingly turning to insights gained from the experimental method as a means of informing public policies. Whether—and to what extent—insights from a research study scale to the level of the broader public is, in many situations, based on blind faith. This scale-up problem can lead to a vast waste of resources, a missed opportunity to improve people’s lives, and a diminution in the public’s trust in the scientific method’s ability to contribute to policymaking. This study provides a theoretical lens to deepen our understanding of the science of how to use science. Through a simple model, we highlight three elements of the scale-up problem: (1) when does evidence become actionable (appropriate statistical inference); (2) properties of the population; and (3) properties of the situation. We argue that until these three areas are fully understood and recognized by researchers and policymakers, the threats to scalability will render any scaling exercise as particularly vulnerable. In this way, our work represents a challenge to empiricists to estimate the nature and extent of how important the various threats to scalability are in practice, and to implement those in their original research.
Keywords: scalability; experimental methods; public policy; evidence-based policymaking
JEL Codes: C9; C90; C91; C92; C93; D03
Edges that are evidenced by causal inference methods are in orange, and the rest are in light blue.
| Cause | Effect |
|---|---|
| Competition among researchers (O36) | Selection bias (C24) |
| Selection bias (C24) | Biased treatment effect estimates (C21) |
| Efficacy of interventions observed in small-scale studies (C90) | Efficacy in larger populations (C90) |
| Scaling interventions (C32) | Diminished treatment effects (C22) |
| Variations in implementation quality (L15) | Reduced effectiveness (I14) |
| Increasing number of participants (C90) | Maintaining quality (L15) |
| Strategic selection of populations (R23) | Inflated treatment effects (C22) |
| Inflated treatment effects (C22) | Hold true in broader populations (C12) |