Working Paper: NBER ID: w24834
Authors: Amanda E. Kowalski
Abstract: A fundamental concern for researchers who analyze and design experiments is that the estimate obtained from the experiment might not be externally valid for other policies of interest. Researchers often attempt to assess external validity by comparing data from an experiment to external data. In this paper, I discuss approaches from the treatment effects literature that researchers can use to begin the examination of external validity internally, within the data from a single experiment. I focus on presenting the approaches simply using stylized examples.
Keywords: external validity; experiments; treatment effects; selection
JEL Codes: C9; C93; H0
Edges that are evidenced by causal inference methods are in orange, and the rest are in light blue.
| Cause | Effect |
|---|---|
| running experiments (C90) | eliminate selection bias (C52) |
| selection can still occur within experiments (C90) | inform external validity (C90) |
| treatment effects may vary across different groups (C21) | inform applicability of treatment effects to other policies (C90) |
| wellness plan reduces costs for compliers (I10) | costs may increase for always takers (D41) |