Working Paper: NBER ID: w18373
Authors: Hunt Allcott
Abstract: "Site selection bias" can occur when the probability that a program is adopted or evaluated is correlated with its impacts. I test for site selection bias in the context of the Opower energy conservation programs, using 111 randomized control trials involving 8.6 million households across the U.S. Predictions based on rich microdata from the first ten replications substantially overstate efficacy in the next 101 sites. Several mechanisms caused this positive selection. For example, utilities in more environmentalist areas are more likely to adopt the program, and their customers are more responsive to the treatment. Also, because utilities initially target treatment at higher-usage consumer subpopulations, efficacy drops as the program is later expanded. The results illustrate how program evaluations can still give systematically biased out-of-sample predictions, even after many replications.
Keywords: site selection bias; program evaluation; energy conservation; randomized control trials
JEL Codes: C93; D12; L94; O12; Q41
Edges that are evidenced by causal inference methods are in orange, and the rest are in light blue.
| Cause | Effect |
|---|---|
| site selection bias (C52) | overestimation of program efficacy (C90) |
| utilities in more environmentally conscious areas (L99) | adoption of energy conservation programs (Q48) |
| positive selection bias (C52) | larger treatment effects in early sites (C90) |
| targeting of higher-usage households (R22) | positive selection bias (C52) |
| utility ownership structures (L97) | positive selection bias (C52) |
| site characteristics (R53) | treatment effects (C22) |