Working Paper: NBER ID: w21447
Authors: Christopher Blattman; Julian C. Jamison; Tricia Koroknay-Palicz; Katherine Rodrigues; Margaret Sheridan
Abstract: Field experiments rely heavily on self-reported data, but subjects may misreport behaviors, especially sensitive ones such as crime. If treatment influences survey responses, it biases experimental estimates. We develop a validation technique that uses intensive qualitative work to assess survey measurement error. Subjects were assigned to receive cash, therapy, both, or neither. According to survey responses, receiving both treatments dramatically reduced crime and other sensitive behaviors. Local researchers spent several days with a random subsample of subjects following their endline surveys, building trust and seeking verbal confirmation of six behaviors: theft, drug use, homelessness, gambling, and two expenditures. This validation suggests that subjects in the control and cash only groups underreported sensitive behaviors and expenditures in the survey relative to the other treatment arms. We bound survey-based treatment effects estimates, and find the impacts of cash and therapy on crime may be larger than suggested by surveys alone.
Keywords: measurement error; survey data; qualitative validation; field experiments; sensitive behaviors
JEL Codes: C81; C93; I32; K4; O1
Edges that are evidenced by causal inference methods are in orange, and the rest are in light blue.
| Cause | Effect |
|---|---|
| self-reported data on sensitive behaviors (C83) | biased experimental estimates (C51) |
| treatment influences survey responses (C83) | biased experimental estimates (C51) |
| validation technique (C52) | mitigate systematic measurement error (C83) |
| systematic measurement error correlated with treatment status (C32) | biased experimental estimates (C51) |
| underreporting of sensitive behaviors in control and cash-only groups (C92) | larger impacts of cash and therapy on crime (K42) |
| qualitative validation (C52) | enhance credibility of survey-based treatment effect estimates (C83) |