Working Paper: NBER ID: w30940
Authors: Eli Benmichael; Avi Feller; Jesse Rothstein
Abstract: In a pilot program during the 2016-17 admissions cycle, the University of California, Berkeley invited many applicants for freshman admission to submit letters of recommendation. This proved controversial within the university, with concerns that this change would further disadvantage applicants from disadvantaged groups. To inform this debate, we use this pilot as the basis for an observational study of the impact of submitting letters of recommendation on subsequent admission, with the goal of estimating how impacts vary across pre-defined subgroups. Understanding this variation is challenging in an observational setting because estimated impacts reflect both actual treatment effect variation and differences in covariate balance across groups. To address this, we develop balancing weights that directly optimize for “local balance” within subgroups while maintaining global covariate balance between treated and control units. Applying this approach to the UC Berkeley pilot study yields excellent local and global balance, unlike more traditional weighting methods, which fail to balance covariates within subgroups. We find that the impact of letters of recommendation increases with applicant strength. However, we find little average difference for applicants from disadvantaged groups, although this result is more mixed. In the end, we conclude that soliciting letters of recommendation from a broader pool of applicants would not meaningfully change the composition of admitted undergraduates.
Keywords: Letters of Recommendation; College Admissions; Equity; Observational Study
JEL Codes: C1; I23
Edges that are evidenced by causal inference methods are in orange, and the rest are in light blue.
| Cause | Effect |
|---|---|
| applicant strength (J68) | benefit derived from submitting letters of recommendation (LOR) (I24) |
| letters of recommendation (LOR) (Y30) | college admissions (I23) |
| applicant strength (J68) | impact of letters of recommendation (LOR) on college admissions (I24) |
| disadvantaged applicants (I24) | impact of letters of recommendation (LOR) (I24) |
| strong credentials (Y80) | impact of letters of recommendation (LOR) for non-disadvantaged applicants (I24) |
| soliciting LOR from a broader pool of applicants (O36) | demographic composition of admitted undergraduates (J19) |