Working Paper: NBER ID: w24324
Authors: Ran Abramitzky; Roy Mill; Santiago Pérez
Abstract: Linking individuals across historical datasets relies on information such as name and age that is both non-unique and prone to enumeration and transcription errors. These errors make it impossible to find the correct match with certainty. In the first part of the paper, we suggest a fully automated probabilistic method for linking historical datasets that enables researchers to create samples at the frontier of minimizing type I (false positives) and type II (false negatives) errors. The first step guides researchers in the choice of which variables to use for linking. The second step uses the Expectation-Maximization (EM) algorithm, a standard tool in statistics, to compute the probability that each two records correspond to the same individual. The third step suggests how to use these estimated probabilities to choose which records to use in the analysis. In the second part of the paper, we apply the method to link historical population censuses in the US and Norway, and use these samples to estimate measures of intergenerational occupational mobility. The estimates using our method are remarkably similar to the ones using IPUMS’, which relies on hand linking to create a training sample. We created an R code and a Stata command that implement this method.
Keywords: Historical Data; Record Linkage; Intergenerational Mobility
JEL Codes: C10; J01; J10; N00
Edges that are evidenced by causal inference methods are in orange, and the rest are in light blue.
| Cause | Effect |
|---|---|
| automated linking method (C45) | reduction of false positives (C52) |
| automated linking method (C45) | accuracy of occupational mobility estimates (J62) |
| quality of linked samples (C83) | accuracy of occupational mobility estimates (J62) |
| flexibility of the method (C90) | tradeoff between match quality and sample size (C78) |