Working Paper: NBER ID: w25058
Authors: Christopher Snyder; Ran Zhuo
Abstract: The increasing demand for empirical rigor has led to the growing use of auxiliary tests (balance, specification, over-identification, placebo, etc.) in assessing the credibility of a paper’s main results. We dub these “sniff tests” because rejection is bad news for the author and standards for passing are informal. Using a sample of nearly 30,000 published sniff tests collected from scores of economics journals, we study the use of sniff tests as a screen in the publication process. For the subsample of balance tests in randomized controlled trials, our structural estimates suggest that the publication process removes 46% of significant sniff tests, yet only one in ten of these is actually misspecified. For other tests, we estimate more latent misspecifiation and less removal. Surprisingly, more authors would be justified in attributing significant sniff tests to random bad luck.
Keywords: sniff tests; publication bias; empirical rigor; economics journals
JEL Codes: A14; B41; C18
Edges that are evidenced by causal inference methods are in orange, and the rest are in light blue.
| Cause | Effect |
|---|---|
| Significant sniff tests (C52) | Well-specified studies (C90) |
| Publication process (A19) | Favoring studies with better luck on sniff tests (C92) |
| Publication process (A19) | Retention of studies based on sniff test results (C52) |
| Publication process (A19) | Misspecified studies (C50) |
| Publication process (A19) | Removal of p-values below 0.15 threshold (C29) |
| Removal of p-values below 0.15 threshold (C29) | Misspecified studies (C50) |