Working Paper: CEPR ID: DP16751
Authors: Albert Menkveld; Anna Dreber; Felix Holzmeister; Juergen Huber; Magnus Johannesson; Michael Kirchler; Sebastian Neusess; Michael Razen; Utz Weitzel; Christian C. P. Wolff
Abstract: In statistics, samples are drawn from a population in a data generating process (DGP). Standard errors measure the uncertainty in sample estimates of population parameters. In science, evidence is generated to test hypotheses in an evidence generating process (EGP). We claim that EGP variation across researchers adds uncertainty: non-standard errors. To study them, we let 164 teams test six hypotheses on the same sample. We find that non-standard errors are sizeable, on par with standard errors. Their size (i) co-varies only weakly with teammerits, reproducibility, or peer rating, (ii) declines significantly after peer-feedback, and (iii) is underestimated by participants.
Keywords: No keywords provided
JEL Codes: No JEL codes provided
Edges that are evidenced by causal inference methods are in orange, and the rest are in light blue.
| Cause | Effect |
|---|---|
| nonstandard errors (NSEs) (C20) | peer feedback (C92) |
| peer feedback (C92) | nonstandard errors (NSEs) (C20) |
| nonstandard errors (NSEs) (C20) | team quality (L15) |
| nonstandard errors (NSEs) (C20) | workflow quality (L15) |
| peer feedback (C92) | accuracy of reported findings (C90) |
| researchers' awareness (C90) | nonstandard errors (NSEs) (C20) |