Working Paper: CEPR ID: DP12488
Authors: Erich Battistin; Marco Ovidi
Abstract: We use the UK's 2014 Research Excellence Framework (REF) to study which attributes characterize a top-scoring (four-star) publication in Economics and Econometrics. We frame the analysis as a classification problem and, using information in official documents, derive conditions to infer the unobservable score that panellists awarded to each publication. Juxtaposing institutions’ submissions with REF outcomes provides information on the unobservable pass-marks used for assigning quality levels, which respond to journal prestige measured by the Thomson Reuters Article Influence Score. We explore this statistical feature in the econometric analysis, which reveals the limited contribution to awarded quality made by other publication attributes, possibly unobservable to us, conditional on the Article Influence Score. We conclude that, in large-scale and costly evaluations such as the REF, the time-consuming task of peer reviews should be devoted to publications not in academic outlets with unambiguously top-scoring bibliometric indicators of journal impact. Our model also predicts a ranking of academic journals consistent with the classification of REF panellists.
Keywords: education; policy; higher education; journal rankings; research funding
JEL Codes: H52; H83; I23; I28
Edges that are evidenced by causal inference methods are in orange, and the rest are in light blue.
| Cause | Effect |
|---|---|
| Thomson Reuters Article Influence Score (AIS) (A14) | quality classification awarded by REF panellists (Q50) |
| number of submissions in a journal exceeds number of world-leading outputs (A14) | publication not consistently awarded a world-leading ranking (A14) |
| Thomson Reuters Article Influence Score (AIS) (A14) | contribution of other bibliometric indicators becomes statistically insignificant (C29) |
| reliance on bibliometric indicators (A14) | efficiency of peer review process in identifying quality publications (A14) |
| publications with clear bibliometric indicators of high quality (A14) | peer reviews may not be necessary (Y30) |