Working Paper: CEPR ID: DP16099
Authors: Lina Anaya; Nagore Iriberri; Pedro Rey Biel; Gema Zamarro
Abstract: Standardized assessments are widely used to determine access to educational resources with important consequences for later economic outcomes in life. However, many design features of the tests themselves may lead to psychological reactions influencing performance. In particular, the level of difficulty of the earlier questions in a test may affect performance in later questions. How should we order test questions according to their level of difficulty such that test performance offers an accurate assessment of the test taker's aptitudes and knowledge? We conduct a field experiment with about 19,000 participants in collaboration with an online teaching platform where we randomly assign participants to different orders of difficulty and we find that ordering the questions from easiest to most difficult yields the lowest probability to abandon the test, as well as the highest number of correct answers. Consistent results are found exploiting the random variation of difficulty across test booklets in the Programme for International Student Assessment (PISA), a triannual international test, for the years of 2009, 2012, and 2015, providing additional external validity. We conclude that the order of the difficulty of the questions in tests should be considered carefully, in particular when comparing performance between test-takers who have faced different order of questions
Keywords: question order; difficulty; test performance
JEL Codes: C93; D81; I20; J16
Edges that are evidenced by causal inference methods are in orange, and the rest are in light blue.
| Cause | Effect |
|---|---|
| more difficult part of the test (C52) | abandonment rates (J63) |
| more difficult part of the test (C52) | lower scores in subsequent parts (D29) |
| order of question difficulty (C69) | probability of test abandonment (C12) |
| order of question difficulty (C69) | number of correct answers (C52) |
| difficult questions first (Y20) | probability of test abandonment (C12) |
| easy questions first (Y20) | probability of test abandonment (C12) |