Guess Again: The Effect of Correct Guesses on Scores in an Operational CAT Program

March 14, 2011


Previous theoretical research shows that recovery of ability estimate on computerized adaptive tests after initial correct responses depends on factors such as the item selection method and test length.Because it would be difficult to evaluate all factors present in an operational setting, data from an actual live testing program and examinees were used to simulate the effect of strings of correct responses in different positions on the exam. Differences interacted relative to string length, position, section, and examinee ability. Findings refuted the myth that the beginning items are the most important. The greatest limitation of the study is that the use of specific constraints for an operational program yields results that are not generalizable. Future research may include looking at incorrect responses and evaluating how increased time pressure at the end of the test would affect any benefit accrued from better performance at the beginning of the test.