We describe a retrospective study of the responses to the Brief Electricity and Magnetism Assessment (BEMA) collected from a large population of 3480 students at a large public university. Two different online testing setups were employed for administering the BEMA. Our tale focuses on the interpretation of the data collected from these two testing setups. Our study brings new insight and awareness to the effect of the randomization on the BEMA test results. Starting from an a priori common

more »

... e model, we show simple methods to detect and separate guessing from the genuinely thought responses. In addition, we show that the group of responses with low scores (7 or less out of 30) had different answer choice strategies than the groups with average or high scores, and that the time-in-testing is an essential parameter to be reported. Our results suggest that the data should be cleaned to insure that only valid times-in-testing are included before reporting and comparing statistics. We analyze in detail the effect of the answer-options randomization and the effect of the shuffling of the questions order. Our study did not detect any significant effect of the option randomization alone on the BEMA scores, for valid times-in-testing. We found clear evidence that shuffling the order of the independent questions does not affect the scores of the BEMA test, while shuffling the order of the dependent questions does.