On the evaluation of IR systems

S.E. Robertson, M.M. Hancock-Beaulieu
1992 Information Processing & Management  
The paper highlights the ever increasing complexity in the evaluation of IR systems which has arisen over the last decade. Relevance, cognition, user behaviour, interaction and a changing view of the boundaries of the system are considered to be contributory factors. Issues such as laboratory versus operational systems, black-box versus diagnostic experiments, and qualitative and quantitative methods, are discussed and supported by examples drawn from three groups of evaluative experiments:
more » ... hted searching on a front end system, information seeking behaviour and the use of OPACs, and the OKAPI experimental retrieval system. Published in: Information Processing and Management 28, 457-466 (1992). Published version The volume edited by Sparck Jones and published in 1981, Information Retrieval Experiment, remains the one substantial work on the evaluation of IR systems. The first chapter, by one of the present authors (Robertson, 1981) , ended with the thought that the succeeding twenty years might see as much change in this field as the previous twenty: If, in 2001, this entire chapter is obsolete, so much the better! As we are now half way through that period, an overview is appropriate. It is the contention of the present paper that the field has changed substantially in ten years.
doi:10.1016/0306-4573(92)90004-j fatcat:qpi33c2y25fyfftdtwe2ggehca