Usage based effectiveness measures

Leif Azzopardi
2009 Proceeding of the 18th ACM conference on Information and knowledge management - CIKM '09  
The aim of an Information Retrieval (IR) application is to support the user accessing relevant information effectively and efficiently. It is well known that system performance, in terms of finding relevant information, experienced by the user is heavily dependent upon the application itself (i.e. the IR system exposed through the application's interface) as well as how the application is used by the user (i.e. how the user interacts with the system through the interface). Thus, a very
more » ... us, a very pragmatic evaluation question that arises at the application level is: what is the effectiveness experienced by the user during the usage of the application? To be able to answer this question, we represent the usage of an application by the stream of documents the user encounters while interacting with the application. This representation enables us to monitor and track the performance over time/usage. By taking a stream-based, time-centric view of the IR process, instead of a rank-list, topic/task centric view, the evaluation can be performed on any IR based application. To illustrate the difference and the utility of this approach, we demonstrate how a new suite of usage based effectiveness measures can be applied. This work provides the conceptual foundations for measuring, monitoring and modeling the performance of any IR application which needs to be evaluated over time and in context.
doi:10.1145/1645953.1646034 dblp:conf/cikm/Azzopardi09 fatcat:4mtqneg3bjgonl6rxis5quxmxi