Effective Time Ratio: A Measure for Web Search Engines with Document Snippets [chapter]

Jing He, Baihan Shu, Xiaoming Li, Hongfei Yan
2010 Lecture Notes in Computer Science  
The dominant method for evaluating search engines is the Cranfield paradigm, but the existing metrics do not consider some modern search engines features, such as document snippets. In this paper, we propose a new metric effective time ratio for search engine evaluation. Effective time ratio measures the ratio between effective time a user spent on getting relevant information and the total search time. For retrieval system without presenting document snippet, its value is identical to
more » ... . For search engine with snippet, some theoretical analysis proves that its value can reflect both retrieval system performance and snippet quality. We further deploy a real user study, showing that effective time ratio can reflect users' satisfaction better than the existing metrics based on document relevance and/or snippet relevance.
doi:10.1007/978-3-642-17187-1_7 fatcat:lvfhnlprarbkhkj4k2gjs42tsq