Randomised pair comparison

Alexander Eichhorn, Pengpeng Ni, Ragnhild Eg
2010 Proceedings of the 20th international workshop on Network and operating systems support for digital audio and video - NOSSDAV '10  
Subjective quality perception studies with human observers are essential for multimedia system design. Such studies are known to be expensive and difficult to administer. They require time, a detailed knowledge of experimental designs and a level of control which can often only be achieved in a laboratory setting. Hence, only very few researchers consider running subjective studies at all. In this paper we present Randomised Pair Comparison (r/pc), an easy-to-use, flexible, economic and robust
more » ... conomic and robust extension to conventional pair comparison methods. r/pc uses random sampling to select a unique and small subset of pairs for each assessor, thus separating session duration from the experimental design. With r/pc an experimenter can freely define the duration of sessions and balance between costs and accuracy of an experiment. On a realistic example study we show that r/pc is able to create stable results with an accuracy close to full factorial designs, yet much lower costs. We also provide initial evidence that r/pc can avoid unpleasant fatigue and learning effects which are common in long experiment sessions.
doi:10.1145/1806565.1806582 dblp:conf/nossdav/EichhornNE10 fatcat:cngonyzebrbexl6bonurfro2ye