Assessing the speaker recognition performance of naive listeners using mechanical turk

Wade Shen, Joseph Campbell, Derek Straub, Reva Schwartz
2011 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)  
In this paper we attempt to quantify the ability of naive listeners to perform speaker recognition in the context of the NIST evaluation task. We describe our protocol: a series of listening experiments using large numbers of naive listeners (432) on Amazon's Mechanical Turk that attempts to measure the ability of the average human listener to perform speaker recognition. Our goal was to compare the performance of the average human listener to both forensic experts and state-ofthe-art automatic
more » ... ofthe-art automatic systems. We show that naive listeners vary substantially in their performance, but that an aggregation of listener responses can achieve performance similar to that of expert forensic examiners.
doi:10.1109/icassp.2011.5947708 dblp:conf/icassp/ShenCSS11 fatcat:or7ws4dgdjaczdtizzxp255xxq