Benefits and barriers of user evaluation in software engineering research

Raymond P.L. Buse, Caitlin Sadowski, Westley Weimer
2011 Proceedings of the 2011 ACM international conference on Object oriented programming systems languages and applications - OOPSLA '11  
In this paper, we identify trends about, benefits from, and barriers to performing user evaluations in software engineering research. From a corpus of over 3,000 papers spanning ten years, we report on various subtypes of user evaluations (e.g., coding tasks vs. questionnaires) and relate user evaluations to paper topics (e.g., debugging vs. technology transfer). We identify the external measures of impact, such as best paper awards and citation counts, that are correlated with the presence of
more » ... th the presence of user evaluations. We complement this with a survey of over 100 researchers from over 40 different universities and labs in which we identify a set of perceived barriers to performing user evaluations.
doi:10.1145/2048066.2048117 dblp:conf/oopsla/BuseSW11 fatcat:wtivmbojh5bztfdgw55ly765jq