Principles of survey research

Shari Lawrence Pfleeger, Barbara A. Kitchenham
2001 Software engineering notes  
Surveys are probably the most commonly-used research method world-wide. Survey work is visible not only because we see many examples of it in software engineering research, but also because we are often asked to participate in surveys in our private capacity, as electors, consumers, or service users. This widespread use of surveys may give us the impression that surveybased research is straightforward, an easy option for researchers to gather important information about products, context,
more » ... cts, context, processes, workers and more. In our personal experience with applying and evaluating research methods and their results, we certainly did not expect to encounter major problems with a survey that we planned, to investigate issues associated with technology adoption. This article and subsequent ones in this series describe how wrong we were. We do not want to give the impression that there is any way of turning a bad survey into a good one; if a survey is a lemon, it stays a lemon. However, we believe that learning from our mistakes is the way to make lemonade from lemons. So this series of articles shares with you our lessons learned, in the hope of improving survey research in software engineering. We began our investigation by reviewing what is known about technology transfer. We found that, a few years ago, Zelkowitz, Wallace and Binkley (1998) surveyed practitioners to determine their confidence in different types of empirical evaluations as the basis for technology adoption decisions. Their findings confirmed what we have long suspected: the evidence produced by the research community to support technology adoption is not the kind of evidence being sought by practitioners. To build on Zelkowitz et al.'s work, we contacted the publisher of Applied Software Development, a newsletter whose readership included primarily software project managers. We wanted to do a followup survey of managers, to find out what kinds of evaluations they make of proposed technologies, and what kinds of evidence they rely on for their technology decisions. However, when we administered our survey and analyzed the results, we realized that we had made errors in survey design, construction, administration and analysis that rendered our results inconclusive at best. Nevertheless. our lemon of a survey gave us some useful insights into the right and wrong ways to do survey research. This article is the first of six parts about how to do useful survey research. We use our own work, plus the work of several other researchers, to illustrate the attractions and pitfalls of the survey technique. To understand how surveys fit in the larger scheme of empirical investigation, we invite you to read the series we wrote, starting in December 1995, in Software Engineering Notes. There, we described several empirical methods, including case studies, factor analyses, experiments and surveys, to help you decide which technique is appropriate in which situation. Here, we assume you have decided to do a survey, and we focus on how to organize, administer and analyze one so that you get useful, meaningful results. What our series will discuss This first installment of our series describes three surveys: two reported in the literature and the one we attempted recently. We provide you with a general overview of each survey; we will use particulars of each survey in subsequent installments as examples of the state of the practice and how it might be improved. We also suggest general reference material for studying survey techniques. Subsequent installments will focus on these issues: • Survey design, including discussions of sample size and response rate • Questionnaire design and construction • Issues in survey administration, including motivating respondents, surveyor bias, pre-and pilot tests, survey documentation, and survey reliability and validity • Data analysis. Population and sampling
doi:10.1145/505532.505535 fatcat:p2hipncb3faixoq2yr2k5qi3eu