Editorial for the special section on Empirical Studies in Software Engineering Selected, and extended papers from the Eighteenth International Conference on Evaluation and Assessment in Software Engineering, May 13th–14th 2014, London, UK

Tracy Hall, Steve Counsell, Ingunn Myrtveit
2015 Information and Software Technology  
While it could be argued that every study of a computer-based system is empirical to a degree, the past twenty-five years or so has seen the assessment of systems and their associated practices emerge as a mainstream software engineering discipline. The growing depth and breadth of studies in the area demonstrates the value that exploring systems using quantitative and qualitative techniques can give. Empirical studies allow new software engineering techniques to be assessed, old techniques to
more » ... e revisited, old and new theories to be tested and past assumptions to be challenged and, potentially, revised. Four papers comprise the special section of this Journal issue and they cover diverse areas. Two papers in the special section relate to replications of previous studies and provide insights into the state-of-the-art. Such studies are an important way in which a body of software engineering knowledge can be strengthened and new perspectives gained. Moreover, replication can validate existing findings and test those findings by applying a study's methods to other systems, but replication is not without its problems. The unique context of each empirical study and variations between original study and replication often present difficulties. In their paper: "Investigations about replication of empirical studies in software engineering: A systematic mapping study", Cleyton de Magalhães, Fabio da Silva, Ronnie Santos and Marcos Suassuna provide a comprehensive assessment of replication studies dating back to 1996. Six research questions were posed and then answered relating to trends in the thirty-seven studies extracted from the search process. The research questions included detailed examination of the frequency of studies over that period, the composition of authors in the studies, the topics addressed and how study results had been presented. The paper provides insightful reflection; the number of replication studies is still very small and there is still a lack of consistency in the terminology used and understanding of the replication concept. A range of discussion points and questions are also raised by the study. Most notably, what is the exact definition of a replication? How do we gauge a successful replication? Finally, how should replications be reported? The authors suggest that 'replication work is being conducted without a solid conceptual background about replications, which could result in low quality results'. The study is therefore a reality check for the empirical software engineering community -replication is still under-valued and under-used and a change in attitudes towards replication is needed. While there are considerable practical benefits to replication if done well, more effort needs to be invested in the methodology and definitions underlying replication studies. Finally, the authors report that the number of authors engaged in replication is small; while in itself this is not a problem, a lack of dissemination activities may be hindering progress in the area. The second replication paper: "On the Usefulness of Ownership Metrics in Open-Source Software Projects" by
doi:10.1016/j.infsof.2015.02.015 fatcat:bncl5d6pm5d3jl4wqiewrmb3f4