The Evolution of Continuous Experimentation in Software Product Development: From Data to a Data-Driven Organization at Scale

Aleksander Fabijan, Pavel Dmitriev, Helena Holmstrom Olsson, Jan Bosch
2017 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE)  
Software development companies are increasingly aiming to become data-driven by trying to continuously experiment with the products used by their customers. Although familiar with the competitive edge that the A/B testing technology delivers, they seldom succeed in evolving and adopting the methodology. In this paper, and based on an exhaustive and collaborative case study research in a large software-intense company with highly developed experimentation culture, we present the evolution
more » ... of moving from ad-hoc customer data analysis towards continuous controlled experimentation at scale. Our main contribution is the "Experimentation Evolution Model" in which we detail three phases of evolution: technical, organizational and business evolution. With our contribution, we aim to provide guidance to practitioners on how to develop and scale continuous experimentation in software organizations with the purpose of becoming data-driven at scale. A/B testing; continuous experimentation; data science; customer feedback; continuous product innovation; Experimentation Evolution Model; product value; Experiment Owner I.
doi:10.1109/icse.2017.76 dblp:conf/icse/FabijanDOB17 fatcat:wzitigxflvhrlefwt63ispla3u