Skoll: A Process and Infrastructure for Distributed Continuous Quality Assurance
IEEE Transactions on Software Engineering
Software engineers increasingly emphasize agility and flexibility in their designs and development approaches. They increasingly use distributed development teams, rely on component assembly and deployment rather than green field code writing, rapidly evolve the system through incremental development and frequent updating, and use flexible product designs supporting extensive end-user customization. While agility and flexibility have many benefits, they also create an enormous number of
... s number of potential system configurations built from rapidly changing component implementations. Since today's quality assurance (QA) techniques do not scale to handle highly configurable systems, we are developing and validating novel software QA processes and tools that leverage the extensive computing resources of user and developer communities in a distributed, continuous manner to improve software quality significantly. This paper provides several contributions to the study of distributed, continuous QA (DCQA). First, it shows the structure and functionality of Skoll, which is an environment that defines a generic around-the-world, around-the-clock QA process and several sophisticated tools that support this process. Second, it describes several novel QA processes built using the Skoll environment. Third, it presents two studies using Skoll: one involving user testing of the Mozilla browser and another involving continuous build, integration, and testing of the ACE+TAO communication software package. The results of our studies suggest that the Skoll environment can manage and control distributed continuous QA processes more effectively than conventional QA processes. For example, our DCQA processes rapidly identified problems that had taken the ACE+TAO developers much longer to find and several of which they had not found. Moreover, the automatic analysis of QA results provided developers information that enabled them to quickly find the root cause of problems. benefits of this approach include: massive parallelization of QA processes, greatly expanded access to resources and environment not easily obtainable in-house, and (depending on the specific QA process being executed), visibility into actual fielded usage patterns. This section describes our initial steps towards realizing our vision. A. Distributed Continuous QA processes At a high level, distributed continuous QA processes resemble certain traditional distributed computations as shown in Figure 1 . As implemented in Skoll, tasks are QA activities, Computing Nodes Cluster #1 Computing Nodes Cluster #2 Skoll Coordinator Site Computing Nodes Cluster #3 Subtask 1.1 Subtask 1.2 Subtask 1.3 Task 1 QA Task 1 is split into three subtasks (1.1, 1.2, 1.3) and allocated to computing node clusters 1, 2, and 3 respectively.