Six Sigma Applied Throughout the Lifecycle of an Automated Decision System

Angie Patterson, Piero Bonissone, Marc Pavese
2005 Quality and Reliability Engineering International  
Automated decision-making systems have been deployed in many industrial, commercial, and financial applications. The needs for such systems are usually motivated by requirements for variation reduction, capacity increase, cost and cycle time reduction, and end-to-end traceability of the transaction or product. Before we can use any automated decision-making system in a production environment we must develop a strategy to insure high quality throughout its entire lifecycle. We need to guarantee
more » ... ts performance through a rigorous Design for Six Sigma process (DFSS). This process includes validation, tuning, and production testing of the system. Once the system is in production we must monitor and maintain its performance over its lifecycle. In this paper we will outline the Six Sigma process that led to the deployment of an automated decision-making system in one of the General Electric Financial Assurance businesses. In the development of any type of decision algorithm, we usually face several design trade-offs. Among the most common, we find: (1) accuracy versus coverage; (2) accuracy versus interpretability; (3) run-time efficiency versus configuration-driven architecture. These trade-offs are always present no matter what the application of decision engine technology. In the define phase of the project, enough insight into the requirements of a specific application must be gathered in order to be able to make the appropriate trade-off for that situation. In the DFSS build process for the insurance underwriting engine described below, each of these trade-offs was determined in this way. The first trade-off is similar to the precision versus recall compromise found in the design of information retrieval systems. We could tune a classifier to maximize its number of correct decisions, declining to make any commitment if we are not confident about the conclusion. This behavior would increase accuracy at the expense of coverage. Alternatively, we could tune the same classifier to always issue a decision for each probe, increasing coverage at the expense of accuracy. The second trade-off, typically dictated by legal or compliance regulations, constrains the underlying technologies used to implement the classifier 1,2 . In our approach we have used soft computing (SC) techniques, a collection of computational paradigms (probabilistic, fuzzy, neural, and evolutionary) in which the equation 'model = structure + parameters' takes a different connotation, as we have a much richer repertoire to represent the structure, to tune the parameters, and to iterate this process 3 . This repertoire enables us to choose among different trade-offs between the model's interpretability and its accuracy. For instance, one approach aimed at maintaining the model's transparency usually starts with knowledge-derived linguistic models, in which domain knowledge is translated into an initial structure and parameters. The model's accuracy is further improved by using global or local data-driven search methods to tune the structure and/or parameters. An alternative approach, aimed at building more accurate models, might start with data-driven search methods. Then, we could embed domain knowledge into the search operators to control or limit the search space, or to maintain the model's interpretability. Post-processing approaches can also be used to extract explicit structural information from the models. The third trade-off is related to the use of configuration files to drive the behavior of the classifiers, instead of hard-coding their logical behavior. The essential idea here is that the actual coded software would implement a more generic approach to solving the problem, which could then be specialized not within the code itself but by reading parameters from a configuration file. In fact, any external data source, such as a database table, could also be used to supply engine parameters. While slightly less efficient at run-time, the use of a common decision engine driven by configuration files produces a more maintainable classifier than one whose parametric values are intertwined in the engine's code. This additional computational cost is introduced for the purpose of lifecycle benefits. Its software implementation is described in Aggour and Pavese 4 . Description of the underwriting classifier for an insurance product To illustrate the key issues in developing and deploying an automated decision engine (ADE), we will use a representative, challenging classification problem: the process of underwriting insurance applications. Insurance underwriting is a complex decision-making task that is traditionally performed by trained individuals. An underwriter must evaluate each insurance application in terms of its potential risk for generating a claim,
doi:10.1002/qre.629 fatcat:tlq5xy5sbneynev3bjs5dhfdbq