Satisfaction of assumptions is a weak predictor of performance

1988 International Journal of Approximate Reasoning  
Abstracts 345 Much of the controversy about methods for automated decision making has focused on specific calculi for combining beliefs or propagating uncertainty. The authors broaden the debate by (1) exploring the constellation of secondary tasks surrounding any primary decision problem and (2) identifying knowledge engineering concerns that present additional representational trade-offs. They argue on pragmatic grounds that the attempt to support all of these tasks wihtin a single calculus
more » ... misguided. In the process, they note several uncertain reasoning objectives that conflict with the Bayesian ideal of complete specification of probabilities and utilities. In response, they advocate treating the uncertainty calculus as an object language for reasoning mechanisms that support the secondary tasks. Arguments against Bayesian decision theory are weakened when the calculus is relegated to this role. Architectures for uncertainty handling that take statements in the calculus as objects to be reasoned about offer the prospect of retaining normative status with respect to decision making while supporting the other tasks in uncertain reasoning. This study examined the effects of "tuning" the parameters of the incremental function of MYCIN, the independent function of PROSPECTOR, a probability model that assumes independence, and a simple additive linear equation. The parameters of each of these models were optimized to provide solutions that most nearly approximated those from a full probability model for a large set of simple networks. Surprisingly, MYCIN, PROSPECTOR, and the linear equation performed equivalently; the independence model was clearly more accurate on the networks studied. This paper demonstrates a methodology for examining the accuracy of uncertain inference systems (UIS) after their parameters have been optimized, and uses it for several common UISs. This methodology may be used to test the accuracy when either the prior assumptions or updating formulas are not exactly satisfied. Surprisingly, these UISs were revealed to be no more accurate on the average than a simple linear regression. Moreover, even on prior distributions that were deliberately biased so as to give very good accuracy, they were less accurate than the simple probabilistic model which 346 Abstracts assumes marginal independence between inputs. This demonstrates that the importance of updating formulas can outweigh that of prior assumptions. Thus, when UISs are judged by their final accuracy after optimization, completely different results are obtained than when they are judged by whether or not their prior assumptions are perfectly satisfied.
doi:10.1016/0888-613x(88)90168-5 fatcat:z3h3g5ddm5gf7bg2ystmf6ik4u