A bias/variance decomposition for models using collective inference

Jennifer Neville, David Jensen
2008 Machine Learning  
Bias/variance analysis is a useful tool for investigating the performance of machine learning algorithms. Conventional analysis decomposes loss into errors due to aspects of the learning process, but in relational domains, the inference process used for prediction introduces an additional source of error. Collective inference techniques introduce additional error, both through the use of approximate inference algorithms and through variation in the availability of test-set information. To date,
more » ... the impact of inference error on model performance has not been investigated. We propose a new bias/variance framework that decomposes loss into errors due to both the learning and inference processes. We evaluate the performance of three relational models on both synthetic and real-world datasets and show that (1) inference can be a significant source of error, and (2) the models exhibit different types of errors as data characteristics are varied. Keywords Statistical relational learning · Collective inference · Evaluation Introduction Bias/variance analysis (Geman et al. 1992) has been used for a number of years to investigate the mechanisms behind model performance. This analysis has focused on loss as a measure of classification performance-a loss function L(t, y) defines the penalty for predicting class y for an instance x when its true class label value is t . Bias/variance analysis decomposes the expected loss (E[L(t, y)]) for an instance x into three components of error: bias, variance, and noise. Overall model loss can be reduced by reducing either bias
doi:10.1007/s10994-008-5066-6 fatcat:vkyukzo2v5c3dc42ewqgqb3syu