Feature Attributions and Counterfactual Explanations Can Be Manipulated [article]

Dylan Slack, Sophie Hilgard, Sameer Singh, Himabindu Lakkaraju
<span title="2021-06-25">2021</span> <i > arXiv </i> &nbsp; <span class="release-stage" >pre-print</span>
As machine learning models are increasingly used in critical decision-making settings (e.g., healthcare, finance), there has been a growing emphasis on developing methods to explain model predictions. Such explanations are used to understand and establish trust in models and are vital components in machine learning pipelines. Though explanations are a critical piece in these systems, there is little understanding about how they are vulnerable to manipulation by adversaries. In this paper, we
more &raquo; ... cuss how two broad classes of explanations are vulnerable to manipulation. We demonstrate how adversaries can design biased models that manipulate model agnostic feature attribution methods (e.g., LIME & SHAP) and counterfactual explanations that hill-climb during the counterfactual search (e.g., Wachter's Algorithm & DiCE) into concealing the model's biases. These vulnerabilities allow an adversary to deploy a biased model, yet explanations will not reveal this bias, thereby deceiving stakeholders into trusting the model. We evaluate the manipulations on real world data sets, including COMPAS and Communities & Crime, and find explanations can be manipulated in practice.
<span class="external-identifiers"> <a target="_blank" rel="external noopener" href="https://arxiv.org/abs/2106.12563v2">arXiv:2106.12563v2</a> <a target="_blank" rel="external noopener" href="https://fatcat.wiki/release/6eidicjv2vaxdb6f6vftjscp64">fatcat:6eidicjv2vaxdb6f6vftjscp64</a> </span>
<a target="_blank" rel="noopener" href="https://web.archive.org/web/20210625142930/https://arxiv.org/pdf/2106.12563v1.pdf" title="fulltext PDF download [not primary version]" data-goatcounter-click="serp-fulltext" data-goatcounter-title="serp-fulltext"> <button class="ui simple right pointing dropdown compact black labeled icon button serp-button"> <i class="icon ia-icon"></i> Web Archive [PDF] <span style="color: #f43e3e;">&#10033;</span> <div class="menu fulltext-thumbnail"> <img src="https://blobs.fatcat.wiki/thumbnail/pdf/4b/29/4b29b460d73929aee1dd57998563a03090eeb125.180px.jpg" alt="fulltext thumbnail" loading="lazy"> </div> </button> </a> <a target="_blank" rel="external noopener" href="https://arxiv.org/abs/2106.12563v2" title="arxiv.org access"> <button class="ui compact blue labeled icon button serp-button"> <i class="file alternate outline icon"></i> arxiv.org </button> </a>