Adoption of Artificial Intelligence (AI) algorithms into the clinical realm will depend on their inherent trustworthiness, which is built not only by robust validation studies but is also deeply linked to the explainability and interpretability of the algorithms. Most validation studies for medical imaging AI report performance of algorithms on study level labels and lay little emphasis on measuring the accuracy of explanations generated by these algorithms in the form of heat maps or bounding

more »

... oxes, especially in true positive cases. We propose a new metric, Explainability Failure Ratio (EFR), derived from Clinical Explainability Failure (CEF) to address this gap in AI evaluation. We define an Explainability Failure as a case where the classification generated by an AI algorithm matches with study level ground truth but the explanation output generated by the algorithm is inadequate to explain the algorithms output. We measured EFR for two algorithms that automatically detect consolidation on chest X rays to determine the applicability of the metric and observed a lower EFR for the model that had lower sensitivity for identifying consolidation on chest X rays, implying that trustworthiness of a model should be determined not only by routine statistical metrics but also by novel clinically oriented models.