Automated training systems (ATS) for a range of skills are an emerging artificial intelligence (AI) technology that aims to deliver automatic assessment and feedback in the absence of human expert input. However, most current systemsare not designed to explain the assessment to help trainees improve their motivation and performance. We propose that explanations on the assessments and predictions made by these automated systems can act as feedback providing tangible informational insights, and

more »

... creasing the trainee's effectiveness and performance. Explainable AI (XAI) methods promoting human understanding of the AI models' decisions have the potential to satisfy these informational needs of the feedback. However, explainable ATS must adhere to the main elements of feedback seen in traditional manual methods (e.g., actionable, clear impact, withexamples). This paper discusses the validity of the explanation as feedback and its challenges in designing and implementing explainable ATS.