XAI is the idea of creating Artificial Intelligence (AI) algorithms capable of explaining their purpose, rationale and decision-making process in a human understandable way. In this article, Prof Alexiei Dingli, from the Department of AI, University of Malta, delves further into the topic by giving an overview of the challenges which currently exist in the field and why it is vital for the progress of AI algorithms.