A bank declines a loan. A hospital tool flags a patient as high risk. A hiring system ranks candidates. In each case, people reasonably ask: why? Explainable AI (XAI) is the set of methods and practices aimed at making the reasoning of AI systems understandable to humans. It matters for trust, debugging, fairness and, increasingly, for legal and regulatory requirements.
Why explanations matter
- Trust and adoption: users are more willing to rely on systems they can question.
- Debugging: explanations reveal when a model uses the wrong signals, such as hospital watermarks on X-rays instead of the lung.
- Fairness: they help detect when sensitive attributes or their proxies drive decisions.
- Accountability: decision makers need to justify actions affecting people.
- Regulation: laws in some regions require meaningful information about automated decisions affecting individuals.
- Improvement: understanding failures guides better data and features.
Interpretable models versus post-hoc explanations
Inherently interpretable models
Some models are transparent by design.
- Linear and logistic regression: each feature has a weight that shows direction and strength of influence.
- Decision trees (small): a path of if-then rules.
- Rule lists and scoring systems: simple points-based tools, common in medicine and credit.
- Generalised additive models: show each feature's effect curve.
If an interpretable model performs well enough, it is often the best choice for high-stakes decisions.
Post-hoc explanation of complex models
For black-box models such as gradient boosting ensembles and deep neural networks, we use methods that explain predictions after the fact.
Global versus local explanations
- Global: how the model behaves overall. "Income and payment history are the most influential factors."
- Local: why this one prediction was made. "This application was declined mainly because of two recent missed payments."
Popular techniques
Feature importance
Measures how much each feature contributes to model performance. Permutation importance shuffles one feature and checks how much accuracy drops. It is simple and model-agnostic, but can mislead when features are correlated.
Partial dependence and ICE plots
Show how the prediction changes as one feature varies, averaging over others (partial dependence) or for individual examples (ICE curves).
LIME
Local Interpretable Model-agnostic Explanations fits a simple model around a single prediction, by perturbing the input and observing outputs, to approximate the complex model locally.
SHAP
Based on Shapley values from game theory, SHAP assigns each feature a contribution to a specific prediction so that contributions add up to the difference from the average prediction. It provides both local and global views and is widely used with tree models.
Saliency and attention for images and text
Gradient-based methods and Grad-CAM highlight which pixels or words influenced the output. These can be useful but also misleading, so they need validation.
Counterfactual explanations
"If your income were 20,000 higher, the loan would be approved." Counterfactuals tell people what would need to change, which is practical and actionable, but they must be realistic and respect what can actually be changed.
Example-based explanations
Show similar past cases or prototypes. "This image was classified as healthy because it resembles these examples."
A SHAP example in Python
import shap
import pandas as pd
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import train_test_split
data = load_breast_cancer(as_frame=True)
X_train, X_test, y_train, y_test = train_test_split(data.data, data.target, random_state=0)
model = GradientBoostingClassifier(random_state=0).fit(X_train, y_train)
explainer = shap.TreeExplainer(model)
shap_values = explainer(X_test)
# Why did the model predict this particular patient the way it did?
shap.plots.waterfall(shap_values[0])
# Which features matter most overall?
shap.plots.bar(shap_values)The waterfall plot shows how each feature pushes the prediction up or down from the baseline. Use it as a clue, not as definitive proof of causation.
Limits and risks of explanations
- Explanation is not causation. A feature that drives the model's output does not necessarily cause the real-world outcome.
- Correlated features split or hide credit among themselves.
- Instability: small changes in the input or method can alter explanations.
- Persuasive but wrong: a plausible explanation may not reflect what the model truly does (unfaithful explanations).
- Different audiences need different explanations: a data scientist, a regulator and a customer have different needs.
- Overtrust: explanations can create false confidence in a flawed model.
- Gaming: revealing too much may allow people to manipulate the system.
Large language models add another twist: they can write fluent self-explanations that do not necessarily reflect their internal computation.
Explanations in practice
- Know the audience and purpose: debugging, user-facing, audit.
- Choose a model appropriate to the stakes: simple and transparent for high-stakes, regulated areas when performance permits.
- Use more than one method and check consistency.
- Test explanations: do domain experts agree they make sense? Do they change when you change the data?
- Present them clearly: use plain language, limit to the most important factors and show uncertainty.
- Offer recourse: people should know how to appeal or correct inaccurate data.
- Document the system: model cards and data sheets describing purpose, training data, performance and limitations.
Fairness link
Explanations can reveal that postal code or language is acting as a proxy for protected characteristics. Combine them with fairness metrics and with human review of outcomes.
Frequently Asked Questions
Is explainability the same as interpretability?
They are often used interchangeably. Interpretability usually refers to a model being understandable by design; explainability includes methods to explain complex models afterwards.
Do all AI systems need explanations?
Not all, but the more a decision affects people's rights, safety or money, the more important explanations and oversight become.
Can I trust SHAP or LIME completely?
They are useful tools with limitations. Treat them as evidence to investigate rather than final truth.
How can I explain a large language model's answer?
Ask for sources and verify them, use retrieval so answers cite documents, and test behaviour systematically. Do not rely only on the model's own explanation.
Conclusion
Explainable AI combines interpretable models, explanation methods and clear communication so people can question and trust automated decisions. Use it to debug, check fairness and support accountability, while remembering that explanations have limits.