Here is the organized content with headings and subheadings:
Path Attribution Methods
Path attribution methods are a gradient-based way of explaining deep models. These methods require choosing a hyperparameter known as the baseline input. What does this hyperparameter mean, and how important is it? In this article, we investigate these questions using image classification networks as a case study. We discuss several different ways to choose a baseline input and the assumptions that are implicit in each baseline.
Introduction
If you are in the business of training neural networks, you might have heard of the integrated gradients method, which was introduced at ICML two years ago. The method computes which features are important to a neural network when making a prediction on a particular data point. This helps users understand which features their network relies on. Since its introduction, integrated gradients has been used to interpret networks trained on a variety of data types, including retinal fundus images and electrocardiogram recordings.
Image Classification
We focus on image classification as a task, as it will allow us to visually plot integrated gradients attributions and compare them with our intuition about which pixels we think should be important. We use the Inception V4 architecture, a convolutional neural network designed for the ImageNet dataset, in which the task is to determine which class an image belongs to out of 1000 classes. On the ImageNet validation set, Inception V4 has a top-1 accuracy of over 80%. We download weights from TensorFlow-Slim and visualize the predictions of the network on four different images from the validation set.
A Better Understanding of Integrated Gradients
As you look through the attribution maps, you might find some of them unintuitive. Why does the attribution for “goldfinch” highlight the green background? Why doesn’t the attribution for “killer whale” highlight the black parts of the killer whale? To better understand this behavior, we need to explore the assumptions behind each baseline input.
Baselines in Path Attribution
Of course, we modified MNIST in this example specifically so that expected gradients attributions of an accurate model would look exactly like those of a randomly initialized model. The way we did this is similar to the decoy MNIST dataset, except instead of the top left corner encoding the class label, we randomly scattered noise throughout each training and test image where the intensity of the noise encodes the true class label. Generally, you would run these kinds of saliency method sanity checks on un-modified data.
Conclusion
So what should be done? We have many baselines and no conclusion about which one is the “best.” Although we don’t provide extensive quantitative results comparing each baseline, we do provide a foundation for understanding them further. At the heart of each baseline is an assumption about missingness in our model and the distribution of our data. In this article, we shed light on some of those assumptions, and their impact on the corresponding path attribution.
Related Methods
This work focuses on a specific interpretability method: integrated gradients and its extension, expected gradients. We refer to these methods as path attribution methods because they integrate importances over a path. However, path attribution methods represent only a tiny fraction of existing interpretability methods. We focus on them here both because they are amenable to interesting visualizations, and because they provide a springboard for talking about missingness.
FAQs
What is path attribution?
Path attribution methods are a gradient-based way of explaining deep models. They require choosing a hyperparameter known as the baseline input.
What is the baseline input?
The baseline input is a hyperparameter in path attribution methods that requires choosing a specific input to the model.
Why is the constant black image a “natural baseline” for image data?
The constant black image is a “natural baseline” for image data because it is a common and intuitive choice for a baseline input in path attribution methods.
How do I choose a baseline input?
The choice of baseline input depends on the specific problem and data type. In this article, we discuss several different ways to choose a baseline input and the assumptions that are implicit in each baseline.
What is missingness in machine learning models?
Missingness refers to the concept of representing missing data in machine learning models. In this article, we discuss how missingness is related to path attribution methods and the choice of baseline input.
What are some other related methods?
There are many other related methods, including SHAP, DeepSHAP, layer-wise relevance propagation, LIME, RISE, and Grad-CAM, among others. Many of these methods define some notion of baseline or missingness, as missingness and explanations are closely related.
