Visualizing Features

Feature Visualization by Optimization

Neural networks are, generally speaking, differentiable with respect to their inputs. If we want to find out what kind of input would cause a certain behavior, whether that’s an internal neuron firing or the final output behavior, we can use derivatives to iteratively tweak the input towards that goal.

While conceptually simple, there are subtle challenges in getting the optimization to work. We will explore them, as well as common approaches to tackle them in the section “The Enemy of Feature Visualization”/>.

Optimization Objectives

What do we want examples of? This is the core question in working with examples, regardless of whether we’re searching through a dataset to find the examples or optimizing images to create them from scratch. We have a wide variety of options in what we search for:

If we want to understand individual features, we can search for examples where they have high values—either for a neuron at an individual position, or for an entire channel. We used the channel objective to create most of the images in this article.

Conclusion

Neural feature visualization has made great progress over the last few years. As a community, we’ve developed principled ways to create compelling visualizations. We’ve mapped out a number of important challenges and found ways of addressing them.

In the quest to make neural networks interpretable, feature visualization stands out as one of the most promising and developed research directions. By itself, feature visualization will never give a completely satisfactory understanding. We see it as one of the fundamental building blocks that, combined with additional tools, will empower humans to understand these systems.

There remains still a lot of important work to be done in improving feature visualization. Some issues that stand out include understanding neuron interaction, finding which units are most meaningful for understanding neural net activations, and giving a holistic view of the facets of a feature.

lucid.

FAQs

Q: What is the goal of feature visualization?
A: The goal of feature visualization is to understand what a neural network is looking for by generating examples of what the network is trying to find.

Q: What are the two main threads of research in neural network interpretability?
A: The two main threads of research in neural network interpretability are feature visualization and attribution.

Q: How do we optimize the input to activate a particular neuron?
A: We can use derivatives to iteratively tweak the input towards that goal, allowing us to find out what kind of input would cause a certain behavior.

Q: What are the different optimization objectives in feature visualization?
A: The different optimization objectives in feature visualization include searching for examples of individual features, entire channels, or output classes from a classifier.

Q: What is the difference between maximizing class logits and class probabilities?
A: Maximizing class logits before the softmax can produce images of better visual quality, but it can also be prone to the issue of pushing down evidence for other classes, whereas maximizing class probabilities can be more effective in terms of the final output.

Q: What is the role of preconditioning in feature visualization?
A: Preconditioning can change the direction of descent, making the optimization problem easier, and can be used to deal with issues such as high-frequency patterns or correlated features.