Making Sense of Hidden Layers
Much of the recent work on interpretability is concerned with a neural network’s input and output layers.
Arguably, this focus is due to the clear meaning these layers have: in computer vision, the input layer represents values for the red, green, and blue color channels for every pixel in the input image, while the output layer consists of class labels and their associated probabilities.
How Trustworthy Are These Interfaces?
In order for interpretability interfaces to be effective, we must trust the story they are telling us.
We perceive two concerns with the set of building blocks we currently use.
First, do neurons have a relatively consistent meaning across different input images, and is that meaning accurately reified by feature visualization?
Semantic dictionaries, and the interfaces that build on top of them, are premised off this question being true.
Second, does attribution make sense and do we trust any of the attribution methods we presently have?
Much prior research has found that directions in neural networks are semantically meaningful.
One particularly striking example of this is “semantic arithmetic” (e.g. “king” – “man” + “woman” = “queen”).
We explored this question, in depth, for GoogLeNet in our previous article and found that many of its neurons seem to correspond to meaningful ideas.
For more details, see the article’s appendix and the guided tour in @ch402’s Twitter thread.
We’re actively investigating why GoogLeNet’s neurons seem more meaningful.
Conclusion & Future Work
There is a rich design space for interacting with enumerative algorithms, and we believe an equally rich space exists for interacting with neural networks.
We have a lot of work left ahead of us to build powerful and trustworthy interfaces for interpretability.
But, if we succeed, interpretability promises to be a powerful tool in enabling meaningful human oversight and in building fair, safe, and aligned AI systems.
FAQs
Q: What is the goal of this article?
A: The goal is to explore the importance of interpretability in neural networks and to discuss the need for building interfaces that can effectively communicate the meaning of neural networks to humans.
Q: What are the challenges in building these interfaces?
A: The challenges include developing techniques that can accurately reify the meaning of neural networks and ensuring that the interfaces do not mislead users.
Q: Why are semantic dictionaries important?
A: Semantic dictionaries are important because they allow us to map neurons to meaningful concepts, making it easier to understand the meaning of neural networks.
Q: What are the limitations of current attribution methods?
A: Current attribution methods have limitations, including the problem of path-dependent attribution, where the output of a function can be the result of non-linear interactions between its inputs.
Q: What is the future of interpretability research?
A: The future of interpretability research is promising, with the potential to develop powerful and trustworthy interfaces that can enable meaningful human oversight and build fair, safe, and aligned AI systems.
