Let’s start with generating new strokes based on your handwriting input
Neural networks are an extremely successful approach to machine learning, but it’s tricky to understand why they behave the way they do. This has sparked a lot of interest and effort around trying to understand and visualize them, which we think is so far just scratching the surface of what is possible.
In this article, we will try to push forward in this direction by taking a generative model of handwriting and visualizing it in a number of ways. The model is quite simple (so as to run well in the browser) so the generated output mostly produces gibberish letters and words (albeit, gibberish that look like real handwriting), but it is still useful for our purposes of exploring visualization techniques.
Looking at the Output of the Model
Our first experiment is the most obvious: when we want to see how well someone has learned a task, we usually ask them to demonstrate it. So, let’s ask our model to write something for us and see how well it does.
Most of the marks are gibberish, but many of them are surprisingly convincing. Some real (or real-ish) words even start to appear. One surprising thing you’ll notice is that the general style of handwriting is more or less consistent within a sample. This is because the type of architecture used for this model (LSTM) has a mechanism for remembering previous strokes. It is therefore able to remember things like how loopy or jerky the handwriting is, or which letter preceeded the current one. (For more on LSTMs and how they can remember, Chris Olah has a good primer
Examining the Internals of the Model
Our model has 500 cells which act as a sort of memory which it will use as part of its input when deciding what to generate. If we can see what those cells are doing as the model progresses we may be able to gain some intuitive understanding about what the model is doing.
We begin by showing the activation of the cells over time. Each column in the heatmap below represents one line segment of the handwriting. Each row represents one cell of the model and is colored by its activations on that part of the stroke. By inspecting the diagram you may be able to see some patterns in the way certain cells activate for certain types of strokes.
Conclusion
The black box reputation of machine learning models is well deserved, but we believe part of that reputation has been born from the programming context into which they have been locked. The experience of having an easily inspectable model available in the same programming context as the interactive visualization environment (here, javascript) proved to be very productive for prototyping and exploring new ideas for this post.
As we are able to move them more and more into the same programming context that user interface work is done, we believe we will see richer modes of human-ai interactions flourish. This could have a marked impact on debugging and building models, for sure, but also in how the models are used. Machine learning research typically seeks to mimic and substitute humans, and increasingly it’s able to. What seems less explored is using machine learning to augment humans. This sort of complicated human-machine interaction is best explored when the full capabilities of the model are available in the user interface context.
FAQs
Q: What is the purpose of this article?
A: The purpose of this article is to explore ways to visualize a generative model of handwriting and to gain a better understanding of how it works.
Q: What is a generative model of handwriting?
A: A generative model of handwriting is a type of machine learning model that is trained to generate new handwritten text based on a dataset of existing handwritten text.
Q: What is the main goal of this article?
A: The main goal of this article is to demonstrate how to visualize a generative model of handwriting and to provide a deeper understanding of how it works.
Q: What are some of the key findings of this article?
A: Some of the key findings of this article include the ability to visualize the output of the model, the ability to examine the internals of the model, and the potential for machine learning to augment humans in complex tasks.
Q: What are some potential applications of this technology?
A: Some potential applications of this technology include generating handwritten text for use in documents, such as signatures, and for use in art and design.
