Deconvolution and Checkerboard Artifacts
When we look very closely at images generated by neural networks, we often see a strange checkerboard pattern of artifacts. It’s more obvious in some cases than others, but a large fraction of recent models exhibit this behavior.
Mysteriously, the checkerboard pattern tends to be most prominent in images with strong colors. What’s going on? Do neural networks hate bright colors? The actual cause of these artifacts is actually remarkably simple, as is a method for avoiding them.
Deconvolution & Overlap
When we have neural networks generate images, we often have them build them up from low-resolution, high-level descriptions. This allows the network to describe the rough image and then fill in the details.
In order to do this, we need some way to go from a lower resolution image to a higher one. We generally do this with the deconvolution operation. Roughly, deconvolution layers allow the model to use every point in the small image to "paint" a square in the larger one.
Overlap & Learning
Thinking about things in terms of uneven overlap is – while a useful framing – kind of simplistic. For better or worse, our models learn weights for their deconvolutions.
In theory, our models could learn to carefully write to unevenly overlapping positions so that the output is evenly balanced. This is a tricky balancing act to achieve, especially when one has multiple channels interacting. Avoiding artifacts significantly restricts the possible filters, sacrificing model capacity. In practice, neural networks struggle to learn to completely avoid these patterns.
Better Upsampling
To avoid these artifacts, we’d like an alternative to regular deconvolution ("transposed convolution"). Unlike deconvolution, this approach to upsampling shouldn’t have artifacts as its default behavior. Ideally, it would go further, and be biased against such artifacts.
One approach is to make sure you use a kernel size that is divided by your stride, avoiding the overlap issue. This is equivalent to "sub-pixel convolution," a technique which has recently had success in image super-resolution.
Another approach is to separate out upsampling to a higher resolution from convolution to compute features. For example, you could use a convolutional neural network to compute features at a lower resolution, and then use a separate upsampling operation to bring the image to the desired resolution.
Conclusion
In this article, we’ve seen how deconvolution can lead to checkerboard artifacts in neural network-generated images. We’ve also discussed how these artifacts can be caused by uneven overlap, and how this can be a problem for learning models. Finally, we’ve seen that there are alternative approaches to upsampling that can avoid these artifacts.
Frequently Asked Questions
Q: What is the cause of checkerboard artifacts in neural network-generated images?
A: The cause of checkerboard artifacts is uneven overlap in deconvolution.
Q: How can I avoid checkerboard artifacts?
A: You can avoid checkerboard artifacts by using an alternative to regular deconvolution, such as sub-pixel convolution or separating out upsampling from convolution.
Q: Why do neural networks struggle to avoid checkerboard artifacts?
A: Neural networks struggle to avoid checkerboard artifacts because of the problem of uneven overlap in deconvolution, which can be difficult to overcome.
