Image Inpainting for Art Restoration Using Generative Adversarial Networks
A study of Generative Adversarial Networks (GANs) for the purpose of Image Inpainting, filling in missing or obscured sections of images.
PROBLEM
The problem of Image Inpainting aims to reconstruct missing parts of images. For this Independent Study, I have analyzed the Image Inpainting task for the purpose of restoring damaged artwork. I chose art restoration as my area of focus based on my previous coursework studying medieval art when I studied abroad in Siena, Italy. Much of the art produced during that time period was made with fragile plaster, which is subject to damage over time. I was inspired by the damaged sections of art to tackle the problem using deep learning.
DATA
To achieve the best results on Image Inpainting for artwork, I landed upon using a curated dataset that features art by the same artist in the same style to add uniformity to the dataset. I found that using artwork across artists, styles, and media leads to worse results. The dataset I used consists of around 1,000 training images and 100 validation images of Pierre-Auguste Renior’s Impressionist Paintings downloaded from an open source art dataset, Wikiart. I used Renior’s Impressionist works because they were the largest in number for the same artist with the same style out of the Wikiart dataset. In the future, this dataset could be expanded so that the model would be trained on all types of images. I decided to not do this in my Independent Study because I was constrained for time, and focusing on refining the model was a higher priority task. The uniform dataset helped me refine the model in a more efficient manner given I knew irregularities in the data were not causing problems while training.
RESULTS
Figure 1. Inpainting trained on a singular random mask generated by Perlin Noise and tested on the same random mask. After 200 epochs of training.
The best results were created when the system was trained on the same mask that it was tested on as seen in Figure 1. The system is able to learn colors and edges of the input images after only 200 epochs of training.
However, if each image is trained on a random mask, the system is not able to learn the colors and edges of the training images as seen in Figure 2.
Figure 2. Inpainting trained on on singular random mask generated by Perlin Noise and tested on a circle. After 200 epochs of training.
CHALLENGES & MOVING FORWARD
Ideally, the system would be able to accept any mask to fill, even if it was not trained on that exact mask. However, as presented above, the model is not able to successfully train on multiple random masks. If training on multiple masks continues to be a problem, there could be a few reasons for why the system is failing under these circumstances. One option is that there is not enough data. The system is only being trained on around 1,000 images, which may not be enough to learn the necessary patterns.
It is also possible that artwork images are not regular enough for pattern recognition. To test this theory, the generator could be trained on the CelebA dataset, which offers more regularity. A graph plotting the generator and discriminator errors would show if the losses are converging on this dataset and not the Renoir dataset, meaning the data is the problem. Another issue could be that the model is not training for long enough which could be tested by checking error convergence on training the model for longer.
The problem of art restoration itself also presents some inherent challenges. The major consideration when repairing art is the style of the piece. This style needs to be applied to the reconstructed section. To tackle the problem of style application, I added a function to extract the style layers from the training images. It takes certain layers from the pre-trained VGG-19 network which uses 5 convolutional blocks. The style layers are the first convolutional layer in each block. These style layers could be used to inform the losses of the generator and discriminator going forward. I was not able to implement the use of the style layers for training during my independent study due to time constraints.