A team of researchers from Tel Aviv University has unveiled HyperStyle, an inverted version of NVIDIA's StyleGAN2 machine learning system that has been redesigned to recreate missing parts when editing real-life images. The code is written in Python using the PyTorch framework and distributed under the MIT license.
If StyleGAN allows you to synthesize realistic-looking new faces of people by setting parameters such as age, gender, hair length, smile pattern, nose shape, skin color, glasses and photo angle, then HyperStyle makes it possible to change similar parameters in existing photos without changing their characteristic features and retaining the recognizability of the original face. For example, using HyperStyle, you can simulate a change in the age of a person in a photo, change a hairstyle, add glasses, a beard or mustache, make an image look like a cartoon character or a hand-drawn picture, make a sad or cheerful expression. In this case, the system can be trained not only to change the faces of people, but also for any objects, for example, for editing images of cars.
The proposed method is aimed at solving the problem with the reconstruction of the missing parts of the image when editing. In the previous methods, the compromise between reconstruction and editability was solved by fine-tuning the image generator to substitute parts of the target image when recreating initially missing editable areas. The disadvantage of such approaches is the need for long-term targeted training of the neural network for each image.
The method based on the StyleGAN algorithm makes it possible to use a typical model, previously trained on common collections of images, to generate elements characteristic of the original image with a level of confidence comparable to algorithms that require individual training of the model for each image. Among the advantages of the new method, the possibility of modifying images with a performance close to real time is also noted.
The pre-trained models are prepared for human, car and animal faces based on the collections of Flickr-Faces-HQ (FFHQ, 70k high-quality PNG images of human faces), Stanford Cars (16k images of cars) and AFHQ (photos of animals). In addition, tools are provided for training their models, as well as ready-made trained models of typical encoders and generators suitable for use with them. For example, generators are available to create Toonify-style images, Pixar characters, sketching, and even styling them like Disney princesses.
Source: opennet.ru