NVIDIA has published the source code for the SPADE (GauGAN) machine learning system, which can synthesize realistic landscapes from rough sketches, as well as untrained models associated with the project. The system was demonstrated in March at the GTC 2019 conference, but the code was only published yesterday. The developments are open under a free license CC BY-NC-SA 4.0 (Creative Commons Attribution-NonCommercial-ShareAlike 4.0), allowing use only for non-commercial purposes. The code is written in Python using the PyTorch framework.
The sketches are drawn up in the form of a segmented map that determines the placement of approximate objects on the scene. The nature of the generated objects is specified using color marks. For example, a blue fill transforms into the sky, blue into water, dark green into trees, light green into grass, light brown into stones, dark brown into mountains, gray into snow, a brown line transforms into a road, and a blue line into a river. Additionally, based on the selection of reference images, the overall composition style and time of day are determined. The proposed tool for creating virtual worlds can be useful to a wide range of specialists, from architects and urban planners to game developers and landscape designers.
Objects are synthesized by a generative adversarial neural network (GAN), which creates realistic images based on a schematic segmented map, borrowing details from a model pre-trained on several million photographs. Unlike previously developed image synthesis systems, the proposed method is based on the use of adaptive spatial transformation followed by transformation based on machine learning. Processing a segmented map instead of semantic markup allows you to achieve exact match results and control the style.
To achieve realism, two competing neural networks are used: a generator and a discriminator (Discriminator). The generator generates images based on mixing elements of real photographs, and the discriminator identifies possible deviations from real images. As a result, a feedback is formed, on the basis of which the generator begins to compose more and more high-quality samples, until the discriminator no longer distinguishes them from the real ones.
Source: opennet.ru