NVIDIA has opened the code for a machine learning system that synthesizes landscapes from sketches

NVIDIA Company опубликовала source texts of the machine learning system SWORDS (GauGAN), which allows you to synthesize realistic landscapes based on rough sketches, as well as those associated with the project trained models. The system was demonstrated in March at the GTC 2019 conference, but the code was only published yesterday. Developments open under a proprietary license CC BY-NC-SA 4.0 (Creative Commons Attribution-NonCommercial-ShareAlike 4.0) for non-commercial use only. The code is written in Python using the framework PyTorch.

Sketches are made in the form of a segmented map that determines the placement of approximate objects on the scene. The nature of the generated objects is set using color labels. For example, blue fills are converted to sky, blue to water, dark green to trees, light green to grass, light brown to rocks, dark brown to mountains, gray to snow, brown line to road, and blue line to river. Additionally, based on the choice of reference images, the general style of the composition and the time of day are determined. The proposed tool for creating virtual worlds can be useful to a wide range of professionals, from architects and urban planners to game developers and landscape designers.

Objects are synthesized by a generative adversarial neural network (GAN), which creates realistic images based on a schematic segmented map, borrowing details from a model previously trained on several million photographs. Unlike previously developed image synthesis systems, the proposed method is based on the application of adaptive spatial transformation followed by transformation based on machine learning. Processing a segmented map instead of semantic markup allows you to achieve an exact match of the result and control the style.

To achieve realism, two competing neural networks are used: a generator and a discriminator (Discriminator). The generator generates images based on mixing elements of real photographs, and the discriminator identifies possible deviations from real images. As a result, a feedback is formed, on the basis of which the generator begins to compose more and more high-quality samples, until the discriminator no longer distinguishes them from the real ones.

Source: opennet.ru