Stability AI has released the second edition of the machine learning system Stable Diffusion, capable of synthesizing and modifying images based on a provided template or natural language text description. The code for training the neural network and generating images is written in Python using the PyTorch framework and is published under the MIT license. Pre-trained models are open under the permissive license Creative ML OpenRAIL-M, allowing commercial use. Additionally, a demo online image generator is available.
Key improvements in the new edition of Stable Diffusion:
- A new image synthesis model based on text descriptions has been created — SD2.0-v, supporting image generation at a resolution of 768×768. The new model has been trained using the LAION-5B dataset, which includes 5.85 billion images with text descriptions. The model uses the same set of parameters as the Stable Diffusion 1.5 model, but differs by transitioning to a fundamentally different encoder, OpenCLIP-ViT/H, which has significantly improved the quality of the resulting images.
- A simplified version SD2.0-base has been prepared, trained on 256×256 images using a classic noise prediction model and supporting image generation at a resolution of 512×512.
- The ability to use super-resolution technology to increase the resolution of the original image without a loss in quality has been provided by using spatial scaling and detail reconstruction algorithms. The provided image processing model (SD20-upscaler) supports a fourfold increase in scale, allowing for the creation of images with a resolution of 2048×2048.
- The model SD2.0-depth2img has been proposed, taking into account the depth and spatial arrangement of objects. A monocular depth estimation system, MiDaS, is used. The model allows for the synthesis of new images using another image as a template, which can radically differ from the original while maintaining overall composition and depth. For example, one can use the pose of a person in a photograph to create a different character in the same pose.
- The model for image modification has been updated — SD 2.0-inpainting, allowing parts of an image to be replaced and altered using text prompts.
- The models have been optimized for use on standard systems with a single GPU.
Source: opennet.ru
