The source texts of the PIXIE machine learning system, which allows you to create 3D models and animated avatars of the human body from a single photo, are open. Realistic textures of the face and clothes can be attached to the resulting model, which differ from those depicted in the original photo. The system can be used, for example, to draw from a different viewpoint, create animation, reconstruct the body according to the shape of the face, and form a 3D model of the fingers. The code is written in Python using the Pytorch framework and distributed under a license that allows for non-commercial use only.
It is stated that compared to similar projects, PIXIE allows you to more accurately recreate the contours of the body, originally hidden by clothing in the photograph, the shape of the face and the position of the joints of the hands. The method is based on the use of a neural network that extracts face, body, and hand parameters from a pixel image. The work of the neural network is coordinated by a special regulator, which, based on the analysis of illumination, adds information about the weight coefficients of various parts of the body to eliminate the definition of unnatural poses. When creating a model, anatomical differences between the male and female bodies, pose parameters, lighting, surface reflectivity and face rotation in a three-dimensional plane are taken into account.
PIXIE Features:
- The reconstructed 3D model of the body, as well as information about the pose, position of the hands and facial expressions are saved as a set of SMPL-X parameters, which can later be used in the Blender modeling system through a plugin.
- Detailed information about the shape and expression of the face, as well as its features, such as the presence of wrinkles, is determined from the photograph (the DECA machine learning system developed by the same authors is used to build the head model).
- When generating a face texture, the object's albedo is estimated.
- The constructed body model can later be animated or presented in a different pose.
- Support for building a model from ordinary photographs in which a person is taken in natural conditions. PIXIE does a good job of defining different poses, lighting conditions, and obscuring the visibility of parts of an object.
- High performance, suitable for dynamic processing of the camera image.
Source: opennet.ru