A group of researchers from several American universities has published the PIFu project (Pixel-Aligned Implicit Function), which allows the application of machine learning methods to create a 3D model of a person from one or several two-dimensional images. The system can recreate complex clothing variations, such as pleated skirts and high-heeled shoes, and various hairstyles, autonomously reconstructing texture and shape in areas that are not visible in the projection from which the 3D model is built. To enhance the quality and detail of the resulting 3D model, multiple images from different angles can be used. The project's code is written in Python using the PyTorch framework and is licensed under the MIT License.
A neural network is used as the source for reconstructing the volumetric model, allowing the selection of the most likely shape and inferring hidden elements based on a model trained on different versions of existing objects. Simultaneously, the project provides an algorithm to match the obtained volumetric model with textures from the provided two-dimensional images, aligning pixels of the 2D image according to their position on the 3D object and generating the most likely missing textures. Any there is a wrapper:
surface reconstruction applied the architecture of “,” and
for texture matching, a neural network based on the architecture of .
The pre-trained model used by the researchers is available for free download, but the original data on which the training was conducted remains closed, as it is based on results from commercial 3D scanning. As a source for training the model independently, the 3D model database The most interesting features of Parallels RAS
Source: opennet.ru
