The ChatTTS project has released a model and associated machine learning tools for emotional speech synthesis. The ChatTTS project is specifically optimized for use in dialogue systems, such as interactive assistants, and aims to replicate the properties of natural emotional communication. It supports interaction with multiple speakers and the construction of interactive dialogue. Prosodic elements such as laughter, pauses, and interjections are correctly tracked and reproduced during synthesis.
Approximately 40,000 hours of speech recordings were used to train the model (100,000 hours in the non-public version). According to the developers, the model surpasses all previously available open-source speech synthesis models in its ability to generate intonations. Currently, only token substitution is supported for emotional control during synthesis, for example, "[laugh]" for laughter. To generate a 30-second recording, a GPU with 4GB of memory is required. With a NVIDIA GeForce RTX 4090 GPU, the generation speed is approximately 7 semantic tokens per second. Male and female voice synthesis is supported in English and Chinese (for Russian, the TTS framework and XTTS-v2 model are recommended, which support both synthesis and voice cloning from a short speech sample, including synthesis in another language).
The ChatTTS model is published under the CC BY-NC-ND 4.0 license (Creative Commons Attribution-NonCommercial-NoDerivatives 4.0), which allows for free distribution with attribution to the author, but prohibits the creation of derivative works and use in commercial projects. Additionally, for protection against using the model for fraudulent and criminal activities, high-frequency noise substitution was employed during model training, and maximum audio compression was utilized using the MP3 format.
Source: opennet.ru
