{"id":117781,"date":"2024-07-18T03:43:48","date_gmt":"2024-07-18T01:43:48","guid":{"rendered":"https:\/\/prohoster.info\/blog\/novosti-interneta\/novaya-statya-praktikum-po-ii-risovaniyu-chast-devyataya-sd3m-troechka-na-troechku"},"modified":"2024-07-18T03:43:48","modified_gmt":"2024-07-18T01:43:48","slug":"novaya-statya-praktikum-po-ii-risovaniyu-chast-devyataya-sd3m-troechka-na-troechku","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/news\/novaya-statya-praktikum-po-ii-risovaniyu-chast-devyataya-sd3m-troechka-na-troechku","title":{"rendered":"New Article: AI Drawing Practicum, Part Nine: SD3M \u2014 a \u2018three\u2019 for the \u2018three\u2019","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

It's amusing how the history of the release of open models for converting text prompts into images, developed and trained by Stability AI, resembles a series of ups and downs much like the successive versions of Microsoft's operating systems. After the legendary success of XP, we saw the problematic Vista; then came the magnificent 'seven' \u2014 followed by the fruitless Windows 8. Following the initially unremarkable version 1.5, which was gradually refined by enthusiasts, came the outright unsuccessful SD 2.0 \u2014 unsuccessful primarily because it included the atypical OpenCLIP encoder for models of this kind. trained on rather ambiguously selected images<\/a><\/noindex> from from the open dataset LAION-5B<\/a><\/noindex>During this selection process, not only inappropriate (NSFW) visual references were filtered out, but also artworks created by popular artists like the infamous Greg Rutkowski. This last point completely infuriated enthusiasts: while for SD 1.5, even in its original version without using specially trained checkpoints, simple prompts with styles \u2014 'epic medieval fantasy landscape, in the style of Greg Rutkowski' \u2014 produced impressive results, SD 2.0 stopped 'recognizing' the names of the most widely known illustrators, whose copyright on their created works is still relevant and who did not consent to allow these works for AI training. It became necessary to describe the desired more elaborately, and with overly long prompts, the model performed poorly.<\/p>\n

<\/center> \t\t\t\t
<\/center> \"New <\/p>\n

\u2193<\/a><\/noindex>#<\/a><\/noindex>Sat down \u2014 stood up, sat down \u2014 stood up<\/h2>\n

Combined with the modified encoder (the converter that transforms text prompts into digital tokens, which the model operates on afterwards), the inability to apply named styles simply deprived enthusiasts of the motivation to refine the 'two' on their own. I mean, here one would need to simultaneously figure out how to adapt the existing prompt construction techniques to the new encoder, and further train the model to recognize those images and themes with which the creators had not acquainted it during the initial training phase\u2014after all, the generated images from the 'two' did not guarantee any qualitative difference from the 'one and a half'. Yes, there was a significant leap in quality from the standard size of 512\u00d7512 for SD 1.5 to 768\u00d7768, but by that time the community was already actively using upscalers, outpainters, and other tools to increase the final image size, so SD 2.0 largely went unnoticed. returned to the standard<\/a><\/noindex> (developed by OpenAI and used, in particular, by the DALL-E project) CLIP encoder<\/a><\/noindex> \u2014 its code is also open, but the database on which it is trained is unlike OpenCLIP<\/a><\/noindex>, proprietary. Additionally, the standard canvas size for 'Oversize' has increased to 1024\u00d71024, plus a whole range of additional improvements has emerged\u2014so enthusiasts took pleasure in further developing it. As of now, it is indeed SDXL (and its recently emerged derivatives that are less demanding on hardware, such as SDXL Turbo<\/a><\/noindex> and SDXL Lightning<\/a><\/noindex>) can confidently be considered the most popular open-source AI image generator. However, staunch supporters of SD 1.5 argue with this, pointing out that crucial tools like ControlNet have not been adequately transferred to SDXL. As of June 12, 2024, when the model code allowing for local generations was 'released into the wild', it was supposed to mark the time for the 'open' version of SD 3 \u2014 to be more precise,<\/a><\/noindex>.<\/p>\n

And since June 12, 2024, with the 'wild release' of the model's code allowing for local generations, it was supposed to be the time for the 'open' version of SD 3\u2014more precisely, (SD3M or SD3 2B) with 2 billion working parameters. Conventionally, we remind you that this number corresponds to the total amount of weights at the inputs of all perceptrons in the model. Even earlier, in April, Stability AI refined and proposed an 8-million parameter version for commercial use<\/a><\/noindex> Stable Diffusion 3 Large Stable Diffusion 3 Large<\/a><\/noindex>, also known as SD3 8B. Let us remember that for SDXL 1.0\u2014 3.5 billion parameters<\/a><\/noindex>, however, SD3M, according to the developers, is the 'most sophisticated model for generating images of all that we have created so far.'<\/a><\/noindex>It was meant that with lower video memory requirements than those of 'Oversize', it should still generate images 'with a new level of photorealism' even in response to simple prompts. Among the advantages of the 'trio' were also mentioned 'unprecedented typography quality of the generated text in images', 'deeper understanding of prompts thanks to the combined efforts of three encoders', and 'readiness for effective fine-tuning even on limited datasets'.<\/p>\n

\"New <\/p>\n

It was presumably expected that the community of enthusiasts, having received a long-awaited new toy, would rush to refine it with the same zeal as they did with the 'halves' and 'Oversize' in their time. However, from the very beginning, things did not go as planned: SD3M faced its audience, and not once but twice. The first time was due to an inexplicable idiosyncrasy regarding prompts that included the seemingly innocent phrase 'lying on\/in the grass'; the second time was due to the incredibly vague formulations in the user agreement, which even professional lawyers could not immediately understand. And although from the perspective of a regular AI art enthusiast this seems insignificant, the legal aspect will have a direct impact on the further development of the model \u2014 to the point that no development may follow at all. its audience, and not just once. The first time was due to an inexplicable idiosyncrasy with prompts that included the seemingly innocent phrase 'lying on\/in the grass'; the second was the incredibly vague wording in the user agreement, which even professional lawyers couldn't quickly make sense of. And while from the perspective of an ordinary AI art enthusiast this last point may seem insignificant, it will have a direct impact on the future development of the model\u2014up to the point where no development may follow at all.<\/a><\/noindex> with characteristics such as 'perpetual, worldwide, non-exclusive, free, gratuitous, irrevocable copyright license for reproduction, preparation, public demonstration, public performance, sublicensing and distribution of additional materials for both the model itself and its derivatives'. The 'trio' provides for two types of licensing:<\/p>\n

CreativeML Open RAIL++-M License CreativeML Open RAIL++-M License<\/a><\/noindex> \u2014 with rather lenient formulations like 'Stability AI grants you a non-exclusive, worldwide, non-transferable, non-sublicensable, revocable, gratuitous, and limited license for intellectual property' \u2014 and for non-commercial use<\/a><\/noindex> Stable Diffusion 3 \u2014 this is not Open Source a much more burdensome commercial one.<\/a><\/noindex>.<\/p>\n

\u2193<\/a><\/noindex>#<\/a><\/noindex>The grass won't lead to good.<\/h2>\n

After a very short time, the community reached a consensus that Stable Diffusion 3 \u2014 this is not Open Source<\/a><\/noindex>Essentially, it has boycotted the developing company, unwilling to spend time and effort on training a clearly raw and poorly polished model \u2013 being aware that Stability AI can at any moment take back the previously granted license from a specific enthusiast carrying out such training at the slightest whim of their media managers. The licensing agreement is formulated in such a way that non-lawyers studying it are left with the impression that following the revocation of the permission for commercial use of SD3M, the former licensee will be obliged to delete all derivative works created from the licensed intellectual property, including not only the trained models (LoRA, text inversions, entire checkpoints) but also their derivatives (quote: \u201cUpon termination of this Agreement, you shall delete and cease use of any Software Products or Derivative Works\u201d) \u2013 meaning the fruits of the labor of other people who used these derivative models as a starting point for their own work; moreover, work that was neither paid for nor commissioned, executed out of pure enthusiasm.<\/p>\n

Soon after the wave of outrage on this matter reached stratospheric heights, reports started appearing from professional lawyers, that not everything is so bad.<\/a><\/noindex> And that the revocation along with the prohibition of further use should actually pertain only to some auxiliary products with closed code, which Stability AI will hand over to a commercial user (say, for accelerating and optimizing the same training of SD3M) \u2013 yet the company has still not provided final clarifications on this matter. The very fact of such oppressive silence for already three weeks (at the time of writing this article) since the release of the 'trio' into the public domain is harming the developer's reputation much more significantly than the generated content has affected the girls created by its invention.<\/p>\n

\"New <\/p>\n

As for the infamous grass, which literally became a meme in just a few hours, first on Hugging Face,<\/a><\/noindex>, and later on practically all more or less specialized platforms on the Internet, it turned out that the presence in the suggestion of a phrase like «a girl lying on the grass» leads to the appearance in the results of «triple» not just hallucinations, but nightmarish creations of a sick \u2014 in the medical sense \u2014 imagination of artists and filmmakers who specialize in body horror (we urge you, if your sanity and life are dear to you, — stay away from such things and don't even try typing this phrase into the image search window with the safe filter turned off). Meanwhile, images of standing \u2014 and to a slightly lesser extent sitting \u2014 people score a confident four plus from the «triple», while portraits sometimes turn out to be flawless; at least, no worse than the basic model SDXL 1.0; the catch here is precisely some internal taboo on the horizontal position of the human body.<\/p>\n

Judging by the comment from Emad Mostaque, the founder and former (until March 2024) head of Stability AI, who left the company to «engaging in decentralized projects in the field of artificial intelligence»<\/a><\/noindex>the gross violence against SD3M right before the opening of its weights for limited non-commercial use (the API for the larger SD3 8B model for online generation, let\u2019s remember, has been available through partner sites since April,<\/a><\/noindex>, but its weights remain hidden) is a consequence of the current leadership's pursuit of safety — «due to regulatory obligations»<\/a><\/noindex>, articulated back in March this year as Acceptable Use Policy<\/a><\/noindex>. It is within the framework of this policy that users are not allowed to «commit, promote, facilitate, ease, encourage, plan, incite or further violence, terrorism or the creation of hate content that discriminates against or threatens a protected group of people (whether by gender, ethnicity, sexual identity or orientation, religion or otherwise)»; so no images of pandas in their natural state fighting wild dragons for you! Exclusively kittens in funny hats, dogs in cute jackets, bottles with unknown contents, and fresh cookies straight from the oven!<\/p>\n

Technically speaking, the model's de-emphasis may have involved simply excluding images of people lying down and other pictures that could be interpreted as inappropriate from the training dataset. As a result, the 'third model' simply does not 'understand' the meaning of the word 'lie.' Alternatively, the updated architecture of SD3 may have confidently allowed the identification of weights on the perceptrons activated during the generation of 'unsafe' images\u2014these weights were selectively zeroed out just before the release, inadvertently causing 'forced hallucination.' Probably, this can be likened to a lobotomy.<\/a><\/noindex>: the encoder correctly converts text into tokens, but in the 'sanitized' latent space, these tokens no longer refer to anything specific, resulting in the pixels being positioned on the image somewhat randomly.<\/p>\n

\"New <\/p>\n

Considering that the full but closed SD3 8B model's API has been available for several months, along with the enthusiastic assurances from Stability AI's media managers that the 'compactified' 2B will prove to be almost equal in text perception and image generation quality,<\/a><\/noindex>, AI drawing enthusiasts greeted the public release of the weights for SD3M on June 12 with great enthusiasm: within the first 24 hours, the model was downloaded 2.7 million times.<\/a><\/noindex>. Currently, it can still be obtained, although it is a slightly more circuitous route than is common for SDXL and SD 1.5 checkpoints. The conventional route is to go to the Civitai website, from where most models can be downloaded without registration, but from June 17 and until the publication of this article, the page dedicated to the 'third model' on this site temporarily banned<\/a><\/noindex>. And the reason is simple: the commercial license for SD3M is written in such vague language that even Civitai's lawyers have paused for a closer examination. If an enthusiast trains a LoRA for the 'third model' on their PC and uploads it to Civitai, and Stability AI suddenly decides that the result is inappropriate and revokes the license from the offender, what should the host site do? It doesn't just host checkpoints, auxiliary cycle graphs, and models; it also provides visitors with the ability to generate images in the cloud and train the same LoRA, text inversions, and more. In general, while this legal matter is pending, the model files and the three accompanying token converters can only be obtained by from Stability AI's own page on the Hugging Face portal<\/a><\/noindex>.<\/p>\n

\u2193<\/a><\/noindex>#<\/a><\/noindex>Let's begin<\/h2>\n

However, there is a nuance: to access the download links, you must log in to the website and then confirm acceptance of the previously mentioned draconian license agreement \u2014 this procedure is free and fully accessible from Russia. There are a total of four model options and four token text prompt encoders to choose from, plus three standard workflow cycles for execution in the ComfyUI environment. which we have discussed previously<\/a><\/noindex>:<\/p>\n

models:<\/p>\n