Databricks announced the launch of the large language model DBRX, which can be used to create chatbots that answer questions in natural language, solve proposed mathematical problems, generate content on specified topics, and write code in various programming languages. The model was developed by Mosaic ML, which was acquired by Databricks for $1.3 billion. A cluster of 3072 NVIDIA H100 Tensor Core GPUs was used for training. It is recommended to have 320GB of memory for deploying the model.
The model was trained using the MoE (Mixture of Experts) architecture, enabling more accurate expert evaluation, along with a collection of texts and code totaling 12TB. The context size considered by the DBRX model is 32,000 tokens (the number of tokens that the model can process and remember when generating text). In comparison, the context size for Google Gemini and OpenAI GPT-4 models is 32,000 tokens, Google Gemma is 8,000, while the GPT-4 Turbo model accommodates 128,000 tokens.
The model encompasses 132 billion parameters and is divided into 16 expert networks, of which no more than 4 can be used when processing a request (covering no more than 36 billion parameters per token). For comparison, the GPT-4 model reportedly includes 1.76 trillion parameters, the recently unveiled X/Twitter model Grok (X/Twitter) has 314 billion, GPT-3.5 has 175 billion, YaLM (Yandex) has 100 billion, LLaMA (Meta) has 65 billion, GigaChat (Sber) has 29 billion, and Gemma (Google) has 7 billion.
The model and its associated components are distributed under the Databricks Open Model License, which allows for use, reproduction, copying, modification, and creation of derivative products, but with certain restrictions. For instance, the license prohibits using DBRX, derivative models, and any output based on them to enhance other language models distinct from DBRX. The license also forbids the model's use in areas that violate laws and regulations. Derivative models must be distributed under the same license. When used in products and services that reach over 700 million users per month, a separate permission is required.
According to the creators of the model, DBRX surpasses the characteristics and capabilities of the GPT-3.5 model from OpenAI and Grok-1 from Twitter, and can compete with the Gemini 1.0 Pro model when tested for language comprehension, coding abilities in programming languages, and solving mathematical problems. In some applications, such as generating SQL queries, DBRX approaches the efficiency of the GPT-4 Turbo model, which leads the market. Additionally, the model stands out from competing services with its very fast performance, allowing for almost instantaneous responses. Specifically, DBRX can generate text at a speed of up to 150 tokens per second per user, which is about twice as fast as the LLaMA2-70B model.
Additionally, it is worth noting the publication of the technical description of the open large language model InternLM2, which is distributed under the Apache 2.0 license and is available in versions with 20, 7, and 1.8 billion parameters. The model is being developed by a Shanghai artificial intelligence laboratory with the participation of several Chinese universities and is notable for its context handling of up to 200K tokens and support for both English and Chinese languages. In many tests, the model is close to GPT-4.
Furthermore, there are reports of the development of 84 new matrix multiplication cores for the llamafile toolkit, which is being developed by Mozilla and allows for the creation of universal executable files for running large machine learning language models (LLM). The changes have significantly accelerated the performance of models in llamafile when executed on CPU. For instance, executing models using llamafile is now faster by 30% to 500% compared to using llama.cpp, depending on the environment, and matrix operations that fit in L2 cache execute twice as fast in the new implementation compared to the MKL library.
Source: opennet.ru
