The company xAI, founded by Elon Musk and which has received about a billion dollars for the development of technologies related to artificial intelligence, announced the discovery of a large Grok language model used in a chatbot integrated into the social network X (Twitter). The set of weighting coefficients, neural network architecture, and use cases are published under the Apache 2.0 license. A ready-to-use archive with the model, 296 GB in size (magnet), is available for download.
The Grok model is pre-trained on a large collection of text data using xAI's proprietary learning stack and spans approximately 314 billion parameters, making it the largest open large language model available. For comparison, the recently opened Gemma model by Google has 7 billion parameters, Sber GigaChat - 29 billion parameters, Meta LLaMA - 65 billion, Yandex YaLM - 100 billion, OpenAI GPT-3.5 - 175 billion, and the market leader, the GPT-4 model, supposedly includes 1.76 trillion parameters.
The open version of the Grok-1 model is published in a basic representation and does not include optimizations for certain areas of use, such as organizing dialog systems. For testing, a GPU with a large amount of memory is required (exactly what kind of memory is not specified). A static cast of the model is publicly available, while one of the features of the Grok chatbot being developed for Twitter is dynamic adaptation to emerging new content (integration with the X/Twitter platform is used to access new knowledge).
Built on Grok, the chatbot outperforms GPT-3.5 in tests for solving high school math problems (GSM8k), generating answers to interdisciplinary questions (MMLU), completing Python code (HumanEval), and solving university math problems described in LaTeX format (MATH ).
Source: opennet.ru