Researchers from Nijmegen University (Netherlands) have prepared a ranking of the openness of 40 large language models and 7 models for generating images from text descriptions, which are claimed by their producers to be open. Due to the fact that the criteria for the openness of machine learning models are still being developed, there is currently a situation where models that have licensing restrictions on their use (for example, many models prohibit application in commercial projects) are circulated under the guise of being open. Often, manufacturers also do not provide access to the training data used, do not disclose implementation details, or do not fully open the accompanying code.
Most models marketed as 'open' should actually be viewed as 'open weight coefficients' or more accurately 'available weight coefficients,' as they are distributed under restrictive licenses that prohibit use in commercial products. Third-party researchers can experiment with such models, but they do not have the ability to adapt the model to their needs or inspect its implementation. More than half of the models do not provide detailed information about the data used for training, nor do they publish information about their internal structure and architecture.
The most open models recognized include BloomZ, AmberChat, OLMo, Open Assistant, and Stable Diffusion, which are published under open licenses along with their source data, code, and API implementation. Models from Google (Gemma 7B), Microsoft (Orca 2), and Meta (Llama 3), marketed by their manufacturers as open, have ranked closer to the bottom, as they do not provide access to the source data, do not disclose technical implementation details, and distribute the model's weight coefficients under licenses that limit their use. The popular model Mistral 7B placed approximately in the middle of the ranking, as it is provided under an open license but is only partially documented, does not disclose the training data used, and has not fully opened its accompanying code.
Researchers have proposed 14 criteria for the openness of AI models, covering conditions for the distribution of code, training data, weights, data variants, and coefficients optimized through reinforcement learning (RL), as well as the availability of ready-to-use packages, APIs, documentation, and detailed implementation descriptions.
According to the draft definition of open AI proposed by the Open Source Initiative (OSI), the main criteria for the openness of an AI system include providing the ability to use it for any purpose without needing special permission; studying how the system works and inspecting its components; making modifications for any purpose; and transferring both the original version and the modified versions to others.
To allow for modifications, an AI system must include:
- Detailed information about the data used for training and the training methodology. This information should be sufficient for a professional developer to recreate an equivalent AI system using the same or similar data.
- The availability of source code that allows for both the operation of the AI system and the execution of its training process (in the aforementioned table, many models indicate "~" in the "code" column, which implies partial availability of code, where code for running the model is available, but code for training or creating the model is not). The code should also cover areas such as preprocessing, data validation, and tokenization. Additionally, a detailed description of the model's architecture should be provided.
- Model parameters (weights), indicating the availability of a ready-to-use snapshot after training or a finalized optimized version of the model.
Source: opennet.ru
