The non-profit organization OpenSearch Software Foundation, governed by the Linux Foundation, has released version 3.0 of the OpenSearch project, which is a fork of the search, analytics, and data storage platform Elasticsearch and the Kibana web interface. Companies such as Amazon, SAP, Uber, Aryn, Atlassian, Canonical, DigitalOcean, and NetAp are participating in the development of the fork. The code is distributed under the Apache 2.0 license.
The fork was created in 2021 in response to the transition of the Elasticsearch project to a non-open license SSPL (Server Side Public License) and the cessation of publishing changes under the older Apache 2.0 license. Despite Elasticsearch's return to using an open license, the OpenSearch project remains relevant, as it continues to use the permissive Apache 2.0 license instead of the AGPLv3 license adopted by Elasticsearch, and it is developing a number of specific extensions that were previously offered by Amazon in a separate distribution called Open Distro for Elasticsearch, replacing paid components of Elasticsearch.
OpenSearch includes the OpenSearch storage and search engine, the OpenSearch Dashboards web interface and data visualization environment, and a set of plugins for machine learning, SQL support, notification generation, cluster performance diagnostics, traffic encryption, role-based access control (RBAC), authentication via Active Directory, Kerberos, SAML, and OpenID, single sign-on (SSO) implementation, and detailed logging for auditing.
Changes in OpenSearch 3.0 include:
- A vector engine (OpenSearch Vector Engine) has been added, which can be used to store and work with data utilized in machine learning systems. To speed up vector searches, GPU computations are employed, which have increased indexing speed by 9.3 times and reduced operational costs by 3.75 times compared to solutions that rely solely on CPU. Support for the MCP (Model Context Protocol) has been implemented to facilitate interaction with data sources, LLM applications, and AI platforms. Integration with AI agents from companies like Anthropic, LangChain, and OpenAI is supported.
- Optimization has been added, allowing for a one-third reduction in the storage size of k-NN (k-nearest neighbors) vectors, as well as reducing delays by up to 30 times during query execution immediately after startup (cold start) by eliminating redundant secondary information and using primary data to reconstruct the necessary information.
- An experimental feature has been added that uses the gRPC protocol (protobuf over gRPC) for data transmission between clients, servers and storage nodes. Compared to JSON, using gRPC reduces serialization overhead and increases performance by allowing simultaneous sending of different requests over a single TCP connection.
- A pull mode for data retrieval has been added, where OpenSearch directly requests data from streaming sources such as Apache Kafka and Amazon Kinesis.
- The cluster now has the ability to separate traffic related to indexing and searching. An API has been added to disable write operations and keep the index available only for search in order to optimize data handling for configurations where data is written once and read multiple times.
- Integration with Apache Calcite has been expanded and the ability to use the PPL (Piped Processing Language) query language for searching, filtering, and merging operations has been implemented.
- Automatic detection of index types has been ensured. Specific optimizations for log-related data indices have been employed to accelerate log analysis operations.
- The Lucene full-text search engine has been updated to version 10, which enhances index handling and improves the performance of parallel task processing.
- Support for Java modules (Java Platform Module System) has been added for separating components into individual libraries. Java 21 has been declared as the minimum version.
- The performance of operations with ranges and fields containing dates and numbers has been accelerated (the throughput of the Big5 test set increased by 25%). Data aggregation operations have been accelerated (in the p90 latency test, delays have been reduced by 75%). For k-NN vectors, the parallel search mode for segments is enabled by default, resulting in a 2.5 times increase in query performance.
Source: opennet.ru
