The release of the distributed DBMS rqlite 6.0 is presented, which uses SQLite as a storage engine and allows organizing the operation of a cluster from storages synchronized with each other. Of the features of rqlite, it is noted that it is easy to install, deploy and maintain a distributed fault-tolerant storage, somewhat similar to etcd and Consul, but using a relational data model instead of the key / value format. The project code is written in Go and distributed under the MIT license.
The Raft consensus algorithm is used to keep all nodes in a synchronized state. Rqlite uses the original SQLite library and the standard go-sqlite3 driver, on top of which a layer is launched that processes client requests, performs replication to other nodes and monitors the achievement of consensus on the choice of the leader node.
Changes to the database can only be made by the node that is selected as the leader, but connections with write operations can also be directed to other nodes of the cluster, which will return the address of the leader to repeat the request (in the next version, they promise to add automatic forwarding of the call to the leader). The main emphasis is on fault tolerance, so the DBMS scales only on reads, and writes are the bottleneck. It is possible to run an rqlite cluster from a single node and such a solution can be used to provide access to SQLite over HTTP without providing fault tolerance.
The SQLite data on each node is not stored in a file, but in memory. At the level of the layer with the implementation of the Raft protocol, a log of all SQLite commands that lead to a change in the database is kept. This log is used during replication (replication at the level of replaying queries on other nodes), starting a new node, or recovering from a loss of connectivity. To reduce the size of the log, automatic packaging is used, which starts after a specified number of changes and leads to the fixation of a snapshot on the disk, relative to which a new log is started (the state of the database in memory is identical to the snapshot + the accumulated change log).
rqlite features:
- Ease of deployment of a cluster, without the need for a separate installation of SQLite.
- The ability to quickly get replicated SQL storage.
- Ready for use in working projects (Production-grade).
- The presence of an HTTP(S) API that allows you to update data in batch mode and determine the leading node of the cluster. It also provides a command line interface and the ability to use various client libraries built for SQLite.
- The presence of a service for determining other nodes, allowing you to create clusters dynamically.
- Support for encryption of data exchange between nodes.
- Ability to adjust the level of checking the relevance and consistency of data when reading.
- The optional ability to connect nodes in read-only mode that do not take part in the determination of consensus and are used to increase the scalability of the cluster for read operations.
- Support for a native form of transactions based on combining commands in a single request (transactions based on BEGIN, COMMIT, ROLLBACK, SAVEPOINT and RELEASE are not supported).
- Support for creating hot backups.
The new release includes significant architectural changes to improve cluster reliability by improving the process of directing read and write requests to the correct cluster nodes. rqlite nodes can now multiplex multiple logical connections between themselves using TCP connections established between nodes by the Raft protocol. If the request requires the authority of the leading node, but is sent to a secondary node, the secondary node can determine the leader's address and pass it to the client, without performing the Raft protocol consensus calculation.
The change also eliminated the separate component for metadata synchronization and eliminated the separate handling of Raft state and metadata. Secondary nodes now send requests to the leader node only when necessary, when they need to know the address of the leader node. The API provides the ability to obtain information about the state of other nodes in the cluster. The ".sysdump" command has been added to the CLI.
Source: opennet.ru