Alibaba, one of the largest Chinese IT companies, has open-sourced PolarDB, a distributed database based on PostgreSQL. PolarDB extends the capabilities of PostgreSQL with tools for distributed data storage with integrity and support for ACID transactions in the context of the entire global database, distributed across different nodes of the cluster. PolarDB also supports distributed SQL query processing, fault tolerance, and redundant data storage to replenish information after one or more nodes fail. If you need to expand storage, you can simply add new nodes to the cluster. The code is open under the Apache 2.0 license.
PolarDB consists of two components - extensions and a set of patches for PostgreSQL. The patches extend the capabilities of the PostgreSQL core, and the extensions include components implemented separately from PostgreSQL, such as a distributed transaction management engine, global services, a distributed SQL query processor, additional metadata, tools for managing a cluster, deploying a cluster, and simplifying the transfer of existing systems to it.
The patches add to the PostgreSQL kernel a distributed version of the parallel data access control mechanism using multiversion concurrency control (MVCC, Multiversion concurrency control) for different isolation levels. Much of PolarDB's functionality has been moved into extensions, which reduces the dependency on PostgreSQL and makes it easier to upgrade and implement PolarDB-based solutions (simplifies the transition to new versions of PostgreSQL and maintains full compatibility with PostgreSQL). To manage the cluster, the pgxc_ctl toolkit is used, based on a similar utility from PostgreSQL-XC and PostgreSQL-XL.
There are three basic components in a cluster: database nodes (DN), cluster manager (CM), and transaction management service (TM). Additionally, a proxy load balancer can be used. Each of the components is a separate process and can be run on a different server. DB nodes serve SQL queries from clients and at the same time act as coordinators of distributed query execution with the participation of other DB nodes. The cluster manager monitors the status of each database node, stores the cluster configuration, and provides tools for managing, backing up, load balancing, updating, starting, and stopping the nodes. The transaction management service is responsible for maintaining the overall integrity of the entire cluster.
PolarDB is based on the Shared-nothing distributed computing architecture, in accordance with which data is distributed during storage to different nodes, without using a common storage for all nodes, and each node is responsible for the portion of data attached to it and executes queries related to the data. Each table is segmented into parts (sharding) using hashing by the primary key. If the request covers data located on different nodes, a distributed transaction execution mechanism and a transaction coordinator are connected to ensure atomicity, consistency, isolation and reliability (ACID).
For fault tolerance, each shard is replicated to at least three nodes. To save resources, the full data includes only two replicas, and one is limited to storing the lazy log (WAL). One of the two nodes with full replicas is chosen as the leader and participates in the processing of requests. The second node acts as a spare for the data segment under consideration, and the third takes part in the choice of the leading node and can be used to restore information in the event of failure of two nodes with full replicas. Data replication between cluster nodes is organized using the Paxos algorithm, which provides a consistent definition of consensus in a network with potentially unreliable nodes.
It is noted that the full functionality of the PolarDB DBMS is planned to be disclosed within three releases: In the first version, replication, high availability and cluster management tools will be published. The second release will implement a distributed transaction execution system that supports per-node ACID and distributed SQL execution. The third release will include a plugin for PostgreSQL and tools for flexible distribution of data across nodes, including adaptive shard placement for optimal performance and the ability to expand the cluster by adding new nodes.
Source: opennet.ru