The CAP theorem is the cornerstone of distributed systems theory. Of course, debates around it continue: its definitions are not canonical, and there is no strict proof... Nevertheless, firmly standing on the position of common sense™, we intuitively understand that the theorem is true.
The one thing that is not obvious is the meaning of the letter 'P'. When a cluster splits, it decides whether to not respond until a quorum is reached, or to provide the data that is available. Depending on the results of this choice, the system is classified as either CP or AP. Cassandra, for example, can behave either way, depending not even on the cluster settings but on the parameters of each specific request. But if the system is not 'P', and it has split, then what?
The answer to this question is somewhat unexpected: a CA cluster cannot split.
What sort of cluster is it that cannot split?
An essential attribute of such a cluster is a shared data storage system. In the overwhelming majority of cases, this means connecting through SAN, which limits the use of CA solutions to large enterprises that can maintain SAN infrastructure. For several servers to work with the same data, a clustered file system is required. Such file systems are available in the portfolios of HPE (CFS), Veritas (VxCFS), and IBM (GPFS).
Oracle RAC
The Real Application Cluster option first appeared in 2001 with the release of Oracle 9i. In such a cluster, multiple instances server operate on the same database.
Oracle can work with both a clustered file system and its own solution – ASM, Automatic Storage Management.
Each instance maintains its own log. A transaction is executed and recorded by one instance. In the event of a failure of an instance, one of the surviving nodes of the cluster (instances) reads its log and recovers the lost data – this ensures availability.
All instances maintain their own cache, and the same pages (blocks) can be present in the caches of multiple instances simultaneously. Moreover, if one instance needs a certain page, and it is in the cache of another instance, it can acquire it from the 'neighbor' using the cache fusion mechanism instead of reading from disk.
But what happens if one of the instances needs to change data?
A feature of Oracle is that it does not have a dedicated locking service: if the server wants to lock a row, the lock record is placed directly on the memory page where the locked row is located. Thanks to this approach, Oracle is a performance champion among monolithic databases: the locking service never becomes a bottleneck. However, in a clustered configuration, such an architecture can lead to intensive network exchange and mutual blocking.
As soon as a record is locked, the instance notifies all other instances that the page containing this record is monopolized. If another instance needs to change a record on the same page, it must wait until the changes on the page are committed, meaning the change information is written to the disk log (while the transaction may continue). It can also happen that the page is modified sequentially by multiple instances, and then, when writing the page to disk, it has to determine which instance holds the current version of that page.
Randomly updating the same pages through different RAC nodes leads to a sharp drop in database performance — to the point that the cluster's performance can be lower than that of a single instance.
The proper use of Oracle RAC is the physical partitioning of data (for example, using partitioned tables) and accessing each set of partitions through a dedicated node. The main purpose of RAC has become not horizontal scaling but ensuring fault tolerance.
If a node stops responding to the heartbeat, the node that detects this first initiates a voting procedure on the disk. If the lost node does not respond here either, one of the nodes takes on the responsibility of data recovery:
- it "freezes" all the pages that were in the cache of the lost node;
- reads the redo logs of the lost node and reapplies the changes recorded in those logs, while checking if any other nodes have fresher versions of the modified pages;
- rolls back unfinished transactions.
To simplify the switching between nodes, Oracle introduces the concept of a service – a virtual instance. An instance can serve multiple services, and a service can move between nodes. An application instance that serves a specific part of the database (for example, a group of clients) works with one service, while the service responsible for that part of the database migrates to another node in the event of a node failure.
IBM Pure Data Systems for Transactions
The clustered solution for DBMS appeared in the Blue Giant's portfolio in 2009. Ideologically, it is a descendant of the Parallel Sysplex cluster built on 'regular' hardware. The DB2 pureScale product was released in 2009, representing a software package, and in 2012, IBM offered a software-hardware bundle (appliance) called Pure Data Systems for Transactions. It should not be confused with Pure Data Systems for Analytics, which is simply a rebranded Netezza.
The pureScale architecture resembles Oracle RAC at first glance: similarly, several nodes are connected to a shared data storage system, and each node runs its own DBMS instance with its own memory areas and transaction logs. However, unlike Oracle, DB2 has a dedicated locking service represented by a set of processes db2LLM*. In the clustered configuration, this service is moved to a separate node, which in Parallel Sysplex is called coupling facility (CF), and in Pure Data – PowerHA.
PowerHA provides the following services:
- lock manager;
- global buffer cache;
- inter-process communication area.
To transfer data from PowerHA to the database nodes and back, remote memory access is used, so the cluster interconnect must support the RDMA protocol. PureScale can use both Infiniband and RDMA over Ethernet.
If a node needs a page that is not in the cache, it requests the page from the global cache, and only if it is not there either does it read it from disk. Unlike Oracle, the request goes only to PowerHA, not to neighboring nodes.
If an instance is about to change a row, it locks it in exclusive mode, and the page where the row is located in shared mode. All locks are recorded in the global lock manager. When the transaction completes, the node sends a message to the lock manager, which copies the modified page to the global cache, releases the locks, and invalidates the modified page in other nodes' caches.
If the page containing the modifiable row is already locked, the lock manager will read the modified page from the memory of the node that made the changes, release the lock, invalidate the modified page in the caches of other nodes, and grant the page lock to the requesting node.
‘Dirty’, that is, modified, pages can be written to disk both from a regular node and from PowerHA (castout).
In the event of a failure of one of the nodes, pureScale recovery is limited to only those transactions that were not yet completed at the time of the failure: pages modified by this node in completed transactions are present in the global cache on PowerHA. The node is restarted in a reduced configuration on one of the cluster servers, rolls back the uncompleted transactions, and frees the locks.
PowerHA operates on two servers, and the primary node synchronously replicates its state. In the event of a failure of the primary node, the PowerHA cluster continues to operate with the backup node.
Of course, accessing the dataset through a single node will yield higher overall performance for the cluster. PureScale can even notice that a certain area of data is being processed by a single node, and then all locks related to that area will be handled by the node locally without communications with PowerHA. However, as soon as the application tries to access that data through another node, centralized lock processing will resume.
IBM's internal load tests, consisting of 90% read and 10% write, which closely resembles real industrial loads, show almost linear scaling up to 128 nodes. Unfortunately, the testing conditions are not disclosed.
HPE NonStop SQL
Hewlett-Packard Enterprise also has its own high-availability platform. This is the NonStop platform, launched in 1976 by Tandem Computers. In 1997, the company was acquired by Compaq, which in turn merged with Hewlett-Packard in 2002.
NonStop is used to build critical applications, such as HLR or credit card processing. The platform is delivered as a software-hardware complex (appliance), which includes computing nodes, storage systems, and communication equipment. The ServerNet network (Infiniband in modern systems) serves both for communication between nodes and for access to the storage system.
In early versions of the system, proprietary processors were used that were synchronized with each other: all operations were executed synchronously by multiple processors, and as soon as one of the processors encountered an error, it was shut down, while the other continued to operate. Later, the system transitioned to standard processors (initially MIPS, then Itanium, and finally x86), and other mechanisms were used for synchronization:
- messages: each system process has a duplicate 'shadow', to which the active process periodically sends messages about its state; upon failure of the main process, the shadow process begins to operate from the point indicated by the last message;
- voting: the storage system has a special hardware component that receives several identical requests and executes them only if they match; instead of physical synchronization, processors operate asynchronously, and the results of their work are compared only at input/output moments.
Since 1987, a relational DBMS has been operating on the NonStop platform – first SQL/MP, and later SQL/MX.
The entire database is divided into parts, with each part managed by its own Data Access Manager (DAM) process. It handles data writing, caching, and locking mechanisms. Data processing is carried out by executor processes (Executor Server Process) running on the same nodes as the corresponding data managers. The SQL/MX scheduler divides tasks among the executors and combines the results. For making coordinated changes, a two-phase commit protocol provided by the TMF (Transaction Management Facility) library is used.
NonStop SQL can prioritize processes to ensure that long analytical queries do not interfere with transaction execution. However, its primary purpose is to handle short transactions, not analytics. The developer guarantees the availability of the NonStop cluster at five nines, meaning downtime is only 5 minutes per year.
SAP HANA
The first stable release of the HANA DBMS (1.0) was in November 2010, and the SAP ERP package transitioned to HANA in May 2013. The platform is based on acquired technologies: the TREX Search Engine (for searching in columnar storage), the P*TIME DBMS, and MAX DB.
The term 'HANA' is an acronym for High-performance ANalytical Appliance. This DBMS is delivered as code that can run on any x86 servers; however, industrial installations are only permitted on certified hardware. Solutions are available from HP, Lenovo, Cisco, Dell, Fujitsu, Hitachi, and NEC. Some Lenovo configurations even allow operation without SAN, where a GPFS cluster on local disks acts as a shared storage system.
Unlike the previously mentioned platforms, HANA is an in-memory DBMS, meaning the primary data is stored in RAM, while only logs and periodic snapshots are written to disk for recovery in case of a failure.
Each node in the HANA cluster is responsible for its portion of the data, and the data map is stored in a special component called the Name Server, located on the coordinator node. Data is not duplicated between nodes. Information about locks is also stored on each node, but the system has a global deadlock detector.
The HANA client, when connecting to the cluster, loads its topology and can subsequently access any node directly depending on the data it needs. If a transaction involves data from a single node, it can be executed locally by that node. However, if data from multiple nodes is being modified, the initiating node contacts the coordinating node, which opens and coordinates the distributed transaction, committing it using an optimized two-phase commit protocol.
The coordinating node is duplicated, so in the event of a coordinator failure, a backup node immediately takes over. If a data node fails, the only way to access its data is to restart the node. Typically, HANA clusters keep a spare server to quickly restart the lost node.
Source: habr.com
