PayPal has opened source code for JunoDB, a fault-tolerant DBMS that manipulates data in key-value format. The system was originally designed with high security, horizontal scalability, fault tolerance and the ability to handle hundreds of thousands of simultaneous connections with predictable delays in mind. In PayPal, almost all services, from user login to processing financial transactions, are tied to JunoDB. The project code is written in Go (Java client library) and distributed under the Apache 2.0 license. In further development, corrections, improvements and changes from the community will be accepted.
The JunoDB architecture is based on the use of a load balancer that accepts requests from client applications and distributes them among proxy servers that simultaneously access a group of storage servers when a request is made. Each proxy establishes connections to all storage servers at once and redirects requests to a group of storage servers based on the partition index stored in the distributed storage system of the etcd configuration.
Data is partitioned and bound to storage nodes using hashing to reduce data movement as nodes grow or shrink in a cluster. To ensure fault tolerance, each portion of data is replicated on several storage nodes, which allows you to save information when individual servers fail. The creation of geographically distributed storages is supported, in which groups of nodes are located in different data centers.
On storage nodes, data is placed in RAM or in local storage based on the RocksDB library. With persistent storage, data is stored in encrypted form (the encryption key can be determined both by the client and set at the proxy level).
To access the database from applications, a client library is supplied that provides an API for applications in Java, Go, and C++. The client part is simplified as much as possible, and complex logic and settings, if possible, are moved to the side of the DBMS. The interaction between the client and the balancer or proxy is carried out through an encrypted communication channel. You can use a command line interface to manage and send requests, which mimics the full functionality of the client API.
The system is designed to process requests with predictable low latency, for example, a cluster of three storage nodes and one proxy, formed from n1-highmem-32 environments (32 Intel Xeon 2.30GHz CPUs, 214G RAM and 450G SSD-based storage), was able to provide fixed delays not exceeding 2.5 ms in 95% of cases and 16 ms in 99% when processing 200 thousand simultaneous TLS connections and a flow of 15 thousand requests per second (with 3000 simultaneous connections and a flow of 80 thousand requests per second, delays did not exceed 6 ms in 95% of cases and 15ms in 99%). At PayPal, JunoDB-based services serve about 350 billion requests per day.
Source: opennet.ru