XTX Markets has open-sourced the TernFS file system, designed to create distributed storage spanning multiple geographically distributed data centers and capable of storing tens of exabytes of data, trillions of files, and hundreds of billions of directories, all accessed simultaneously by millions of clients. The project's code is written in C++ and Go and is licensed under the GPLv2. The protocol specifications and client libraries are licensed under the Apache 2.0 license.
The file system is optimized for workloads typical of machine learning systems—writing and reading files that are immutable after creation and contain more than a few megabytes of data. The file system is not designed for frequent creation of new directories or frequent file transfers between directories. Features include recovery of accidentally deleted files or directories, as well as configuration of policies for permanent deletion. File system contents can be replicated between multiple data centers in different regions. Storage nodes can include various types of storage (disk-based and flash memory).
One of the design requirements was the absence of a single point of failure and high storage reliability. Power outages should not damage the file system, and files cannot be partially written (only fully written files are accessible). The system is resilient to failure of individual drives, metadata nodes, and storage nodes. Storage maintenance is possible without interrupting operations.
XTX Markets began implementing TernFS-based storage in the summer of 2023, after a year and a half of development. By mid-2024, all XTX Markets machine learning systems were migrated to TernFS. The storage currently comprises over 30 disks and 10 flash drives, housed in three data centers. The total stored data volume is estimated at 500 petabytes, with peak performance reaching several petabytes per second. The storage is used in a cluster of over 100 computing nodes. No incidents resulting in data loss have occurred in two years of operation.
TernFS limitations include: written files cannot be modified; low efficiency when working with files smaller than 2 MB; low performance of directory creation and deletion operations; lack of permissions separation.
The components that make up a TernFS storage cluster are:
- Metadata storage nodes responsible for managing information about directory structures and file attributes.
- Cross-Directory Coordinator (CDC) nodes that perform transactions across different directories.
- Storage services that provide storage of file contents.
- A registry that contains information about all other services and monitors their operation.
- Client services for users to access the FS:
- ternweb — access to storage via Web API.
- terncli is a command-line interface.
- ternfs.ko — kernel module Linux, which allows you to mount the storage to work as a local file system.
- ternfuse is a client implementation for working with FS, using the FUSE subsystem.
- terns3 is an implementation of the Amazon S3 API.
- Background processes:
- GC is a garbage collector that frees expired snapshots (deleted but not purged files) and cleans up blocks for permanently deleted files.
- scrubber — data degradation detection and recovery.
- migrator - exclude failed disks from storage.
Source: opennet.ru
