XTX Markets has opened the source code of the TernFS file system, designed to create a distributed storage solution spanning several geographically separated data centers capable of storing up to tens of exabytes of data, trillions of files, and hundreds of billions of directories, which millions of clients can interact with simultaneously. The project code is written in C++ and Go and is distributed under the GPLv2 license. Protocol specifications and client libraries are provided under the Apache 2.0 license.
The file system is optimized for workloads typical of machine learning systems—writing and reading files that are immutable after creation and contain more than a few megabytes of data. It is not designed for frequent creation of new directories or for frequent moving of files between directories. There are capabilities for both recovering mistakenly deleted files or directories and for configuring policies for complete irreversible deletion. The contents of the file system can be replicated across multiple data centers in different regions. Storage nodes can include different types of storage devices (both disk-based and flash).
One of the design requirements was the absence of a single point of failure and ensuring high reliability of storage—power outages should not lead to file system corruption, and files cannot be partially written (only fully written files are available for reading). The system is resistant to failures of individual storage devices, metadata nodes, and storage nodes. Maintenance of the storage can be performed without stopping operations.
The implementation of the TernFS-based storage at XTX Markets began in the summer of 2023, after a year and a half of file system development. By mid-2024, all XTX Markets machine learning systems were transitioned to TernFS. Currently, the storage comprises over 30,000 disks and 10,000 flash drives, housed in three data centers. The total volume of stored data is estimated at 500 petabytes, with peak performance reaching several petabytes per second. The storage is used in a cluster comprising over 100,000 compute nodes. There have been no incidents leading to data loss over two years of operation.
Among the limitations of TernFS: written files cannot be modified; low efficiency when working with files smaller than 2 MB; low performance in directory creation and deletion operations; lack of access rights separation.
The components that make up the TernFS storage cluster:
- Metadata storage nodes responsible for managing information about the directory structure and file attributes.
- Coordination nodes (CDC — Cross-Directory Coordinator) that execute transactions spanning different directories.
- Storage services that provide file content storage.
- Registry that contains information about all other services and monitors their operation.
- Client services for user access to the FS:
- ternweb — access to storage via Web API.
- terncli — command-line interface.
- ternfs.ko — kernel module for Linux that allows mounting the storage to operate like a local FS.
- ternfuse — implementation of a client for working with the FS, utilizing the FUSE subsystem.
- terns3 — implementation of the Amazon S3 API.
- Background processes:
- GC — garbage collector that releases expired snapshots (deleted but not cleaned files) and cleans blocks for irreversibly deleted files.
- scrubber — detection of data degradation and recovery.
- migrator — exclusion of failed disks from storage.
Source: opennet.ru
