{"id":143998,"date":"2025-09-23T05:12:23","date_gmt":"2025-09-23T03:12:24","guid":{"rendered":"https:\/\/prohoster.info\/blog\/novosti-interneta\/otkryt-kod-raspredelyonnoj-fajlovoj-sistemy-ternfs"},"modified":"2025-09-23T05:12:23","modified_gmt":"2025-09-23T03:12:24","slug":"otkryt-kod-raspredelyonnoj-fajlovoj-sistemy-ternfs","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/news\/otkryt-kod-raspredelyonnoj-fajlovoj-sistemy-ternfs","title":{"rendered":"The code for the distributed file system TernFS has been opened.","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

XTX Markets has opened the source code of the TernFS file system, designed to create a distributed storage solution spanning several geographically separated data centers capable of storing up to tens of exabytes of data, trillions of files, and hundreds of billions of directories, which millions of clients can interact with simultaneously. The project code is written in C++ and Go and is distributed under the GPLv2 license. Protocol specifications and client libraries are provided under the Apache 2.0 license. <\/p>\n

The file system is optimized for workloads typical of machine learning systems \u2014 reading and writing files that are immutable after creation and contain more than a few megabytes of data. The file system is not designed for frequent creation of new directories and frequent movement of files between directories. It includes capabilities for recovering accidentally deleted files or directories, as well as for configuring policies for complete irreversible deletion. The file system's contents can be replicated across multiple data centers in different regions. Storage nodes may include various types of storage devices (disk-based and flash memory). <\/p>\n

One of the design requirements was to avoid a single point of failure and ensure high reliability of storage \u2014 power outages should not lead to corruption of the file system, and files cannot be partially written (only fully written files are available for reading). The system is resilient against the failure of individual storage devices, metadata nodes, and storage nodes. Storage maintenance can be performed without stopping operations. <\/p>\n

The implementation of storage based on TernFS at XTX Markets began in the summer of 2023, after one and a half years of developing the file system. By mid-2024, all machine learning systems at XTX Markets were migrated to TernFS. Currently, the storage comprises over 30,000 disks and 10,000 flash drives located in three data centers. The total volume of stored data is estimated at 500 petabytes, with peak performance reaching several petabytes per second. The storage is used in a cluster consisting of more than 100,000 computing nodes. No incidents resulting in data loss have occurred in two years of operation. <\/p>\n

Among the limitations of TernFS: written files cannot be modified; low efficiency when working with files smaller than 2 MB; low performance in directory creation and deletion operations; lack of access rights separation. <\/p>\n

The components that make up the TernFS storage cluster: <\/p>\n