storage , optimized for storing multi-dimensional arrays and data used in scientific calculations. TileDB is mentioned as applicable in various systems processing genetic information, spatial and financial data, i.e., systems operating on or continuously filled multi-dimensional arrays. TileDB provides a C++ library for transparent abstraction of data and metadata access in applications, taking care of all the low-level organization for efficient storage. The project code is written in C++ and licensed under MIT. It is supported on Linux, macOS, and Windows.
Key features of TileDB:
- Efficient storage methods for sparse arrays where data does not follow continuously, the array is filled in fragments, and most elements remain empty or take the same value.
- The ability to access data in key-value format or sets of columns ();
- Support for integration with cloud storage solutions like AWS S3, Google Cloud Storage, and Azure Blob Storage;
- Support for mosaic (block) arrays;
- The ability to use various compression and encryption algorithms for data;
- Support for integrity checks using checksums;
- Multi-threaded operation with input/output parallelization;
- Support for versioning of stored data, including the ability to fetch the state at a specific moment in the past or atomic updates of large datasets.
- Ability to attach metadata;
- Support for data grouping;
- Integration modules for use as a low-level storage engine in Spark, Dask, MariaDB, GDAL, PDAL, Rasterio, gVCF, and PrestoDB;
- Wrapper libraries over C++ API for Python, R, Java, and Go.
Release 2.0 is notable for supporting the concept of 'DataFrame', which allows storing data as columns of values of arbitrary length, linked to specific attributes. The storage is also optimized for handling sparse arrays of heterogeneous sizes (cells can hold different types of data, and operations on merging columns of different types can be performed, such as those containing names, time, and price). Support for columns with string data has been added. Modules for integration with Google Cloud Storage and Azure Blob Storage have been introduced. The API for R has been redesigned.
Source: opennet.ru
