The code for the BlazingSQL SQL engine, which uses GPU for acceleration, has been opened.

Announced about the release of the SQL engine source code BlazingSQL, which uses GPU to accelerate data processing. BlazingSQL is not a full-fledged DBMS, but positions itself as an engine for analyzing and processing large datasets, comparable in its tasks to Apache Spark. The code is written in Python and is open under the Apache 2.0 license.

BlazingSQL is suitable for executing single analytical queries over large datasets (tens of gigabytes), stored in tabular formats (e.g., logs, NetFlow statistics, etc.). BlazingSQL can execute queries from raw files in formats such as CSV and Apache Parquet, located in network and cloud file systems like HDFS and AWS S3, directly transferring the results to GPU memory. Thanks to parallel processing on the GPU and the use of faster video memory, query execution in BlazingSQL is done up to 20 times faster than in Apache Spark.

For working with the GPU, a library set developed with the participation of NVIDIA is used open-source called RAPIDS, which allows creating applications for data processing and analytics executed entirely on the GPU side (a Python interface is provided for using low-level CUDA primitives and parallelizing computations).

BlazingSQL provides the ability to use SQL instead of the data processing API cuUDF (based on Apache Arrow), used in RAPIDS. BlazingSQL is an additional layer working over cuDF and uses the cuIO library to read data from disk. SQL queries are translated into cuUDF function calls, allowing data to be loaded into the GPU and operations like merging, aggregating, and filtering to be performed. Support for creating distributed configurations covering thousands of GPUs is included.

BlazingSQL significantly simplifies working with data — instead of hundreds of cuDF function calls, one SQL query suffices. The use of SQL allows for integration of RAPIDS with existing analytics systems without the need to write specific handlers and without loading data into an additional DBMS, while
maintaining full compatibility with all components of RAPIDS, translating existing functionality into SQL and providing performance on the level of cuDF. This includes support for integration with libraries XGBoost and cuML for solving analytics and machine learning tasks.

Source: opennet.ru