After one and a half years of development, the Apache Software Foundation release , a free platform for organizing distributed processing of large volumes of data using the paradigm, where a task is divided into many smaller isolated fragments, each of which can be run on a separate node in the cluster. A Hadoop-based storage can cover thousands of nodes and contain exabytes of data.
Hadoop includes the implementation of the Hadoop Distributed Filesystem (HDFS), which automatically provides data redundancy and is optimized for running MapReduce applications. To simplify data access in Hadoop storage, the HBase database and the SQL-like language Pig have been developed, which acts as a kind of SQL for MapReduce, with queries that can be parallelized and processed across multiple Hadoop platforms. The project is considered fully stable and ready for industrial operation. Hadoop is actively used in large industrial projects, providing capabilities comparable to the Google Bigtable/GFS/MapReduce platform, while Google has officially Hadoop and other Apache projects the right to use technologies that are covered by patents related to the MapReduce method.
Hadoop ranks first among Apache repositories in the number of changes made and fifth in codebase size (about 4 million lines of code). Notable implementations of Hadoop include Netflix (storing over 500 billion events per day), Twitter (a real-time cluster of 10,000 nodes holds over a zettabyte of data and processes over 5 billion sessions per day), and Facebook (a cluster of 4,000 nodes stores over 300 petabytes and grows by 4 petabytes daily).
Key in Apache Hadoop 3.3:
- Support for ARM architecture-based platforms has been added.
- The implementation of the (Protocol buffers), used for serializing structured data, has been updated to release 3.7.1 due to the end of life of the protobuf-2.5.0 branch.
- S3A connector capabilities have been expanded: authentication support using tokens has been added (), improved support for caching responses with a 404 code, increased performance of S3guard, and enhanced reliability.
- The ABFS file system has resolved issues with automatic tuning.
- Built-in support for the Tencent Cloud COS file system has been added for accessing the COS object storage.
- Full support for Java 11 has been added.
- The HDFS RBF (Router-based Federation) implementation has been stabilized. Security management tools have been added to the HDFS Router.
- A DNS Resolution service has been added for clients to identify servers through DNS by node names, allowing for avoiding listing all hosts in the settings.
- Support for launch scheduling has been added. through a centralized resource manager (ResourceManager), including the ability to allocate containers based on each node's load.
- An application catalog for YARN (Yet Another Resource Negotiator) has been added with search capabilities.
Source: opennet.ru
