{"id":33630,"date":"2019-10-31T21:53:49","date_gmt":"2019-10-31T18:53:49","guid":{"rendered":"https:\/\/prohoster.info\/blog\/chto-osobennogo-v-cloudera-i-kak-ee-gotovit\/"},"modified":"2019-10-31T21:53:49","modified_gmt":"2019-10-31T18:53:49","slug":"chto-osobennogo-v-cloudera-i-kak-ee-gotovit","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/chto-osobennogo-v-cloudera-i-kak-ee-gotovit","title":{"rendered":"What is special about Cloudera and how to prepare it","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

The market for distributed computing and big data, according to reports statistics<\/a><\/noindex>, is growing by 18-19% annually. This makes the choice of software for these purposes a relevant issue. In this post, we will start with the reasons why distributed computing is necessary, delve into how to choose the software, discuss the use of Hadoop with Cloudera, and finally talk about hardware selection and how it impacts performance in various ways.<\/p>\n

\"What<\/a><\/noindex>
\nWhy is distributed computing needed in regular business? It's simple yet complex at the same time. Simple because, in most cases, we perform relatively straightforward calculations on a unit of information. Complex because there is a vast amount of such information. Consequently, we have to process terabytes of data across 1000 streams.<\/a><\/noindex>Thus, the use cases are quite universal: calculations can be applied wherever there's a need to consider a large number of metrics over an even larger dataset.<\/p>\n

A recent example: the Dodo Pizza chain determined<\/a><\/noindex> based on an analysis of the customer order database that when choosing a pizza with random toppings, users typically operate with only six basic sets of ingredients plus a couple of random ones. Accordingly, the pizzeria adjusted its purchasing strategy. Moreover, they were able to better recommend additional items to users during the ordering process, which improved profitability.<\/p>\n

Another example: an analysis<\/a><\/noindex> of product items allowed the H&M store to reduce the range of products in certain shops by 40%, while maintaining sales levels. This was achieved by eliminating poorly selling items, taking seasonality into account.<\/p>\n

The choice of tool<\/h2>\n

\nThe industry standard for such computing is Hadoop. Why? Because Hadoop is an excellent, well-documented framework (even Habr has numerous detailed articles on this topic) that comes with a whole set of utilities and libraries. You can input massive datasets, both structured and unstructured, and the system will distribute them among the computing resources itself. Moreover, these resources can be increased or decreased at any time \u2014 that's horizontal scalability in action. <\/p>\n

In 2017, the influential consulting firm Gartner has entered into<\/a><\/noindex>, predicted that Hadoop would soon become obsolete. The reason is quite simple: analysts believe that companies will start to migrate en masse to the cloud, where they can pay based on actual usage of computing resources. A second important factor that could supposedly 'bury' Hadoop is its speed. Options like Apache Spark or Google Cloud DataFlow operate faster than the MapReduce underlying Hadoop.<\/p>\n

Hadoop rests on several pillars, the most notable of which are MapReduce (a data distribution system for computations across servers) and the HDFS file system. The latter is specifically designed to store information distributed across the cluster's nodes: each fixed-size block can be placed on multiple nodes, and thanks to replication, the system's resilience to individual node failures is ensured. Instead of a traditional file table, a special server called NameNode is used.<\/p>\n

The illustration below shows the operation scheme of MapReduce. In the first stage, data is divided based on certain criteria, in the second, it is distributed across computing resources, and in the third, calculations take place. <\/p>\n

\"What
\nInitially, MapReduce was created by Google for its search needs. Then it became open source, and the Apache project took it over. Meanwhile, Google gradually migrated to other solutions. An interesting nuance: currently, Google has a project called Google Cloud Dataflow, positioned as the next step after Hadoop, serving as its fast replacement.<\/p>\n

Upon closer inspection, it is clear that Google Cloud Dataflow is based on a variant of Apache Beam, which includes a well-documented framework called Apache Spark. This allows for virtually identical execution speeds for solutions. Moreover, Apache Spark works exceptionally well on the HDFS file system, enabling its deployment on Hadoop servers.<\/p>\n

When we factor in the volume of documentation and ready-made solutions for Hadoop and Spark compared to Google Cloud Dataflow, the choice of tool becomes evident. Furthermore, engineers can decide for themselves which code\u2014Hadoop or Spark\u2014they will execute, depending on the task, their experience, and qualifications.<\/p>\n

Cloud or local server<\/h2>\n

\nThe trend towards a universal shift to the cloud has even created such an interesting term as Hadoop-as-a-service. In this scenario, server administration becomes very important. Because, unfortunately, despite its popularity, pure Hadoop is quite complex to configure, as much must be done manually. For example, individually configuring servers, monitoring their metrics, meticulously adjusting many parameters. Overall, it requires a certain level of expertise, and there is a significant chance of making mistakes or overlooking something.<\/p>\n

As a result, various distributions that come with convenient deployment and administration tools have gained great popularity. One of the most popular distributions that supports Spark and simplifies everything is Cloudera. It has both paid and free versions\u2014the latter provides access to all core functionalities without limitations on the number of nodes. <\/p>\n

\"What
\n
\nDuring the setup, Cloudera Manager will connect to your servers via SSH. An interesting point: during installation, it is better to specify that the installation be done using what are called parcels<\/i>: special packages, each containing all the necessary components configured to work together. Essentially, it is an improved version of a package manager.<\/p>\n

After installation, we receive a cluster management console, where you can view telemetry data for the clusters, installed services, plus you will be able to add or remove resources and edit the cluster configuration. <\/p>\n

\"What
\n
\nAs a result, you are faced with the shell of the rocket that will take you into the bright future of Big Data. But before we say \"let's go,\" let's take a look under the hood.<\/p>\n

Hardware Requirements<\/h2>\n

\nOn its website, Cloudera mentions various possible configurations. The general principles on which they are built are illustrated:<\/p>\n

\"What
\nThis optimistic picture can be overshadowed by MapReduce. Looking again at the diagram from the previous section, it becomes clear that in almost all cases, a MapReduce job may encounter a \"bottleneck\" when reading data from disk or over the network. This is also noted in Cloudera's blog. As a result, for any fast computations, including via Spark, which is often used for real-time calculations, input\/output speed is crucial. Therefore, when using Hadoop, it is very important that balanced and fast machines are part of the cluster, which, to put it mildly, is not always ensured in cloud infrastructure.<\/p>\n

Balance in load distribution is achieved by using OpenStack virtualization on servers with powerful multi-core CPUs. Data nodes are allocated their own processing resources and specific disks. In our solution Atos Codex Data Lake Engine<\/i> a broad virtualization is achieved, which benefits us both in terms of performance (minimizing the impact of the network infrastructure) and TCO (eliminating unnecessary physical servers).<\/p>\n

\"What
\nIn the case of using BullSequana S200 servers, we achieve a fairly uniform load, free of some bottlenecks. The minimum configuration includes 3 BullSequana S200 servers, each with two JBODs, plus optionally additional S200s connected, containing four data nodes each. Here's an example of the load in the TeraGen test:<\/p>\n

\"What
\n
\nTests with various data volumes and replication values show similar results in terms of load distribution among the cluster nodes. Below is a graph of disk access distribution based on performance tests.<\/p>\n

\"What
\n
\nCalculations were performed based on the minimum configuration of 3 BullSequana S200 servers. This includes 9 data nodes and 3 master nodes, as well as reserved virtual machines for deploying protection based on OpenStack Virtualization. TeraSort test result: block size of 512 MB with a replication factor of three, including encryption, is 23.1 minutes.<\/p>\n

How can the system be expanded? Various types of extensions are available for the Data Lake Engine:<\/p>\n