{"id":92508,"date":"2020-08-28T07:42:10","date_gmt":"2020-08-28T05:42:10","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/kak-my-organizovali-vysokoeffektivnoe-i-nedorogoe-datalake-i-pochemu-imenno-tak"},"modified":"2020-08-28T07:42:10","modified_gmt":"2020-08-28T05:42:10","slug":"kak-my-organizovali-vysokoeffektivnoe-i-nedorogoe-datalake-i-pochemu-imenno-tak","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/kak-my-organizovali-vysokoeffektivnoe-i-nedorogoe-datalake-i-pochemu-imenno-tak","title":{"rendered":"How we organized a highly effective and cost-efficient DataLake and why we did it this way","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p>We live in an amazing time when it is possible to quickly and easily connect several ready-made open tools, configure them with 'disconnected consciousness' based on advice from stackoverflow, without getting bogged down in 'wordiness', and launch them into commercial operation. And when it's time to update\/expand or if someone accidentally restarts a couple of machines, one realizes that some obsessive nightmare has started in reality, everything has suddenly become overwhelmingly complicated, there's no way back, the future is hazy, and instead of programming, it's safer to raise bees and make cheese.<\/p>\n<p>It's no wonder that more experienced colleagues, with their heads gray from countless bugs, watching the improbably rapid deployment of 'containers' in 'cubes' across dozens of servers with 'trendy languages' featuring built-in support for asynchronous non-blocking I\/O \u2014 smile modestly. And they quietly continue to reread 'man ps', strain their eyes digging into the sources of 'nginx', and write-write-write unit tests. Colleagues know that the most interesting part lies ahead when 'all this' one day becomes a stake on New Year\u2019s Eve. And only a deep understanding of the nature of Unix, memorized TCP\/IP state tables, and basic sorting-searching algorithms will help them to revive the system at the stroke of midnight.<br \/>\n<noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><br \/>\nAh yes, I got a bit sidetracked, but I hope I conveyed the sense of anticipation.<br \/>\nToday, I want to share our experience in deploying a convenient and affordable stack for DataLake that addresses most analytical tasks in the company across completely different structural divisions.<\/p>\n<p>Some time ago, we came to understand that companies increasingly need insights from both product and technical analytics (not to mention the cherries on top in the form of machine learning), and in order to grasp trends and risks, it's necessary to collect and analyze growing amounts of metrics.<\/p>\n<h3>Basic technical analytics in 'Bitrix24'<\/h3>\n<p>\nSeveral years ago, alongside the launch of the \"Bitrix24\" service, we actively invested time and resources in creating a simple and reliable analytics platform that would help quickly identify infrastructure issues and plan the next steps. Naturally, it was preferable to use ready-made tools that were as simple and understandable as possible. As a result, we chose Nagios for monitoring and Munin for analytics and visualization. Now we have thousands of checks in Nagios, hundreds of graphs in Munin, and colleagues use them successfully on a daily basis. The metrics are clear, the graphs are understandable, the system has been reliable for several years and new tests and graphs are regularly added: when we introduce a new service into operation, we add several tests and graphs. Here\u2019s to a good start.<\/p>\n<h3>Keeping a Finger on the Pulse \u2014 Advanced Technical Analytics<\/h3>\n<p>\nThe desire to receive information about issues \"as quickly as possible\" led us to actively experiment with simple and understandable tools \u2014 Pinba and Xhprof.<\/p>\n<p>Pinba sent us statistics on the performance of parts of web pages on PHP in UDP packets, allowing us to see in real-time a short list of problems in the MySQL storage (the Pinba has its own MySQL engine for fast event analytics) and respond to them. And Xhprof automatically allowed us to gather execution graphs of the slowest PHP pages for clients and analyze what could have led to this \u2014 calmly, with a cup of tea or something stronger.<\/p>\n<p>Some time ago, the toolkit was supplemented with another quite simple and understandable engine based on reverse indexing algorithms, excellently implemented in the legendary Lucene library \u2014 Elastic\/Kibana. The straightforward idea of multi-threaded document recording into Lucene's inverted index based on events in logs and quick search through them using faceted division turned out to be quite useful.<\/p>\n<p>Despite the rather technical appearance of the visualizations in Kibana, with low-level concepts like \"bucket\" and a newly reinvented language not yet forgotten from relational algebra \u2014 the tool has helped us well in the following tasks:<\/p>\n<ul>\n<li>How many PHP errors did the Bitrix24 client have on portal p1 in the last hour, and what were they? Understand, forgive, and quickly rectify.<\/li>\n<li>How many video calls were made on the portals in Germany over the past 24 hours, what was the quality, and were there any issues with the channel\/network?<\/li>\n<li>How well is the system functionality (our extension in C for PHP) working, compiled from source code in the latest service update and rolled out to clients? Are there any segfaults?<\/li>\n<li>Are client data stored in PHP memory? Are there any errors related to exceeding the allocated process memory: 'out of memory'? Identify and neutralize them.<\/li>\n<\/ul>\n<p>\nHere's a concrete example. Despite thorough multi-level testing, a client encountered an annoying and unexpected error due to a very non-standard case with corrupted input data; the siren went off, and the process of rapid correction began:<\/p>\n<p><img decoding=\"async\" alt=\"How we organized a highly effective and cost-efficient DataLake and why we did it this way\" src=\"\/wp-content\/uploads\/2020\/08\/8a802dba41b5d1a85c0dc41dfbf8b84e.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nAdditionally, Kibana allows organizing alerts based on specific events, and in a short time, dozens of employees from different departments \u2014 from technical support and development to QA \u2014 began using the tool.<\/p>\n<p>The activity of any department within the company has become easy to track and measure \u2014 instead of manually analyzing logs on the servers, it's enough to set up log parsing and send them to the elastic cluster once, to enjoy, for example, watching the dashboard in Kibana showing the number of sold two-headed kittens printed on a 3D printer over the past lunar month.<\/p>\n<h3>Basic Business Analytics<\/h3>\n<p>\nEveryone knows that often business analytics in companies starts with the extremely active use of, yes, Excel. But the main thing is that it doesn\u2019t end there. Cloud-based Google Analytics also adds fuel to the fire \u2014 you quickly get used to good things.<\/p>\n<p>In our harmoniously growing company, 'prophets' of more intensive work with larger data sets began to appear here and there. There has been a regular need for deeper and more comprehensive reports, and through the efforts of people from different departments, a simple and practical solution was organized some time ago \u2014 the integration of ClickHouse and PowerBI.<\/p>\n<p>For quite a long time, this flexible solution helped excellently, but gradually it became clear that ClickHouse is not elastic and can't be abused like that.<\/p>\n<p>It is important to understand that ClickHouse, like Druid, Vertica, and Amazon RedShift (which is based on Postgres), is an analytical engine optimized for convenient analytics (sums, aggregations, min-max on columns, and a bit of joining), as they are organized for efficient storage of columns in relational tables, unlike known MySQL and other (row-oriented) databases.<\/p>\n<p>Essentially, ClickHouse is just a more spacious 'database', with not-so-convenient point inserts (that's by design, all is well), but pleasant analytics and a set of interesting powerful data manipulation functions. Yes, you can even create a cluster \u2014 but you understand that hammering nails with a microscope is not quite right, and we started looking for other solutions.<\/p>\n<h3>Demand for Python and analysts<\/h3>\n<p>\nIn our company, there are many developers who write code almost every day for 10-20 years in PHP, JavaScript, C#, C\/C++, Java, Go, Rust, Python, Bash. There are also many experienced system administrators who have survived more than one completely unbelievable catastrophe that defies statistical laws (for example, when most disks in a RAID-10 are destroyed by a strong lightning strike). In such conditions, it has long been unclear what 'a Python analyst' is. Python is like PHP, just a slightly longer name and slightly fewer traces of substances altering consciousness in the interpreter's source code. However, as more and more analytical reports were created, experienced developers increasingly recognized the importance of narrow specialization in tools like numpy, pandas, matplotlib, seaborn.<br \/>\nThe decisive role was probably played by the sudden fainting of employees at the combination of the words 'logistic regression' and the demonstration of effective report building on large datasets using, yes, pyspark.<\/p>\n<p>Apache Spark and its functional paradigm, which fits well with relational algebra, made such an impression on developers accustomed to MySQL that the need to reinforce the ranks with experienced analysts became as clear as day.<\/p>\n<h3>Further attempts at Apache Spark\/Hadoop to take off and what went not quite according to the script<\/h3>\n<p>\nHowever, it soon became clear that there was something fundamentally wrong with Spark, or perhaps we just needed to wash our hands better. While the Hadoop\/MapReduce\/Lucene stack was crafted by experienced programmers\u2014evident if you closely examine the Java sources or Doug Cutting's ideas in Lucene\u2014Spark, unexpectedly, is written in a rather controversial and currently stagnant language called Scala from a practical standpoint. The frequent crashes of computations on Spark clusters due to illogical and opaque memory allocation for reduce operations (many keys arrive at once) created an aura around it suggesting there is much room for improvement. Additionally, a large number of strange open ports, temporary files growing in inexplicable places, and a jar-dependency bloat exacerbated the situation\u2014evoking a familiar and deep-seated feeling in system administrators: intense loathing (maybe we should have been washing our hands with soap).<\/p>\n<p>As a result, we \"survived\" several internal analytical projects that actively utilized Apache Spark (including Spark Streaming, Spark SQL) and the Hadoop ecosystem (and more). Despite the fact that over time we learned to prepare and monitor \"it\" quite well and \"it\" nearly stopped crashing unexpectedly due to changes in data characteristics and RDD's uneven hashing, the desire to use something ready-made, updatable, and managed somewhere in the cloud grew stronger and stronger. It was during this time that we tried using Amazon Web Services' ready-made cloud build\u2014 <noindex><a rel=\"nofollow\" href=\"https:\/\/aws.amazon.com\/ru\/emr\/\">EMR<\/a><\/noindex> and subsequently aimed to tackle tasks on it. EMR is a pre-packaged Apache Spark from Amazon with additional software from the ecosystem, somewhat like Cloudera\/Hortonworks builds.<\/p>\n<h3>A \"rubber\" data warehouse for analytics\u2014this is an urgent need.<\/h3>\n<p>\nThe experience of \"preparing\" Hadoop\/Spark with burns on various parts of the body did not go without consequence. The need for a single, inexpensive, and reliable data storage solution that is resilient to hardware failures and capable of storing files in different formats from various systems while allowing for efficient and timely queries for reports became increasingly clear.<\/p>\n<p>We also wanted the software update for this platform not to turn into a New Year's nightmare of reading 20-page Java traces and analyzing kilometers of detailed logs from the cluster using Spark History Server and a magnifying glass with a spotlight. We wanted a simple and transparent tool that doesn't require regular dives under the hood, especially when a developer's standard MapReduce query fails due to the reduce worker's memory running out because of a poorly chosen data partitioning algorithm.<\/p>\n<h3>Is Amazon S3 a candidate for DataLake?<\/h3>\n<p>\nExperience with Hadoop\/MapReduce taught us that we need a scalable, reliable file system and scalable workers that come closer to the data to avoid moving data over the network. Workers should be able to read data in different formats but, preferably, without reading unnecessary information and allowing data to be stored in convenient formats for the workers in advance.<\/p>\n<p><b>Once again \u2014 the main idea.<\/b> There is no desire to \"dump\" large data into a single clustered analytical engine that will inevitably choke sooner or later and will need to be sharded awkwardly. We want to store files, just files, in a clear format and run efficient analytical queries on them with various but understandable tools. And there will be more and more files in different formats. It's better to shard the source data, not the engine. We decided we need a scalable and universal DataLake...<\/p>\n<p>What if we store files in the familiar and well-known scalable cloud storage Amazon S3, avoiding the hassle of preparing our own Hadoop meals?<\/p>\n<p>Of course, personal data is \"not allowed\", but what about moving other data there and \"processing it efficiently\"?<\/p>\n<h3>The clustered big data analytics ecosystem of Amazon Web Services \u2014 in very simple terms.<\/h3>\n<p>\nBased on our experience with AWS, Apache Hadoop\/MapReduce has long been actively used there under various guises, for instance, in the DataPipeline service (I envy my colleagues; they really know how to set it up right). Here we configured backups from various services from DynamoDB tables:<br \/>\n<img decoding=\"async\" alt=\"How we organized a highly effective and cost-efficient DataLake and why we did it this way\" src=\"\/wp-content\/uploads\/2020\/08\/058dc54ed032a7bf3e9e129646202440.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nAnd they have been running regularly on built-in Hadoop\/MapReduce clusters like clockwork for several years now. \"Set it up and forget it\":<\/p>\n<p><img decoding=\"async\" alt=\"How we organized a highly effective and cost-efficient DataLake and why we did it this way\" src=\"\/wp-content\/uploads\/2020\/08\/a6569da8cafdb96c63250bb32bf51704.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nAdditionally, one can effectively engage in data science by launching Jupiter notebooks in the cloud for analysts and using AWS SageMaker for training and deploying AI models. Here's how it looks with us:<\/p>\n<p><img decoding=\"async\" alt=\"How we organized a highly effective and cost-efficient DataLake and why we did it this way\" src=\"\/wp-content\/uploads\/2020\/08\/c825d979c9278a8edf8e1e747ef6def8.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nAnd yes, you can set up a notebook in the cloud or for an analyst and connect it to a Hadoop\/Spark cluster, perform calculations, and then 'nail it all down':<\/p>\n<p><img decoding=\"async\" alt=\"How we organized a highly effective and cost-efficient DataLake and why we did it this way\" src=\"\/wp-content\/uploads\/2020\/08\/73cea18c54d2a9ce8d0441463991808b.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nIt's indeed convenient for individual analytical projects, and for some, we have successfully used the EMR service for large-scale calculations and analytics. But what about a systematic solution for DataLake? Would it work? At that moment, we were on the verge of hope and despair and continued our search.<\/p>\n<h3>AWS Glue is a neatly packaged Apache Spark 'on steroids.'<\/h3>\n<p>\nIt turned out that AWS has its own version of the Hive\/Pig\/Spark stack. The role of Hive, i.e., the catalog of files and their types in DataLake, is performed by the 'Data Catalog' service, which does not hide its compatibility with the Apache Hive format. In this service, you need to add information about where your files are located and in what format they are. The data can be stored not only in S3 but also in a database, but that's not the topic of this post. Here's how our DataLake data catalog is organized:<\/p>\n<p><img decoding=\"async\" alt=\"How we organized a highly effective and cost-efficient DataLake and why we did it this way\" src=\"\/wp-content\/uploads\/2020\/08\/adb45d09698fdacbf41c86bbadde8bb2.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nFiles are registered, great. If the files are updated, we either run crawlers manually or on a schedule, which will refresh the information from the lake and save it. Next, the data from the lake can be processed and the results exported somewhere. In the simplest case, we export them to S3 as well. Data processing can be done anywhere, but it's suggested to set up the processing on an Apache Spark cluster using advanced features via the AWS Glue API. Essentially, you can take your trusty old Python code using the pyspark library and configure it to run on N nodes of a cluster with some power, along with monitoring, without digging into the depths of Hadoop and dragging around Docker containers and resolving dependency conflicts.<\/p>\n<p><b>Once again \u2014 a simple idea.<\/b> There's no need to configure Apache Spark; you just need to write Python code for pyspark, test it locally on your desktop, and then run it on a large cloud cluster, specifying where the source data is and where to place the results. Sometimes this is necessary and useful, and here's how it's set up with us:<\/p>\n<p><img decoding=\"async\" alt=\"How we organized a highly effective and cost-efficient DataLake and why we did it this way\" src=\"\/wp-content\/uploads\/2020\/08\/dc03181573bb3f5cfcc3a8760bc7e07b.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nThus, if you need to perform calculations on a Spark cluster with data in S3, you write Python\/pyspark code, test it, and off you go to the cloud.<\/p>\n<p>What about orchestration? What if a task fails and is lost? Yes, a nice pipeline in the style of Apache Pig is suggested, and we've tried it, but we've decided to continue using our deeply customized orchestration in PHP and JavaScript (I understand there\u2019s a cognitive dissonance, but it has been working flawlessly for years).<\/p>\n<p><img decoding=\"async\" alt=\"How we organized a highly effective and cost-efficient DataLake and why we did it this way\" src=\"\/wp-content\/uploads\/2020\/08\/e00ed2047c5c6492e36fccc82e7278a3.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<\/p>\n<h3>The format of files stored in the lake is key to performance.<\/h3>\n<p>\nIt is very, very important to understand two more key points. To ensure that data queries in the lake are executed as quickly as possible and performance does not degrade with the addition of new information, you need to:<\/p>\n<ul>\n<li>Store file columns separately (so you don't have to read all rows to understand what is in the columns). For this, we adopted the parquet format with compression.<\/li>\n<li>It is crucial to shard files into folders such as: language, year, month, day, week. Engines that understand this type of sharding will only look into the relevant folders, without sifting through all the data indiscriminately.<\/li>\n<\/ul>\n<p>\nEssentially, in this way, you are providing the original data in the most efficient form for the analytical engines applied on top, which can selectively access sharded folders and read only the necessary columns from the files. There\u2019s no need to 'upload' data anywhere (the storage will just burst) \u2014 simply place it smartly in the filesystem in the correct format. Of course, it should be clear that storing a huge CSV file in a DataLake, which needs to be read line by line to extract columns \u2014 is not very sensible. Think about the two aforementioned points once more, if you're still unclear on why this matters.<\/p>\n<h3>AWS Athena \u2014 the 'magic' from the box.<\/h3>\n<p>\nAnd here, while creating the lake, we stumbled upon Amazon Athena. It turned out that neatly arranging our large log files into sharded folders in the correct (parquet) columnar format allows for rapid, highly informative queries and report generation WITHOUT an Apache Spark\/Glue cluster.<\/p>\n<p>The Athena engine, working on data in S3, is based on the legendary. <noindex><a rel=\"nofollow\" href=\"https:\/\/aws.amazon.com\/ru\/big-data\/what-is-presto\/\">Presto<\/a><\/noindex> \u2014 a representative of the MPP (massive parallel processing) family of data processing approaches, which retrieves data from where it resides, from s3 and Hadoop to Cassandra and regular text files. You just need to ask Athena to execute an SQL query, and everything \"works quickly and on its own.\" It's important to note that Athena is \"smart,\" only accessing the necessary sharded folders and reading only the columns needed for the query.<\/p>\n<p>The pricing for queries to Athena is also interesting. We pay for <noindex><a rel=\"nofollow\" href=\"https:\/\/aws.amazon.com\/ru\/athena\/pricing\/\">the volume of scanned data<\/a><\/noindex>. That is, not by the number of machines in the cluster per minute, but\u2026 for the actual data scanned that is necessary to execute the query across 100-500 machines.<\/p>\n<p>By querying only the required columns from the correctly sharded folders, it turned out that the Athena service costs us tens of dollars a month. Well, it's great, almost free compared to analytics on clusters!<\/p>\n<p>Here, by the way, is how we shard our data in s3:<\/p>\n<p><img decoding=\"async\" alt=\"How we organized a highly effective and cost-efficient DataLake and why we did it this way\" src=\"\/wp-content\/uploads\/2020\/08\/00bd9ae48c1cd13f3c4f7d32692c9209.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nAs a result, in a short time, completely different departments in the company, from information security to analytics, began actively querying Athena and quickly, in seconds, receiving useful answers from \"big\" data over quite lengthy periods: months, half-years, etc.<\/p>\n<p>But we went further and started fetching answers from the cloud <noindex><a rel=\"nofollow\" href=\"https:\/\/docs.aws.amazon.com\/athena\/latest\/ug\/connect-with-odbc.html\">through the ODBC driver<\/a><\/noindex>: an analyst writes an SQL query in the familiar console, which \"cheaply\" sifts through the data in s3 across 100-500 machines and usually returns an answer in a matter of seconds. Convenient. And fast. It's hard to believe.<\/p>\n<p>As a result, having decided to store data in s3, in an efficient columnar format and with reasonable data sharding across folders\u2026 we created a DataLake and a fast and affordable analytical engine \u2014 for free. It became very popular in the company, as it understands SQL and works orders of magnitude faster than via launches\/stops\/settings of clusters. \"And if the result is the same, why pay more?\"<\/p>\n<p>A query to Athena looks something like this. Of course, if desired, a sufficiently <noindex><a rel=\"nofollow\" href=\"https:\/\/prestodb.io\/docs\/0.172\/index.html\">complex and multi-page SQL query<\/a><\/noindex>, can be formed, but we will limit ourselves to a simple grouping. Let's see what response codes the client had a few weeks ago in the web server logs and ensure that there are no errors:<\/p>\n<p><img decoding=\"async\" alt=\"How we organized a highly effective and cost-efficient DataLake and why we did it this way\" src=\"\/wp-content\/uploads\/2020\/08\/30028991467b9e52f597faa617d374b5.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<\/p>\n<h3>Conclusions<\/h3>\n<p>\nHaving gone through a journey that wasn't long but was painful, constantly assessing risks, complexity, and support costs, we found a solution for DataLake and analytics that continues to delight us with its speed and cost of ownership.<\/p>\n<p>It turned out that building an efficient, fast, and cost-effective DataLake for the needs of completely different company departments is well within reach, even for experienced developers who have never worked as architects and don\u2019t know how to draw boxes on boxes with arrows, nor do they know 50 terms from the Hadoop ecosystem.<\/p>\n<p>At the beginning, my head was spinning from the overwhelming variety of both open and closed-source software and the weight of responsibility I felt towards future generations. Just start building your DataLake with simple tools: nagios\/munin -&gt; elastic\/kibana -&gt; Hadoop\/Spark\/s3 ..., gathering feedback and deeply understanding the physics of the processes involved. Anything complex and murky should be left to your rivals and competitors.<\/p>\n<p>If you don\u2019t want to go to the cloud and love maintaining, updating, and patching open-source projects, you can build a setup similar to ours locally on inexpensive office machines with Hadoop and Presto on top. The main thing is not to stop and keep moving forward, calculating, searching for simple and clear solutions, and everything will definitely work out! Good luck to everyone and see you again!<br \/>\n<br \/>Source: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/bitrix\/blog\/516374\/\">habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u041c\u044b \u0436\u0438\u0432\u0435\u043c \u0432 \u0443\u0434\u0438\u0432\u0438\u0442\u0435\u043b\u044c\u043d\u043e\u0435 \u0432\u0440\u0435\u043c\u044f, \u043a\u043e\u0433\u0434\u0430 \u043c\u043e\u0436\u043d\u043e \u0431\u044b\u0441\u0442\u0440\u043e \u0438 \u043f\u0440\u043e\u0441\u0442\u043e \u0441\u043e\u0441\u0442\u044b\u043a\u043e\u0432\u0430\u0442\u044c \u043d\u0435\u0441\u043a\u043e\u043b\u044c\u043a\u043e \u0433\u043e\u0442\u043e\u0432\u044b\u0445 \u043e\u0442\u043a\u0440\u044b\u0442\u044b\u0445 \u0438\u043d\u0441\u0442\u0440\u0443\u043c\u0435\u043d\u0442\u043e\u0432, \u043d\u0430\u0441\u0442\u0440\u043e\u0438\u0442\u044c \u0438\u0445 \u0441 \u00ab\u043e\u0442\u043a\u043b\u044e\u0447\u0435\u043d\u043d\u044b\u043c \u0441\u043e\u0437\u043d\u0430\u043d\u0438\u0435\u043c\u00bb \u043f\u043e \u0441\u043e\u0432\u0435\u0442\u0430\u043c stackoverflow, \u043d\u0435 \u0432\u043d\u0438\u043a\u0430\u044f \u0432 \u00ab\u043c\u043d\u043e\u0433\u043e\u0431\u0443\u043a\u0432\u00bb, \u0437\u0430\u043f\u0443\u0441\u0442\u0438\u0442\u044c \u0432 \u043a\u043e\u043c\u043c\u0435\u0440\u0447\u0435\u0441\u043a\u0443\u044e \u044d\u043a\u0441\u043f\u043b\u0443\u0430\u0442\u0430\u0446\u0438\u044e. \u0410 \u043a\u043e\u0433\u0434\u0430 \u043d\u0443\u0436\u043d\u043e \u0431\u0443\u0434\u0435\u0442 \u043e\u0431\u043d\u043e\u0432\u043b\u044f\u0442\u044c\u0441\u044f\/\u0440\u0430\u0441\u0448\u0438\u0440\u044f\u0442\u044c\u0441\u044f \u0438\u043b\u0438 \u043a\u0442\u043e-\u0442\u043e \u0441\u043b\u0443\u0447\u0430\u0439\u043d\u043e \u043f\u0435\u0440\u0435\u0437\u0430\u0433\u0440\u0443\u0437\u0438\u0442 \u043f\u0430\u0440\u0443 \u043c\u0430\u0448\u0438\u043d \u2014 \u043e\u0441\u043e\u0437\u043d\u0430\u0442\u044c, \u0447\u0442\u043e \u043d\u0430\u0447\u0430\u043b\u0441\u044f \u043a\u0430\u043a\u043e\u0439-\u0442\u043e \u043d\u0430\u0432\u044f\u0437\u0447\u0438\u0432\u044b\u0439 \u0434\u0443\u0440\u043d\u043e\u0439 \u0441\u043e\u043d \u043d\u0430\u044f\u0432\u0443, \u0432\u0441\u0435 \u0440\u0435\u0437\u043a\u043e \u0443\u0441\u043b\u043e\u0436\u043d\u0438\u043b\u043e\u0441\u044c \u0434\u043e [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":92509,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-92508","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.2 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u041c\u044b \u0436\u0438\u0432\u0435\u043c \u0432 \u0443\u0434\u0438\u0432\u0438\u0442\u0435\u043b\u044c\u043d\u043e\u0435 \u0432\u0440\u0435\u043c\u044f, \u043a\u043e\u0433\u0434\u0430 \u043c\u043e\u0436\u043d\u043e \u0431\u044b\u0441\u0442\u0440\u043e \u0438 \u043f\u0440\u043e\u0441\u0442\u043e \u0441\u043e\u0441\u0442\u044b\u043a\u043e\u0432\u0430\u0442\u044c \u043d\u0435\u0441\u043a\u043e\u043b\u044c\u043a\u043e \u0433\u043e\u0442\u043e\u0432\u044b\u0445 \u043e\u0442\u043a\u0440\u044b\u0442\u044b\u0445 \u0438\u043d\u0441\u0442\u0440\u0443\u043c\u0435\u043d\u0442\u043e\u0432, \u043d\u0430\u0441\u0442\u0440\u043e\u0438\u0442\u044c \u0438\u0445 \u0441 \u00ab\u043e\u0442\u043a\u043b\u044e\u0447\u0435\u043d\u043d\u044b\u043c \u0441\u043e\u0437\u043d\u0430\u043d\u0438\u0435\u043c\u00bb \u043f\u043e \u0441\u043e\u0432\u0435\u0442\u0430\u043c stackoverflow, \u043d\u0435 \u0432\u043d\u0438\u043a\u0430\u044f \u0432 \u00ab\u043c\u043d\u043e\u0433\u043e\u0431\u0443\u043a\u0432\u00bb, \u0437\u0430\u043f\u0443\u0441\u0442\u0438\u0442\u044c \u0432.\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/kak-my-organizovali-vysokoeffektivnoe-i-nedorogoe-datalake-i-pochemu-imenno-tak\" \/>\n\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.2\" \/>\n\t\t<meta property=\"og:locale\" content=\"en_US\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47\u041a\u0430\u043a \u043c\u044b \u043e\u0440\u0433\u0430\u043d\u0438\u0437\u043e\u0432\u0430\u043b\u0438 \u0432\u044b\u0441\u043e\u043a\u043e\u044d\u0444\u0444\u0435\u043a\u0442\u0438\u0432\u043d\u043e\u0435 \u0438 \u043d\u0435\u0434\u043e\u0440\u043e\u0433\u043e\u0435 DataLake \u0438 \u043f\u043e\u0447\u0435\u043c\u0443 \u0438\u043c\u0435\u043d\u043d\u043e \u0442\u0430\u043a | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u041c\u044b \u0436\u0438\u0432\u0435\u043c \u0432 \u0443\u0434\u0438\u0432\u0438\u0442\u0435\u043b\u044c\u043d\u043e\u0435 \u0432\u0440\u0435\u043c\u044f, \u043a\u043e\u0433\u0434\u0430 \u043c\u043e\u0436\u043d\u043e \u0431\u044b\u0441\u0442\u0440\u043e \u0438 \u043f\u0440\u043e\u0441\u0442\u043e \u0441\u043e\u0441\u0442\u044b\u043a\u043e\u0432\u0430\u0442\u044c \u043d\u0435\u0441\u043a\u043e\u043b\u044c\u043a\u043e \u0433\u043e\u0442\u043e\u0432\u044b\u0445 \u043e\u0442\u043a\u0440\u044b\u0442\u044b\u0445 \u0438\u043d\u0441\u0442\u0440\u0443\u043c\u0435\u043d\u0442\u043e\u0432, \u043d\u0430\u0441\u0442\u0440\u043e\u0438\u0442\u044c \u0438\u0445 \u0441 \u00ab\u043e\u0442\u043a\u043b\u044e\u0447\u0435\u043d\u043d\u044b\u043c \u0441\u043e\u0437\u043d\u0430\u043d\u0438\u0435\u043c\u00bb \u043f\u043e \u0441\u043e\u0432\u0435\u0442\u0430\u043c stackoverflow, \u043d\u0435 \u0432\u043d\u0438\u043a\u0430\u044f \u0432 \u00ab\u043c\u043d\u043e\u0433\u043e\u0431\u0443\u043a\u0432\u00bb, \u0437\u0430\u043f\u0443\u0441\u0442\u0438\u0442\u044c \u0432.\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/kak-my-organizovali-vysokoeffektivnoe-i-nedorogoe-datalake-i-pochemu-imenno-tak\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2020-08-28T05:42:10+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-08-28T05:42:10+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47How we organized a highly efficient and inexpensive DataLake and why exactly this way | ProHoster","description":"We live in an amazing time when you can quickly and easily connect several ready-made open-source tools, configure them with 'disconnected consciousness' based on stackoverflow advice, without getting bogged down in 'many letters', and launch them.","canonical_url":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/kak-my-organizovali-vysokoeffektivnoe-i-nedorogoe-datalake-i-pochemu-imenno-tak","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"en_US","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u041a\u0430\u043a \u043c\u044b \u043e\u0440\u0433\u0430\u043d\u0438\u0437\u043e\u0432\u0430\u043b\u0438 \u0432\u044b\u0441\u043e\u043a\u043e\u044d\u0444\u0444\u0435\u043a\u0442\u0438\u0432\u043d\u043e\u0435 \u0438 \u043d\u0435\u0434\u043e\u0440\u043e\u0433\u043e\u0435 DataLake \u0438 \u043f\u043e\u0447\u0435\u043c\u0443 \u0438\u043c\u0435\u043d\u043d\u043e \u0442\u0430\u043a | ProHoster","og:description":"\u041c\u044b \u0436\u0438\u0432\u0435\u043c \u0432 \u0443\u0434\u0438\u0432\u0438\u0442\u0435\u043b\u044c\u043d\u043e\u0435 \u0432\u0440\u0435\u043c\u044f, \u043a\u043e\u0433\u0434\u0430 \u043c\u043e\u0436\u043d\u043e \u0431\u044b\u0441\u0442\u0440\u043e \u0438 \u043f\u0440\u043e\u0441\u0442\u043e \u0441\u043e\u0441\u0442\u044b\u043a\u043e\u0432\u0430\u0442\u044c \u043d\u0435\u0441\u043a\u043e\u043b\u044c\u043a\u043e \u0433\u043e\u0442\u043e\u0432\u044b\u0445 \u043e\u0442\u043a\u0440\u044b\u0442\u044b\u0445 \u0438\u043d\u0441\u0442\u0440\u0443\u043c\u0435\u043d\u0442\u043e\u0432, \u043d\u0430\u0441\u0442\u0440\u043e\u0438\u0442\u044c \u0438\u0445 \u0441 \u00ab\u043e\u0442\u043a\u043b\u044e\u0447\u0435\u043d\u043d\u044b\u043c \u0441\u043e\u0437\u043d\u0430\u043d\u0438\u0435\u043c\u00bb \u043f\u043e \u0441\u043e\u0432\u0435\u0442\u0430\u043c stackoverflow, \u043d\u0435 \u0432\u043d\u0438\u043a\u0430\u044f \u0432 \u00ab\u043c\u043d\u043e\u0433\u043e\u0431\u0443\u043a\u0432\u00bb, \u0437\u0430\u043f\u0443\u0441\u0442\u0438\u0442\u044c \u0432.","og:url":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/kak-my-organizovali-vysokoeffektivnoe-i-nedorogoe-datalake-i-pochemu-imenno-tak","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-08-28T05:42:10+00:00","article:modified_time":"2020-08-28T05:42:10+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"92508","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 12:07:39","updated":"2022-10-01 09:50:53","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/92508","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/comments?post=92508"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/92508\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/media\/92509"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/media?parent=92508"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/categories?post=92508"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/tags?post=92508"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}