Do we need a data lake? And what about data storage?

This article is a translation of my article on Medium — Getting Started with Data Lake, which turned out to be quite popular, probably due to its simplicity. So I decided to write it in Russian and add a bit to help an ordinary person, who is not a data specialist, understand what data warehousing (DW) is, what a data lake (Data Lake) is, and how they coexist together.

Why did I want to write about data lakes? I have been working with data and analytics for over 10 years, and currently, I'm definitely working with big data at Amazon Alexa AI in Cambridge, which is near Boston, even though I live in Victoria on Vancouver Island and often travel to Boston, Seattle, and Vancouver, and sometimes even speak at conferences in Moscow. I also write occasionally, but mainly in English, and I have already written a few books, and I have a need to share analytics trends from North America, and I sometimes write on Telegram.

I have always worked with data warehouses, and since 2015, I have been deeply involved with Amazon Web Services, and I've transitioned to cloud analytics (AWS, Azure, GCP). I have observed the evolution of analytics solutions since 2007, even working at the data warehouse vendor Teradata and implementing it at Sberbank, when Big Data emerged with Hadoop. Everyone started saying that the era of warehouses was over and that now everything was on Hadoop, and later on, discussions about Data Lake began, claiming that data warehousing was now definitely finished. But fortunately (perhaps unfortunately for those who made a lot of money setting up Hadoop), data warehousing has not disappeared.

In this article, we will examine what a data lake is. The article is aimed at people with little or no experience with data warehouses.

In the picture is Lake Bled, one of my favorite lakes, although I've only been there once, it left a lasting impression. But we will talk about another type of lake — a data lake. Many of you may have heard this term more than once, but another definition won't hurt anyone.

First of all, here are the most popular definitions of a Data Lake:

“A file repository for all types of raw data that is accessible for analysis by anyone in the organization” — Martin Fowler.

If you think of a data warehouse as a bottle of water—purified, packaged, and bottled for convenient consumption—then a data lake is like a huge reservoir of water in its natural state. Users can scoop water for themselves, dive deep, and explore. — James Dixon.

Now we certainly know that a data lake is about analytics; it allows us to store large volumes of data in their original form and provides us with the necessary and convenient access to that data.

I often like to simplify things; if I can explain a complex term in simple words, it means I've understood how it works and why it's needed. For instance, I was tinkering with my iPhone's photo gallery, and it struck me—this is a real data lake. I even created a slide for conferences:

It's all very simple. We take a photo with our phone, the photo is saved on the phone and can also be stored in iCloud (cloud storage). The phone also collects metadata about the photo: what is depicted, geo tags, time. As a result, we can use the convenient interface of the iPhone to find our photo, and we even see metrics. For example, when I search for photos with the word 'fire', I find 3 pictures of a campfire. For me, this works just like a Business Intelligence tool that operates very quickly and precisely.

And of course, we must not forget about security (authorization and authentication), otherwise our data could easily become publicly accessible. There are so many news stories about large corporations and startups that had their data exposed due to developers' negligence and failure to follow simple rules.

Even such a simple image helps us visualize what a data lake is, how it differs from traditional data storage, and its main elements:

1. Data Loading (Ingestion) — a key component of a data lake. Data can enter the data storage in two ways—batch (loading at intervals) and streaming (real-time data flow).
2. File Storage (Storage) — the main component of the Data Lake. We need the storage to be easily scalable, extremely reliable, and low-cost. For example, in AWS, this is S3.
3. Catalog and Search (Catalog and Search) — to avoid a Data Swamp (which occurs when all data is dumped into one pile, making it impossible to work with), we need to create a metadata layer for classifying data, so users can easily find the information they need for analysis. Additionally, we can leverage other search solutions, such as ElasticSearch. Search helps users find the necessary data through a user-friendly interface.
4. Processing (Process) — this step is responsible for processing and transforming data. We can transform data, change its structures, clean it up, and much more.
5. Security (Security) — it is important to take the time to design the security of the solution. For example, encrypting data during storage, processing, and transmission. It’s crucial to use authentication and authorization methods. In conclusion, an auditing tool is necessary.

From a practical standpoint, we can characterize a data lake by three attributes:

1. Collect and store anything — a data lake contains all data, both raw unprocessed data over any period of time and processed/cleaned data.
2. Deep analysis — a data lake allows users to explore and analyze data.
3. Flexible access — a data lake provides flexible access to various data and different scenarios.

Now we can discuss the difference between a data warehouse and a data lake. People often ask:

• What about data warehouses?
• Do we replace the data warehouse with a data lake, or do we expand it?
• Can we do without a data lake at all?

In short, there is no clear answer. It all depends on the specific situation, the skills within the team, and the budget. For example, migrating a data warehouse to Oracle on AWS and creating a data lake by Amazon's subsidiary — Woot — Our data lake story: How Woot.com built a serverless data lake on AWS.

On the other hand, the vendor Snowflake claims that you no longer need to think about data lakes, as their data platform (previously a data warehouse until 2020) allows you to combine both a data lake and a data warehouse. I've worked a bit with Snowflake, and it is indeed a unique product that can do that. The cost is another matter.

In conclusion, my personal opinion is that we still need a data warehouse as the primary data source for our reporting, and everything that doesn’t fit there is stored in a data lake. The entire role of analytics is to provide convenient access for the business to make decisions. Regardless, business users work more efficiently with a data warehouse than with a data lake; for example, in Amazon, there is Redshift (analytical data warehouse) and Redshift Spectrum/Athena (SQL interface for the data lake in S3 based on Hive/Presto). The same applies to other modern analytical data warehouses.

Let’s consider a typical data warehouse architecture:

This is a classical solution. We have source systems, and using ETL/ELT, we copy data into the analytical data warehouse and connect it to a Business Intelligence solution (my favorite is Tableau; what’s yours?).

Such a solution has the following disadvantages:

• ETL/ELT operations require time and resources.
• Typically, memory for storing data in an analytical data warehouse is not cheap (e.g., Redshift, BigQuery, Teradata), since we need to purchase an entire cluster.
• Business users have access to cleaned and often aggregated data and do not have the opportunity to access raw data.

Of course, it all depends on your case. If you have no issues with your data warehouse, then you absolutely do not need a data lake. But when problems arise with a lack of space, power, or cost is a key factor, then it might be worth considering a data lake. That’s why data lakes are very popular. Here’s an example of a data lake architecture:

Using the data lake approach, we load raw data into our data lake (batch or streaming), and then we process the data as needed. A data lake allows business users to create their own data transformations (ETL/ELT) or analyze data in Business Intelligence solutions (if the necessary driver is available).

The goal of any analytical solution is to serve business users. Therefore, we must always work from the business requirements. (At Amazon, this is one of the principles — working backward).

By working with both the data warehouse and the data lake, we can compare both solutions:

The main takeaway is that data storage does not compete with a data lake, but rather complements it. However, it is up to you to decide what suits your case. It's always interesting to try it yourself and draw the right conclusions.

I would also like to share one of the cases when I started using the data lake approach. It's quite straightforward; I tried using the ELT tool (we had Matillion ETL) and Amazon Redshift. My solution worked, but it didn't meet the requirements.

I needed to take web logs, transform them, and aggregate them to provide data for two cases:

1. The marketing team wanted to analyze bot activity for SEO.
2. IT wanted to look at the metrics for website performance.

Very simple, very basic logs. Here’s an example:

https 2018-07-02T22:23:00.186641Z app/my-loadbalancer/50dc6c495c0c9188 
192.168.131.39:2817 10.0.0.1:80 0.086 0.048 0.037 200 200 0 57 
"GET https://www.example.com:443/ HTTP/1.1" "curl/7.46.0" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 
arn:aws:elasticloadbalancing:us-east-2:123456789012:targetgroup/my-targets/73e2d6bc24d8a067
"Root=1-58337281-1d84f3d73c47ec4e58577259" "www.example.com" "arn:aws:acm:us-east-2:123456789012:certificate/12345678-1234-1234-1234-123456789012"
1 2018-07-02T22:22:48.364000Z "authenticate,forward" "-" "-"

One file weighed between 1 to 4 megabytes.

But there was one difficulty. We had 7 domains across the globe, and 7000 files were created in one day. That’s not a huge volume, just 50 gigabytes. However, the size of our Redshift cluster was also small (4 nodes). Loading a single file the traditional way took about a minute. In other words, the problem could not be solved directly. And that was the case when I decided to use the data lake approach. The solution looked something like this:

It is fairly simple (I want to note that the advantage of cloud operations is simplicity). I used:

• AWS Elastic Map Reduce (Hadoop) as the computing power.
• AWS S3 as the file storage with data encryption and granular access control.
• Spark as the InMemory computing power and PySpark for logic and data transformation.
• Parquet as the output of the Spark process.
• AWS Glue Crawler as a metadata collector for new data and partitions.
• Redshift Spectrum as the SQL interface to the data lake for existing Redshift users.

The smallest EMR+Spark cluster processed the whole batch of files in 30 minutes. There are also other cases for AWS, especially many related to Alexa, where the data volume is quite large.

I recently learned about one of the drawbacks of a data lake — GDPR. The issue arises when a client requests deletion of their data, and it is stored in one of the files, as we cannot use Data Manipulation Language and the DELETE operation like in a database.

I hope this article clarified the difference between a data warehouse and a data lake. If it was interesting, I can translate more of my articles or articles from professionals I read, as well as share insights about the solutions I work with and their architecture.

Source: habr.com