Hello again! The article title speaks for itself. In anticipation of the course launch, we will explore who data engineers really are. The article contains many useful links. Enjoy your reading.
A simple guide on how to ride the wave of Data Engineering without being swept away.
It seems that nowadays everyone wants to become a data scientist. But what about Data Engineering? Essentially, it is a hybrid of a data analyst and a data scientist; a data engineer is typically responsible for managing workflows, processing pipelines, and ETL processes.Given the significance of these functions, it is currently another popular professional jargon that is gaining momentum.
High salaries and immense demand are just part of what makes this job incredibly attractive! If you want to join the ranks of heroes, it is never too late to start learning. In this post, I have gathered all the necessary information to help you take your first steps.
So, let's get started!
What is Data Engineering?
Honestly, there is no better explanation than this:
"A scientist can discover a new star, but cannot create it. They will have to ask an engineer to do it for them."
– Gordon Lindsay Glegg
Thus, the role of a data engineer is quite significant.
As the name suggests, data engineering is all about data, specifically its delivery, storage, and processing. Accordingly, the primary task of engineers is to ensure a reliable infrastructure for data. If we look at the AI hierarchy of needs, data engineering occupies the first 2–3 stages: collection, movement, and storage, data preparation.
What does a data engineer do?
With the emergence of big data, the scope of responsibilities has changed drastically. Previously, these experts would write large SQL queries and transfer data using tools like Informatica ETL, Pentaho ETL, Talend, but now the requirements for data engineers have increased.
Most companies with job openings for data engineers have the following requirements:
- Excellent knowledge of SQL and Python.
- Experience with cloud platforms, particularly Amazon Web Services.
- Knowledge of Java/Scala is preferred.
- A good understanding of SQL and NoSQL databases (data modeling, data storage).
Keep in mind, this is just the essentials. From this list, one can assume that data engineers are specialists in software development and backend systems.
For example, if a company starts generating a large volume of data from different sources, your task as a data engineer is to organize the collection, processing, and storage of that information.
The list of tools used in this case may differ; it all depends on the volume of this data, the speed of its arrival, and its heterogeneity. Most companies do not deal with big data at all, so a SQL database (PostgreSQL, MySQL, etc.) can be used as a centralized repository, along with a small set of scripts that direct the data into storage.
IT giants like Google, Amazon, Facebook, or Dropbox have higher requirements: knowledge of Python, Java, or Scala.
- Experience with big data: Hadoop, Spark, Kafka.
- Knowledge of algorithms and data structures.
- Understanding the fundamentals of distributed systems.
- Experience with data visualization tools like Tableau or ElasticSearch would be a significant plus.
This indicates a clear shift towards big data, particularly in processing under high loads. These companies have increased requirements for system resilience.
Data Engineers vs. Data Scientists
Okay, that was a simple and fun comparison (nothing personal), but in reality, it’s much more complex.
First of all, you should know that there is quite a lot of ambiguity in the delineation of roles and skills between data scientists and data engineers. This means you can easily be perplexed by what skills are actually needed for a successful data engineer. Of course, there are certain overlapping skills between the two roles. But there is also a whole range of diametrically opposed skills.
Data science is a serious business, but we are moving towards a world of functional data science, where practitioners are capable of doing their own analytics. To engage data pipelines and integrated data structures, you need data engineers, not scientists.
Is a data engineer in greater demand than a data scientist?
— Yes, because before you can bake a carrot cake, you need to gather, peel, and stock up on carrots first!
A data engineer understands programming better than any data scientist, but when it comes to statistics, the opposite is true.
But here’s the advantage of a data engineer:
without him/her, the value of a prototype model, often consisting of a snippet of poor-quality code in a Python file obtained from a data scientist and somehow yielding results, tends to approach zero.
Without a data engineer, that code will never become a project, and no business problem will be effectively solved. The data engineer aims to turn it all into a product.
Key knowledge that a data engineer should have
So, if this job ignites a spark in you and you are full of enthusiasm — you have the ability to learn this, you can acquire all the necessary skills and become a true rock star in data development. And yes, you can achieve this even without programming skills or other technical knowledge. It’s challenging, but possible!
What are the first steps?
You need to have a general understanding of what’s what.
First of all, Data Engineering relates to computer science. Specifically — you should understand efficient algorithms and data structures. Secondly, since data engineers work with data, it’s essential to understand the principles of database operation and the underlying structures.
For example, common B-tree SQL databases are based on the B-Tree data structure, and in modern distributed repositories, LSM-Tree and other modifications of hash tables.
* These steps are based on a remarkable article . So, if you know Russian, support this author and read .
1. Algorithms and Data Structures
Using the right data structure can significantly improve the performance of an algorithm. Ideally, we should all learn about data structures and algorithms in our schools, but this is rarely covered. In any case, it's never too late to familiarize yourself.
So, here are my favorite free courses to learn about data structures and algorithms:
Plus, don't forget about the classic work on algorithms by Thomas Cormen — . It's the perfect reference when you need to refresh your memory.
- To improve your skills, use .
You can also dive into the world of databases with amazing videos from Carnegie Mellon University on YouTube:
- .
- .
2. Learning SQL
Our whole life is data. And to extract this data from the database, you need to 'speak' to it in one language.
SQL (Structured Query Language) is the language for communicating in the data field. No matter what anyone says, SQL has lived, is living, and will continue to live for a long time.
If you've been in development for a while, you've probably noticed that rumors about the imminent death of SQL come up periodically. The language was developed in the early 1970s and is still very popular among analysts, developers, and just enthusiasts.
Without knowledge of SQL, there's not much you can do in data engineering, as you'll inevitably need to create queries to extract data. All modern big data warehouses support SQL:
- Amazon Redshift
- HP Vertica
- Oracle
- SQL Server
… and many others.
To analyze large volumes of data stored in distributed systems like HDFS, SQL mechanisms such as Apache Hive, Impala, etc. were invented. You see, it's not going anywhere.
How to learn SQL? Just practice it.
For this, I would recommend checking out an excellent tutorial, which, by the way, is free, from .
The standout feature of these courses is the interactive environment where you can write and execute SQL queries right in your browser. The resource is also a great addition. And you can apply this knowledge in in the Database section.
3. Programming in Python and Java/Scala
I've already discussed why you should learn the Python programming language in the article . As for Java and Scala, most tools for storing and processing large amounts of data are written in these languages. For example:
- Apache Kafka (Scala)
- Hadoop, HDFS (Java)
- Apache Spark (Scala)
- Apache Cassandra (Java)
- HBase (Java)
- Apache Hive (Java)
To understand how these tools work, you need to know the languages they are written in. The functional approach of Scala effectively addresses parallel data processing tasks. Unfortunately, Python can't boast of speed and parallel processing. Overall, knowledge of multiple languages and programming paradigms positively impacts the breadth of approaches to problem-solving.
To dive into the Scala language, you can read by the creator of the language. Additionally, Twitter published a good introductory guide — .
As for Python, I believe is the best mid-level book.
4. Tools for Big Data
Here’s a list of the most popular tools in the big data world:
- Apache Spark
- Apache Kafka
- Apache Hadoop (HDFS, HBase, Hive)
- Apache Cassandra
You can find more information on building large data blocks in this amazing . The most popular tools are Spark and Kafka. They are definitely worth studying, preferably to understand how they work internally. Jay Kreps (co-author of Kafka) published a monumental work in 2013 titled , by the way, the foundational ideas from this tome were used to create Apache Kafka.
- An introduction to Hadoop can be found in .
- The most comprehensive guide to Apache Spark for me is — .
5. Cloud platforms
Knowledge of at least one cloud platform is listed among the basic requirements for data engineer candidates. Employers favor Amazon Web Services, followed by Google Cloud Platform, and Microsoft Azure rounds out the top three.
You should be well-versed in Amazon EC2, AWS Lambda, Amazon S3, DynamoDB.
6. Distributed Systems
Working with big data implies having clusters of independently operating computers, connected via a network. The larger the cluster, the greater the likelihood of its nodes failing. To become a top expert in the data field, you need to delve into the challenges and existing solutions for distributed systems. This field is both old and complex.
Andrew Tanenbaum is considered a pioneer in this field. For those unafraid of theory, I recommend his book , it may seem complex for beginners, but it will truly help you sharpen your skills.
I believe is the best introductory book. By the way, Martin has a wonderful . His work will help you systematize knowledge about building modern infrastructure for storing and processing big data.
For those who enjoy watching videos, there is a course on Youtube .
7. Data Pipelines
Data pipelines are essential for anyone working as a data engineer.
Most of the time a data engineer builds what is called a data pipeline, meaning they create a process for delivering data from one place to another. This can involve user scenarios that go to an API of an external service or make SQL queries, enrich the data, and place it into a centralized repository (data warehouse) or unstructured data storage (data lakes).
In summary: the main checklist for a data engineer
To sum up, a good understanding of the following is necessary:
- Information systems;
- Software development (Agile, DevOps, Design Techniques, SOA);
- Distributed systems and parallel programming;
- Database fundamentals — planning, design, operation, and troubleshooting;
- Experiment design — A/B testing to validate concepts, gauge reliability, assess system performance, and develop robust paths for the operational delivery of good solutions.
These are just a few requirements to become a data engineer, so explore and familiarize yourself with data systems, information systems, continuous delivery/deployment/integration, programming languages, and other topics in computer science (not all subject areas).
And finally, the last but very important thing I want to say.
The path to becoming a Data Engineer is not as simple as it may seem. It can be unforgiving, frustrating, and you must be prepared for that. There may be moments in this journey that push you to quit. But it is a genuine effort and a learning process.
Just don’t sugarcoat it from the start. The whole point of the journey is to learn as much as possible and be ready for new challenges.
Here’s a great image I came across that illustrates this point well:
And yes, don’t forget to avoid burnout and take breaks. That’s important too. Good luck!
What do you think of the article, friends? We invite you to , which will take place today at 8:00 PM. During the webinar, we will discuss how to build an efficient and scalable data processing system for a small company or startup with minimal costs. As a practical exercise, we’ll familiarize ourselves with Google Cloud data processing tools. See you there!
Source: habr.com
