Plan for leveling up to become a Data Engineer

For the past eight years, I have been working as a project manager (not writing code at work), which has naturally negatively affected my tech background. I decided to bridge my technological gap and pursue the profession of Data Engineer. The main skill of a Data Engineer is the ability to design, build, and maintain data warehouses.

I created a study plan that I believe will be useful not only for me. The plan is aimed at self-study of courses. Priority is given to free courses in Russian.

Sections:

• Algorithms and Data Structures. A key section. Master this and everything else will follow. It's important to get hands-on experience writing code and using basic data structures and algorithms.
• Databases and Data Warehouses, Business Intelligence. From algorithms, we transition to data storage and processing.
• Hadoop and Big Data. When the database doesn’t fit on a hard drive, or when data needs to be analyzed but Excel can’t load it, big data begins. In my opinion, you should only move to this section after thoroughly studying the previous two.

Algorithms and Data Structures

I included studying Python, reviewing the fundamentals of mathematics and algorithmization in my plan.

• Programming in Python
• Python: basics and application
• Linear Algebra
• Basics of Discrete Mathematics
• Algorithms: Theory and Practice. Methods
• Algorithms: Theory and Practice. Data Structures

Databases and Data Warehouses, Business Intelligence

• Book: Martin Kleppmann — Designing Data-Intensive Applications. Programming, Scaling, Supporting. This book describes how different data models work, their internal implementation, limitations, and choices based on the task.
• Introduction to Databases
• Diving into DBMS
• Introduction to NoSQL Databases

Topics related to building data warehouses, ETL, and OLAP cubes strongly depend on the tools, so I do not provide links to courses in this document. It makes sense to study such systems while working on a specific project in a specific company. For an introduction to ETL, you can try Talend eller Airflow.

In my opinion, it is important to study the modern methodology of data warehouse design, Data Vault. link 1, link 2. Den beste måten å lære det på er å ta det i bruk gjennom et enkelt eksempel. GitHub har flere eksempler på implementering av Data Vault. lenke. Moderne bok om datalager: Modeling the Agile Data Warehouse with Data Vault av Hans Hultgren.

For å bli kjent med Business Intelligence-verktøyene for sluttbrukere kan man bruke den gratis rapportbyggeren, dashbordet, og mini datalageret Power BI Desktop. Læringsmateriell: link 1, link 2.

Hadoop og Big Data

• Man må begynne med å implementere MapReduce selv uten tredjepartsbiblioteker. Dette vil gjøre det lettere å forstå flertrådet implementering senere. Et utmerket eksempel på Python er beskrevet here.
• Hadoop. Et system for behandling av store datamengder.
• Introduksjon til big data engineering

Konklusjon

Ikke alt man lærer kan brukes i arbeidet. Derfor er det nødvendig med et diplomprosjekt der man får prøve ut nye kunnskaper.

Planen inneholder ikke emner relatert til dataanalyse og maskinlæring, da dette mer hører til yrket Data Scientist. Det er heller ikke emner knyttet til skyene AWS, Azure, da disse temaene er sterkt avhengige av plattformvalget.

Spørsmål til fellesskapet:
Hvor rimelig er min oppgraderingsplan? Hva bør fjernes eller legges til?
Hvilket prosjekt vil du anbefale som et diplomarbeid?

Kilde: habr.com