Skill enhancement plan to become a Data Engineer.

For the last eight years, I have been working as a project manager (not coding at work), which has naturally negatively affected my technical background. I decided to reduce my technological lag and pursue a career as a Data Engineer. The core skill of a Data Engineer is the ability to design, build, and maintain data warehouses.

I created a learning plan that I believe will be useful not only for me. The plan is designed for self-study of courses. Priority is given to free courses in Russian.

Sections:

• Algorithms and Data Structures. A key section. If you master this, everything else will follow. It is crucial to practice coding and using basic structures and algorithms.
• Databases and Data Warehouses, Business Intelligence. From algorithms, we move on to data storage and processing.
• Hadoop and Big Data. When the database cannot fit on the hard drive, or when data needs to be analyzed but Excel can no longer handle it, big data begins. In my opinion, one should only transition to this section after thoroughly studying the two previous ones.

Algorithms and data structures

In my plan, I included learning Python, revisiting the basics of mathematics, and algorithm design.

• Programming in Python
• Python: Fundamentals and Applications
• Linear Algebra
• Introduction to Discrete Mathematics
• Algorithms: Theory and Practice. Methods
• Algorithms: Theory and Practice. Data Structures

Databases and data warehouses, Business Intelligence

• Book: Martin Kleppmann — Designing Data-Intensive Applications. This book describes how different data models work, their internal implementations, limitations, and choices based on the task.
• Introduction to Databases
• Immersion in DBMS
• Introduction to NoSQL Databases

Topics related to building data warehouses, ETL, and OLAP cubes heavily depend on tools, so I won't provide links to courses in this document. It is advisable to study such systems while working on a specific project in a specific company. To get acquainted with ETL, one can try Talend or Airflow.

In my opinion, it is important to learn the modern methodology of designing data warehouses, Data Vault. link 1, link 2. And the best way to learn it is to take and implement it with a simple example. There are several examples of Data Vault implementation on GitHub. link. A modern book on data warehousing: Modeling the Agile Data Warehouse with Data Vault by Hans Hultgren.

To get acquainted with Business Intelligence tools for end users, you can use the free report builder, dashboards, and mini data warehouses in Power BI Desktop. Educational materials: link 1, link 2.

Hadoop and Big Data

• You should start with implementing MapReduce independently without third-party libraries. This will help you better understand multi-threaded implementations in the future. A great example in Python is described here.
• Hadoop. A system for processing large volumes of data.
• Introduction to Big Data Engineering

Conclusion

Not everything you learn can be applied at work. Therefore, a diploma project is necessary, where you will try to apply new knowledge.

The plan does not include topics related to data analysis and Machine Learning, as these are more related to the Data Scientist profession. Also, there are no topics related to cloud services AWS and Azure, as these topics heavily depend on the choice of platform.

Questions to the community:
How reasonable is my development plan? What should I remove or add?
What project would you recommend as a thesis project?

Source: habr.com