According to , currently the data engineer profession is one that is growing faster than any other. Data engineers play a critically important role in organizations – they create and maintain pipelines and databases that are used for processing, transforming, and storing data. What skills are essential for professionals in this field? Is the list different from what is required of data scientists? You will learn all about this in my article.
I analyzed job openings for the position of data engineer as they appeared in January 2020 to understand which technological skills are in the highest demand. I then compared the results with statistics on job vacancies for data scientists – revealing some interesting differences.
Let's skip the lengthy preambles – here are the top ten technologies mentioned most frequently in job postings:
Mentions of technologies in job postings for data engineers in 2020
Let's break it down.
Responsibilities of a data engineer
Today, the work performed by data engineers is of immense importance to organizations – these individuals are responsible for storing information and structuring it so that other employees can work with it. Data engineers build pipelines to facilitate data retrieval, whether streaming or in batches, from multiple sources. The pipelines then execute extraction, transformation, and loading operations (in other words, ETL processes), making the data more suitable for further use. After that, the data is passed on to analysts and data scientists for deeper processing. Finally, the data concludes its journey on dashboards, reports, and machine learning models.
I searched for information that would allow me to draw conclusions about which technologies are currently in demand for data engineers.
Methods
I gathered information from three job search websites – , and and looked at which keywords appeared in conjunction with 'data engineer' in job postings aimed at residents of the U.S. For this task, I used two Python libraries – and I included both keywords from the previous list for analyzing data scientist job positions and those I manually selected while reading job postings for data engineers. LinkedIn was not among the sources since I was banned there after my last attempt to collect data.
For each keyword, I calculated the percentage of occurrences from the total number of texts on each site separately, and then computed the average value across the three sources.
Results
Below are thirty technical terms from the field of data engineering with the highest indicators across all three job sites.
Here are the same figures, but presented in a table:
Let's proceed in order.
Overview of Results
Both SQL and Python appear in more than two-thirds of the examined job postings. These are the two technologies worth studying first. is a very popular programming language used for data work, creating websites, and writing scripts. stands for Structured Query Language; it implies a standard implemented by a group of languages and is used for extracting data from relational databases. It has been around for a long time and has established itself as very reliable.
Spark is mentioned in about half of the job postings. is a unified analytics engine for big data processing with built-in modules for streaming, SQL, machine learning, and graph processing. It's particularly popular among those working with large databases.
AWS appears in approximately 45% of job texts. This is Amazon's cloud computing platform, which holds the largest market share among all cloud platforms.
Next are Java and Hadoop, each appearing in just over 40% of postings. is a widely-used, battle-tested language that in the ranked tenth among the languages that cause dread among developers. In contrast, Python was second among the most loved languages. Oracle is behind Java, and everything you need to know about it can be understood from this screenshot of the official page from January 2020.
As if I traveled in a time machine
uses the MapReduce programming model with server clusters for big data. This model is increasingly being abandoned.
Next, we see Hive, Scala, Kafka, and NoSQL – each of these technologies is mentioned in a quarter of the job listings available. Apache Hive is a data warehouse program that 'simplifies reading, writing, and managing large datasets located in distributed storage using SQL.' is a programming language actively used for big data. In particular, Spark was created using Scala. In the already mentioned ranking of highly sought-after languages, Scala ranks eleventh. is a distributed platform for processing stream messages. It is very popular as a means of streaming data.
positional against SQL. They are non-relational, unstructured, and possess horizontal scalability. NoSQL has gained some popularity, but the frenzied enthusiasm for this approach, including prophecies that it would replace SQL as the dominant storage paradigm, seems to be behind us now.
Comparison of terms in data scientist job listings
Here are thirty technological terms most commonly found among employers in the data science field. This list was obtained in the same way I described earlier for data engineering.
Mentions of technologies in job listings for data scientist positions in 2020
In terms of the overall number, compared to the earlier reviewed set, there were 28% more job listings (12,013 versus 9,396). Let's look at which technologies are less frequently found in job listings for data scientists compared to data engineers.
More popular in data engineering
The chart below shows keywords with an average difference in values greater than 10% or less than -10%.
The largest differences in keyword frequency between data engineers and data scientists
AWS shows the most significant increase: it appears in data engineering 25% more frequently than in data science (approximately 45% and 20% of total job listings, respectively). The difference is noticeable!
Here is the same data presented slightly differently – in the chart, the results for the same keyword in job listings for data engineer and data scientist positions are placed side by side.
The largest differences in keyword frequency between data engineers and data scientists
The next largest increase I noted for Spark – data engineers often have to work with big data. also increased by 20%, which is almost four times compared to the data scientist job results. Data transmission is one of the key responsibilities of a data engineer. Finally, the number of mentions turned out to be 15% higher in the field of data engineering for Java, NoSQL, Redshift, SQL, and Hadoop.
Less popular in data engineering
Now let's take a look at which technologies are less popular among data engineer job postings.
The sharpest decline compared to the field of data science was : where it appeared in about 56% of job postings, here – only in 17%. Impressive. R is a programming language that is popular among scientists and statisticians, and also ranks eighth in the list of most dreaded languages.
also appears in data engineer job postings significantly less – the difference is 14%. SAS is a proprietary language designed for statistical and data work. An interesting point: judging by the results , it has recently lost ground significantly – more than any other technology.
In demand both in data engineering and in data science
It's worth noting that eight out of the top ten positions in both sets overlap. SQL, Python, Spark, AWS, Java, Hadoop, Hive, and Scala made the top ten for both the data engineering and data science sectors. In the graph below, you can see the fifteen most popular technologies among data engineer employers, along with their figures for data scientists’ job postings.
Recommendations
If you want to get into data engineering, I would recommend mastering the following technologies – I list them in order of approximate priority.
Learn SQL. I particularly recommend PostgreSQL because it has open-source code, great popularity in the community, and is in a growth phase. You can learn how to use the language from the book My Memorable SQL – its pilot version is available .
Master Python, even if not at the hardcore level. The book My Memorable Python is aimed at beginners. It can be purchased at , either as an electronic copy or a physical one, your choice, or you can download it in pdf or epub format .
Once you become familiar with Python, move on to pandas – a Python library used for data cleaning and processing. If you're aiming to work in a company that requires Python skills (which is the majority), you can be assured that knowledge of pandas will be expected by default. I'm currently finishing an introductory guide to working with pandas – you can , so you won't miss the release.
Master AWS. If you want to become a data engineer, you can't do without a cloud platform in your toolkit, and AWS is the most popular one. Courses from , helped me when I was learning , and I think they have good materials on AWS as well.
If you’ve already mastered this entire list and want to further enhance your standing with employers as a data engineer, I suggest adding Apache Spark for big data. Although my research on data science job postings showed a decline in interest, it still appears in nearly every second data engineer job listing.
Finally,
I hope this overview of the most in-demand technologies for data engineers has been useful to you. If you're interested in job prospects for analysts, read my . Happy engineering!
Source: habr.com
