{"id":55327,"date":"2020-01-18T00:00:00","date_gmt":"2020-01-17T21:00:00","guid":{"rendered":"https:\/\/prohoster.info\/blog\/blog_prohoster\/samye-vostrebovannye-navyki-v-professii-data-engineer"},"modified":"2020-02-18T14:03:26","modified_gmt":"2020-02-18T11:03:26","slug":"samye-vostrebovannye-navyki-v-professii-data-engineer","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/samye-vostrebovannye-navyki-v-professii-data-engineer","title":{"rendered":"The most in-demand skills in the data engineer profession","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

According to statistics of 2019<\/a><\/noindex>, currently the data engineer profession is one that is growing faster than any other. Data engineers play a critically important role in organizations \u2013 they create and maintain pipelines and databases that are used for processing, transforming, and storing data. What skills are essential for professionals in this field? Is the list different from what is required of data scientists? You will learn all about this in my article.<\/p>\n

I analyzed job openings for the position of data engineer as they appeared in January 2020 to understand which technological skills are in the highest demand. I then compared the results with statistics on job vacancies for data scientists \u2013 revealing some interesting differences.<\/p>\n

Let's skip the lengthy preambles \u2013 here are the top ten technologies mentioned most frequently in job postings:<\/p>\n

\"The<\/p>\n

Mentions of technologies in job postings for data engineers in 2020<\/i><\/p>\n

Let's break it down.
\n<\/a><\/noindex><\/p>\n

Responsibilities of a data engineer<\/h4>\n

\nToday, the work performed by data engineers is of immense importance to organizations \u2013 these individuals are responsible for storing information and structuring it so that other employees can work with it. Data engineers build pipelines to facilitate data retrieval, whether streaming or in batches, from multiple sources. The pipelines then execute extraction, transformation, and loading operations (in other words, ETL processes), making the data more suitable for further use. After that, the data is passed on to analysts and data scientists for deeper processing. Finally, the data concludes its journey on dashboards, reports, and machine learning models.<\/p>\n

I searched for information that would allow me to draw conclusions about which technologies are currently in demand for data engineers.<\/p>\n

Methods<\/h4>\n

\nI gathered information from three job search websites \u2013 SimplyHired<\/a><\/noindex>, Indeed<\/a><\/noindex> and Monster<\/a><\/noindex> and looked at which keywords appeared in conjunction with 'data engineer' in job postings aimed at residents of the U.S. For this task, I used two Python libraries \u2013 Requests<\/a><\/noindex> and Beautiful Soup<\/a><\/noindex>I included both keywords from the previous list for analyzing data scientist job positions and those I manually selected while reading job postings for data engineers. LinkedIn was not among the sources since I was banned there after my last attempt to collect data.<\/p>\n

For each keyword, I calculated the percentage of occurrences from the total number of texts on each site separately, and then computed the average value across the three sources.<\/p>\n

Results<\/h4>\n

\nBelow are thirty technical terms from the field of data engineering with the highest indicators across all three job sites.<\/p>\n

\"The<\/p>\n

Here are the same figures, but presented in a table:<\/p>\n

\"The<\/p>\n

Let's proceed in order.<\/p>\n

Overview of Results<\/b><\/p>\n

Both SQL and Python appear in more than two-thirds of the examined job postings. These are the two technologies worth studying first. Python<\/a><\/noindex> is a very popular programming language used for data work, creating websites, and writing scripts. SQL<\/a><\/noindex> stands for Structured Query Language; it implies a standard implemented by a group of languages and is used for extracting data from relational databases. It has been around for a long time and has established itself as very reliable.<\/p>\n

Spark is mentioned in about half of the job postings. Apache Spark<\/a><\/noindex> is a unified analytics engine for big data processing with built-in modules for streaming, SQL, machine learning, and graph processing. It's particularly popular among those working with large databases.<\/p>\n

AWS appears in approximately 45% of job texts. This is Amazon's cloud computing platform, which holds the largest market share among all cloud platforms.
\nNext are Java and Hadoop, each appearing in just over 40% of postings. Java.<\/a><\/noindex> is a widely-used, battle-tested language that in the 2019 Stack Overflow Developer Survey<\/a><\/noindex> ranked tenth among the languages that cause dread among developers. In contrast, Python was second among the most loved languages. Oracle is behind Java, and everything you need to know about it can be understood from this screenshot of the official page from January 2020.<\/p>\n

\"The
\n
\nAs if I traveled in a time machine<\/i>
\nApache Hadoop<\/a><\/noindex> uses the MapReduce programming model with server clusters for big data. This model is increasingly being abandoned.<\/p>\n

Next, we see Hive, Scala, Kafka, and NoSQL \u2013 each of these technologies is mentioned in a quarter of the job listings available. Apache Hive is a data warehouse program that 'simplifies reading, writing, and managing large datasets located in distributed storage using SQL.' Scala<\/a><\/noindex> is a programming language actively used for big data. In particular, Spark was created using Scala. In the already mentioned ranking of highly sought-after languages, Scala ranks eleventh. Apache Kafka<\/a><\/noindex> is a distributed platform for processing stream messages. It is very popular as a means of streaming data.<\/p>\n

NoSQL Databases<\/a><\/noindex> positional against SQL. They are non-relational, unstructured, and possess horizontal scalability. NoSQL has gained some popularity, but the frenzied enthusiasm for this approach, including prophecies that it would replace SQL as the dominant storage paradigm, seems to be behind us now.<\/p>\n

Comparison of terms in data scientist job listings<\/h4>\n

\nHere are thirty technological terms most commonly found among employers in the data science field. This list was obtained in the same way I described earlier for data engineering.<\/p>\n

\"The<\/p>\n

Mentions of technologies in job listings for data scientist positions in 2020<\/i><\/p>\n

In terms of the overall number, compared to the earlier reviewed set, there were 28% more job listings (12,013 versus 9,396). Let's look at which technologies are less frequently found in job listings for data scientists compared to data engineers.<\/p>\n

More popular in data engineering<\/b><\/p>\n

The chart below shows keywords with an average difference in values greater than 10% or less than -10%.<\/p>\n

\"The<\/p>\n

The largest differences in keyword frequency between data engineers and data scientists<\/i><\/p>\n

AWS shows the most significant increase: it appears in data engineering 25% more frequently than in data science (approximately 45% and 20% of total job listings, respectively). The difference is noticeable!<\/p>\n

Here is the same data presented slightly differently \u2013 in the chart, the results for the same keyword in job listings for data engineer and data scientist positions are placed side by side.<\/p>\n

\"The<\/p>\n

The largest differences in keyword frequency between data engineers and data scientists<\/i><\/p>\n

The next largest increase I noted for Spark \u2013 data engineers often have to work with big data. Kafka<\/a><\/noindex> also increased by 20%, which is almost four times compared to the data scientist job results. Data transmission is one of the key responsibilities of a data engineer. Finally, the number of mentions turned out to be 15% higher in the field of data engineering for Java, NoSQL, Redshift, SQL, and Hadoop.<\/p>\n

Less popular in data engineering<\/b><\/p>\n

Now let's take a look at which technologies are less popular among data engineer job postings.
\nThe sharpest decline compared to the field of data science was R<\/a><\/noindex>: where it appeared in about 56% of job postings, here \u2013 only in 17%. Impressive. R is a programming language that is popular among scientists and statisticians, and also ranks eighth in the list of most dreaded languages.<\/p>\n

SAS<\/a><\/noindex> also appears in data engineer job postings significantly less \u2013 the difference is 14%. SAS is a proprietary language designed for statistical and data work. An interesting point: judging by the results of my job posting research for data scientists<\/a><\/noindex>, it has recently lost ground significantly \u2013 more than any other technology.<\/p>\n

In demand both in data engineering and in data science<\/b><\/p>\n

It's worth noting that eight out of the top ten positions in both sets overlap. SQL, Python, Spark, AWS, Java, Hadoop, Hive, and Scala made the top ten for both the data engineering and data science sectors. In the graph below, you can see the fifteen most popular technologies among data engineer employers, along with their figures for data scientists\u2019 job postings.<\/p>\n

\"The<\/p>\n

Recommendations<\/h4>\n

\nIf you want to get into data engineering, I would recommend mastering the following technologies \u2013 I list them in order of approximate priority.<\/p>\n

Learn SQL. I particularly recommend PostgreSQL because it has open-source code, great popularity in the community, and is in a growth phase. You can learn how to use the language from the book My Memorable SQL \u2013 its pilot version is available here<\/a><\/noindex>.<\/p>\n

Master Python, even if not at the hardcore level. The book My Memorable Python is aimed at beginners. It can be purchased at Amazon<\/a><\/noindex>, either as an electronic copy or a physical one, your choice, or you can download it in pdf or epub format on this site<\/a><\/noindex>.<\/p>\n

Once you become familiar with Python, move on to pandas \u2013 a Python library used for data cleaning and processing. If you're aiming to work in a company that requires Python skills (which is the majority), you can be assured that knowledge of pandas will be expected by default. I'm currently finishing an introductory guide to working with pandas \u2013 you can subscribe<\/a><\/noindex>, so you won't miss the release.<\/p>\n

Master AWS. If you want to become a data engineer, you can't do without a cloud platform in your toolkit, and AWS is the most popular one. Courses from Linux Academy<\/a><\/noindex>, helped me when I was learning data engineering on Google Cloud<\/a><\/noindex>, and I think they have good materials on AWS as well.<\/p>\n

If you\u2019ve already mastered this entire list and want to further enhance your standing with employers as a data engineer, I suggest adding Apache Spark for big data. Although my research on data science job postings showed a decline in interest, it still appears in nearly every second data engineer job listing.<\/p>\n

Finally,<\/h4>\n

\nI hope this overview of the most in-demand technologies for data engineers has been useful to you. If you're interested in job prospects for analysts, read my other article<\/a><\/noindex>. Happy engineering!
\n
Source: habr.com<\/a><\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"

\u0421\u043e\u0433\u043b\u0430\u0441\u043d\u043e \u0441\u0442\u0430\u0442\u0438\u0441\u0442\u0438\u043a\u0435 2019 \u0433\u043e\u0434\u0430, data engineer \u043d\u0430 \u0434\u0430\u043d\u043d\u044b\u0439 \u043c\u043e\u043c\u0435\u043d\u0442 \u044f\u0432\u043b\u044f\u0435\u0442\u0441\u044f \u043f\u0440\u043e\u0444\u0435\u0441\u0441\u0438\u0435\u0439, \u0441\u043f\u0440\u043e\u0441 \u043d\u0430 \u043a\u043e\u0442\u043e\u0440\u0443\u044e \u0440\u0430\u0441\u0442\u0435\u0442 \u0431\u044b\u0441\u0442\u0440\u0435\u0435 \u0432\u0441\u0435\u0445 \u043f\u0440\u043e\u0447\u0438\u0445. Data engineer \u0438\u0433\u0440\u0430\u0435\u0442 \u0432 \u043e\u0440\u0433\u0430\u043d\u0438\u0437\u0430\u0446\u0438\u0438 \u043a\u0440\u0438\u0442\u0438\u0447\u0435\u0441\u043a\u0438 \u0432\u0430\u0436\u043d\u0443\u044e \u0440\u043e\u043b\u044c \u2013 \u0441\u043e\u0437\u0434\u0430\u0435\u0442 \u0438 \u043f\u043e\u0434\u0434\u0435\u0440\u0436\u0438\u0432\u0430\u0435\u0442 \u0432 \u0440\u0430\u0431\u043e\u0447\u0435\u043c \u0441\u043e\u0441\u0442\u043e\u044f\u043d\u0438\u0438 \u043f\u0430\u0439\u043f\u043b\u0430\u0439\u043d\u044b \u0438 \u0431\u0430\u0437\u044b \u0434\u0430\u043d\u043d\u044b\u0445, \u043a\u043e\u0442\u043e\u0440\u044b\u0435 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u0443\u044e\u0442\u0441\u044f \u0434\u043b\u044f \u043e\u0431\u0440\u0430\u0431\u043e\u0442\u043a\u0438, \u0442\u0440\u0430\u043d\u0441\u0444\u043e\u0440\u043c\u0430\u0446\u0438\u0438 \u0438 \u0445\u0440\u0430\u043d\u0435\u043d\u0438\u044f \u0434\u0430\u043d\u043d\u044b\u0445. \u041a\u0430\u043a\u0438\u0435 \u043d\u0430\u0432\u044b\u043a\u0438 \u043d\u0443\u0436\u043d\u044b \u043f\u0440\u0435\u0434\u0441\u0442\u0430\u0432\u0438\u0442\u0435\u043b\u044f\u043c \u044d\u0442\u043e\u0439 \u043f\u0440\u043e\u0444\u0435\u0441\u0441\u0438\u0438 \u0432 \u043f\u0435\u0440\u0432\u0443\u044e \u043e\u0447\u0435\u0440\u0435\u0434\u044c? \u041e\u0442\u043b\u0438\u0447\u0430\u0435\u0442\u0441\u044f […]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-55327","post","type-post","status-publish","format-standard","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\n","aioseo_head_json":{"title":"\ud83e\udd47The most in-demand skills in the data engineer profession | ProHoster","description":"According to 2019 statistics, data engineering is currently a profession in high demand.","canonical_url":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/samye-vostrebovannye-navyki-v-professii-data-engineer","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"en_US","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u0421\u0430\u043c\u044b\u0435 \u0432\u043e\u0441\u0442\u0440\u0435\u0431\u043e\u0432\u0430\u043d\u043d\u044b\u0435 \u043d\u0430\u0432\u044b\u043a\u0438 \u0432 \u043f\u0440\u043e\u0444\u0435\u0441\u0441\u0438\u0438 data engineer | ProHoster","og:description":"\u0421\u043e\u0433\u043b\u0430\u0441\u043d\u043e \u0441\u0442\u0430\u0442\u0438\u0441\u0442\u0438\u043a\u0435 2019 \u0433\u043e\u0434\u0430, data engineer \u043d\u0430 \u0434\u0430\u043d\u043d\u044b\u0439 \u043c\u043e\u043c\u0435\u043d\u0442 \u044f\u0432\u043b\u044f\u0435\u0442\u0441\u044f \u043f\u0440\u043e\u0444\u0435\u0441\u0441\u0438\u0435\u0439, \u0441\u043f\u0440\u043e\u0441 \u043d\u0430.","og:url":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/samye-vostrebovannye-navyki-v-professii-data-engineer","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-01-17T21:00:00+00:00","article:modified_time":"2020-02-18T11:03:26+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"55327","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 13:45:53","updated":"2022-09-30 07:37:35","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/55327","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/comments?post=55327"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/55327\/revisions"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/media?parent=55327"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/categories?post=55327"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/tags?post=55327"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}