Who are DevOps?

Currently, this is arguably the most expensive position on the market. The buzz around DevOps engineers surpasses all imaginable limits, especially with Senior DevOps engineers.
I work as the head of the integration and automation department; guess the English abbreviation—DevOps Manager. Does the English title accurately reflect our daily activities? Probably not, but the Russian version is more precise in this case. Due to my role, it is essential for me to interview potential team members, and over the past year, around 50 people have crossed my path, with just as many filtered out at the prescreening stage with my colleagues.

We are still looking for colleagues since underneath the label DevOps lies a vast array of different types of engineers.

Everything written below is my personal opinion; you are not obliged to agree with it, but I admit it might color your view on the topic. Despite the risk of backlash, I publish my opinion because I believe it deserves to be shared.

Companies have various understandings of who DevOps engineers are and, in order to quickly hire resources, they attach this label to everyone. The situation is quite strange, as companies are willing to pay unrealistic rewards to these individuals, often getting just an admin-tool specialist in return.

So who are DevOps engineers?

Let's start with the history of its emergence — Development Operations appeared as another step towards optimizing collaboration in small teams to increase product delivery speed, as an expected consequence. The idea was to empower the development team with knowledge of procedures and approaches in managing the product environment. In other words, a developer should understand how their product works under various conditions, know how to deploy their product, and which environment characteristics to adjust to improve performance. Thus, over time, developers with a DevOps approach emerged. DevOps developers wrote build and packaging scripts to simplify their activities and the operability of the production environment. However, the complexity of solution architectures and the interdependencies of infrastructure components began to degrade environment performance over time, requiring a deeper understanding of various components with each iteration, ultimately reducing a developer's productivity due to the additional effort needed to understand components and tune systems for specific tasks. The cost of developers increased, along with the product's price, sharply raising the demands for new developers in the team, as they needed to cover the responsibilities of the 'stars' of development, who naturally became less available. It's also worth noting that, in my experience, few developers are interested in the specifics of packet processing by the operating system's kernel, packet routing rules, or host security aspects. A logical step was to involve an administrator, someone familiar with these issues, and assign such responsibilities to them, which, thanks to their experience, allowed achieving similar results at a lower cost compared to the cost of a 'star' developer. These administrators were placed in teams, with the primary task of managing test and production environments according to the specific rules of their team, utilizing resources allocated to that team. Thus, this is how DevOps appeared in the perception of most.

Over time, either partially or completely, system administrators began to understand the specific needs of this development team — how to simplify life for developers and testers, how to roll out updates without being stuck in the office late on a Friday fixing deployment errors. As time passed, system administrators became the 'stars,' understanding what developers wanted. To minimize impacts, management tools began to emerge, and everyone recalled the old and reliable OS-level isolation methods that minimized security, network management requirements, and overall host configuration, consequently reducing the burden on the new 'stars.'

A 'wonderful' thing emerged — Docker. Why wonderful? Simply because creating isolation in chroot or jail, as well as OpenVZ, required non-trivial knowledge of the OS, while Docker allows for the easy creation of an isolated application environment on a host with everything necessary inside, and hands back control to developers while enabling system administrators to manage just a single host, ensuring its security and high availability — a logical simplification. However, progress does not stand still, and systems are becoming increasingly complex, with more and more components; a single host no longer meets system demands, and it becomes necessary to build clusters, bringing us back to system administrators capable of constructing these systems.

Cycle after cycle, various systems emerge to simplify development and/or administration; orchestration systems come into play, which are easy to use as long as there is no need to deviate from the standard process. Microservices architecture also arose to simplify everything described above — fewer interconnections, easier management. In my experience, I have not fully encountered microservices architecture; I would say it's a 50-50 split — 50 percent microservices, black boxes where input comes in and processed output comes out, and the other 50 percent being a fragmented monolith, services unable to function independently of other components. All of this has imposed further limitations on the knowledge required of both developers and administrators.

Such "swings" of expert knowledge of various resources continue to this day. But we've digressed a bit; there are several points worth discussing.

Build Engineer/Release Engineer

Highly specialized engineers who emerged as a means to standardize software build processes and releases. During the widespread adoption of Agile, it seemed they had fallen out of demand; however, this is far from the truth. This specialization arose as a method to standardize specifically software assembly and delivery on an industrial scale, meaning using standard techniques for all company products. With the advent of DevOps, developers partially lost some of their functions, as developers began preparing the product for delivery. Given the changing infrastructure and the approach to the fastest possible delivery without regard for quality, they gradually became the bottleneck for changes, as adherence to quality standards inevitably slows down deliveries. Thus, over time, parts of the Build/Release engineers' functionality have shifted onto the shoulders of system administrators.

Different types of Ops

As we move forward, a wide array of responsibilities and the lack of qualified personnel drive us towards rigid specialization, leading to the emergence of various Operations roles like mushrooms after rain:

• TechOps — system administrators known as HelpDesk Engineers
• LiveOps — system administrators primarily responsible for production environments
• CloudOps — system administrators specializing in public clouds like Azure, AWS, GCP, etc.
• PlatOps/InfraOps/SysOps — infrastructure system administrators.
• NetOps — network administrators
• SecOps — system administrators specializing in information security — PCI compliance, CIS compliance, patching, etc.

DevOps is a person who understands the entire development cycle — development, testing, the architecture of the product, able to assess security risks, familiar with automation approaches and tools, at least at a high level, and also understands pre-release and post-release product support. This person can act as an advocate for both Operations and Development, facilitating favorable cooperation between these two pillars. They comprehend the processes of work planning by teams and managing customer expectations.

To perform such tasks and responsibilities, this person must have management tools not only for development and testing processes but also for managing the product infrastructure and planning resources. In this context, DevOps cannot be confined to IT, R&D, or even PMO; they must have influence across all these areas — a company's technical director, Chief Technical Officer.

Is this the case in your company? — I doubt it. In most cases, it's either IT or R&D.

A lack of resources and the ability to influence at least one of these three areas will shift the weight of problems toward where changes can be more easily applied, such as imposing technical constraints on releases due to 'dirty' code according to static analysis tools. This means that when PMO sets a hard deadline for delivering functionality, R&D cannot provide a quality result in that time frame and delivers whatever it can, leaving refactoring for later; DevOps related to IT may block the release with technical means. A lack of authority to change the situation leads to hyper-responsibility among responsible employees for matters they cannot influence, especially if these employees understand and see the mistakes and how to correct them — 'Ignorance is bliss,' leading to burnout and loss of these employees.

The DevOps Resource Market

Let's look at several DevOps job vacancies from different companies.

We are ready to meet with you if you:

1. Are proficient in Zabbix and know what Prometheus is;
2. Understand Iptables;
3. Are a BASH aspirant;
4. Are a Professor of Ansible;
5. Are a Linux Guru;
6. Are you skilled at debugging and identifying application issues (php/java/python) in collaboration with developers?
7. Routing doesn’t make you panic;
8. You pay significant attention to system security;
9. You back up 'everything and anything' and are successful in restoring this 'everything and anything';
10. You know how to configure the system to get the most out of the least;
11. You set up replication before bed on Postgres and MySQL;
12. Configuring and adjusting CI/CD is as essential for you as breakfast/lunch/dinner.
13. You have experience working with AWS;
14. You are ready to grow alongside the company;

So:

• 1-6 — System Administrator
• 7 — a bit of network administration, which also fits within the scope of a Middle SysAdmin
• 8 — a bit of security, which is essential for a Middle SysAdmin
• 9-11 — Middle System Administrator
• 12 — Depending on the tasks assigned, either a Middle System Administrator or a Build Engineer
• 13 — Virtualization — Middle System Administrator or what is known as CloudOps, with advanced knowledge specifically of services on a given platform, for effective financial management and reduced maintenance load hosting To summarize this vacancy, it can be said that candidates should ideally be a Middle/Senior System Administrator.

By the way, it’s not worth overly distinguishing between Linux/Windows admins. I understand that the services and systems of these two worlds differ, but the foundation for both is the same, and any self-respecting admin is familiar with both, and even if not, a competent admin can easily familiarize themselves with it.

Let's consider another vacancy:

Experience in building high-load systems;

1. Excellent knowledge of Linux OS, general system software, and web stack (Nginx, PHP/Python, HAProxy, MySQL/PostgreSQL, Memcached, Redis, RabbitMQ, ELK);
2. Experience with virtualization systems (
3. , VMWare, LXC/Docker);KVMProficiency in scripting languages;
4. Understanding of network protocols and how networks operate;
5. Understanding the principles of building fault-tolerant systems;
6. Independence and initiative;
7. Breaking down:

1 — Senior System Administrator

• 2 — Depending on the meaning behind this stack — Middle/Senior System Administrator
• 3 — Work experience can also mean — 'I haven't set up a cluster, but I created and managed VMs, there was one Docker host, and I routed access to containers' — Middle System Administrator
• 3 — Work experience can include aspects such as, 'I did not lift the cluster, but I created and managed virtual machines, had a single Docker host, set up access to the containers' — Middle System Administrator
• 4 — Junior System Administrator — yes, an admin who cannot write basic automation scripts, regardless of the language, is not an admin — just a helpdesk technician.
• 5 — Middle System Administrator
• 6 — Senior System Administrator

In summary — Middle/Senior System Administrator

One more thing:

1. Experience in DevOps;
2. Experience with one or more products for CI/CD process implementation. GitLab CI will be an advantage;
3. Working with containers and virtualization; If you have used Docker – great, but if you have experience with Kubernetes – even better!
4. Experience working in an agile team;
5. Knowledge of any programming language;

Let's see:

• 1 — Hmm... What do they mean? =) Most likely, they themselves do not know what is behind this.
• 2 — Build Engineer
• 3 — Middle System Administrator
• 4 — Soft skills, we won’t consider just yet, although Agile is another thing that people interpret as they like.
• 5 — Too vague — this could be a scripting language or a compiled one. Interestingly, would they be satisfied if I wrote in Pascal and Basic back in school? =)

I would also like to add a remark regarding point 3, to strengthen the understanding of why this point is covered by the system administrator. Kubernetes is merely orchestration, a tool that wraps direct commands to networking drivers and virtualization/isolation hosts in a few commands and allows for abstract communication with them, that’s all. For example, take the 'build framework' Make, which I don’t consider a framework, by the way. Yes, I know about the trend to shove Make everywhere, where it’s needed and not needed — like wrapping Maven in Make for example, seriously?
Essentially, Make is just a wrapper over shell, simplifying compilation, linking, and compilation environment commands, just like Kubernetes.

Once, I interviewed a guy who used Kubernetes in his work on top of OpenStack, and he talked about how he deployed services on it, however, when I specifically asked about OpenStack, it turned out that it is administered and set up by system administrators. Do you really think that a person who has set up OpenStack, regardless of which platform he uses behind it, is not capable of using Kubernetes? =)
This candidate is actually not a DevOps but a System Administrator, and to be more precise, a Kubernetes Administrator.

Let’s summarize once again — Middle/Senior System Administrator will be sufficient for them.

How much to weigh in grams

The range of proposed salaries for the specified vacancies is 90k-200k
Now, I would like to draw a parallel between the monetary rewards of System Administrators and DevOps Engineers.

In principle, for simplification, experience-based grades can be classified, although this will not be precise; for the purposes of this article, it will suffice.

Experience:

1. up to 3 years — Junior
2. up to 6 years — Middle
3. more than 6 years — Senior

The job search website offers:
System Administrators:

1. Junior — 2 years — 50k rub.
2. Middle — 5 years — 70k rub.
3. Senior — 11 years — 100k rub.

DevOps Engineers:

1. Junior — 2 years — 100k rub.
2. Middle — 3 years — 160k rub.
3. Senior — 6 years — 220k rub.

For the experience of DevOps, any experience at least touching SDLC was used.

From the above, it follows that companies do not actually need DevOps, and that they could save at least 50 percent of the originally planned costs by hiring an Administrator. Moreover, they could more clearly define the responsibilities of the desired candidate and faster fulfill the need. It is also important to remember that a clear division of responsibilities allows lowering personnel requirements and creates a more favorable atmosphere in the team due to the absence of overlaps. In the overwhelming majority of job postings, there are utilities and DevOps labels, however, they do not actually have true requirements for a DevOps Engineer, only requests for a tooling administrator.

The process of training DevOps engineers is also limited to a set of specific tasks and utilities, not providing a general understanding of processes and their dependencies. It is good when a person can deploy AWS EKS using Terraform, in conjunction with a Fluentd sidecar in this cluster, and the AWS ELK stack for logging within 10 minutes, using just one command in the console. However, if they do not understand the principle of log processing and their purpose, do not know how to collect metrics from them and track service degradation, they will just be another handyman who knows how to use certain utilities.

Demand, however, creates supply, and we see an extremely overheated market for the DevOps position, where requirements do not match the actual role but only allow system administrators to earn more.

So, who are they? DevOps or greedy system administrators? =)

How to move forward?

Employers should formulate requirements more precisely and look for exactly the people they need instead of scattering labels. If you don't know what DevOps do, then you don't need them in that case.

Employees should learn. Continuously improve their knowledge, look at the overall picture of processes, and track the path to the set goal. You can become whoever you want, you just need to try.

Source: habr.com