At the moment, this is almost the most expensive position on the market. The fuss around "DevOps" engineers is beyond all imaginable limits, and even worse with Senior DevOps engineers.
I work as the head of the integration and automation department, guess the English transcript - DevOps Manager. Whether it reflects the English transcript of our daily activities is unlikely, but the Russian version in this case is more accurate. By the nature of my activity, it is natural that I need to interview future members of my team and, over the past year, 50 people passed through me, and the same number were cut off on the screen with my employees.
We are still looking for colleagues, because behind the DevOps label there is a very large layer of various kinds of engineers.
Everything written below is my personal opinion, you are not obliged to agree with it, but I admit that it will add a shade to your attitude to the topic. Despite the risk of falling out of favor, I am posting my opinion because I think there is a place for it.
Companies have different understandings of what DevOps engineers are, and for the sake of quickly hiring a resource, they hang this label on everyone. The situation is quite strange, since companies are ready to pay unrealistic rewards to these people, receiving, in most cases, an admin-toolzist for them.
So who are DevOps engineers?
Let's start with the history of the emergence - Development Operations appeared as another step towards optimizing the interaction in small teams to increase the speed of product production, as an expected consequence. The idea was to strengthen the development team with knowledge of the procedures and approaches in managing the product environment. In other words, the developer must understand and know how his product works in certain conditions, must understand how to deploy his product, what characteristics of the environment to tweak in order to increase performance. So, for some time, there have been developers with a DevOps approach. DevOps developers wrote assembly and packaging scripts to simplify their activities and the performance of a productive environment. However, the complexity of the solution architecture and the mutual influence of the infrastructure components over time began to worsen the performance of the environments, with each iteration, an ever deeper understanding of certain components was required, reducing the productivity of the developer himself due to the additional costs of understanding the components and tuning systems for a specific task. . The developer’s own value grew, the cost of the product along with it, the requirements for new developers in the team jumped sharply, because they also had to cover the duties of the development “star” and, naturally, the “stars” became less and less available. It is also worth noting that, in my experience, few developers are interested in the specifics of packet processing by the operating system kernel, packet routing rules, and host security aspects. The logical step was to attract an administrator who is familiar with this particular format and assign duties of this format to him, which, thanks to his experience, made it possible to achieve the same indicators at a lower cost compared to the cost of a development “star”. Such administrators were placed in a team and their main task was to manage test and production environments, on the rules of a particular team, with resources allocated to this particular team. So, in fact, DevOps appeared in the view of the majority.
Partially or completely, over time, these system administrators began to understand the needs of this particular team in the field of development, how to make life easier for developers and testers, how to roll out an update and not stay overnight at the office on Friday, fixing deployment errors. Time passed, now system administrators became "stars", understanding what developers want. In order to minimize the impact, management utilities began to pull up, everyone remembered the old and reliable methods of isolating the OS level, which made it possible to minimize the requirements for security, network part management, and the host configuration as a whole and, as a result, reduce the requirements for new “stars”.
A “wonderful” thing appeared - docker. Why wonderful? Yes, only because the creation of isolation in a chroot or jail, as well as OpenVZ, required non-trivial knowledge of the OS, in a counter utility that allows you to simply create an isolated application environment on a certain host with everything you need inside and transfer the reins of control to development again, and the system administrator can only manage only one host, ensuring its security and high availability - a logical simplification. But progress does not stand still and the systems are again becoming more and more complex, there are more and more components, one host no longer meets the needs of the system and it is necessary to build clusters, we are again returning to system administrators who are able to build these systems.
Cycle after cycle, various systems appear that simplify development and / or administration, orchestration systems appear that, exactly as long as you do not need to deviate from the standard process, are easy to use. Microservice architecture also appeared with the aim of simplifying everything described above - fewer interconnections, easier to manage. In my experience, I did not find a fully microservice architecture, I would say 50 to 50 - 50 percent of microservices, black boxes, it came in, it came out processed, the other 50 is a torn monolith, services unable to work separately from other components. All this again imposed restrictions on the level of knowledge of both developers and administrators.
Similar "swings" of the level of expert knowledge of a particular resource continue to this day. But we digress a little, there are many points that are worth highlighting.
Build Engineer/Release Engineer
Highly specialized engineers who emerged as a means of standardizing software build processes and releases. In the process of introducing wholesale Agile, it would seem that they have ceased to be in demand, but this is far from being the case. This specialization appeared as a means of standardizing the assembly and delivery of software on an industrial scale, i.e. using standard techniques for all company products. With the advent of DevOps, developers partially lost their functions, since it was the developers who began to prepare the product for delivery, and given the changing infrastructure and approach to delivering as quickly as possible without regard to quality, over time it turned into a change stopper, since following quality standards inevitably slows down deliveries. So, gradually, part of the functionality of Build / Release engineers migrated to the shoulders of system administrators.
Ops are so different
We move on and again, the presence of a large range of responsibilities and the lack of qualified personnel pushes us to a rigid specialization, like mushrooms after rain, various Operations appear:
- TechOps - enikei system administrators aka HelpDesk Engineer
- LiveOps - system administrators primarily responsible for production environments
- CloudOps - system administrators specializing in public "clouds" Azure, AWS, GCP, etc.
- PlatOps/InfraOps/SysOps - infrastructure system administrators.
- NetOps - Network Administrators
- SecOps - system administrators specializing in information security - PCI compliance, CIS compliance, patching, etc.
DevOps - (in theory) a person who understands firsthand all the processes of the development cycle - development, testing, understands the product architecture, is able to assess security risks, is familiar with automation approaches and tools, at least at a high level, in addition to this, he also understands pre- and post- release support for the product. A person who is able to act as an advocate for both Operations and Development, which allows building a favorable cooperation between these two pillars. Understanding the processes of planning work by teams and managing customer expectations.
To perform this kind of work and duties, this person must have the means to manage not only the development and testing processes, but also the management of the product infrastructure, as well as resource planning. DevOps in this sense cannot be in IT, or in R&D, or even in PMO, it must have influence in all these areas - the company's technical director, Chief Technical Officer.
Is this the case in your company? - I doubt. In most cases, this is either IT or R&D.
The lack of funds and the ability to influence at least one of these three areas of activity will shift the weight of problems towards where these changes are easier to apply, such as the application of technical restrictions on releases in connection with "dirty" code according to static analyzer systems. That is, when PMO sets a strict deadline for the release of functionality, R&D cannot produce a quality result within these deadlines and produces it as best it can, leaving refactoring for later, DevOps related to IT blocks the release by technical means. The lack of authority to change the situation, in the case of responsible employees, leads to the manifestation of hyper-responsibility for what they cannot influence, all the more so if these employees understand and see mistakes, and how to correct them - “Happiness in ignorance”, and as a result to burnout and loss of these employees.
DevOps resource market
Let's look at a few vacancies for a DevOps position from different companies.
We are ready to meet with you if you:
- Own Zabbix and know what Prometheus is;
- iptables;
- PhD student B.A.S.H.;
- Professor Ansible;
- Guru Linux;
- You know how to use debugging and, together with developers, find application problems (php / java / python);
- Routing does not make you hysterical;
- Pay significant attention to system security;
- Backup “everything and everything”, and also successfully restore this “everything and everything”;
- You know how to set up the system in such a way as to squeeze the maximum out of the minimum;
- Set up replication before going to bed on Postgres and MySQL;
- Setting up and adjusting CI/CD for you is a necessity like breakfast/lunch/dinner.
- Do you have experience with AWS?
- Ready to develop together with the company;
So:
- 1 to 6 - system administrator
- 7 - a bit of network administration, which also fits into the system administrator, the Middle level
- 8 - a bit of security, which is a must for a Middle level sysadmin
- 9-11 Middle System Administrator
- 12 - Depending on the tasks, either Middle System Administrator or Build Engineer
- 13 - Virtualization - Middle System Administrator, or the so-called CloudOps, extended knowledge of the services of a particular hosting site, for the efficient use of funds and reducing the load on maintenance
Summarizing this vacancy, we can say that the guys have enough Middle/Senior System Administrator.
By the way, you should not strongly divide administrators into Linux / Windows. Of course, I understand that the services and systems of these two worlds differ, but they all have the same basis and any self-respecting admin is familiar with both one and the other, and even if he is not familiar, it will not be difficult for a competent admin to get acquainted with this.
Consider another job:
- Experience in building high-load systems;
- Excellent knowledge of Linux OS, general system software and web stack (Nginx, PHP/Python, HAProxy, MySQL/PostgreSQL, Memcached, Redis, RabbitMQ, ELK);
- Experience with virtualization systems (KVM, VMWare, LXC/Docker);
- Knowledge of scripting languages;
- Understanding the principles of operation of networks of network protocols;
- Understanding the principles of building fault-tolerant systems;
- Independence and initiative;
Parsing:
- 1 - Senior System Administrator
- 2 - Depending on the meaning invested in this stack - Middle / Senior System Administrator
- 3 - Experience, including, may mean - "The cluster did not raise, but created and managed virtual machines, there was one Docker host, access to containers was natal" - Middle System Administrator
- 4 - Junior System Administrator - yes, an admin who cannot write elementary automation scripts, regardless of language, not an admin - enikey.
- 5 - Middle System Administrator
- 6 - Senior System Administrator
In summary - Middle/Senior System Administrator
Another one:
- devops experience;
- Experience in using one or more products to build CI/CD processes. Gitlab CI would be an advantage;
- Working with containers and virtualization; If you used docker - good, but if k8s - great!
- Experience in an agile team;
- Knowledge of any programming language;
We'll see:
- 1 – Hmm… What do the guys mean? =) Most likely they themselves do not know what lies behind this
- 2 - Build Engineer
- 3 - Middle System Administrator
- 4 - Soft skill, we will not consider yet, although Agile is another thing that is interpreted in a convenient way.
- 5 - Too lengthy - it can be a scripting language, or a compiled one. Interestingly, did writing at school in Pascal and Basic suit them? =)
I would also like to leave a remark regarding point 3 in order to strengthen the understanding of why this point is covered by the system administrator. Kubernetes is just orchestration, a tool that wraps direct commands to network drivers and virtualization / isolation hosts into a couple of commands and allows you to make communication with them abstract, that's all. For example, let's take 'build framework' Make, which, by the way, I don't consider to be a framework. Yes, I know about the fashion to shove Make anywhere, where it is necessary and not necessary - wrap Maven in Make, for example, seriously?
In fact, Make is just a shell wrapper that simplifies compilation commands, linking commands, compilation environments, just like k8s.
Once, I interviewed a guy who used k8s in his work on top of OpenStack, and he told how he deployed services on it, however, when I asked specifically about OpenStack, it turned out that he was administered, as well as raised by system administrators. Do you really think that a person who has raised OpenStack, no matter what platform he uses behind him, is not able to use k8s? =)
This applicant is not actually DevOps, but the same System Administrator and, to be more precise, Kubernetes Administrator.
We summarize once again - Middle / Senior System Administrator will be enough for them.
How much to hang in grams
The range of salaries offered for the specified vacancies is 90k-200k
Now I would like to draw a parallel between the monetary rewards of System Administrators and DevOps Engineers.
In principle, to simplify, you can scatter grades based on work experience, although this will not be accurate, it will be enough for the purposes of the article.
An experience:
- up to 3 years old – Junior
- up to 6 years — Middle
- more than 6 – Senior
The recruiting site offers:
System Administrators:
- Junior - 2 years - 50k rubles.
- Middle - 5 years - 70k rubles.
- Senior - 11 years old - 100k rubles.
DevOps Engineers:
- Junior - 2 years - 100k rubles.
- Middle - 3 years - 160k rubles.
- Senior - 6 years old - 220k rubles.
According to the experience of "DevOps"'s, the experience was used, at least somehow affecting the SDLC.
It follows from the above that companies do not really need DevOps, and also that they could save at least 50 percent of their originally planned costs by hiring the Administrator, moreover, they could clearly define the responsibilities of the person they were looking for and close the need faster. Do not forget that a clear division of responsibility allows you to reduce the requirements for staff, as well as create a more favorable atmosphere in the team, due to the absence of intersections. The vast majority of vacancies are full of utilities and DevOps labels, however, they do not really have requirements for a DevOps Engineer, only requests for a tool administrator.
The process of training DevOps engineers is also limited to a set of specific jobs, utilities, and does not provide a general understanding of processes and their dependencies. It's certainly good when a person can deploy AWS EKS using Terraform, in conjunction with the Fluentd sidecar in this cluster and the AWS ELK stack for the logging system in 10 minutes, using only one command in the console, but if he does not understand the processing principle itself logs and what they are for, not knowing how to collect metrics on them and track the degradation of the service, then it will be the same enikey that can use some utilities.
Demand, however, breeds supply, and we see a very overheated market for the DevOps position, where the requirements do not correspond to the real role, but only allow system administrators to earn more.
So who are they? DevOps or greedy sysadmins? =)
How to live further?
Employers should formulate requirements more precisely and look for exactly those who are needed, and not scatter labels. You don't know what DevOps do - you don't need them in this case.
Employees - Learn. Constantly improve your knowledge, look at the overall picture of the processes and track the path to the goal. You can be whoever you want, you just have to try.
Source: habr.com