At the time of writing this article, a search on a popular job site for the phrase "Network Engineer" yielded around three hundred job listings across Russia. In comparison, a search for the phrase "System Administrator" returns nearly 2,500 listings, while "DevOps Engineer" shows almost 800.
Does this mean that network specialists are no longer needed in the age of cloud computing, Docker, Kubernetes, and ubiquitous public Wi-Fi?
Let's dig deeper (c)
Let's get acquainted. My name is Alexey, and I am a network specialist.
I have been working in networks for over 10 years and have over 15 years of experience with various *nix systems (I've dabbled in both Linux and FreeBSD). I have worked at communication operators, large companies often regarded as 'enterprise', and most recently in a 'young and daring' fintech company, where clouds, DevOps, Kubernetes, and other intimidating terms will surely render me and my colleagues obsolete someday. Perhaps.
disclaimer: "In our lives, not everything is always and everywhere, and some things happen sometimes and in certain places" (c) Maxim Dorofeyev.
Everything written below can and should be considered the personal opinion of the author, not claiming to be the absolute truth or a comprehensive study. All characters are fictional, and any resemblance to real persons is purely coincidental.
Welcome to my world.
Where can network specialists be found?
1. Communication operators, service companies, and other integrators. It's simple: networks are their business. They either sell connectivity directly (operators) or provide services for launching/maintaining networks for their clients.
There is plenty of experience here, but not much money (unless you are a director or a successful sales manager). Nevertheless, if you like networking and are just starting out, a career in support at a small to mid-sized operator would, even now, be an ideal starting point (everything is highly scripted in federal companies, with little room for creativity). Stories of moving from a support engineer to a C-level manager in just a few years are also quite real, though rare for obvious reasons. There is always a demand for personnel because turnover does exist. This is both good and bad at the same time — there are always job openings, but on the other hand, the most active/smart ones tend to leave quickly, either for promotions or for other, more 'comfortable' positions.
2. Conditional 'enterprise'It doesn't matter whether their main activity is related to IT or not. What matters is that they have their own IT department that ensures the operation of the company's internal systems, including the network in the offices, communication channels to branches, etc. In such companies, the functions of a network engineer may be performed by a system administrator (if the network infrastructure is small, or it is managed by an external contractor), while a network specialist, if present, may also oversee telephony and SAN (of course). Salaries vary widely — it heavily depends on the business's profitability, company size, and structure. I have worked with companies where the Cisco devices were regularly 'loaded in barrels' and with companies where the network was built from poop, sticks, and blue duct tape, and servers were hardly ever updated (needless to say, there were no backups either). The experience here is much less, and it will almost certainly involve dealing with harsh vendor lock or 'how to do something out of nothing'. Personally, I found it incredibly boring, though many enjoy it — everything is quite measured and predictable (if we're talking about large companies), 'dorahka-bakhato', etc. At least once a year, some major vendor announces that they have created another mega-super-duper system that will automate everything, and all sysadmins and network engineers can be laid off, leaving a couple of people to click buttons in a beautiful interface. The reality is that, even if we abstract from the cost of solutions, network engineers aren't going anywhere. Yes, perhaps instead of a console, there will be a web interface again (but now not for a specific piece of hardware, but for a large system that manages dozens and hundreds of such devices), but the knowledge of 'how everything is arranged internally' will still be needed.
3. Product-Based Companies, whose profit comes from the development (and often, operation) of some kind of software or platform — the actual product. Usually, they are small and agile, far from the scale of enterprises and their bureaucratization. This is where those devops, kubernetes, docker, and other scary words can be found in large numbers, which will certainly make networks and network engineers an unnecessary remnant.
What is the difference between a network engineer and a system administrator?
To non-IT people, it's nothing. Both look at a black screen and type some spells, sometimes quietly swearing.
To programmers, it is only a matter of domain. System administrators manage servers, network engineers manage switches and routers. Sometimes they manage poorly, and everything crashes. In any strange cases, network engineers are also to blame. Just because fuck you, that’s why.
In reality, the main difference is the approach to work. It seems that among network engineers, the "If it works, don’t touch it!" mentality is the most common. Usually, you can only do a certain task (within a single vendor) in one way, and the entire configuration is right there, in your hands. The cost of a mistake is high, and sometimes very high (for example, you might have to travel hundreds of kilometers to reboot a router while several thousand people are out of service — a quite common situation for a telecom operator).
In my opinion, this is why network engineers are extremely motivated to maintain network stability (and changes are the main enemy of stability), and their knowledge tends to go deeper rather than broader (you don't need to configure dozens of different demons, but rather know technologies and their implementation by specific hardware manufacturers). This is why a sysadmin who googles how to set a VLAN in Cisco is not yet a network engineer. And he is unlikely to be able to effectively support (or troubleshoot) even a somewhat complex network.
But why would you need a network engineer if you have a host provider?
For an additional fee (and if you are a very large and favored client — maybe even for free, "out of friendship") data center engineers will configure your switches to meet your needs, and possibly even help set up BGP peering with providers (if you have your own subnet for announcements). IP addresses for the announcement).
The main issue is that a data center is not your IT department; it is a separate company whose goal is to make a profit, including from you as a client. The data center provides racks, powers them, cools them, and gives some 'default' connectivity to the internet. Based on this infrastructure, the data center can host your equipment (colocation), lease you a server (dedicated server), or provide managed services (for example, OpenStack or K8s). However, managing client infrastructure is usually not part of a data center's business model because this process is quite labor-intensive, poorly automated (whereas in a proper data center, everything possible is automated), even less standardized (each client is unique), and can lead to complaints ('You set up my server, and now it’s down; you are to blame!!!111'). Therefore, if a host does help you, they will likely try to keep it as simple and straightforward as possible. Doing complex tasks is not cost-effective, at least from the perspective of the engineers’ labor costs at that host (but situations can vary; see the disclaimer). This does not mean that the host will necessarily do everything poorly. However, it is not guaranteed that they will provide exactly what you actually needed.
It may seem quite obvious, but I have encountered several times in my practice that companies began to rely on their hosting provider a little more than they should, and this did not lead to anything good. I had to explain in detail that no SLA covers losses from downtime (there are exceptions, but generally, this is very, VERY costly for the client) and that the host is not informed about what is happening in the clients' infrastructure (other than very general metrics). The host also does not make backups on your behalf. The situation is even worse if you have more than one host. In case of problems between them, they definitely will not figure out for you what went wrong.
The motives here are exactly the same as when choosing between "in-house admin team vs outsourcing". If risks are assessed, quality is satisfactory, and the business is on board—why not give it a try? On the other hand, the network is one of the most fundamental layers of infrastructure, and it's hardly wise to hand it over to outsiders if you already manage everything else yourself.
When is a network engineer needed?
Next, we will talk specifically about modern product companies. With operators and enterprises, everything is more or less clear—there hasn't been much change in recent years, and network engineers have always been needed there, both in the past and now. However, with the so-called 'young and bold', it's not so straightforward. Often, they host their entire infrastructure in the cloud, so they don't necessarily need admins—other than the cloud admins, of course. The infrastructure is, on the one hand, quite simple in its structure, but on the other hand, it is well automated (Ansible/Puppet, Terraform, CI/CD... well, you know). Yet even here, there are situations where a network engineer is indispensable.
Example 1, classic
Suppose a company starts with one server that has a public IP address located in a data center. Then the servers become two. Then more... Eventually, the need arises for a private network between the servers. Because external traffic is limited both in bandwidth (no more than 100Mbps, for example) and in the volume of data downloaded/uploaded per month (different hosts have various rates, but bandwidth to the outside world is generally much more expensive than a private network).
The host adds additional network cards to the servers and connects them to their switches in a separate VLAN. A 'flat' local area network emerges between the servers. Convenient!
The number of servers is growing, and so is the traffic in the private network — backups, replications, etc. The host offers to separate you into individual switches so that you don't interfere with other clients and they don't interfere with you. The host sets up certain switches and configures them in such a way — most likely leaving a flat network between all your servers. Everything works well, but at a certain point, problems begin: delays between hosts occasionally increase, there are warnings in the logs about too many ARP packets per second, and a pentester gained access to your entire local network by compromising just one server during an audit.
What needs to be done?
Divide the network into segments — VLANs. Set up distinct addressing in each VLAN, assign a gateway that will route traffic between networks. On the gateway, configure ACLs to restrict access between segments, or even place a separate firewall next to it.
Example 1, continuation
Servers are connected to the local network with a single cable. The switches in the racks are somehow interconnected, but if there's a failure in one rack, three neighboring ones also go down. Schemas exist, but their relevance is questionable. Each server has its own public address, assigned by the host and linked to the rack. Thus, when moving a server, the address must be changed.
What needs to be done?
Connect the servers using LAG (Link Aggregation Group) with two cables to the switches in the rack (these also need to be redundant). Reserve the connections between racks, redesign to a 'star' topology (or the trendy CLOS) so that the failure of one rack does not affect others. Identify 'central' racks where the network core will be located, and to which other racks will connect. At the same time, tidy up the public addressing, obtain a subnet from the host (or from RIR, if possible) that can be announced to the world either independently or through the host.
Can all this be done by an 'ordinary' system administrator without deep knowledge of networks? I'm not sure. Will the host do it? They might, but you will need to provide a fairly detailed technical specification, which someone will also need to draft and later ensure everything is done correctly.
Example 2. Cloud
Suppose you have a VPC in a public cloud. To gain access from your office or an on-premises part of the infrastructure to the local network inside the VPC, you need to set up a connection via IPSec or a dedicated channel. On one hand, IPSec is cheaper because you don't need to purchase additional hardware; you can configure a tunnel between your server with a public address and the cloud. However, there are latency issues, limited performance (since the channel needs to be encrypted), and uncertain connectivity (as access goes through the regular internet).
What needs to be done?
Establish a connection through a dedicated channel (for example, AWS calls this Direct Connect). To do this, find a partner operator who will connect you, determine the nearest access point (both for you to the operator and the operator to the cloud), and finally set everything up. Can all this be done without a network engineer? Certainly, yes. However, troubleshooting if issues arise without one is not so clear.
There may also be availability issues between clouds (if you have a multi-cloud setup) or latency issues between different regions, etc. Certainly, many tools have emerged that improve visibility into what is happening in the cloud (such as Thousand Eyes), but these are all tools for network engineers, not a replacement for them.
I could list a dozen such examples from my experience, but I think it's clear that in a team, starting from a certain level of infrastructure development, there should be someone (or even better, more than one person) who understands how the network works, can configure networking equipment, and troubleshoot issues if they arise. Trust me, they will have plenty to do.
What should a network engineer know?
It is not at all necessary (and sometimes, even harmful) for a network engineer to focus only on networking and nothing else. Even if we don't consider the option of infrastructure that almost entirely lives in the public cloud (which is, whether we like it or not, becoming more and more popular), and take, for example, on-premises or private clouds, where mere 'CCNP-level knowledge' is not enough.
In addition to networking itself — where there is an endless field for study, even if one focuses only on a specific direction (provider networks, enterprise, data centers, Wi-Fi...)
Of course, many of you will recall Python and other "network automation" tools, but that's just a necessary, not sufficient condition. For a network engineer to "successfully integrate into the team," they need to be able to communicate in the same language with both developers and fellow admins/devOps. What does that mean?
- They should not only know how to work in Linux as a user but also be able to administer it, at least at a junior sysadmin level: installing necessary software, restarting a crashed service, and writing a simple systemd unit.
- They need to understand (at least in general terms) how the network stack works in Linux, how networking is structured in hypervisors and containers (lxc/docker/kubernetes).
- Of course, they should be able to work with ansible/chef/puppet or another SCM system.
- It is also necessary to mention SDN and networks for private clouds (for example, TungstenFabric or OpenvSwitch). This is yet another huge area of knowledge.
In brief, I’ve described a typical T-shaped specialist (as it is fashionable to say nowadays). It may not seem new, however, based on my interview experience, not all network engineers can boast knowledge in at least two of the topics listed above. In practice, the lack of knowledge in "related fields" greatly complicates not only communication with colleagues but also understanding the requirements that the business places on the network as the most fundamental infrastructure of the project. Without this understanding, it becomes harder to convincingly advocate your viewpoint and 'sell' it to the business.
On the other hand, the habit of "understanding how the system works" gives network professionals a significant advantage over various "general specialists" who know about technologies from articles on Habr/Medium and chats in Telegram, but have no real grasp of the principles behind particular software. And as we know, understanding certain patterns often successfully replaces the need to know numerous facts.
Conclusions, or simply TL;DR
- A network administrator (like a DBA or VoIP engineer) is a specialist in a fairly narrow field (unlike sysadmins/devops/SREs), and the need for such an expert does not arise immediately (and may not arise for a long time, in fact). However, when the need does arise, it is unlikely that their expertise can be substituted by external support (outsourcing or regular admins who also "oversee the network"). What is somewhat more unfortunate is that the demand for such specialists is low; for instance, in a company with 800 programmers and 30 devops/admins, there may be only two network engineers who manage their responsibilities excellently. That is, the market has been and remains quite small, and the opportunities for a good salary are even more limited.
- On the other hand, a good network engineer in the modern world must not only understand networks (and how to automate their configuration) but also know how operating systems and software interact with those networks. Without this knowledge, it will be extremely difficult to understand what your colleagues are asking of you and to convey your wishes/requirements to them (justifiably).
- There is no cloud, it’s just someone else’s computer. It’s important to understand that using public/private clouds or services from a hosting provider "that does everything turnkey for you" does not eliminate the fact that your application still relies on a network, and any issues with it will affect your application’s performance. Your choice is where the center of competence will be located, which will be responsible for the network of your project.
Source: habr.com
