Hello, Habr! For ten years, I have been maintaining Highload IT systems. I won't discuss the issues of configuring nginx for 1000+ RPS or other technical details in this article. Instead, I'll share observations about the challenges in the processes that arise in the support and operation of such systems.
Monitoring
Technical support doesn’t wait for a ticket saying, "Why... is the site down again?" Support should identify the problem and start resolving it within a minute after the site goes down. But the website is just the tip of the iceberg.Its availability is monitored as one of the first aspects.
What to do when the inventory of an online store stops coming from the ERP system? Or the CRM system, which calculates discounts for customers, stops responding? The site, at a glance, appears to be functioning. Conditioned Zabbix receives a 200 response. The on-call team hasn't received any alerts from monitoring and happily continues watching the first episode of the new season of "Game of Thrones."
Monitoring often only measures memory state, RAM, and CPU load. servers. But for businesses, it’s much more important to ensure product availability on the website. A failure of one virtual machine in the cluster can result in traffic being redirected away and increased load on other servers. The company won’t lose money.
Therefore, in addition to monitoring technical parameters of operating systems on servers, it is crucial to set up business metrics. Metrics that directly affect revenue. Various interactions with external systems (CRM, ERP, etc.). The number of orders over a specific period. Successful and unsuccessful customer logins, and other metrics.
Interaction with external systems.
Any website or mobile application with an annual turnover of over a billion rubles interacts with external systems. From the aforementioned CRMs and ERPs to the transfer of sales data to external Big Data systems for analyzing purchases and suggesting products that the customer will definitely buy (not really). Each of these systems has its own support. Often, communication with these systems can be painful, especially when the issue is global and requires analysis across different systems.
Some systems provide the phone number or Telegram of their admins. In some cases, you need to write emails to managers or visit the bug trackers of these external systems. Even within a large company, different systems often operate under different request tracking systems. Tracking the status of a request can sometimes become impossible. You receive a request in one hypothetical Jira. Then, in the comments of that first Jira, you link to a task in another Jira. In the second Jira, someone has already commented on the request, stating that you need to call the hypothetical admin Andrei to resolve the issue. And so on.
The optimal solution to this problem would be to create a unified communication space, for example in Slack. Inviting all participants involved in the operation of external systems. As well as a single tracker, to avoid duplicating requests. Requests should be tracked in one place, from monitoring alerts to the resolution of bugs in production. You may say this is unrealistic and that historically, we have worked in one tracker while they work in another. Different systems have emerged, and they had their own autonomous IT teams. I agree, and thus the issue must be addressed from the top level at the CIO or product owner level.
Each system you interact with should provide support as a service, with a clear SLA for priority problem resolution. Not just when the hypothetical admin Andrei finds a minute for you.
A bottleneck person
Is there someone on the project (or product) whose vacation sends the management into a panic? It could be a DevOps engineer, an analyst, or a developer. After all, only the DevOps engineer knows which servers have which containers installed, how to restart a container in case of problems, and generally, any complex issue cannot be resolved without him. The analyst is the only one who knows how your complex mechanism works. What data streams go where, under what request parameters to which services, and what responses we will receive.
Who will quickly understand why there are errors in the logs and promptly fix a critical bug in production? Of course, it's that same developer. There are others, but for some reason, only he understands how different modules of the system are structured.
The root of this problem is the lack of documentation.If all the services in your system were documented, it would be possible to resolve issues without an analyst. If a DevOps engineer could spare a couple of days from their packed schedule to document all servers, services, and standard issue resolution instructions, then problems could be tackled in their absence. There's no need to rush to finish your beer on the beach during vacation while searching for Wi-Fi to solve issues.
Competence and responsibility of support staff
In large projects, companies are generous with developer salaries. They hunt for expensive mid-level or senior developers from similar projects. The situation with support is somewhat different. These costs are usually curtailed. Companies hire inexpensive, inexperienced technicians and confidently move forward. This strategy is feasible if it's about a simple brochure site for a factory in Zelenograd.
When it comes to a large online store, every hour of downtime costs more than the monthly salary of a junior admin. Let's take as a starting point an annual revenue of 1 billion rubles. This is the minimum turnover for any online store in the top-100 ranking for 2018. . Dividing this amount by the number of hours in a year gives us more than 100,000 rubles in direct losses. If we exclude nighttime hours, this figure can easily be doubled.
But money isn't everything, right? (No, of course money is important) There are also reputational losses. An hour of downtime for a well-known online store can trigger a wave of comments on social media and publications in specialized media. Conversations among friends at the kitchen table like "Don’t buy anything there; their site is constantly down" are impossible to quantify.
Now let's discuss responsibility. In my experience, there was a case where a duty administrator failed to respond in time to the monitoring system’s alert about the website being down. On a nice summer Friday evening, a well-known online store’s site in Moscow was quietly down. By Saturday morning, the product manager of that site wondered why it wasn’t loading, and there was silence in the support chats and urgent notifications in Slack. This mistake cost us a six-figure sum and significant trouble for the duty admin.
Responsibility is a skill that is hard to develop. A person either has it or does not. Therefore, during interviews, I try to identify its presence through various questions that indirectly show whether the person is used to taking responsibility. If the candidate answers that they chose a university because their parents said so, or they change jobs because their wife said they're earning too little, it's better not to engage with such individuals.
Interaction with the development team
When users encounter minor issues in production, support resolves them independently. They attempt to reproduce the problem, analyze logs, and so on. But what happens when a bug appears in production? In this case, support creates a ticket for the developers, and that’s where things get interesting.
Developers are constantly overloaded. They are focused on creating new features. Fixing bugs from production is not the most engaging task, especially with deadlines looming for the next sprint. And then unpleasant people from support arrive and say: 'Drop everything, we have problems.' The priority of such tasks is minimal, especially when the issue isn’t the most critical and the main functionality of the site is operational, and when the release manager isn't running around with wide eyes insisting: 'We need to include this task in the next release or hotfix immediately.'
Tasks with normal or low priority are carried over from release to release. When asked 'When will the task be completed?', you'll receive responses like: 'Sorry, there are a lot of tasks right now, ask the team leads or release manager.'
Problems in production take higher priority than creating new features. Negative feedback will not be long in coming if users keep encountering bugs. A tarnished reputation is hard to restore.
The questions of interaction between development and support are addressed by DevOps. This abbreviation is often used to refer to a specific person who helps create test environments for development, builds CICD pipelines, and quickly deploys tested code into production. DevOps is an approach to software development where all participants in the process interact closely with one another and help to create and update software products and services faster. I mean analysts, developers, testers, and support.
Support and development are not separate departments with their own goals and tasks in this approach. Development is involved in operation and vice versa. The famous phrase from distributed teams, "The problem isn't on my side," is now less frequently seen in chats, making end users a bit happier.
Source: habr.com
