Emergency mode (also referred to as IPKVM), allowing connection to VPS without RDP directly from the hypervisor level, saves 15–20 minutes a week.
First and foremost — do not annoy people. Support worldwide is divided into tiers, and the first-level employee must try typical solutions. If the issue surpasses their capabilities — it gets escalated to the second level. Among VDS administrators, there are frequently individuals who can think critically, unlike many other support teams. At least, this happens significantly more often. They structure tickets well, clearly detailing everything needed. If the first level 'loses focus' and mistakenly asks to turn something on and off in response to a detailed inquiry — that's a failure.
The task is very simple: to provide adequate support for our VDS hosting with minimal costs. Because we are the fast food of hosting providers: no special 'polishing', low prices, decent quality. There was already a story about how with the emergence of Instagram influencers trying to automate account management and small business owners with remote accounting and other people not too tech-savvy, communication 'as an admin with an admin' stopped working. We had to change the communication style.
Now I'll talk a bit more about the processes — and the inevitable blunders that come with them.
Do not annoy people #1
Any support is a conveyor process. A request comes in, and the first-level employee immediately tries to identify a typical situation that has occurred a thousand times before and will occur a thousand times more. There is a 90% chance that the request is standard, and it can be answered by literally pressing a couple of buttons to apply a template. Usually, you'll just need to fill in a couple of words in the template — and it's done. Or access the management interface and click a few buttons. In more complex cases (such as transfers between zones), specific actions according to an algorithm must be carried out.
What annoys people the most, regardless of the support quality, is the typical reaction to an atypical request. A ticket arrives detailing everything, with plenty of necessary data for three questions ahead; the client anticipates a dialogue... And from the first words, the support staff, on autopilot, starts typing the template response: 'try rebooting, it should help.'
This is exactly what drives people crazy, and it’s after such situations that the most negative reviews and angry comments are left. It’s clear that we made mistakes, which is why we know the statistics. We’ve made different mistakes, but cases like these are always just wild. Including for ourselves. We would, of course, like to eliminate these occurrences entirely. But that’s not very feasible in practice: every few weeks, a tired employee, weary of the monotony, may accidentally press the cheerful buttons.
Annoying People, Part 2
The second thing that equally drives people crazy is when no one responds to a ticket for a long time. In Europe, such support behavior is normal: three days before an incident is taken in for work is more than acceptable. Even if you have something urgent and on fire—there are no social media, no phone, no messenger, just email, and you wait your turn. In Russia, this is much less common, but still, some tickets get 'forgotten.' At the very beginning, we set an SLA for first response at 15 minutes. And this is with a 24/7 honest approach. It’s clear that when VDS hosting becomes large, this comes into play. But questionable service providers don’t have this. And we were just a questionable provider at the start, and only later became somewhat larger. Well, more or less average.
The first line consists of operators who have been given scripts and trained to react to typical situations. They quickly sort out problems and strive to either respond with a typical action within 15 minutes or inform that the ticket is being attended to and pass it to the second line.
The second line comprises hosting administrators who can do almost everything manually. Also, there is a support manager who can do everything and a bit more. The third line includes developers; tickets like 'please fix this in the interface' or 'the parameter is incorrectly accounted for here' go to them.
Reduce the number of requests.
For obvious reasons, if you want to provide support cheaply, you need to enhance automation instead of increasing the first line of support, allowing people with scripts to resolve issues more quickly. The goal is to replace humans with actual scripts. Therefore, one of the first things we did was automate processes such as launching virtual machines and scaling resources (including disk up and down, but not CPU frequency) and similar tasks. The more users can do through the interface, the easier it is for the first line of support, and the fewer personnel are needed. When a user requests something found in their account dashboard, we need to show them how to accomplish it independently.
If you don't need support, it means everything is working well.
Another feature that saves a lot of time is the thorough documentation of the knowledge base. If a user encounters a problem that is not included in the list of supported actions (often questions like 'how to set up a Minecraft server' or 'where to configure VPS in Windows Server'), we write an article for the knowledge base. A similarly detailed article is created for every unique request. For instance, if a user asks support to disable the built-in Windows Server firewall, we direct them to read about the consequences of actually turning it off and how to configure permissions for selected software. Usually, the issue arises because something cannot connect due to settings rather than the firewall itself. However, explaining this repeatedly in dialogues is quite challenging. Disabling the firewall isn't ideal, as we may soon lose either the virtual machine or the client.
If a particular software issue in the knowledge base becomes highly visited, we can add a distribution to the marketplace to offer a service like 'launch a server with this already installed.' This is how Docker became available, and the same applies to the Minecraft server. Again, a single 'make this work for me' button in the interface can save up to a hundred tickets a year.
Emergency mode
After these actions, the most serious failures requiring manual intervention usually occur when the user has lost remote access to the guest OS in the hypervisor for some reason. The most common scenario is simply an incorrect firewall configuration; the second most common is bugs that prevent Windows from starting properly, forcing a reboot into Safe Mode. And in Safe Mode, RDP is not available by default.
We have created a recovery mode for this situation. Generally, to access a VDS machine, some kind of remote client is needed. Most often this involves console access, RDP, VNC, or something similar. The drawback of these methods is that they do not work without an OS. However, at the hypervisor level, we can obtain the screen's image and send keyboard inputs! The downside is that this heavily loads the CPU (due to actual video streaming), but it allows us to achieve the desired result.
Therefore, we provided access to the recovery mode for all users, but it is limited in terms of continuous usage time. Fortunately, as practice shows, this amount of time is usually sufficient to reboot and make necessary adjustments.
The result is even fewer support tickets. And where admins can fix issues themselves in the ticket, support does not need to intervene directly.
Remaining issues
Users often think that support is trying to sell them something. Unfortunately, there is nothing we can do about this (or we haven't found a solution). The two most common examples are resource limits and DDoS protection.
Each virtual machine has limits on disk load, memory, and permissible traffic. The ability to set limits is outlined in the offer; the limits themselves are configured so that the majority of users can operate smoothly without being aware of them. However, if you suddenly start heavily stressing the channel and disk, the algorithms automatically notify the user. Since April of last year, we have removed the auto-blocking feature. Instead, we implement soft limits for variable periods.
It used to be this way: a warning first, then if the user didn’t comply, an automatic block. At that moment, people would get upset: "What are you doing? Your system is glitching, I didn’t receive any notifications!" — and then you could either try to figure out the application software or suggest upgrading their plan. We don’t have the ability to investigate the application software issues because that’s beyond support. Although we handled the first few cases with users together. One case that stood out was where a viewer booster on YouTube had a built-in trojan, and this trojan was leaking memory. In the end, we concluded that these weren’t just random bugs, but real problems faced by users; otherwise, we would have been flooded with similar requests. But not a single person has admitted that they might have exceeded their limits themselves.
A similar story occurs with DDoS: we inform the user that they are under attack. We kindly ask them to activate protection. And the user responds: "You are the ones attacking me!" Of course, we are not actually targeting one user with DDoS just to trick them into paying 300 rubles. That would be quite the profitable scheme. Yes, I know that many larger hosting companies in the more expensive category include this protection in their plans, but we can’t do that: the fast-food economy imposes different minimum prices.
Users are also frequently dissatisfied with the support team when we have deleted their data. This is in the sense that we legitimately deleted it after the paid term expired. If someone doesn’t renew their VDS rental, they receive several notifications explaining what will happen next. At the end of the payment period, the virtual machine is stopped, but its image is preserved. Another notification is sent, followed by a couple more. The image is stored for an additional seven days and only then is permanently deleted. There’s a category of people who are very unhappy about this. Starting from "the admin has left, notifications were sent to his email, restore the data," to accusations of fraud and threats of physical harm. The reason is the same pricing structure applied to all other users. If we were to store the data for a month, it would require more storage space. This would mean higher prices for each individual client. And the fast-food economy... Well, you get the picture. As a result, we receive reviews on forums like "they took my money, deleted my data, fraudsters."
I would like to note that we have a range of premium plans. The situation there, of course, is different, as we take into account client requests and flexibly adjust both the limits and the removal in case of non-payment (we allow it to go into negative, just to avoid blocking). Economically, it makes sense there, as anything can happen, and retaining a large permanent client is costly.
Sometimes users are malicious. We have experienced system failures a few times that resulted in the blocking of hundreds of virtual machines due to clearly illegitimate actions by clients. In fact, it was precisely because of such situations that we needed our own network drivers to monitor network activity and see if a user was launching an attack from their server. Monitoring like this is important to ensure that the boundaries of neighboring virtual machines are not violated by unruly individuals.
There are those who simply spam, mine, or otherwise breach the agreement. Then they contact support and ask what went wrong and why their machine is blocked. If the process in the ticket is named "spammer.exe" in the screenshot, then something is probably going wrong. About once every two weeks, we receive complaints from companies like Sony or Lucasfilm (now Disney) that someone from our virtual machine within our range of IP addresses is distributing pirated films. For this, there’s an immediate block and refund of any remaining balance under the agreement (I remind you that our quantization is per second, so the remainder will always be accurate). To get a refund, one must show an ID: this is a measure against money laundering. For some reason, pirates often write that we have stolen their money instead of showing their ID, forgetting to mention certain circumstances.
Oh, yes. Our best request of the year is: "Can I test the virtual machine on the 30 rubles per month plan for a few days before purchase?".
Summary
The first line sorts tickets and responds with typical actions. The most dissatisfaction arises here. It won't be possible to fix this because the foundation for correction lies in hosting automation, which involves a huge backlog. Yes, we have more than many in the market, but it's still not enough. Therefore, the best thing we can do is to establish monitoring for the first line. Support service monitoring means achieving KPI for the first line. In real time, SLA delays can be viewed: who is messing up, and often why. Requests never get lost due to such alerts. Yes, a ticket might get a response with an unrelated template, but we find this out through feedback.
If a client urgently requests it, a second-line specialist can access the server and do what is necessary for the client (the condition is confirmation by email in which they will provide access details to the server).
We do this very rarely and trust such work only to the best, because we want to ensure that user data will not be compromised. The best are the second line of support.
The first line has a knowledge base that can be referred to for more complex issues.
A feature-rich personal account plus a knowledge base — and we managed to reduce the number of requests to an average of 1-1.5 per year per client.
The second line usually processes complex requests that require manual labor. Interestingly, the more expensive the plan, the fewer such requests per virtual machine. Usually, this is because those who can afford an expensive plan either have specialists on staff or simply don't encounter many problems due to adequate configurations. I still remember that hero who installed a not-so-old Windows Server on a configuration with 256 MB of RAM.
The second line has a set of distributions and a set of automation scripts. Both can be updated as needed.
The second line and personal managers for VIP plans can add notes to the client's profile. If they are a Linux admin — we will note that. This serves as a hint for the first line: the user surely knows that this will not be a shot in the foot, but rather a controlled breakdown.
The third line governs the strangest things. For example, we had a bug where one of the features in the personal account couldn't be accessed in Firefox. A user directly threatened, 'If you don't fix this within 12 hours, I'll write negative reviews on all hosting sites.' It turned out the issue was with a custom adblocker. On the user's side, strangely enough. Often, complicated errors come without details, and they can't reproduce them again. Sometimes we get detectives with screenshots: 'Why have you been fixing this for a month?' — 'We've just been looking for your bug all this time,' 'Oh, I encountered it again today, but I couldn't reproduce it once more'...
You never really know where a screenshot of a support dialogue will end up, and if someone is reaching out to support, then they have a problem. We can improve customer relations. At the very least, we should try.
Yes, we know our support isn't perfect, but I like to believe it combines sufficient speed with adequate quality. And it doesn't raise prices for those who can manage without it.
Source: habr.com
