In general, I want to say right away that the opinion about hell and the fact that many of the two thousandths have service is a value judgment. In fact, of course, they come from Russia. In fact, of course, we are also good, and I will also talk about these spots in the biography. In fact, in recent years, the same support for many has become much better. But still, someone's pedigree pops up here and there.
Let me go over the problems that often directly hurt hosting customers, tell you what is good and bad with us and how it looks in other hostings in Russia and abroad (but there, obviously, I know less about the internals).
The first story is iron. Customers are unrealistically infuriated when a RAID controller has flown or several disks have taken off at once, and support makes it easy to replace. We had one client who was first ricocheted by DDoS on a neighboring VDS in the same server, then two hours later scheduled work with the network adapter began, and then the raid went into rebuild after being turned on and rebooted. We will return to the issue of didos, by the way.
So, you can take cheap “near-household” iron and often repair it, or you can use server hardware - we have Huawei of the corporate line. As far as I know, we and two other players on the Russian market have professional server hardware. Correct me if I'm wrong. This is because at the start we thought that we would live for more than five years and decided to write off the old hardware at least five years after the start of operation. By the way, again, this is how the tariff for 30 rubles for VDS appeared, you know?
The Iron Dilemma
So, we have an enterprise class Huawei. Usually, hosters in Russia have self-assembly, which is bought in wholesale stores with office and home desktops for components, and then assembled and operated using various dendritic methods. This affects the frequency of breakdowns and the cost of services. If everything is more or less obvious with the frequency of breakdowns (the worse the hardware, the higher the chance of downtime), then with the cost of services everything is more interesting. With our cycle of five to six years, it turns out to be cheaper to buy servers and network devices of corporate lines for data centers.
Yes, they are more expensive to buy. Yes, they have a very expensive warranty (we have an extended warranty for all new devices on the next business day, plus an extended warranty for not the most successful series far beyond the time limit). Yes, you need to keep a repair kit on site: we change the same drives, RAID controllers, RAM strips and sometimes power supplies from our own spare parts in all ten data centers. Somewhere there are more spare parts, somewhere less, depending on the objective number and age of the servers there.
When we were just starting the business, we immediately decided to take more reliable hardware. Because there was a case to check: before RUVDS, we were engaged in algorithmic trading and used self-assembled cheap hardware. And it turned out that the difference is really very big. Consumables are bought simply by centners. Naturally, if hosting has such costs or a shorter iron write-off cycle, then the price of tariffs rises. And since prices for more or less identical configurations are more or less fixed throughout the market, something else usually degrades. As a rule, not support, but either the quality of communication, or information security.
Of course, I may be mistaken, but the assessment is this: whoever does not directly indicate a partnership with an iron vendor and a professional line of iron on their website uses the “near-household” one. Perhaps someone is just hiding their cool equipment.
We made cheap (but not the cheapest) Therefore, we carefully considered and continue to consider operating costs. I don’t really understand the models of other companies, but it seems that the point is that they have planning horizons of two or three years, and we just have more. Perhaps we are wrong, and in Russia it is not worth planning so far, but so far, pah-pah, we have won on this and continue to grow as a company.
Location of the data center
Most VDS hosting has one or two locations. We have ten, and not only in Moscow, but also close to major Russian cities (Ekaterinburg, Novosibirsk), which is important for Minecraft and Counter-Strike servers, and there are Switzerland, England and Germany. And at the same time, Russian-speaking support is everywhere.
Why a second location is needed is understandable - services need to be geo-distributed. But why data centers are needed in other countries is a very interesting question.
Firstly, the data center in Switzerland is considered more reliable than the Russian one. This is not an objective assessment, but the opinion of most of our customers. I must say that yes, of course, and there can be epic gouges, as elsewhere, but in general they have much more carefully followed maintenance procedures and a very strong external security perimeter. That is, they should have fewer problems.
Secondly, of course, outside of Russia. It is important for someone to trade closer to the key points where applications are processed. It is important for someone because of their own VPNs (I think at least a third of our servers were bought specifically for organizing VPN tunnels through other jurisdictions). Well, there are people who found mask shows in their data centers in Russia and now they simply prefer not to store data with us. Although, in theory, no one is immune from this either. It's just that the defaults for hitting the data center are different.
I must say right away that some of our commercial data centers are no worse than those in the UK or Switzerland. For example, in the site is almost without jambs (and definitely without serious ones) and complies with Uptime Institute (T3) standards. Well guarded. That is, objectively, it is very good, but among the clients there is somehow a belief that it is safer abroad. And those Russian hosters who do not provide a foreign location do not immediately fit into the needs of the market a little.
Changing the server configuration and billing
We did surveys and studied what is important to customers. It turned out that a very high place is occupied by such parameters as the quantization unit in the tariff and the ability to quickly change the server configuration. We know that somewhere a virtual machine is created manually in one or two hours upon request, the configuration changes in a day upon request for support.
We automated the processes until the median for creating a virtual machine was four minutes, and the average interval from application to launch was 10-11 minutes. This is because some complex applications are still done by hand in about 20 minutes.
Our billing is per second (not hourly or daily). You can create a server, look at it and immediately delete it, saving your money (we ask for a monthly advance payment, but return it if it doesn’t work). Most Russian sites need to separately rent a license for the OS. We have WinServer delivered to all machines for free and included in the tariff (but the desktop version of Windows is not possible).
The server configuration changes in about ten minutes from the interface, both up and down. Two exceptions - down the disk is not always possible automatically (if the space is occupied by something), and when transferring from 2,2 GHz to 3,5 GHz, it is done through a ticket. Manual applications have an SLA for the first response of 15 minutes, processing time of 20-30 minutes (maybe more, depending on the amount of data being copied). In the tariffs, by the way, where we have an HDD, everywhere, in fact, SSD with restrictions up to HDD speeds (it turned out to be cheaper, and we switched completely to SSD about a year and a half ago). You can take a car with a video card. There is a recycling rate (there is a complex formula from the processor, RAM, disks and traffic) - if you have peak computing, it's cheaper, but there are also customers who do not fully predict their consumption correctly and pay twice the usual rate sometimes. Well, someone saves.
Yes, it all requires the cost of automation. But as practice shows, this also allows you to save a lot on support and retain customers due to the quality of service.
The negative point is that sometimes we advise you to take 10 GB more for certain software. Or sometimes, in correspondence with a client, we understand what kind of software he has and see that there is simply not enough RAM or processor cores and we advise you to buy it, but many people think that this is some kind of wiring from support.
Marketplaces
Overseas, there has been a tendency to provide not only VDS, but also a set of pre-installed software at once. In one form or another All major hosting companies have it and are often missing from smaller ones. Our providers still often sell empty cars, as in Europe.
The first candidate for the marketplace after WinServer was . Our technical specialists immediately said that the marketplace is not needed, because the admins are not so handless. Installing Docker is a couple of minutes, and you shouldn’t consider them so lazy that they won’t do it. But we deployed the marketplace and put Docker there. And they began to use, because laziness. Saves time! Small, but saves. This is not a vital necessity for customers, of course, but already the next standard of the market.
On the other hand, we do not have the same Kuber. But recently appeared . He is more in demand. There are interesting directions for VPS with preinstalled software: there is a configuration with a stripped-down Win (so that it does not eat performance), there is a configuration with OTRS already preinstalled. We provide pre-installed software, and how you activate it is up to you, we do not see this.
The coolest marketplaces in the world, in my opinion, are Amazon, Digital Ocean and Vultr. Startups want to come to the Amazon marketplace: if you made some kind of tool like Elasticsearch, but didn’t get into the marketplace, no one will know, no one will buy it. And if it hit, then a distribution channel appeared.
DDoS
Every hosting is attacked. These are usually weak non-targeted attacks that are similar to the natural microflora of the Internet. But when they start to put some particular client, problems begin for those neighboring him on the same “branch”. As a rule, these are those who are served from the same network device.
More than 99% of clients do not experience problems, but some are not lucky. This is a common reason why customers do not like us - due to server downtime due to DDoS on a neighbor. We tried to minimize these stories for a very long time, but, of course, we could not avoid them completely. We cannot include DDoS protection in the cost of the tariff for everyone in a row, then the services will rise in price on the lower lines by about two times. When support recommends that a client take protection under DDoS (paid, of course), the client sometimes thinks that we put it on purpose in order to sell something. And, most importantly, there is no way to explain, but the neighbors suffer. As a result, we had to get deeper into the stuffing of network adapters and write our own drivers for them. It is the driver for the hardware, yes, you heard right. The second circuit - there is a double protection system that can switch routes in minutes. If you get into antiphase of checks, you can get a maximum of four minutes of downtime. Right now, switching still creates some problems in virtual switches and switches, we are finishing the stack.
Support
Russian support is one of the best in the world. I'm serious now. The fact is that many large European VDS hosting simply do not bother to take on many issues. The situation when someone works only in the mode of answering letters is ubiquitous. Even constantly appearing small Russian hosting of two or three people usually have either a chat on the site, or a phone, or the ability to knock on the messenger. And in Europe, large hosting companies have support for several days (especially if the application is before the weekend) considers the ticket, and it is unrealistic to call or write to them on the social network.
Our clients, by the way, choose locations in their cities, as our support jokes, in order to also fill their faces on occasion. In fact, several people stopped by on their way home to the office.
And here it is time to start talking about our epic jambs.
Our shoals
The smallest is disk crashes, RAM and controller raids. It's easy to come up and replace, but when a server crashes, several clients suffer at once. Yes, we tried to do what we could, and yes, reliable hardware is cheaper in the long run, but it's still a lottery, and if you get such a breakdown, then, of course, it's a shame. The same Amazon is also not insured against anything like this, and breakdowns happen quite regularly there, but for some reason, customers expect perfection from us every time. Forgive us for the physics and bad randomness if it hit your virtual machine.
Then the aforementioned DDoS. December 2018 and December 2019. Then in January and March 2020. In the latter case, several servers stopped responding (the physical machines were dead, and the virtual machines were on them) - a hard reboot was needed to make the network adapters come to life. Deploying back is not the most fun procedure, and a couple of people got downtime in hours, not minutes. Attacks happen every day, and in 99,99% of all circuits work out normally, and no one notices it, but there are times when something goes wrong.
In December 2018, during a four-hour attack, a network switch failed. The second one did not pick up due to some kind of mysticism, when trying to reanimate it, looped traffic appeared, and while we were figuring out what was happening, a simple one appeared. There was surprisingly little negative, everyone understood that DDoS happens. Although we raised the network for quite a long time by our standards. If you suddenly fell into this incident, then forgive us, and thank you for understanding everything correctly then.
Another important point: DDoS is always local. It has never been that problems in one data center developed simultaneously with problems in another. So far, the worst thing that has happened locally is a multi-machine switch reboot.
In order to finally reassure our hacking clients, we have insured liability with AIG. If they break us, and the clients suffer, the insurers must compensate. This turned out to be not very expensive in terms of a single tariff, but somehow gives confidence.
Support. We tried to make with different features to choose from and sufficient reliability. This means that our support does not do two things: it does not talk to the client in long polite phrases and does not climb into the application software. The second backfired on us last year, when numerous Instagram divas came who bought VDS to install like boosters and post automators. It is impressive how some people, extremely far from IT, have ways to competently figure out how to install software on a virtual machine. There is no such instruction that a fitonyasha will not master for a 30% increase in subscribers. But they broke down on setting up outgoing traffic inside their software for some reason. Perhaps the instructions did not provide for this. We cannot be responsible for the operation of third-party software. And the problems there are not only that the user does not understand how to configure it, but also in stability. For example, a person installed auxiliary software for cheating views on YouTube. And it comes from some forum complete with a trojan. And there is a bug in the Trojan, its memory is leaking. And we don't fix bugs in Trojans. If we install software, then this is a product out of the box.
This problem has been tackled . There are three stages: we do not know what kind of software is there, and we politely answer that we do not support such things. The second stage: there are several such requests, we understand one or two and write instructions, put it in our knowledge base and send it to it. The third stage: there are many such calls, and we start the distribution kit in .
And then, as we worked with more and more "non-admins", we began to encounter the second rake. Support always tried to work quickly and answered shortly and dryly. And some perceived it as passive aggression. What is acceptable in a dialogue between two admins is completely unsuitable for an ordinary user who has taken VDS for his small business. And over the years, there have been more such users. And the problem there is not that the support says something wrong, but how she says it. We are now doing a lot of work on updating the templates - we include in each one not only something in the spirit of "we do not support, sorry", but a detailed description of what to do and how, why we do not support, what now, and all this is polite and understandable . More details and explanations and more etiquette, instead of three-letter abbreviations, simpler explanations of what is there. We rolled it out for a week, let's see what happens. Before the pandemic, the priority was not to lick the client, but to solve the problem as quickly as possible. According to the philosophy of the enterprise, we are like McDonald's: you cannot choose to roast meat, support does quickly only what is included in standard requests. In general, the lesson is that if you answer dryly, people will often feel that they are being somewhat rude to them. We didn't think until last year, honestly. Well, we didn't mean to offend anyone, of course. In this regard, we lag behind the developed support services on the market: many have the goal of being very careful with the client, and we have just started working with this priority.
Rate. Well, our most epic fail is problems on a 30-ruble tariff. We have a special line of already weak iron, where VDS stands . Enjoys huge popularity. They immediately said in the description that there would be full stuffing, the tariff was not for work, but for training. In general, AS IS, and this IS will often be very scary.
As it turned out, such a description of the tariff stopped few people. 30 rubles is still cheaper than an ipv4 address, and then there’s a virtual machine with it right away. It seems to me that many bought just to buy, because we open it in waves. The first time everything went more or less normal, but then we did not pay due attention to the fact that after three or four months, recycling began to gradually increase - projects did not unfold there immediately, and by the end of the year the load became less comfortable for the average client, there were long queues for writing to disk, for example. Yes, there is an SSD, but we limit it on the tariff to HDD speeds, and this is not NVMe, but cheap Intel disks specially purchased for experiments for server configurations. We changed the disks to bigger and more normal ones, this allowed us to get at least some performance.
The second discovery of this tariff brought us thousands of Chinese users. They wrote scripts that are burning our site, because about 800 cars were bought by the fraternal people in the window between the appearance of the news on the site and the distribution, and this is just a few minutes. I can't say exactly what they were doing there, but judging by the nature of the traffic, they were dissidents who were bypassing the Great Firewall of China. We banned under the terms of the action to buy a car except for citizens of the Russian Federation. In order to protect Kwaimyeon, we had to suspend the creation of virtual machines. First, Russian users thanked us, then support - some of the users "in the process" had to be completed by hand. Well, there was a negative, because many people were waiting, and when they received the letter, the tariff had already ended.
Now we have several thousand active clients on a 30-ruble tariff. If the admin has straight arms, he makes the cheapest VPN in the world. Someone knocked on support with Linux screenshots with some kind of GUI (I don’t remember what was there, but the very fact of a GUI on such machines with limited RAM is already cool), someone installed an ISP panel, and so on . Someone really used to learning. We will do this action again, taking into account the mistakes, but just know that somewhere out there, in the Celestial Empire, there is a small forum for about a million registered participants who are subscribed to a thread about our servers.
The main lesson of this story is that the machines initially ran faster than expected, and people formed wrong expectations about performance. When she began to fall to the promised level, complaints began in support, and she was bombarded with negativity. Now, of course, we will explain more precisely what awaits at such a tariff. Again, forgive us if you are offended by this story.
This is how my vision of different moments in the market looks like. And now I want to ask you to tell me what infuriated you in the market and how it can be fixed for earthly money. If it is economically justified, we will try. Well, other hosters will look at this section of comments, and maybe they will do it too.
Source: habr.com