🥇A Day in the Data Center: The Unnoticed Details from 7 Years of Operation. And the Continuation about the Rat

Let me tell you right away: that rat in the delivered server, which we fed tea after it got electrocuted a couple of years ago, most likely escaped. Because we saw its friend during one of our rounds.

Now the cursed land surrounds the data center: no birds settle on the building, and probably all the moles and worms have fled. We worried that the sound might cause HDD failure, but we checked and the frequencies were off.

The next story is much funnier. We once received a piece of hardware worth a couple of million rubles in a box with sensors for tilt, vibration, and humidity. Everything was intact. We carefully unwrapped it, and the hardware was bent. A mystery.

The case was like a bow. Quite beautiful.

A detective

We wouldn't have paid much attention to it, because the bent metal case looked almost like a design piece. Quite beautiful, with no chips. And if it weren't for other similar pieces of hardware nearby, we wouldn't have even thought something was wrong during unpacking. But there were others that were not bent, with the correct geometric shape.

Luckily, unboxing such hardware is recorded on video (I recommend everyone to adopt this habit), so we managed to prove to the manufacturer that it arrived that way. An intact package and a carefully bent case isn't a result of a mishap by the freight handlers. Most likely, it was damaged before it left for Russia.

The vendor says, "Hey guys, let's just replace it under warranty right away." And then we faced an epic trap.

The thing is, customs allows us to import such equipment on the documents without the right to export. So we can bring it in, but we can't resell it outside of Russia. When we send back a burnt power supply, for example, it’s clear. That's a spare part, a power supply.

But here we needed to send everything back:
- Guys, look, we are sending the hardware back to the manufacturer.
- The whole equipment?
— Yes.
- Is the model such-and-such?
— Yes.
- Can it work?
- We don't know, we haven't turned it on.
- So it's whole equipment.
- But it doesn't work.
- But look, it's whole equipment of this model. No right to re-export. We won't allow it.

In general, there were a lot of squats before we figured out that we weren't exporting it, but returning it. In the end, we managed to get everything done.

There were also shoe covers.

Years ago we had our first automatic shoe cover dispenser, a dream for any admin. You load packs of shoe covers into it, and it unpacks, opens, and places them right where you just step in. Snap-snap, and it's done.

About six months later, it chewed through around a hundred packs of shoe covers and choked. It turned out there were so many moving parts that we either had to repair it every month given our workload (we have a lot of engineers on site since we're a commercial data center) or buy a new one.

The second problem was that during one of our regular cleanings, we found a 'blue rag' lying on the grid of one of the test stands. The forensic expert, an engineer from the X-team, identified it as a fragment of a shoe cover. It turned out that at the clinic, shoe covers are convenient: you walk around for half an hour—and that’s it. But some engineers can work with hardware all day long, shuffling their feet. Really shuffling. And the shoe covers wear down into these little scraps that float around the machine room.

We almost immediately bought a new shoe cover dispenser. We took a thermal shoe cover dispenser: it's a machine where you load a film, and it carefully heat-shrinks the film over the shoe. It's neat, efficient, and sturdy. It scatters less. It sat with us for quite a while, but we had to change the shrink film about every 1-2 hours because the sole tended to come off by itself.

At first, we thought we were just unlucky while others somehow solved this issue. But no. We asked our Western colleagues—the same story. As a result, we started thinking about how to do this properly. Going back from the machine room for new shoe covers—let's just say, that's a rather poor idea. We found industrial cleaners for construction sites and manufacturing. It’s like mats that the shift walks on to enter the workshop. These mats with a bunch of rollers clean everything, and they're designed to catch and clean your shoes whether you want it to or not. They cost between five hundred thousand and a million rubles. We looked around and found one for 200 thousand, but you have to stick your foot in it yourself. It’s dimensionally similar to a shoe cleaning machine. You approach, shove your foot in, it chews on it, and gives it back clean. We installed it at the entrance of the data center.

Works great except for two nuances. The first — it quickly became evident that it suits us engineers well. However, in practice, various people visit the data center, including top executives from large companies. With shoes made from dragon leather. Even their shoe cream brush costs more than my training sneakers; they specifically select the pile for that. They refused to shove their feet into our miracle device. The second problem arose in winter: when shoes are very muddy, they might not be able to remove everything from the deep tread. Then you walk around the hall, leaving ectoplasm traces.

We solved it simply. We placed a roll-up shoe cover dispenser next to it. Anyway, we have to duplicate everything by standard.

A new problem emerged. Observing the behavior of the engineers-clients, we saw the following picture: they first shoved their foot into the cleaning machine, and then grabbed shoe covers from the roll-up dispenser. Now we’ve hung a sign that says either one or the other option is preferable to clean up, but if life principles prohibit cleaning shoes, then wear the covers. It seemed like a ticket that would last a couple of days, but it stretched out for a long time and has now closed. Here is the device:

Twice "ku"

According to PCI DSS requirements, it is necessary to visually distinguish the roles of people in the data center. Not by scrutinizing the pass and reading something there, but directly visually, like the military distinguish each other by their ranks, only even more vividly. We decided not to show off and used the old good chatlan method — color differentiation of the pants. Specifically, we started making pass ribbons of different colors. The admins immediately took the green ones like their favorite.

Sounds simple, but it caused three unexpected effects:

We needed retractors for automatically retracting these passes when worn (these are devices that automatically adjust the length of the ribbon). We wrote up specifications that took into account all the wishes from all departments. This was a strategic mistake. Color, format, material, retractors not plastic, fishing line made from metal, to print the logo, and it had to be sewn into the ribbon. The things turned out to be so expensive that we later had to cut back on the requirements and change the format.
Once the differentiation of the bands was implemented, it became very convenient. Contractors have one type of band, external admins have another, and our admins have a third. It's clear who has what role. In electrical work, only gray bands are used; in air conditioning, blue ones. And then we needed bands for the drivers (this is a separate role; they can enter the unloading zone but cannot leave it except outside). Drivers do not need passes. Initially, we gave them bands without passes. Then, the safety officers decided that this was quite odd and demeaning to drivers' dignity. They have their own military logic, so now drivers come to receive passes right away with the band, but this pass does not allow them anywhere. From a security perspective, it serves as a marker that this person has been checked by security.
One of our engineers suggested replacing the band with green uniform sweaters. He submitted a rationalization proposal. We did it halfway: we kept the passes with bands and also actually sewed green uniform sweaters. Now we have admin uniforms. The safety officers supported the joke and included it in the regulations. Now it is mandatory (pants, shirt, sweater, but the sweater can be removed).

Our clients often complained about the incorrect routes in the maps leading to our data center 'Kompressor'. You enter the address, but the road shows incorrectly. As a result, visitors would turn the wrong way because there is a railway and behind it a traffic jam, and they could not turn around. Initially, we wanted to hang signs above the road. The city offers such a service—placing yellow additional signs under regular ones; these are considered advertising. The cost is like advertising: one sign on Enthusiasts Highway costs a million rubles a year. Meanwhile, we wrote to Yandex, and they surprisingly responded. They stopped their mischief. You can even indicate the gate's diodes: entrance through one, exit through another.

Google, if you are reading this, know: you still have a problem, and we don’t know whom to tell about it to be heard.

In the invitation emails, we included links not just to the address but to the address with the route plotted from the user's geolocation. As a result, mistakes in navigation decreased.

Gobo projectors and other minor items.

Do you know what gobo projectors are? We didn't know either. We were thinking about how to label the rows of racks. The racks themselves are marked with special quick-release labels, but they can only be seen from a distance of 1-2 meters. The room is 500 square meters, so it’s easy to get lost. So we tackled the labeling of the rows. A brainstorming session began. How to label, with what, and where? On the floor, on the wall, ceiling signs, etc. Then our colleague noticed that IKEA used to have wear-resistant stickers on the floor, but then they introduced light arrows. So we decided to take a simple approach: go to IKEA and check one of the projectors. We couldn't retrieve it: while we were moving chairs, a sales associate asked what we were doing. He immediately helped, saying it was a gobo. It turns out that it’s not just the projector, but a plate or lens for colored images. This color filter is the gobo. One projector costs around 40,000 rubles (it has a powerful lamp for daytime use), and we have 14 rows in each of the four machine rooms. So we opted for stickers.

We also have diagrams on the walls that fade over the years. We replaced them with laminated versions, with "built-in" special pockets for the Revizorro. Revizorro in our case is the chief engineer, whose duty is to check the accuracy of all diagrams posted in the data center. So, all diagrams must be checked and signed annually by such a Revizorro. The presence of a special small journal in the diagram's pocket simplifies this procedure and eliminates the need to replace the diagram every three years. Profit!

We performed rotary cleaning of the false floor from the outside. We have regular cleaning schedules, methods, and times for cleaning. But the heavy rack wheels leave marks. We did the cleaning. Now we are nervous: it doesn't look significantly cleaner, but there are reflections at certain angles for certain people, you know, those with their own markers to their taste. We're now contemplating and searching for some chemicals that will bleach the floor and give it a sheen. So that even the chosen ones have no questions.

Have you seen the console stands? They are like portable buffet tables, but instead of drinks, there’s a terminal for connecting to the stand. The wheels on these console stands keep falling off and getting stuck, just like the carts in a supermarket. It has been incredibly frustrating. Ultimately, the only way to fix them is to buy a new wheel. However, we couldn't find wheels specifically for our models, despite surveying all suppliers. In the end, we designed a stand tailored to our needs, focusing on ease of movement across the machine hall and repairability. It turned out really well.

There was a story with synthetic socks. There's something called anti-static bracelets. It's when you approach the stand, connect the bracelet to the ground on the stand, and it links with the potential equalization system. So, while the stand is grounded, it might turn out that the engineer isn’t grounded. Colleagues from previous workplaces recounted a couple of times they saw sparks on surveillance footage, so we decided to mandate everyone to use them as a precaution.

Critical incidents

One serious incident involved all chillers shutting down at once. Our chillers are not protected by UPS because we trust physics, and as a backup for temperature, we have a pool filled with chilled water. If something goes offline, we don't need batteries to power the chillers that cool the water; we just need the cold water itself, which is already prepared. It’s convenient and simple, but there’s a catch. The chillers have protective automation that shuts them down when electrical parameters become dangerous. If the input power goes out, we start the DG set, and then the chillers are powered by it. Everything would have been fine if we did not live in Russia. We have experienced network drops many times, but everything was okay. However, one time there was an abrupt spike — first down, then sharply up, and then back down again — in just a few seconds, input parameters changed by about 4 times. The chillers shut down, of course. We first tried to turn them on remotely, but they had securely protected themselves in emergency mode. I had to go up to the roof and turn them on manually. Importantly, according to Tier III standards, such a situation is a legitimate reason to stop a data center. We did not stop because the on-site staff are competent and trained. For this, we were regularly checked by UI to ensure compliance with TIII Operational standards. By the way, we have passed UI recertification for TIII Gold — Operational Sustainability. In the Russian commercial data center market, nothing is better than ours, and only one other facility has the same achievement. Data CenterI should note that recertification is more challenging than obtaining a certificate from scratch, as they assess the previous period as well, requiring much more evidence.

There was an interesting incident with cameras. We decided to recount the blind spots just in case, drawn intersections, built diagonal sight lines on the plan, and suddenly found a dead zone about 30 centimeters by 15 meters right in the center of one of the halls. Narrow and long. There is nothing like this in the adjacent hall. It turned out that the rotating camera had slowly shifted over the years, so it ended up showing about one and a half degrees to the left of where it should have been in its extreme position.

There was also a major case with the replacement of the UPS.