I’ll say right away: that rat in the brought server, which we gave tea to drink after an electric shock a couple of years ago, most likely escaped. Because we saw her friend once on the round. And immediately decided to put ultrasonic repellers.
Now there is cursed land around the data center: birds do not land on the building, and, probably, all the moles and worms have escaped. Worried that cause failure of the HDD, but checked, the frequencies are not the same.
The next story is much more fun. We somehow got a piece of iron for a couple of million rubles in a box with tilt, vibration and humidity sensors. Everything is whole. They carefully removed the packaging, and the piece of iron was bent. Mystic.
The body is straight arc. So beautiful.
Detective
We would not attach any importance to this, because the curved metal case was almost a design image. So beautiful, no chips. And if it were not for other similar pieces of iron nearby, we would not even have thought at unpacking that something was wrong. But nearby were the same, only of a more regular geometric shape.
Fortunately, the unpacking of such hardware is filmed (I recommend everyone to get into this habit), so we managed to prove to the manufacturer that it arrived like that. A whole package and a neatly bent body are not a blow from movers. Most likely, it was she who was damaged even before leaving for Russia.
The vendor says: "Um, guys, let's change it for you right away under warranty." And then an epic ambush awaited us.
The fact is that customs allows us to import such equipment according to documents without the right to export. That is, you can bring it, but you can’t resell it to someone outside of Russia. When we send back a burned-out power supply, for example, everything is clear. This is a spare part, a power supply.
And then it took to send everything back:
- Guys, look, we send the piece of iron back to the manufacturer.
- Whole equipment?
- Yes.
- What is the model?
- Yes.
- Can work?
We don't know, they didn't.
“So it’s a piece of equipment.
Well, she doesn't work.
— Well, look, the whole equipment is of this model. Without the right to re-export. Let's not let.
In general, there were many squats before we figured out that we were not exporting it, but giving it back. In the end, everything was done.
There were also shoe covers
We first had the first automatic many years ago, an admin's dream. You load shoe covers there in a pack, she unpacks them herself, opens them and puts them in a position where you just need to step on them. Bang-bang and you're done.
About six months later, she chewed on about a hundred packs of shoe covers and choked. It turned out that there are so many moving parts that we either need to repair it once a month at our load (we have a lot of customer engineers walking around the facility, because we are a commercial data center), or we need to buy a new one.
The second problem is that later, at one of the regular cleanings, we somehow found a “little blue rag” hanging on the grate of one of the racks of our testing. The medical examiner, represented by an X-team engineer, identified a fragment of the shoe cover's body. It turned out that it was convenient in the clinic with shoe covers: I walked around for half an hour - and that's it. And some engineers can work with iron all day long. Scuffing your feet. Strongly shuffling. And at the same time, shoe covers wear out into these little shreds that fly around the turbine hall.
We almost immediately bought a new shoe cover. We took a thermosleeve: this is a machine into which the film is loaded, and it gently heat-shrinks this film over the boot. Beautiful, efficient, durable. It shatters less. We had it for quite a long time, but it was necessary to change the shrink film about once every 1-2 hours, because the sole had a tendency to fall off by itself.
At first we thought that we were not lucky, but people somehow solve this problem. But no. They asked Western colleagues - the same story. As a result, they began to think how to do it right. Returning from the turbine hall for new shoe covers is, frankly, a so-so idea. Found industrial cleaners for construction sites and industries. This is something like the paths along which the shift enters the workshop. Tracks with a bunch of rollers clean everything, and are made in such a way that you like it or not - it will catch and clean it. They cost half a million to a million rubles. We dug around and found the same one for 200 thousand, but you have to stick your foot into it yourself. Similar in size to a shoe polisher. You come up, shove your leg in there, she chews it and gives it back clean. Put at the entrance to the data center.
Works great with two exceptions. First, it quickly became clear that it was normal for us engineers. But in practice, a variety of people come to see the data center, among them are the tops of large companies. With shoes made of leather from the ass of a dragon. And they even have a brush for applying cream on shoes that costs more than my training sneakers, they specially pick up the pile there. It was they who refused to shove their legs into our miracle device. The second problem arose in winter: when the shoes are right in the mess, they can’t get everything out of the deep tread. Then you walk around the hall, leaving traces of ectoplasm.
We decided simply. They put a roll-up shoe cover nearby. All the same, we need to duplicate everything according to the standard.
A new problem has arisen. Observing the behavior of customer engineers, we saw the following picture: they first put their foot into the machine for cleaning, and then hooked shoe covers from a roll shoe cover. Now they put up a sign that it’s either one or the other, and it’s better to clean up, but if life principles forbid cleaning shoes, then put on shoe covers. It seems that the ticket, which was for a couple of days, but stretched out for a long time, closed. Here is the device:
Two times "ku"
According to the requirements of PCI DSS, you need to be able to visually distinguish the roles of people located in the data center. Not looking closely at the pass and reading something there, but directly visually, how the military distinguishes each other by shoulder straps, only even brighter. We decided not to show off and used the good old Chatlan method - the color differentiation of pants. Specifically, they began to make pass ribbons of different colors. Green as their favorite was immediately taken by our admins.
Sounds simple, but it produced three unexpected effects:
- Retractors were needed to automatically retract these passes when worn (these are things that themselves adjust the length of the tape). We wrote a TOR, where we took into account all the wishes of all departments. This was a strategic mistake. Color, format, material, the retractor is not plastic, a metal fishing line to apply the logo, so that it is sewn into the tape. The pieces turned out to be so expensive that I had to cut the requirements later and change the format.
- When the differentiation of the pants worked, it became very convenient. Contractors have one tape, external admins have others, our admins have third. You can see who has what role. In electrics - only gray, in air conditioning - blue. And then ribbons were needed for drivers (this is a separate role, they can enter the unloading area, but cannot leave it except outside). However, drivers do not need a pass. At first we gave them ribbons without passes. Then the security guards decided that it was completely strange and degrading to the human dignity of the drivers. They have their own military logic, so now drivers come to get a pass immediately with a ribbon, but this pass does not let them go anywhere. From the point of view of security, it turns out a marker that the security of this person has checked.
- One of our engineers suggested that instead of the ribbon, we make uniform sweaters in green. And sent a rationalization proposal. They did it halfway: they left passes with a ribbon, plus they actually sewed green uniform sweaters. Now we have an admin uniform. The security guards supported the joke and made it into the regulations. Now it is mandatory (pants, shirt, sweater, but the sweater can be removed).
Our customers also often complained about crooked routes in the maps before entering our Compressor data center. You fill in the address, and the road is shown incorrectly. As a result, visitors taxied in the wrong direction, because there is a railway there, and there is a traffic jam behind it, and there is no turning around. At first we wanted to hang signs over the road. There is such a service at the city - under the usual signs to place additional yellow ones, they are considered advertising. And the price for them is like advertising: on the Enthusiasts Highway, one plate costs a million rubles a year. In parallel, we wrote to Yandex, and they even suddenly answered. And they stopped messing around. You can even specify gate diodes: entry through one, exit through others.
Google, if you follow us, then know that you still have a problem, and we don't know who to tell about it to hear us.
In invitation letters, links were inserted not just to an address, but to an address with a route from the user's geolocation. As a result, there were fewer misses.
Gobo projectors and other little things
Do you know what gobo projectors are? We didn't know either. Somehow they thought about how to mark the rows of racks. The racks themselves, of course, are marked with special quick-release markings, but they can be seen from a distance of 1-2 meters. The hall itself is 500 sq.m, so there is where to get lost. Therefore, they nevertheless approached the marking of the rows. The brainstorm has begun. How to mark, with what and where? On the floor, on the wall, pointers on the ceiling, etc. And then our colleague saw that in Ikea there used to be wearable stickers on the floor, and then light arrows appeared. Well, we decided to reverse it in a simple way: go to Ikea and turn one of the projectors out to look. It was not possible to get it: while we were wearing chairs, the seller asked what we were doing. And he immediately helped, saying that it was a gobo. It turns out that this is not the projector itself, but a plate or lens for a color image. This light filter is the gobo. One projector costs from 40 thousand rubles (there is a powerful lamp for daytime use), and we have 14 rows in each of the four turbine halls. So put stickers on.
We also have diagrams on the walls that fade over the years. We changed them to laminated ones, with “sewn-in” special pockets for revizorro. Revizorro in our case is the chief engineer, whose duties include checking for the relevance of all schemes located in the data center. So, all schemes must be checked annually and signed by such an auditor. And the presence of a special small magazine in the pocket of the circuit makes this procedure easier and does not require the replacement of the circuit itself every three years. Profit!
They did a rotary cleaning of the raised floor from the outside. We have regular cleanings, there are methods and times for cleaning. But from the wheels of heavy racks there are traces. They did a cleaning. Now we are nervous: it didn’t look much neater, but glare appeared under certain angles for certain people, well, who have their own felt-tip pens to their taste. Now we are thinking and looking for some kind of chemistry that will whiten the floor and add gloss. So that even the chosen ones do not have questions.
Have you seen console racks? These are traveling buffet tables, only instead of drinks there is a terminal for connecting to the rack. So, at these cantilever racks, the wheels fall off and wedge, like carts in a supermarket. It got us incredibly. As a result, the only possible way to reanimate is to buy a new wheel. But it was no longer possible to get wheels specifically for our models, we interviewed all the contractors. As a result, they themselves designed the rack for themselves, focusing on ease of movement around the turbine hall and maintainability. It worked out very well.
There was a story with synthetic socks. There is such a thing - antistatic bracelets. This is when you approach the rack, connect the bracelet to the ground on the rack, and it contacts the potential equalization system. So, the rack is grounded, but it may turn out that the engineer is not grounded. Colleagues from previous places of work told how they saw sparks on video surveillance a couple of times, and we decided from sin to oblige everyone to use it directly according to the regulations.
Critical Incidents
From a serious point of view, there was a situation when all the chillers were cut off at once. Our chillers are not protected by UPS, because we believe in physics, and as a temperature reserve we have a cold water pool. If something goes out, you don’t need batteries to power the chillers that cool the water, but just the cold water itself, already ready. Convenient and simple, but there is a nuance. The chillers are equipped with protective automation, which, in case of dangerous parameters of the electrical network, turns them off. If the input is turned off, we turn on the DGU, and then the chillers are powered by them. Everything would be fine if we did not live in Russia. We had many network drawdowns, but everything was fine. But once the jump was sharp, first down, then sharply up, then down again - in a few seconds, the input parameters changed by about 4 times. Chillers turned off, of course. We first tried to turn them on remotely, but they protected themselves very reliably, in an emergency way. I had to change legs on the roof and turn them on manually. What is important, according to the TierIII standard, such a situation is a legitimate reason for stopping the data center. We didn’t have a stop, because people are on the ground with a head, and there is a drill with teachings. For this, the UI just fucked us regularly to be sure about the TIII Operational. If anything, we have passed the UI recertification for TIII Gold - Operational Sustainability. There is nothing cooler on the Russian commercial data center market, except for ours, only one has the same achievement . I note that it is harder to re-certify than to get a certificate from scratch, since they still check the previous period as if they are not themselves, and much more evidence is required.
There was an interesting incident with the cameras. We decided to recalculate the blind zones just in case, drew intersections, plotted diagonals of viewing angles on the plan, and suddenly found a dead zone of about 30 centimeters by 15 meters right in the center of one of the halls. Narrow and long. There is none in the next room. It turned out that the PTZ camera had slowly moved over the years so that it began to show about one and a half degrees to the left than it should be in the extreme position.
Another big case was in the post .
references
- и
- My mail: [email protected]
Source: habr.com