The biggest transportation problem in St. Petersburg is the bridges. At night, you have to rush out of the pub without finishing your beer. Or pay twice as much for a taxi. In the morning, you must carefully plan your time so that as soon as the bridge is lowered, you can make it to the station like a nimble mongoose. We do not consider the option of 'staying somewhere in the center, near the station' as ideologically harmful.
... The 'Sapsan,' leaving at 5:30, was conceived by people who hate owls. Of course, it's very convenient to be in Moscow by nine in the morning, and St. Petersburg is incredibly beautiful during the white nights, but at half past four, when I got into the taxi, I wanted to turn my head 270 degrees to the lark who invented this morning train.
'All good data centers are alike' — I thought when I learned that we were invited by the guys from DataLine.
It turned out that each data center is good (or bad) in its own way, especially after you've collected all the bruises, gathered every stumbling block around, and learned lessons from your own (and, somewhat, others') rich experience. So we (consisting of five leading linkmeup members along with their friends and sympathizers) decided to check how good (or not so good) the OST and NORD data centers are.
Having worked a bit on the road (there could have been an advertisement for a mobile operator here, but no — the coverage remained as poor as before), we disembark at the Leningrad Station. Well, at least not to Gazmanov.
We began our tour of the 'green' data centers with OST — 'that one' that recently 'caught fire.' The quotation marks are for later. For now, we stand, smoke, and wait for the delayed ones. We take photos of the neighboring RT office. There are many funny stories about this neighborhood — starting with laying fresh asphalt and painting curbs before the arrival of High Personalities, and ending with discounts at nearby cafes that were offered to DataLine employees due to the similarity of badges.
The trumpet of totalitarian propaganda
Data center with strict (pass) regime
Pass, rules of conduct, meat grinder at the entrance — we saw almost the same one . Have you seen many coffee shops in data centers? We now know a couple. Employees and clients have 'coffee' deposits on their badges that are regularly replenished, so within reasonable limits, you can enjoy coffee completely for free. As esteemed guests, we were treated as well.
I can't say anything bad about the coffee — it’s quite typical for a 'chain' café, but the location is indeed appealing, and the interior is right in line with modern trends.
The interior of the local coffee shop
Photos of NORD data centers
The tour starts from the office. Besides the well-known phrases on the walls and the white-green color scheme, there are also various amusing items. These are props left over from corporate theater performances (can you imagine the technical director of a network of data centers performing in a play? We know him personally).
Indeed
Photos from performances
The data center (or more precisely, four data centers) is located in a complex of industrial buildings constructed back in the 19th century (then it was the 'Citroën' factory, and during the Soviet era - 'Hydrodrive' factory), so on one hand - there's a harsh industrial atmosphere, metal beams, a loft vibe, and industrial artifacts from various eras that are still found during various work. On the other hand, the hermetic zones of the data center need to be integrated into the existing structures, often requiring compromises with reality. However, I wouldn't say these compromises have affected the quality — but the team surely had to work hard during the design phase. Also, rumor has it that the Metro-2 passes somewhere nearby, but that's not confirmed.
In addition to the theater, there is also a museum here. The exhibits, naturally, are archaeological finds.
A few words about Yes, the building (or rather, the roof) did indeed catch fire. The external cooling system blocks and refrigerant pipelines were damaged. Somewhere, wiring burned out and the air conditioners were left without electricity. In some places, refrigerant pipes exploded. However, despite this, the hermetic zones withstood the tons of water poured onto them, and the equipment survived. Recovery efforts were managed by the 'headquarters' represented by senior management, and everyone worked together — they gathered all the copper pipes from Moscow and the surrounding areas (even bringing them in at night on BMW X5s) and all the sober installers. Critical equipment from the affected data halls (affected by the lack of cooling, not by the fire!) was quickly relocated to other rooms, so by five in the morning, not only were all our services restored, but also those of our clients.
We personally checked the hall that was the most 'damaged' — other than the smell of smoke and the replaced tiles of the false floor, there is nothing to remind of the fire (except for the equipment racks that were 'temporarily' taken to another data hall, and remain there).
Status Panel
Walking through the corridors, we see a touchscreen display showing something that looks very much like a layout of the data center server rooms. This is the status panel. They are scattered throughout the corridors and allow on-site engineers to quickly find, for example, the necessary air conditioner, and simply provide the current status.
Monitoring is built on Nagios (now there are several independent clusters) and covers all data centers (both OST and NORD). Additionally, there are applications for Android and iOS (for clients and employees), and a status page right on the website.
On the display, I notice an unusually high PUE — it's not common to calculate this everywhere in Russia, but still, a PUE of 1.90 is quite high. The reason for this is purely economic — in 2012-2013, racks in data centers were selling like hot cakes, so energy efficiency was not a priority. We installed what was on the market, did it as quickly as possible, and didn't care about these PUE numbers at all, because why bother? During the design stage, 80-90% of the racks were purchased, and legislation in Russia doesn't care about energy efficiency. Though some commission in Amsterdam would be shocked by these figures.
Nothing personal. Just business.
Another feature is the labeling of data halls. In each facility, data halls not only have an 'official' alphanumeric coding but also 'human' aliases from the naval alphabet (Alpha-Bravo-Charlie-Delta…). For some reason, all clients love the Whiskey hall.
In client care, there are no small details.
Such ‘fences’ are set up for certain clients. We weren't allowed to take photos in the machine rooms, so here's a photo from DataLine ©.
DataLine clients are all large and reputable companies, and they do not like their equipment being photographed. Especially if the client's name (or logo) is prominently displayed on the rack in large letters. They simply don’t like it. Therefore, we photographed the now-famous walls with various inscriptions and the engineering area. Not all of the engineering area can be photographed everywhere, but we tried. Since among our clients are large and respected firms, their requests can be specific—special racks (or even some kind of super storage solution from several vendor racks), their own fences, their access control systems, their own video surveillance, and a rapid response team with rifles. In general, one of the unspoken (and spoken as well) rules of DataLine is to take care of the client and make things convenient for them. Starting with vests and equipped workplaces with internet access and chargers, and ending with various events for clients. The latter are also very useful because the same people move from company to company and bring them to DataLine. Simply because everything here is cozy, convenient, and reliable. DataLine’s business is not just about renting rack spaces. It's also about cloud services, simultaneously certified under both Law 152-FZ and PCI-DSS (we're planning to do a podcast about it, so stay tuned for announcements), as well as providing connectivity (own fibers between facilities and from each of them to M9, their own optical ring around the city, and all the way to organizing the last mile to the client's office).
M stands for Motivation.
Speaking of inscriptions, ideas come not only from employees but also from partners and clients, after which the best ones can be voted on at DataLine on FB. You can also suggest your idea there.
Meeting on the Elbe
Cross-country (photo provided by DataLine)
We stop at the Meet-Me-Room, where the operators are located. In the MMR (there are three pairs in OST, corresponding to the number of fiber optic main inputs, for active and passive setups), all the connections from the data halls are aggregated. The cross-connect area is boring and neat. However, where the operator's active equipment is located, you remember the best years and scenes from some BM18: a tangle of wires and a variety of equipment (DLink, Extreme, Juniper, Mikrotik… and so on up to ASR9k). There are more than fifty operators here, so everything is colorful. From the MMR, fiber optics and copper spread out to the machine halls. At least two paths for each (variations are possible), so you don't have to worry about connection reliability. The MMR is called 'Elba,' which is quite a clever name.
Need IBM Power?
Another find is the IBM P-series, which has never been turned on and has been sitting in one of the corridors for several years. The machine (along with a tape library and storage system) was ordered by one of the clients, but it 'wasn't needed,' so it was donated to the data center. It has been sitting there ever since.
It must be nice to be wealthy guys and make such gifts, right? (Or maybe it wasn't donated but forgotten. Or lost in a card game. But it has been there for a long time.)
This handsome machine is located in NORD-4.
Let's go to the power center — three (or even four) DGUs for each data center (reminder: there are four in OST). The diesel reaches full power in 30-40 seconds and delivers 1900 kVA. N+1 redundancy, synchronization, and more are included. The longest time running on diesel in OST is 12 hours, while in NORD it is 2.5 days (after all, a storage capacity of 85 tons is no small feat).
Of course, there are agreements with various fuel companies for fuel delivery. Switching to diesel occurs regularly, ensuring both fuel consumption and contract checks are maintained. By the way, according to colleagues, during the testing of DGUs in NORD, which usually takes several hours, grass even starts to grow in the spot where the hot air from the diesels is expelled, even in winter.
After a quick snack at the familiar café, we head to the NORD site. This isn't just one data center but four independent buildings on the same territory. Each has its own power inputs, its own diesels, and its own cooling systems. NORD-1 and NORD-2 have passed the Uptime Institute M&O audit. The newest (and best DataLine data center) is NORD-4, so that's where we're headed.
Despite the seemingly huge number of data centers in Moscow, there is still a shortage. Of course, you can get one or two racks without any problems. They might even give you twenty. More than that, however, becomes a question (and DataLine has clients with such needs — not just one or two). Specifically, there is currently space available in NORD-4 because a major client moved to their own newly built data center, but in other locations, luck may not be on your side. Nevertheless, the design of NORD-5 is actively underway, and it is promised to launch in early 2020.
But let’s return to our tour. Since we were traveling by car, we were allowed into the gateway, where armed guards took aim at us, checked our vehicle's registration number against the application, and asked all passengers to exit to receive passes (of course, presenting passports). Because B stands for security.
NORD-4 is the pride of the company and a culmination of many years of experience in the construction and operation of data centers. It has Uptime Institute Tier III certifications for its design and facility (i.e., the constructed data center), as well as Operational Sustainability — the team approached the project with maximum responsibility. Inside, there are four independent hermetic zones with two halls within a single building. Each has its own energy center (diesel generators, battery rooms), cooling contours, and fire suppression systems. The only shared components are the optical inputs (two for the building) and the MMR.
Modding a healthy engineer
At the entrance to the data center, a familiar coffee shop is waiting for us (well, why not?). We head up to the monitoring center. There are currently two main ones — in OST and NORD-4 (NORD-1, 2, and 3 also have their own duty shifts). From each center, you can monitor both sites, ensuring redundancy n+1 even here. On the video wall, we see the dashboards in a green color scheme (naturally, there are many more parameters on it than on the status display). We chat a bit about monitoring — no trendy Prometheus and Grafana, just Nagios, hardcore, and custom visualization. Over ten years (actually, even more) of usage have resulted in many custom checks and scripts, so switching to something else seems quite adventurous. About once a month, another company appears offering to resolve all issues "quickly and painlessly," but so far, it hasn't worked out. By the way, the video wall also runs on a self-assembled solution, which turned out to be several times cheaper than commercial alternatives.
Local Command Center
When you design, build, and operate data centers for many years, you become less afraid to do things yourself rather than just buy off-the-shelf solutions. DataLine, for example, isn't shy about saying that they, simply due to their scale, have hit the maximum number of pitfalls, and each new data center is a product of difficult mistakes and experiences amassed over more than ten years. Of course, each new data center has its own innovations, and while walking through NORD, you often catch yourself thinking that people just took the initiative and did well. Not for the sake of show, nor for ratings and certificates — just a routine engineering process where, with each iteration, you grow and improve. An outside observer might say this is boring and conservative, but clients, on the contrary, support such stability with their own funds. To achieve quality, it's not always necessary to spend a fortune on engineering systems or software (even though sometimes it's unavoidable). Often, experience, knowledge, and engineering ingenuity make the difference.
For example, the NORD-4 building has 3.5 floors. The first three are full-fledged, with ceilings of 4.5 meters, where the machine rooms are located. The last one is a technical floor, only 1.8 meters high, dedicated solely to the cooling system. Since it is a technical floor, the requirements for it are much lower, which allowed for a simplified design and saved costs.
View of the triangular data center
Max has a special kung fu for not being caught in photos
We finish the tour on the roof — from here there is a great view of the nearby CHPP and residential area, as well as other NORD buildings. Two of them are triangular. Colleagues joked that 'DataLine is foolish for building a triangular data center' to accommodate triangular racks and triangular servers. But it was built, successfully operated, and effectively uses the available space (which is not cheap in Moscow). Another surprising fact for me is that most commercial clients want to be located specifically in Moscow. The site over fifty kilometers down the Minsk Highway (even if it has all the necessary capacities and communications) is of little interest to anyone. So everyone is cramped in a limited space.
Cooling towers are no longer a data center but a CHPP.
And so our journey has come to an end.
Afterwards, there was a long and productive discussion about joint plans and future podcasts (I hope we won't have to wait long), so stay tuned for news, there will soon be more interesting content.
For those who wish to see everything with their own eyes, DataLine invites you for a tour. Completely free, but by appointment only. Those who mention the password 'linkmeup' will earn extra karma.
Stay tuned!
Source: habr.com
