My colleagues and I thought: why not recall some horror stories related to data center construction instead of successes and interesting projects before our favorite horror holiday? So, turn off the lights, play some eerie music, and here come the tales that still occasionally make us wake up in a cold sweat.
The Office Ghost
In one office-building, we set up a server room and various automation systems for climate control, including blinds with motors. On the roof, there's a weather station that determines the direction of sunlight and closes the blinds if it's too bright. We delivered the project and forgot about it until, some time later, we received a call asking:
— Could you please re-automate the blinds? We want to close them ourselves.
— Why?!
— Our cleaners get scared. And we do too — it feels like there's a ghost in there.
This is exactly what a nightmare looks like for an automation specialist: at first, the client wants automation and energy efficiency, and you happily provide it, only to find out later that the manager prefers to take control himself. In the room where climate management is fully automated, everything ends up working manually.
In the offices of various executives, we often create touch panels from which one can control lighting scenarios, air conditioning, ventilation, and blinds. One particularly conservative executive said: I don't want algorithms, I just want two buttons: 'turn everything on' and 'turn everything off'. The programmer came, cried while removing the standard control interface, drew two buttons instead, and went home to cry.
Where are our diesels?
On a dark, dark night, in the dark, dark room of technician Oleg, his mobile phone rang.
— We have too high a coolant temperature in the diesel installation. I'll send you a screenshot.
It was the dispatcher from the data center that we had delivered a month ago with a clear conscience. He was unfazed by the fact that it was three o'clock in the morning or that the system was showing the same 'temperature' in the diesel and in the room. Because it wasn't actually a temperature at all, but an error code 'no connection with the sensor'. Oleg honestly told the dispatcher where he ought to go at night with such inquiries. Literally:
— Go check the diesel engines; most likely, the battery in the sensor has died. The battery module in the diesel units powers this control panel, and there’s a switch — if anyone touched it, it needs to be turned back.
In fact, the dispatcher is precisely the person who should know the facility as well as an experienced mill operator knows his three fingers, but then an astonishing question arose:
— And where are the diesel engines?
— Go up to the second floor, ask the electricians; they will show you.
For the next 20 minutes, Oleg worked remotely as a navigator, trying to connect the dispatcher and the electricians, who weren't particularly keen on giving a tour in the middle of the night.
The Cycle of Troubles in the Data Center
In a faraway kingdom, in a thirty-state empire, our frequency converter went down somehow to A vendor's service technician rushed in, threw up his hands, and said the patient was more likely dead, and waiting for a new board would take at least a month. The commission was already at the doorstep, and there were few options. The first was to unscrew the frequency converter from the "healthy" chiller and put it on the "sick one," then switch them back after the tests were completed. It's not cheating, by the way: per the testing regulations, one of the three chillers is still in reserve, so the scenario is quite feasible. The second option was to try to find a new frequency converter in the remaining one and a half hours. We called our refrigeration specialist in Moscow. The refrigeration technician contacted a known insider at the vendor's Russian office. That person, in turn, pressured the manufacturer's representative in the Netherlands, and... an hour later, they were already installing a new board for us. The certification went off brilliantly.
Incomplete Work
Long or short, there always comes a moment at the construction site when all subcontractors leave, and you're left with: unfinished work. This refers to incomplete tasks, not those who left them behind, just so you know. The last one left is the one who has to deal with it.
Whether long or short, there always comes a time on the construction site when all the subcontractors leave, and what remains are the unfinished tasks. This refers to the incomplete work, not to those who left it behind, just to clarify. The last one standing is the one who has to clean up.
Once we created a data center in a basement: racks in two rows, aligned against one side to allow the door to open. Because of this, a gap formed between one row and the wall, and the client refused to sign the acceptance certificate due to that gap. The tech man and the project manager hurriedly went to 'Leroy Merlin' to get foam, paint, fasteners and diligently filled the gap to match the racks. We submitted it.
And once after the heating subcontractors left, an inconsistency was found in the project: there should be 7 radiators, but there were only 6. We went and counted — it was indeed correct; they had failed to install one radiator in the corridor. It was too late to drink 'Borjomi', everything was already installed and pressure tested. The client was pulling their hair out because the acceptance certificates had already been signed. Once again, 'Leroy Merlin' came to the rescue — we bought an electric heater there, extended a group of cables into the corridor, and on Sunday morning, we installed it ourselves, making the client happy.
We were once left with a magical nothing instead of a fire barrier. In the data center, the supply air duct from the second floor goes to the basement through a room that, according to fire safety classifications, belongs to a different category than those above and below it. In practice, this means that there should be a fire damper in the duct, and a fire barrier around it. Around our shining air duct with a fire damper was a hole; guess who patched it up? 'Leroy Merlin' did not sponsor this post, what a pity.
Dale Carnegie is nervously smoking.
A long time ago, when the grass was greener and the dollar was around 30-something, we were creating a data center for a bank in the historical center of Moscow. The timeline was extremely tight. But the streets were narrow, the building’s openings were also small, and lifting several tons of equipment to the necessary floor via stairs was nearly impossible. We told the client that we needed to load the crane directly onto the roof; the client responded in the spirit of 'you’re the cowboy — you jump.' So, a good idea, but now you need to coordinate the arrival of a 120-ton crane with the authorities like the traffic police. And preferably, yesterday. Good luck with that; if you don’t hurry, there will be a fine.
The situation was deadlocked, time was running out, and we decided to take a risk; after all, the fines from the traffic police are nothing compared to the penalties for missing deadlines — just flowers. On Saturday morning, a 16-meter crane was brought in, hoping we could get everything done quickly. A couple of hours later, the local police officer arrived and timidly asked for permission. We, of course, didn't have it. And who knows how it would have ended if a salesperson with exceptional diplomatic skills hadn't been with us.
He took the officer aside, explained something to him for 5 minutes, during which the policeman's expression changed several times, but in the end, he got into his UAZ and shouted that if anything happened, he would come and help us himself. What arguments were used, the salesperson still keeps to himself.
Lift me up, people!
Beyond the hills and fields, but within the boundaries of the Third Transport Ring, there stood… no, not a hut, but a serious government construction project. A complicated night shift, unloading equipment. The last truck with a 15-ton piece of metal in its bed was leaving the only intersection on the territory when something made a loud noise and the machine sank into the mud. At 5 AM, the construction site was slowly waking up; the concrete mixer driver behind our truck, cursing fallen women, unambiguously inquired if we wouldn't mind clearing the road. After all, his concrete was hardening. And we would gladly comply because at 7 AM, one of the numerous clients of one of the ministers would arrive, and if he saw this mess, there would be hell to pay: from engineers to top management.
The tech guy ran to the local crane operator, desperately asking him to lift the metal piece from the truck bed so we could slide another truck underneath. But he refused, even for money. The situation was resolved by our engineer — he negotiated through the supervisor of that crane operator. We eventually delivered the metal where it needed to go, but we were a little greyer for it.
And then we became much grayer at the same site. An epic fail happened in the most literal sense.
There's a thing called a telehandler. It's used when there isn't enough space on the construction site but you need to lift a load up and place it carefully. We needed it to unload a 1.5-ton module through a hole in the wall. Everything seemed fine: according to the specifications, the machine should have been able to handle over 2 tons. But when we were just half a meter from the window, the 'forks' on the telehandler broke, and the metal piece fell from above. There was nothing to do—so we called the manufacturer for someone to come and conduct repairs. Their technicians arrived and... refused to enter the construction site. This was because they had to go along a path where an excavator was digging. We were used to it: we waited 10 seconds while the boom was turned away from the path, and then we ran. The guys were shocked. We had to present this situation as an exciting attraction, a sort of 'Fort Boyard', and they did fix the module for us.
Instant Karma
Soon the tale will be told, but getting things done takes time, especially when it comes to signing acceptance certificates for completed work. One time, we built an excellent data center for a company. But the client, our 'king,' decided to set us one last test:
— Your specifications state 100 nuts per tray, but I counted 97. Correct the specifications and estimates, or I won't sign anything.
And each time we ventured far and wide, counting with the client in unison for either the fasteners for the air ducts, the nuts, or the bolts. It turned out that it wasn't 97 but 99, and so on. And we never found peace. In the end, we racked up so many internal costs that our management couldn't take it anymore. They said: let them do what they want—no one is going back there again. So we ended up without signatures.
...And a year later, the client came by himself and politely asked where he could sign. It turned out that an auditing office had come, and he had unaccounted equipment worth seven zeros.
Wingardium Leviosa!
Once upon a time, there was a good client who decided to buy an old building for a data center. But his joy was short-lived: something was amiss in the battery room. He called us for help, to take a look and consult. We arrived, went into the battery room, and there... the walls were levitating above the floor on three sides. No wonder: they simply placed 4 tons of batteries on the floor — and it began to sink into the ground. This is a common problem with batteries: it’s crucial to calculate the load on the structure correctly and provide support frames, so the ceilings don’t collapse like a house of cards.
But the cherry on top of the architectural nightmare in this building was the complete lack of a foundation. The walls were simply standing on a sandy leveling layer, under which was not the most forgiving soil. We started to think about how to save the structure, and ultimately proposed a complex silication system: this involves drilling the ground in several locations and injecting a reinforcing solution. This didn’t restore the floor to its previous heights, but at least it stopped sinking.
The Battle of Two Yokozuna
In a certain kingdom, in a certain office state, we were doing dispatching: for the data center on one of the floors, for climate control — for everything. Dozens of automation cabinets, kilometers of low-voltage cables!
A feature of that office was its own SPA complex with a sauna. When we were developing the project, it was assumed that the steam room would not be used frequently. But the engineers think, while the client has other ideas: the management team got so into healthy living that they stopped turning off the sauna altogether — it takes too long to heat up.
The result: the automation detects the temperature rise from the sauna — it turns on the air conditioner more forcefully to compensate. It helps only marginally. The air conditioner continues to strain — the atmosphere heats up because the system doesn’t suspect that anyone could use the sauna so much. The AC falls into paranoia, deciding: 'It's not them, it’s me. All efforts are in vain. I think I’m broken,' which it communicates to the dispatch control panel. The dispatcher sighs and presses the 'task cleared' button. And this happens every single day.
With a light wave of the hand
We still have many stories in our archives, but not all of them can fit here. Here are three of the latest tales: about clumsy hands.
The first tale is about how we provided air support in the elevator lobbies and shafts of a building besides the server room. We finished the work, left the site, waited for another contractor to complete the finishing works, and returned for system testing. When we turned on the air support, the false ceiling suddenly inflated, and the panels crashed to the floor, instantly shifting the design to a 'loft' style. It turned out that the guys who installed the ceiling had forgotten to include the panels with perforations for air. They had no idea they existed; they just took from the open pack without noticing the packaging with the perforated ones and didn't look at the project.
The second epic is also about the ceiling. In one data center, there was a rather narrow corridor, and the air exchange in the room with the battery racks was supposed to be substantial, so the communications ran along the entire ceiling. The brave young men who made the false ceiling decided not to bother and… screwed the hangers directly into our poor air ducts. Any air duct vibrates a bit when the ventilation units operate, but when it has a bunch of holes and irregularities, the sounds of Apocalypse are guaranteed. We gave those glorious installation warriors such benevolent reprimands that the wounds on the air ducts miraculously healed. At their expense, of course.
The third tale occurred when we were working on the automation for the technological air conditioning system. We handed the client’s contractor a temperature sensor and thoughtlessly asked him to install it on the heater in the intake unit. There is always a heater for outside air in intake units: either water or electric. This wonderful person followed all instructions without question. He just took and screwed a self-tapping screw directly into the pipe of the water heater, with all the ensuing (literally) consequences. Fortunately, there were air conditioning specialists with MAP gas on site—the leak was quickly soldered.
Links:
- Write here if needed: build@croc.ru
Source: habr.com
