Tales from the data center: Halloween horror stories about diesel engines, diplomacy and self-tapping screws in the heater

My colleagues and I thought: before our favorite horror holiday, why not, instead of successes and interesting projects, remember all sorts of horror films that people encounter in property development. So, turn off the lights, turn on the disturbing music, now there will be stories from which we still sometimes wake up in a cold sweat.

Tales from the data center: Halloween horror stories about diesel engines, diplomacy and self-tapping screws in the heater

Ghost of the office

In one office building we made a server room and all kinds of automation for climate systems, including curtains with drives. There is a weather station on the roof that determines which side the sun is shining from and closes the curtains if it is too bright. The object was handed over and forgotten, after a while they call and ask:

— Could you de-automate the curtains again? We want to close everything ourselves.
- Why ?!
“Our cleaners are scared.” And so do we – it feels like there’s a ghost there.

This is exactly what an automation specialist’s nightmare looks like: first the customer wants automation and energy efficiency, you happily do all this for him, and then it turns out that the manager likes to lead himself. And in a room where climate control is fully automated, everything ends up working in manual mode.

In the offices of all kinds of top executives, we often make a touch panel with which you can control lighting scenarios, air conditioning, ventilation, and curtains. One particularly conservative top said: I don’t want algorithms, I want two buttons: “turn everything on” and “turn everything off.” The programmer came, cried, removed the standard control interface, drew two buttons instead, and went home crying.

Where are our diesel engines?

On a dark, dark night, in the dark, dark room of technical man Oleg, his cell phone rang.

— The coolant temperature in our diesel unit is too high. I'll send you a printscreen now.

It was the data center manager, which we handed over a month ago with a pure soul. He was not embarrassed either by the fact that it was three o’clock in the morning, or by the fact that the system showed the same “temperature” in the diesel engine and in the room. Because it wasn’t the temperature at all, but the error code “no connection with the sensor.” Oleg honestly told the dispatcher where he should go at night with such requests. Literally:

— Go to the diesel engine and have a look, most likely the battery in the sensor is simply dead. The battery module in diesels powers this control panel, there is a switch there - if one of yours touched it, it needs to be turned back.

In general, the dispatcher is exactly the person who should know the object like an experienced milling operator with his three fingers, but then an amazing question was asked:

— Where are the diesel engines?
— Go up to the second floor, ask the electricians, they will take you.

For the next 20 minutes, Oleg worked remotely as a navigator, trying to bring together the dispatcher and electricians, who didn’t really want to give someone a tour in the middle of the night.

The circulation of stars in the data center

In the distant kingdom, the thirtieth state, we somehow got covered frequency operator on chiller. Exactly three hours before certification at the Uptime Institute. It would take a long time to tell a fairy tale what a chiller and frequency generators are if you don’t know. So just believe: they must work like chimes, otherwise certification will turn into a pumpkin, and the customer will turn into an evil stepmother. And, the most offensive thing, he will be right, because the commission charges a lot of money for a visit, and if something goes wrong, no one will return it.

The vendor's service technician rushed over, threw up his hands and said that the patient was most likely dead, and the wait for a new board would be at least a month. The commission is already on the doorstep, there are few ways out. The first is to unscrew the frequency switch from the “healthy” chiller and put it on the “sick” one, and then change places until the tests are over. This is not cheating, if anything: according to the testing regulations, one of the three chillers is still redundant, so the scenario is quite workable. The second way out is to try to find a new frequency generator in the remaining hour and a half. We called our refrigeration specialist in Moscow. The refrigerator dialed a Russian representative of the vendor to a friend inside. He, in turn, put pressure on the manufacturer’s representative office in Holland and... half an hour later they were already screwing on a new board for us. The certification went well.

Defects

Whether it's long or short, there always comes a moment on a construction site when all the subs leave and they remain: unfinished work. This is about unfinished works, and not about those who left them, if that. Whoever is last is the one who rake.

We once made a data center in the basement: racks in two rows, aligned along one of the edges so that the door could open. Because of this, a gap formed between one row and the wall, and the customer refused to sign the acceptance certificate because of this gap. The technician and the project manager went to Leroy early in the morning for foam plastic, paint, fasteners and conscientiously sealed the gap so that it matched the racks. Passed.

And one day, after the heating contractors left, a discrepancy was discovered in the project: there should be 7 radiators, but there were 6 of them. We went and did the math - everything was right, they really didn’t fit one radiator into the corridor. It’s too late to drink Borjomi, everything is already installed and pressed. The customer sprinkles ashes on his head because the acts have already been signed. I saved Leroy again - we bought an electric heater there, ran a group of cables into the corridor and installed it ourselves on Sunday morning, the customer is happy.

They also once left us a magical nothingness instead of a fire barrier. In a data center, the supply ventilation duct from the second floor goes to the basement, through a room that, according to fire classification, belongs to a different category than those above and below it. In practice, this means that there must be a fire damper in the duct, and a fire barrier around it. There was a hole around our shiny fire damper duct, guess who fixed it and how? Leroy Merlin did not sponsor this post, which is a pity.

Dale Carnegie smokes nervously

A long time ago, when the grass was greener and the dollar was 30-something, we built a data center for one bank in the historical center of Moscow. It was necessary to meet it in an extremely short time frame. But the streets there are narrow, the openings near the building are also narrow, and lifting several tons of equipment to the desired floor using stairs is almost impossible. They told the customer that they had to load it with a crane straight onto the roof, the customer responded in the spirit of “you’re a cowboy, you jump.” Well, that’s a good idea, now coordinate the arrival of a 120-ton crane with authorities like the traffic police. And preferably yesterday. Good luck in your endeavors; if you don’t have time, you’ll be fined.

The situation is stalemate, time is running out, and we decided to take a risk; after all, traffic police fines compared to fines for failure to meet deadlines are just flowers. On Saturday morning we brought in a 16-meter crane, hoping that we would have time to do everything quickly. A couple of hours later a local police officer arrived and timidly asked permission. Of course, we don't have it. And it is unknown how everything would have ended if we had not had a salesman with extraordinary diplomatic skills with us.

He took the district police officer aside, explained something to him for 5 minutes, the policeman’s face changed several times during this time, but in the end he sat down in his UAZ and shouted that if something happened, he himself would come and help us. What kind of arguments were there, the salesman still doesn’t give a damn.

Lift it up for me, people!

Behind the mountains, behind the fields, but within the Third Transport, stood... no, not a hut, but quite a serious government construction site. Difficult night shift, loading equipment. The last truck with a 15-ton piece of iron in the back drives out to the only intersection on the territory, something bursts with a loud sound, and the colossus bottoms out in the mud. 5 am, the construction site is gradually coming to life, the driver of the concrete mixer behind our truck, remembering the fallen women, clearly asks: would we please get out of the way? His concrete, they say, is getting cold. And we would be glad, because at 7 am one of the many deputies of one of the ministers will arrive and, if he sees this disgrace, everyone will fly in: from engineers to top executives.

The technician runs to the local crane operator and tearfully asks him to lift the piece of iron from the back so that we can slide another truck under it. And he won’t do anything, even for money. Our engineer corrected the situation - he made an agreement through the supervisor of that crane operator. We still got the piece of iron to where it needed to be, but we were already a little gray.

And then they became a lot grayer, at the same facility. There was an epic failure in the most literal sense.

There is such a thing: a telescopic forklift. It is used when there is nowhere to turn around at a construction site, and the load needs to be lifted up and carefully placed. With its help, we needed to unload a 1,5-ton module through a hole into a window. Nothing foreshadowed: according to the specification, the machine was supposed to withstand 2 tons with a hook. But when there was exactly half a meter left to the window, the “forks” of the loader broke off, and the piece of iron came over from a height. There is nothing to do - we call the manufacturer to come and carry out restoration work. Their service workers arrived and... refused to enter the construction site. Because you had to go there along a path, and on the path there was an excavator digging. We are familiar: we waited 10 seconds until the arrow was turned in the direction opposite to the path, and ran. And the guys were shocked. We had to present this feature as an exciting attraction, a kind of “Fort Boyard”, and they finally fixed the module for us.

Non-instant karma

Soon the fairy tale will tell, but it won’t be done soon, especially when it comes to signing acceptance certificates for completed work. We once built an excellent data center for one company. But the king-priest-customer decided to give us one last test:
— Your specifications say 100 nuts per tray, but I counted 97. Correct the specifications and estimates, or I won’t sign anything.

And each time we went to distant lands, and together with the Tsar-Father we counted fasteners for air ducts, then nuts, then bolts. Each time it turned out that it was not 97, but 99, etc. And we had no peace. As a result, we accumulated so many internal costs that our bosses couldn’t stand it. They said: let them do what they want - no one else should go there. So they remained without signatures.

...And a year later the customer comes himself and politely asks where he can sign? It turned out that the accounting chamber came, and he had unaccounted equipment worth seven zeros.

Wingardium Leviosa!

Once upon a time there was a good customer, and he decided to buy himself an old building for a data center. Only his joy did not last long: something strange began to happen in the battery room. He then called us to help, to look at the marvelous thing and to advise. We come to visit, go into the battery room, and there... the walls above the floor are levitating on three sides. Of course: 4 tons of batteries were simply placed on the floor - and it began to go underground. This is a common problem with batteries: it is important to correctly calculate the load on the structure and provide unloading frames so that the floors do not collapse like a house of cards.

But the icing on the cake of this building's architectural nightmare was its complete lack of foundation. The walls stood stupidly on a sandy screed, under which there was not the most friendly soil. They began to think about how to save the patient, and in the end they proposed a complex system of silicification: this is when the soil is drilled in several places and a strengthening solution is injected there. This did not return the floor to its previous heights, but at least it stopped collapsing.

Battle of two yokozuna

In some kingdom, in some office state, we did dispatching: for a data center on one of the floors, for climate control - for everything. Dozens of automation cabinets, kilometers of low-current cable!

A special feature of that office was its own SPA complex with a sauna. When we developed the project, it was assumed that the steam room would be used infrequently. But the engineer suggests, and the customer has it: the management team became so involved in a healthy lifestyle that, just in case, they stopped turning off the sauna altogether - it takes too long to warm up.

Bottom line: the automation detects the increase in temperature from the sauna and turns on the air conditioner harder to compensate. It helps itself. The air conditioner continues to strain - the atmosphere is heating up, because the automation does not suspect that someone can sweat so much. Condey falls into paranoia and decides: “It’s not about them, it’s about me. All efforts are in vain. It seems that I’m broken,” which is reported to the dispatcher’s console. The dispatcher sighs and presses the “task cleared” button. And so every single day.

Flick of the wrist

We have many more stories in our bins, but not all of them will fit here. Here are the last three tales: about crooked hands.

The first story is about how in one building, in addition to the server room, we supplied air pressure to the elevator halls and shafts. They did it, left the site, waited for another contractor to finish the finishing work, and returned to test the system. We start the backup - the false ceiling inflates sharply, the slabs fly out onto the floor with a roar and everything is designed in a “loft” style in a second. It turned out that the guys who installed the ceiling forgot to put in slabs with perforations for air. They had no idea about their existence at all, they simply took them from an open pack - and that’s it, they didn’t pay attention to the packaging with perforated ones, they didn’t look at the project.

The second epic is also about the ceiling. In one data center there was a rather narrow corridor, and the air exchange in the room with battery racks had to be strong, so communications ran along the entire ceiling. The good fellows who made the false ceiling didn’t bother and...screwed the hanger fasteners directly into our unfortunate air ducts. Any air duct already vibrates a little during the operation of ventilation units, and when there are a lot of holes and irregularities in it, you are guaranteed the sounds of the Apocalypse. We gave such life-giving slaps to the glorious knights-installers that the wounds on the air ducts miraculously healed. At their expense, of course.

The third tale happened when we were making automation for the technological air conditioning system. We handed the customer’s contractor a temperature sensor and without a second thought asked to put it on the heater in the supply air. In air supply units there is always a heater for outdoor air: either water or electric. This wonderful man did everything without question. That is, he took and screwed a self-tapping screw directly into the water heater tube, with everything that (literally) follows from here. It’s good that there were air conditioners with MAPP gas at the site - the leak was quickly repaired.

Links:

Source: habr.com

Add a comment