I dedicate this post to those people who lied in their certificates, because of which we almost installed sparklers in our halls.
The stories are more than four years old, but I am publishing now because the NDA has ended. Then we realized that we had loaded the data center (which we rent out) almost completely, and its energy efficiency had not improved much. Previously, the hypothesis was that the more we fill it, the better, because the engineer is distributed to everyone. But it turned out that we were deceiving ourselves in this regard, and with a good load, there were losses somewhere. We worked in many directions, but our brave team took over the refrigeration.
The real life of a data center is a little different from that in the project. Constant tweaks from the maintenance service to improve efficiency and optimize settings for new tasks. Take the mythical middle stance. In practice, it does not happen, the distribution of the load is uneven, somewhere dense, somewhere empty. So I had to reconfigure something for better energy efficiency.
Our data center The compressor is needed for a variety of customers. Therefore, there, in the middle of the usual two to four kilowatt racks, there may be quite a 23-kilowatt or more. Accordingly, the air conditioners were set to cool them, and the air simply rushed past through the less powerful racks.
The second hypothesis was that warm and cold corridors do not mix. After measurements, I can say that this is an illusion, and the real aerodynamics differs from the model in almost everything.
Examination
First we started watching the air currents in the halls. Why did they go there? Since they understood that the data center is designed for five or six kW per rack, but they knew that in fact they are from 0 to 25 kW. It is almost impossible to regulate all this with tiles: the very first measurements showed that they pass almost equally. And there are no 25 kW tiles at all, they must be not just empty, but with a liquid vacuum.
We bought an anemometer and began to measure the flows between the racks and above the racks. In general, it is necessary to work with it in accordance with GOST and a bunch of standards that are difficult to implement without stopping the turbine hall. We were not interested in accuracy, but in the fundamental picture. That is, measured approximately.
According to the measurement, out of 100 percent of the air that comes out of the tiles, 60 percent gets into the racks, the rest flies by. This is due to the fact that there are heavy racks of 15-25 kW, on which the cooling is built.
We cannot turn off the air conditioners, because it will be quite warm on the warm racks near the upper servers. At this moment, we understand that it is necessary to isolate something from something so that the air does not jump from row to row and that the thermal exchange in the block still occurs.
In parallel with this, we ask ourselves whether it is financially feasible.
We are surprised to find that we have the energy consumption of the data center as a whole, but we simply cannot calculate the fan coil units for a particular room. That is, analytically we can, but in fact - no. And we are not able to estimate the savings. The task becomes more and more interesting. If we save 10% of air conditioner power, how much money can we put aside for insulation? How to count?
We went to the automators who finished the monitoring system. Thanks to the guys: they had all the sensors, it was only necessary to add the code. They began to output separately chillers, UPS, lighting. With the new stray, it became possible to watch how the situation is changing according to the elements of the system.
Experiments with curtains
In parallel, we begin experiments with curtains (fences). We decide to mount them on the pins of cable trays (there is nothing else anyway), since they should be light. With canopies or combs decided quickly.
The ambush is that we have previously worked with a bunch of vendors. Everyone has solutions for their own data centers of companies, but there are essentially no ready-made solutions for a commercial data center. We have customers coming and going all the time. We are one of the few "heavy" data centers with no limit on the width of the rack with the ability to accept these thresher servers up to 25 kW. No infrastructure planning in advance. That is, if we take vendors' modular caging systems, there will always be holes for two months. That is, the turbine hall will never be energy efficient in principle.
We decided to do it ourselves, since we have our own engineers.
The first thing they took was tapes from industrial refrigerators. These are such flexible polyethylene snots that can be staggered. You must have seen them somewhere at the entrance to the meat department of the largest grocery stores. They began to look for non-toxic and non-combustible materials. Found, bought two rows. Hung up, began to see what happens.
We knew it wouldn't be very good. But it turned out in general very directly very not very much. They begin, like pasta, to flutter on streams. Found magnetic tapes like fridge magnets. We glued them on these strips, glued them to each other, the wall turned out to be moderately monolithic.
We began to figure out what will be in the hall.
We went to the builders, we show our project. They look, they say: your curtains are something very heavy. 700 kilograms throughout the turbine hall. Go, they say, good people, to hell. More precisely, to the SCS team. Let them count how many noodles they have in the trays, because 120 kg per square meter is the maximum.
SCS say: do you remember, one big customer came to us? It has tens of thousands of ports in one room. Itβs still fine along the edges of the turbine hall, but it wonβt be possible to fasten it closer to the cross line: the trays will fall off.
The builders also asked for a certificate for the material. I note that before that we worked on the parole of the supplier, since it was just a test run. We turned to this supplier, we say: OK, we are ready to go into beta, let's get all the paperwork. They send something not very established pattern.
We say: listen, where did you get this piece of paper? They are: our Chinese manufacturer sent us in response to requests. On paper, this thing does not burn at all.
At this point, we realized that it was time to stop and check the facts. We go to the girls from the fire safety of the data center, they call us a laboratory that tests flammability. Quite earthly money and terms (however, we cursed everything while we made up the required number of pieces of paper). There, scientists say: bring the material, we will do tests.
In conclusion, it was written that about 50 grams of ash remains from a kilogram of the substance. The rest burns brightly, flows down and supports combustion very well in a puddle.
We understand - it's good that they did not buy. We started looking for other material.
Found polycarbonate. He got tougher. Transparent sheet - two mm, doors - from four millimeters. Basically, it's plexiglass. Together with the manufacturer, we start a conversation with fire safety: give us a certificate. They send. Signed by the same institution. We call there, we say: well, what, guys, did you check this?
They say yes, they did. First, they burned it at home, then they only brought it for tests. There, from a kilogram of material, approximately 930 grams of ash remains (if it is burned with a burner). It melts and drips, but the puddle will not burn.
We immediately check our magnets (they are on a polymer lining). They burn surprisingly badly.
Assembly
From this we begin to collect. Polycarbonate is great because it's lighter than polyethylene and doesn't bend as well. True, they bring sheets of 2,5 by 3 meters, and the supplier does not care what to do with it. And we need 2,8 with a width of 20β25 centimeters. The doors were sent to offices that cut the sheet as it should. And they cut the slats themselves. The cutting process itself costs twice as much as the sheet.
Here's what happened:
The bottom line is that the cabling system pays off in less than a year. This is how we saved 200-250 kW permanently on the power of fan coil units. How much more is on the chillers, how much exactly - we do not know. Servers suck at a constant speed, fan coils blow. And the chillers are turned on and off with a comb: it is difficult to extract data from it. It is impossible to stop the turbine hall for tests.
We are glad that at one time there was a rule to install 5x5 racks in modules so that their average consumption was six kW maximum. That is, the warm is not concentrated by the island, but distributed throughout the turbine hall. But there is a situation where 10 pieces of 15-kilowatt racks are nearby, but there is a lot of storage opposite. He is cold. Balanced.
Where there is no rack, you need a fence to the floor.
And we also have some customers isolated with bars. They also had a few features.
They cut it into lamellas, because the width of the racks is not fixed, and the frequency of the fastening comb is determined: there will always be three or four cm either to the right or to the left. If you have a block 600 under the rack, then it will not stand up with a probability of 85 percent. And short and long lamellae coexist and stick together. Sometimes we cut the lamella with the letter G along the contours of the racks.
SENSORS
Before reducing the power of the fan coil units, it was necessary to set up very accurate temperature monitoring at different points in the hall so as not to catch any surprises. This is how wireless sensors were born. Wired - you need to hang your own thing on each row to cross these sensors and sometimes extension cords on it. It turns into a garland. Very bad. And when these wires enter the cages for customers, the security guards immediately get excited and ask to explain with a certificate what is being filmed through these wires. The nerves of the security guards must be protected. For some reason, they do not touch wireless sensors.
And more racks come and go. The sensor on the magnet is easier to remount, because it must be hung higher or lower each time. If the servers are in the lower third of the rack, they must be hung down, and not according to the standard, one and a half meters from the floor on the door of the rack in the cold corridor. It is useless to measure there, it is necessary to measure what is in the gland.
One sensor for three racks - more often you can not hang it. The temperature is no different. They feared that there would be air being drawn through the racks themselves - it did not happen. But we still give a little more cold air than the calculated values. We made windows in lamellas 3, 7 and 12, we make a hole above the counter. When going around, we put an anemometer in it: we look that the flow is going where it needs to be.
Then they hung bright strings: an old practice for snipers. It looks strange, but allows you to detect a possible problem faster.
Funny
While we were doing all this in silence, a vendor came who produces an engineer for data centers. He says: let's come and talk about energy efficiency. They come and start talking about the non-optimal hall, air currents. We nod in understanding. Because we have three years as established.
They hang three sensors on each rack. Pictures of monitoring are zashibnye, beautiful. More than half of this solution in price is software. At the level of "alert in Zabbix", but proprietary and very expensive. The ambush is also in the fact that they have sensors, software, and then they are looking for a contractor on the spot: they donβt have their own vendors for cabling.
It turns out that their hands cost five to seven times more than we did.
references
- .
- .
- .
- .
- .
- My mail: [email protected]
Source: habr.com