🥇How we focused on energy efficiency in the machine room

I dedicate this post to those people who lied on their certificates, which almost led us to install Bengal lights in our halls.

The story is more than four years old, but I publish it now because the NDA has expired. At that time, we realized that we had almost completely filled the data center (which we rent out), but its energy efficiency had not really improved. Earlier, the hypothesis was that the more we fill it, the better it will be, as the engineering load is distributed among everyone. However, it turned out we were deceiving ourselves in this regard, and with good occupancy, there were some losses. We worked on many fronts, but our brave team tackled cooling.

The real life of a data center is somewhat different from what is shown in the project. There are constant tweaks from the operations service to improve efficiency and optimize settings for new tasks. Take, for example, the mythical average rack. In practice, it doesn’t exist; the load distribution is uneven, dense in some areas and sparse in others. So, we had to reconfigure some things for better energy efficiency.

Our Compressor data center is needed for a variety of customers. Therefore, in the midst of ordinary two–four kilowatt racks, there may well be a 23-kilowatt rack or even larger. Accordingly, the air conditioners were configured to cool them, and the air simply rushed past the less powerful racks.

The second hypothesis was that warm and cold aisles do not mix. After measurements, I can say that this is an illusion, and true aerodynamics differ from the model in almost every way.

Survey

At first, we began to look at airflows in the halls. Why did we delve into this? Because we understood that the data center is designed for five to six kW per rack, but we knew that they actually range from 0 to 25 kW. Regulating all this with tiles is almost impossible: the first measurements showed that they let air through almost equally. And there are no tiles for 25 kW; they must not only be empty but have liquid vacuum.

We bought an anemometer and started measuring airflow between racks and above them. In general, one must work with it according to GOST and a bunch of standards, difficult to fulfill without stopping the machine room. We were interested not in accuracy but in the fundamental picture. That is, we measured approximately.

According to measurements, out of 100 percent of the air that exits from the tiles, 60 percent reaches the racks, while the rest passes by. This is due to the presence of heavy racks ranging from 15 to 25 kW, which handle the cooling.

We cannot simply turn off the air conditioners because it would get too warm around the upper server racks. At that moment, we realize that we need to isolate something to prevent the air from bouncing back and forth and to ensure proper thermal exchange within the unit.

At the same time, we are asking ourselves whether this financial approach is justified.

We are surprised to find that overall, we have data on the data center's energy consumption, but we struggle to calculate the fan coils for a specific room. We can analyze it theoretically, but in practice, we can't. Evaluating potential savings becomes unmanageable. The question arises: if we save 10% on the air conditioners, how much can we set aside for insulation? How do we calculate this?

We approached the automation team that was refining the monitoring system. Thanks to the guys, they had all the sensors; we just needed to write additional code. They started displaying information for chillers, UPS systems, and lighting separately. With the new equipment, we now have the ability to monitor how the situation changes across the system elements.

Experiments with curtains

Simultaneously, we started experiments with curtains (partitions). We decided to attach them to the cable tray rods (since there’s nothing else to attach to), as they need to be lightweight. We quickly settled on canopies or combs.

The problem is that we had previously worked with many vendors. All of them have solutions for their own companies' data centers, but there are essentially no ready-made solutions for commercial data centers. Our clients come and go constantly. We are one of the few 'heavy' data centers without width restrictions on racks, able to accommodate these powerful servers up to 25 kW. There's no prior planning of the infrastructure. If we were to adopt modular cage systems from vendors, there would always be gaps for two months. This means the machine hall will never be energy-efficient in principle.

We decided to do it ourselves, as we have our own engineers.

The first thing we took was the strips from industrial refrigerators. These are flexible polyethylene strips that can pack quite a punch. You’ve probably seen them at the entrance of the meat section in the biggest grocery stores. We started looking for non-toxic and non-combustible materials. We found and bought enough for two rows. We hung them up and began to see how it turned out.

We knew it wouldn't be great. But it turned out to be really not great at all. They started, like pasta, flapping in the currents. We found magnetic strips, like the magnets on the fridge. We stuck them to these strips and attached them to each other, and the wall turned out to be sufficiently monolithic.

We began to estimate what would be in the hall.

We went to the builders, showed them our project. They looked and said: your curtains are quite heavy. 700 kilograms across the machine hall. They said, go find someone from the SCS team and have them calculate how much noodle they have in the trays because 120 kg per square meter is the maximum.

The SCS team said: remember when a large client came to us? They had tens of thousands of ports in one hall. At the edges of the machine hall it’s okay, but closer to the cross-connect it won’t work: the trays will fall off.

The builders also asked for a certificate for the material. I should mention that until then we had been working on the supplier's good word since it was just a test run. We approached this supplier and said: okay, we are ready to go into beta, let’s have all the papers. They sent us something that wasn't very standardized.

We said: listen, where did you get this paper? They replied: our Chinese manufacturer sent it to us in response to requests. According to the document, this stuff doesn’t burn at all.

At that moment we realized it was time to stop and check the facts. We went to the fire safety girls at the data center; they named a laboratory that tests combustibility. The costs and timelines were quite reasonable (although we cursed everything while preparing the required paperwork). The scientists there said: bring the material, we’ll conduct tests.

The conclusion stated that from one kilogram of the substance, about 50 grams remain as ash. The rest burns brightly, flows down, and supports combustion very well in a pool.

We realized it was good we didn’t buy it. We started looking for another material.

We found polycarbonate. It turned out to be stiffer. The clear sheet is two mm thick, while the doors are made of four mm. Essentially, it's plexiglass. We're starting a conversation about fire safety with the manufacturer: let's get that certificate. They send it over, signed by the same institute. We call them and ask: well, guys, have you checked this?

They say: yes, we checked it. First, they burned it themselves, and only then did they bring it for testing. From a kilogram of material, about 930 grams of ash remains (if you burn it with a torch). It melts and drips, but a puddle will not catch fire.

Immediately, we check our magnets (they're on a polymer base). Surprisingly, they burn poorly.

Building

We start assembling from that. Polycarbonate is great because it’s lighter than polyethylene and bends much worse. The problem is that they deliver sheets of 2.5 by 3 meters, and the supplier doesn’t care what to do with that. We need it to be 2.8 wide, 20–25 centimeters. The doors were sent to companies that cut sheets as needed. And we cut the slats ourselves. The cutting process itself costs about twice as much as the sheet.

Here's what we ended up with:

As a result, the caging system pays off in less than a year. We saved about 200–250 kW consistently on the capacity of the fan coils. We can save a bit more on chillers, but we're not sure how much. Servers consume at a constant rate, fan coils blow. Chillers turn on and off in a staggered manner: it's hard to extract data from them. The machine hall can't be stopped for testing.

We’re pleased that at one time there was a rule to install racks in 5x5 modules so that their average consumption was a maximum of six kW. This keeps the warmth distributed rather than concentrated in a single spot in the machine hall. However, there is a situation where there are 10 15-kilowatt racks next to each other, but across from them is a cold storage. It balances out.

Where there is no rack, a partition to the floor is needed.

Also, some of our customers are isolated with grids. There were a few peculiarities with them too.

Slats were cut because the width of the racks is not fixed, and the spacing of the grid mounts is determined: three to four cm either to the right or to the left will always be the case. If you have a block of 600 under the rack space, it has an 85 percent chance of not fitting. Short and long slats coexist and are glued together. Sometimes we cut slats in a 'G' shape following the contours of the racks.

Sensors

Before reducing the power of fan coils, it was necessary to set up very precise temperature monitoring at different points in the room to avoid any surprises. This led to the use of wireless sensors. For wired ones, you need to install a device for cross-connection for each row, and sometimes you need extensions for it. This turns into a garland of wires. It's very problematic. And when these wires go into clients' racks, the safety officers immediately get concerned and ask for an explanation with certification on what is being pulled through those wires. You have to keep the safety officers calm. For some reason, they don't bother with wireless sensors.

And the racks come and go. A magnetic sensor is easier to relocate because you need to adjust its height each time. If the servers are in the lower third of the rack, it should be positioned lower rather than the standard one and a half meters from the floor by the rack's doors in the cold corridor. Measuring there is useless; you need to measure what's in the hardware.

One sensor for three racks is often sufficient. The temperature doesn’t differ much. We were concerned that there would be air being pulled through the racks, but it didn’t happen. Nevertheless, we still provide slightly more cold air than the calculated values. We made small openings in the slats at 3, 7 and 12, and create a hole above the rack. During inspections, we place an anemometer in it to check if the airflow is directed correctly.

Then we hung bright strings: an old practice for snipers. It looks strange but allows for quicker detection of potential problems.

Amusing

While we were quietly doing all this, a vendor who produces engineering solutions for data centers visited. He said: let us come and discuss energy efficiency. They arrived, started talking about the suboptimal room, and airflow. We nodded understandingly because we've had it set up like that for three years.

They hang three sensors for each rack. The monitoring visuals are impressive and beautiful. More than half of this solution's cost is in the software. It's at the level of 'alert in Zabbix,' but proprietary and expensive. The catch is that they have sensors and software, but then they look for a contractor on site: they have no vendors for cage installations.

It turns out that their installations cost five to seven times more than what we implemented.