In one of the chats, I was asked the question:
— Is there anything to read on how to properly pack servers in racks?
I realized that I didn't know such a text, so I wrote my own.
Firstly, this text is about physical servers in physical data centers (DCs). Secondly, we assume that there are quite a few servers: hundreds to thousands; for a smaller number, this text doesn’t make sense. Thirdly, we consider that we have three constraints: physical space in the racks, power supply per rack, and let's say the racks are arranged in rows, so we can use one ToR switch to connect servers in adjacent racks.
The answer to the question heavily depends on which parameter we are optimizing and what we can vary to achieve the best result. For example, we may only need to take up minimal space to leave more room for future growth. Or maybe we have the freedom to choose rack height, power per rack, outlets in the PDU, the number of racks in the switch group (one switch for 1, 2, or 3 racks), cable lengths, and installation work (this is critical at the ends of the rows: with 10 racks in a row and 3 racks on a switch, we will need to run cables to another row or underuse ports on the switch), etc., etc. Separate stories: choosing servers and selecting a DC; let's assume they have been chosen.
It would be good to understand some nuances and details, in particular, the average/maximum consumption of servers, and how electricity is supplied to us. So, if we have Russian power at 230V and a single phase per rack, then a 32A circuit breaker can handle about 7kW. Let's say we nominally pay for 6kW per rack. If the provider measures our consumption only across a row of 10 racks, rather than by each rack, and if the circuit breaker is set at a threshold of about 7kW, technically, we could consume 6.9kW in one rack, 5.1kW in another, and everything would be fine — without penalties.
Usually, our main goal is to minimize costs. The best criterion for measurement is to reduce TCO (total cost of ownership). It consists of the following components:
- CAPEX: purchasing DC infrastructure, servers, networking equipment, and cabling.
- OPEX: DC rent, consumed electricity, maintenance. OPEX depends on the service life. It is reasonable to assume it is equal to 3 years.
Depending on the size of individual pieces in the overall pie, we need to optimize the most expensive part, while the rest can use all remaining resources as efficiently as possible.
Let's assume we already have an existing data center, a rack height of H units (for example, H=47), power per rack Prack (Prack=6kW), and we decided to use h=2U two-unit servers. We will remove 2..4 units from the rack for switches, patch panels, and organizers. Thus, physically, we can fit Sh=rounddown((H-2..4)/h) servers in the rack (i.e., Sh = rounddown((47-4)/2)=21 servers per rack). Let's remember this Sh.
In a simple case, all servers in the rack are identical. So, if we fill the rack, servers, we can allocate power of Pserv=Prack/Sh (Pserv = 6000W/21 = 287W) on average per server. For simplicity, we will ignore the consumption of the switch here.
Let's take a step back and define what the maximum consumption of a server Pmax is. If we simplify it greatly, very inefficiently, and completely safely, it's what’s indicated on the server's power supply — that’s it.
If we want to be more precise and effective, we take the TDP (thermal design package) of all components and sum them up (this isn't entirely correct, but it can work).
Usually, we don't know the TDP of components (except for CPU), so we take the most accurate but also the most complex approach (requiring a lab) — we use an experimental server with the desired configuration and load it, for example, with Linpack (for CPU and memory) and fio (for disks), and measure the consumption. If we want to be serious, we also need to create the warmest environment in the cold aisle during testing, because it affects both fan consumption and CPU consumption. This gives us the maximum consumption of a specific server with a specific configuration under those specific conditions and load. Just keep in mind that a new firmware update, a different software version, or different conditions can influence the result.
In summary, let's return to Pserv and how to compare it with Pmax. This is a matter of understanding how services work and how strong the nerves of your tech director are.
If you want to play it completely safe, we assume that all servers can simultaneously start consuming their maximum. At the same time, a single input to the data center may occur. The infrastructure under these conditions must provide the service, hence Pserv ≡ Pmax. This is an approach where reliability is absolutely crucial.
If the tech director considers not only perfect security but also the company's finances and is bold enough, it can be concluded that
- we start managing our vendors, specifically prohibiting scheduled maintenance during planned peak loads to minimize the decrease from one input;
- and/or our architecture allows us to lose a rack/row/data center, while the services continue to operate;
- and/or we effectively distribute the load horizontally across racks, ensuring our services never reach maximum consumption all at once in any single rack.
It's very useful not just to guess but to monitor consumption and know how servers actually consume electricity under normal and peak conditions. Therefore, after some analysis, the tech director condenses everything he has and states: "we make a decisive choice that the achievable average from the maximum server consumption per rack is **so much** lower than the maximum consumption," conditionally Pserv=0.8*Pmax.
As a result, in a rack with 6kW, we can fit not 16 servers with Pmax = 375W, but 20 servers with Pserv = 375W * 0.8 = 300W. This means 25% more servers. This is a significant saving—after all, we also need 25% fewer racks (and we can save on PDUs, switches, and cables as well). A serious drawback of such a decision is that we need to constantly monitor whether our assumptions are still valid. We need to ensure that the new firmware version does not significantly alter the operation of the fans and consumption, and that development hasn’t suddenly started using the servers much more efficiently with the new release (in other words, achieving greater load and higher consumption per server). After all, if that happens, both our initial assumptions and conclusions become incorrect right away. This is a risk that must be responsibly accepted (or avoided, paying for obviously underloaded racks instead).
An important note — it's worth trying to distribute servers from different services horizontally across racks, if possible. This is necessary to avoid situations where a batch of servers for one service arrives, and the racks get densely packed vertically for increased 'density' (because it's easier that way). In reality, one rack ends up filled with identical low-load servers from one service, while another is filled with equally high-load ones. The likelihood of failure for the latter is significantly higher since the load profile is the same, and all servers in that rack start consuming equally much as the load increases.
Let's return to the distribution of servers in racks. We have considered the physical space limitations in the rack and power supply constraints; now let's look at networking. You can use 24/32/48 port switches N (for example, we have 48-port ToR switches). Fortunately, there aren't too many options if you don't think about breakout cables. We consider scenarios where we have one switch per rack, one switch for two or three racks in an Rnet group. I think having more than three racks in a group is excessive because the cabling issue between racks becomes significantly greater.
So, for each network scenario (1, 2, or 3 racks in a group), we distribute servers across the racks:
Srack = min(Sh, rounddown(Prack/Pserv), rounddown(N/Rnet))
Thus, for the option with 2 racks in a group:
Srack2 = min(21, rounddown(6000/300), rounddown(48/2)) = min(21, 20, 24) = 20 servers per rack.
We similarly calculate the other options:
Srack1 = 20
Srack3 = 16
And we are almost there. We calculate the number of racks needed to distribute all our servers S (let's say 1000):
R = roundup(S / (Srack * Rnet)) * Rnet
R1 = roundup(1000 / (20 * 1)) * 1 = 50 * 1 = 50 racks
R2 = roundup(1000 / (20 * 2)) * 2 = 25 * 2 = 50 racks
R3 = roundup(1000 / (16 * 3)) * 3 = 25 * 2 = 63 racks
Next, we calculate TCO for each option based on the number of racks, the necessary number of switches, cabling, etc. We choose the option where TCO is lower. Profit!
Note that while the required number of racks for options 1 and 2 is the same, their costs will differ, as the number of switches for the second option is half and the length of necessary cables is greater.
P.S. If there is a possibility to play around with rack power and rack height, the variability increases. However, the process can be reduced to the above-mentioned by simply evaluating options. Yes, there will be more combinations, but still a quite limited number — the power supply for the rack can be increased in increments of 1 kW, standard racks come in a limited number of sizes: 42U, 45U, 47U, 48U, 52U. For calculations, Excel's What-If analysis in Data Table mode can be helpful. We look at the resulting tables and select the minimum.
Source: habr.com
