Hello, Habr! I am Taras Chirkov, the director of the Linxdatacenter in Saint Petersburg. Today, in our blog, I will discuss the role of maintaining cleanliness in the standard operation of a modern data center, how to measure it correctly, and how to achieve and sustain it at the required level.
Cleanliness Trigger
Once, a client from our data center in Saint Petersburg approached us regarding a layer of dust in the lower part of the rack housing equipment. This became the starting point of our investigation, initially hypothesizing the following:
- dust enters the server rooms through the shoes of employees and clients of the data center,
- is carried in through the ventilation system,
- and both of these sources.
Blue Shoe Covers – A Thing of the Past
We started with shoes. At that time, cleanliness was traditionally maintained through a container of shoe covers at the entrance. The effectiveness of this approach did not meet our expectations: it was difficult to control their use by data center guests, and the format itself was inconvenient. We quickly abandoned it in favor of a more advanced technology in the form of a shoe cover machine. The first model we installed failed: the machine frequently ripped the covers when trying to put them on shoes, and its use was more annoying than it was helpful.
Consulting with colleagues in Warsaw and Moscow did not solve the problem, and ultimately we chose the technology of thermoplastic film for shoes. With the help of the film, you can apply 'shoe covers' to shoes of any sole – even thin women's heels. Yes, the film can also slip off occasionally, but much less often than the classic blue covers, and the technology itself is significantly more convenient and modern for visitors. Another important benefit (for me) is that the film easily covers the largest shoe sizes, unlike traditional covers, which tend to tear when attempting to fit size 45. For the sake of modernity, we also installed waste bins with automatically opening lids triggered by motion sensors.
This process looks like this:
Guests immediately appreciated the innovation.
Dust in the Wind
After addressing the most obvious channel for potential contamination of the space, we turned our attention to more subtle matters – the air. It is quite possible that a significant portion of dust enters the server rooms through ventilation due to insufficient filtration, or it is carried in from the outside. Or is it all due to poor cleaning quality? The investigation continued.
We decided to measure the concentration of particles in the air inside the data center and invited a laboratory specializing in air quality control in clean rooms for this work.
The laboratory staff measured the number of control points (20), created a sampling schedule to track dynamics and create the most accurate picture possible. The total cost of the measurements process by the laboratory was about 1 million rubles, which seemed completely unreasonable to us, but it gave us several ideas for independent implementation. It also became clear that while using a laboratory is good, conducting analyses dynamically and frequently relying on their services is extremely inconvenient.
After reviewing the planned actions of the laboratory, we decided to look for more utilitarian devices for self-monitoring. In the end, we managed to find the necessary tool for this task – an air quality analyzer. Here it is:
The device shows the concentration of particles of different diameters (in micrometers).
Reviewing the standards
This device analyzes the quantity of particles, temperature, humidity, and displays results in measurement units according to ISO standards for this parameter. The display shows the levels of particle concentration with different diameters in the air sample.
In parallel, we suspected the filters: at that time we were using G4 filter models inside the server rooms. This model provides coarse air cleaning, so we admitted the possibility of missing particles that could lead to contamination. We decided to purchase F5 fine filtration filters for testing, which are used in air conditioning and ventilation systems as second-stage filters (post-filtration).
The investigation has been conducted – we can proceed with the control measurements. As a reference, we decided to use the requirements of the ISO 14644-1 standard concerning the number of airborne particles.
Classification of clean rooms based on the number of airborne particles.
It may seem straightforward – measure and compare against the table. However, it turned out to be quite challenging in practice to find air cleanliness standards for the data center server rooms. This is not explicitly detailed by any organization or industry institution. Only on the internal Uptime Inside Track forum (accessible to those who have completed training programs from the Uptime Institute) was there a separate discussion on this topic. After studying it, we leaned towards the ISO 8 standard – the second last in classification.
The first measurements showed that we had underestimated ourselves – the results of the indoor air checks corresponded to ISO 5 requirements for indoor spaces, significantly exceeding the desired norms set by participants of Uptime Inside Track. Moreover, it was a considerable margin. We have a data center, not a biological laboratory, of course, but to achieve a particle concentration in the air equivalent to ISO 8 – that would mean a facility classified at least as a 'cement plant'. Thus, it is unclear how the same standard can apply to a data center. At the same time, we achieved ISO 5 results while measuring with G4 air filters. This means dust cannot enter the racks through the air; F5 filters turned out to be excessive, and we didn't even use them.
A negative result is still a result: we continued to search for the source of contamination in other directions, and included air quality control in quarterly checks, combining it with checks of BMS sensors by accredited devices (ISO 9000 requirements and client audits).
Below is an example of the report that is filled out based on the data obtained from the measurements. For greater accuracy, measurements are taken with two devices – Testo 610 and a BMS sensor. The header of the table indicates the limit values for the devices. Deviations in the specified parameters are automatically highlighted in color to facilitate the identification of problematic areas or time periods.
We have everything clear: the difference in instrument readings is minimal, and the particle concentration is well below the limit.
Through the back entrance
Since there were other entrances to the clean rooms besides the main entrance for clients, where we installed the bootie machine, there remained a necessity to prevent dirt from entering the data center through them.
Putting on/taking off booties during equipment unloading procedures is inconvenient, so we found a machine for cleaning soles. It's convenient, functional, but the human factor comes into play with the non-compulsory approach to this machine. Essentially, it's the same situation as with the booties at the main entrance.
To solve the problem, we started looking for cleaning options that couldn't be avoided: sticky mats with removable layers worked best. When authorizing at the entrance doors, visitors must step on such a mat, removing excess dust from the soles of their shoes.
Cleaners remove the top layer of such a mat every day; it has 60 layers in total – enough for about 2 months.
After visiting the Ericsson data center in Stockholm, among other things, I noticed how they address these issues there: in addition to removable layers, they use reusable antibacterial mats from Dycem. I liked the idea due to the principle of reusability and the ability to cover a larger area.
A magical antibacterial mat. Too bad it’s not an airplane, it could have been – at that price!
We struggled to find representatives of the company in Russia and evaluated the cost of the solution for our data center. In the end, we received a figure nearly 100 times more expensive than the solution with multi-layer mats – about the same 1 million rubles as in the air quality measurement project. Additionally, it turned out there was a need to use special cleaning agents, naturally available only from this manufacturer. This solution was also ruled out, so we settled on the multi-layer option.
Manual labor
It is particularly important to note that all specified measures did not eliminate the use of cleaning staff. In preparation for the certification of the Linxdatacenter data center under the Uptime Institute Management & Operations standard, it was necessary to clearly regulate the actions of the cleaning service personnel on the premises of the data center. Detailed instructions were created, outlining where, what, and how they must perform their tasks.
Here are some excerpts from the instructions:
As can be seen, everything is specified, literally every aspect of work in a specific room, permissible cleaning agents, materials, etc. No detail is overlooked, even the smallest one. Training is documented with the signature of each cleaning staff member. In server rooms, electrical rooms, etc., cleaning can only occur in the presence of authorized data center personnel, such as the on-duty engineer.
But that’s not all
Also included in the list of measures ensuring cleanliness in the data center are rounds with visual inspections of the premises, including weekly inspections of racks to identify any leftover wires, packaging remnants from equipment and components. An incident report is created for each such episode, and the client is notified of the need to rectify violations promptly.
Additionally, we have created a separate room for unpacking and setting up equipment — this is also part of the company's cleaning policy.
Another measure we adopted from Ericsson’s practice is maintaining a constant air pressure in server rooms: the pressure inside is higher than outside to prevent airflow into the space — we will provide more details about this solution in a separate article.
Finally, we have acquired robotic assistants for areas that are restricted from access by cleaning personnel.
The top grille not only provides a +10 to the robot's protection but also prevents it from getting stuck under vertical cable trays of the racks.
An unexpected find as a conclusion
Cleanliness in the data center is crucial for the operation of server and network equipment, which draws air through itself. Exceeding permissible dust levels can lead to dust accumulation on components and a total temperature increase of up to 1 degree Celsius. Dust reduces cooling efficiency, which can translate into noticeable indirect costs over the year, as well as affect the overall reliability of the facility.
It can be said that this is a theoretical assumption, but experts from the Uptime Institute, who certified the Linxdatacenter data center according to the Management & Operations quality standard, pay serious attention to cleanliness. It was especially gratifying to receive the highest praise in this area: the certification requirements that our data center in St. Petersburg exceeds significantly. The institute's expert called us "the cleanest data center he has ever seen" and, furthermore, our data center is used by Uptime as an example of how to address cleanliness in server rooms. Additionally, we easily pass any client audit on this parameter – the most stringent requirements from the most particular clients are met with excellence.
Let's return to the beginning of the story. Where did the contamination that triggered the cleanliness project in the data center come from? The part of the client's rack that was the cause of the project was contaminated from the moment it was brought in and installed in the data center. The client did not clean the rack before bringing it into the server room – inspection of adjacent racks installed at the same time revealed the same dust situation there. This situation led to the addition of a cleaning control point in the client's rack installation checklist. One must never forget about the likelihood of such occurrences = forewarned is forearmed. This is all about "cleanliness and discipline" in our data center; in the next article, I will discuss pressure sensors, but for now, feel free to ask questions in the comments.
Source: habr.com
