Hey Habr! I'm Taras Chirkov, director of the Linxdatacenter data center in St. Petersburg. And today in our blog I will talk about the role of maintaining cleanliness in the room in the normal operation of a modern data center, how to properly measure it, achieve and maintain it at the right level.
Purity Trigger
Once a client of a data center in St. Petersburg also contacted us about a layer of dust on the bottom of an equipment rack. This was the starting point of the investigation, the first hypotheses of which suggested the following:
- dust enters the server rooms from the soles of the shoes of employees and clients of the data center,
- carried through the ventilation system,
- both.
Blue shoe covers - to the dustbin of history
We started with shoes. At that time, the problem of cleanliness was solved in the traditional way: a container with shoe covers at the entrance. The effectiveness of the approach did not reach the desired level: it was difficult to control their use by the guests of the data center, and the format itself was inconvenient. It was quickly abandoned in favor of more advanced technology in the form of a shoe cover machine. The first model of such a device we installed failed: the machine tore shoe covers too often when trying to put them on shoes, its use was more annoying than making life easier.
Referring to the experience of colleagues in Warsaw and Moscow did not solve the problem, and as a result, the choice was made in favor of the technology of fusing thermal film on shoes. With the help of thermal film, you can put on "shoe covers" on shoes with any sole - up to a thin female heel. Yes, the film also sometimes slips off, but much less often than the classic blue shoe covers, and the technology itself is many times more convenient for the visitor and more modern. Another important (for me) plus is that the film easily covers the largest shoe sizes, unlike traditional shoe covers that tear when trying to put them on the 45th size. To make the process more modern, bins were installed with an automated opening of the lid by a motion sensor.
This process looks like this:
Guests immediately appreciated the innovation.
Dust in the wind
Having put in order the most obvious channel of possible pollution of space, we took up more subtle matters - air. It is likely that a significant part of the dust enters the server rooms through ventilation due to insufficient filtration, or is brought in from the street. Or is it all about the poor quality of cleaning? The investigation continued.
We decided to take measurements of the content of particles in the air inside the data center and invited a laboratory specializing in air quality control in special-purpose clean rooms to carry out this work.
The laboratory staff measured the number of control points (20), compiled a sampling schedule to track the dynamics and create the most accurate picture. The cost of the entire measurement process by the laboratory was about 1 million rubles, which seemed completely inappropriate to us, but gave a number of ideas for independent implementation. Along the way, it became clear that the laboratory is good, but the analyzes must be carried out in dynamics and it is extremely inconvenient to constantly turn to their services.
After looking at the planned activities of the laboratory, we decided to look at more utilitarian devices for independent work. As a result, we managed to find the tool necessary for this task - an air quality analyzer. Here's one:
The device shows the content of particles of different diameters (in micrometers).
Redefining standards
This device analyzes the number of particles, temperature, humidity and shows the results in units of measure in accordance with the ISO standards for this parameter. The display shows the levels of particles with different diameters in the air sample.
At the same time, they sinned against filters: at that time, G4 filter models were used inside the server rooms. This model provides a rough air purification, therefore, the possibility of passing particles that led to pollution was assumed. We decided to purchase F5 fine filters for testing, which are used in air conditioning and ventilation systems as filters of the second stage of purification (post-treatment).
The investigation has been carried out - you can proceed to control measurements. We decided to use the requirements of the ISO 14644-1 standard for the amount of suspended particles as a guideline.
Classification of clean rooms by the number of suspended particles.
It would seem - measure and compare according to the table. But not everything is so simple: in practice, it turned out to be quite difficult to find air purity standards for data center server rooms. This is not explicitly stated anywhere, not by any organization or industry institute. And only on the internal forum Uptime Inside Track (persons who have been trained under the programs of the Uptime Institute have access to it) there was a separate discussion on this topic. Based on the results of its study, they tended to focus on the ISO 8 standard - the penultimate one in the classification.
The very first measurements showed that we underestimated ourselves - the results of indoor air tests showed compliance with ISO 5 requirements in indoor areas, which significantly exceeded the standards desired by Uptime Inside Track participants. At the same time - with a large margin. We have a data center, not a biological laboratory, of course, but in order for the concentration of particles in the air to be equal to ISO 8, it must be an object of at least a “cement plant” class. And how you can apply the same standard to the data center is not very clear. At the same time, we obtained the result at ISO 5 by taking measurements during air filtration with G4 filters. That is, dust cannot enter the racks through the air, the F5 filters turned out to be redundant, and they were not even used.
A negative result is also a result: we continued to search for the cause of pollution in other directions, and included air quality control in quarterly checks, combined with BMS sensor checks by verified devices (ISO 9000 requirements and customer audits).
Below is an example of a report that is filled out based on the data obtained during the measurement. For greater accuracy, measurements are made by two devices - Testo 610 and BMS sensor. The heading of the table shows the limit values for the devices. The deviation of the specified parameters is automatically highlighted in color to facilitate the identification of problem areas or time periods.
Everything is clear with us: the difference in the performance of devices is minimal, and the concentration of particles is much lower than the limit.
Through the back door
Since there were other entrances to the cleanrooms in addition to the main entrance for customers, where we installed a shoe cover machine, it remained necessary to prevent dirt from entering the data center through them.
Putting on / taking off shoe covers during the procedures for unloading equipment is inconvenient, so we found a machine for cleaning soles. Convenient, functional, but the human factor affects in the form of an optional approach to this device. In fact, the same thing as with shoe covers at the main entrance.
To solve the problem, they began to look for cleaning options that could not be avoided: sticky carpets with peelable layers did the best job. In the process of authorization at the entrance door, the visitor must stand on such a rug, removing excess dust from the soles of his shoes.
Cleaners tear off the top layer of such a rug every day, there are 60 layers in total - enough for about 2 months.
Having visited the Ericsson data center in Stockholm, among other things, I paid attention to how these issues are solved there: along with tearable layers, Dycem reusable antibacterial carpets are used in Sweden. I liked the idea because of the principle of reusability and the ability to provide a large coverage area.
Magic antibacterial carpet. It's a pity, not a plane, but it could - at such a price!
With difficulty, we found representatives of the company in Russia and made an estimate of the cost of the solution for our data center. As a result, we got a figure almost 100 times more expensive than the solution with multilayer carpets - about the same 1 million rubles as in the project with air purity measurements. In addition, it turned out the need to use special cleaning products, naturally available only from this manufacturer. The decision also disappeared by itself, we settled on a multi-layered version.
Manual labor
I would especially like to draw attention to the fact that all these measures did not cancel the use of the labor of cleaners. As part of the preparation for the certification of the Linxdatacenter data center according to the Uptime Institute Management & Operations standard, it was necessary to clearly regulate the actions of cleaning service employees on the territory of the data center. Detailed instructions were drawn up, prescribing where, what and how they should do it.
A couple of excerpts from the manual:
As you can see, everything is prescribed, literally every aspect of work in a particular room, cleaning products, materials, etc., acceptable for use. Not a single detail, even the smallest, is left without attention. Briefing - under the signature of each employee of the service. In server rooms, electrical rooms, etc. they are removed only in the presence of authorized data center employees, for example, an engineer on duty.
But that's not all
Also on the list of measures to ensure cleanliness in the data center: walks with a visual inspection of the premises, including weekly inspections of racks to detect wire scraps left inside them, the remains of packages from equipment and components. For each such episode, an incident is started, the client receives a notification about the need to eliminate violations as soon as possible.
Also, for unpacking and setting up the equipment, we have created a separate room - this is also part of the company's cleaning policy.
Another measure that we learned from Ericsson's practice is maintaining a constant air supply in server rooms: inside the rooms there is more pressure than outside, so that there is no draft inside - we will talk about this solution in more detail in a separate article.
Finally, we got ourselves robotic assistants for the premises, which are excluded from the list available for cleaning personnel to visit.
The grate on top not only gives +10 to the protection of the robot, but also allows it not to get stuck under the vertical cable trays of the racks.
An unexpected find as a conclusion
Cleanliness in the data center is important for the operation of server and network equipment, which draws air through itself. Exceeding dust limits will cause dust to accumulate on components and result in an overall temperature rise of up to 1 degree Celsius. Dust reduces cooling efficiency, which in terms of a year can turn into significant indirect costs, as well as affect the fault tolerance of the facility as a whole.
We can say that this is a speculative assumption, but the Uptime Institute experts who certified the Linxdatacenter data center for the quality standard for operational management (Management & Operations) pay the most serious attention to cleanliness. And it was all the more pleasant to receive the most flattering assessments in this area: our data center in St. Petersburg seriously exceeds the certification requirements. An institute expert called us “the cleanest data center he has seen”, moreover, our data center is used by Uptime as an example of solving the issue of server room cleanliness. Also, we easily pass any client audit on this parameter - the most serious requirements of the most capricious clients are met in excess.
Let's go back to the beginning of the story. Where did the pollution come from in the same complaint from the beginning of the article? The part of the customer's rack that triggered the entire "data center cleanliness" project was contaminated from the moment the rack was brought in and installed in the data center. The client had not cleaned the rack by the time it was brought into the server room - when checking neighboring racks installed at the same time, it turned out that the situation with dust was the same there. This situation prompted the addition of a cleaning control point to the customer's rack installation checklist. The likelihood of such things should also never be forgotten = forewarned, therefore armed. This is all about "purity and dictatorship" in our data center, in the next article I will talk about pressure sensors, but for now, ask questions in the comments.
Source: habr.com