Hi all. As we promised, we are immersing Habr's readers into the details of the production of Russian hardware platforms for Aerodisk Vostok storage systems based on Elbrus processors. In this article, we will describe step by step the production of the Yakhont-UVM E124 platform, which effectively accommodates 5 disks in 124 units, can operate at a temperature of +30 degrees Celsius and not only works, but works well.
Also on 05.06.2020/XNUMX/XNUMX, we are organizing a webinar, where we will talk in detail about the technical nuances of Vostok storage systems and answer any questions. You can register for the webinar using this link:
So there you go!
Before diving into the process that is organized now, a little historical background from two years ago. At the time of the beginning of the development of the platforms described in this article, the conditions for their production were, to put it mildly, none. There are reasons for this, they are all known: mass production (namely, production, and not re-gluing stickers) of server platforms in Russia was absent as a class. There were separate factories that could produce individual components, but at the same time very limitedly and often on the basis of outdated technologies. Therefore, we had to start virtually from scratch and at the same time raise the production of server solutions in Russia to a qualitatively new level.
So, the process of any production begins with a need, which is then transformed into general requirements. Such requirements are initially formed by the developers of NORSI-TRANS in Nizhny Novgorod. Requirements, of course, are not taken out of thin air, but from the needs of customers. This is not yet a technical task, as it may mistakenly seem. At the stage of general requirements, it is impossible to make a full-fledged TOR, because there are too many unknown conditions for production.
Target model development: from idea to implementation
After the general requirements are formed, the selection of the element base begins. It follows from the historical background that the element base does not exist, that is, it must be created.
To do this, from what is on the open market, a pilot sample is assembled, which is at least a little similar to the target one. Further, standard tests of this sample are carried out for its performance. If all is well, then the next step is to develop the target model (2D and 3D).
Then the search for Russian enterprises ready to begin production of this pilot begins. The developers carry out the necessary refinement of each of the elements of the product, based on the capabilities of a particular enterprise.
During the design process, the necessary refinement of each of the elements of the product is carried out. For example, in the process of working with a prototype, classic 12G SAS expanders with a large number of wires were used (with a very large number, given the number of disks). It is not cheap, inconvenient for this particular platform, and besides, the expanders are enemy foreign. But this is a temporary solution in order to test the sample as a whole and move on to the next step. However, for the final version in a specific server platform, using SAS expanders is not good.
We don't need enemy expanders, we'll make our own backplane with blackjack and sh...
Taking into account future plans for production volumes (thousands of servers), it was decided to develop our own SAS backplane for this product (and, of course, for the next ones), which is much more functional than the expander in relation to this solution. The design and programming of the backplane is carried out by the same development team, and the boards are manufactured at the Microlit plant in the Moscow region (we promise a separate article about this plant and how motherboards for Elbrus processors are printed there).
Here, by the way, is his first prototype, now it already looks completely different.
And then they do programming
An interesting fact: when the development of the backplane was started, and the designers turned to the developer of the SAS3 chip for a reference design of the board, it turned out that not a single company in Europe could develop their backplanes. Previously, there was a Fujitsu-Siemens joint venture, but after Siemens Nixdorf Informations systeme AG left the joint venture and the complete closure of the computer business at Siemens, competencies in this area in Europe were lost.
Therefore, the chip developer initially did not immediately take the developments of NORSI-TRANS seriously, which is why there were delays in the development of the final design. True, later, when the seriousness of the intentions and competence of the NORSI-TRANS company became obvious, and the backplane was developed and printed, his attitude changed for the better.
How to cool down 124 disks and a server in 5 units and stay alive?
A separate quest was with food and cooling. The fact is that, based on the requirements, the E124 platform must operate at a temperature of 30 degrees Celsius, and there, for a minute, there are 124 well-heated mechanical disks in 5 units, and besides, a motherboard with a processor (i.e. this is not stupid JBOD, but a full-fledged storage controller with disks).
For cooling (except for the small fans inside), in the end we decided to use three rather large fans at once in the rear of the case, with the possibility of hot swapping each. For normal operation of the system, two are enough (the temperature does not change at all), so you can safely plan work on replacing fans and not think about temperature. If, however, two fans are turned off (for example, according to the law of meanness, while one was being changed, the second one broke), then the system is also able to work normally with one fan, but the temperature will increase by 10-20% percent, which is permissible if at least one more is installed fan soon.
Fans (like almost everything else) were also unique. The reason for the uniqueness was one cost. Under certain conditions, it may happen that the fans, instead of sucking out air, blowing through the entire case from the inside, can start to suck it in, and then “goodbye”, that is, the platform will quickly overheat. Therefore, in order to prevent such a problem, they made changes to the fan design and added their own “know-how” - a check valve. This check valve quietly allows air to be sucked out of the platform, but at the same time blocks the very possibility of air being sucked back in any case.
At the stage of piloting the cooling system, there were many failures, various elements of the system heated up and burned, but in the end, the platform developers managed to achieve better cooling than even world-famous competitors.
"You can't break the diet."
There was a similar story with power supplies, i.e. they were made specifically for this platform and the reason here is banal. Each unit is a lot of money, which is why such a super-dense platform was developed and, if I'm not mistaken (correct in the comments if I'm wrong), so far this is a world record, because. there are no servers and JBODs with a large number of disks for 5 units yet.
Thus, in order to provide power to the platform and at the same time organize the possibility of replacing the power supply in the normal mode, the total power of the active units had to be 4 kilowatts (of course, there are no such solutions on the market), so they were made to order with the launch of a production line for mass production ( Let me remind you that there are thousands of such servers in the plans).
As one of the main designers of the platform put it, “Here, such currents are like in a welding machine - this is not khukhr-mukhr for you :-)”
When designing, it was also possible to operate the power supply not only at 220V, but also at 48V, i.e. in the OPC architecture, which is now very important for telecom operators and large data centers.
As a result, the solution with power supply repeats the logic of the solution with cooling, the platform can comfortably work with two power supplies, which allows you to perform work on replacing them in normal mode. If, in the event of an accident, only one of the three PSUs remains, it will be able to pull out the work of the platform at peak load, but, of course, it is impossible to leave the platform in this form for a long time.
Metal and plastic: not everything, it turns out, is so simple.
There are many nuances in the process of platform development. A similar situation was not only with electronic components (risers, backplane, motherboards, etc.), but also with ordinary metal and plastic: for example, with a case, rails, and even with disc carriages.
With the case and other less smart elements of the platform, it would seem that there should be no problems. But in practice, everything is different. When the platform developers first turned to various Russian factories with the need for production, it turned out that most of them work with rather outdated methods, which ultimately affects both the quality and quantity of products.
The very first results of the production of cases were proof of this. Incorrect geometry, rough welds, inaccurate holes and similar costs made the product unusable.
Most of the factories that could make server cases worked then (let me remind you that “then” means 2 years ago) “the old fashioned way”, that is, a bunch of design documentation was issued, in accordance with it, the operator manually corrected the operation of the machines, also often instead of riveting metal welding was used. As a result, the low degree of automation, the human factor and the excessive bureaucracy of production have borne fruit. It turned out long, bad and expensive.
We must pay tribute to the factories: many of them have greatly modernized their production since that time. They improved the quality of welding, mastered riveting, and also often began to use machine tools with numerical control (CNC). Now, instead of tons of documents, product data is loaded directly from 3D and 2D models to the CNC.
CNC reduces the intervention of the machine operator in the manufacturing process of the product to a minimum, so the human factor does not interfere with life. The main concern of the operator is mainly preparatory and final operations: installation and removal of the product, setting tools, etc.
Cases of the appearance of new parts no longer enter production into a stupor, for their manufacture it is enough to make changes to the CNC software. Accordingly, the production time for parts under new projects at factories has been reduced from months to weeks, which cannot but rejoice. And, of course, the accuracy has also increased greatly.
Motherboards and processor: no adventure
Processors and motherboards come as a set from the factory. This production is already quite well established, so NORSI carries out standard input control and output control at the level of finished platforms.
Each set of motherboard and processor is tested by software tools received from MCST.
In the event of certain problems (thank God, there are very few of them with the motherboard and processor), a well-established chain of returning modules to the manufacturer and replacing them works.
Assembly and final control
In order for our balalaika to start playing, it remains to be assembled and tested. Now production has been put on stream, the system is assembled in a standard way in Moscow.
Each system is equipped with bootable SSDs (for the OS) and fully stocked with spindles (for future data).
After that, the entrance testing of both the platform itself and the disks installed in it begins. To do this, all system disks are loaded with auto-tests for at least an hour.
Automatically reads and writes to each drive, and records read, write, and temperature readings for each drive. In normal mode, the average temperature should be around 30-35 degrees Celsius. In peaks, each individual disk can “bounce” up to 40 degrees. If the temperature gets higher or the speed drops below the read/write thresholds, the drive will turn red and not be culled. The tested components are packed for further use.
Conclusion
There is such a myth, which is actively supported by various figures, that "in Russia they know nothing but how to pump oil." Unfortunately, this myth eats into the heads of even respected and intelligent people.
Recently, a remarkable story happened to my colleague. He was driving from one of the shows of the storage system Vostok and in the trunk of his car this storage system was just lying (not E124, of course, simpler). On the way, he captured one of the customer’s representatives (a very important person, works in a high position in one of the state structures), and in the car they had something like the following conversation:
My colleague: “Now they showed storage systems on Elbrus, the results are good, everyone was happy, for your industry, by the way, this storage system will also be useful”
Customer: “I know that you have storage systems, but what else is Elbrus, what are you talking about?”
My colleague: “Well, how ... the Russian processor Elbrus, they recently released the 8-ku, it is quite good in terms of storage systems, we, accordingly, made a new line of storage systems on it, called Vostok”
Customer: Elbrus is a mountain! And you don’t voice fairy tales about the Russian processor in a decent society, this is all done just to master the budgets, in fact there is nothing and never will be ”
My colleague: "In terms of? But nothing, what exactly is this storage system in my trunk? Let's stop right now, I'll show you!"
Customer: “It’s good to suffer nonsense, we go further, there are no “Russian storage systems” - this is basically impossible”
At that moment, the important person did not want to hear anything more about Elbrus. Of course, later, when he clarified the information, he admitted that he was wrong, but still, until the last moment, he did not believe in the veracity of this information.
In fact, after the collapse of the USSR, our country actually stopped in the development of microelectronics production. Something was taken out stolen in favor of transnational corporations, something was stolen by the local privatization company, something, of course, was invested, but mainly in favor of the same transnational corporations. The tree was cut down, but the root remained.
After almost 30 years of illusions on the topic “the West will help us”, it has become obvious to almost everyone that we can only help ourselves, so we need to restore our production not only in the field of microelectronics, but also everywhere in all industries.
At the current moment, in the context of a global pandemic in a situation of an actual stoppage of transnational production chains, it is already becoming clear that the restoration of local production is no longer the development of budgets, but a condition for the survival of Russia as an independent state.
Therefore, we will continue to look for and use Russian equipment in our lives and tell you about what our companies are actually doing, what problems they face and what titanic efforts they make to solve them.
It is quite difficult to cover all aspects of production within the framework of one article, therefore, as a bonus, we will organize an online discussion in the format of a webinar on this topic. At this webinar, we will talk in detail and in colors about the technical aspects of the production of Yakhont platforms for Vostok storage systems and answer all, even the most tricky questions, online.
Our interlocutor will be a representative of the platform developer, NORSI-TRANS. The webinar will take place on 05.06.2020/XNUMX/XNUMX, those wishing to participate can register using the link: .
Thank you all, as usual, we look forward to constructive comments.
Source: habr.com