Hello, Habr readers. We want to share some very pleasant news. We have finally waited for the actual serial production of the new generation of Russian Elbrus 8C processors. Officially, serial production was supposed to start back in 2016, but in reality, mass production only began in 2019, and to date, around 4,000 processors have already been produced.
Almost immediately after the start of serial production, these processors became available to us at Aerodisk, for which we want to thank NORSI-TRANS for kindly providing us with their Yakhont UVM hardware platform, which supports Elbrus 8C processors, to carry out the software porting of the storage system. This is a modern platform that meets all the requirements of MCST. Currently, the platform is used by special consumers and telecommunications operators to ensure actions during operational-search activities.
At this point, the porting has been successfully completed, and the AERODISK storage system is now available with Russian Elbrus processors.
In this article, we will talk about the processors themselves, their history, architecture, and of course, our implementation of the storage system on Elbrus.
History
The history of Elbrus processors dates back to the Soviet Union. In 1973, the development of multiprocessor computing complexes called 'Elbrus' began at the V.A. Steklov Institute of Computational Mathematics and Mathematical Physics. The project was led by Vsevolod Sergeyevich Burts, and Boris Artashesovich Babayan, one of the deputy chief designers, also actively participated in its development.
Vsevolod Sergeyevich Burts
Boris Artashesovich Babayan
The main customer of the project was, of course, the Soviet Armed Forces, and this series of computers was successfully used in the creation of command computing centers and firing complexes for missile defense systems, as well as other special purpose systems.
The first computer "Elbrus" was completed in 1978. It had a modular architecture and could include from 1 to 10 processors based on medium integration schemes. The speed of this machine reached 15 million operations per second. The total amount of RAM shared among all 10 processors was up to 2 to the power of 20 machine words or 64 MB.
Later, it became clear that many of the technologies used in developing the "Elbrus" were being explored in the world at the same time, and they were handled by the International Business Machines (IBM) company. However, the work on those projects, unlike the work on "Elbrus," was never completed and ultimately did not lead to the creation of a finished product.
According to Vsevolod Burtsev, Soviet engineers sought to apply the most advanced practices from both domestic and foreign developers. The architecture of the "Elbrus" computers was also influenced by computers from Burroughs, developments from Hewlett-Packard, as well as the experience of the BESM-6 developers.
Nevertheless, many of the developments were original. The most interesting aspect of the "Elbrus-1" was indeed its architecture.
The created supercomputer became the first computer in the USSR to employ a superscalar architecture. The widespread use of superscalar processors abroad only began in the 1990s with the introduction of affordable Intel Pentium processors to the market.
In addition, to organize the transfer of data streams between peripheral devices and RAM, special input-output processors could be used in the computer. There could be up to four of these processors in the system; they operated in parallel with the central processor and had their own dedicated memory.
Elbrus-2
In 1985, "Elbrus" received its logical continuation with the creation and initiation of serial production of the "Elbrus-2" computer. In architecture, it differed little from its predecessor, but it used a new element base, which allowed for nearly a tenfold increase in overall performance – from 15 million operations per second to 125 million. The RAM of the computer increased to 16 million 72-bit words or 144 MB. The maximum throughput of the input-output channels of the "Elbrus-2" was 120 MB/s.
The "Elbrus-2" was actively used in nuclear research centers in Chelyabinsk-70 and Arzamas-16 in the MCC, in the A-135 missile defense system, as well as at other military facilities.
The creation of the "Elbrus" was duly recognized by the leaders of the Soviet Union. Many engineers were awarded orders and medals. The chief designer Vyacheslav Burtsov and several other specialists received state awards. Boris Babayan was awarded the Order of the October Revolution.
These awards were more than deserved; Boris Babayan later said:
"In 1978, we created the first superscalar machine, Elbrus-1. Now, in the West, they only make superscalars of that architecture. The first superscalar in the West appeared in 1992, ours was in 1978. Moreover, the variant of the superscalar that we made is analogous to the Pentium Pro, which Intel made in 1995."
These words about historical precedence are confirmed in the USA; Keith Diefendorff, a developer of the Motorola 88110, one of the first Western superscalar processors, wrote:
"In 1978, almost 15 years before the first Western superscalar processors appeared, Elbrus-1 utilized a processor capable of issuing two instructions per clock cycle, reordering instructions, renaming registers, and executing based on speculation."
Elbrus-3
It was 1986, and almost immediately after completing work on the second "Elbrus," development of the new "Elbrus-3" system began at ITMiVT, using a fundamentally new processor architecture. Boris Babayan called this approach "post-superscalar." This architecture, later named VLIW/EPIC, was eventually used in Intel Itanium processors in the mid-90s (while those developments in the USSR started in 1986 and finished in 1991).
This computing complex was the first to bring to life the ideas of explicit control of operation parallelism through the compiler.
In 1991, the first and, unfortunately, only computer "Elbrus-3" was released, which could not be finalized, and after the collapse of the Soviet Union, it became unnecessary, leaving the developments and plans on paper.
The prerequisites for a new architecture
The team that worked at ITMiVT on the creation of Soviet supercomputers did not disband but continued as a separate company under the name MCST (Moscow Center of SPARC Technologies). In the early 1990s, active collaboration began between MCST and Sun Microsystems, where the MCST team participated in the development of the UltraSPARC microprocessor.
It was during this period that the E2K architecture project was initiated, which was initially funded by Sun. Later, the project became fully independent and all intellectual property rights remained with the MCST team.
"If we had continued working with Sun in this area, everything would belong to Sun. Although 90% of the work was completed before Sun even appeared." (Boris Babayan)
E2K Architecture
When we discuss the architecture of the Elbrus processors, we often hear the following statements from our colleagues in the IT industry:
"Elbrus is a RISC architecture"
"Elbrus is an EPIC architecture"
"Elbrus is a SPARC architecture"
In reality, none of these statements are completely accurate, and if they are, then only partially.
The E2K architecture is a distinct, original processor architecture whose main qualities are energy efficiency and excellent scalability, achieved through explicit parallelism in operations. The E2K architecture was developed by the MCST team and is based on a post-superscalar architecture (similar to EPIC) with some influences from SPARC architecture (with a RISC background). Moreover, MCST was directly involved in the creation of three out of the four basic architectures (Superscalar, post-superscalar, and SPARC). Indeed, the world is small.
To avoid confusion in the future, we have drawn a simple diagram that, although simplified, clearly shows the roots of the E2K architecture.
Now, a bit more about the name of the architecture, for which there is also some misunderstanding.
In various sources, you may encounter different names for this architecture: "E2K", "Elbrus", "Elbrus 2000", ELBRUS ("ExpLicit Basic Resources Utilization Scheduling", i.e., explicit scheduling of the use of basic resources). All these names refer to the same architecture, but in official technical documentation and on technical forums, the term E2K is used to denote the architecture. Therefore, going forward, when we refer to the processor architecture, we will use the term "E2K", and when referring to a specific processor, we will use the name "Elbrus".
Technical Features of the E2K Architecture
In traditional architectures like RISC or CISC (x86, PowerPC, SPARC, MIPS, ARM), a stream of instructions is fed into the processor, designed for sequential execution. The processor can detect independent operations and execute them in parallel (superscalarity) and even reorder them (out-of-order execution). However, dynamic dependency analysis and support for out-of-order execution have limitations on the number of instructions that can be launched and analyzed in one cycle. Furthermore, the corresponding blocks within the processor consume a significant amount of energy, and their complex implementation can sometimes lead to stability or security issues.
In the E2K architecture, the compiler takes on the primary work of analyzing dependencies and optimizing the order of operations. The processor receives so-called wide instructions, where each instruction encodes commands for all execution units of the processor that should be activated during the given cycle. The processor does not need to analyze dependencies between operands or rearrange operations among wide instructions; the compiler handles all this based on the analysis of the source code and resource planning for the processor. As a result, the hardware component of the processor can be simpler and more energy-efficient.
The compiler can analyze the source code much more thoroughly than the RISC/CISC hardware of the processor, enabling it to find more independent operations. Therefore, the E2K architecture has more parallel execution units than traditional architectures.
Current Capabilities of the E2K Architecture:
- 6 arithmetic-logic unit (ALU) channels operating in parallel.
- A register file consisting of 256 84-bit registers.
- Hardware support for loops, including pipelining. Enhances CPU resource utilization efficiency.
- Programmable asynchronous data prefetching device with separate read channels. Allows hiding memory access delays and fully utilizes the ALU.
- Support for speculative computing and single-bit predicates. Reduces branching and enables concurrent execution of multiple program branches.
- A broad instruction set capable of executing up to 23 operations in a single cycle (over 33 operations when operand packing into vector instructions).
x86 emulation
From the early architecture design stage, the developers understood the importance of supporting software written for the Intel x86 architecture. They implemented a dynamic (i.e., runtime) translation system for translating x86 binary codes into E2K processor codes. This system can operate either in application mode (similar to WINE) or in a hypervisor-like mode (allowing for the entire guest OS for x86 architecture to run).
Thanks to multiple optimization levels, high speed is achieved for the translated code. The quality of x86 emulation is confirmed by successful launches on the Elbrus computing complexes of over 20 operating systems (including several versions of Windows) and hundreds of applications.
Protected mode for program execution
One of the most interesting ideas inherited from the Elbrus-1 and Elbrus-2 architectures is the so-called secure program execution. Its essence lies in ensuring that a program operates only with initialized data, checking all memory accesses for compliance with the allowed range of addresses, and providing inter-module protection (for example, protecting the calling program from errors in a library). All these checks are performed at the hardware level. A full-fledged compiler and runtime support library are available for the secure mode. However, it should be understood that the imposed restrictions lead to the inability to organize the execution of, for instance, C++ code.
Even in the regular, 'unprotected' mode of operation of Elbrus processors, there are features that enhance system reliability. For example, the stack of linking information (the chain of return addresses during procedural calls) is separated from the user data stack and is inaccessible to such attacks used in viruses as return address spoofing.
The developments made over the years not only allow us to catch up with and ultimately surpass competing architectures in terms of performance and scalability but also ensure protection against errors that plague x86/amd64. Vulnerabilities like Meltdown (CVE-2017-5754), Spectre (CVE-2017-5753, CVE-2017-5715), RIDL (CVE-2018-12126, CVE-2018-12130), Fallout (CVE-2018-12127), ZombieLoad (CVE-2019-11091), and others of their kind.
Modern protection against the vulnerabilities found in x86/amd64 architecture is based on patches at the operating system level. This is why the performance drop in current and previous generations of processors of these architectures is so noticeable, ranging from 30% to 80%. As active users of x86 processors, we are aware of this, we suffer from it, and we continue to 'eat the cactus'. However, having a solution to these problems at the root is undoubtedly beneficial for us (and ultimately for our clients), especially if the solution is Russian.
Technical Specifications
Below are the official technical specifications of the Elbrus processors of the past (4C), current (8C), new (8CV), and future (16C) generations compared to similar Intel x86 processors.
Even a brief glance at this table shows (and this is very encouraging) that the technological gap of domestic processors, which seemed insurmountable 10 years ago, now appears quite small, and in 2021, with the launch of the "Elbrus-16S" (which, among other things, will support virtualization), it will shrink to minimal distances.
AERODISK storage systems on Elbrus 8S processors
From theory to practice. As part of the strategic alliance of MCST, Aerodisk, Basalt SPO (formerly Alt Linux), and NORSI-TRANS, a data storage system has been developed and prepared for operation that, at present, in terms of security, functionality, cost, and performance, is if not the best, then undoubtedly a worthy solution that can ensure an adequate level of technological independence for our Homeland.
Now for the details...
Hardware components
The hardware part of the storage system is based on the universal platform Yakhont UVM from NORSI-TRANS. The Yakhont UVM platform has received the status of telecommunications equipment of Russian origin and is included in the unified register of Russian radio-electronic products. The system consists of two separate storage controllers (2U each), connected to each other via a 1G or 10G Ethernet interconnect, as well as to shared disk shelves using SAS connections.
Of course, this is not as aesthetically pleasing as the "Cluster in a box" format (when controllers and disks with a shared backplane are installed in one 2U chassis), which we usually use, but it will also be available in the near future. The main thing is that it works well, and we can think about the "decorations" later.
Under the hood, each controller has a single-processor motherboard with four slots for RAM (DDR3 for the 8S processor). Each controller also features four 1G Ethernet ports (two of which are used by AERODISK ENGINE as service ports) and three PCIe slots for Back-end (SAS) and Front-end (Ethernet or FibreChannel) adapters.
As boot disks, we use Russian SATA SSDs from GS Nanotech, which we have repeatedly tested and used in projects.
Upon our first encounter with the platform, we thoroughly examined it. There were no questions regarding the quality of assembly and soldering; everything was done meticulously and reliably.
Operating system
The OS used is the ALT 8SP version for certification. We plan to create a plug-in and continuously updated repository for ALT OS with the AERODISK storage software soon.
This version of the distribution is built on the current stable version of the Linux 4.9 kernel for E2K (the long-term support branch ported by experts from MCST), enhanced with patches for functionality and security. All packages in ALT OS are compiled directly on "Elbrus" using the original transactional build system of the ALT Linux Team project, which allowed us to reduce the labor costs for the porting process and focus more on the quality of the product.
Any release of the ALT OS for Elbrus can be significantly expanded in terms of functionality by utilizing the available repository (from about 6 thousand source packages for the eighth version to about 12 thousand for the ninth).
The choice was also made because Basalt SPU, the developer of ALT OS, actively collaborates with other software and hardware developers across various platforms, ensuring seamless interaction within hardware-software complexes.
Storage System Software
During the porting process, we immediately abandoned the idea of using x86 emulation supported in E2K and started working directly with processors (thankfully, ALT already has the necessary tools for this).
Among other things, the native execution mode provides better security (the same three hardware stacks instead of one) and increased performance (there's no need to allocate one or two cores out of eight for the binary translator, and the compiler performs its work better than JIT).
In fact, the implementation of AERODISK ENGINE on E2K supports a large part of the existing functionality of storage systems available in x86. The current version of AERODISK ENGINE (A-CORE version 2.30) is used as the storage system software.
Without any significant issues, the following functions have been established and tested for production use on E2K:
- Fault tolerance up to two controllers and multipath input-output (mpio)
- Block and file access with thin volumes (RDM pools, DDP; FC, iSCSI, NFS, SMB protocols including integration with Active Directory)
- Various RAID levels up to triple parity (including the ability to use a RAID builder)
- Hybrid storage (combining SSD and HDD within a single pool, i.e., caching and tiering)
- Space-saving options through deduplication and compression
- ROW snapshots, clones, and various replication options
- And other small but useful features such as QoS, global hotspare, VLAN, BOND, etc.
In fact, we managed to deploy all our functionality on E2K, except for multi-controller support (more than two) and the multithreaded I/O scheduler, which increases the performance of all-flash pools by 20-30%.
But we will certainly add these useful features; it's just a matter of time.
A bit about performance
After successfully passing the basic functionality tests of the storage system, we, of course, started performing load tests.
For instance, on a dual-controller storage system (2xCPU E8C 1.3 GHz, 32 GB RAM + 4 SAS SSD 800GB 3DWD), where RAM caching was disabled, we created two DDP pools with primary RAID-10 level and two LUNs of 500G each, connecting these LUNs via iSCSI (10G Ethernet) to a Linux host. We executed one of the basic hourly tests with small blocks of sequential load using the FIO program.
The initial results were quite positive.
The CPU load was on average at 60%, which is the basic level at which the storage system can operate smoothly.
Yes, this is far from high-load and will be clearly insufficient for high-performance DBMS used in billing systems, but, as our experience shows, these characteristics are enough for 80% of general tasks for which storage systems are utilized.
Later, we plan to return with a detailed report on the load tests of Elbrus as a platform for storage systems.
Bright future
As we mentioned earlier, mass production of Elbrus 8C actually started just recently – in early 2019, and by December, about 4000 processors had been produced. In comparison, only 5000 processors of the previous generation Elbrus 4C were produced throughout their entire production period, so the progress is evident.
It's clear that this is just a drop in the ocean, even for the Russian market, but the path will be paved by those who walk it.
In 2020, the release of several tens of thousands of Elbrus 8C processors is planned, which is already a significant figure. Furthermore, during 2020, the Elbrus-8SV processor is to be brought to mass production by the MCST team.
Such production plans indicate a claim to a very significant share of the entire domestic server processor market.
As a result, here and now we have a good and modern Russian processor with a clear and, in our opinion, correct development strategy, based on which there is a maximally secure and certified data storage system produced in Russia (and in the future, a virtualization system on Elbrus-16C). The system is as Russian as is physically possible under current conditions.
We often see in the news the latest epic failures of companies that proudly call themselves Russian manufacturers, but in reality, they engage in rebranding foreign products without adding any intrinsic value to them apart from their markup. Unfortunately, such companies cast a shadow on all real Russian developers and manufacturers.
With this article, we want to clearly demonstrate that in our country, there have been, are, and will be companies that genuinely and qualitatively create modern complex IT systems and actively develop, and that import substitution in IT is not a sham, but a reality in which we all live. This reality can be disliked or criticized, but it can also be worked on and improved.
The collapse of the USSR at one time hindered the team of Elbrus creators from becoming a notable player in the world of processors and forced the team to seek funding for their developments abroad. It was found, work was done, and intellectual property was preserved, for which we want to give a huge thanks to these people!
That's all for now; please write your comments, questions, and, of course, criticisms. We are always happy to hear from you.
On behalf of the entire Aerodisk team, I would also like to congratulate the entire Russian IT community on the upcoming New Year and Christmas, wishing 100% uptime – and may backups not be needed in the new year!)))
Materials Used
An article with a general description of technologies, architectures, and personalities:
A Brief History of Computers Named 'Elbrus':
General Article on e2k Architecture:
Article Specifically About the 4th Generation (Elbrus-8C) and 5th Generation (Elbrus-8SV, 2020):
Specifications for the Next 6th Generation Processors (Elbrus-16SV, 2021):
Official Description of Elbrus Architecture:
Developers' Plans for the 'Elbrus' Hardware-Software Platform to Create an Exaflop Supercomputer:
Russian 'Elbrus' Technologies for Personal Computers, Servers, and Supercomputers:
An Old Article by Boris Babayan, Still Relevant:
An Old Article by Mikhail Kuzminsky:
MCST Presentation, General Information:
Information About Alt OS for Elbrus Platform:
Source: habr.com
