Good day, dear Khabrovites!
I want to tell you a long and, as I hope, fascinating, and maybe useful story of assembling a “village supercomputer” from a Dell server node board, an Nvidia Tesla K20 GPU and what was bought here and there in all sorts of online stores or in computer stores in your city.
The story began with the fact that my programmer friend, who is also an astronomer, started studying neural networks. Their "full-time specialist" quit and the topic was hung up on the "closest specialist". I myself am not a programmer, just “a radio mechanic for repairing computer equipment (with) my diploma”, therefore, assembling all sorts of interesting computer hardware is an interesting and enjoyable experience for me. Unfortunately, I work in a different field.
To more clearly formulate the problem, I created a topic on the Iron Ghosts of the Past forum, where it was discussed for a long time. At first, there was a rather naive idea “to assemble 4-way SLI on a GTX 580 3Gb” which gradually transformed into an understanding - you need to assemble a server! Prices for server motherboards were shamelessly biting until I came across an interesting video on Youtube about the launch of a Chinese server board on 2 non-standard format processors.
Here's the video:
I was especially pleased with the budget price of the system in this video.
However, consultation with more knowledgeable comrades who dealt with Chinese servers convinced me - “We don’t need Chinese happiness!” According to their reviews, Chinese servers were simply monstrously unreliable. And I began to look for options with Dell server boards on Avito. I have two laptops from this company and they have only positive impressions. Very reliable technology.
On Avito, a Dell PowerEdge C6220 server node board was found in the process of communicating with the seller of which - he suggested to me an excellent site where there was a publication about how one craftsman launched such a board, . And there was a link to an American forum, where powerful workstations were assembled on such boards..
I read the whole topic from beginning to end, I decided on the goals, objectives and ways to achieve them. The task was formulated as follows - "Assemble a two-processor server on a Dell PowerEdge C8220 node board with a Tesla K10 or K20 GPU." The choice for specialized GPUs fell after a discussion with the person for whom the system was actually assembled - having "cards" that could conduct long calculations with double precision and error control of the ECC memory, he could use them for his scientific activities, and not just for training neural networks. Which made him very happy.
To discuss and capture the history of the assembly process on the Iron Ghosts of the Past forum, I created a corresponding topic, where I actually wrote about the process and posted photos. .
The task was set and I started looking for components. At the time when it all started, I did not yet have an eBay registration, and at first the necessary spare parts were bought by my friends, to whom I paid the purchase and shipping costs. Later, I myself registered there and began to buy directly, although sometimes I have to ask for help from those who have accounts on Shopotam and similar services. Not all the necessary spare parts are directly sent from the USA to Russia.
The first motherboard I bought from eBay was Dell PowerEdge C8220 0083N0. According to Dell's documentation, it refers to version 1.2 boards with 3 PCI-E 16x slots. Two ordinary ones near the power button and the third on the other side of the board is non-standard, for the so-called GPGPU riser, which was included in the so-called Edge Slot.
Photo of the board, the same 0083N0, photo from eBay.
And this is my photo, a ruler is attached to the board to understand the scale.
By that time, a riser for GPGPU also arrived in the same Edge Slot.
Here is a photo where it is connected for testing to its regular place.
At the same time, a power adapter was bought on eBay, from ATX to this C6100 power connector. They are sold on eBay in two types, 12 and 18 pins. We need the latter, and another DC-DC boost to turn +5VSB from an ATX PSU into +12VSB from a Dell server. And of course, the mating block “mother” into the connector to install the jumper needed to start the board and output the PS_ON signal from it. By the way, it has a non-standard contact pitch of 2.0 mm. Of course, desperate guys can weave a screwdriver or a nail directly into the board connector, but I preferred to do everything culturally.
In addition, for a trial run of the board, the cheapest Xeon E5-2604 V1 was bought from Aliexpress and a pair of DDR3 ECC REG memory sticks from eBay, which were sold as compatible with the Dell PowerEdge C8220. I first used the Alpine 20 Plus C0 coolers for LGA 2011, which I had to modify - their edges were filed by a grinder that rested on the memory slots, spring washers were removed from the fastening screws, and a pair of nuts were screwed onto the threads - so as not to screw the screws too deep and don't break the board. The LGA 2011 server sockets are arranged a little differently than the usual ones, and the threads of the radiator screws should be short there. By the way, the coolers worked well, despite the fact that they are pure aluminum.
And so, the moment came when the processors arrived, I captured their installation in a photo as a keepsake.
And here are the very Alpine aluminum coolers installed.
Assembled and running system.
My old faithful Chiftek 550 W power supply was connected to the system, a USB hub for 4 devices, which included a keyboard, a mouse and a USB flash drive with Ubuntu, a card reader was connected to the USB card reader connector on the board, into which I plugged a Chinese USB sound, I also connected a VGA monitor and a patch cord to the 100 Mbit IPMI port, which is called Delicated-NIC. Next to it are two 10Gbe ports that work over a conventional copper twisted pair and fully support a regular 100/1000 network.
In this form, the system was launched and it turned out that the board checks the memory for a very long time at startup. And in the BIOS splash screen, she called herself Dell DCS 6220.
On this, I will finish the first part of my story so as not to tire the grateful readers.
Link to part 2:
Source: habr.com