Good day, dear Khabrovites!
I want to continue my story about assembling a "village supercomputer". And I will explain why it is so named - the reason is simple. I myself live in the countryside. And the name is a light trolling over those who shout on the Internet “There is no life beyond the Moscow Ring Road!”, “The Russian village has drunk itself and is dying!” So, somewhere it may be so, and I will be an exception to the rule. I don’t drink, I don’t smoke, I do things that not every “city kreakl (s)” can afford. But let's get back to our sheep, or rather, the server, which, at the end of the first part of the article, was already "showing signs of life."
The board was lying on the table, I climbed through the BIOS, setting it up to my liking, dashed off Ubuntu 16.04 Desktop for simplicity and decided to connect a video card to the “super-machine”. But at hand was only a GTS 250 with a hefty non-native fan glued on. Which I installed in the PCI-E 16x slot near the power button.
“I shot on a pack of Belomor (s)”, so please do not kick for the quality of the photo. I'd rather comment on what is captured on them.
Firstly, it turned out that when installed in a slot, even a short video card rests on the memory slots, in which, in this case, it cannot be installed, and you even have to lower the latches. Secondly, the iron mounting plate of the video card covers the power button, so it had to be removed. By the way, the power button itself is illuminated by a two-color LED that lights up green when everything is in order and blinks orange if there are any problems, a short circuit and the PSU protection has worked or the + 12VSB power supply is too high or too low.
In fact, this motherboard is not designed to have video cards "directly" connected to its PCI-E 16x slots, they are all attached to the risers. To install an expansion card in the slots near the power button, there are corner risers, a low one for installing short cards up to the first processor heatsink, and a high corner one with an additional +12V power connector for installing a video card "above" the regular low 1U cooler. It can include large video cards like the GTX 780, GTX 980, GTX 1080 or specialized Nvidia Tesla K10-K20-K40 GPGPU cards or Intel Xeon Phi 5110p "computing cards" and the like.
But in the GPGPU riser, the card included in the EdgeSlot can be turned on directly only by connecting additional power with the same connector as on the high angle riser. Who cares - on eBay this flexible riser is called "Dell PowerEdge C8220X PCI-E GPGPU DJC89" and costs about 2.5-3 thousand rubles. Angle risers with additional power are much rarer and I had to arrange for them to be bought from a specialized server parts store through Shopotam. They came out at 7 thousand apiece.
I’ll say right away, “risk guys (tm)” can even connect a pair of GTX 980s to the board with Chinese flexible 16x risers, as one person did at the “That Very Forum”, the Chinese, by the way, make crafts that work quite well on PCI-E 16x 2.0 in the style of Thermaltek flexible risers, but if you once burn out the power circuits on the server board from this, you will only blame yourself. I did not risk expensive equipment and used original risers with additional power and one Chinese flexible one, believing that connecting one card “directly” would not burn the board.
Then the long-awaited connectors for connecting additional power arrived and I made a tail for my riser in EdgeSlot. And the same connector, but with a different pinout, is used to supply additional power to the motherboard. This connector is right next to this very EdgeSlot connector, there is an interesting pinout. If there are 2 wires +12 and 2 common on the riser, then there are 3 wires +12 and 1 common on the board.
That's actually the same GTS 250 included in the GPGPU riser. By the way, I take additional power to the risers and the motherboard - from the second + 12V power connector of the CPU of my PSU. I decided that it would be more correct to do so.
The fairy tale quickly affects, but slowly the parcels to Russia from China and other places on the globe go. Therefore, there were large gaps in the assembly of the “supercomputer”. But finally, a server Nvidia Tesla K20M with a passive heatsink came to me. Moreover, it is completely zero, from storage, sealed in its own box, in its own package, with warranty papers. And the suffering began, how to cool it down?
First, I bought a custom cooler from England with two small "turbines", here it is in the photo, with a homemade cardboard diffuser.
And they turned out to be complete crap. They made a lot of noise, the mount did not fit at all, they blew weakly and gave such a vibration that I was afraid that the components would fall off the Tesla board! Why were they sent to the trash almost immediately.
By the way, in the photo under Tesla, you can see LGA 2011 1U server copper radiators with a snail from Coolerserver, bought from Aliexpress, installed on the processors. Quite worthy though noisy coolers. They fit perfectly.
But actually, while I was waiting for a new cooler for Tesla, this time I ordered a large BFB1012EN snail from Australia with a mount printed on a 3D printer, it came to the storage system of the server. The server board has a mini-SAS connector through which 4 SATA and 2 more SATA are output with regular connectors. All SATA 2.0 standard but it suits me.
The integrated RAID intel C602 chipset is not bad and, most importantly, it skips the TRIM command for SSDs, which many inexpensive external RAID controllers cannot do.
On eBay, I bought a meter long mini-SAS to 4 SATA cable, and on Avito I bought a 5,25″ hot-swap basket for 4 x 2,5″ SAS-SATA. So when the cable and the basket arrived - 4 terabyte Seagates were installed in it, RAID5 for 4 devices was assembled in the BIOS, I started installing the server Ubuntu ... and stumbled upon the fact that the disk partitioning program did not allow me to make a swap partition on the raid.
I solved the problem head-on - I bought an ASUS HYPER M.2 x 2 MINI and M.4 SSD Samsung 2 EVO 960 Gb adapter in DNS PCI-E to M.250, decide that you need to select the fastest device for swap, since the system will work with high computational load, and the memory is still obviously less than the size of the data. And the memory was 256 GB more expensive than this SSD.
Here is the same adapter with an SSD installed in a low angle riser.
Anticipating questions - “Why not make the entire system on M.2 and have a maximum access speed higher than that of a SATA raid?” - I will answer. Firstly, for 1 or more TB, M2 SSDs are too expensive for me. Secondly, even after updating the BIOS to the latest version 2.8.1, the server still does not support booting NVE devices into M.2. I did experience when the system put /boot on USB FLASH 64 Gb and everything else on M.2 SSD, but I didn't like it. Although in principle such a bundle is quite efficient. If large-capacity M.2 NVEs become cheaper, I may return to this option, but for now SATA RAID as a storage system suits me quite well.
When I decided on the disk subsystem, I came up with a combination of 2 x SSD Kingston 240 Gb RAID1 "/" + 4 x HDD Seagate 1 Tb RAID5 "/home" + M.2 SSD Samsung 960 EVO 250 Gb "swap" it was time to continue my experiments with GPU. I already had a Tesla and just arrived an Australian cooler with an “evil” snail eating as much as 2.94A at 12V, the second slot was occupied by M.2 and for the third I borrowed a GT 610 “for experiments”.
Here in the photo all 3 devices are connected, and M.2 SSD through a flexible Thermaltek riser for video cards that works without errors on the 3.0 bus. It is like this, from many separate "ribbons" in the likeness of those from which SATA cables are made. PCI-E 16x risers made from a monolithic flat cable, sort of like the old IDE-SCSI - into the furnace, they torture you with errors due to mutual interference. And as I said, the Chinese now also make such risers as Termaltek's but shorter.
In combination with the Tesla K20 + GT 610, I tried a lot of things, at the same time finding out that when an external video card is connected and the output is switched to it, vKVM does not work in the BIOS, which did not really upset me. Anyway, I did not plan to use external video on this system, there are no video outputs on Teslas, and the remote admin panel via SSH and without X-owls works fine when you slightly remember what a command line without a GUI is. But IPMI + vKVM greatly simplifies management, reinstallation and other things with a remote server.
In general, this IPMI board is gorgeous. A separate 100 Mb port, the ability to reconfigure packet injection to one of the 10 Gb ports, a built-in Web server for power management and server control, downloading directly from it the vKVM Java client and the client for remote mounting disks or images for reinstallation ... The only thing is that clients under the old Java Oracle, which is no longer supported in Linux, and for the remote admin panel, I had to get a laptop with Win XP SP3 with this most ancient Toad. Well, the client is slow, there is enough for the admin panel and all that, but you can’t play toys remotely, the FPS is small. Yes, and the ASPEED video that is integrated with IPMI is weak, only VGA.
In the process of dealing with the server, I learned a lot and learned a lot in the field of professional server hardware from Dell. What I do not regret at all, as well as about the time and money spent with benefit. The continuation of the informative story about the actual assembly of the frame with all the components of the server will be later.
Link to part 3:
Source: habr.com