Good day, Khabrovites! I will continue my story of assembling a "supercomputer in the village."
I will begin the third part by expressing sincere gratitude to my friends who supported me in difficult times, motivated me, helped me with money by sponsoring this rather expensive business for a long time and even helped with the purchase of components from abroad in cases when I could not buy them on direct myself. For example, if a company selling server spare parts in the USA or Canada simply did not send to Russia. Without their long and regular help, my progress would have been much more modest.
Also, thanks to their requests, I decided and started an account on Youtube, bought an old Lumia 640 smartphone, which I use exclusively as a video camera, and started making educational videos, both about assembling a “village supercomputer” and about other aspects and projects of my village life.
Village supercomputer playlist:
Those who wish to spoilers can familiarize themselves, although of course it is better to do this in the process of reading my story or even after.
The second part of my story was interrupted by the fact that I connected the Tesla K20M, GT 610 and M.2 NVE SSD + disk array to the system. By the way, what else is good about this Dell board - it has a built-in “disk shelf”, albeit for only 6 devices, and RAID is not “the most sophisticated in the world”, but unlike its more professional external counterparts, it skips the TRIM command on SSD. Which is also important if you use non-professional server SSDs intensively.
By the way, there is also one interesting and important point about this board. Chipset heatsinks are low with small fins. This works well when the board is in the original rack, where powerful turbines blow it along. But when using the board separately, you need to remove the plastic sticker from the radiator closest to the expansion slots, and it is advisable to replace the far one with any suitable radiator from the chipset of the old motherboard with large fins, because the chip located under it heats up the most on the board.
Having removed the video card from the system, I began to assemble a frame for my server, in the test version everything was on electrical tape, matchboxes and other plastic props, but for the full use of 24/7/365 this option did not seem acceptable to me. It was necessary to make a normal frame from an aluminum corner. Aluminum corners from Leroy Merlin were used, which were sent to me by a friend from the Moscow region, in his nearby city they simply were not sold anywhere at all!
In addition to the corners, M5 countersunk screws and nuts, M3 screws and nuts, small furniture corners, aluminum rivets for 5 mm holes, a riveter, a hacksaw for metal, a screwdriver, a 5.0 mm drill for metal, a file, a Phillips screwdriver, cable ties and hands growing not from the ass.
The corners were used to fasten the board to the frame and some other elements. This, of course, added some height to the entire system, because the board turned out to be raised quite high above the bottom plane of the frame, but I decided that this was acceptable for me. I didn’t fight for every gram of weight, millimeter of height, after all, this is not an on-board computer of an aircraft where the standard is “15 G in 3 axes, shocks up to 1000 G and vibration”.
The board is installed, the risers are screwed, the adapter with SSD M.2 is screwed on.
The board, SSD, risers and Tesla are installed in their places. DC-DC has not yet been screwed into place and it hangs on the wires behind the scenes. This is server version 1.0, still on one Tesla K20M.
Here DC-DC is already fixed on the frame, there it is a small scarf on the side behind the motherboard under the power “tails”.
And this is the system already assembled, top view. Above Tesla, one more time from the corners in which a pair of SSDs are screwed side by side, above them is an HDD basket, and on top of the frame that closes the frame hangs a PSU 850 W Thermaltek modular. The PSU is fashionable, gaming, with RGB backlighting, which I turned off so that it does not blink like a Christmas tree. The only powerful modular PSU at that time in the stores of a nearby city.
Side view of the server version 1.0.
View of the server from the "front end". I made connectors and rivers for disks on one side, as in server systems, so that for all manipulations it was not necessary to turn the entire system back and forth. On the “cutout bar”, a takeaway with two USB 2.0 is screwed on, which I connected instead of a card reader, and an M.2 adapter board is screwed to its bottom.
Here it is shown how the DC-DC and the board are fixed, the very corners that I spoke about.
View from the other side, how the GPGPU riser is fixed, which is EdgeSlot.
The same high angle riser with additional power for the GPGPU that I bought through Shopotam from America.
The machine was assembled, the operating system was installed, the drivers were installed, the CUDA Toolkit was configured ...
Here is a short video about her.
In this form, the system with one Tesla K20M 5 GB worked for half a year, while my astronomer friend considered his tasks. Then he went on vacation and suddenly eBay showed up server Tesla K20X 6 GB for 6000 used, like a sale from a data center in England. And we decided to assemble the second version of the "supercomputer" already on 3 Tesla K20X.
Teslas were bought, a second motherboard was bought exactly the same, only they decided to save on shipping and chose delivery by the eBay service. Who took her to SPAIN and gave her away to some completely leftist type. A dispute was opened on eBay, the seller from the USA supported me and the money was returned, and already the third board went to me by the usual but reliable USPS. Other spare parts also arrived and here is a video about the beginning of the assembly of the “village supercomputer” 2.0.
Video about spare parts for this very "machine".
Launch board and some features.
Here I began to assemble the frame of the second version of the server.
Tesla K20X arrived, the first video.
Informative video about the Tesla K20X, about the design of the card and its cooling system, and a bummer with a water block from the GTX 780 Ti.
Continuation of the video about Tesla K20X, scanned its board on the scanner, if anyone suddenly needs it.
Front side with GPU chip.
Backside.
As you can see, Tesla K20, although it is “in general terms” similar to the GTX 780 GTX 780 ti GTX TITAN on the GK110 Kepler GPU, is nevertheless not compatible with them in terms of the board and cooling system. If I have a Quadro K5200 K6000 GK110 Kepler, then I will compare its board with the Tesla K20 board, but so far I don’t have any of the above Quadros.
And here is the continuation of the assembly of the server 2.0
Again 1U coolers with snails and other things that are required for a server with more power than the first one. By the way, I had to disassemble the first server in order to assemble the second one, while my friend did not have an urgent need to count.
A little cable management...
And the second Tesla is installed in its place.
And here I was comprehended by an offensive bummer. It turned out that the system does not pull 3 Tesla K20 pieces. At the start of the BIOS, such an error pops up and that's it, the third Tesla does not work in any way. Even updating the BIOS to version 2.8.1 did not help, after which the board from the Dell DCS 6220 turned into the Dell C6220 2.8.1. I turned on and off various options in the BIOS, even tried to tape some of the Tesla contacts with tape to make them 8x - nothing helped. I had to accept and stop at a configuration of 2 Tesla K20X + NVE SSD. By the way, in version 2.0 of the server, all SATA drives live in one Chinese basket with 6 bays. Now there is a pair of Samsung 860 EVO 500 Gb + 4 terabyte Seagates. I bought Samsungs on Ali for 3600 apiece. OEM wheels, but I'm fine with it.
Now "supercomputer 2.0" is fully assembled and ready to go.
In other matters, the spare parts bought for the second system arrived and I picked up the first one back, here about this video.
And I invite readers to vote what to do with the first board? What is interesting to collect on its basis? Or if someone wants to buy it like Tesla K20M and K20X with or without snail coolers, I'm ready, write.
Here is such a story, I hope it turned out to be interesting and useful to dear readers.
PS: For those who had the patience to read to the end - subscribe to my YouTube channel, comment, rate like / dislike - this will motivate me for further publications and shooting new educational videos.
Source: habr.com