Mhoro Habr! Maseti edata eBig Data uye kudzidza muchina ari kukura zvakanyanya uye isu tinofanirwa kufambirana navo. Yedu chinyorwa nezve imwe tekinoroji yehunyanzvi mumunda weiyo yepamusoro performance computing (HPC, High Performance Computing), inoratidzwa paKingston booth pa. . Uku ndiko kushandiswa kweHi-End data kuchengetedza masisitimu (SDS) mumaseva ane graphic processing units (GPU) uye GPUDirect Storage bhazi tekinoroji. Kutenda kuchinjisa data pakati peiyo yekuchengetedza sisitimu neGPU, ichipfuura iyo CPU, kurodha data muGPU accelerators inokwidziridzwa nekurongeka kwehukuru, saka Big Data maapplication anomhanya pakunyanya kuita kunopihwa neGPU. Nekudaro, vagadziri veHPC system vanofarira kufambira mberi mumasisitimu ekuchengetera ane epamusoro I / O kumhanya, senge akagadzirwa naKingston.
Kuita kweGPU kunodarika kurodha data
Sezvo CUDA, GPU-based hardware uye software parallel computing architecture yekugadzira zvakajairika-chinangwa maapplication, yakagadzirwa muna 2007, hunyanzvi hwehardware hweGPU pachahwo hwakakura zvisingaite. Nhasi, maGPU ari kuwedzera kushandiswa mumashandisirwo eHPC akadai seBig Data, kudzidza muchina (ML), uye kudzidza kwakadzama (DL).
Ziva kuti kunyangwe kufanana kwemazwi, maviri ekupedzisira ari algorithmically akasiyana mabasa. ML inodzidzisa komputa zvichibva pane yakarongeka data, nepo DL inodzidzisa komputa zvichienderana nemhinduro kubva kune neural network. Muenzaniso wekubatsira kunzwisisa misiyano iri nyore. Ngatifungei kuti komputa inofanirwa kusiyanisa pakati pemifananidzo yekatsi nembwa dzakatakurwa kubva kune yekuchengetedza system. Kune ML, iwe unofanirwa kuendesa seti yemifananidzo ine akawanda ma tag, imwe neimwe inotsanangura chimwe chimiro chemhuka. Kune DL, zvakakwana kurodha nhamba yakakura kwazvo yemifananidzo, asi ine tag imwe chete "iyi katsi" kana "iyi imbwa". DL yakafanana zvikuru nekudzidziswa kwevana vaduku - vanongoratidzwa mifananidzo yembwa nekatsi mumabhuku uye muupenyu (kazhinji, pasina kana kutsanangura mutsauko wakajeka), uye uropi hwemwana pachahwo hunotanga kuona rudzi rwemhuka mushure. imwe nhamba yakaoma yemifananidzo yekuenzanisa ( Maererano nekufungidzira, tiri kutaura pamusoro pezviratidziro zana chete kana maviri muhuduku hwose). DL algorithms haisati yakwana: kuti neural network ishande zvakabudirira pakuziva mifananidzo, zvinodikanwa kudyisa nekugadzirisa mamirioni emifananidzo muGPU.
Pfupiso ye preface: zvichibva paGPUs, unogona kuvaka HPC zvikumbiro mumunda weBig Data, ML uye DL, asi pane dambudziko - seti yedata yakakura zvekuti nguva inopedzerwa kurodha data kubva kune yekuchengetedza system kuenda kuGPU. inotanga kuderedza kushanda kwese kwekushandisa. Mune mamwe mazwi, maGPU anomhanyisa anoramba asingashandiswe nekuda kwekunonoka I/O data rinouya kubva kune mamwe ma subsystems. Musiyano muI/O kumhanya kweGPU uye bhazi kuenda kuCPU / yekuchengetedza system inogona kuve yakarongeka yehukuru.
GPUDirect Storage tekinoroji inoshanda sei?
Iyo I/O maitiro inodzorwa neCPU, sezvakaita maitiro ekurodha data kubva pakuchengeterwa kuenda kumaGPU kuti iwedzere kugadziriswa. Izvi zvakakonzera kuchikumbiro chetekinoroji yaizopa mukana wakananga pakati peGPU neNVMe madhiraivha kuti akurumidze kutaurirana. NVIDIA yaive yekutanga kupa tekinoroji yakadai uye yakaidaidza kuti GPUDirect Storage. Muchokwadi, uku ndiko mutsauko weGPUDirect RDMA (Remote Direct Memory Kero) tekinoroji yavakambogadzira.
Jensen Huang, CEO weNVIDIA, acharatidza GPUDirect Storage seyakasiyana yeGPUDirect RDMA paSC-19. Kunobva: NVIDIA
Musiyano uripo pakati peGPUDirect RDMA neGPUDirect Storage uri mumidziyo iri pakati peinoitwa kero. GPUDirect RDMA tekinoroji yakadzoserwa kufambisa data zvakananga pakati pemberi-yekupedzisira network interface kadhi (NIC) uye GPU ndangariro, uye GPUDirect Storage inopa yakananga data nzira pakati penzvimbo kana kure yekuchengetedza seNVMe kana NVMe pamusoro peFabric (NVMe-oF) uye GPU ndangariro.
Ose ari maviri GPUDirect RDMA neGPUDirect Kuchengeta anodzivirira kusingakoshi kufamba kwedata kuburikidza nebuffer muCPU ndangariro uye bvumidza yakananga memory yekuwana (DMA) nzira yekufambisa data kubva kunetiweki kadhi kana chengetedzo yakananga kana kubva kuGPU ndangariro - zvese zvisina mutoro pakati peCPU. YeGPUDirect Storage, nzvimbo yekuchengetedza haina basa: inogona kunge iri NVME dhisiki mukati meGPU unit, mukati meraki, kana yakabatana pamusoro petiweki seNVMe-oF.
Scheme yekushanda kweGPUDirect Storage. Kunobva: NVIDIA
Hi-End ekuchengetedza masisitimu paNVMe ari kudiwa mumusika weHPC application
Uchiziva kuti nekuuya kweGPUDirect Storage, kufarira kwevatengi vakakura kuchakwezvwa kupa masisitimu ekuchengetera ane I/O kumhanya anoenderana nekukura kweGPU, paSC-19 kuratidzwa Kingston akaratidza demo rehurongwa hunosanganisira yekuchengetedza sisitimu yakavakirwa paNVMe disks uye unit ine GPU, iyo yakaongorora zviuru zvemifananidzo yesatellite pasekondi. Isu takatonyora nezve yakadaro yekuchengetedza sisitimu yakavakirwa pagumi DC10M U.1000 NVMe madhiraivha. .
Sitimu yekuchengetera yakavakirwa pa10 DC1000M U.2 NVMe madhiraivha anokwana zvakakwana sevha ine magiraidhi accelerators. Kunobva: Kingston
Iyi sisitimu yekuchengetera yakagadzirwa se1U kana yakakura rack unit uye inogona kuyerwa zvichienderana nehuwandu hweDC1000M U.2 NVMe madhiraivha, imwe neimwe iine 3.84-7.68 TB. Iyo DC1000M ndiyo yekutanga NVMe SSD modhi muU.2 fomu chinhu muKingston's mutsara wedata center drive. Iyo ine chiyero chekutsungirira (DWPD, Dhiraivha inonyora pazuva), ichiibvumira kunyorazve data kusvika pakuzara kwayo kamwe pazuva kwehupenyu hwakavimbiswa hwekutyaira.
Muiyo fio v3.13 bvunzo paUbuntu 18.04.3 LTS inoshanda sisitimu, Linux kernel 5.0.0-31-generic, sampu yekuchengetera yekuratidzira yakaratidza kumhanya kwekuverenga (Kuverengera Kuverengera) kwe5.8 miriyoni IOPS ine inogoneka throughput (Sustained Bandwidth. ) ye23.8 Gbit/s.
Ariel Perez, maneja webhizinesi reSSD kuKingston, akataura nezvehurongwa hutsva hwekuchengetera: "Takagadzirira kugadzirira chizvarwa chinotevera chemaseva neU.2 NVMe SSD mhinduro kubvisa akawanda emabhodhoro ekutamisa data ayo agara achibatanidzwa nekuchengetedza. Iko kusanganiswa kweNVMe SSD madhiraivha uye yedu yekutanga Server Premier DRAM inoita kuti Kingston ive imwe yeindasitiri yakawandisa-yekupedzisira-yekupedzisira data mhinduro vanopa. "
Iyo gfio v3.13 bvunzo yakaratidza kubuda kwe23.8 Gbps yedemo yekuchengetedza system paDC1000M U.2 NVMe madhiraivha. Kunobva: Kingston
Ndeipi yakajairika sisitimu yeHPC maapplication inoita senge kushandisa GPUDirect Storage kana yakafanana tekinoroji? Ichi chivakwa chine kupatsanurwa kwemuviri kwemayuniti anoshanda mukati meraki: imwe kana maviri mauniti e RAM, akati wandei eGPU neCPU computing node, uye imwe kana anopfuura mayuniti ekuchengetedza masisitimu.
Nechiziviso cheGPUDirect Storage uye nekubuda kwematekinoroji akafanana kubva kune vamwe vatengesi veGPU, kudiwa kwaKingston kwemasisitimu ekuchengetedza akagadzirirwa kushandiswa mumakomputa epamusoro ari kuwedzera. Mucherechedzo uchave kumhanya kwekuverenga data kubva mugadziriro yekuchengetedza, inofananidzwa nekubuda kwe40- kana 100-Gbit network makadhi pamusuwo wekombuta unit ine GPU. Nekudaro, ekupedzisira-yepamusoro-yekumhanyisa masisitimu ekuchengetedza, kusanganisira yekunze NVMe kuburikidza neFabric, ichabva kubva kune exotic kuenda kune mainstream kune HPC application. Pamusoro pesainzi uye kuverenga kwemari, vachawana kushanda mune dzimwe nzvimbo dzakawanda dzinoshanda, senge masisitimu ekuchengetedza paSafe City metropolitan level kana nzvimbo dzekutarisa dzekufambisa, uko kunofanirwa kuzivikanwa nekuzivikanwa nekumhanya kwemamiriyoni emifananidzo yeHD pasekondi, "akadaro. iyo musika niche yepamusoro Kuchengeta system
Rumwe ruzivo nezveKingston zvigadzirwa zvinogona kuwanikwa pa kambani.
Source: www.habr.com