Indlela yokwakha ingqalasizinda corp. ikilasi ngokusetshenziswa kwe-Dell R730xd E5-2650 v4 amaseva abiza u-9000 euros ngepeni?

Kulesi sihloko sizokutshela ukuthi yiziphi izixazululo zengqalasizinda amakhasimende ethu azisebenzisayo zisebenzisa izinkundla zeseva ye-Dell R730xd nokuthi kungani intengo yokuqasha le nkundla ingu Izinga le-TierIII+ lesikhungo sedatha saseYurophu ngeziteshi ezinhle kakhulu zokuxhumana e-Ukraine naseRussia, kanye ezindaweni ezingu-9 e-USA, esivele inokubekwa nokuxhumeka ngentengo kusuka ku-$249 / ngenyanga nge-2 x Intel Dodeca-Core Xeon E5-2650 v4 128GB DDR4 6x480 SSD 1Gbps isibe ngokoqobo. Sizokwabelana ngezixazululo ezingenzeka ngokusekelwe kulawa mapulatifomu sisebenzisa i-vlan yangasese, inethiwekhi yendawo ye-10G kanye ne-hardware Firewall evela ku-CISCO, etholakala kumakhasimende ethu uma ecelwa. Futhi, emasikweni angcono kakhulu, sizonikeza ibhonasi ngendlela yesikhathi samahhala sokusebenzisa amaseva we-Dell R730xd kubafundi be-Habrahabr.

Indlela yokwakha ingqalasizinda corp. ikilasi ngokusetshenziswa kwe-Dell R730xd E5-2650 v4 amaseva abiza u-9000 euros ngepeni?

Muva nje, besilokhu sithola izicelo ezengeziwe zokwakhiwa kwengqalasizinda ehlukahlukene yezinkampani, futhi lokhu, ngeshwa, akuxhumene kakhulu neqiniso lokuthi sinikeza izixazululo zekhwalithi ephezulu kakhulu, kodwa ngentengo yalezi zixazululo kanye nezinga ukuphepha kanye nokubusa komthetho, okunikezwa eNetherlands nase-USA, kodwa, maye, ngokuvamile akutholakali e-Ukraine naseRussia. Lapho, ngeshwa, izixazululo ezinjalo zibiza imali β€œyezinkanyezi” nje, njengoba umqondo wokuthi "imali yesikhathi eside" ungaziwa emazweni angemva kweSoviet, ngokuphambene nesizinda sezinye izingozi noma ngenxa yokuntuleka kwengqalasizinda edingekayo kanye nezinga isitifiketi.

Isibonelo esihle lapha kungaba umehluko wentengo yehadiwe. Isb, Ipulatifomu ye-Dell R730xd, ithengelwe amakhasimende ethu, ekucushweni okuyisisekelo, i-2 x Intel Dodeca-Core Xeon E5-2650v4 128GB DDR4 6x480GB SSD ibiza cishe ama-euro ayi-9000. Kuyacaca ukuthi angeke kube khona umbuzo wanoma iyiphi intengo ephansi yokuqasha le seva e-Ukraine noma e-Russia, njengoba izindleko zokuqasha zisekelwe enkokhelweni yezindleko zesikhulumi phakathi nenkathi yezinyanga ezingu-12-18. Lokhu kusho ukuthi inani elincane elingase liqashiswe, ngaphandle kwezindleko zendawo yokuhlala, ugesi kanye neziteshi zokuxhumana, lizoba ku-$500-800 / ngenyanga, kuye ngezinga lobungozi bomhlinzeki kanye nohlelo lwebhizinisi. Udinga futhi ukuthola isikhungo sedatha esihle esinezinga elidingekayo lesitifiketi neziteshi zokuxhumana. Hhayi-ke, ungakhohlwa ngokucabangela izingozi zokubanjwa ngokungemthetho kwemishini phakathi nezenzo zophenyo ezingekho emthethweni ezingase zivele ngenxa yokuncintisana kanye nemininingwane yokwenza ibhizinisi endaweni ethile.

Ngakho-ke, akumangazi ukuthi isiphakamiso sethu sivuse isithakazelo esikhulu, ikakhulukazi njengoba izikhungo zedatha lapho sihlinzeka khona ngamaseva ahlongozwayo zinezitifiketi eziningi ezidingekayo nezibalulekile emkhakheni wezinkampani - ISO 27001, I-PCI DSS, I-SOC 1, I-HIPAA ΠΈ I-NEN7510.

Indlela yokwakha ingqalasizinda corp. ikilasi ngokusetshenziswa kwe-Dell R730xd E5-2650 v4 amaseva abiza u-9000 euros ngepeni?

Manje ake sibheke izimo ezimbalwa ezingase zibe ezithakazelisayo kwabaqalayo namaphrojekthi amancane, kanye nabahlanganisi besistimu abakhulu abakha izixazululo emahhovisi abo ezindaweni zokuqukethwe noma ezikhungweni zedatha.

Izinzuzo zamapulatifomu e-Dell R730xd lapho kwakhiwa izindawo zokugcina idatha kusetshenziswa i-Red Hat Ceph

Akuyona imfihlo ukuthi izidingo zokugcinwa nokucubungula idatha zikhula njalo, futhi ngesivinini esisheshayo. Uma eminyakeni embalwa edlule kwakwanele ukuba ne-1TB yokugcina, enikeze amakhulu amaningana e-IOPS, manje izidingo ziye zanda zaba amashumi ezinkulungwane ze-IOPS nama-petabytes esikhala. Lezi zidingo zomthamo nokusebenza zigqugquzelwa ngokwengxenye ukwanda komthamo wedatha engahlelekile, okuhlanganisa umculo, izithombe, amavidiyo, izipele zolwazi olugciniwe, amafayela okungena emininingwaneni nezinye izingobo zomlando, idatha yezezimali neyezokwelapha - idatha eyaziwa ngokuthi "Idatha Enkulu". Ingasaphathwa eyezidingo ezikhulayo zokugcinwa kwedatha ngenxa yokwanda nokunwetshwa kwe-inthanethi kanye nezinsiza ezintsha ze-inthanethi. Ngakho-ke, ngazo zonke lezi zimfuno zamandla, okulindelwe ngamakhasimende ngokuthembeka okuphezulu nokusebenza okuphezulu kukhulu kunangaphambili.

Nakuba izinkampani ze-IT zizama ukuxazulula izinkinga zokuphatha ama-petabytes ngisho nama-exabytes edatha, imodeli yokugcina ifu iya ngokuya ivame kakhulu ezikhungweni zedatha zanamuhla. Kulokhu kubhalwa izinhlelo ezintsha zesoftware ezikuvumela ukuthi ulungiselele ukusebenzisana kwemvelo yamafu nehadiwe ngendlela efanele; enye yentuthuko enjalo yiCeph.

I-Ceph iwumthombo ovulekile wesistimu yesitoreji esabalalisiwe eklanyelwe ukuhlinzeka ngokusebenza okuphezulu, ukwethembeka kanye nokulinganisa. I-Ceph isebenzisa ukugcinwa kwento kuqoqo lekhompiyutha esabalalisiwe futhi ihlinzeka ngendawo yokusebenzela yokugcina izinto, amabhlogo namafayela. I-Ceph inikeza isitoreji esisatshalaliswe ngokugcwele ngaphandle kwephuzu elilodwa lokwehluleka kanye nobukhulu bezinga le-petabyte. I-Ceph iphinda idatha futhi ngaleyo ndlela inikeze ukubekezelelana kwamaphutha. Uhlelo luklanyelwe ngendlela yokuthi lunganikezeli nje kuphela ukutakula okuzimele, kodwa futhi nokuphatha, okusiza ukugwema izindleko zokulondoloza ezingadingekile. Ngenxa yokuthi i-Ceph isebenzisa i-hardware yenjongo evamile elawulwa isofthiwe nemisebenzi yokuphatha itholakala ngokusebenzisa i-application programming interfaces (APIs), ihlukaniswa njenge-software-defined storage (SDS).

I-Red Hat Ceph Storage iwuhlelo lokugcina oselulungele ukusetshenziswa, isitoreji esichazwe nge-software esivulekile, esivumelana nezimo, esikalekayo futhi esisekelwa yonke indawo. Ihlanganisa izinto ezintsha ezivela ekuthuthukisweni komthombo ovulekile nomongo wobuchwepheshe nokusekelwa okuvela ku-Red Hat. Isixazululo sihlinzeka ngokuhlanganiswa okuqinile ne-OpenStack futhi yakhelwe kusukela phansi ukuze inike amandla isitoreji sesizukulwane esilandelayo kuzo zombili izindawo zamafu neminye imisebenzi.

Lapha sizobheka isibonelo sokuqalisa lesi sixazululo kumaseva akwa-Dell, ikakhulukazi i-Dell PowerEdge R730xd, esiyihlinzeka ngokuqashiswa, futhi sicabangele izinzuzo zesitoreji sokwakha ngokusekelwe kulawa mapulatifomu. Lolu lwazi lungase lube usizo kubaklami bezakhiwo, onjiniyela, nabaphathi be-IT abafuna ukuhlola izinzuzo zokusebenzisa i-Red Hat Ceph Storage kumaseva e-Dell PowerEdge futhi okufanele baklame futhi bahlele ukuqaliswa besebenzisa imikhuba emihle efakazelwe.

Kodwa ekuqaleni:

Okuncane mayelana nenkundla ngokwayo, iyiphi inketho esiyinikezayo futhi kungani ibiza kakhulu?

I-Dell PowerEdge R730xd ingenye yezinkundla ezingcono kakhulu zemisebenzi yebhizinisi nangale kwalokho, isiwine imiklomelo eminingi ehlukene futhi iyisixazululo esisebenzayo sokulondoloza, njengoba inganikeza isitoreji esiminyana kakhulu ngemali ephusile.

Indlela yokwakha ingqalasizinda corp. ikilasi ngokusetshenziswa kwe-Dell R730xd E5-2650 v4 amaseva abiza u-9000 euros ngepeni?

I-PowerEdge R730xd itholakala ekucushweni kwe-chassis okuthathu okunezici ezihlukahlukene zefomu kanye nezinhlanganisela zedrayivu, kanye neyunithi yokwandisa yangemuva engakhethwa:

- Amatheku angu-24" afinyeleleka ngaphambili angu-2,5" we-SAS, SATA noma amadrayivu e-SAS aseduze kanye namatheku angu-2 angakhethwa angu-2,5" angemuva. I-chassis engu-2,5-intshi ingasekela ngokuzithandela kufikela ku-4 PCIe Express Flash Drives ukusuka kuDell ngaphambili.
- 12 ngaphambili okufinyeleleka kuyo 3,5" SAS, SATA noma eduze ne-SAS drive izindawo ezinezikhethelo ezingu-4 zangaphakathi ezingu-3,5" zokushayela ezishisayo, kanye namatheku amabili ongawakhetha angu-2,5" ngemuva.
- Amatheku angu-18" e-SATA afinyeleleka ngaphambili angu-1,8, amatheku angu-8 3,5" amadrayivu e-SAS, e-SATA noma aseduze ne-SAS, kanye nezingu-2 ozikhethela zona ezingu-2,5" zangemuva.

Kubukeka sengathi i-chassis ngayinye ilungele uhlobo lwayo lomsebenzi. Lokhu kuqinisekile. Kodwa ingabe abiza ngokulinganayo ngempela?

Indlela yokwakha ingqalasizinda corp. ikilasi ngokusetshenziswa kwe-Dell R730xd E5-2650 v4 amaseva abiza u-9000 euros ngepeni?

Sikhethe inketho yesibili ukuze sanelise izidingo zohlu olubanzi ngangokunokwenzeka lwabasebenzisi ku-chassis enama-drive bays angu-12. Ngoba siyibheka njengephumelela kakhulu. Futhi yingakho. Ukusebenza kahle kwezomnotho kwesixazululo sekuvele kubonakaliswe ekusebenzeni kwayo okuhlukahlukene - amadrayivu ezinto ezahlukahlukene angafakwa kule chassis futhi ishibhile ukuthenga, futhi, okubalulekile, ungathola umkhiqizo omkhulu lapho usebenza nedatha usebenzisa amapulatifomu amaningana axhumene kunalokho. kunesisodwa esinomthamo omkhulu wesitoreji kanye nokukhiqiza okuphansi ngokusobala.

Sikholelwa ukuthi ngokuxhuma amapulatifomu amaningana afanayo kunethiwekhi yendawo ye-multi-gigabit (kungenzeka ukuxhuma indawo ngayinye kunethiwekhi yendawo ngesivinini esingu-20 Gbit / s noma ngaphezulu, usebenzisa amakhadi we-dual-port ten-gigabit Intel X540-T2, esikunikeza ngaphezu kwalokho), singakwazi ukuthola imiphumela engcono kakhulu nokusebenza kunezingxenyekazi ezinokuminyana kwesitoreji esiphezulu. Lokhu kubaluleke kakhulu ezingqalasizinda ezisebenzisa lezi zinkundla ukuze zisebenze nesizindalwazi. Maye, ngenani elikhulu lokushayela, isilawuli singalayishwa ngokweqile, futhi isilawuli esengeziwe enguqulweni ye-xd, maye, asitholakali. Ukusebenza okungafinyelelwa uma usebenzisa amapulatifomu anama-bay angu-12 kanye nenethiwekhi yendawo ye-multi-gigabit izoba phezulu kakhulu, futhi isisombululo, sizosatshalaliswa kakhulu futhi sithembeke. Ngamafuphi - kuyabiza!

Izici zesakhiwo nokuklama, ukusetshenziswa kwamakhadi wevidiyo

Isiteji seDell PowerEdge R730xd, esikala i-2U kuphela, sisekela amaphrosesa angu-2 we-Intel Xeon E5-2600 v3 nangaphezulu, okungukuthi, ikuvumela ukuthi ukhuphukele kuma-cores angu-36 uma usebenzisa ama-processor angu-18-core. Sikhethe inketho emaphakathi, kepha isizukulwane sakamuva - iprosesa engu-12-core E5-2650 v4 (ama-cores angama-24 onawo esewonke, futhi kucatshangelwa ukucutshungulwa okuningi - ama-cores angama-48), ngoba kube yindleko enkulu kakhulu. -esebenzayo. Ngakho-ke, esizukulwaneni sesine, imiyalelo yeprosesa esebenza kahle isetshenziswa, isibonelo, i-AES, enesibopho sokubethelwa kwedatha, i-70% isebenza kahle kakhulu kunamaphrosesa afanayo, kodwa isizukulwane sesithathu. Ngesikhathi esifanayo, iphrosesa neplatifomu ingasekela kufika ku-1,54 TB ye-RAM, okubalulekile kwezinye izimo eziqondile kakhulu. Sikhethe inketho ethandwa kakhulu, engabizi kakhulu ngokwentengo nesivinini sokusebenza - 128GB DDR4 RAM futhi sinikeze ithuba lokuthuthukisa ngesicelo sababhalisile.

Ephaneli elingaphambili le-R730xd kunezinkomba zesimo sesistimu ezingu-6 ezingakwazisa ngezinkinga ezihlukahlukene, ukuze ukwazi ukugwema izinkinga eziningi ezibucayi ngokuthatha izinyathelo ezifanele ngesikhathi esifanele. Izikhala ze-DIMM ze-RAM zitholakala ngqo ebhodini lomama. I-R730xd isekela ama-DIMM okulungisa amaphutha abhalisiwe kanye nama-LRDIMM (Layisha Amamojula Ancishisiwe Enkumbulo Ekabili Yangaphakathi Komugqa), uhlobo olusha lwememori uma kuqhathaniswa namaseva. Kodwa asiyisebenzisi, njengoba ama-LRDIMM eba usizo kuphela ngenani elikhulu lenkumbulo, lapho inhloso kuwukukhuphula isivinini sokusebenza.

I-Internal Dual-SD Module (IDSDM) ivumela amakhasimende ukuthi asebenzise ubuchwepheshe be-Virtualization obubekezelela amaphutha ka-Dell, obuhlinzeka ngokuphindaphindeka kwe-failover kuma-hypervisors ashumekiwe. Nakuba i-PowerEdge R730 isekela ama-GPU, angaba wusizo kakhulu ezindaweni zasehhovisi (amadeskithophu angokoqobo), kanye nasekusebenzeni okuphezulu kwekhompuyutha nokusebenza ngokubambisana, i-R730xd ayiwasekeli ama-GPU ngoba ukupholisa okufanele akukwazi ukuhlinzekwa ngamakhadi evidiyo. Nokho, kuze kube manje sinesidingo esincane sale sevisi futhi oyedwa kuphela kwababhalisile bethu o-ode ikhadi levidiyo leseva. Kungalesi sizathu ukuthi asizange si-ode amapulatifomu ezinkulungwane ezingama-R730 ngobuningi, kodwa singawadiliva lapho esicelile, sifakwe elinye lamakhadi anconyiwe.

Ngenxa yalokhu, ngeshwa, intengo yokuqasha isixazululo ayikwazi ukukhanga kakhulu futhi ikhuphuka izikhathi ezingaphezu kwezingu-2, kuye ngokuthi isikhathi sokukhokha kanye nesikhathi senkontileka. Sinikeza ukuthenga amakhadi ngokwawo ngomphakeli wethu eNetherlands, amanani anikezwe ngezansi futhi kuphela kulolu hlu olunconyiwe (lezi kwakuyizincomo zikaDell uqobo lapho senza isicelo), mhlawumbe lolu lwazi luzoba usizo kumuntu othile:

I-NVIDIA Tesla M10 GPU CusKit: 2,884.98 EUR
I-NVIDIA Tesla M40 GPU: 4,913.33 EUR
I-NVIDIA Tesla M40 24GB GPU, Cust Kit: 6,458.95 EUR
I-NVIDIA M60 GPU, I-Passive, Idinga I-GRID 2.0 SW Yomsebenzi We-VDI, Ikhithi Ye-Cust: 5,094.95 EUR

Amalayisense:
Ilayisense ye-Nvidia GRID vApps Yokubhalisa iminyaka emi-3, 1 CCU: 20 EUR
Ilayisense Yokubhalisa ye-Nvidia GRID vPC iminyaka emi-3, 1 CCU: 95 EUR
Ilayisense Yokubhalisa ye-Nvidia GRID vWS 3 iminyaka, 1 CCU: 480 EUR

Ngakho-ke, uma usukulungele okungenani inkontileka yeminyaka emi-2 yokuqasha iseva ye-Dell R730 (hhayi i-xd, yingakho isisombululo sibiza kakhulu) - thintana naye [i-imeyili ivikelwe], sizokujabulela ukukusiza! Usebenzisa elinye lamakhadi evidiyo anconywe ngenhla, ku-DELL R730 2 x E5-2650 v4 / 128GB / 6 x 480GB SSD / 1Gbps 100TB + GPU ukucushwa kanye nenkontileka yeminyaka emi-2, kuzobiza cishe u-$6816 ngonyaka, esikhundleni sika-$2988. ngonyaka esimweni I-platform ye-Dell R730xd + okuxoxwe ngayo esihlokweni izodinga inkokhelo yezindleko zekhadi levidiyo nelayisense, ngoba lokhu kuyimishini ecacile kakhulu.

Indlela yokwakha ingqalasizinda corp. ikilasi ngokusetshenziswa kwe-Dell R730xd E5-2650 v4 amaseva abiza u-9000 euros ngepeni?

Kodwa-ke, ngisho nale ntengo iyathandeka kakhulu kunamanani entengo izikhungo zedatha e-Ukraine naseRussia ezinganikeza izixazululo ezinjalo, uma zifuna ukunikeza izixazululo ezinjalo nhlobo ... Ngokuba, ngokungavamile, eNetherlands, lapho ukubambisana noDell. isungulwa ngokuqondile, isikhathi esithenjisiwe sokulethwa kweseva enekhadi levidiyo sasicishe sibe izinyanga ezi-2 kusukela ngesikhathi iklayenti lethu lifaka i-oda (iklayenti livumile ukulinda, njengoba zingekho ezinye izindlela emakethe), ngenxa yokucaciswa okuqinile kwalo mkhiqizo U-Dell umane wayengenayo esitokweni. Noma kunjalo, ukulethwa kwaqedwa esikhathini esingaphansi kwenyanga. Sibonga umnyango wokuhlinzeka ngesikhungo sedatha kanye nezinsizwa ezivela kwa-Dell ngokusebenza kahle kwazo. Kodwa kimina lokhu okuhlangenwe nakho kwakungajwayelekile, njengoba amapulatifomu ajwayelekile e-Dell R730xd alethwa phakathi nezinsuku ezimbalwa.

I-PERC Controller Controls kanye Namakhono

Iseva ingaphathwa nge-iDRAC8 elula (I-Integrated Dell Remote Access Controller 8) ene-Dell Lifecycle Controller, enciphisa isikhathi esichithwa emisebenzini yokuphatha, inciphisa amathuba okuba namaphutha, ithuthukise ukuphepha futhi ithuthukise ukusebenza kahle konke kwendawo yakho ye-IT.

Indlela yokwakha ingqalasizinda corp. ikilasi ngokusetshenziswa kwe-Dell R730xd E5-2650 v4 amaseva abiza u-9000 euros ngepeni?

Kwababhalisile bethu, ukufinyelela ku-iDRAC8 kunikezwa nge-RMI (I-Remote Management Interface), efinyeleleka ngokukhethekile kunethiwekhi yesikhungo sedatha yangasese ngezinjongo zokuphepha, ukufinyelela esikunikeza ngomhubhe we-Open VPN mahhala. Uma isingenile, i-iDRAC ibonisa uhlolojikelele lwesistimu kanye nokubukwa kwekhonsoli ebonakalayo nge-iKVM.

Ngingathanda ukunaka ngokukhethekile ukuqapha; i-iDRAC8 ikuvumela ukuthi uthole izibalo zokusetshenziswa kwamandla zehora lokugcina, usuku noma isonto futhi ukhawulele ukusetshenziswa kwamandla okuphezulu:

Indlela yokwakha ingqalasizinda corp. ikilasi ngokusetshenziswa kwe-Dell R730xd E5-2650 v4 amaseva abiza u-9000 euros ngepeni?

Isistimu yokuqapha izinga lokushisa ne-iDRAC yokurekhoda ikuvumela ukuthi uqoqe idatha kusuka kuzinzwa zokushisa futhi unqume ububanzi lapho itholakala khona. Ngakho-ke, kubhekwa njengokwamukelekayo ukuthi abacubungulayo babe sebangeni lokushisa Lesexwayiso kuphela ngo-10% wesikhathi phakathi nonyaka, futhi Okubalulekile ku-1%. Isikhathi kubhendi ebucayi siphinde sithinte isikhathi esivumelekile kubhendi yesixwayiso. Ukuqoqwa kwedatha yezinga lokushisa kuqala lapho isistimu ivuliwe ngemva kokuphuma embonini futhi ayikwazi ukusethwa kabusha.

Indlela yokwakha ingqalasizinda corp. ikilasi ngokusetshenziswa kwe-Dell R730xd E5-2650 v4 amaseva abiza u-9000 euros ngepeni?

Isizukulwane se-13 samaseva we-Dell PowerEdge ifakwe izilawuli ze-PERC9, lapho ubuchwepheshe be-LSI CacheCade obusetshenziswa ngaphambilini kuzilawuli ze-PERC8 buthathelwe indawo obusebenza kahle kakhulu - I-DAS Cache evela ku-SanDisk.

Indlela yokwakha ingqalasizinda corp. ikilasi ngokusetshenziswa kwe-Dell R730xd E5-2650 v4 amaseva abiza u-9000 euros ngepeni?

Ezivivinyweni zokwenziwa, Inqolobane ye-DAS isheshisa kakhulu ukusebenza kwamalungu afanayo e-HDD ezimeni zokucushwa kwe-HDD+SSD eyingxube. Ngakho-ke, endabeni yohlu lwama-HDD angu-5 ku-RAID6 (izinga lesi-6 likhethiwe ukuze linikeze umthamo omkhulu) kanye ne-5 SSD RAID10 (4 + 1 hot spare SSD ukuze kwandiswe ukusebenza kohlu) uma usebenzisa i-DAS Cache, ukusebenza kwe uhlu luyi-5 HDD RAID6 + DAS Cache kuhlu olucacisiwe lwe-SSD kuvele ukuthi lisondele ekusebenzeni kohlu lwe-SSD ngokwalo:

Indlela yokwakha ingqalasizinda corp. ikilasi ngokusetshenziswa kwe-Dell R730xd E5-2650 v4 amaseva abiza u-9000 euros ngepeni?

Indlela yokwakha ingqalasizinda corp. ikilasi ngokusetshenziswa kwe-Dell R730xd E5-2650 v4 amaseva abiza u-9000 euros ngepeni?

Ngakho-ke, njengoba sesijwayele kancane ngesiteji, manje sesingakwazi ukubheka izinzuzo zalesi sigcawu lapho sakha izindawo zokugcina.

Yini eyenza iplatifomu ye-Dell i-R730xd ibe ilungele ukuthunyelwa kwe-warehouse data, ikakhulukazi i-Red Hat Ceph?

Indawo Yesitoreji Se-Red Hat Ceph isebenzisa amaseva ajwayelekile emboni ukuze alinganisele, aqine, futhi asebenze. Izindlela zokuvikela idatha zidlala indima ebalulekile ezindlekweni zesixazululo. I-Ceph ivumela umsebenzisi ukuthi akhethe izindlela zokuphepha ezihlukene zamachibi okugcina ahlukene.

Amachibi okugcina aphindaphindwe akhiqiza amakhophi aphelele ezinto ezigciniwe futhi alungele ukutholwa ngokushesha kanye nokukhohlisa idatha. Endabeni yechibi eliphindiwe lokugcina, i-Ceph ishintsha ibe yisici sokuphindaphinda kokuthathu lapho amakhophi amathathu edatha ehlala ezindaweni ezintathu ezihlukene ze-Ceph.

Amachibi okugcina amelana nenkohlakalo ahlinzeka ngekhophi eyodwa yedatha ngokulinganayo, okuwusizo lapho kudingeka ukugcinwa kwedatha yesikhathi eside futhi kubiza izindleko.

I-PowerEdge R730xd iyiseva yerack yamayunithi amabili eguquguqukayo futhi eyingozi eletha ukucutshungulwa kokusebenza okuphezulu kanye nohlu olubanzi lwezinketho zesitoreji zasendaweni ezilungiselelwe umsebenzi, okuhlanganisa ama-hybrid tiers. Ukuthuthukiswa kufaka phakathi ukucushwa okubanzi, okulungele i-Ceph.

β€” I-R730xd inikeza amandla okudala izixazululo zengqalasizinda ezisebenza kahle kakhulu kumabhizinisi amancane namakhulu.
β€” U-R730xd wehlisa isikhathi sokusebenzisa ngamakhono okuthumela okuzenzakalelayo, okunciphisa okokufaka komsebenzisi futhi kunciphise amaphutha abantu.
β€œAmaseva e-PowerEdge athuthukisa ukusebenza kwe-IT yesikhungo sedatha ngamathuluzi okuphatha amasha afana ne-iDRAC Quick Sync kanye ne-iDRAC Direct, ehlinzeka ngokubonakala kwezempilo kwesistimu okuphelele nokuthunyelwa ngokushesha.
β€œAmaseva e-PowerEdge athuthukisa ukusetshenziswa kwamandla, alethe ukusebenza okungcono nge-watt ngayinye nokulawula okunembe kakhudlwana kokusetshenziswa kwamandla nokupholisa.

I-Dell PowerEdge R730xd inikeza izinzuzo eziqinisekisa ukukhiqiza okuphezulu ngokwenza lokhu:

- Sheshisa ukusebenza kahle kohlelo lokusebenza ngokusebenzisa ubuchwepheshe bamuva kanye nesitoreji sasendaweni esinamandla.
β€œUkukala kulula futhi kuyashesha ngesitoreji sokufinyelela ngaphambili esisekela izinhlobonhlobo zamadrayivu ukusuka kuma-SATA HDD abiza kancane kuya kuma-SSD ashesha kakhulu angu-2.5”, kanye namadrayivu e-latency aphansi kakhulu njenge-PowerEdge Express Flash NVMe PCIe SSD.
- I-R730xd iyifanele kahle imiklamo yesitoreji esiyingxubevange ngenxa yokusetshenziswa kobuchwepheshe obusha bokulondolozwa kwesikhashana kusilawuli se-PERC, okwandisa ukusebenza kahle kwezindleko zesixazululo se-Ceph.

Isibonelo sokuqaliswa kwengqalasizinda nge-Red Hat Ceph Storage, equkethe amaseva angu-5 Dell R730xd, ungabona ngezansi:

Indlela yokwakha ingqalasizinda corp. ikilasi ngokusetshenziswa kwe-Dell R730xd E5-2650 v4 amaseva abiza u-9000 euros ngepeni?

Ukuhlolwa okuzimele kokucushwa kwepulatifomu ye-Dell R730xd okuhlukahlukene kuveze imiphumela elandelayo:

Indlela yokwakha ingqalasizinda corp. ikilasi ngokusetshenziswa kwe-Dell R730xd E5-2650 v4 amaseva abiza u-9000 euros ngepeni?

Indlela yokwakha ingqalasizinda corp. ikilasi ngokusetshenziswa kwe-Dell R730xd E5-2650 v4 amaseva abiza u-9000 euros ngepeni?

Inguqulo yethu yesikhulumi ibonisa imiphumela emihle kakhulu, futhi esimweni sokufunda, mhlawumbe okungcono kakhulu. Futhi uma sicabangela iqiniso lokuthi si-ode lezi zinkundla ngobuningi futhi ngenxa yalokhu singanikeza intengo izikhathi ezingaphezu kwezingu-2 eziphansi kunezinye izinketho zeplathifomu - kuvela ukuthi iyona engcono kakhulu ehlanganiswe ngokufunda nokubhala ngokwemibandela. yepharamitha eyongayo. Ekugcineni, akukho lutho olukuvimbela ekwakheni iqoqo lamapulatifomu angu-7-8, esikhundleni se-5. Ngesikhathi esifanayo, uthola ukubekezelelana kwamaphutha amakhulu nakakhulu.

Ingabe amaswishi e-10-Gigabit ayadingeka lapho kwakhiwa izingqalasizinda ezinjalo? Cha, akudingekile, elinye lamakhasimende ethu lakhe ingqalasizinda, nakuba lisuka kumaseva angu-3, ​​lisebenzisa lolu hlelo olulandelayo lokuxhumana kwenethiwekhi yendawo:

Indlela yokwakha ingqalasizinda corp. ikilasi ngokusetshenziswa kwe-Dell R730xd E5-2650 v4 amaseva abiza u-9000 euros ngepeni?

Okumvumele ukuthi onge kancane - $350 / ngenyanga ekuqashweni kokushintsha, njengoba amaswishi e-10-Gigabit emba eqolo, futhi sinikeza okungcono kakhulu kuphela - i-Arista 7050tx-48-r. Asiboni iphuzu kuzixazululo zamachweba amaningi.

Yiziphi izinkundla ze-Dell R730xd ezikwazi ukwenza ngokomgomo?

Esikhathini esingeside esidlule, iMicrosoft yenza ukuhlolwa okuthakazelisayo kokusebenza kwalawa mapulatifomu esimweni sokwakha izindawo zokugcina idatha ngenani elincane lama-node.

Sisebenzise amanodi angu-4 e-Dell R730xd, ehlanganiswe kunethiwekhi yendawo ye-100-gigabit sisebenzisa i-32-port ye-Arista DCS-7060CX-32S 100Gb switch, esebenzisa inguqulo ye-EOS 4.15.3FX-7060X.1.

Ukucushwa kwama-node asetshenzisiwe kungokulandelayo:

2x Xeon E5-2660v3 2.6Ghz (10c20t)
256GB DRAM (16x 16GB DDR4 2133 MHz DIMM)
4x Samsung PM1725 3.2TB NVME SSD (PCIe 3.0 x8 AIC)
I-Dell HBA330
4x Intel S3710 800GB SATA SSD
I-12x Seagate 4TB Enterprise Capacity 3.5” SATA HDD
I-2x Mellanox ConnectX-4 100Gb (Imbobo Ekabili 100Gb PCIe 3.0 x16)
I-Mellanox FW v. 12.14.2036/XNUMX/XNUMX
I-Mellanox ConnectX-4 Driver v. 1.35.14894
Idivayisi ye-PSID MT_2150110033
Imbobo eyodwa exhunyiwe / i-adaptha

Kusetshenziswa i-VMFleet, imishini engama-virtual engama-20 yaphakanyiswa endaweni ngayinye, okungukuthi, isamba semishini engama-virtual engama-80. Umshini ngamunye obonakalayo walungiswa nge-1vCPU. I-VMFleet yabe isetshenziselwe ukusebenzisa insiza ye-DISKSPD ukuhlola ukusebenza kwesitoreji endaweni ngayinye yama-80, insiza iyatholakala mahhala. lapha. Amapharamitha wokuhlola ami kanje - uchungechunge olu-1, 512KiB wokufundwa okulandelanayo kokusebenza okungu-4 kwe-I/O.

Indlela yokwakha ingqalasizinda corp. ikilasi ngokusetshenziswa kwe-Dell R730xd E5-2650 v4 amaseva abiza u-9000 euros ngepeni?

Njengoba ubona esithombeni esingenhla, sikwazile ukuthola umphumela omangalisayo wamagigabhayithi angama-60 ngomzuzwana usuwonke, ocishe ulingane nezinguqulo ezi-5 zesiNgisi zeWikipedia ngendlela ecindezelwe (11.5GiB), ezilandwa njalo ngomzuzwana. Futhi ijubane elivela emshinini ngamunye obonakalayo lalicishe libe yi-CD eyodwa ngomzuzwana - 750 MB.

Lokhu kuhlola kubonisa kahle ukuthi isixazululo singasebenza kahle kangakanani uma izingxenye ezintathu zokubala, isitoreji, kanye nenethiwekhi zibhalansiswa, kuncishiswe izithiyo ezingaba khona ezingaba khona ohlelweni olungalinganiseli.

Kuthiwani ngamapulatifomu avela kwabanye abakhiqizi?

Ukuqhathaniswa kwe-Dell R730xd ne-HP ProLiant DL380

Ukuze siqhathanise ukusebenza, sikhethe izingxenyekazi zekhompuyutha ezibuthakathaka kunalezo esizinikeza ukuqashiswa - amaseva anephrosesa ye-E5-2620v3, kune-E5-2650v4, ekhiqiza kakhulu. Indlela yokuhlola eyinhloko esetshenziswe kulokhu kuqhathanisa kwakuwukukala inani le-IOPS. Kusetshenziswe ukulingiswa okuningana okuhlukene komthwalo womsebenzi, okungukuthi konke Funda kanye no-30% Funda / 70% Bhala (okufana nohlelo lwe-OLTP, uhlelo lokuthengiselana, lapho kucutshungulwa ukuthengiselana okuncane, kodwa ukugeleza okukhulu, futhi amaklayenti adinga ukuqinisekisa isikhathi esincane sokuphendula).

Ukuhlolwa kwenziwa izikhathi eziningana ukuze kuqedwe umthelela omubi wezinto ezifihliwe ekusebenzeni kwesistimu. Okokuqala, senze izivivinyo eziyisisekelo ku-Dell R730xd kanye ne-HP ProLiant DL380 sisebenzisa kuphela isitoreji se-HDD (5x1TB HDD RAID5) ukuze sinikeze ibhentshimakhi yokusebenza eyisisekelo. Isethi efanayo yokuhlola yabe iqhutshwa ngesitoreji esingu-5x1TB HDD RAID-5 kusetshenziswa inqolobane ye-DAS efakwe kuma-SAS SSD (2xSAS 480GB Samsung SSD RAID1) kuseva ye-Dell, nakuseva ye-HP esebenzisa i-HPE SmartCache enesethi efanayo yamadrayivu ukuze igcinwe. kanye nokulondoloza isikhashana. Ekugcineni, isethi yokugcina yokuhlola yaqhutshwa kuseva ye-Dell kusetshenziswa i-DAS Cache kumadrayivu e-NVMe PCIe SSD (2x400GB Samsung NVMe ku-software RAID) ukuze kuboniswe umnikelo omkhulu wokugcinwa kwesikhashana ekuzuzweni kokusebenza kohlelo lokusebenza ezimweni ezinzima zokufunda nokubhala. Ukuhlola okufanayo akwenziwanga kuseva ye-HP ngoba i-Smart Cache ayisekeli amadrayivu e-NVMe ukuze afakwe kunqolobane.

Ukucushwa kwamasistimu:

I-Dell PowerEdge R730xd (13th Generation)
I-HP ProLiant DL380 (isizukulwane sesi-9)

Iseva
CPU: x86-64 - Intel Xeon CPU E5-2620 v3 @2.40GHz;
Imemori: 32GB DDR4.
CPU: x86-64 - Intel Xeon CPU E5-2620 v3 @2.40GHz;
Imemori: 32GB DDR4.

Uhlelo olusebenzayo/isofthiwe
Windows 2012 R2 SP1;
I-SanDisk DAS Cache v1.4.
Windows 2012 R2 SP1;
I-HPE SmartCache.

Isitoreji/inqolobane
5x1TB HDD RAID5;
2xSAS 480GB Samsung SSD RAID1;
2x400GB Samsung NVMe Software RAID.
5x1TB HDD RAID5;
2xSAS 480GB I-Samsung SSD RAID1.

Ukuhlolwa
Lingisa umsebenzi wokufunda nokubhala we-OLTP;
300GB usayizi wesizindalwazi;
Izisebenzi ezi-4 ngesikhathi esisodwa ze-IOMETER ezinomugqa wokushona okungama-32.

Lingisa umsebenzi wokufunda nokubhala we-OLTP;
300GB usayizi wesizindalwazi;
Izisebenzi ezi-4 ngesikhathi esisodwa ze-IOMETER ezinomugqa wokushona okungama-32.

Ukuhlolwa okuningana okungahleliwe kokufunda/ukubhala kwenziwa ngosayizi abahlukene bebhulokhi - 4 kanye no-8 KB. Ngosayizi ngamunye webhulokhi, futhi, ukuhlolwa kwenziwa ekusebenzeni kokufunda okungu-100%, kanye naku-70% ukufunda imisebenzi kanye nokusebenza kokubhala okungu-30%. Ukuhlolwa obekuhlobene nokusheshisa ukusebenza kusetshenziswa ukugcinwa kwesikhashana kwaqhutshwa amasekhondi angu-900 (amaminithi angu-15) ukuze kunikezwe isikhathi esanele sokuthi inqolobane isebenze ngokugcwele futhi ikhiqize imiphumela enembe kakhulu.

Isilinganiso sokusebenza ku-IOPS (imisebenzi yokokufaka/yokukhiphayo ngomzuzwana):

Iseva/ukulungiselelwa ngaphansi kokuhlolwa
4KB RR
Kufundiwe 100%.
4KB RR
70% Funda 30% Bhala
8KB RR
Kufundiwe 100%.
8KB RR
70% Funda 30% Bhala

I-Dell R730xD (ayikho inqolobane)
1650
974
1540
1040

I-HP ProLiant DL380 (ayikho inqolobane)
1370
628
1322
630

I-Dell R730xD
nge-DAS Cache
138884
66483
98368
56641

I-HP ProLiant DL380
nge-SmartCache
41273
33534
35984
39396

I-Dell R730xD nge
I-DAS Cache kanye
I-NVMe PCIe SSD
264750
158157
257150
104490

Isilinganiso sokusebenza ku-% ukusetshenziswa kweprosesa:

Iseva/ukulungiselelwa ngaphansi kokuhlolwa
4KB RR
Kufundiwe 100%.
4KB RR
70% Funda 30% Bhala
8KB RR
Kufundiwe 100%.
8KB RR
70% Funda 30% Bhala

I-Dell R730xD (ayikho inqolobane)
0,4%
0,28%
0,42%
0,3%

I-HP ProLiant DL380 (ayikho inqolobane)
0,8%
0,5%
0,8%
0,5%

I-Dell R730xD
nge-DAS Cache
13%
8,8%
11,34%
7,83%

I-HP ProLiant DL380
nge-SmartCache
6%
6%
5%
5%

I-Dell R730xD nge
I-DAS Cache kanye
I-NVMe PCIe SSD
16%
10,1%
16%
5,78%

Imiphumela yokuhlolwa ibonisa izinzuzo eziphawulekayo zokusebenza kwezinkundla ze-Dell R730xd lapho kusetshenziswa ukugcinwa kwenqolobane ye-SanDisk DAS uma kuqhathaniswa ne-HPE SmartCache, nokusekelwa kwe-NVMe PCIe SSD kunikeza izinzuzo ezinkulu nakakhulu. Ezilinganisweni ze-OLTP, i-R730xd ibonise i-IOPS ephindwe izikhathi ezi-3 ngenxa yokusetshenziswa kwe-SanDisk DAS Cache nama-SAS SSD futhi izikhathi ezingu-6 ngaphezulu ngenxa yokusetshenziswa kwe-SanDisk DAS Cache + NVMe PCIe SSDs uma kuqhathaniswa ne-HPE's SmartCache. Ukusetshenziswa okuphezulu kwe-CPU nge-SanDisk DAS Cache, kanye nokusetshenziswa kwe-CPU ephakeme nakakhulu nge-SanDisk DAS Cache + NVMe PCIe SSDs, kumane kuwumphumela wokubala okuphezulu kakhulu kwe-IOPS. futhi, ngokuphambene nalokho, kukhombisa ukusetshenziswa okungcono kakhulu kwezinsiza zokucubungula.

Ngakho-ke, uma ku-HP ngokusetshenziswa okungu-6% umphumela we-IOPS uyizinkulungwane ezingama-41, okungukuthi, i-1% yezinsiza zokucubungula isetshenziswa nge-IOPS eyinkulungwane eyi-0,15, ngakho-ke esimweni se-Dell R730xd sinokusetshenziswa okungu-16% ku-264 ayizinkulungwane, okuyinto inikeza inkomba esebenza kahle kakhulu - 0,06% wezinsiza zokucubungula ngo-1000 IOPS ngayinye.

Okusho ukuthi, ngokuya ngokusetshenziswa kweprosesa, iDell ibuye ibe ngcono (izikhathi ezingama-2,5), yize inamanani wokusetshenziswa kweprosesa aphezulu ngo-%, kodwa, njengoba sikwazile ukuqinisekisa, lokhu kungumphumela wokucubungula. inani elikhulu kakhulu lemisebenzi, kanjalo nokukhiqiza okungcono.

Ngakho-ke, isiteji se-Dell R730xd sibonakala sisebenza kahle kakhulu kune-HP ProLiant DL380 (izikhathi eziningi), kokubili ngokusebenza nokusebenza kahle kwezindleko.

Imiphumela yokuhlolwa kwe-Geekbanch 3 i-Dell R730xd ene-E5-2640 v4 processor noma kungani imiyalelo yokucubungula kanye nenqolobane yesilawuli kubalulekile

Njengoba sibona ngenhla, singathola imiphumela ephezulu kuphela uma izingxenye ezintathu - amandla ekhompiyutha, isitoreji kanye nenethiwekhi - zilinganiselwe, njengoba kuncishiswa izithiyo ezingaba khona ohlelweni olungalinganiseli.

Kepha futhi okubaluleke kakhulu ukwesekwa komprosesa wesikhulumi semiyalo eminingi nokusebenza kwayo kahle. Sengike ngasho ngenhla ukuthi umehluko obonakala ungelutho ezizukulwaneni ungadlala indima ebalulekile kwezinye izimo. Ngakho-ke, ngokwemiphumela yokuhlolwa okuzimele, i-E5-2650 v4 esiyinikezayo isebenza kahle ngo-70% esimweni sokubethela (imiyalo ye-AES) kune-E5-2650 v3.

Kuthiwani-ke ngezixazululo "ezihlwabusayo" ezivela kubancintisana nezindleko eziphansi, kodwa hhayi ingxenye ye-premium? Kungani izixazululo zethu zingcono? Kungani izisombululo zethu ziyi-premium? Impendulo ilula - amaseva ethu ayekhona futhi abhalansile. Ngisho noma ngaphambilini bezinamafrikhwensi aphansi noma amanani amancane enkumbulo, bezihlale zibhalansiswa ngokwamapharamitha amathathu achazwe ngenhla. Kubandakanya isikhungo sedatha esithembekile esinezitifiketi ezidingwa ingxenye yebhizinisi kanye neziteshi ezinhle kakhulu zokuxhumana zangaphandle ezinikeza ukubambezeleka okuncane okuvela e-Netherlands, kokubili kulo lonke elaseYurophu, naseRussia nase-Ukraine, ngisho ukubambezeleka okuphansi kakhulu olwandle phansi e-USA!

Kodwa manje senze isipho sokuncintisana ngokwemibandela yalezi zinkomba, ezindaweni ezingu-10 ngesikhathi esisodwa, eyodwa eNetherlands nezindawo ezingu-9 e-USA, kodwa masingakhulumi ngathi, ake sibheke ukuhlolwa. Ngeshwa, lokhu kuhlolwa okwamanje kutholakala kuphela kunguqulo ye-E5-2640 v4, ene-10, hhayi ama-cores angu-12, yesikhulumi sethu.

https://browser.primatelabs.com/v4/cpu/768278 β€” Ρ€Π΅Π·ΡƒΠ»ΡŒΡ‚Π°Ρ‚Ρ‹ тСста Geekbanch 3 ΠΏΠ»Π°Ρ‚Ρ„ΠΎΡ€ΠΌΡ‹ Dell R730xd c процСссором Π•5-2640 v4

Yiqiniso, akuyona inhloso kakhulu, njengoba ingacabangi zonke izinzuzo zesikhulumi, i-cache efanayo ye-SanDisk DAS esikhulume ngayo, izinkomba zokusebenza zangempela zingaba ngcono kakhulu kunezokwenziwa endaweni ethile. icala, kakhulu! Kodwa kusengenzeka ukuthola ukuqonda okuthile.

Ake sinake iziyalezo eziningi. Okubonisa kakhulu kungase kube i-AES eshiwo kakade, cishe kumafrikhwensi afanayo umehluko phakathi kwamanani we-1st core ungaba izikhathi ezingu-1000 kuye ngokuthi iphrosesa, imiyalelo esele eyaqhathaniswa yabonisa umehluko wamanani kusuka izikhathi eziningana kuya kweziyi-100.

Ungakwazi ngokwakho faka iphrosesa ekusesheni futhi wenze isiqhathaniso, kodwa ungakhohlwa ukuthi lokhu kuhlolwa akubonisi ngokuphelele, ngoba akunaki i-hardware iyonke, kodwa ikakhulukazi kubonisa iphrosesa.

Noma kunjalo, ngithemba ukuthi le miphumela ingaba usizo kuwe lapho uhlaziya ukusebenza kwesixazululo esithile.

Ukusebenzisa amaseva e-Dell R730xd njenge-Bare Metal, kungcono nje!

Amanye amakhasimende ethu anqume ukuyeka ukusetshenziswa kwezixazululo zefu ze-VMware ukuze avune izixazululo ze-Bare Metal (amaseva ensimbi) futhi inkundla ye-Dell R730xd yabasiza kakhulu kulokhu. Futhi lapha asikhulumi kuphela ngamafu omphakathi, kodwa futhi nangasese.

Kungani abantu benza izinqumo ezinjalo ngokwandayo? Mayelana namafu omphakathi, yonke into isobala kakhulu. Mina ngokwami ​​ngibheka amafu omphakathi njengokumaketha okukhulu, uma okuthile kuwa lapho (futhi wonke amafu awela ngokushesha noma kamuva), awela lapho isikhathi eside. Isibonelo i-Amazon efanayo eyaziwayo, eyake yalala phansi izinsuku ezimbalwa kanye nenkonzo ebalulekile yokuqoqa idatha yezokwelapha nge-telemetric, okuhlanganisa nedatha ngesimo senhliziyo seziguli ezisebenzisa i-pacemaker, ezaziqashwe odokotela, abaningana. abantu bafa ... Futhi maduzane, lapho i-Bitrix yethu yenkampani ilele phansi, kwavela ukuthi isizathu sasiwukuthi abasebenzi, ababecisha eminye yemishini, ngenxa ye-typo, bavala ama-node amaningi kunalokho okudingekayo, okuthinta ama-node okuphatha, ngenxa yalokho imishini ebonakalayo yaqala kabusha phakathi namahora angu-5 ngenxa yobukhulu beqoqo elikhulu ... Kodwa lokhu akuyona inkinga engenzeka kuphela; kungase kwenzeke ukuthi uthole "ibhodlela" ngokuzumayo ngenxa yokusetshenziswa. yefu ngamanye amaphrojekthi noma ukuntula amandla okuphatha ngokushesha ingqalasizinda yayo ukuze ihlangabezane nezidingo zakho.

Uma kukhulunywa ngamafu ayimfihlo, ngokombono, i-virtualization yenza kube lula ukutholakala kanye nokubuyiselwa kwezinhlekelele. Empeleni, kunezimo eziningi lapho lokhu kungenangqondo:

Uma kunezinkinga zobuchwepheshe ku-node, isikhathi sokuphumula samaminithi angu-15-30 siyamukeleka.
Akuwona wonke amaphrojekthi adinga isikhathi esicishe sibe ngu-100%, ngaphandle kwezingane zesikole ezijwayele ukukhokha idola futhi ezifuna u-100%. Kwamanye amaphrojekthi, isikhathi esiqinisekisiwe esingu-99,9% singaphezu kwenketho eyamukelekayo. Ngoba i-0,1% ukungatholakali kwemizuzu engama-44 ngenyanga, okungenzeka ngenxa yezizathu ezihlukahlukene ezingahleliwe - ngenxa yokuhluleka kwe-hardware noma ukungatholakali kwenethiwekhi. Siqinisekisa isikhathi sokuphumula senethiwekhi ku-99,99% ngezixazululo ezijwayelekile, okuvumela ukungatholakali kwemizuzu emi-4 kuphela ngenyanga. Uma iseva ixhunywe kumashaneli angu-2 azimele adlula ezindaweni ezihlukene zokuxhumana futhi ithrafikhi yawo icutshungulwa ngama-cores azimele, isikhathi sokuphumula senethiwekhi esiqinisekisiwe sikhuphuka izikhathi eziningana ngaphezulu. Lokhu kwenza kube nokwenzeka ukwanelisa izidingo zesikhathi esengeziwe zamabhange ngisho nokungatholakali kwawo cishe imizuzu engama-40 ngonyaka okwamukelekayo, ngokuhambisana nesitifiketi. Eqinisweni, izikhathi zokungatholakali ziyivelakancane kakhulu. Eminyakeni engu-5 yokusebenza nesikhungo sedatha, akukaze kube khona isikhathi lapho wonke amaseva engatholakali ngesikhathi esifanayo ngenxa yezinkinga zenethiwekhi noma ugesi. Ngisho nalapho kuba nokucinywa komhlaba wonke eNetherlands futhi ezinye izikhungo zedatha eziningi zazingatholakali - ezinye zakhohlwa ukufaka amafutha ezinjinini zikadizili, ezinye zazingenawo amandla anele e-UPS, esikhungweni sethu sedatha angaphansi kwephesenti kuphela lamaseva awazange atholakale isikhashana. isikhathi. Amanye amaseva aqashwa ngamakhasimende kithi, ngisho noma esephelelwe yisikhathi, i-hardware ebonakala isiphelelwe yisikhathi, okungenzeka ukuthi yehluleke kunezixazululo ezintsha ezinophawu, awakaze aqalwe kabusha ngisho nakanye eminyakeni emi-3, njengoba nje eminyakeni emi-3 engakaze alahleke inetha. . Ingabe ukungatholakali kwemizuzu engama-30 kwamukelekile iminyaka emi-3? Ngaphezu kwalokho, ngisho namabhange.

Futhi kuphakama umbuzo onengqondo, kungani-ke ukhokha ngokweqile? Kuyacaca ukuthi akufanele uthembele entuthukweni futhi kufanele uhlale usuka kumgomo wokubhukha okuphusile. Umthetho oyinhloko engiwutholile yilo olandelayo: ukubhukha akufanele kubize ngaphezu kokulahlekelwa okungenzeka uma kwenzeka isikhathi sokungatholakali ngenxa yokuntuleka kwalokho kubhuka. Okusho ukuthi, uma isikhathi sokuphumula esiqinisekisiwe sivumela ukungatholakali kwemizuzu engama-40 ngenyanga, futhi uma kwenzeka ukungatholakali phakathi nesikhathi esinjalo, ukulahlekelwa kuzoba ngamakhulu ambalwa noma inkulungwane yamadola - ukuqasha isisombululo esengeziwe sokwandisa isiqinisekiso sokuphumula kalula. akunangqondo. Ngoba empeleni, amaseva ngeke atholakale njalo ngenyanga imizuzu engama-40; imizuzu engama-40 yokungatholakali isiqinisekiso, isimo esibi kakhulu. Okungenziwa nyanga zonke, njengoba umkhuba ubonisile.

Ngale ndlela, amaseva angabaluleka ngaphandle kokubaluleka komgomo, futhi lapho abanikazi bephrojekthi bengenandaba nokushintsha umthwalo kwenye iseva ezinikezele, singagwema ubunzima bokwakha iqoqo le-faillover kanye nesitoreji esabiwe se-failover.

Ukukhiqiza okuphansi akwamukelekile ngesikhathi sokusebenza. Amakhasimende ethu amaningi akhetha izixazululo zensimbi ezingenalutho kuDell ngenxa yalesi sizathu; ishibhile futhi kunenzuzo enkulu ukusebenzisa i-hardware ephelele ngesikhathi sokusebenza okujwayelekile, okungekho ngaphansi kokungafinyeleleki, njengoba sesixoxile kakade, kunokukhokha ngokweqile ifu. inketho kanye nokusebenza okufanayo efwini, okuzobiza izikhathi eziningana kunesixazululo esizinikezele. Ifu nokuhlanganisa akukwazi nje ukwanelisa izidingo zensiza futhi ngeke kubize kakhulu kumakhasimende anjalo.

Ababhalisile abafuni ukukhokha imali eningi ngesevisi enamandla alinganiselwe. Ikakhulu amakhasimende ethu yizinkampani ezincane ezingenayo iminyango yazo yezobuchwepheshe kanye nezimali zokuthenga i-hardware yazo. Ukuthenga i-hardware kungaba igalelo langempela kubo futhi kunganikezeli ukusebenza okwanele, njengoba ngokuhamba kwesikhathi i-hardware ivame ukuphelelwa yisikhathi, futhi kuyihlazo ukulahla i-hardware ebizayo ngisho nangemva kweminyaka emi-3, ngesikhathi lapho inkampani isikhulile futhi kukhona okungcono kakhulu. okudingekayo. Uma uqasha i-Dell R730xd kithi, unethuba lokushintshela kwenye iseva nganoma yisiphi isikhathi noma ngemva konyaka uma ukhetha inketho yokuqasha ngenkontileka yonyaka. Ngaphezu kwalokho, kunethuba lokonga ekukhokheleni amalayisense e-VMware.

Ukusebenzisa amaseva e-Dell R730xd kumaseva eDatabase

Iningi lamakhasimende ethu asebenzisa lezi ziphakeli zedathabheyisi seliyithanda le nkundla. Futhi hhayi kuphela ngoba, ngokungafani ne-Dell R730, inguqulo ye-xd ingahlinzekwa ngokusekelwa kwamadrayivu e-NVMe PCIe SSD, ahlinzeka ngokubambezeleka okuncane, okubaluleke kakhulu kuzixazululo ezinjalo. Kodwa futhi ngoba isilawuli sisebenza kahle ngendlela emangalisayo uma sisebenza ngisho nama-SSD avamile angu-2,5" esiwanikeza amakhasimende.

Vele, kukhona ne-drawback ku-Dell R730xd - kukhona isilawuli esisodwa kuphela, kuyilapho i-Dell R1 enamadrayivu angama-730 isebenzisa izilawuli ezi-26 zehadiwe. Kodwa lokhu akuyona inkinga ephawulekayo, njengoba sikhethe isiteji esinama-drive bays angu-2 ngakho-ke ngeke uthole ibhodlela ngendlela yokusebenza kwesilawuli esinganele lapho usebenza nale nkundla kanye nenani elikhulu lama-SSD. Futhi inzuzo ngendlela yekhono lokufaka i-NVMe PCIe SSD ihlanganisa ngokuphelele le nkinga.

Njengoba kushiwo ngenhla, akuwona wonke umuntu odinga amaqoqo edatha eyinkimbinkimbi; ukusebenza okungahlinzekwa yile nkundla kubaluleke kakhulu. Elinye lamakhasimende ethu lenze lokho kanye, lashiya ukweqisa kokwenziwa kwezinto ezibonakalayo kanye nokwakha amaqoqo ayinkimbinkimbi, laqasha i-VPS enamadrayivu anikezelwe amakhophi edatha egciniwe avela kuseva enkulu ye-Dell R730xd: I-VPS (KVM) – E5-2650 v4 (24 Cores) / 40GB DDR4 / 4x240GB RAID10 SSD 1Gbps 40TB – $99. Yiqiniso, uma kwenzeka kunenkinga nge-node eyinhloko, kuzoba nokwehla okuthile ekusebenzeni futhi kuzoba nesidingo sokushintsha ngesandla, noma kunjalo, lokhu kuyisixazululo esingaphezu kwezindleko, kunikezwe isabelomali esilinganiselwe.

Asikhuthazi wonke umuntu ukuthi akonge ngale ndlela; kwezinye izimo kufaneleka kakhulu. Kodwa-ke, ngokwandisa isabelomali ngo-$230 kuphela ngenyanga, ungakwazi ukukhokhela i-node efanayo ye-E5-2650 v4 futhi uyixhume ndawonye kunethiwekhi yendawo ye-20-Gigabit, okuzokuvumela ukuthi wakhe iqoqo elincane eline-latency eyamukelekayo. ngisho nangaphandle kokusebenzisa amadrayivu e-PCIe NVMe SSD.

Isixazululo ngokwaso sizobukeka kanje (ngesethi eyisisekelo yamadrayivu):

I-Dell R730xd 2 x Intel Dodeca-Core Xeon E5-2650 v4 128GB DDR4 6x480GB SSD 1Gbps 100 TV + Intel X540-T2 20GBPS LAN – $289/ngenyanga
I-Dell R730xd 2 x Intel Dodeca-Core Xeon E5-2650 v4 128GB DDR4 6x480GB SSD 1Gbps 100 TV + Intel X540-T2 20GBPS LAN – $289/ngenyanga

Ukwandisa isibalo sabo ukuze kwandiswe ukukhiqiza akwenzi mqondo ngaso sonke isikhathi. Ngakho-ke, ngesikhathi sokuhlolwa ngemininingwane yolwazi, sithole ukuthi isixazululo esinama-SSD ayi-8 sisebenza kahle ngo-4% kunesixazululo esinama-SSD ama-35.

Ngokuqondene namazinga e-RAID anconyiwe, i-RAID5 ingase ibe yehle kakhulu kwezinye izimo. Impela, i-RAID10 ihlinzeka ngokusebenza okungcono, kepha kuthiwani uma amadrayivu eshesha kangangokuba ukuwasebenzisa ku-RAID10 kungabangela isilawuli ukuthi sibe yibhodlela? Ezimweni ezinjalo, ukwakha uhlu lwe-HW RAID RAID5 kunenzuzo enkulu usebenzisa amadrayivu ashisayo-ayisipele kune-RAID10 (uma inhloso kungekona ukulahlekelwa ingxenye yesabelo se-SSD).

Ingabe ngidinga inqolobane yesilawuli ukuthi inikwe amandla noma kungcono ukuyikhubaza? Isilawuli se-RAID sinenani elilinganiselwe lememori, iyini i-1GB uma usebenza ne-SSD? Ezimweni ezinama-SSD asheshayo, akukho phuzu ekwenzeni inqolobane yokufunda inikwe amandla, ngoba ngokugcwalisa inkumbulo ngemisebenzi efundwayo, engeke isheshe, njengoba ama-SSD asevele eshesha ngokwanele, asishiyi inqolobane yokubhala eyanele, futhi , njengoba kwaziwa, kuma-SSD angabizi - i-bottleneck yimisebenzi yokubhala, ngenxa yokwehla kokusebenza ngenxa yokunganele kokunikezwa okungaphezu kokwanele kanye no-100% wokusetshenziswa kwesabelo samahhala (ukwehla okuphindwe kashumi kungenzeka). Ngakho-ke, sincoma ukushiya inqolobane yokubhala kuphela enikwe amandla. Ngaleyo ndlela kuqinisekisa ukusebenza kahle okukhulu kwalesi sixazululo.

Ingabe usayizi we-block ye-NTFS unendaba ngempela? Ezinqolobaneni eziningi, i-64KB ikhonjiswe njengosayizi webhulokhi ofanele kakhulu wokusebenza okuphezulu. Kodwa ngokuvamile sihlangabezana nezimo lapho iklayenti selivele lisebenzisa iseva egciniwe, lisebenzisa ukumisa kokuqala ngosayizi ojwayelekile weqoqo le-NTFS elingu-4KB. Ukuze ulungise lokhu udinga ukufometha kabusha amadrayivu, kodwa ingabe umehluko mkhulu kangako esimweni esivamile? Ingabe udinga ukuzichaya esikhathini sokuphumula okungenzeka kanye nokuphazamiseka okuhlobene nokufuduka?

Isilawuli kuseva ye-Dell R730xd sinikeza ukusebenza okuhle okunjalo ngisho namadrayivu angu-4 we-SSD kanye nenqolobane yokubhala inikwe amandla (iphuzu elibaluleke kakhulu, i-cache efundwayo kufanele ikhutshazwe ngoba ayinikezi noma yiziphi izinzuzo, kodwa kunciphisa kuphela ukusebenza kahle kokubhala imisebenzi efanele. ekuntulekeni kwesikhala senqolobane egcwele, njengoba kukhonjisiwe ngenhla), ukuthi ngisho nosayizi webhulokhi ojwayelekile ongu-4KB, sithola izinkomba zokusebenza ezinhle kakhulu, ezinhle kakhulu.

Noma kunjalo, sihlala sizimisele ukuhlangana futhi silungele ukunikeza inkundla efanayo isikhathi esithile ukuze ababhalisile benze umsebenzi uma okuthile bengakunakanga.

I-RAMdrive isabonakala ishesha kunamadrayivu e-SSD. Uma imibuzo yakho ingacutshungulwa ku-TempDB, njengokuhlunga noma ukuhlanganisa, bese usebenzisa i-RAMdisk (uhlelo olushintsha ingxenye ye-RAM yakho ibe isitoreji) kungaba wusizo kakhulu. Ngakho-ke, uma siqhathanisa ijubane lamadrayivu angu-8 we-SSD ku-RAID10 nge-RAMdrive, sithole ukuthi lishesha cishe izikhathi ezi-4 ekusebenzeni kokubhala kwe-4K ngokujula komugqa okungama-32, okulingisa ncamashi ukusebenza kwamafayela e-TempDB. Ngisho nokuhlanganisa ama-SSD angu-8 ku-RAID0 kwakungenakwenzeka ukushaya imiphumela ye-RAMdrive.

I-RAID5 esimweni samadrayivu angu-8 we-SSD ingasebenza kangcono kune-RAID10. Ukufundwa kwe-4K nokujula komugqa okungu-32 cishe kushesha ngo-40%, okungamangazi uma kubhekwa izici ze-RAID5. I-RAID5 iwina ekusebenzeni kwezinye izivivinyo, isibonelo, ngokubhala okulandelanayo ingu-20%, iphansi kuphela emisebenzini yokubhala engu-4K ngokujula komugqa omkhulu (kithi 32) futhi kuvela ingasebenzi kahle ngo-30%. Ngakho-ke ukusetshenziswa kwe-RAID5 kwezinye izimo kungathethelelwa ngokwezomnotho.

Ukulondoloza okwengeziwe akusho ukuthi ngokushesha. Uma sibhekene nemisebenzi emincane engahleliwe, ukwengeza amadrayivu kumalungu afanayo angaphezu komkhawulo othile akuthuthukisi ukusebenza nhlobo. Kungabonakala kungani? Kuvela ukuthi konke kuncike emandleni omlawuli, ophoqeleka ukuthi abhale idatha kuqoqo lonke lokushayela. Kungakho sithenge amapulatifomu anenani eliphakeme lamadrayivu ayi-12, ukuze sinciphise amathuba okuba nebhodlela. Kodwa-ke, udinga ukubheka ukuqina kokusebenza ngaphakathi kwephrojekthi ngayinye, kwesinye isikhathi amadrayivu ama-4 amakhulu, yithi i-960GB, angaba yisixazululo esingcono nesingambi kakhulu kunenqwaba yamadrayivu angama-480GB.

Ukusebenzisa amaseva e-Dell R730xd kumaseva edatha we-Aerospike NoSQL

I-Aerospike iwumthombo ovulekile wedatha ye-NoSQL ene-flash elungiselelwe isivinini nokuqina. I-Aerospike iklanyelwe ukunikeza ukubambezeleka okuphansi kakhulu kwezicelo zokufunda kanye nemithwalo esindayo yokubhala, enelisa izidingo zebhizinisi zesimanje ezidinga ikhono lokusingatha izinombolo ezinkulu zemisebenzi ye-I/O. I-Aerospike ilawula ngokuqondile isitoreji sendawo kuma-cluster node ayo, kungakhathaliseki ukuthi yiziphi izinhlelo zefayela zomthombo wemidiya, kungaba i-DRAM, i-flash noma amadrayivu aqinile endabuko. Lokhu kuvumela idatha ebalulekile ukuthi igcinwe kumidiya eshesha kakhulu lapho ukucubungula okusheshayo kubaluleke kakhulu, kuyilapho imvelo esabalalisiwe ye-database ye-Aerospike iqinisekisa izinga eliphezulu lokuthembeka uma kwenzeka ukuhluleka kwe-cluster node.

Amaseva kaDell wesizukulwane se-13 se-PowerEdge R730xd aletha amandla amangalisayo ekhompiyutha ngamaphrosesa akamuva e-E5-2650 v4 kanye ne-DDR4 SDRAM esheshayo. Futhi amandla okusebenzisa ama-NVMe SSD, afana ne-Samsung SM1715, angasiza ekutholeni ukusebenza okuhle kakhulu nge-latency encane kuma-node anemithwalo ephezulu. Lokhu kwenza kube nokwenzeka ukuhambisana nezidingo ze-SLA zesimanje zesikhathi sokucubungula sokuthengiselana okukodwa ngezinga elithembekile lokuvikelwa ekukhwabaniseni ngisho nasezingomeni zolwazi ezinkulu kakhulu, okwenza isisombululo samukeleke ngisho nakumkhakha wezezimali kanye nokudluliselwa kwebhange, lapho ukuthengiselana ngezinye izikhathi kwenziwa khona. phakathi kwamahora ambalwa noma ngisho nezinsuku.

Ngomkhakha wamabhange, singanikeza isixazululo esisekelwe kulawa mapulatifomu anenethiwekhi yendawo engu-40 kanye ne-100 Gbit / s phakathi kwama-cluster node kanye nokusetshenziswa kwamaswishi afanelekile.

Ukusebenzisa amaseva e-Dell R730xd kumanethiwekhi e-DMZ

Siyajabula ukunikeza ababhalisile bethu izindinganiso eziphakeme kakhulu zokuphepha. Izikhungo zedatha lapho sinikeza khona lawa maseva zinezitifiketi eziningi: ISO 27001, I-PCI DSS, I-SOC 1, I-HIPAA ΠΈ I-NEN7510.

Kodwa ukuze kuhlangatshezwane nezidingo ze-Google efanayo, amanye amaklayenti adinga ukwakhiwa okuthiwa amanethiwekhi asuswe impi (amanethiwekhi e-DMZ) - amanethiwekhi anokuphepha okukhulisiwe, okusho ukuba khona kwe-firewall yehadiwe ekhawulela ukufinyelela kumishini futhi ikuvumela ukuthi fihla izingxenye ezidingekayo ngemuva kwe-firewall futhi uthuthukise ukuphepha okwengeziwe.

Futhi konke kulungile lapho isixazululo esinjalo siyalwa ngokushesha, kodwa kunzima kakhulu ukuqalisa inguquko ngokungabi nasikhathi sokuphumula kule nqubo, esanda kwenzeka kwelinye lamakhasimende ethu, ngephrojekthi yokucubungula izicelo ze-visa zezakhamizi ezivela emazweni ahlukene. zomhlaba, lapho kungenakufinyeleleka khona ngisho naphakathi namahora kungaholela ekulahlekelweni amashumi ezinkulungwane zamaRandi. Kodwa sathola isixazululo, saya emhlanganweni, sabeka imishini eyengeziwe ekamelweni, sihambisa izinto ezingabalulekile kangako, futhi savula i-firewall ngokuvumelana nesimiso, njengoba iklayenti lalifisa, ukuze likwazi ukufeza ukufuduka ngaphandle kokuphumula.

Kwaba / Kwaba

Indlela yokwakha ingqalasizinda corp. ikilasi ngokusetshenziswa kwe-Dell R730xd E5-2650 v4 amaseva abiza u-9000 euros ngepeni?Indlela yokwakha ingqalasizinda corp. ikilasi ngokusetshenziswa kwe-Dell R730xd E5-2650 v4 amaseva abiza u-9000 euros ngepeni?

Mhlawumbe lolu hlelo luzophinde lube usizo kumuntu lapho enza ukufuduka okunjalo, njengoba onjiniyela bezikhungo zedatha abafuni njalo ukubona izinto ezisobala, kodwa bagcizelela ukuthi isikhathi sokuphumula samahora angu-2 ukuze ushintshe ngemuva kwe-firewall, kodwa ngiyakucela ngomusa - oda i-firewall yehadiwe konke kusengaphambili uma uyidinga. Siyavumelana nezimo futhi sihlala sizimisele ukwamukela, kodwa kwezinye izimo amandla ethu angase alinganiselwe.

Ezincwadini ezizayo, sizozama ukwenza ezinye izivivinyo ezengeziwe futhi sabelane ngolwazi olungeziwe sisebenzisa lezi nkundla ezizothuthukisa ukuqonda kokukhethwa kwesixazululo esisebenza kahle kakhulu. Phela, kuyamangaza ukubona iqiniso lokuthi i-RAID0 yamadrayivu amabili e-SSD, ezimeni zolwazi ezinomthwalo omkhulu, ingase ihambe kancane kune-drive drive ehlukile, yingakho sithola izicelo ze-NVMe PCIe SSD kusukela ngesikhathi. kusikhathi. Kungenzeka ukuthi sizobonisa izimo ezisebenza kahle kakhulu zokuqalisa izixazululo, sabelane ngolwazi olwengeziwe lapho sakha amalungu afanayo amaningi e-RAID ngaphakathi kwesilawuli esisodwa se-RAID sisebenza kahle, futhi sibonise ukuthi yiziphi izimo izinhlobo ezimbili ezimele ze-RAID10 zamadrayivu angu-4 zizoba ngcono kunohlu olulodwa lwe-RAID10 lwe 8. Kuzosebenza nini ukusebenzisa i-RAID1 ku-TempDB, kanye ne-RAID5 kukho konke okunye. Futhi ungenza kanjani ngaphandle kwe-NVMe PCIe SSD uma unesabelomali esilinganiselwe.

I-Dell R730xd: ibhonasi yabafundi be-Habrahabr

Sifuna ukukunikeza okungcono kakhulu futhi senze konke ngalokhu ngokukunikeza amanani ahlukile we-Dell R730xd:

ua-hosting.company/serversnl - eNetherlands
ua-hosting.company/serversus - nase-USA

Ukucushwa kungahluka kakhulu, isibonelo, Okulandelayo kuyatholakala ukuze kusebenze ngokushesha e-Netherlands::

I-Dell R730xd 2 x Intel Dodeca-Core Xeon E5-2650 v4 128GB DDR4 6Γ—480 SSD 1Gbps 100 TV -*$249
I-Dell R730xd 2 x Intel Dodeca-Core Xeon E5-2650 v4 128GB DDR4 12Γ—240 SSD 1Gbps 100 TV -*$249
I-Dell R730xd 2 x Intel Dodeca-Core Xeon E5-2650 v4 128GB DDR4 4x4TB 4x480 SSD 1Gbps 100 TV β€” *$249
I-Dell R730xd 2 x Intel Dodeca-Core Xeon E5-2650 v4 128GB DDR4 12x4TB SATA 1Gbps 100 TV β€” *$249

Futhi i-1Gbps ezinikele ethembekile engalinganisiwe (ngaphandle kwethrafikhi) iyatholakala kubo ngo-+$120/ngenyanga. Kanye nethuba lokuthenga i-firewall, amakhadi enethiwekhi, ukushintsha nokwakha inethiwekhi yendawo yamagigabhithi amaningi. Kodwa isimo senani sisebenza kwinkontileka yonyaka.

Sifuna ukuthuthukisa isimo futhi sinikeze ibhonasi yezinkokhelo zesikhashana esifushane kanye nokuqashisa ngaphandle kwenkontileka, okwenza lokho okunikezwayo kufinyeleleke kakhudlwana. Noma ubani o-oda futhi akhokhele noma yiziphi izinketho ezihlongozwayo e-Netherlands isikhathi esiyinyanga engu-1, sizohlinzeka iseva efanayo ncamashi inyanga engu-1 mahhala, noma sizobala kabusha ukucushwa oku-odiwe futhi sibale iseva e-odiwe ngentengo efanayo neyenkontileka yonyaka ($249 / ngenyanga esikhundleni sika-$369 / ngenyanga), sibuyisela umehluko kubhalansi. Ukuze wenze lokhu, vele ushiye inombolo yakho ye-oda kumazwana. Kulabo abangakavuthwa ekuxazululeni izixazululo ezizinikezele, sinikeza ukuqasha noma iyiphi i-VPS (KVM) enamadrayivu azinikele e-Netherlands noma e-USA kulawa mapulatifomu, isibonelo, VPS (KVM) – E5-2650 v4 (6 Cores) 10GB DDR4 240GB SSD 1Gbps 10TB – $29, futhi uthole ibhonasi yezinyanga ezingu-1-4 zokusetshenziswa lapho ukhokhela izinyanga ezingu-1, 3, 6, 12, ngokulandelana, okubonisa inombolo ye-oda emazwaneni. Manje ukwakha ingqalasizinda sekulula!

Source: www.habr.com

Engeza amazwana