Ukuhlolwa kokuphahlazeka kwesistimu yesitoreji ye-AERODISK ENGINE N2, ukuhlolwa kwamandla

Ukuhlolwa kokuphahlazeka kwesistimu yesitoreji ye-AERODISK ENGINE N2, ukuhlolwa kwamandla

Sanibonani nonke! Ngalesi sihloko, i-AERODISK ivula ibhulogi ku-Habré. Hurray, bakwethu!

Izindatshana ezedlule ezimayelana noHabré zaxoxa ngemibuzo mayelana nezakhiwo kanye nokucushwa okuyisisekelo kwezinhlelo zokugcina. Kulesi sihloko sizocabangela umbuzo ongakaze uhlanganiswe ngaphambili, kodwa uvame ukubuzwa - mayelana nokubekezelelana kwephutha kwezinhlelo zokugcina ze-AERODISK ENGINE. Ithimba lethu lizokwenza konke ukuqinisekisa ukuthi isistimu yokugcina i-AERODISK iyeka ukusebenza, i.e. phule.

Kwenzeka ukuthi izihloko eziphathelene nomlando wenkampani yethu, mayelana nemikhiqizo yethu, kanye nesibonelo sokuqaliswa okuphumelelayo sezivele zilenga ku-Habré, okuyinto Sibonga kakhulu kozakwethu - i-TS Solution kanye nezinkampani zeSoftline.

Ngakho-ke, ngeke ngiqeqeshe amakhono okuphatha amakhophi nokunamathisela lapha, kodwa ngizomane nginikeze izixhumanisi kokwangempela kwalezi zihloko:

Ngifuna nokukuxoxela izindaba ezimnandi. Kodwa ngizoqala, kunjalo, ngenkinga. Thina, njengomthengisi osemusha, phakathi kwezinye izindleko, sihlala sibhekene neqiniso lokuthi onjiniyela abaningi nabaphathi bamane nje abazi ukuthi balusebenzise kanjani kahle uhlelo lwethu lokugcina.
Kuyacaca ukuthi ukuphatha izinhlelo eziningi zokugcina kubukeka kucishe kufane ngokombono womlawuli, kodwa umenzi ngamunye unezici zakhe. Futhi nathi asihlukile lapha.

Ngakho-ke, ukuze kube lula umsebenzi wokuqeqesha ongoti be-IT, sinqume ukunikela kulo nyaka emfundweni yamahhala. Ukuze senze lokhu, emadolobheni amaningi amakhulu aseRussia sivula inethiwekhi ye-AERODISK Competence Centers, lapho noma yimuphi uchwepheshe wezobuchwepheshe onentshisekelo angathatha isifundo mahhala futhi athole isitifiketi sokuphatha izinhlelo zokugcina IZINJINI ze-AERODISK.

Esikhungweni ngasinye sobuhlakani sizofaka isitendi sedemo esigcwele esiphuma ohlelweni lokugcina lwe-AERODISK kanye nesiphakeli esiphathekayo, lapho uthisha wethu azoqhuba khona ukuqeqeshwa ubuso nobuso. Sizoshicilela uhlelo lokusebenza lwezikhungo ze-Competence lapho zibukeka, kodwa sesivele sivule isikhungo e-Nizhny Novgorod futhi idolobha laseKrasnodar lilandelayo. Ungabhalisela ukuqeqeshwa usebenzisa izixhumanisi ezingezansi. Nalu ulwazi olwaziwayo okwamanje mayelana namadolobha namadethi:

  • Nizhny Novgorod (SEKUVULIWE - ungabhalisa lapha https://aerodisk.promo/nn/);
    Kuze kube ngu-Ephreli 16, 2019, ungavakashela isikhungo nganoma yisiphi isikhathi sokusebenza, futhi ngo-Ephreli 16, 2019, kuzohlelwa isifundo esikhulu sokuqeqesha.
  • I-Krasnodar (IYAVULA MADUZE - ungabhalisa lapha https://aerodisk.promo/krsnd/ );
    Kusukela ngo-April 9 kuya ku-April 25, 2019, ungavakashela isikhungo nganoma yisiphi isikhathi sokusebenza, futhi ngo-April 25, 2019, kuzohlelwa isifundo esikhulu sokuqeqesha.
  • Екатеринбург (KUYAVULA MADUZE, landela imininingwane ekuwebhusayithi yethu noma ku-Habré);
    Meyi-Juni 2019.
  • Новосибирск (landela imininingwane ekuwebhusayithi yethu noma ku-Habré);
    Okthoba 2019.
  • Красноярск (landela imininingwane ekuwebhusayithi yethu noma ku-Habré);
    Novemba 2019.

Futhi-ke, uma iMoscow ingekude nawe, ungavakashela ihhovisi lethu eMoscow nganoma yisiphi isikhathi futhi uthole ukuqeqeshwa okufanayo.

Konke. Sesiqedile ngezokumaketha, asiqhubekele kwezobuchwepheshe!

Ku-Habré sizoshicilela njalo izindatshana zobuchwepheshe mayelana nemikhiqizo yethu, izivivinyo zokulayisha, iziqhathaniso, izici zokusetshenziswa nokusetshenziswa okuthakazelisayo.

Ukuhlolwa kokuphahlazeka kwesistimu yesitoreji ye-AERODISK ENGINE N2, ukuhlolwa kwamandla

ISEXWAYISO! Ngemva kokufunda lesi sihloko, ungasho ukuthi: Yebo, umthengisi uzozihlola ukuze yonke into isebenze "nge-bang," izimo ze-greenhouse, njll. Ngizophendula: ayikho into enjalo! Ngokungafani nezimbangi zethu zakwamanye amazwe, sitholakala lapha, eduze nawe, futhi ungakwazi njalo ukuza kithi (eMoscow noma kunoma iyiphi iKomidi Elikhulu) futhi uhlole uhlelo lwethu lokugcina nganoma iyiphi indlela. Ngakho-ke, akunangqondo ukuthi silungise imiphumela ibe yisithombe esihle somhlaba, ngoba Kulula kakhulu ukuhlola. Kulabo abavilapha kakhulu ukuhamba futhi abangenaso isikhathi, singahlela ukuhlola okukude. Sinelebhu ekhethekile yalokhu. Xhumana nathi.

ACHTUNG-2! Lokhu kuhlolwa akukona ukuhlolwa komthwalo, ngoba lapha sinendaba kuphela nokubekezelela amaphutha. Emasontweni ambalwa, sizolungisa ukuma okunamandla futhi siqhube ukuhlolwa komthwalo wesistimu yokugcina, sishicilela imiphumela lapha (ngendlela, izicelo zokuhlolwa ziyamukelwa).

Ngakho-ke, asihambe siyiphule.

Isitendi sokuhlola

Isitendi sethu siqukethe ihadiwe elandelayo:

  • I-1 x Uhlelo lwe-Aerodisk Engine N2 yokugcina (izilawuli ezingu-2, i-cache engu-64GB, amachweba we-8xFC 8Gb / s, amachweba angu-4xEthernet 10Gb / s SFP+, amachweba we-4xEthernet 1Gb / s); Amadiski alandelayo afakiwe ohlelweni lokugcina:
  • 4 x SAS SSD amadiski 900 GB;
  • 12 x SAS 10k amadiski 1,2 TB;
  • 1 x Iseva ebonakalayo ene-Windows Server 2016 (2xXeon E5 2667 v3, 96GB RAM, 2xFC port 8Gb/s, 2xEthernet port 10Gb/s SFP+);
  • 2 x iswishi ye-SAN 8G;
  • 2 x LAN 10G switch;

Sixhume iseva ohlelweni lwesitoreji ngamaswishi nge-FC ne-10G Ethernet. Umdwebo wokuma ungezansi.

Ukuhlolwa kokuphahlazeka kwesistimu yesitoreji ye-AERODISK ENGINE N2, ukuhlolwa kwamandla

Izingxenye esizidingayo, njengeMPIO kanye ne-iSCSI initiator, zifakwe ku-Windows Server.
Izindawo zilungiswa ekushintsheni kwe-FC, ama-VLAN ahambisanayo ayalungiswa ekushintshanisweni kwe-LAN, futhi i-MTU 9000 ifakiwe kumachweba okugcina, amaswishi, kanye nomsingathi (ukwenza konke lokhu kuchazwe kumadokhumenti ethu, ngakho ngeke sichaze. lolu hlelo lapha).

Indlela yokuhlola

Uhlelo lokuhlola ukuphahlazeka lumi kanje:

  • Ihlola ukwehluleka kwezimbobo ze-FC ne-Ethernet.
  • Ukuhlola ukuhluleka kwamandla.
  • Ukuhlola ukuhluleka kwesilawuli.
  • Ihlola ukwehluleka kwediski eqenjini/echibini.

Zonke izivivinyo zizokwenziwa ngaphansi kwezimo zokwenziwa komthwalo, esizozikhiqiza ngohlelo lwe-IOMETER. Ngokuhambisanayo, sizokwenza izivivinyo ezifanayo, kodwa ngaphansi kwezimo zokukopisha amafayela amakhulu ohlelweni lokugcina.

I-IOmeter config imi kanje:

  • Funda/Bhala - 70/30
  • Vimba - 128k (sanquma ukugeza izinhlelo zokugcina ngamabhulokhi amakhulu)
  • Inani lezintambo - 128 (elifana kakhulu nomthwalo okhiqizayo)
  • Okungahleliwe Okugcwele
  • Inombolo Yabasebenzi - 4 (2 ye-FC, 2 ye-iSCSI)

Ukuhlolwa kokuphahlazeka kwesistimu yesitoreji ye-AERODISK ENGINE N2, ukuhlolwa kwamandla
Ukuhlolwa kokuphahlazeka kwesistimu yesitoreji ye-AERODISK ENGINE N2, ukuhlolwa kwamandla

Ukuhlolwa kunezinjongo ezilandelayo:

  1. Qinisekisa ukuthi ukulayisha kokwenziwa kanye nenqubo yokukopisha ngeke kuphazamise noma kubangele amaphutha ngaphansi kwezimo ezihlukahlukene zokuhluleka.
  2. Qinisekisa ukuthi inqubo yokushintsha izimbobo, izilawuli, njll. izenzakalela ngokwanele futhi ayidingi izenzo zomlawuli uma kwenzeka ukwehluleka (okungukuthi, ngesikhathi sokuhluleka, asikhulumi ngokuhluleka, kunjalo).
  3. Qiniseka ukuthi ulwazi olukulogi luboniswa ngendlela efanele.

Ilungiselela umsingathi kanye nesistimu yokugcina

Silungise ukufinyelela kwebhulokhi kusistimu yesitoreji sisebenzisa izimbobo ze-FC ne-Ethernet (FC ne-iSCSI, ngokulandelana). Abafana abavela ku-TS Solution bachaze ngokuningiliziwe ukuthi bangakwenza kanjani lokhu esihlokweni esedlule (https://habr.com/ru/company/tssolution/blog/432876/). Futhi-ke, akekho okhansele amamanyuwali nezifundo.

Sakha iqembu eliyingxubevange sisebenzisa wonke amadrayivu esasinawo. Amadiski angu-2 we-SSD angeziwe kunqolobane, amadiski angu-2 we-SSD angeziwe njengesigaba sokugcina esingeziwe (i-Online-tier). Siqoqe amadrayivu angu-12 e-SAS10k kwaba i-RAID-60P (i-triple parity) ukuze sihlole ukwehluleka kwamadrayivu amathathu eqenjini ngesikhathi esisodwa. Idiski eyodwa ishiywe ukuze ifakwe esikhundleni.

Ukuhlolwa kokuphahlazeka kwesistimu yesitoreji ye-AERODISK ENGINE N2, ukuhlolwa kwamandla

Sixhume ama-LUN amabili (enye nge-FC, enye nge-iSCSI).

Ukuhlolwa kokuphahlazeka kwesistimu yesitoreji ye-AERODISK ENGINE N2, ukuhlolwa kwamandla

Umnikazi wawo womabili ama-LUN uyisilawuli se-Engine-0

Ukuhlolwa kokuphahlazeka kwesistimu yesitoreji ye-AERODISK ENGINE N2, ukuhlolwa kwamandla

Asiqale isivivinyo

Sivumela i-IOMETER ngokulungiselelwa okungenhla.

Ukuhlolwa kokuphahlazeka kwesistimu yesitoreji ye-AERODISK ENGINE N2, ukuhlolwa kwamandla

Sirekhoda ukuphuma kwe-1.8 GB/s kanye nokubambezeleka kwama-millisecond angu-3. Awekho amaphutha (Inani Eliphelele Lephutha).

Ngesikhathi esifanayo, kusukela kudrayivu yendawo ethi "C" yomphathi wethu, ngokufanayo siqala ukukopisha amafayela amabili amakhulu angu-100GB ku-FC kanye ne-iSCSI storage LUNs (ishayela u-E no-G ku-Windows), sisebenzisa ezinye izixhumi ezibonakalayo.

Ngenhla inqubo yokukopisha ku-LUN FC, ngezansi ku-iSCSI.

Ukuhlolwa kokuphahlazeka kwesistimu yesitoreji ye-AERODISK ENGINE N2, ukuhlolwa kwamandla

Isivivinyo #1: Ukukhubaza izimbobo ze-I/O

Sisondela ohlelweni lwesitoreji ngemuva))) futhi ngokuhamba kancane kwesandla sikhipha zonke izintambo ze-FC ne-Ethernet 10G kusilawuli se-Engine-0. Kunjengokungathi owesifazane ohlanzayo osebenzisa i-mop wadlula futhi wanquma ukugeza phansi lapho kulele khona i-snot kanye nezintambo (okungukuthi isilawuli sisasebenza, kodwa amachweba e-I/O afile).

Ukuhlolwa kokuphahlazeka kwesistimu yesitoreji ye-AERODISK ENGINE N2, ukuhlolwa kwamandla

Ake sibheke i-IOMETER futhi sikopishe amafayela. Ukudlulisa kwehle kwafika ku-0,5 GB/s, kodwa kwabuyela ngokushesha ezingeni layo langaphambilini (cishe imizuzwana emi-4-5). Awekho amaphutha.

Ukuhlolwa kokuphahlazeka kwesistimu yesitoreji ye-AERODISK ENGINE N2, ukuhlolwa kwamandla

Ukukopisha amafayela akumile, kukhona ukwehla kwejubane, kodwa akubalulekile neze (kusuka ku-840 MB/s kwehle kuya ku-720 MB/s). Ukukopisha akukakayeki.

Sibheka amalogi esistimu yesitoreji futhi sibona umlayezo mayelana nokungatholakali kwezimbobo kanye nokuthutha okuzenzakalelayo kweqembu.

Ukuhlolwa kokuphahlazeka kwesistimu yesitoreji ye-AERODISK ENGINE N2, ukuhlolwa kwamandla

Ithimba lezolwazi liphinde lisitshele ukuthi yonke into ayilungile neze ngamachweba eFC.

Ukuhlolwa kokuphahlazeka kwesistimu yesitoreji ye-AERODISK ENGINE N2, ukuhlolwa kwamandla

Isistimu yokulondoloza iphumelele ekuhlulekeni kwezimbobo ze-I/O ngempumelelo.

Inombolo yokuhlola 2. Ikhubaza isilawuli sesitoreji

Cishe ngokushesha (ngemuva kokuxhuma izintambo emuva ohlelweni lwesitoreji) sinqume ukuqeda isistimu yokugcina ngokukhipha isilawuli ku-chassis.

Siphinde futhi sisondele ohlelweni lokugcina ngemuva (esithandile))) futhi kulokhu sikhipha isilawuli se-Engine-1, okwamanje umnikazi we-RDG (lapho iqembu lithuthele khona).

Isimo ku-IOmeter simi kanje. I/O ime cishe imizuzwana emi-5. Amaphutha awaqoqi.

Ukuhlolwa kokuphahlazeka kwesistimu yesitoreji ye-AERODISK ENGINE N2, ukuhlolwa kwamandla

Ngemva kwamasekhondi angu-5, i-I/O iphinde yaqala kabusha cishe ngokuphuma okufanayo, kodwa ngokubambezeleka kwama-millisecond angu-35 (ukubambezeleka kulungiswe ngemva kwemizuzu embalwa). Njengoba kungabonwa ezithombeni-skrini, inani lenani lephutha eliphelele lingu-0, okusho ukuthi, awekho amaphutha okubhala noma okufunda.

Ukuhlolwa kokuphahlazeka kwesistimu yesitoreji ye-AERODISK ENGINE N2, ukuhlolwa kwamandla

Ake sibheke ukukopisha amafayela ethu. Njengoba ubona, akuzange kuphazamiseke, kube nokwehla kancane ekusebenzeni, kodwa sekukonke konke kubuyele ku-~ 800 MB/s efanayo.

Ukuhlolwa kokuphahlazeka kwesistimu yesitoreji ye-AERODISK ENGINE N2, ukuhlolwa kwamandla

Siya ohlelweni lwesitoreji futhi sibone isiqalekiso kuphaneli yolwazi ukuthi isilawuli se-Engine-1 asitholakali (impela, siyibulele).

Ukuhlolwa kokuphahlazeka kwesistimu yesitoreji ye-AERODISK ENGINE N2, ukuhlolwa kwamandla

Siphinde sibone ukungena okufanayo kumalogi.

Ukuhlolwa kokuphahlazeka kwesistimu yesitoreji ye-AERODISK ENGINE N2, ukuhlolwa kwamandla

Isilawuli sesitoreji naso sisindile ekwehlulekeni ngempumelelo.

Inombolo yokuhlola 3: Ukunqamula ukunikezwa kwamandla.

Uma kwenzeka, siqale ukukopisha amafayela futhi, kodwa asizange simise i-IOMETER.
Sidonsa iyunithi yokuphakela amandla.

Ukuhlolwa kokuphahlazeka kwesistimu yesitoreji ye-AERODISK ENGINE N2, ukuhlolwa kwamandla

Esinye isexwayiso sengeziwe ohlelweni lwesitoreji kuphaneli yolwazi.

Ukuhlolwa kokuphahlazeka kwesistimu yesitoreji ye-AERODISK ENGINE N2, ukuhlolwa kwamandla

Nakwimenyu yezinzwa siyabona ukuthi izinzwa ezihambisana nogesi okhishiwe seziphenduke zababomvu.

Ukuhlolwa kokuphahlazeka kwesistimu yesitoreji ye-AERODISK ENGINE N2, ukuhlolwa kwamandla

Isistimu yokugcina iyaqhubeka nokusebenza. Ukwehluleka kweyunithi yokuphakela amandla akuthinti nganoma iyiphi indlela ukusebenza kwesistimu yesitoreji; ngokombono womsingathi, isivinini sokukopisha nezikhombisi ze-IOMETER zihlale zingashintshile.

Ukuhlolwa kokuhluleka kwamandla kuphumelele ngempumelelo.

Ngaphambi kokuhlolwa kokugcina, sinqume ukubuyisela isistimu yokugcina iphile kancane, sibuyisele isilawuli kanye neyunithi yokuphakela amandla, futhi sibeke izintambo ngokulandelana kwazo, uhlelo lwesitoreji esazisa ngalo ngenjabulo ngezithonjana eziluhlaza kuphaneli yayo yezempilo. .

Ukuhlolwa kokuphahlazeka kwesistimu yesitoreji ye-AERODISK ENGINE N2, ukuhlolwa kwamandla

Isivivinyo No. 4. Ukuhluleka kwamadiski amathathu eqenjini

Ngaphambi kwalokhu kuhlolwa, senze esinye isinyathelo sokulungiselela. Iqiniso liwukuthi uhlelo lokugcina lwe-ENGINE luhlinzeka ngento ewusizo kakhulu - izinqubomgomo ezihlukene zokwakha kabusha. I-TS Solution ibhale ngalesi sici phambilini, kodwa ake sikhumbule ingqikithi yaso. Umlawuli wesitoreji angacacisa ukubaluleka kokwabiwa kwensiza ngesikhathi sokwakha kabusha. Kuphakathi kokuqondiswa kokusebenza kwe-I/O, okungukuthi, ukwakha kabusha kuthatha isikhathi eside, kodwa akukho ukwehla kokusebenza. Noma ekuqondeni kwejubane lokwakha kabusha, kepha ukukhiqiza kuzoncishiswa. Noma inketho elinganiselayo. Njengoba ukusebenza kwesitoreji ngesikhathi sokwakhiwa kabusha kweqembu lediski kuhlale kuyikhanda lomqondisi, sizohlola inqubomgomo ngokuchema ekusebenzeni kwe-I/O futhi ngezindleko zokuvuselela isivinini.

Ukuhlolwa kokuphahlazeka kwesistimu yesitoreji ye-AERODISK ENGINE N2, ukuhlolwa kwamandla

Manje ake sihlole ukwehluleka kwediski. Futhi sivumela ukurekhoda kuma-LUNs (amafayela kanye ne-IOMETER). Njengoba sineqembu elinama-triple parity (RAID-60P), lokhu kusho ukuthi uhlelo kufanele lumelane nokwehluleka kwamadiski amathathu, futhi ngemva kokwehluleka, ukushintshwa okuzenzakalelayo kufanele kusebenze, idiski eyodwa kufanele ithathe indawo yeyodwa ehlulekile. ku-RDG, futhi ukwakha kabusha kufanele kuqale kuyo.

Qala. Okokuqala, ngokusebenzisa isikhombimsebenzisi sesitoreji, ake sigqamise amadiski esifuna ukuwakhipha (ukuze singaphuthelwa futhi sidonse i-autochange disk).

Ukuhlolwa kokuphahlazeka kwesistimu yesitoreji ye-AERODISK ENGINE N2, ukuhlolwa kwamandla

Sihlola inkomba ku-hardware. Konke kulungile, sibona amadiski amathathu agqanyisiwe.

Ukuhlolwa kokuphahlazeka kwesistimu yesitoreji ye-AERODISK ENGINE N2, ukuhlolwa kwamandla

Futhi sikhipha lawa madiski amathathu.

Ukuhlolwa kokuphahlazeka kwesistimu yesitoreji ye-AERODISK ENGINE N2, ukuhlolwa kwamandla

Ake sibheke ukuthi yini ekusokhaya. Futhi lapho... akukho okukhethekile okwenzekile.

Ukuhlolwa kokuphahlazeka kwesistimu yesitoreji ye-AERODISK ENGINE N2, ukuhlolwa kwamandla
Ukuhlolwa kokuphahlazeka kwesistimu yesitoreji ye-AERODISK ENGINE N2, ukuhlolwa kwamandla

Izinkomba zokukopisha (ziphakeme kunasekuqaleni, ngoba i-cache ifudumala) futhi i-IOMETER ayishintshi kakhulu lapho isusa amadiski futhi iqala ukwakha kabusha (ngaphakathi kwe-5-10%).

Ake sibheke ukuthi yini ekusistimu yokugcina.

Ukuhlolwa kokuphahlazeka kwesistimu yesitoreji ye-AERODISK ENGINE N2, ukuhlolwa kwamandla

Esimeni seqembu, sibona ukuthi inqubo yokuhlelwa kabusha isiqalile futhi isiseduze nokuqedwa.

Ukuhlolwa kokuphahlazeka kwesistimu yesitoreji ye-AERODISK ENGINE N2, ukuhlolwa kwamandla

Ku-skeleton ye-RDG ungabona ukuthi amadiski angu-2 asesimweni esibomvu, futhi eyodwa isivele ishintshiwe. I-autoreplacement disk ayisekho; ithathe indawo yediski yesithathu ehlulekile. Ukwakhiwa kabusha kuthathe imizuzu embalwa, ukubhala amafayela lapho ama-disks angu-3 ehluleka awazange aphazamiseke, futhi ukusebenza kwe-I/O akuzange kushintshe kakhulu.

Ukuhlolwa kokuphahlazeka kwesistimu yesitoreji ye-AERODISK ENGINE N2, ukuhlolwa kwamandla

Ukuhlolwa kokuphahlazeka kwesistimu yesitoreji ye-AERODISK ENGINE N2, ukuhlolwa kwamandla

Ukuhlolwa kokuhluleka kwediski kuphumelele nakanjani ngempumelelo.

isiphetho

Kuleli qophelo, sinqume ukumisa udlame olubhekiswe kumasistimu okulondoloza. Ake sifinyeze:

  • Ukuhlolwa kokuhluleka kwembobo ye-FC - kuphumelele
  • Ukuhlola ukuhluleka kwembobo ye-Ethernet - kuphumelele
  • Ukuhlola ukwehluleka kwesilawuli - kuphumelele
  • Ukuhlolwa Kokuhluleka Kwamandla - Kuphumelele
  • Ukuhlola ukwehluleka kwediski ku-grouppool - kuphumelele

Akukho noyedwa ukwehluleka okuyekile ukurekhoda noma okudale amaphutha emthwalweni wokwenziwa; vele, kube khona ukushaya kokusebenza (futhi siyazi ukuthi singakunqoba kanjani, esizokwenza maduze), kodwa uma kubhekwa ukuthi lena imizuzwana, kuyamukeleka. Isiphetho: ukubekezelelana kwephutha kuzo zonke izingxenye zesistimu yokugcina i-AERODISK esebenza ezingeni, kwakungekho amaphuzu okuhluleka.

Ngokusobala, esihlokweni esisodwa asikwazi ukuhlola zonke izimo zokuhluleka, kodwa sizame ukuhlanganisa ezidume kakhulu. Ngakho-ke, sicela uthumele imibono yakho, iziphakamiso zokushicilelwa kwesikhathi esizayo futhi, yiqiniso, ukugxeka okwanele. Sizokujabulela ukuxoxa (noma okungcono, woza ekuqeqeshweni, ngiphindaphinda uhlelo uma kwenzeka)! Kuze kube ukuhlolwa okusha!

  • Nizhny Novgorod (SEKUVULIWE - ungabhalisa lapha https://aerodisk.promo/nn/);
    Kuze kube ngu-Ephreli 16, 2019, ungavakashela isikhungo nganoma yisiphi isikhathi sokusebenza, futhi ngo-Ephreli 16, 2019, kuzohlelwa isifundo esikhulu sokuqeqesha.
  • I-Krasnodar (IYAVULA MADUZE - ungabhalisa lapha https://aerodisk.promo/krsnd/ );
    Kusukela ngo-April 9 kuya ku-April 25, 2019, ungavakashela isikhungo nganoma yisiphi isikhathi sokusebenza, futhi ngo-April 25, 2019, kuzohlelwa isifundo esikhulu sokuqeqesha.
  • Екатеринбург (KUYAVULA MADUZE, landela imininingwane ekuwebhusayithi yethu noma ku-Habré);
    Meyi-Juni 2019.
  • Новосибирск (landela imininingwane ekuwebhusayithi yethu noma ku-Habré);
    Okthoba 2019.
  • Красноярск (landela imininingwane ekuwebhusayithi yethu noma ku-Habré);
    Novemba 2019.

Source: www.habr.com

Engeza amazwana