Waizonzwa sei kana rimwe zuva rakanaka rezhizha nzvimbo yedata ine midziyo yako yakaita seizvi?
Mhoroi mose! Zita rangu ndiDmitry Samsonov, ndinoshanda semutungamiri wehurongwa pa "" Mufananidzo wacho unoratidza imwe yenzvimbo ina dzedhata panoiswa midziyo inoshandira purojekiti yedu. Kuseri kwemadziro aya kune zviuru zvina zvezvimedu zvemidziyo: maseva, data rekuchengetedza masisitimu, network zvishandiso, nezvimwe. - ingangoita ⅓ yemidziyo yedu yese.
Masevha mazhinji ari Linux. Kune zvakare akati wandei maseva paWindows (MS SQL) - nhaka yedu, yatave tichisiya zvine hungwaru kwemakore mazhinji.
Saka, musi waChikumi 5, 2019 na14:35, mainjiniya pane imwe nzvimbo yedu yedata vakashuma aramu yemoto.
Negation
14:45. Zviitiko zvidiki zveutsi munzvimbo dze data zvinowanzoitika kupfuura zvaunofunga. Zviratidzo mukati mehoro zvaive zvakajairika, saka maitiro edu ekutanga aive akadzikama: vakaunza kurambidzwa kwebasa nekugadzira, ndiko kuti, pane chero shanduko yekuchinja, pakuburitsa shanduro nyowani, nezvimwe, kunze kwebasa rine chekuita nekugadzirisa chimwe chinhu.
Hasha
Wakamboedza kutsvaga kubva kune vanodzima moto paya pakaitika moto padenga, kana kuti upinde padenga rinopfuta iwe pachako kuti uongorore mamiriro? Chii chichava chiyero chekuvimba mumashoko anogamuchirwa kuburikidza nevanhu vashanu?
14: 50. Ruzivo rwakagamuchirwa rwekuti moto wave kusvika painotonhora. Asi zvichauya here? Iyo system administrator pabasa inobvisa traffic yekunze kubva kumberi kweiyi data center.
Parizvino, mativi emasevhisi edu ese akadhindwa munzvimbo nhatu dzedhata, kuenzanisa kunoshandiswa padanho reDNS, izvo zvinotibvumira kubvisa kero dzeimwe data data kubva kuDNS, nekudaro kudzivirira vashandisi kubva kumatambudziko anogona kuitika nekuwana masevhisi. . Kana matambudziko atoitika munzvimbo yedata, inosiya kutenderera otomatiki. Unogona kuverenga zvakawanda pano:
Moto hausati watikanganisa nenzira ipi zvayo - kana vashandisi kana midziyo haina kukuvara. Iyi itsaona here? Chikamu chekutanga chegwaro "Accident Action Plan" inotsanangura pfungwa ye "Accident", uye chikamu chinopera seizvi:
«Kana paine kusava nechokwadi kuti kune tsaona here kana kuti kwete, saka itsaona!»
14:53. Murongi wechimbichimbi anogadzwa.
Murongi ndiye munhu anodzora kutaurirana pakati pevatori vechikamu vese, anoongorora chiyero chetsaona, anoshandisa Emergency Action Plan, anokwezva vashandi vanodiwa, anotarisa kupedzwa kwekugadzirisa, uye zvinonyanya kukosha, anotumira chero mabasa. Mune mamwe mazwi, uyu ndiye munhu anotarisira iyo yese emergency mhinduro maitiro.
Kukurukurirana
15:01. Isu tinotanga kudzima maseva asina hukama nekugadzirwa.
15:03. Isu tinodzima nemazvo ese akachengetwa masevhisi.
Izvi zvinosanganisira kwete mafronts chete (ayo nevashandisi venzvimbo iyi havachakwanise kuwana) uye masevhisi avo ebetsero (bhizinesi logic, caches, nezvimwewo), asiwo akasiyana dhatabhesi ane replication factor 2 kana kupfuura (, , , etc.).
15: 06. Ruzivo rwakagamuchirwa kuti moto uri kutyisidzira imwe yenzvimbo dze data data. Isu hatina midziyo muimba ino, asi chokwadi chokuti moto unogona kupararira kubva padenga kusvika kuhoro inoshandura zvikuru mufananidzo wezviri kuitika.
(Gare gare zvakazooneka kuti pakanga pasina tyisidziro yokunyama pahoro yacho, sezvo yakanga yakavharwa kubva padenga.
15:07. Isu tinobvumira kurairwa kwekuraira pamaseva mune yakakurumidza modhi pasina mamwe macheki ().
15:08. Tembiricha mumahoro iri mukati memiganhu yakajairika.
15: 12. Kuwedzera kwekushisa muhoro kwakanyorwa.
15:13. Inopfuura hafu yemaseva ari munzvimbo yedata inodzimwa. Ngatienderere mberi.
15:16. Sarudzo yakaitwa yekudzima midziyo yese.
15:21. Isu tinotanga kudzima simba kumaseva asina mamiriro pasina kudzima nemazvo application uye system yekushandisa.
15:23. Boka revanhu vane basa reMS SQL rakagoverwa (kune vashoma vavo, kutsamira kwesevhisi pavari hakuna kunaka, asi nzira yekudzorera mashandiro inotora nguva yakareba uye yakaoma kupfuura, semuenzaniso, Cassandra).
Kushungurudzika
15: 25. Ruzivo rwakagamuchirwa pamusoro pesimba raidzimwa muhoro ina kubva mu16 (No. 6, 7, 8, 9). Midziyo yedu iri muhoro 7 ne8. Hapana ruzivo nezve horo dzedu mbiri (Nhamba. 1 ne3).
Kazhinji, panguva yemoto, magetsi anodzimwa pakarepo, asi munyaya iyi, nekuda kwebasa rakarongeka revanodzima moto uye vashandi vezvenyanzvi ve data data, harina kudzimwa kwose kwose uye kwete pakarepo, asi sezvinodiwa.
(Zvakazoonekwa gare gare kuti simba rakanga risina kudzimwa muhoro 8 ne9.)
15:28. Tiri kutanga kuendesa MS SQL databases kubva kune backups mune dzimwe nzvimbo dzedata.
Zvichatora nguva yakareba sei? Pane network yakakwana here yenzira yese?
15: 37. Kuvharwa kwezvimwe zvikamu zvetiweki kwakanyorwa.
Management uye network yekugadzira zvakaparadzaniswa kubva kune mumwe nemumwe. Kana iyo network yekugadzira iripo, saka unogona kuenda kune server, misa application uye wodzima OS. Kana isiripo, saka unogona kupinda mukati kuburikidza neIPMI, kumisa application uye kudzima OS. Kana pasina network, iwe haugone kuita chero chinhu. "Thanks, Cap!", iwe unofunga.
"Uye kazhinji, kune mhirizhonga yakawanda," iwe unogona kufungawo.
Chinhu ndechekuti maseva, kunyangwe pasina moto, anogadzira huwandu hukuru hwekupisa. Kunyanya, kana paine kutonhora, vanogadzira kupisa, uye kana pasina kutonhora, vanogadzira gehena remoto, iro rinonyanya kunyungudutsa chikamu chemidziyo uye kudzima chimwe chikamu, uye zvakanyanya ... kukonzera moto mukati. horo, iyo inenge yakavimbiswa kuparadza zvose.
15:39. Isu tinogadzirisa matambudziko neiyo conf database.
Iyo conf dhatabhesi ndiyo yekuseri kwesevhisi yezita rimwe chete, iro rinoshandiswa neese zvigadzirwa zvekugadzira kukurumidza kuchinja marongero. Pasina chigadziko ichi, hatigone kudzora kushanda kweiyo portal, asi iyo portal pachayo inogona kushanda.
15:41. Tembiricha sensors paCore network midziyo rekodhi kuverenga padyo nepamusoro inobvumidzwa. Iri ibhokisi rinogara rakazara rack uye rinovimbisa kushanda kwese network mukati me data data.
15:42. Issue tracker uye wiki hazviwanikwe, chinja kune standby.
Uku hakusi kugadzira, asi kana pakaitika tsaona, kuwanikwa kwechero hwaro hweruzivo kunogona kuve kwakakosha.
15:50. Imwe yeamonitor system yakadzimwa.
Kune akati wandei awo, uye iwo ane mutoro weakasiyana masevhisi. Mamwe acho akarongedzerwa kuti ashande akazvimirira mukati mega yega data data (kureva kuti, ivo vanongotarisa yavo vega data data), mamwe anosanganisira akagoverwa zvikamu zvinopona pachena kurasikirwa kwechero data data.
Pakadai yakamira kushanda , iyo inoshanda mu master-standby mode. Kuchinjira kune standby.
Kugamuchirwa
15:51. Masevha ese kunze kweMS SQL akadzimwa kuburikidza neIPMI asina kuvhara nemazvo.
Wakagadzirira hukuru hwesevha manejimendi kuburikidza neIPMI kana zvichidikanwa?
Iyo nguva chaiyo apo kununurwa kwemidziyo mu data data kunopedzwa panguva ino. Zvose zvaigona kuitwa zvakaitwa. Vamwe vaunoshanda navo vanogona kuzorora.
16: 13. Ruzivo rwakagamuchirwa kuti mapaipi efreon kubva kumhepo inoputika akaputika padenga - izvi zvinononoka kuvhurwa kwenzvimbo yedata mushure mekunge moto wabviswa.
16:19. Zvinoenderana nedata yakagamuchirwa kubva kuvashandi vehunyanzvi ve data data, kuwedzera kwekushisa muhoro kwakamira.
17:10. Iyo conf database yakadzoserwa. Iye zvino tinogona kushandura zvirongwa zvekushandisa.
Sei izvi zvakakosha kana zvese zvichishivirira kukanganisa uye zvinoshanda kunyangwe pasina imwe nzvimbo yedata?
Chekutanga, hazvisi zvese zvinoshivirira kukanganisa. Kune akasiyana masevhisi masevhisi anga asati apona pakutadza kwedata centre zvakanaka zvakakwana, uye kune dhatabhesi mune master-standby mode. Iko kugona kugadzirisa zvigadziriso kunotendera iwe kuti uite zvese zvinodiwa kuti uderedze kukanganisa kwemhedzisiro yetsaona kune vashandisi kunyangwe mumamiriro ezvinhu akaoma.
Chechipiri, zvakava pachena kuti kushanda kwenzvimbo yedata hakuzodzoserwe zvizere mumaawa anouya, saka zvaive zvakafanira kutora matanho ekuona kuti kusawanikwa kwenguva refu kwezvinyorwa hakukonzerese mamwe matambudziko akadai semadhisiki akazara mukati. iyo yakasara data centers.
17:29. Pizza nguva! Tinoshandisa vanhu, kwete marobhoti.
Kugadzirisa
18:02. Muhoro nhamba 8 (yedu), 9, 10 uye 11 tembiricha yakagadzikana. Imwe yeayo anoramba ari offline (No. 7) anogara midziyo yedu, uye tembiricha ikoko kunoramba kuchikwira.
18:31. Vakapa mvumo yekutanga midziyo muhoro nhamba 1 ne3 - horo idzi hadzina kukanganiswa nemoto.
Parizvino, maseva ari kutangwa muhoro Nha. Kushanda kwemasevhisi ese anomhanya kunotariswa. Pachine matambudziko nehoro Nhamba 1.
18:44. Vashandi vezvenyanzvi ve data data vakawana kuti muimba Nhamba 7 (apo chete midziyo yedu iripo) mavhavha akawanda haana kudzimwa. Zvinoenderana nedata redu, maseva makumi maviri nenhanhatu anoramba ari online ipapo. Mushure mekutarisa kwechipiri, tinowana 26 maseva.
20:18. Vanamazvikokota veData centre vanofuridza mweya nemukamuri isina air-conditioned kuburikidza nenhare dzinofamba nemumikoto.
23:08. Admin wekutanga akaendeswa kumba. Munhu anoda kurara husiku kuti aenderere mberi nebasa mangwana. Tevere, isu tichaburitsa mamwe ma admins uye vanogadzira.
02:56. Takaparura zvese zvaigona kutangwa. Isu tinoita yakawanda yekutarisa masevhisi ese tichishandisa otomatiki bvunzo.
03:02. Air conditioning mune yekupedzisira, horo yechinomwe yakadzorerwa.
03:36. Isu takaunza kumberi mu data center mukutenderera muDNS. Kubva panguva ino traffic yemushandisi inotanga kusvika.
Tiri kutumira vazhinji vechikwata chekutonga kumba. Asi tinosiya vanhu vashoma.
Diki FAQ:
Q: Chii chakaitika kubva pa18:31 kusvika 02:56?
A: Kutevera "Chirongwa Chekuita Njodzi", tinotangisa masevhisi ese, kutanga neakanyanya kukosha. Muchiitiko ichi, murongi munhaurirano anoburitsa sevhisi kune akasununguka maneja, anotarisa kana OS uye application zvatanga, kana paine zvikanganiso, uye kana zviratidzo zvakajairika. Mushure mekunge kuvhurwa kwapera, anoshuma kune chat kuti akasununguka uye anogamuchira sevhisi nyowani kubva kumurongi.
Maitiro acho anowedzera kudzikiswa neakakundikana hardware. Kunyangwe kana kumisa OS uye kuvhara maseva kwakafamba nemazvo, mamwe maseva haadzoke nekuda kwekutadza kungoerekana kwaitika kwema disks, memory, uye chassis. Kana simba rikarasika, mwero wekukundikana unowedzera.
Mubvunzo: Sei usingakwanise kungomhanyisa zvese kamwechete, wozogadzirisa zvinouya mukutarisisa?
A: Zvese zvinofanirwa kuitwa zvishoma nezvishoma, nekuti pane zvinotsamira pakati pemasevhisi. Uye zvinhu zvose zvinofanira kuongororwa pakarepo, pasina kumirira kutarisa - nokuti zviri nani kugadzirisa matambudziko pakarepo, pasina kumirira kuti zviwedzere.
7:40. Admin wekupedzisira (coordinator) akaenda kunorara. Basa rezuva rokutanga rapera.
8:09. Vagadziri vekutanga, mainjiniya epa data nevatungamiriri (kusanganisira murongi mutsva) vakatanga basa rekudzosera.
09:37. Takatanga kusimudza horo Nhamba 7 (yekupedzisira).
Panguva imwecheteyo, tinoenderera mberi nekudzorera izvo zvisina kugadziriswa mune mamwe makamuri: kutsiva disks / chiyeuchidzo / maseva, kugadzirisa zvinhu zvose "zvinopisa" mukutarisa, kushandura mabasa kumashure mune master-standby schemes nezvimwe zvinhu zviduku, izvo zviripo. zvisinei zvakawanda.
17:08. Isu tinobvumira zvese zvenguva dzose kushanda nekugadzira.
21:45. Basa rezuva rechipiri rinopedzwa.
09:45. Nhasi iChishanu. Pachine matambudziko mashoma mashoma mukutarisisa. Weekend iri mberi, munhu wese anoda kuzorora. Isu tinoenderera mberi nekugadzirisa zvakanyanya zvese zvatinogona. Regular admin mabasa angadai akamiswa akamiswa. Murongi mutsva.
15:40. Pakarepo hafu yeCore network yemidziyo stack muIMWE dhata data yakatangazve. Mafaro aitorwa kunze kwekutenderera kuti aderedze njodzi. Iko hakuna maitiro kune vashandisi. Zvakazoonekwa kuti yaive chassis yakashata. Murongi ari kushanda pakugadzirisa tsaona mbiri panguva imwe chete.
17:17. Kushanda kwetiweki mune imwe nzvimbo yedata yakadzorerwa, zvese zvakatariswa. Iyo data data inoiswa mukutenderera.
18:29. Basa rezuva rechitatu uye, kazhinji, kudzorerwa mushure mokunge njodzi yapera.
Afterword
04.04.2013 , "Vanodzidza navo" -kwemazuva matatu portal yakanga isipo zvachose kana kuti zvishoma. Munguva yese iyi, vanhu vanopfuura 100 vanobva kumaguta akasiyana, kubva kumakambani akasiyana (akawanda kutenda zvakare!), kure uye zvakananga munzvimbo dzedata, pamaoko uye otomatiki, vakagadzirisa zviuru zvemaseva.
Takatora mhedziso. Kuti izvi zvisaitike zvakare, takaita uye tinoramba tichiita basa rakakura kusvika nhasi.
Ndeupi musiyano mukuru pakati petsaona iripo ne404?
- Tine “Accident Action Plan”. Kamwe kota, tinoita maekisesaizi - tinotamba-tamba mamiriro ekukurumidzira, ayo boka revatariri (vese pamwe chete) vanofanira kubvisa vachishandisa "Emergency Action Plan". Vanotungamira sisitimu maneja vanochinjana kutamba basa remurongi.
- Kwetatu, mukuyedza mode, tinoparadzanisa nzvimbo dzedata (dzose dzichichinjana) kuburikidza neLAN neWAN network, izvo zvinotibvumira kuona nekukurumidza mabhodhoro.
- Madhisiki mashoma akatyoka, nekuti isu takasimbisa zviyero: mashoma ekushanda maawa, akaomesesa zvikumbaridzo zveSMART,
- Isu takasiya zvachose BerkeleyDB, dhatabhesi yekare uye isina kugadzikana yaida nguva yakawanda yekupora mushure mekutanga sevha.
- Isu takadzikisa huwandu hwemaseva neMS SQL uye nekudzora kutsamira pane asara.
- Isu tine zvedu , kwatave tichishingaira kutamisa masevhisi ose kwemakore maviri zvino. Gore rinorerutsa zvakanyanya kutenderera kwese kwekushanda nechishandiso, uye kana pakaitika tsaona inopa maturusi akasiyana se:
- kumira kwakaringana kwese maapplication nekudzvanya kumwe chete;
- kutama kuri nyore kwemaapplication kubva kune akakundikana maseva;
- otomatiki chinzvimbo (mukurongeka kwekutanga kwemasevhisi) kuvhurwa kwese data data.
Tsaona yarondedzerwa munyaya ino ndiyo hombe kubva pazuva rechi404. Chokwadi, hazvisi zvinhu zvose zvakafamba zvakanaka. Semuenzaniso, panguva yekushaikwa kwenzvimbo yedata remoto mune imwe nzvimbo yedata, dhisiki pane imwe sevha yakatadza, ndiko kuti, imwe chete yeatatu akadzokorora muCassandra cluster yakaramba ichiwanikwa, ndosaka 4,2% ye mobile application. vashandisi havana kukwanisa kupinda. Panguva imwecheteyo, vashandisi vakatobatana vakaenderera mberi nekushanda. Pakazara, semugumisiro wetsaona, matambudziko anopfuura makumi matatu akaonekwa - kubva kune banal bugs kusvika kune zvikanganiso mukuvakwa kwebasa.
Asi mutsauko wakanyanya kukosha pakati petsaona iripo ne404 ndeyekuti isu tichibvisa mhedzisiro yemoto, vashandisi vaive vachiri kutumira mameseji uye kuita mafoni efoni , vaitamba mitambo, vakateerera mimhanzi, vakapana zvipo, vakatarira mavhidhiyo, nhevedzano yeTV uye zviteshi zveTV mukati , ndokupindawo mukati .
Tsaona dzako dzinofamba sei?
Source: www.habr.com