Kodi mungamve bwanji ngati tsiku lina labwino lachilimwe malo opangira data okhala ndi zida zanu akuwoneka chonchi?
Moni nonse! Dzina langa ndine Dmitry Samsonov, ndimagwira ntchito ngati woyang'anira dongosolo ku "" Chithunzichi chikuwonetsa imodzi mwa malo anayi a data pomwe zida zomwe zimagwira ntchito yathu zimayikidwa. Kuseri kwa makoma awa pali zida za 4: ma seva, makina osungira deta, zida zama network, ndi zina zambiri. - pafupifupi ⅓ pazida zathu zonse.
Ma seva ambiri ndi Linux. Palinso ma seva angapo pa Windows (MS SQL) - cholowa chathu, chomwe takhala tikuchisiya mwadongosolo kwa zaka zambiri.
Chifukwa chake, pa Juni 5, 2019 nthawi ya 14:35, mainjiniya pa imodzi mwamalo athu opangira data adanenanso za alamu yamoto.
Zosokoneza
14:45. Zochitika zazing'ono za utsi m'malo opangira data ndizofala kuposa momwe mukuganizira. Zizindikiro za mkati mwa holo zinali zachilendo, kotero kuti zomwe tinachita poyamba zinali zodekha: adayambitsa chiletso cha ntchito ndi kupanga, ndiko kuti, pakusintha kulikonse, potulutsa matembenuzidwe atsopano, ndi zina zotero, kupatulapo ntchito yokhudzana ndi kukonza chinachake.
Mkwiyo
Kodi munayamba mwayesapo kufufuza kuchokera kwa ozimitsa moto kumene moto unachitikira padenga, kapena kuti mukwere nokha padenga loyaka moto kuti muwone momwe zinthu zilili? Kodi kudalirika kotani m'zidziwitso zolandilidwa ndi anthu asanu?
14: 50. Zambiri zalandilidwa kuti moto ukuyandikira njira yozizirira. Koma ibwera? Woyang'anira dongosolo ali pantchito amachotsa magalimoto akunja kuchokera kumalire a data center iyi.
Pakadali pano, mbali za mautumiki athu onse zimabwerezedwa m'malo atatu a data, kusanja kumagwiritsidwa ntchito pamlingo wa DNS, zomwe zimatilola kuchotsa maadiresi a malo amodzi kuchokera ku DNS, potero kuteteza ogwiritsa ntchito kumavuto omwe angakhalepo ndi mwayi wopeza ntchito. . Ngati zovuta zachitika kale mu data center, zimasiya kuzungulira kokha. Mutha kuwerenga zambiri apa:
Motowu sunatikhudzebe mwanjira ina iliyonse - palibe ogwiritsa ntchito kapena zida zomwe zawonongeka. Kodi iyi ndi ngozi? Gawo loyamba lachikalatacho "Ndondomeko Yochita Ngozi" likulongosola lingaliro la "Ngozi", ndipo gawolo limatha motere:
«Ngati pali chikaiko ngati pachitika ngozi kapena ayi, ndiye kuti ndi ngozi!»
14:53 . Wogwirizanitsa zadzidzidzi amasankhidwa.
Wogwirizanitsa ndi munthu amene amayendetsa kuyankhulana pakati pa onse omwe akugwira nawo ntchito, amayesa kukula kwa ngozi, amagwiritsa ntchito Emergency Action Plan, amakopa antchito ofunikira, amayang'anira kumalizidwa kwa kukonzanso, ndipo chofunika kwambiri, amapereka ntchito iliyonse. Mwa kuyankhula kwina, uyu ndi munthu amene amayang'anira ntchito yonse yoyankha mwadzidzidzi.
Mgwirizano
15:01. Timayamba kuletsa ma seva omwe sakugwirizana ndi kupanga.
15:03. Timazimitsa bwino ntchito zonse zosungidwa.
Izi zikuphatikiza osati malire okha (omwe pakadali pano ogwiritsa ntchito sathanso kufikira) ndi ntchito zawo zothandizira (malingaliro abizinesi, ma cache, ndi zina), komanso ma database osiyanasiyana okhala ndi replication factor 2 kapena kupitilira apo (, , , etc.).
15: 06. Zambiri zalandilidwa kuti moto ukuwopseza imodzi mwaholo za data center. Tilibe zipangizo m'chipinda chino, koma kuti moto ukhoza kufalikira kuchokera padenga kupita ku maholo amasintha kwambiri chithunzi cha zomwe zikuchitika.
(Kenako zinapezeka kuti panalibe chiwopsezo chakuthupi ku holoyo, popeza kuti inali yotsekeredwa padenga la nyumbayo. Chiwopsezocho chinali kokha ku makina ozizirira a holoyo.)
15:07. Timalola kulamula kwa ma seva mumayendedwe ofulumizitsa popanda macheke owonjezera ().
15:08. Kutentha m'maholo kuli mkati mwa malire abwino.
15: 12. Kuwonjezeka kwa kutentha m'maholo kunalembedwa.
15:13 . Zoposa theka la ma seva mu data center azimitsidwa. Tiyeni tipitilize.
15:16 . Chigamulo chinapangidwa kuti azimitsa zida zonse.
15:21 . Timayamba kuzimitsa mphamvu ku ma seva opanda malire popanda kutseka bwino ntchito ndi makina ogwiritsira ntchito.
15:23 . Gulu la anthu omwe ali ndi udindo wa MS SQL lagawidwa (pali ochepa a iwo, kudalira kwa mautumiki pa iwo sikuli kwakukulu, koma ndondomeko yobwezeretsa ntchito imatenga nthawi yayitali ndipo ndi yovuta kuposa, mwachitsanzo, Cassandra).
Kusokonezeka maganizo
15: 25. Chidziwitso chinalandiridwa ponena za mphamvu zozimitsidwa m'maholo anayi mwa 16 (No. 6, 7, 8, 9). Zida zathu zili muholo 7 ndi 8. Palibe chidziwitso chokhudza maholo athu awiri (No. 1 ndi 3).
Nthawi zambiri, pamoto, magetsi amazimitsidwa nthawi yomweyo, koma pakadali pano, chifukwa cha ntchito yogwirizana ya ozimitsa moto ndi akatswiri a data center, sichinazimitsidwe kulikonse komanso osati nthawi yomweyo, koma ngati kuli kofunikira.
(Pambuyo pake zinadziwika kuti magetsi sanazimitsidwe muholo 8 ndi 9.)
15:28. Tikuyamba kuyika ma database a MS SQL kuchokera ku zosunga zobwezeretsera kumalo ena a data.
Kodi zitenga nthawi yayitali bwanji? Kodi pali netiweki yokwanira panjira yonseyi?
15: 37. Kutsekedwa kwa mbali zina za netiweki kunajambulidwa.
Utsogoleri ndi maukonde opanga ndi olekanitsidwa mwakuthupi. Ngati maukonde opanga alipo, ndiye kuti mutha kupita ku seva, kuyimitsa kugwiritsa ntchito ndikuzimitsa OS. Ngati sichipezeka, ndiye kuti mutha kulowa kudzera pa IPMI, kuyimitsa kugwiritsa ntchito ndikuzimitsa OS. Ngati palibe maukonde, ndiye kuti simungathe kuchita chilichonse. "Zikomo, Kapu!", mungaganize.
"Ndipo kawirikawiri, pamakhala chipwirikiti," mungaganizenso.
Chinthucho ndi chakuti ma seva, ngakhale opanda moto, amapanga kutentha kwakukulu. Zowonjezereka, pakakhala kuzizira, zimapanga kutentha, ndipo pamene palibe kuziziritsa, zimapanga chiwombankhanga cha gehena, chomwe chidzasungunula mbali ina ya zipangizo ndikuzimitsa gawo lina, ndipo poipa kwambiri ... kuyambitsa moto mkati. holoyo, yomwe yatsimikizika kuti iwononga chilichonse.
15:39. Timakonza zovuta ndi database ya conf.
Conf database ndiye kumbuyo kwa ntchito ya dzina lomwelo, lomwe limagwiritsidwa ntchito ndi mapulogalamu onse opanga kuti asinthe masinthidwe mwachangu. Popanda maziko awa, sitingathe kuwongolera magwiridwe antchito a portal, koma portal yokha imatha kugwira ntchito.
15:41 . Zowunikira kutentha pa Core network zida zowerengera zowerengera pafupi kwambiri ndi zovomerezeka. Ili ndi bokosi lomwe limakhala ndi rack yonse ndikuwonetsetsa kuti maukonde onse mkati mwa data center.
15:42. Issue tracker ndi wiki palibe, sinthani ku standby.
Izi sizopanga, koma pakachitika ngozi, kupezeka kwa chidziwitso chilichonse kungakhale kofunikira.
15:50. Njira imodzi yowunikira yazimitsidwa.
Pali angapo a iwo, ndipo ali ndi udindo pazinthu zosiyanasiyana za mautumiki. Zina mwazo zimapangidwira kuti zizigwira ntchito mokhazikika mkati mwa malo aliwonse a data (ndiko kuti, amangoyang'anira malo awo okha), zina zimakhala ndi zigawo zomwe zimagawidwa zomwe zimapulumuka mwachisawawa kutayika kwa deta iliyonse.
Pamenepa idasiya kugwira ntchito , yomwe imagwira ntchito mu master-standby mode. Kusintha kukhala standby.
Kutengera ana
15:51. Ma seva onse kupatula MS SQL adazimitsidwa kudzera pa IPMI osatseka bwino.
Kodi mwakonzekera kuwongolera kwakukulu kwa seva kudzera pa IPMI ngati kuli kofunikira?
Nthawi yomweyo pamene kupulumutsidwa kwa zida mu data center kumalizidwa panthawiyi. Zonse zomwe zikanatheka zachitika. Anzathu ena amatha kupuma.
16: 13. Chidziwitso chalandiridwa kuti mapaipi a freon ochokera ku air conditioners anaphulika padenga - izi zidzachedwetsa kukhazikitsidwa kwa data center moto utachotsedwa.
16:19 . Malinga ndi zomwe adalandira kuchokera kwa ogwira ntchito zamakono a data center, kuwonjezeka kwa kutentha m'maholo kwasiya.
17:10. Conf database yabwezeretsedwa. Tsopano titha kusintha makonda a pulogalamu.
Chifukwa chiyani izi zili zofunika kwambiri ngati zonse ndizololera zolakwika ndipo zimagwira ntchito ngakhale popanda malo amodzi a data?
Choyamba, sizinthu zonse zomwe zimalolera zolakwika. Pali mautumiki osiyanasiyana achiwiri omwe sanapulumuke kulephera kwa data center mokwanira, ndipo pali ma database omwe ali mu master-standby mode. Kutha kuyang'anira makonda kumakupatsani mwayi wochita chilichonse chofunikira kuti muchepetse zotsatira za ngozi kwa ogwiritsa ntchito ngakhale pamavuto.
Kachiwiri, zinawonekeratu kuti ntchito ya data center sichidzabwezeretsedwa mokwanira mu maola akubwera, kotero kunali koyenera kuchitapo kanthu kuti atsimikizire kuti kusapezeka kwa nthawi yaitali kwa replicas sikunabweretse mavuto ena monga ma disks athunthu mu. malo otsala a data.
17:29. Nthawi ya pizza! Timalemba ntchito anthu, osati maloboti.
Kukonzanso
18:02. M'maholo No. 8 (athu), 9, 10 ndi 11 kutentha kwakhazikika. Mmodzi mwa omwe amakhala osalumikizidwa (No. 7) amakhala ndi zida zathu, ndipo kutentha komweko kukupitilira kukwera.
18:31. Iwo adapereka chilolezo choyambitsa zida m'maholo Nambala 1 ndi 3 - Nyumbazi sizinakhudzidwe ndi moto.
Pakalipano, ma seva akuyambitsidwa m'maholo No. 1, 3, 8, kuyambira ndi ovuta kwambiri. Kugwira ntchito moyenera kwa mautumiki onse omwe akuyendetsa kumawunikiridwa. Pali zovuta ndi holo nambala 7.
18:44. Ogwira ntchito zamakono a data center adapeza kuti mu chipinda No. Malinga ndi deta yathu, ma seva 7 amakhalabe pa intaneti pamenepo. Pambuyo pofufuza kachiwiri, timapeza ma seva 26.
20:18. Akatswiri akumalo opangira data amawuzira mpweya kudzera m'chipinda chopanda mpweya kudzera m'manjira oyenda m'njira zodutsamo.
23:08. Admin woyamba adatumizidwa kunyumba. Wina amafunika kugona usiku kuti apitirize ntchito mawa. Kenako, tidzamasula ma admins ndi omanga ena.
02:56. Tinayambitsa zonse zomwe zingayambitsidwe. Timayang'ana kwambiri ntchito zonse pogwiritsa ntchito kuyesa kodziwikiratu.
03:02. Zoziziritsa mpweya m'malo omaliza, holo yachisanu ndi chiwiri yabwezeretsedwa.
03:36. Tinabweretsa malire mu data center kuti azisinthasintha mu DNS. Kuyambira nthawi ino kuchuluka kwa ogwiritsa ntchito kumayamba kufika.
Tikutumiza ambiri a gulu la oyang'anira kunyumba. Koma timasiya anthu ochepa.
Mafunso Aang'ono:
Q: Chinachitika ndi chiyani kuyambira 18:31 mpaka 02:56?
A: Potsatira "Ndondomeko ya Masoka", timayambitsa ntchito zonse, kuyambira ndi zofunika kwambiri. Pankhaniyi, wogwirizira pamacheza akupereka chithandizo kwa woyang'anira waulere, yemwe amawunika ngati OS ndi kugwiritsa ntchito kwayamba, ngati pali zolakwika, komanso ngati zizindikirozo ndizabwinobwino. Kukhazikitsa kukamalizidwa, amauza macheza kuti ali mfulu ndipo amalandira ntchito yatsopano kuchokera kwa wogwirizanitsa.
Njirayi imachepetsedwanso ndi zida zolephera. Ngakhale kuyimitsa OS ndikutseka ma seva kupita molondola, ma seva ena samabwerera chifukwa cha kulephera kwadzidzidzi kwa ma disks, kukumbukira, ndi chassis. Mphamvu ikatha, kulephera kumawonjezeka.
Q: Chifukwa chiyani simungangoyendetsa chilichonse nthawi imodzi, ndikukonza zomwe zikubwera pakuwunika?
A: Chilichonse chiyenera kuchitika pang'onopang'ono, chifukwa pali zodalira pakati pa mautumiki. Ndipo zonse ziyenera kuyang'aniridwa nthawi yomweyo, osadikirira kuyang'anira - chifukwa ndi bwino kuthana ndi mavuto nthawi yomweyo, osadikirira kuti achuluke.
7:40. Admin womaliza (coordinator) adagona. Ntchito ya tsiku loyamba yatha.
8:09. Oyambitsa oyambirira, akatswiri opanga deta ndi olamulira (kuphatikizapo wogwirizanitsa watsopano) anayamba ntchito yobwezeretsa.
09:37. Tinayamba kukweza holo Nambala 7 (yomaliza).
Panthawi imodzimodziyo, tikupitiriza kubwezeretsa zomwe sizinakhazikitsidwe m'zipinda zina: kusintha ma disks / kukumbukira / ma seva, kukonza chirichonse chomwe "chowotcha" poyang'anira, kusintha maudindo mmbuyo mu ndondomeko zoyimilira ndi zinthu zina zazing'ono, zomwe zilipo. komabe kwambiri.
17:08. Timalola ntchito zonse wamba ndi kupanga.
21:45. Ntchito ya tsiku lachiwiri yatha.
09:45. Lero ndi Lachisanu. Palinso zovuta zochepa pakuwunika. Mapeto a sabata ali patsogolo, aliyense akufuna kupuma. Tikupitiriza kukonza kwambiri zonse zomwe tingathe. Ntchito zokhazikika za admin zomwe zikanayimitsidwa zidayimitsidwa. Wogwirizanitsa ndi watsopano.
15:40. Mwadzidzidzi theka la zida za netiweki za Core mu data ENA zinayambikanso. Mapiri adachotsedwa mozungulira kuti achepetse zoopsa. Palibe zotsatira kwa ogwiritsa ntchito. Kenako zinapezeka kuti inali galimoto yolakwika. Wogwirizanitsa ntchitoyo akukonzekera kukonza ngozi ziwiri nthawi imodzi.
17:17 . Kugwira ntchito pa intaneti mu malo ena a data kwabwezeretsedwa, chirichonse chafufuzidwa. Deta ya data imayikidwa mozungulira.
18:29. Ntchito ya tsiku lachitatu ndipo, kawirikawiri, kubwezeretsa pambuyo ngozi yatha.
Pambuyo pake
04.04.2013 , "Ophunzira nawo" -kwa masiku atatu chitsekocho chinali chosapezeka kwathunthu kapena pang'ono. Panthawi yonseyi, anthu opitilira 100 ochokera kumizinda yosiyanasiyana, ochokera kumakampani osiyanasiyana (zikomo kwambiri kachiwiri!), Patali komanso mwachindunji m'malo opangira ma data, pamanja komanso mwachangu, adakonza ma seva masauzande ambiri.
Tapeza mfundo. Kuti izi zisadzachitikenso, tachita ndikupitiliza kugwira ntchito yayikulu mpaka lero.
Kodi pali kusiyana kotani pakati pa ngozi yapano ndi 404?
- Tili ndi "Ndondomeko Yochita Ngozi". Kamodzi kotala, timachita masewera olimbitsa thupi - timasewera zochitika zadzidzidzi, zomwe gulu la oyang'anira (onsewo) liyenera kuthetsa pogwiritsa ntchito "Emergency Action Plan". Oyang'anira machitidwe otsogola amasinthana kukhala ogwirizanitsa.
- Kotala lililonse, poyesa, timapatula malo opangira data (onse motsatana) kudzera pamanetiweki a LAN ndi WAN, zomwe zimatilola kuzindikira zomwe zimalepheretsa mwachangu.
- Ma disks ochepa osweka, chifukwa talimbitsa miyezo: maola ochepera ogwirira ntchito, malo okhwima a SMART,
- Tidasiyiratu BerkeleyDB, nkhokwe yakale komanso yosakhazikika yomwe idafunikira nthawi yochulukirapo kuti tiyambirenso seva itayambiranso.
- Tinachepetsa kuchuluka kwa ma seva ndi MS SQL ndikuchepetsa kudalira otsalawo.
- Tili ndi zathu zathu , komwe takhala tikusuntha mautumiki onse mwachangu kwa zaka ziwiri tsopano. Mtambo umathandizira kwambiri kuzungulira kwa ntchito ndi pulogalamuyo, ndipo pakachitika ngozi umapereka zida zapadera monga:
- kuyimitsa koyenera kwa mapulogalamu onse ndikudina kamodzi;
- kusamuka kosavuta kwa mapulogalamu kuchokera ku maseva olephera;
- kusanjidwa mokhazikika (motengera kufunikira kwa ntchito) kukhazikitsidwa kwa malo onse a data.
Ngozi yomwe yafotokozedwa m’nkhaniyi inali yaikulu kwambiri kuyambira tsiku la 404. N’zoona kuti si zonse zinayenda bwino. Mwachitsanzo, pakusapezeka kwa malo owonongeka ndi moto kumalo ena a data, disk pa imodzi mwa maseva inalephera, ndiye kuti, imodzi yokha mwazojambula zitatu mumagulu a Cassandra idakhalabe yopezeka, chifukwa chake 4,2% ya mafoni. ogwiritsa ntchito sanathe kulowa. Nthawi yomweyo, ogwiritsa ntchito omwe adalumikizidwa kale adapitilizabe kugwira ntchito. Pazonse, chifukwa cha ngoziyi, mavuto oposa 30 adadziwika - kuchokera ku nsikidzi za banal kupita ku zolakwika mu zomangamanga za utumiki.
Koma kusiyana kwakukulu pakati pa ngozi yamakono ndi 404th ndikuti pamene tikuchotsa zotsatira za moto, ogwiritsa ntchito amalemberabe mameseji ndikuyimba mavidiyo. , ankasewera masewera, kumvetsera nyimbo, anapatsana mphatso, anaonera mavidiyo, masewero a pa TV ndi ma channel TV , komanso adalowa mkati .
Kodi ngozi zanu zimayenda bwanji?
Source: www.habr.com