Webinar ilibe mawu osamveka bwino, chifukwa chake tidapanga zolembedwa.
Dzina langa ndine Medvedev Eduard. Lero ndilankhula za SRE, momwe SRE idawonekera, njira zogwirira ntchito za akatswiri a SRE, pang'ono za njira zodalirika, pang'ono pakuwunika kwake. Tipita pamwamba, chifukwa simungathe kunena zambiri mu ola limodzi, koma ndikupatsani zida kuti muwunikenso, ndipo tonse tikukuyembekezerani . ku Moscow kumapeto kwa January.
Choyamba, tiyeni tikambirane za SRE - Site Reliability Engineering - ndi. Ndi momwe izo zinkawonekera ngati malo osiyana, monga njira yosiyana. Zonse zidayamba ndi mfundo yakuti m'magulu achitukuko, Dev ndi Ops ndi magulu awiri osiyana, nthawi zambiri amakhala ndi zolinga ziwiri zosiyana. Cholinga cha gulu lachitukuko ndikutulutsa zatsopano kuti zikwaniritse zosowa zamabizinesi. Cholinga cha gulu la Ops ndikuwonetsetsa kuti zonse zikuyenda bwino ndipo palibe chomwe chimasweka. Mwachiwonekere, zolingazi zimatsutsana mwachindunji: kotero kuti chirichonse chimagwira ntchito ndipo palibe chomwe chimasweka, ndi bwino kutulutsa zatsopano pang'ono momwe mungathere. Chifukwa cha izi, mikangano yambiri yamkati imayamba, yomwe njira yomwe tsopano imatchedwa DevOps ikuyesera kuthetsa.
Vuto ndilakuti tilibe tanthauzo lomveka bwino la DevOps komanso kukhazikitsa bwino kwa DevOps. Ndinalankhula pamsonkhano ku Yekaterinburg zaka 2 zapitazo, ndipo mpaka pano gawo la DevOps lidayamba ndi lipoti lakuti "DevOps ndi chiyani." Mu 2017, devops ali ndi zaka pafupifupi 10, koma tikukanganabe kuti ndi chiyani. Ndipo izi ndizovuta kwambiri zomwe Google idayesa kuthetsa zaka zingapo zapitazo.
Mu 2016, Google idatulutsa buku lotchedwa "Site Reliability Engineering." Ndipo kwenikweni, zinali ndi bukhuli pamene gulu la SRE linayamba. SRE ndi njira yapadera yogwiritsira ntchito paradigm ya DevOps mu kampani inayake. Akatswiri a SRE adadziyika okha cholinga chowonetsetsa kuti machitidwe odalirika akugwira ntchito. Amatengedwa makamaka kuchokera kwa opanga, nthawi zina kuchokera kwa olamulira omwe ali ndi chitukuko champhamvu. Ndipo amachita zomwe oyang'anira machitidwe ankachitira, koma maziko amphamvu pa chitukuko ndi chidziwitso cha dongosololi kuchokera pamawonedwe amachitidwe amatsogolera ku mfundo yakuti anthuwa sakonda kugwira ntchito yoyang'anira nthawi zonse, koma amakonda kuchita zokha.
Zikuoneka kuti paradigm ya DevOps m'magulu a SRE imayendetsedwa chifukwa pali mainjiniya a SRE omwe amathetsa zovuta zamapangidwe. Apa pali, kulumikizana komweku pakati pa Dev ndi Ops komwe anthu akhala akulankhula kwa zaka 8. Udindo wa SRE ndi wofanana ndi wa mmisiri wa zomangamanga kuti novices sakhala ma SRE. Anthu kumayambiriro kwa ntchito zawo alibe chidziwitso chilichonse ndipo alibe chidziwitso chofunikira. Chifukwa SRE imafuna chidziwitso chapamwamba kwambiri cha zomwe ndi liti zomwe zingasokonekera. Choncho, chidziwitso chamtundu wina chikufunika pano, monga lamulo, mkati mwa kampani ndi kunja.
Amafunsa ngati kusiyana pakati pa SRE ndi devops kufotokozedwa. Wangofotokozedwa kumene. Titha kulankhula za malo a SRE mu bungwe. Mosiyana ndi njira yachidule ya DevOps, pomwe Ops akadali dipatimenti yosiyana, SRE ndi gawo la gulu lachitukuko. Iwo ali nawo pa chitukuko cha mankhwala. Palinso njira yomwe SRE ndi gawo lomwe limadutsa kuchokera kwa wopanga wina kupita ku wina. Amatenga nawo mbali pazowunikira zamakhodi monga, mwachitsanzo, opanga UX, opanga okha, ndipo nthawi zina oyang'anira malonda. Ma SRE amagwira ntchito pamlingo womwewo. Tikufuna chivomerezo chawo, tifunika kuunikanso, kuti pagawo lililonse la SRE liti: "Chabwino, kutumizidwa uku, izi sizingasokoneze kudalirika. Ndipo ngati itero, idzakhala mkati mwa malire ovomerezeka.β Tikambirananso za izi.
Chifukwa chake, SRE ili ndi veto pakusintha kwamakhodi. Ndipo kawirikawiri, izi zimabweretsanso mikangano yaying'ono ngati SRE ikugwiritsidwa ntchito molakwika. M'buku lomwelo lonena za Site Reliability Engineering, magawo ambiri, ngakhale oposa, amafotokoza momwe angapewere mikangano iyi.
Anthu amafunsa momwe SRE imakhudzira chitetezo chazidziwitso. SRE sichikhudzidwa mwachindunji ndi chitetezo chazidziwitso. Makamaka m'makampani akuluakulu, izi zimachitika ndi anthu, oyesa, ndi owunika. Koma SRE imalumikizananso nawo m'lingaliro lakuti ntchito zina, zina zimachita, kutumizidwa kwina komwe kumakhudza chitetezo kungakhudzenso kupezeka kwa malonda. Chifukwa chake, SRE nthawi zambiri imalumikizana ndi magulu aliwonse, kuphatikiza magulu achitetezo, kuphatikiza owunika. Chifukwa chake, ma SRE amafunikira makamaka poyesa kukhazikitsa DevOps, koma kulemedwa kwa opanga kumakula kwambiri. Ndiko kuti, gulu lachitukuko palokha silingathenso kuthana ndi mfundo yakuti tsopano akufunikanso kukhala ndi udindo wa Ops. Ndipo gawo losiyana likuwonekera. Ntchitoyi ikukonzedwa mu bajeti. Nthawi zina udindo uwu umamangidwa mu kukula kwa gulu, munthu wosiyana amawonekera, nthawi zina amakhala mmodzi wa opanga. Umu ndi momwe SRE yoyamba imawonekera pagulu.
Kuvuta kwa dongosolo komwe kumakhudzidwa ndi SRE, zovuta zomwe zimakhudza kudalirika kwa ntchito, zingakhale zofunikira kapena mwangozi. Kuvuta kofunikira ndi pamene zovuta za chinthucho zimachuluka momwe zinthu zatsopano zimafunira. Kuvuta kwachisawawa ndi pamene zovuta za dongosolo zimawonjezeka, koma mawonekedwe a malonda ndi zofunikira zamabizinesi sizikhudza izi mwachindunji. Zikutheka kuti mwina wopangayo adalakwitsa kwinakwake, kapena ma algorithm si abwino, kapena zokonda zina zimayambitsidwa zomwe zimawonjezera zovuta za chinthucho mopanda chifukwa. SRE yabwino iyenera kupewa izi nthawi zonse. Ndiye kuti, kudzipereka kulikonse, kutumizidwa kulikonse, pempho lililonse lokoka lomwe limawonjezera zovuta chifukwa chowonjezera mwachisawawa liyenera kuletsedwa.
Funso ndiloti bwanji osangolemba ntchito injiniya, woyang'anira dongosolo ndi chidziwitso chochuluka, kuti alowe nawo gululo. Wopanga ntchito ngati mainjiniya, tikuuzidwa, si njira yabwino kwambiri yothanirana ndi anthu. Wopanga mapulogalamu omwe ali ngati injiniya sinthawi zonse omwe ndiye njira yabwino kwambiri yothanirana ndi anthu, koma mfundo apa ndikuti wopanga omwe akuchita nawo Ops ali ndi chikhumbo chochulukirapo, ali ndi chidziwitso chochulukirapo komanso luso lokhazikitsidwa kuti akwaniritse izi. zochita zokha. Ndipo moyenerera, sitichepetsa nthawi yokha ya ntchito zina zapadera, osati zochitika zokhazokha, komanso zofunikira zamalonda monga MTTR (Mean Time To Recovery, nthawi yochira). Choncho, ndipo tidzakambirananso za izi pambuyo pake, timasunga ndalama za bungwe.
Tsopano tiyeni tikambirane mfundo za SRE ntchito. Ndipo choyamba za kudalirika. M'makampani ang'onoang'ono ndi oyambitsa, nthawi zambiri zimachitika kuti anthu amaganiza kuti ngati ntchitoyo yalembedwa bwino, ngati mankhwalawa alembedwa bwino komanso molondola, adzagwira ntchito, sangasweke. Ndizomwezo, timalemba code yabwino, kotero palibe chosokoneza. Khodiyo ndi yophweka kwambiri, palibe chomwe chingaswe. Izi ndi za anthu omwewo omwe amanena kuti sitikusowa mayesero, chifukwa, taonani, awa ndi njira zitatu za VPI, bwanji mukuvutikira?
Izi zonse ndi zolakwika, ndithudi. Ndipo anthu awa nthawi zambiri amavulazidwa ndi mtundu wamtunduwu pochita, chifukwa zinthu zimasweka. Zinthu nthawi zina zimasweka m'njira zosayembekezereka. Nthawi zina anthu amakana, sizidzachitika. Ndipo zimachitikabe. Zimachitika kawirikawiri. Ndipo ndichifukwa chake palibe amene amalimbikira kupezeka kwa 100%, chifukwa kupezeka kwa 100% sikumachitika. Izi ndizokhazikika. Ndipo ndichifukwa chake nthawi zonse timalankhula za zisanu ndi zinayi tikamalankhula za kupezeka kwautumiki. 2 nine, 3 nine, 4 nine, 5 nine. Ngati timasulira izi kukhala nthawi yopuma, ndiye, mwachitsanzo, 5 nines ndi nthawi yocheperapo kuposa mphindi 5 pachaka, 2 nines ndi masiku 3,5 a nthawi yopuma.
Koma n'zoonekeratu kuti panthawi ina pali kuchepa kwa POI ndi kubwerera ku ndalama. Kuchokera pa zisanu ndi zinayi mpaka zisanu ndi zinayi kumatanthauza kuchepetsa nthawi yocheperako ndi masiku atatu. Kuchokera pa zisanu ndi zinayi mpaka zisanu kumachepetsa nthawi yopuma ndi mphindi 3 pachaka. Ndipo zikuwoneka kuti izi sizingakhale zofunikira pabizinesi. Ndipo kawirikawiri, kudalirika kofunikira si nkhani yaumisiri, choyamba, ndi nkhani yamalonda, ndi nkhani ya malonda. Ndi nthawi yanji yochepetsera yomwe imavomerezedwa kwa ogwiritsa ntchito mankhwalawa, amayembekezera chiyani, amalipira ndalama zingati, mwachitsanzo, amataya ndalama zingati, dongosololi limataya ndalama zingati.
Funso lofunika ndiloti kudalirika kwa zigawo zotsalira. Chifukwa kusiyana pakati pa 4 ndi 5 nines sikudzawoneka pa foni yamakono ndi 2 yodalirika nines. Kunena zowona, ngati china chake chasweka pa foni yam'manja muutumiki wanu ka 10 pachaka, nthawi zambiri kusweka kunachitika mbali ya OS. Wogwiritsa ntchito amagwiritsidwa ntchito pa izi, ndipo sangamvetsere nthawi imodzi yowonjezera pachaka. Ndikofunikira kufananiza mtengo wa kudalirika kowonjezereka ndikuwonjezera phindu.
M'buku la SRE muli chitsanzo chabwino chowonjezereka mpaka 4 nines kuchokera ku 3 nines. Zikuoneka kuti kuwonjezeka kupezeka ndi pang'ono zosakwana 0,1%. Ndipo ngati ndalama zautumikizo ndi $ 1 miliyoni pachaka, ndiye kuti kuwonjezeka kwa ndalama ndi $ 900. Ngati kupezeka kowonjezereka ndi zisanu ndi zinayi kumawononga ndalama zosakwana $900 pachaka, kuwonjezeka kumapangitsa ndalama. Ngati zimawononga ndalama zoposa $ 900 pachaka, sizikhalanso zomveka, chifukwa kuwonjezeka kwa ndalama sikulipiritsa ndalama zogwirira ntchito ndi ndalama zothandizira. Ndipo 3 nines adzakhala okwanira kwa ife.
Ichi ndi chitsanzo chosavuta pomwe zopempha zonse ndizofanana. Ndipo kuchokera ku 3 nines mpaka 4 nine ndizosavuta kupita, koma nthawi yomweyo, mwachitsanzo, kuchoka pa 2 nines mpaka 3 ndi ndalama zokwana madola 9, zimatha kupanga ndalama. Mwachibadwa, kwenikweni, kulephera kulembetsa pempho ndikoipa kuposa kulephera kuwonetsa tsamba; zopempha zimakhala ndi zolemera zosiyana. Atha kukhala ndi njira zosiyana kwambiri ndi bizinesi, komabe, monga lamulo, ngati sitikulankhula za mautumiki enaake, uku ndikuyerekeza kodalirika.
Tinalandira funso ngati SRE ndi m'modzi mwa ogwirizanitsa posankha njira yopangira ntchito. Izi ndizovomerezeka pokhudzana ndi kuphatikizidwa muzinthu zomwe zilipo kuti pasakhale kutaya mu kukhazikika kwake. Inde, ma SRE amakhudza zopempha zokoka, kuchita, kutulutsa mwanjira yomweyo; zimakhudza kamangidwe, kukhazikitsidwa kwa ntchito zatsopano, ma microservices, ndikukhazikitsa mayankho atsopano. Chifukwa chiyani ndidanena kale kuti mukufunika chidziwitso, muyenera ziyeneretso. M'malo mwake, SRE ndi amodzi mwamawu otsekereza pamakina aliwonse omanga ndi mapulogalamu. Choncho, SRE monga injiniya ayenera, choyamba, osati kumvetsa kokha, komanso kumvetsetsa momwe zosankha zinazake zidzakhudzire kudalirika, kukhazikika, ndi kumvetsetsa momwe izi zikugwirizanirana ndi zosowa za bizinesi, ndipo kuchokera pamalingaliro otani izi zikhoza kukhala zovomerezeka, ndi zomwe siziri.
Choncho, ino ndi nthawi yoti tikambirane za njira zodalirika, zomwe mu SRE zimatanthauzidwa kuti SLA (Service Level Agreement). Nthawi zambiri ndi mawu odziwika bwino. SLI (Service Level Indicator). SLO (Cholinga cha Utumiki wa Utumiki). Agreement Level Level Agreement mwina ndi nthawi yofunikira, makamaka ngati mwagwira ntchito ndi ma network, othandizira, komanso kuchititsa. Ichi ndi mgwirizano wamba womwe umafotokoza momwe ntchito yanu yonse ikugwiritsidwira ntchito, zilango, zilango zina zolakwa, ma metrics, njira. Ndipo SLI ndiye metric yopezeka yokha. Ndiko kuti, zomwe SLI ikhoza kukhala: nthawi yoyankha kuchokera ku utumiki, chiwerengero cha zolakwika monga peresenti. Izi zitha kukhala bandwidth ngati tikukamba za mtundu wina wa kuchititsa mafayilo. Ngati tikulankhula za ma aligorivimu kuzindikira, chizindikiro akhoza ngakhale, mwachitsanzo, kulondola kwa yankho. SLO (Cholinga cha Utumiki wa Utumiki) ndi, motero, kuphatikiza kwa chizindikiro cha SLI, mtengo wake ndi nthawi.
Tinene kuti SLA ikhoza kukhala chonchi. Ntchitoyi imapezeka 99,95% ya nthawi yonse chaka chonse. Kapena matikiti 99 othandizira ukadaulo adzatsekedwa mkati mwa maola atatu pa kotala. Kapena 3% ya mafunso adzayankhidwa mkati mwa masekondi 85 mwezi uliwonse. Ndiye kuti, pang'onopang'ono tikufika pomvetsetsa kuti zolakwika ndi zolephera ndizabwinobwino. Izi ndizochitika zovomerezeka, tikukonzekera, tikudalira pamlingo wina. Ndiko kuti, SRE imapanga machitidwe omwe amatha kulakwitsa, omwe ayenera kuyankha kawirikawiri ku zolakwika, ndipo zomwe ziyenera kuziganizira. Ndipo ngati n'kotheka, ayenera kuthana ndi zolakwika m'njira yoti wogwiritsa ntchito asawazindikire, kapena kuwazindikira, koma pali njira ina yogwirira ntchito kuti zonse zisawonongeke.
Mwachitsanzo, ngati mutsitsa kanema ku YouTube, ndipo YouTube siyingasinthe nthawi yomweyo, ngati kanemayo ndi yayikulu kwambiri, ngati mawonekedwewo sali bwino, ndiye kuti pempholo silingalephereke ndi nthawi, YouTube siwonetsa 502 cholakwika, YouTube iti: "Tidapanga chilichonse, kanema wanu akukonzedwa. Ikhala yokonzeka pakangotha ββmphindi 10.β Iyi ndi mfundo ya kunyozeka kwachisomo, yomwe imadziwika bwino, mwachitsanzo, kuchokera ku chitukuko chakutsogolo ngati munachitapo izi.
Mawu otsatirawa omwe tidzakambirana, omwe ndi ofunika kwambiri pogwira ntchito modalirika, ndi zolakwika, ndi zoyembekeza, ndi MTBF ndi MTTR. MTBF ndi nthawi yapakati pakati pa zolephera. MTTR Ikutanthauza Nthawi Yochira, nthawi yapakati kuti muchiritse. Ndiye kuti, ndi nthawi yochuluka bwanji yomwe idadutsa kuchokera pomwe cholakwikacho chidadziwika, kuyambira pomwe cholakwikacho chidawonekera mpaka pomwe ntchitoyo idabwezeretsedwanso kuti igwire bwino ntchito. MTBF imakonzedwa makamaka pogwira ntchito pama code. Ndiye kuti, ma SRE amatha kunena kuti "ayi". Ndipo gulu lonse liyenera kumvetsetsa kuti pamene SRE imati "ayi," sakunena kuti ndi wovulaza, osati chifukwa chakuti ndi woipa, koma chifukwa chakuti wina aliyense adzavutika.
Apanso, pali zolemba zambiri, njira zambiri, njira zambiri, ngakhale m'buku lomwe ndimatchula nthawi zambiri, momwe mungatsimikizire kuti otukula ena sayamba kudana ndi SRE. MTTR, kumbali ina, ili pafupi kugwira ntchito pa SLO yanu (Cholinga cha Utumiki Wanu). Ndipo izi ndizochita zokha. Chifukwa, mwachitsanzo, SLO yathu ndi nthawi yowonjezera ya 4 nines pa kotala. Izi zikutanthauza kuti m'miyezi itatu tikhoza kulola mphindi 3 zakupuma. Ndipo zidapezeka kuti MTTR yathu singakhale yopitilira mphindi 13. Ngati titenga mphindi 13 kuti tichite nthawi yochepera 13, izi zikutanthauza kuti tatopa kale bajeti yonse ya kotala. Tikuphwanya SLO. Mphindi 1 kuti muchite ndikuwongolera kulephera ndizovuta pamakina, koma zochepa kwambiri kwa munthu. Chifukwa panthawi yomwe munthu amalandira chenjezo, panthawi yomwe amachitapo kanthu, pofika pozindikira zolakwika, zimakhala kale mphindi zochepa. Mpaka munthu atamvetsetsa momwe angakonzere, zomwe angakonze, zoyenera kuchita, zidzatenga mphindi zingapo. Ndipo kwenikweni, ngakhale mutangofunika kuyambiranso seva, momwe zimakhalira, kapena kukweza mfundo yatsopano, ndiye kuti MTTR imatenga pafupifupi mphindi 13-7. Mukamapanga njira, MTTR nthawi zambiri imafika pa sekondi imodzi, nthawi zina ma milliseconds. Google nthawi zambiri imalankhula za milliseconds, koma zenizeni, zinthu sizili bwino.
Moyenera, SRE iyenera pafupifupi kusinthiratu ntchito yake, chifukwa izi zimakhudza mwachindunji MTTR, ma metrics ake, SLO yautumiki wonse, ndipo, molingana ndi phindu la bizinesi. Ngati nthawi yadutsa, timafunsidwa ngati mlandu uli ndi SRE. Mwamwayi, mlandu suikidwiratu aliyense. Ndipo ichi ndi chikhalidwe chosiyana, chomwe chimatchedwa balmeless postmortem, chomwe sitilankhula lero, koma tidzasanthula pa Slurm. Uwu ndi mutu wosangalatsa kwambiri womwe ungakambidwe kwambiri. Mwachidule, ngati nthawi yoperekedwa pa kotala yadutsa, ndiye kuti aliyense ali ndi mlandu pang'ono, zomwe zikutanthauza kuti kudzudzula aliyense sikupindulitsa, m'malo mwake, mwina, tisamadzudzule aliyense, koma konzani zochitikazo ndikugwira ntchito ndi zomwe tili nazo. Muzochitika zanga, njira iyi ndi yachilendo kwa magulu ambiri, makamaka ku Russia, koma ndizomveka komanso zimagwira ntchito bwino. Chifukwa chake, pamapeto ndikupangira zolemba ndi zolemba zomwe mungawerenge pamutuwu. Kapena bwerani ku Slurm SRE.
Ndiloleni ndifotokoze. Ngati nthawi ya SLO pa kotala idutsa, ngati nthawi yopuma sinali mphindi 13, koma 15, ndani angakhale ndi mlandu pa izi? Zachidziwikire, SRE ikhoza kukhala yolakwa chifukwa idapanga zolakwa zina kapena kutumizidwa. Woyang'anira malo opangira data atha kukhala ndi mlandu pa izi, chifukwa mwina adakonza zosakonzekera. Ngati woyang'anira malo a data ali ndi mlandu pa izi, ndiye kuti munthu wochokera ku Ops nayenso ali ndi mlandu chifukwa chosawerengera kukonza povomereza SLO. Ichi ndi cholakwika cha manejala, wotsogolera zaukadaulo, kapena wina yemwe adasaina mgwirizano wa data center ndipo sanamvere kuti data center SLA sinapangidwe kuti ikhale yofunikira. Chifukwa chake, aliyense ali ndi mlandu pang'ono pazimenezi. Ndipo izi zikutanthauza kuti palibe chifukwa choimbira mlandu wina aliyense makamaka pazochitika izi. Koma ndithudi ziyenera kukonzedwa. Ndicho chifukwa postmortems alipo. Ndipo ngati muwerenga, mwachitsanzo, GitHub postmortems, ndipo iyi nthawi zonse imakhala nkhani yosangalatsa, yaying'ono komanso yosayembekezereka pazochitika zilizonse, mutha kusintha kuti palibe amene anganene kuti munthu uyu ndiye anali ndi mlandu. Mlandu nthawi zonse umayikidwa pazinthu zina zoperewera.
Tiyeni tipitirire ku funso lotsatira. Zochita zokha. Nthawi zambiri, ndikakamba za automation muzinthu zina, nthawi zambiri ndimayang'ana patebulo lomwe limakamba za nthawi yayitali bwanji yogwirira ntchito kuti musatenge nthawi yochulukirapo kuposa momwe mumasungira. Pali kugwira. Chomwe chimagwira ndichakuti ma SRE akamayendetsa ntchito, samangosunga nthawi, amasunga ndalama chifukwa makina amakhudza mwachindunji MTTR. Amapulumutsa, titero, chikhalidwe cha ogwira ntchito ndi omanga, chomwe ndi chida chotha. Amachepetsa chizolowezi. Ndipo zonsezi zimakhala ndi zotsatira zabwino pa ntchito ndipo, chifukwa chake, pa bizinesi, ngakhale zikuwoneka kuti zodzikongoletsera sizimveka bwino pamtengo wa nthawi.
M'malo mwake, zimatero nthawi zonse, ndipo pali zochitika zochepa zomwe sizoyenera kupanga china chake mu gawo la SRE. Kenako tikambirana zomwe zimatchedwa zolakwika bajeti, bajeti ya zolakwika. M'malo mwake, zikuwoneka kuti ngati mukuchita bwino kwambiri kuposa SLO yomwe mudadzipangira nokha, izi sizabwino kwambiri. Izi ndizoyipa kwambiri, chifukwa SLO imagwira ntchito osati ngati yocheperako, komanso ngati yongoyerekeza kumtunda. Mukadziyika nokha SLO ya 99% kupezeka, ndipo muli ndi 99,99%, zimakhala kuti muli ndi malo oyesera, omwe sangawononge bizinesi konse, chifukwa inu nokha mwatsimikiza izi palimodzi, ndipo inu. khalani ndi malowa musagwiritse ntchito. Muli ndi bajeti ya zolakwika, zomwe kwa inu sizimagwiritsidwa ntchito.
Kodi tikuchita nawo chiyani? Timachigwiritsa ntchito kwenikweni pa chilichonse. Pakuyesa m'mikhalidwe yopangira, potulutsa zatsopano zomwe zingakhudze magwiridwe antchito, zotulutsa, zokonza, zochepetsera zokonzekera. Lamulo losiyana limagwiranso ntchito: ngati bajeti yatha, sitingathe kumasula chirichonse chatsopano, chifukwa mwinamwake tidzapitirira SLO. Bajeti yatha kale, tatulutsa china chake, ngati chimasokoneza magwiridwe antchito, ndiye kuti, ngati sikuli koyenera komwe kumawonjezera mwachindunji SLO, ndiye kuti tikupitilira bajeti, ndipo izi ndizovuta. , pamafunika kusanthula , postmortem, ndipo mwina kukonza ndondomeko.
Ndiko kuti, zikuwoneka kuti ngati ntchitoyo siyikuyenda bwino, ndipo SLO imagwiritsidwa ntchito ndipo bajeti imagwiritsidwa ntchito osati pazoyeserera, osati pazotulutsa zilizonse, koma pazokha, ndiye m'malo mwazokonza zosangalatsa, m'malo mosangalatsa. mawonekedwe, m'malo motulutsa zosangalatsa. M'malo mopanga ntchito iliyonse yolenga, muyenera kukonza zosayankhula kuti mubwezeretse bajeti, kapena kusintha SLO, ndipo iyi ndi njira yomwe siyenera kuchitika nthawi zambiri.
Chifukwa chake, zikuwoneka kuti munthawi yomwe tili ndi ndalama zambiri zolakwa, aliyense ali ndi chidwi: onse a SRE ndi opanga. Kwa Madivelopa, bajeti yayikulu ya zolakwika zikutanthauza kuti atha kuthana ndi zotulutsa, zoyeserera, ndi zoyeserera. Kwa ma SRE, bajeti ya zolakwika ndikulowa mu bajeti iyi zikutanthauza kuti akuchita ntchito yabwino. Ndipo izi zimakhudza chilimbikitso cha mtundu wina wa ntchito yolumikizana. Ngati mumvera ma SRE anu ngati omanga, mudzakhala ndi malo ochulukirapo ochitira ntchito zabwino komanso ntchito zochepa.
Zikuoneka kuti zoyeserera pakupanga ndizofunikira kwambiri komanso gawo lofunikira la SRE m'magulu akulu. Ndipo nthawi zambiri imatchedwa chaos engineering, yomwe imachokera ku gulu la Netflix lomwe limatulutsa chida chotchedwa Chaos Monkey.
Chisokonezo Monkey imalumikizana ndi payipi ya CI/CD ndikusokoneza seva popanga. Apanso, mu dongosolo la SRE timanena kuti seva yowonongeka siili yoipa yokha, ikuyembekezeka. Ndipo ngati ikuphatikizidwa mu bajeti, ndizovomerezeka ndipo sizikuvulaza bizinesi. Zachidziwikire, Netflix ili ndi ma seva owonjezera okwanira, kubwereza kokwanira, kuti zonsezi zitha kukhazikitsidwa popanda wogwiritsa ntchito yonse ngakhale kuzindikira, ndipo palibe amene amasiya seva imodzi pa bajeti iliyonse.
Netflix nthawi ina inali ndi zida zambiri zotere, imodzi mwazo, Chaos Gorilla, imayimitsa imodzi mwa magawo omwe amapezeka ku Amazon. Ndipo zinthu zotere zimathandiza kuzindikira, choyamba, kudalira kobisika, pamene sizikudziwika bwino zomwe zimakhudza chiyani, zimadalira chiyani. Ndipo izi, ngati mukugwira ntchito ndi microservice ndipo zolemba sizili zangwiro, izi zitha kukhala zodziwika kwa inu. Ndipo kachiwiri, izi zimathandiza kuti mugwire zolakwika mu code yomwe simungathe kuigwira panthawi yamasewero, chifukwa siteji iliyonse si yolondola, chifukwa chakuti kuchuluka kwa katundu ndi kosiyana, kachitidwe ka katundu ndi kosiyana, zipangizo ndizo, zambiri. mwina, zina. Zokwera kwambiri zimathanso kukhala zosayembekezereka komanso zosayembekezereka. Ndipo kuyesa kotereku, komwe sikudutsanso bajeti, kumathandizira kuti agwire zolakwika pamapangidwe omwe masitepe, ma autotests, ndi mapaipi a CI / CD sangagwire. Ndipo malinga ngati zonsezi zikuphatikizidwa mu bajeti yanu, ziribe kanthu kuti utumiki wanu watsikira pamenepo, ngakhale kuti zingawoneke zowopsya kwambiri, seva yawonongeka, ndizovuta bwanji. Ayi, ndizabwinobwino, zimathandizira kugwira zolakwika. Ngati muli ndi bajeti, mutha kuzigwiritsa ntchito.
Funso: Ndi mabuku ati omwe ndingapangire? Mndandanda uli kumapeto. Pali mabuku ambiri, ndingapangire malipoti angapo. Momwe zimagwirira ntchito komanso ngati SRE imagwira ntchito m'makampani opanda pulogalamu yawoyawo kapena osatukuka pang'ono. Mwachitsanzo, mu bizinesi, pomwe ntchito yayikulu si mapulogalamu. M'bizinesi, pomwe ntchito yayikulu si pulogalamu, SRE imagwira ntchito mofanana ndi kwina kulikonse, chifukwa mubizinesi muyeneranso kugwiritsa ntchito, ngakhale simupanga, mapulogalamu apulogalamu, muyenera kutulutsa zosintha, muyenera kusintha zomangamanga, muyenera kukula, muyenera kukula. Ndipo ma SRE amathandizira kuzindikira ndikudziwiratu zovuta zomwe zingachitike m'njirazi ndikuwongolera pakayamba kukula ndikusintha zosowa zabizinesi. Chifukwa sikofunikira kwenikweni kuchita nawo chitukuko cha mapulogalamu kuti mukhale ndi SRE, ngati muli ndi ma seva angapo ndipo mukuyembekeza kukula pang'ono.
Zomwezo zimapitanso kumapulojekiti ang'onoang'ono, mabungwe ang'onoang'ono, chifukwa makampani akuluakulu ali ndi bajeti ndi malo oyesera. Koma nthawi yomweyo, zipatso zonsezi zoyesera zitha kugwiritsidwa ntchito kulikonse, ndiko kuti, ma SRE, ndithudi, adawonekera mu Google, Netflix, ndi Dropbox. Koma nthawi yomweyo, makampani ang'onoang'ono ndi oyambitsa amatha kuwerenga kale zinthu zofupikitsidwa, kuwerenga mabuku, ndi malipoti owonera. Amayamba kumva za izi pafupipafupi, yang'anani zitsanzo zenizeni, ndikuganiza, chabwino, izi zitha kukhala zothandiza, timafunikiranso izi, zabwino.
Ndiko kuti, ntchito yonse yayikulu pakuyimilira njirazi idakuchitikirani kale. Zomwe muyenera kuchita ndikutanthauzira gawo la SRE makamaka pakampani yanu ndikuyamba kukhazikitsa machitidwe onsewa, omwe afotokozedwa kale. Ndiko kuti, kuchokera ku mfundo zothandiza kwa makampani ang'onoang'ono, izi nthawi zonse ndi tanthauzo la SLA, SLI, SLO. Ngati simukuchita nawo mapulogalamu, ndiye kuti izi zidzakhala ma SLA amkati ndi ma SLO amkati, bajeti yamkati ya zolakwika. Izi nthawi zambiri zimatsogolera ku zokambirana zosangalatsa mkati mwa gulu komanso mubizinesi, chifukwa zitha kuwoneka kuti mukugwiritsa ntchito ndalama zochulukirapo kuposa zofunikira pakumanga, pamtundu wina wazinthu zabwino, mapaipi abwino. Ndipo zisanu ndi zinayi izi zomwe muli nazo mu dipatimenti ya IT, simukuzifuna tsopano. Koma panthawi imodzimodziyo, zinali zotheka kugwiritsa ntchito nthawi, kugwiritsa ntchito bajeti ya zolakwika pa chinthu china.
Chifukwa chake, kuyang'anira ndi kulinganiza zowunikira ndizothandiza pakampani yamtundu uliwonse. Ndipo kawirikawiri, njira iyi yoganizira, pamene zolakwa zimakhala zovomerezeka, kumene kuli bajeti, kumene Zolinga zilipo, ndizothandizanso kwa kampani ya kukula kulikonse, kuyambira pachiyambi cha anthu atatu.
Chomaliza mwazinthu zamakono zomwe tingakambirane ndikuwunika. Chifukwa ngati tilankhula za SLA, SLI, SLO, sitingamvetse popanda kuyang'anitsitsa ngati tikugwirizana ndi bajeti, kaya tikugwirizana ndi Zolinga zathu, ndi momwe timakhudzira SLA yomaliza. Ndawonapo nthawi zambiri kuti kuyang'anira kumachitika motere: pali phindu lina, mwachitsanzo, nthawi ya pempho kwa seva, nthawi yapakati kapena chiwerengero cha zopempha ku database. Ali ndi muyezo wotsimikiziridwa ndi injiniya. Ngati metric ikuchoka pamwambo, imelo imatumizidwa. Zonsezi ndizopanda pake, monga lamulo, chifukwa zimatsogolera ku kuchulukirachulukira kwa machenjezo, kuwonjezereka kwa mauthenga owunikira, pamene munthu, choyamba, ayenera kuwatanthauzira nthawi zonse, ndiko kuti, kudziwa ngati mtengo wa metric umatanthauza kufunika kwa mtundu wina wa zochita. Ndipo chachiwiri, amangosiya kuzindikira zidziwitso zonsezi, pomwe palibe chomwe chikufunika kuchokera kwa iye. Ndiko kuti, lamulo loyang'anira bwino komanso lamulo loyamba pokhazikitsa SRE ndikuti chidziwitso chiyenera kubwera pokhapokha ngati pakufunika kuchitapo kanthu.
Munthawi yokhazikika pali magawo atatu a zochitika. Pali zidziwitso, pali matikiti, pali zipika. Zidziwitso ndi chilichonse chomwe chimafuna kuchitapo kanthu mwachangu kuchokera kwa inu. Ndiko kuti, chirichonse chasweka, chiyenera kukonzedwa pompano. Matikiti ndi chinthu chomwe chimafunikira kudikirira kuchitapo kanthu. Inde, muyenera kuchita chinachake, muyenera kuchita chinachake pamanja, zokha zalephera, koma simukuyenera kuchita mu mphindi zingapo zotsatira. Mitengo ndi chilichonse chomwe sichifuna kuchitapo kanthu, ndipo nthawi zambiri, ngati zinthu zikuyenda bwino, palibe amene angawerenge. Zidzakhala zofunikira kuwerenga zipikazo pokhapokha, poyang'ana kumbuyo, zikuwoneka kuti chinachake chinasweka kwa nthawi ndithu, sitinadziwe za izo. Kapena kufufuza kwina kuyenera kuchitika. Koma kawirikawiri, zonse zomwe sizifuna kuchitapo kanthu zimapita ku zipika.
Monga chotsatira cha zonsezi, ngati tazindikira kuti ndizochitika ziti zomwe zimafunikira kuchitapo kanthu ndipo tafotokoza bwino zomwe ziyenera kukhala, izi zikutanthauza kuti zochitazo zitha kukhala zokha. Ndiko kuti, zomwe zimachitika. Tikuchokera ku chenjezo. Tiyeni tichitepo kanthu. Tiyeni tipite ku malongosoledwe a izi. Kenako timapita ku automation. Ndiye kuti, makina aliwonse amayamba ndi zomwe zimachitika.
Kuchokera pakuwunika timapita ku mawu otchedwa Observability. Pakhalanso hype pang'ono kuzungulira mawu awa kwa zaka zingapo zapitazi. Ndipo ndi anthu ochepa amene amamvetsa tanthauzo la zimenezi. Koma mfundo yayikulu ndiyakuti Observability ndi metric of transparency system. Ngati china chake chalakwika, mungadziwe mwachangu bwanji zomwe zidalakwika komanso momwe dongosololi linalili panthawiyo. Kuchokera pamawonekedwe a code: ndi ntchito iti yomwe inalephera, yomwe inalephera. Zomwe zinali, mwachitsanzo, zosintha zamkati, kasinthidwe. Kuchokera pamawonekedwe a zomangamanga, apa ndi momwe kupezeka kwa malo kulephera kunachitika, ndipo ngati muli ndi mtundu wina wa Kubernetes, ndiye kuti kulephera kunachitika, komwe kunali kotani. Ndipo moyenerera, Observability ali ndi ubale wachindunji ndi MTTR. Kuwoneka kwapamwamba kwa ntchitoyo, ndikosavuta kuzindikira cholakwika, ndikosavuta kukonza cholakwikacho, ndikosavuta kusinthira cholakwikacho, kutsitsa MTTR.
Ngati tipitanso kumakampani ang'onoang'ono, nthawi zambiri amafunsa, ngakhale tsopano, chochita ndi kukula kwa gulu, komanso ngati kuli kofunikira kubwereka SRE yosiyana mu gulu laling'ono. Ndinalankhula kale za izi posachedwa. M'magawo oyamba a chitukuko cha oyambitsa kapena, mwachitsanzo, gulu, izi sizofunikira konse, chifukwa SRE ikhoza kupangidwa kukhala gawo losinthira. Ndipo izi zipangitsa gululo pang'ono, chifukwa pali mitundu ingapo. Ndipo kuphatikiza izo zidzakonzekeretsa anthu kuti ndi kukula, makamaka, maudindo a SRE adzasintha kwambiri. Ngati mumalemba ntchito munthu, ndiye kuti ali ndi ziyembekezo zina. Ndipo ziyembekezozi sizidzasintha pakapita nthawi, koma zofunikira zidzasintha kwambiri. Chifukwa chake, kulemba ntchito SRE kumakhala kovuta koyambirira. Ndikosavuta kukweza nokha. Koma ndi bwino kuganizira.
Chokhacho, mwinamwake, ndi pamene pali zofunikira kwambiri komanso zodziwika bwino za kutalika. Ndiko kuti, pankhani yoyambira, izi zitha kukhala zovuta zamtundu wina kuchokera kwa osunga ndalama, zolosera zamtundu wina wakukula kangapo nthawi imodzi. Ndiye kubwereka SRE nthawi zambiri kumakhala koyenera chifukwa kumatha kulungamitsidwa. Tili ndi zofunikira za kukula, timafunikira munthu yemwe adzakhale ndi udindo wowonetsetsa kuti palibe chomwe chikuphwanyidwa ndi kukula koteroko.
Funso linanso. Zoyenera kuchita ngati otukula kangapo amadula chinthu chomwe chimayesa mayeso, koma kuswa malonda, kunyamula nkhokwe, kuswa zinthu zina, njira yoti agwiritse ntchito. Chifukwa chake, mu nkhani iyi, bajeti ya zolakwika imayambitsidwa. Ndipo ntchito zina, zina zimayesedwa nthawi yomweyo popanga. Izi zitha kukhala canary, pomwe owerengeka ochepa chabe, koma akupanga kale, akugwiritsa ntchito mawonekedwe, koma ndikuyembekeza kuti ngati china chake chasweka, mwachitsanzo, kwa theka la anthu onse ogwiritsa ntchito, chidzakwanirabe mkati mwa bajeti ya zolakwika. Chifukwa chake, inde, padzakhala cholakwika, kwa ogwiritsa ntchito ena chilichonse chidzasweka, koma tanena kale kuti izi ndizabwinobwino.
Panali funso lokhudza zida za SRE. Ndiye kuti, pali china chake chomwe ma SRE angachigwiritse ntchito chomwe wina aliyense sangachite? M'malo mwake, pali zida zapadera kwambiri, pali mapulogalamu ena omwe, mwachitsanzo, amatengera katundu kapena kuyesa kwa canary A/B. Koma kwenikweni, zida za SRE ndizomwe opanga anu akugwiritsa ntchito kale. Chifukwa SRE imalumikizana mwachindunji ndi gulu lachitukuko. Ndipo ngati muli ndi zida zosiyanasiyana, zimatengera nthawi kuti kulunzanitsa. Makamaka ngati ma SRE akugwira ntchito m'magulu akuluakulu, m'makampani akuluakulu komwe pangakhale magulu angapo, kukhazikitsidwa kwa kampani kudzakhala kothandiza kwambiri pano, chifukwa ngati magulu a 50 amagwiritsa ntchito 50 zosiyana, izi zikutanthauza kuti SRE iyenera kuwadziwa onse. Ndipo ndithudi izi sizidzachitika konse. Ndipo khalidwe la ntchito, khalidwe la kulamulira osachepera ena mwa magulu adzachepa kwambiri.
Webinar yathu ikutha pang'onopang'ono. Ndinakwanitsa kukuuzani zinthu zina zofunika. Zachidziwikire, palibe chokhudza SRE chomwe chingawuzidwe ndikumvetsetsa mu ola limodzi. Koma ndikuyembekeza kuti ndinatha kufotokoza mwanjira imeneyi, mfundo zazikuluzikulu. Ndiyeno, ngati mukufuna, mutha kuzama mozama pamutuwu, kuphunzira nokha, ndikuwona momwe ikugwiritsidwira ntchito ndi anthu ena, m'makampani ena. Ndipo motero, koyambirira kwa February, bwerani kwa ife ku Slurm SRE.
Slurm SRE ndi maphunziro amasiku atatu omwe afotokoza pafupifupi zomwe ndikunena pano, koma mozama kwambiri, ndi zochitika zenizeni, ndikuchita, zonsezo zimangogwira ntchito yothandiza. Anthu adzagawidwa m'magulu. Nonse mudzakhala mukugwira ntchito pazochitika zenizeni. Chifukwa chake, tili ndi aphunzitsi ochokera ku Booking.com Ivan Kruglov ndi Ben Tyler. Tili ndi Evgeniy Varabbas wodabwitsa wochokera ku Google, wochokera ku San Francisco. Ndipo inenso ndikuuzeni inu chinachake. Choncho onetsetsani kuti mwabwera kudzatichezera.
Choncho, mndandanda wa maumboni. Pali maulalo pa SRE. pa bukhu lomwelo, kapena m'mabuku a 2 onena za SRE, olembedwa ndi Google. Winanso , kumene mawu ndi ntchito zawo akufotokozedwa mwatsatanetsatane. 3 yotsatira ndi malipoti a SRE m'makampani osiyanasiyana. Choyamba - , iyi ndi mfundo yofunika kwambiri kuchokera kwa Ben Trainer kuchokera ku Google. Chachiwiri - . Chachitatu ndi cha . Lipoti lachinayi kuchokera , yomwe ili ndi antchito ofunikira 5 okha a SRE m'maiko 190. Ndizosangalatsa kwambiri kuyang'ana zonsezi, chifukwa monga momwe DevOps amatanthawuza zinthu zosiyana kwambiri ndi makampani osiyanasiyana komanso ngakhale magulu osiyanasiyana, SRE ili ndi maudindo osiyana kwambiri, ngakhale m'makampani a kukula kwake.
2 maulalo enanso pa mfundo za engineering chaos: , . Ndipo pamapeto pake pali mindandanda ya 3 kuchokera mndandanda wa Awesome Lists , za ndi za . Mndandanda wa SRE ndi waukulu kwambiri, simuyenera kudutsamo, pali pafupifupi 200 zolemba. Ndikupangira kwambiri zolemba zokhudzana ndi kukonza luso komanso postmortem yopanda cholakwika.
Nkhani yosangalatsa:
Zikomo pondimvera nthawi yonseyi. Ndikukhulupirira kuti mwaphunzirapo kanthu. Ndikukhulupirira kuti muli ndi zida zokwanira kuti muphunzire zambiri. Ndipo tidzawonana pambuyo pake. Tikukhulupirira mu February.
Webinar anali ndi Eduard Medvedev.
PS: Kwa iwo omwe amakonda kuwerenga, Eduard adapereka mndandanda wamawu. Iwo amene amakonda kumvetsa izo mu kuchita ndi olandiridwa pa .
Source: www.habr.com