Ubunjineli be-Chaos: ubugcisa bokutshatyalaliswa ngabom. Icandelo lesi-2

Phawula. transl.: Eli nqaku liqhubeka noluhlu olukhulu lwamanqaku avela kumshumayeli we-AWS weteknoloji u-Adrian Hornsby, obeka ukuchaza ngendlela elula necacileyo ukubaluleka kovavanyo lokunciphisa imiphumo yokungaphumeleli kwiinkqubo ze-IT.

Ubunjineli be-Chaos: ubugcisa bokutshatyalaliswa ngabom. Icandelo lesi-2

"Ukuba uyasilela ukulungiselela isicwangciso, ngoko uceba ukusilela." - Benjamin Franklin

В Inxalenye yokuqala Kolu ngcelele lwamanqaku, ndazisa ingqikelelo yobunjineli besiphithiphithi kwaye ndachaza ukuba kunceda njani ukufumana kunye nokulungisa iziphene kwinkqubo ngaphambi kokuba zikhokelele ekusileleni kwemveliso. Iphinde yaxoxa ngendlela ubunjineli besiphithiphithi obukhuthaza ngayo utshintsho oluhle lwenkcubeko phakathi kwemibutho.

Ekupheleni kwecandelo lokuqala, ndithembise ukuthetha "ngezixhobo kunye neendlela zokwazisa ukusilela kwiinkqubo." Yeha, intloko yam yayinezicwangciso zayo kulo mba, kwaye kweli nqaku ndiza kuzama ukuphendula umbuzo odumileyo ovela phakathi kwabantu abafuna ukungena kubunjineli besiphithiphithi: Yintoni yokuqhawula kuqala?

Umbuzo omkhulu! Nangona kunjalo, akabonakali ekhathazwa yile panda ...

Ubunjineli be-Chaos: ubugcisa bokutshatyalaliswa ngabom. Icandelo lesi-2
Sukuphoxana ne-chaos panda!

Impendulo emfutshane: Jonga iinkonzo ezibalulekileyo ecaleni kwendlela yesicelo.

Impendulo ende kodwa ecacileyo: Ukuqonda ukuba ungaqalisa phi ukuzama isiphithiphithi, nikela ingqalelo kwiindawo ezintathu:

  1. Jonga kwi imbali yokuwa kunye nokuchonga iipateni;
  2. Thatha isigqibo ukuxhomekeka okubalulekileyo;
  3. Sebenzisa okubizwa ukuzithemba ngokugqithiseleyo.

Kuyahlekisa, kodwa le nxalenye inokubizwa ngokulula "Uhambo lokuZifumanisa kunye nokuKhanya". Kuyo siya kuqalisa "ukudlala" ngezixhobo ezithile ezipholileyo.

1. Impendulo ikwixa elidlulileyo

Ukuba uyakhumbula, kwicandelo lokuqala ndazisa ingcamango yoLungiso-kweempazamo (COE) - indlela esihlalutya ngayo iimpazamo zethu - iimpazamo kwiteknoloji, inkqubo okanye umbutho - ukuze siqonde isizathu (s) kunye nokuthintela ukuvela kwakhona kwixesha elizayo . Ngokubanzi, kulapho kufuneka uqale khona.

"Ukuze uqonde ngoku, kufuneka wazi ixesha elidlulileyo." - UCarl Sagan

Jonga kwimbali yokungaphumeleli, uthege kwi-COE okanye kwi-postmortems kwaye uhlele. Chonga iipateni eziqhelekileyo ezihlala zikhokelela kwiingxaki, kwaye kwi-COE nganye, zibuze lo mbuzo ulandelayo:

"Ngaba oku bekunokuxelwa kwangaphambili kwaye kuthintelwe ngesitofu esinempazamo?"

Ndikhumbula ukusilela kwam ekuqaleni komsebenzi wam. Ibinokuthintelwa ngokulula ukuba besiqhube imifuniselo embalwa yesiphithiphithi:

Ngaphantsi kweemeko eziqhelekileyo, iimeko zokubuyela umva ziphendula kwiisheke zezempilo ezivela isilinganisi somthwalo (ELB)). I-ELB isebenzisa ezi tshekhi ukuqondisa izicelo kwakhona kwiimeko eziphilileyo. Xa kuvela ukuba umzekelo "awunampilo", i-ELB iyayeka ukuthumela izicelo kuyo. Ngolunye usuku, emva komkhankaso wokuthengisa oyimpumelelo, umthamo wezithuthi unyuke kwaye i-backends yaqala ukuphendula kwiisheke zezempilo ngokuthe ngcembe kunesiqhelo. Kufuneka kuthiwe ezi zitshekisho lwempilo zazinjalo nzulu, oko kukuthi, imeko yokuxhomekeka ihlolwe.

Nangona kunjalo, yonke into yayihamba kakuhle okwethutyana.

Ke, sele iphantsi kweemeko ezicinezelayo, enye yeemeko yaqala ukwenza umsebenzi ongabalulekanga, oqhelekileyo we-ETL cron. Indibaniselwano yetrafikhi ephezulu kunye necronjob ityhale ukusetyenziswa kwe-CPU phantse kwi-100%. Ukugqithiswa kwe-CPU kuqhubele phambili ukucothisa iimpendulo ekuhlolweni kwezempilo, kangangokuba i-ELB yagqiba ekubeni lo mzekelo ujongene neengxaki zokusebenza. Njengoko bekulindelekile, umlinganisi wayeka ukusabalalisa i-traffic kuyo, okwathi, kwakhokelela ekunyuseni komthwalo kwiimeko eziseleyo kwiqela.

Ngequbuliso, zonke ezinye iimeko nazo zaqala ukungaphumeleli ukukhangela impilo.

Ukuqala umzekelo omtsha ofunekayo ukukhuphela kunye nokufaka iipakethi kwaye kuthathe ixesha elide kunokuba kuthathe i-ELB ukuyikhubaza - nganye nganye - kwiqela le-autoscaling. Kucacile ukuba kungekudala yonke inkqubo yafikelela kwinqanaba elibalulekileyo kwaye isicelo saphuka.

Emva koko saziqonda ngonaphakade ezi ngongoma zilandelayo:

  • Ukufakela isofthiwe xa udala umzekelo omtsha kuthatha ixesha elide; Golden AMI.
  • Kwiimeko ezinzima, iimpendulo kwiisheke zezempilo kunye ne-ELB kufuneka zithathe indawo yokuqala - into yokugqibela oyifunayo kukubunzima ubomi kwiimeko eziseleyo.
  • I-caching yendawo yokutshekisha impilo inceda kakhulu (nokuba imizuzwana embalwa).
  • Kwimeko enzima, musa ukuqhuba imisebenzi ye-cron kunye nezinye iinkqubo ezingabalulekanga - gcina izibonelelo kwimisebenzi ebaluleke kakhulu.
  • Xa i-autoscaling, sebenzisa iimeko ezincinci. Iqela leesampuli ezincinci ezili-10 zingcono kuneqela le-4 enkulu; Ukuba umzekelo omnye uyasilela, kwimeko yokuqala i-10% yetrafikhi iya kusasazwa ngaphaya kwamanqaku ali-9, okwesibini - i-25% yezithuthi ngaphaya kwamanqaku amathathu.

Kwaye ke, ngaba oku bekunokubonwa kwangaphambili, kuze ke ngoko kuthintelwe ngokuzisa ingxaki?

ukuba, kwaye ngeendlela ezininzi.

Okokuqala, ngokulinganisa ukusetyenziswa kwe-CPU ephezulu usebenzisa izixhobo ezifana stress-ng okanye cpuburn:

❯ stress-ng --matrix 1 -t 60s

Ubunjineli be-Chaos: ubugcisa bokutshatyalaliswa ngabom. Icandelo lesi-2
uxinzelelo-ng

Okwesibini, ngokulayisha kakhulu umzekelo nge wrk kunye nezinye izinto eziluncedo ezifanayo:

❯ wrk -t12 -c400 -d20s http://127.0.0.1/api/health

Ubunjineli be-Chaos: ubugcisa bokutshatyalaliswa ngabom. Icandelo lesi-2

Imifuniselo ilula ngokwentelekiso, kodwa inokubonelela ngokutya okulungileyo kokucinga ngaphandle kokutyhubela uxinzelelo lokusilela kokwenyani.

Nangona kunjalo, ungayeki apho. Zama ukuvelisa kwakhona ingozi kwindawo yovavanyo kwaye ujonge impendulo yakho kumbuzo "Ngaba oku kwakunokubonwa kwangaphambili kuze ke ngoko kuthintelwe ngokuzisa isiphoso?" Olu luvavanyo lwesiphithiphithi esincinci ngaphakathi kovavanyo lwesiphithiphithi sokuvavanya iingqikelelo, kodwa luqala ngokusilela.

Ubunjineli be-Chaos: ubugcisa bokutshatyalaliswa ngabom. Icandelo lesi-2
Ngaba yayiliphupha okanye yenzeka ngokwenene?

Ngoko funda imbali yokungaphumeleli, hlalutya I-COE, phawula kwaye uhlele ngokuthi "hit radius" -okanye ngokuchanekileyo, inani labathengi abachaphazelekayo-kwaye ukhangele iipateni. Zibuze ukuba ngaba oku bekunokuxelwa kwangaphambili kwaye kuthintelwe ngokuzisa ingxaki. Jonga impendulo yakho.

Emva koko tshintshela kwiipateni eziqhelekileyo ezinoluhlu olukhulu.

2. Yakha imephu yokuxhomekeka

Thatha umzuzwana ucinge ngesicelo sakho. Ngaba kukho imephu ecacileyo yokuxhomekeka kwayo? Ngaba uyazi ukuba yintoni impembelelo abaya kuba nayo xa kukho ukungaphumeleli?

Ukuba awuqhelananga kakhulu nekhowudi yesicelo sakho okanye ibe nkulu kakhulu, kunokuba nzima ukuqonda ukuba ikhowudi yenza ntoni kwaye yintoni exhomekeke kuyo. Ukuqonda ezi zixhomekeke kunye neempembelelo zabo ezinokwenzeka kwisicelo kunye nabasebenzisi kubalulekile ukwazi ukuba ungaqala phi ngobunjineli be-chaos: indawo yokuqala icandelo kunye neyona ndawo inkulu yempembelelo.

Ukuchonga nokubhala ukuxhomekeka kubizwa ngokuba "ukwakha imephu yokuxhomekeka» (imephu yokuxhomekeka). Oku ngokuqhelekileyo kwenziwa kwizicelo ezinesiseko sekhowudi enkulu usebenzisa izixhobo zekhowudi yeprofayili. (ikhowudi yeprofayile) kunye nezixhobo (isixhobo). Ungakha kwakhona imephu ngokujonga itrafikhi yenethiwekhi.

Nangona kunjalo, ayizizo zonke izinto ezixhomekeke kuzo ezifanayo (ezenza nzima ngakumbi inkqubo). Abanye gxeka, enye - yesibini (ubuncinci kwithiyori, kuba ungquzulwano luhlala lusenzeka ngenxa yeengxaki zokuxhomekeka ebezithathwa njengezingabalulekanga).

Ngaphandle kokuxhomekeka okubalulekileyo, inkonzo ayinakusebenza. Ukuxhomekeka okungabalulekanga "Unga» ukuphembelela inkonzo kwimeko yokuwa. Ukuqonda ukuxhomekeka, kufuneka ube nokuqonda okucacileyo kwee-API ezisetyenziswa sisicelo sakho. Oku kunokuba nzima kakhulu kunokuba kubonakala - ubuncinci kwizicelo ezinkulu.

Qala ngokuhamba kuzo zonke ii-APIs. Balaselisa kakhulu ebalulekileyo kwaye ibaluleke kakhulu. Thatha зависимости ukusuka kwindawo yokugcina ikhowudi, yijonge iilog zoqhagamshelwano, emva koko jonga uxwebhu (ngokuqinisekileyo, ukuba ikhona - kungenjalo usenayoоiingxaki ezinkulu). Sebenzisa izixhobo ukuze iprofayile kunye nomkhondo, hluza iminxeba yangaphandle.

Ungasebenzisa iinkqubo ezifana netstat - Usetyenziso lomgca womyalelo obonisa uluhlu lwazo zonke iidibansi zenethiwekhi (iziseko ezisebenzayo) kwinkqubo. Umzekelo, ukudwelisa lonke uqhagamshelo lwangoku, chwetheza:

❯ netstat -a | more 

Ubunjineli be-Chaos: ubugcisa bokutshatyalaliswa ngabom. Icandelo lesi-2

Kwi-AWS ungasebenzisa zikhuni zokuhamba (iilogi zokuhamba) I-VPC yindlela ekuvumela ukuba uqokelele ulwazi malunga nokuhamba kwe-IP ukuya okanye kwi-interfaces yenethiwekhi kwi-VPC. Iilogi ezinjalo zinokunceda neminye imisebenzi - umzekelo, ukufumana impendulo kumbuzo wokuba kutheni i-traffic ethile ingafiki kumzekelo.

Ungasebenzisa kwakhona AWS X-Ray. I-X-Ray ikuvumela ukuba ufumane iinkcukacha, "ekugqibeleni" (ekupheleni ukuya ekupheleni) Isishwankathelo sezicelo njengoko zihamba ngesicelo, kwaye kwakha imephu yamacandelo aphantsi kwesicelo. Ilungele kakhulu ukuba ufuna ukuchonga abaxhomekeke.

Ubunjineli be-Chaos: ubugcisa bokutshatyalaliswa ngabom. Icandelo lesi-2
AWS X-Ray Console

Imephu yokuxhomekeka kuthungelwano sisisombululo nje esingaphelelanga. Ewe, ibonisa ukuba yeyiphi isicelo esinxibelelana nayo, kodwa kukho ezinye izinto ezixhomekeke kuyo.

Izicelo ezininzi zisebenzisa i-DNS ukuqhagamshela kwizixhomekeke, ngelixa ezinye zinokusebenzisa ukufunyanwa kwenkonzo okanye needilesi ze-IP ezifakwe nzima kwiifayile zoqwalaselo (umzekelo. /etc/hosts).

Umzekelo, unokudala DNS umngxuma omnyama ngoncedo iptables kwaye ubone ukuba yintoni eqhekezayo. Ukwenza oku, ngenisa lo myalelo ulandelayo:

❯ iptables -I OUTPUT -p udp --dport 53 -j REJECT -m comment --comment "Reject DNS"

Ubunjineli be-Chaos: ubugcisa bokutshatyalaliswa ngabom. Icandelo lesi-2
DNS umngxuma omnyama

Ukuba ngaphakathi /etc/hosts okanye ezinye iifayile zoqwalaselo, uya kufumana iidilesi ze-IP ongazi nto ngayo (ewe, ngelishwa, oku kuyenzeka), unokuza kuhlangula kwakhona. iptables. Masithi ubhaqile 8.8.8.8 kwaye andazi ukuba le yidilesi yeseva kaGoogle yeDNS kawonke wonke. Ngokusebenzisa iptables Ungavala itrafikhi engenayo nephumayo kule dilesi usebenzisa le miyalelo ilandelayo:

❯ iptables -A INPUT -s 8.8.8.8 -j DROP -m comment --comment "Reject from 8.8.8.8"
❯ iptables -A OUTPUT -d 8.8.8.8 -j DROP -m comment --comment "Reject to 8.8.8.8"

Ubunjineli be-Chaos: ubugcisa bokutshatyalaliswa ngabom. Icandelo lesi-2
Ukuvala ukufikelela

Umgaqo wokuqala ulahla zonke iipakethi kwi-DNS kaGoogle kawonkewonke: ping isebenza, kodwa iipakethi azibuyiswa. Umgaqo wesibini uwisa zonke iipakethi ezisuka kwindlela yakho ukuya kwi-DNS kaGoogle kawonke-wonke-ekuphenduleni ping sifumana Umsebenzi awuvumelekanga.

Qaphela: kule meko kuya kuba ngcono ukusebenzisa whois 8.8.8.8, kodwa lo ngumzekelo nje.

Singangena nzulu phantsi komngxuma womvundla, kuba yonke into esebenzisa i-TCP kunye ne-UDP eneneni ixhomekeke kwi-IP nayo. Kwiimeko ezininzi, i-IP ibotshelelwe kwi-ARP. Ungalibali ngeefirewall...

Ubunjineli be-Chaos: ubugcisa bokutshatyalaliswa ngabom. Icandelo lesi-2
Ukuba uthatha ipilisi ebomvu, uhlala e-Wonderland, kwaye ndiza kukubonisa ukuba unzulu kangakanani umngxuma womvundla."

Indlela engqongqo ngakumbi kukuba nqamla iimoto nganye nganye kwaye ubone ukuba yintoni eyophukileyo ... ibe "yi-chaos monkey." Ewe, iinkqubo ezininzi zokuvelisa azenzelwanga uhlaselo olunjalo lwamandla, kodwa ubuncinci lunokuzanywa kwindawo yovavanyo.

Ukwakha imephu yokuxhomekeka kudla ngokuba luxanduva olude kakhulu. Kutshanje ndithethe nomxhasi ochithe phantse iminyaka emi-2 ephuhlisa isixhobo esivelisa ngokuzenzekelayo iimephu zokuxhomekeka kumakhulu eenkonzo ezincinci kunye nemiyalelo.

Isiphumo, nangona kunjalo, sinomdla kakhulu kwaye siluncedo. Uya kufunda okuninzi malunga nenkqubo yakho, ukuxhomekeka kwayo kunye nokusebenza. Kwakhona, yiba nomonde: luhambo ngokwalo olubaluleke kakhulu.

3. Kulumkele ukuzithemba ngokugqithiseleyo

"Nabani na ophuphayo ngento, ukholelwa kuyo." — Demosthenes

Ngaba wakha weva ukuzithemba ngokugqithiseleyo?

Ngokutsho kweWikipedia, isiphumo sokuzithemba ngokugqithisileyo “lukhetho lokuqonda apho ukuzithemba komntu kwizenzo nakwizigqibo zakhe kungaphezulu kakhulu kunokuchaneka kwenjongo yezo zigwebo, ngakumbi xa umgangatho wokuzithemba uphezulu.

Ubunjineli be-Chaos: ubugcisa bokutshatyalaliswa ngabom. Icandelo lesi-2
Ngokusekwe kwithuku kunye namava...

Kumava am, oku kugqwetheka luphawu oluhle lokuba ungaqala phi ngobunjineli besiphithiphithi.

Mlumkele umsebenzisi ozithembe kakhulu:

UCharlie: "Le nto ayizange iwe kwiminyaka emihlanu, yonke into ilungile!"
Ingozi: "Yima ... ndiza kufika kungekudala!"

I-bias njengesiphumo sokuzithemba ngokugqithisileyo yinto efihlakeleyo kwaye inobungozi ngenxa yezinto ezahlukeneyo eziyiphembelela. Oku kuyinyani ngakumbi xa amalungu eqela athe agalela iintliziyo zawo kwitekhnoloji okanye achitha ixesha elininzi “eyilungisa”.

Ukushwankathela

Ukukhangela indawo yokuqala yobunjineli besiphithiphithi kuhlala kuzisa iziphumo ezingaphezulu kunebezilindelwe, kwaye amaqela aqala ukophula izinto ngokukhawuleza aphulukane nombono wehlabathi kunye neyona nto inomdla ye (isiphithiphithi-)ubunjineli -Usetyenziso oluyilayo iindlela zenzululwazi и ubungqina obunobungqina kuyilo, uphuhliso, ukusebenza, ukugcinwa kunye nokuphuculwa kweenkqubo (software).

Oku kuqukumbela inxalenye yesibini. Nceda ubhale uphononongo, wabelane ngezimvo okanye uqhwabe izandla phakathi. Kwinxalenye elandelayo I ngokwenene Ndiza kuqwalasela izixhobo kunye neendlela zokwazisa ukusilela kwiinkqubo. De!

PS evela kumguquleli

Funda nakwibhlog yethu:

umthombo: www.habr.com

Yongeza izimvo