Qaphela. transl.: Lesi sihloko siqhubeka nochungechunge oluhle lwezihloko ezivela kumshumayeli wezobuchwepheshe be-AWS u-Adrian Hornsby, ozimisele ukuchaza ngendlela elula necacile ngokubaluleka kokuhlola ukuze kuncishiswe imiphumela yokwehluleka ezinhlelweni ze-IT.
“Uma wehluleka ukulungisa uhlelo, kusho ukuthi uhlela ukwehluleka.” - Benjamin Franklin
В Kulolu chungechunge lwezihloko, ngethule umqondo wobunjiniyela besiphithiphithi futhi ngachaza ukuthi kusiza kanjani ukuthola nokulungisa amaphutha ohlelweni ngaphambi kokuholela ekuhlulekeni kokukhiqiza. Iphinde yaxoxa ngokuthi ubunjiniyela besiphithiphithi bukhuthaza kanjani ushintsho oluhle lwamasiko phakathi kwezinhlangano.
Ekupheleni kwengxenye yokuqala, ngethembise ukukhuluma “ngamathuluzi nezindlela zokwethula ukwehluleka ezinhlelweni.” Maye, ikhanda lami lalinezinhlelo zalo mayelana nalokhu, futhi kulesi sihloko ngizozama ukuphendula umbuzo othandwa kakhulu ophakama phakathi kwabantu abafuna ukungena ebunjiniyela be-chaos: Yini okufanele uphule kuqala?
Umbuzo omkhulu! Kodwa-ke, akabonakali ekhathazwa yile panda...
Ungadlali nge-chaos panda!
Impendulo emfushane: Khomba izinsiza ezibalulekile endleleni yokucela.
Impendulo ende kodwa ecacile: Ukuze uqonde ukuthi ungaqala kuphi ukuhlola isiphithiphithi, naka izindawo ezintathu:
- Bheka umlando wokuphahlazeka futhi akhombe amaphethini;
- Nquma ukuncika okubalulekile;
- Sebenzisa okuthiwa ukuzethemba ngokweqile.
Kuyahlekisa, kodwa le ngxenye ingabizwa kalula nje "Uhambo Lokuzitholela Nokuzikhanyisela". Kuyo sizoqala “ukudlala” ngezinsimbi ezithile ezipholile.
1. Impendulo ilele esikhathini esidlule
Uma ukhumbula, engxenyeni yokuqala ngethule umqondo wokulungiswa kwamaphutha (COE) - indlela esihlaziya ngayo amaphutha ethu - amaphutha kwezobuchwepheshe, inqubo noma inhlangano - ukuze siqonde imbangela (izi) zabo futhi sivimbele. ukuphindeka esikhathini esizayo . Ngokuvamile, yilapho kufanele uqale khona.
"Ukuze uqonde okwamanje, udinga ukwazi okwedlule." - UCarl Sagan
Bheka umlando wokwehluleka, ukumaka ku-COE noma kuma-postmortems futhi uwahlukanise. Thola amaphethini avamile avame ukuholela ezinkingeni, futhi ku-COE ngayinye, zibuze umbuzo olandelayo:
"Ingabe lokhu bekungabikezelwa ngakho-ke kuvinjwe ngomjovo wephutha?"
Ngikhumbula iphutha elilodwa ekuqaleni komsebenzi wami. Bekungavinjelwa kalula ukube senze izivivinyo ezimbalwa ezilula zesiphithiphithi:
Ngaphansi kwezimo ezijwayelekile, izimo ezingemuva ziphendula ukuhlolwa kwezempilo kusuka ). I-ELB isebenzisa lokhu kuhlola ukuqondisa kabusha izicelo ezimeni ezinempilo. Uma kuvela ukuthi isigameko "asinampilo", i-ELB iyayeka ukuthumela izicelo kuso. Ngolunye usuku, ngemva komkhankaso wokumaketha ophumelelayo, umthamo wethrafikhi wanda futhi abangemuva baqala ukuphendula ekuhlolweni kwezempilo kancane kancane kunokujwayelekile. Kufanele kushiwo ukuthi lokhu kuhlolwa kwezempilo kwaba , okungukuthi, isimo sokuncika sihloliwe.
Nokho, konke kwakuhamba kahle okwesikhashana.
Ngemuva kwalokho, ngaphansi kwezimo ezicindezelayo, esinye sezimo saqala ukwenza umsebenzi ongabalulekile, ojwayelekile we-ETL cron. Inhlanganisela yethrafikhi ephezulu ne-cronjob iphushe ukusetshenziswa kwe-CPU cishe ku-100%. Ukulayishwa ngokweqile kwe-CPU kuqhubekisele phambili ijubane lokuphendula ekuhlolweni kwezempilo, kangangokuthi i-ELB yanquma ukuthi lesi sibonelo sasinezinkinga zokusebenza. Njengoba bekulindelekile, ibhalansi yayeka ukusabalalisa ithrafikhi kuyo, okwaholela ekwandeni komthwalo ezimweni ezisele eqenjini.
Kungazelelwe, zonke ezinye izimo nazo zaqala ukwehluleka ukuhlolwa kwezempilo.
Ukuqala isenzakalo esisha kudinga ukulanda nokufaka amaphakheji futhi kwathatha isikhathi eside kunalokho okuthathe i-ELB ukuwakhubaza - ngamunye ngamunye - eqenjini le-autoscaling. Kuyacaca ukuthi ngokushesha lonke uhlelo lwafinyelela eqophelweni elibucayi futhi isicelo saphahlazeka.
Sabe sesiqonda phakade la maphuzu alandelayo:
- Ukufaka isofthiwe lapho udala isenzakalo esisha kuthatha isikhathi eside; kungcono ukunikeza okuncamelayo endleleni engaguquki futhi .
- Ezimeni eziyinkimbinkimbi, izimpendulo ekuhlolweni kwezempilo kanye nama-ELB kufanele zize kuqala - into yokugcina oyifunayo ukwenza impilo ibe nzima ezimweni ezisele.
- Ukugcinwa kunqolobane kwasendaweni kokuhlolwa kwezempilo kusiza kakhulu (ngisho nangemizuzwana embalwa).
- Esimeni esinzima, ungasebenzisi imisebenzi ye-cron nezinye izinqubo ezingabalulekile - gcina izinsiza zemisebenzi ebaluleke kakhulu.
- Lapho uscaling ngokuzenzakalela, sebenzisa izimo ezincane. Iqembu lama-specimens amancane ayi-10 lingcono kuneqembu lama-4 amakhulu; uma isigameko esisodwa sihluleka, okokuqala u-10% wethrafikhi uzosatshalaliswa ngaphezu kwamaphoyinti angu-9, okwesibili - u-25% wethrafikhi ngaphezu kwamaphuzu amathathu.
Ngakho-ke, ngabe lokhu bekungabonwa kusengaphambili, ngakho-ke kuvinjwe ngokwethula inkinga?
Yebo, nangezindlela eziningana.
Okokuqala, ngokulingisa ukusetshenziswa okuphezulu kwe-CPU usebenzisa amathuluzi afana noma :
❯ stress-ng --matrix 1 -t 60s
ukucindezela-ng
Okwesibili, ngokulayisha ngokweqile isibonelo nge nezinye izinsiza ezifanayo:
❯ wrk -t12 -c400 -d20s http://127.0.0.1/api/health
Izivivinyo zilula uma kuqhathaniswa, kodwa zinganikeza ukudla okuhle kokucabanga ngaphandle kokubhekana nengcindezi yokwehluleka kwangempela.
Nokho, ungagcini lapho. Zama ukukhiqiza kabusha ukuphahlazeka endaweni yokuhlola bese uhlola impendulo yakho embuzweni othi "Ingabe lokhu bekungabonwa kusengaphambili ngakho-ke kuvinjwe ngokuletha iphutha?" Lesi isilingo sesiphithiphithi esincane esingaphakathi kwesilingo sesiphithiphithi sokuhlola ukucabanga, kodwa siqala ngokwehluleka.
Ingabe kwakuyiphupho, noma kwenzeka ngempela?
Ngakho funda umlando wokwehluleka, hlaziya I-COE, maka futhi uwahlukanise ngokuthi “hit radius”—noma ngokunembile, inani lamakhasimende athintekile—bese ubheka amaphethini. Zibuze ukuthi ngabe lokhu bekubikezelwe futhi kuvinjwe yini ngokwethula inkinga. Hlola impendulo yakho.
Bese ushintshela kumaphethini ajwayeleke kakhulu anobubanzi obukhulu kakhulu.
2. Yakha imephu yokuncika
Thatha isikhashana ucabange ngesicelo sakho. Ingabe likhona imephu ecacile yokuncika kwayo? Uyazi ukuthi bazoba namuphi umthelela uma kukhona ukwehluleka?
Uma ungayazi kahle ikhodi yohlelo lwakho lokusebenza noma isiba nkulu kakhulu, kungaba nzima ukuqonda ukuthi ikhodi yenzani nokuthi incike ini. Ukuqonda lokhu kuncika kanye nomthelela wako ongaba khona kuhlelo lokusebenza nakubasebenzisi kubalulekile ekwazini ukuthi ungaqala kuphi ngobunjiniyela besiphithiphithi: indawo yokuqala ingxenye enendawo enkulu yomthelela.
Ukuhlonza nokubhala okuncikile kubizwa ngokuthi "ukwakha imephu yokuncika» (imephu yokuncika). Lokhu kuvame ukwenzelwa izinhlelo zokusebenza ezinesisekelo esikhulu sekhodi kusetshenziswa amathuluzi wokwenza iphrofayela yekhodi. (iphrofayili yekhodi) kanye nezinsimbi (izinsimbi). Ungakwazi futhi ukwakha imephu ngokuqapha ithrafikhi yenethiwekhi.
Kodwa-ke, akuzona zonke izincika ezifanayo (okwenza inqubo ibe nzima nakakhulu). Abanye okugxekayo, okunye - okwesibili (okungenani ngokombono, njengoba ukuphahlazeka kuvame ukwenzeka ngenxa yezinkinga zokuncika obekubhekwa njengokungabalulekile).
Ngaphandle kokuncika okubalulekile, isevisi ayikwazi ukusebenza. Ukuncika okungabalulekile "akufanele»ukuthonya isevisi uma kwenzeka ukuwa. Ukuze uqonde ukuncika, udinga ukuqonda okucacile kwama-API asetshenziswa uhlelo lwakho lokusebenza. Lokhu kungaba nzima kakhulu kunalokho okubonakala - okungenani ezinhlelweni ezinkulu.
Qala ngokudlula kuwo wonke ama-API. Gqamisa kakhulu ebalulekile futhi ebucayi. Thatha ukuthembela kusuka endaweni yokugcina amakhodi, kuhlole izingodo zokuxhuma, bese ubuka imibhalo (Yebo, uma ikhona - kungenjalo usenayoоizinkinga ezinkulu). Sebenzisa amathuluzi ukuze ukuphrofayili nokulandelela, hlunga amakholi angaphandle.
Ungasebenzisa izinhlelo ezifana netstat - insiza yomugqa womyalo ebonisa uhlu lwazo zonke izixhumi zenethiwekhi (amasokhethi asebenzayo) ohlelweni. Isibonelo, ukuze ufake kuhlu konke ukuxhumana kwamanje, thayipha:
❯ netstat -a | more
Ku-AWS ungasebenzisa (flow logs) I-VPC indlela ekuvumela ukuthi uqoqe ulwazi mayelana nethrafikhi ye-IP eya noma isuka ezindaweni zokusebenzelana zenethiwekhi ku-VPC. Amalogi anjalo angasiza nangeminye imisebenzi - ngokwesibonelo, ukuthola impendulo yombuzo othi kungani ithrafikhi ethile ingafinyeleli esimeni.
Ungasebenzisa futhi . I-X-Ray ikuvumela ukuthi uthole imininingwane, "ekugcineni" (ekugcineni-ekugcineni) Uhlolojikelele lwezicelo njengoba zihamba phakathi kwesicelo, futhi yakha imephu yezingxenye eziyisisekelo zohlelo lokusebenza. Kulula kakhulu uma udinga ukukhomba okuncikile.
I-AWS X-Ray Console
Imephu yokuncika kwenethiwekhi iyisixazululo esiyingxenye kuphela. Yebo, ikhombisa ukuthi yiluphi uhlelo lokusebenza oluxhumana nalo, kepha kukhona okunye ukuncika.
Izinhlelo zokusebenza eziningi zisebenzisa i-DNS ukuxhuma kokuncikile, kuyilapho ezinye zingasebenzisa ukutholwa kwesevisi noma amakheli e-IP anekhodi eqinile kumafayela okumisa (isb. /etc/hosts).
Isibonelo, ungakha ngosizo lwe iptables futhi ubone ukuthi yini ephukayo. Ukuze wenze lokhu, faka umyalo olandelayo:
❯ iptables -I OUTPUT -p udp --dport 53 -j REJECT -m comment --comment "Reject DNS"
Imbobo emnyama ye-DNS
Uma ku /etc/hosts noma amanye amafayela okucushwa, uzothola amakheli e-IP ongazi lutho ngawo (yebo, ngeshwa, lokhu kuyenzeka), ungaphinde usize iptables. Ake sithi uthole 8.8.8.8 futhi angazi ukuthi leli ikheli leseva ye-Google ye-DNS yomphakathi. Ngokusebenzisa iptables Ungakwazi ukuvimba ithrafikhi engenayo nephumayo kuleli kheli usebenzisa imiyalo elandelayo:
❯ iptables -A INPUT -s 8.8.8.8 -j DROP -m comment --comment "Reject from 8.8.8.8"
❯ iptables -A OUTPUT -d 8.8.8.8 -j DROP -m comment --comment "Reject to 8.8.8.8"
Ukuvala ukufinyelela
Umthetho wokuqala ulahla wonke amaphakethe ku-DNS yomphakathi ye-Google: ping iyasebenza, kodwa amaphakethe awabuyiswa. Umthetho wesibili uwisa wonke amaphakethe asuka kusistimu yakho aye ku-DNS yomphakathi ye-Google - ngokuphendula ping sithola Ukusebenza akuvunyelwe.
Qaphela: kulesi simo kungaba ngcono ukusebenzisa whois 8.8.8.8, kodwa lesi isibonelo nje.
Singangena sijule ngisho nasemgodini onogwaja, ngoba yonke into esebenzisa i-TCP ne-UDP empeleni incike ku-IP futhi. Ezimweni eziningi, i-IP iboshelwe ku-ARP. Ungakhohlwa ngama-firewall...
Uma uthatha iphilisi elibomvu, uhlala e-Wonderland, futhi ngizokukhombisa ukuthi ujula kangakanani umgodi onogwaja."
Indlela egqamile iwukuba nqamula izimoto ngayinye ngayinye futhi ubone ukuthi yini ephukile ... abe "i-chaos monkey." Yiqiniso, izinhlelo eziningi zokukhiqiza aziklanyelwe ukuhlasela kwebutho elinonya, kodwa okungenani kungazanywa endaweni yokuhlola.
Ukwakha imephu yokuncika ngokuvamile kuwumsebenzi omude kakhulu. Ngisanda kukhuluma neklayenti elichithe cishe iminyaka emi-2 lithuthukisa ithuluzi elikhiqiza ngokuzenzakalelayo amamephu okuncika ngamakhulu ama-microservices nemiyalo.
Umphumela, nokho, uthakazelisa kakhulu futhi uwusizo. Uzofunda okuningi mayelana nesistimu yakho, ukuncika kwayo kanye nokusebenza. Futhi, bekezela: wuhambo ngokwalo olubaluleke kakhulu.
3. Qaphela ukuzethemba ngokweqile
"Noma ngubani ophuphayo, ukholelwa kukho." - I-Demosthenes
Wake wezwa ngani ukuzethemba ngokweqile?
Ngokuvumelana ne Wikipedia, umphumela wokuzethemba ngokweqile “uwukuchema komqondo lapho ukuzethemba komuntu ezenzweni nasezinqumweni zakhe kukhulu kakhulu kunokunemba okuhlosiwe kwalezo zinqumo, ikakhulukazi lapho izinga lokuzethemba liphezulu uma kuqhathaniswa.”
Ngokusekelwe kumzwelo wemvelo nolwazi...
Ngokuhlangenwe nakho kwami, lokhu kuhlanekezela kuwuphawu oluhle lokuthi ungaqala kuphi ngobunjiniyela besiphithiphithi.
Qaphela u-opharetha ozethemba ngokweqile:
UCharlie: "Le nto ayizange iwe eminyakeni emihlanu, konke kuhamba kahle!"
Ukuphahlazeka: "Ima... ngizofika maduze!"
Ukuchema njengomphumela wokuzethemba ngokweqile kuyinto ecashile futhi eyingozi ngisho nangenxa yezinto ezihlukahlukene ezikuthonyayo. Lokhu kuyiqiniso ikakhulukazi lapho amalungu eqembu ethele izinhliziyo zawo kubuchwepheshe noma echitha isikhathi esiningi “ewulungisa”.
Ukufingqa
Ukuseshwa kwesiqalo sobunjiniyela besiphithiphithi kuhlala kuletha imiphumela eminingi kunalokho obekulindelwe, futhi amaqembu aqala ukwephula izinto ngokushesha alahlekelwa umbono wengqikithi yomhlaba wonke nehehayo ye (isiphithiphithi-)ubunjiniyela - Isicelo sokudala izindlela zesayensi и ubufakazi obunamandla ukuklama, ukuthuthukiswa, ukusebenza, ukugcinwa kanye nokwenza ngcono (isofthiwe) izinhlelo.
Lokhu kuphetha ingxenye yesibili. Sicela ubhale ukubuyekezwa, wabelane ngemibono noma uvele ushaye izandla . Engxenyeni elandelayo I ngempela Ngizobheka amathuluzi nezindlela zokwethula ukwehluleka kumasistimu. Kuze kube!
I-PS evela kumhumushi
Funda futhi kubhulogi yethu:
- «";
- «";
- «";
- «".
Source: www.habr.com