Cherechedza. transl.: Chinyorwa ichi chinoenderera mberi nenhevedzano huru yezvinyorwa kubva kuAWS tekinoroji muevhangeri Adrian Hornsby, uyo anotanga kutsanangura nenzira iri nyore uye yakajeka kukosha kwekuedza kuderedza mhedzisiro yekutadza muIT masisitimu.
"Kana ukatadza kugadzirira chirongwa, saka unoronga kukundikana." - Benjamin Franklin
Π Mune ino nhevedzano yezvinyorwa, ndakaunza iyo pfungwa yemhirizhonga engineering uye ndikatsanangura kuti inobatsira sei kuwana nekugadzirisa zvikanganiso muhurongwa zvisati zvatungamira mukukundikana kwekugadzira. Yakakurukurawo kuti chaos engineering inosimudzira sei shanduko yakanaka yetsika mukati memasangano.
Pakupera kwechikamu chekutanga, ndakavimbisa kutaura nezve "zvishandiso uye nzira dzekuunza kukundikana muhurongwa." Maiwe, musoro wangu waive nehurongwa hwawo mune izvi, uye mune ino chinyorwa ndichaedza kupindura mubvunzo unozivikanwa unomuka pakati pevanhu vanoda kupinda mumhirizhonga engineering: Chii chekutanga kutyora?
Mubvunzo wakanaka! Nekudaro, haaite kunge ari kunyanya kunetseka nepanda iyi ...
Usakanganise nenyonganyonga panda!
Mhinduro pfupi: Tarisira masevhisi akakosha munzira yekukumbira.
Mhinduro refu asi yakajeka: Kuti unzwisise kwekutanga kuyedza mhirizhonga, teerera kunzvimbo nhatu:
- Tarisa uone nhoroondo yekuparara uye kuziva mapatani;
- Sarudza zvakakosha kutsamira;
- Shandisa inonzi overconfidence effect.
Zvinosetsa, asi chikamu ichi chinogona kungodaidzwa zviri nyore "Rwendo rweKuzvitsvaga uye Kuvhenekerwa". Mariri tichatanga "kutamba" nezvimwe zviridzwa zvinotonhorera.
1. Mhinduro iripo kare
Kana iwe uchiyeuka, muchikamu chekutanga ndakaunza pfungwa yeKururamisa-kwe-Mhosho (COE) - nzira yatinoongorora nayo kukanganisa kwedu - kukanganisa mune tekinoroji, maitiro kana sangano - kuti tinzwisise chikonzero chavo (s) uye kudzivirira. kudzokorora mune ramangwana . Kazhinji, apa ndipo paunofanira kutanga.
"Kuti unzwisise zvazvino, unofanirwa kuziva zvakapfuura." β Carl Sagan
Tarisa nhoroondo yekutadza, tag ivo muCOE kana postmortems uye uzviise mumapoka. Ziva mapatani akajairika anowanzo tungamira kumatambudziko, uye kune yega yega COE, zvibvunze iwe unotevera mubvunzo:
"Izvi zvingave zvakafanotaurwa uye nokudaro kudzivirirwa nejekiseni rekukanganisa?"
Ndinoyeuka kumwe kukundikana pakutanga kwebasa rangu. Zvingadai zvakadzivirirwa zviri nyore dai takaita zviyedzo zvakapusa zvemhirizhonga:
Mumamiriro ezvinhu akajairwa, zviitiko zvekumashure zvinopindura kuongororwa kwehutano kubva ) ELB inoshandisa cheki idzi kutungamira zvikumbiro kune zvine hutano zviitiko. Kana zvikazoitika kuti chiitiko "hachina hutano", ELB inomira kutumira zvikumbiro kwairi. Rimwe zuva, mushure mekushambadzira kwakabudirira kwekushambadzira, huwandu hwemotokari hwakawedzera uye mashure akatanga kupindura kuongororwa kwehutano zvishoma nezvishoma kupfuura zvakajairika. Zvinofanira kutaurwa kuti izvi utano cheki vaiva , kureva kuti, mamiriro ekutsamira akatariswa.
Zvisinei, zvinhu zvose zvakanga zvakanaka kwenguva yakati.
Zvino, zvatove pasi pemamiriro ezvinhu anonetsa, imwe yezviitiko yakatanga kuita isiri-yakakosha, yenguva dzose ETL cron basa. Iko kusanganiswa kwepamusoro traffic uye cronjob yakasundira CPU kushandiswa kusvika kunosvika zana%. Kuremerwa kweCPU kwakawedzera kudzikisira mhinduro kumacheki ehutano, zvekuti ELB yakafunga kuti chiitiko ichi chaive nematambudziko ekuita. Sezvaitarisirwa, muchengeti akamira kugovera traffic kwairi, izvo, zvakare, zvakakonzera kuwedzera kwemutoro pane zvakasara zviitiko muboka.
Nenguva isipi, mamwe mamiriro ese akatangawo kutadza kuongororwa kwehutano.
Kutanga chiitiko chitsva chaida kudhawunirodha nekuisa mapakeji uye zvakatora nguva yakareba kupfuura zvakatora ELB kuvadzima - chimwe nechimwe - muboka re autoscaling. Zviri pachena kuti munguva pfupi nzira yese yakasvika padanho rakakosha uye chikumbiro chakadonha.
Ipapo isu takanzwisisa nekusingaperi pfungwa dzinotevera:
- Kuisa software kana uchigadzira chiitiko chitsva kunotora nguva yakareba; zviri nani kupa sarudzo kune isingachinjike nzira uye .
- Mumamiriro ezvinhu akaoma, mhinduro kune hutano-cheki uye ELBs dzinofanira kutora pekutanga - chinhu chekupedzisira chaunoda kuomesa hupenyu kune mamwe mamiriro asara.
- Local caching yehutano cheki inobatsira zvikuru (kunyangwe kwemasekonzi mashoma).
- Mumamiriro ezvinhu akaoma, usamhanya cron mabasa uye mamwe maitiro asina-akakosha - chengetedza zviwanikwa zveakanyanya kukosha mabasa.
- Kana autoscaling, shandisa zvidiki zviitiko. Boka rezvienzaniso gumi zviri nani pane boka remazana mana; kana chiitiko chimwe chikatadza, mune yekutanga 10% yetraffic ichagoverwa pamusoro pe4 mapoinzi, mune yechipiri - 10% yemotokari pamusoro pemapoinzi matatu.
Uye saka, izvi zvingave zvakafanooneswa, uye nokudaro kudziviswa nokuunza dambudziko here?
kuti, uye munzira dzakawanda.
Kutanga, nekutevedzera yakakwira CPU kushandiswa uchishandisa maturusi akadai kana :
β― stress-ng --matrix 1 -t 60s
kunetseka-ng
Chechipiri, nekuremedza muenzaniso ne nezvimwe zvinoshandiswa zvakafanana:
β― wrk -t12 -c400 -d20s http://127.0.0.1/api/health
Iwo maedzo akareruka, asi anogona kupa chikafu chakanaka chekufunga pasina kuenda nekushushikana kwekutadza chaiko.
Zvisinei, usagumire ipapo. Edza kuburitsa tsaona munzvimbo yekuyedza uye tarisa mhinduro yako kumubvunzo "Izvi zvingave zvakafanoona uye nokudaro kudziviswa nokuunza chikanganiso here?" Uku ndiko kuyedza mhirizhonga mukati meyedzo yemhirizhonga yekuyedza fungidziro, asi kutanga nekutadza.
Kwaiva kurota here kana kuti zvakaitika zvechokwadi?
Saka dzidza nhoroondo yekukundikana, ongorora EOC, tag uye uzviise ne "hit radius" - kana zvakanyanya, nhamba yevatengi vanobatwa-uye wozotsvaga maitiro. Zvibvunze kana izvi zvingadai zvakafanotaurwa uye kudziviswa nekuunza dambudziko. Tarisa mhinduro yako.
Wobva wachinja kune akajairika mapatani ane hukuru hukuru.
2. Vaka mepu inotsamira
Tora nguva yekufunga nezvechikumbiro chako. Pane mepu yakajeka yezvainotsamira? Unoziva here kuti chii chavanozoita kana pane kukundikana?
Kana usiri kunyatsoziva kodhi yako yekushandisa kana kuti yakakura kwazvo, zvinogona kunetsa kunzwisisa kuti kodhi inoita sei uye kuti inoenderana nei. Kunzwisisa zvinotsamira izvi uye zvazvingaite pakushandisa uye vashandisi kwakakosha kuti uzive pekutangira nechaos engineering: pekutangira ndicho chikamu chine yakakura kukanganisa radius.
Kuziva uye kunyora zvinoenderana kunonzi "kuvaka mepu yekutsamiraΒ» (dependence mepu). Izvi zvinowanzoitirwa maapplication ane hombe kodhi base uchishandisa kodhi profiling maturusi. (code profiling) uye instrumentation (chiridzwa). Iwe unogona zvakare kuvaka mepu nekutarisa network traffic.
Nekudaro, hazvisi zvese zvinotsamira zvakafanana (izvo zvinowedzera kuomesa maitiro). Vamwe kutsoropodza, zvimwe - yechipiri (zvichida mudzidziso, sezvo kubondera kunowanzoitika nekuda kwezvinetso nezvinotsamira zvaionekwa sezvisina kukosha).
Pasina kutsamira kwakakosha, sevhisi haigone kushanda. Non-critical dependencies "haifanireΒ» kupesvedzera sevhisi muchiitiko chekudonha. Kuti unzwisise kutsamira, iwe unofanirwa kuve nekunzwisisa kwakajeka kweiyo APIs inoshandiswa nechikumbiro chako. Izvi zvinogona kuoma zvakanyanya kupfuura zvinoita - zvirinani kune makuru maapplication.
Tanga nekuenda kuburikidza nemaAPI ese. Simbisa zvakanyanya zvakakosha uye zvakakosha. Tora dependencies kubva kukodhi repository, tarisa kunze matanda ekubatanidza, wobva waona zvinyorwa (zvechokwadi, kana iripo - zvikasadaro uchinayoΠΎmatambudziko makuru). Shandisa zvishandiso kuti profiling nekutsvaga, sefa nhare dzekunze.
Unogona kushandisa mapurogiramu akafanana netstat - yekuraira mutsara utility inoratidza runyorwa rwese network yekubatanidza (inoshingaira masokisi) muhurongwa. Semuenzaniso, kunyora zvese zvinongedzo zvazvino, nyora:
β― netstat -a | more
MuAWS unogona kushandisa (kuyerera matanda) VPC inzira inokubvumira kuti utore ruzivo nezve IP traffic kuenda kana kubva kune network interfaces muVPC. Matanda akadaro anogonawo kubatsira nemamwe mabasa - semuenzaniso, kuwana mhinduro kumubvunzo wekuti nei mamwe traffic isingasviki pamuenzaniso.
Unogonawo kushandisa . X-Ray inokutendera kuti uwane zvakadzama, "yekupedzisira" (kusvika-kumagumo) Kutarisisa kwezvikumbiro pavanenge vachifamba nekushandisa, uye zvakare inovaka mepu yezvishandiso zvemukati mechikamu. Yakanaka kwazvo kana iwe uchida kuziva zvinotsamira.
AWS X-Ray Console
Mepu yekutsamira kunetiweki inongova mhinduro isina kukwana. Ehe, inoratidza kuti ndeipi application inotaurirana nayo, asi pane zvimwe zvinotsamira.
Zvishandiso zvakawanda zvinoshandisa DNS kubatanidza kune zvinotsamira, nepo vamwe vachigona kushandisa kuwanikwa kwesevhisi kana kunyange yakaoma-coded IP kero mumafaira ekugadzirisa (e.g. /etc/hosts).
Somuenzaniso, unogona kugadzira nerubatsiro iptables uye ona zvinoputsika. Kuti uite izvi, isa murairo unotevera:
β― iptables -I OUTPUT -p udp --dport 53 -j REJECT -m comment --comment "Reject DNS"
DNS gomba dema
Kana mukati /etc/hosts kana mamwe mafaera ekugadzirisa, iwe uchawana IP kero iwe yausingazive nezvayo (hongu, zvinosuruvarisa, izvi zvinoitikawo), unogona kuuya kuzonunura zvakare. iptables. Ngatiti wawana 8.8.8.8 uye handizive kuti iyi iGoogle's public DNS server address. Nokushandisa iptables Unogona kuvhara traffic inouya neinobuda kune ino kero uchishandisa mirairo inotevera:
β― iptables -A INPUT -s 8.8.8.8 -j DROP -m comment --comment "Reject from 8.8.8.8"
β― iptables -A OUTPUT -d 8.8.8.8 -j DROP -m comment --comment "Reject to 8.8.8.8"
Kuvhara kupinda
Mutemo wekutanga unodonhedza mapaketi ese kubva kuGoogle yeruzhinji DNS: ping inoshanda, asi mapaketi haadzoserwe. Mutemo wechipiri unodonhedza mapaketi ese anobva kusystem yako akananga kuGoogle yeruzhinji DNS - mukupindura ping tinowana Kushanda hakubvumidzwe.
Cherechedza: munyaya iyi zvingava nani kushandisa whois 8.8.8.8, asi uyu unongova muenzaniso.
Tinogona kuenda zvakadzika pasi pegomba retsuro, nekuti zvese zvinoshandisa TCP uye UDP zvinoenderana neIP futi. Kazhinji, IP inosungirirwa kune ARP. Usakanganwa nezve firewalls ...
Kana ukatora piritsi dzvuku, iwe unogara muWonderland, uye ini ndichakuratidza kuti gomba retsuro rinopinda zvakadii."
Imwe nzira yakanyanyisa ndeye patsanura mota one by one uone zvaparara... iva "chaos monkey." Ehe, mazhinji masisitimu ekugadzira haana kugadzirirwa kurwiswa kwechisimba kwakadaro, asi zvirinani zvinogona kuyedzwa munzvimbo yekuyedza.
Kuvaka mepu yekutsamira kazhinji ibasa rakareba. Ini nguva pfupi yadarika ndakataura nemutengi akapedza makore angangoita maviri achigadzira chishandiso icho semi-otomatiki chinoburitsa mamepu ekuvimba kwemazana emamicroservices nemirairo.
Chigumisiro, zvisinei, chinonakidza zvikuru uye chinobatsira. Iwe unozodzidza zvakawanda nezve system yako, kutsamira kwayo uye mashandiro. Zvakare, iva nemoyo murefu: ndirwo rwendo pacharwo runonyanya kukosha.
3. Ngwarira kuzvivimba zvakanyanya
"Ani nani anorota chii, anotenda mazviri." β Demosthenes
Makambonzwa nezvayo overconfidence effect?
Sekureva kweWikipedia, mhedzisiro yekunyanya kuzvivimba "kusarerekera kwekuziva uko kuvimba kwemunhu mune zvavanoita uye sarudzo kwakakura zvakanyanya kupfuura chinangwa chechokwadi chekutonga ikoko, kunyanya kana danho rekuvimba rakanyanya."
Zvichienderana neruzivo uye ruzivo...
Mune ruzivo rwangu, kukanganisa uku izano rakakura rekuti ungatangira kupi nechaos engineering.
Ngwarira munhu anonyanya kuzvivimba:
Charlie: "Ichi chinhu hachina kudonha mumakore mashanu, zvese zvakanaka!"
Crash: "Mira ... ndichauya nekukurumidza!"
Rusarura semhedzisiro yekunyanya kuzvivimba chinhu chinonyengera uye chine njodzi nekuda kwezvikonzero zvakasiyana zvinozvifurira. Izvi ndezvechokwadi kunyanya kana nhengo dzechikwata dzakadurura mwoyo yavo mune tekinoroji kana kupedza nguva yakawanda "vachigadzirisa".
Kupfupisa
Kutsvaga pekutangira kwechaos engineering kunogara kuchiunza mibairo yakawanda kupfuura yaitarisirwa, uye zvikwata zvinotanga kutyora zvinhu nekukasira zvinotadza kuona zvepasirese uye zvinonakidza essence ye (mhirizhonga-)engineering - kushandiswa kwekugadzira nzira dzesainzi ΠΈ empical evidence yekugadzira, kusimudzira, kushanda, kuchengetedza uye kuvandudzwa kwe (software) masisitimu.
Izvi zvinopedzisa chikamu chechipiri. Ndokumbira unyore wongororo, govana zvaunofunga kana kungoombera maoko . Muchikamu chinotevera I chaizvo Ini ndichafunga maturusi uye nzira dzekuunza kutadza mumasystem. Kusvikira!
PS kubva kumushanduri
Verenga zvakare pablog yedu:
- Β«";
- Β«";
- Β«";
- Β«".
Source: www.habr.com