Cherechedza. transl.: mukutanga kwaNyamavhuvhu, Red Hat yakataura pachena nezvekugadzirisa matambudziko ekuwanikwa ayo vashandisi vebasa rayo vakasangana nawo mumwedzi yapfuura. (yakavakirwa pane registry yemifananidzo yemidziyo, iyo kambani yakagamuchira pamwe nekutenga kweCoreOS). Zvisinei nekufarira kwako basa iri sekudaro, nzira iyo mainjiniya eSRE ekambani akatora kuti aongorore uye kubvisa zvikonzero zvetsaona inodzidzisa.
Musi wa19 Chivabvu, mambakwedza (Eastern Daylight Time, EDT), sevhisi yequay.io yakaparara. Tsaona iyi yakabata vatengi ve quay.io uye Open Source mapurojekiti vachishandisa quay.io sepuratifomu yekuvaka nekuparadzira software. Red Hat inokoshesa kuvimba kwevaviri.
Chikwata chemainjiniya eSRE chakabva chapinda uye kuyedza kudzikamisa sevhisi yeQuay nekukurumidza. Nekudaro, pavainge vachiita izvi, vatengi vakarasikirwa nekugona kusunda mifananidzo mitsva, uye nguva nenguva vaikwanisa kudhonza yaivepo. Nechimwe chikonzero chisingazivikanwe, iyo quay.io dhatabhesi yakavharwa mushure mekuyera iyo sevhisi kusvika kuzere.
Β«Chii chakachinja?"- uyu ndiwo mubvunzo wekutanga unowanzo bvunzwa mumamiriro ezvinhu akadaro. Takaona kuti nguva pfupi nyaya isati yatanga, OpenShift Dedicated cluster (iyo inomhanya quay.io) yakatanga kuvandudza kune shanduro 4.3.19. Sezvo quay.io ichimhanya paRed Hat OpenShift Yakatsaurirwa (OSD), zvigadziriso zvenguva dzose zvaive zvechinyakare uye hazvina kumbokonzera matambudziko. Zvakare, mumwedzi mitanhatu yapfuura, takakwidziridza maQuay masumbu kakawanda pasina kukanganiswa mubasa.
Patakanga tichiedza kudzoreredza sevhisi, mamwe mainjiniya akatanga kugadzirira boka idzva reOSD neshanduro yapfuura yesoftware, kuitira kuti kana chimwe chinhu chaitika, vagone kuendesa zvese pairi.
Root Cause Analysis
Chiratidzo chikuru chekukundikana kwaive kuwanda kwemakumi ezviuru zvekubatanidza dhatabhesi, izvo zvakaita kuti chiitiko cheMySQL chisashande. Izvi zvakaita kuti zviome kuziva dambudziko racho. Isu takaisa muganho pahuwandu hwehuwandu hwekubatanidza kubva kune vatengi kubatsira timu yeSRE kuongorora nyaya. Hatina kuona chero traffic isina kujairika kune dhatabhesi: chokwadi, zvikumbiro zvizhinji zvakaverengwa, uye vashoma chete ndivo vainyora.
Isu takaedzawo kuona pateni mudhatabhesi traffic inogona kukonzera iyi avalanche. Nekudaro, isu hatina kuwana chero mapatani mumatanda. Tichiri kumirira sumbu idzva rine OSD 4.3.18 kuti rigadzirire, takaramba tichiedza kuburitsa quay.io pods. Pese apo cluster yaisvika kuzara, dhatabhesi raitooma. Izvi zvaireva kuti zvaive zvakafanira kutangazve iyo RDS muenzaniso mukuwedzera kune ese quay.io pods.
Pakazosvika manheru, takagadzirisa sevhisi mukuverenga-chete modhi uye takadzima mabasa mazhinji asina kukosha sezvaaigona (semuenzaniso, namespace kuunganidzwa kwemarara) kuderedza mutoro padhatabhesi. Kutonhora kwakamira asi chikonzero hachina kuwanikwa. Iyo itsva OSD cluster yakanga yakagadzirira, uye isu takatama sevhisi, yakabatana traffic uye kuenderera mberi nekutarisa.
Quay.io yakashanda zvakatsiga pane itsva OSD cluster, saka takadzokera kumatanda edatabase, asi hatina kuwana kuwirirana kwaizotsanangura kuvharika. OpenShift mainjiniya akashanda nesu kuti tinzwisise kana shanduko muRed Hat OpenShift 4.3.19 inogona kukonzera matambudziko neQuay. Zvisinei, hapana chakawanikwa, uye Zvakanga zvisingakwanisi kubereka dambudziko mumamiriro e laboratory.
Kukundikana kwechipiri
Musi waMay 28, nguva pfupi masikati asati asvika EDT, quay.io yakaputsika zvakare nechiratidzo chimwe chete: database yakavharwa. Uye zvakare takakanda kuedza kwedu kwese mukuferefeta. Chokutanga pane zvose, zvakanga zvakakodzera kudzorera basa racho. Zvisinei panguva ino kutangazve RDS uye kutangazve quay.io pods hapana chaakaita: imwe avalanche yekubatanidza yakakurira base. Asi nei?
Quay yakanyorwa muPython uye yega yega pod inoshanda senge monolithic mudziyo mumwe. Iyo mudziyo yekumhanyisa nguva inomhanyisa akawanda akafanana mabasa panguva imwe chete. Isu tinoshandisa raibhurari gevent pasi gunicorn kugadzirisa zvikumbiro zvewebhu. Kana chikumbiro chikauya muQuay (kuburikidza neyedu API, kana kuburikidza neDocker's API), inopihwa mushandi wegevent. Kazhinji mushandi uyu anofanira kubata database. Mushure mekutadza kwekutanga, takaona kuti vashandi ve gevent vaibatana nedhatabhesi vachishandisa default marongero.
Tichifunga huwandu hwakakosha hweQuay pods uye zviuru zvezvikumbiro zvinouya pasekondi, nhamba huru yekubatanidza dhatabhesi inogona kukurira chiitiko cheMySQL. Nekuda kwekutarisa, zvaizivikanwa kuti Quay inoita avhareji yezviuru zvishanu zvikumbiro pasekondi. Nhamba yekubatanidza kune dhatabhesi yaive yakafanana. 5 zviuru zvekubatanidza zvaive mukati mekugona kweiyo RDS muenzaniso (isingakwanise kutaurwa nezve makumi ezviuru). Nekuda kwechimwe chikonzero kwaive nemaspikes asingatarisirwe muhuwandu hwekubatanidza, zvisinei, hatina kuona chero kuwirirana nezvikumbiro zvinouya.
Panguva ino takanga takatsunga kutsvaga uye kubvisa kwaikonzera dambudziko, uye kwete kugumira pachedu kutangazve. KuQuay codebase shanduko dzakaitwa kudzikamisa nhamba yekubatanidza kune dhatabhesi yemushandi wega wega gevent. Iyi nhamba yakava parameter mukugadzirisa: zvakave zvichiita kuti ichinje panhunzi pasina kuvaka mufananidzo mutsva wemudziyo. Kuti tizive kuti mangani ekubatanidza anogona kubatwa nemazvo, takamhanyisa bvunzo dzinoverengeka munzvimbo yekutandarira, tichiisa hunhu hwakasiyana kuti tione kuti izvi zvingakanganisa sei mamiriro ekuyedza. Somugumisiro, zvakaonekwa kuti Quay inotanga kukanda zvikanganiso zve502 apo nhamba yekubatanidza inodarika zviuru gumi.
Isu takabva taendesa iyi vhezheni itsva mukugadzira uye takatanga kutarisa iyo database yekubatanidza hurongwa. Kare, chigadziko chakavharwa pasi mushure memaminitsi makumi maviri. Mushure memaminitsi makumi matatu asina dambudziko takava netariro, uye awa gare gare takava nechivimbo. Takadzoreredza traffic kune saiti uye takatanga postmortem ongororo.
Takwanisa kunzvenga dambudziko rinotungamira kuvharika, hatina kuwana zvikonzero zvayo chaizvo. Yakasimbiswa kuti haina hukama nechero shanduko muOpenShift 4.3.19, sezvo chinhu chimwe chete chakaitika pane vhezheni 4.3.18, iyo yakamboshanda neQuay pasina matambudziko.
Zvaiva pachena kuti pane chimwe chinhu chakanga chakavandira muchikwata ichi.
Detailed Study
Quay.io yakashandisa iyo default marongero kuti abatanidze kune dhatabhesi kwemakore matanhatu pasina matambudziko. Chii chakachinja? Zviri pachena kuti nguva yese iyi traffic pa quay.io yanga ichikura zvishoma nezvishoma. Muchiitiko chedu, zvaiita sokunge kuti kumwe kukosha kwakanga kwasvikwa, uko kwakashanda sechinhu chokunyandura kukukurwa kwezvibatanidza. Takaramba tichidzidza zvinyorwa zvemashoko mushure mekukundikana kwechipiri, asi hatina kuwana chero maitiro kana hukama huri pachena.
Zvichakadaro, timu yeSRE yanga ichishanda mukuvandudza kwekuonekwa kweQuay uye hutano hwesevhisi. Metrics matsva uye dashboards akaiswa, kuratidza kuti ndedzipi zvikamu zveQuay zvakanyanya kudiwa kubva kune vatengi.
Quay.io yakashanda zvakanaka kusvika June 9. Mangwanani ano (EDT) takaona zvakare kuwedzera kukuru kwenhamba yekubatanidza database. Panguva iyi pakanga pasina nguva, sezvo parameter itsva yakaganhurira nhamba yavo uye haina kuvabvumira kudarika MySQL throughput. Nekudaro, kweinenge hafu yeawa, vashandisi vazhinji vakaona kunonoka kuita kwequay.io. Isu takakurumidza kuunganidza data rese rinogoneka tichishandisa akawedzera ekutarisa maturusi. Pakarepo muenzaniso wakabuda.
Nguva pfupi isati yatanga kuvhiya kwekubatanidza, nhamba huru yezvikumbiro yakaitwa kuApp Registry API. App Registry chinhu chidiki-chinozivikanwa chequay.io. Iyo inokutendera iwe kuchengetedza zvinhu senge Helm machati uye midziyo ine yakapfuma metadata. Vazhinji vashandisi vequay.io havashande nechinhu ichi, asi Red Hat OpenShift inoishandisa nesimba. OperatorHub sechikamu cheOpenShift inochengetedza vese vashandisi muApp Registry. Ava vashandisi vanoumba hwaro hweiyo OpenShift workload ecosystem uye shamwari-centric inoshanda modhi (Zuva 2 mashandiro).
Imwe neimwe OpenShift 4 cluster inoshandisa maoperator kubva kune yakavakirwa-mukati OperatorHub kuburitsa katalog yevashandisi varipo kuisirwa uye nekupa zvigadziriso kune avo vakatoiswa. Nekukurumbira kuri kukura kweOpenShift 4, huwandu hwemasumbu pairi kutenderera pasirese hwawedzerawo. Rimwe nerimwe remasumbu aya rinodhawunirodha zvirimo opareta kuti imhanye yakavakirwa-mukati OperatorHub, uchishandisa iyo App Registry mukati quay.io seyokumashure. Mukutsvaga kwedu kunobva dambudziko, takapotsa chokwadi chekuti OpenShift payakakura zvishoma nezvishoma mukuzivikanwa, mutoro pane imwe yeasingawanzo shandiswa quay.io mabasa wakawedzera..
Isu takaita ongororo yeApp Registry yekukumbira traffic uye takatarisa iyo registry kodhi. Pakarepo, zvikanganiso zvakaratidzwa, nekuda kwekuti mibvunzo kune database haina kuumbwa zvakakwana. Kana mutoro wacho wakaderera, havana kuunza chinetso, asi apo mutoro wacho wakawedzera, ivo vakava manyuko ezvinetso. Registry yeApp yakave nemagumo maviri ane dambudziko asina kupindura zvakanaka pakuwedzera mutoro: yekutanga yakapa runyoro rwemapakeji ese mune repository, yechipiri yakadzosa mabhurobhu ese pasuru.
Kubviswa kwezvikonzero
Muvhiki rinotevera takapedza kukwenenzvera kodhi yeApp Registry pachayo nenzvimbo yayo. Zviripachena kusashanda SQL mibvunzo yakagadziridzwa uye isingakoshi yekuraira mafoni yakabviswa tar (yaiitwa pese paitorwa mabhurobhu), caching yaiwedzerwa pese pazvaibvira. Isu takazoitisa kuyedza kwakakura kwekuita uye tikaenzanisa kumhanya kweApp Registry isati yatanga uye mushure mekuchinja.
API zvikumbiro zvaimbotora kusvika hafu yeminiti zvapedzwa mumamilliseconds. Vhiki rakatevera takatumira shanduko mukugadzira, uye kubva ipapo quay.io yanga ichishanda zvakatsiga. Munguva iyi, pakanga paine akati wandei akapinza spikes mutraffic paApp Registry endpoint, asi kuvandudzwa kwakaitwa kwakadzivirira kubuda kwedatabase.
Tadzidzei?
Zviri pachena kuti chero sevhisi inoedza kudzivirira kuderera. Kwatiri isu, tinotenda kuti kubuda kwazvino kwabatsira kuita kuti quay.io ive nani. Takadzidza zvidzidzo zvishoma zvinokosha zvatinoda kugovera:
- Dhata pamusoro pekuti ndiani anoshandisa sevhisi yako uye sei haina kumbobvira yakanyanyisa. Nekuti Quay "ichangoshanda," isu hatina kumbopedza nguva tichikwenenzvera traffic uye kubata mutoro. Zvose izvi zvakagadzira pfungwa yenhema yekuchengeteka iyo sevhisi inogona kukwira nekusingaperi.
- Kana sevhisi yadzikira, kuita kuti idzoke uye kushanda ndicho chinhu chinonyanya kukosha.. Nekuti Quay yakaramba ichitambura kubva pakakiyiwa dhatabhesi panguva yekutanga kubuda, maitiro edu akajairwa akange asina mhedzisiro yaaida uye hatina kukwanisa kudzoreredza sevhisi tichiashandisa. Izvi zvakaita kuti pave nemamiriro ezvinhu apo nguva yaifanira kupedzwa kuongorora nekuunganidza data netarisiro yekuwana mudzi wechikonzero - pachinzvimbo chekutarisa kuedza kwese kudzoreredza kushanda.
- Ongorora kushanda kwesevhisi yega yega. Vatengi vaisawanzoshandisa App Registry, saka yanga isiri yekutanga kuchikwata chedu. Kana mamwe maficha echigadzirwa asisashandiswe, mabhugi avo haawanzo kuoneka, uye vanogadzira vanomira kutarisa kodhi. Zviri nyore kuwira munyama yepfungwa isiriyo yekuti iyi ndiyo nzira yazvinofanira kuvaβkusvikira kamwe kamwe basa iroro razviwana riri pakati pechiitiko chikuru.
Chii chinotevera?
Basa rekuona kugadzikana kwebasa harimbomira uye isu tiri kugara tichirivandudza. Sezvo huwandu hwetraffic hunoramba huchikura pa quay.io, tinoziva kuti isu tine basa rekuita zvese zvatinogona kuti tirarame zvinoenderana nevatengi vedu. Naizvozvo, parizvino tiri kushanda pane anotevera mabasa:
- Tumira kuverenga-chete dhatabhesi replicas kubatsira sevhisi kubata yakakodzera traffic muchiitiko chematambudziko neiyo yekutanga RDS muenzaniso.
- Kugadziridza muenzaniso weRSS. Iyo yazvino vhezheni pachayo haisi iyo dambudziko. Asi, tinongoda kubvisa nzira yenhema (iyo yatakatevera panguva yekukundikana); Kuchengeta software iri kusvika parizvino kunobvisa chimwe chinhu muchiitiko chekudzima mune ramangwana.
- Kuwedzera caching pane yese cluster. Isu tinoramba tichitsvaga nzvimbo uko caching inogona kuderedza mutoro pane database.
- Kuwedzera web application firewall (WAF) kuti uone kuti ndiani ari kubatana nequay.io uye nei.
- Kutanga nekuburitswa kunotevera, Red Hat OpenShift masumbu achasiya App Registry vachifarira Operator Catalogs zvichienderana nemidziyo yemifananidzo inowanikwa pa quay.io.
- Kutsiva kwenguva refu kweApp Registry kunogona kunge kuri tsigiro yeOpen Container Initiative (OCI) zvinyorwa zvekugadzira. Ikozvino inoshandiswa seyakasikwa Quay mashandiro uye ichave iripo kune vashandisi kana iyo yakatarwa pachayo yapera.
Zvese zviri pamusoro apa chikamu cheRed Hat chirikuenderera mberi chekudyara mu quay.io sezvatinofamba kubva kudiki "yekutanga-maitiro" timu kuenda kune yakakura SRE-inotyairwa chikuva. Tinoziva kuti vazhinji vevatengi vedu vanovimba ne quay.io mubasa ravo rezuva nezuva (kusanganisira Red Hat!) uye tinoedza kuva pachena sezvinobvira pamusoro pekubuda kwemazuva ano uye kuedza kuenderera mberi kwekuvandudza.
PS kubva kumushanduri
Verenga zvakare pablog yedu:
- Β«";
- Β«";
- Β«".
Source: www.habr.com