Thumela i-Mortem ekungatholakalini kwe-Quay.io

Qaphela. transl.: ekuqaleni kuka-Agasti, i-Red Hat yakhuluma esidlangalaleni mayelana nokuxazulula izinkinga zokufinyeleleka abasebenzisi besevisi yayo ababehlangabezane nazo ezinyangeni ezedlule. I-Quay.io (kusekelwe ekubhaliseni izithombe zeziqukathi, ezitholwe yinkampani kanye nokuthengwa kwe-CoreOS). Kungakhathaleki ukuthi unentshisekelo kangakanani kule sevisi, indlela onjiniyela be-SRE benkampani abayithathile ukuze bahlole futhi baqede izimbangela zengozi iyafundisa.

Thumela i-Mortem ekungatholakalini kwe-Quay.io

Ngomhlaka-19 Meyi, ekuseni kakhulu (Isikhathi Sokukhanya SaseMpumalanga, EDT), isevisi ye-quay.io yaphahlazeka. Ingozi ithinte kokubili abathengi be-quay.io namaphrojekthi womthombo ovulekile asebenzisa i-quay.io njengenkundla yokwakha nokusabalalisa isofthiwe. I-Red Hat ikwazisa ukwethenjwa kwakho kokubili.

Ithimba lonjiniyela be-SRE lazibandakanya ngokushesha futhi lazama ukusimamisa insizakalo ye-Quay ngokushesha okukhulu. Kodwa-ke, ngenkathi benza lokhu, amaklayenti alahlekelwa amandla okuphusha izithombe ezintsha, futhi ngezikhathi ezithile kuphela ayekwazi ukudonsa ezikhona. Ngesizathu esingaziwa, isizindalwazi se-quay.io siye savinjwa ngemva kokukala isevisi ngokugcwele.

Β«Yini eguqukile?"- lo ngumbuzo wokuqala ovame ukubuzwa ezimweni ezinjalo. Siqaphele ukuthi ngaphambi nje kwenkinga, i-OpenShift Dedicated cluster (esebenzisa i-quay.io) yaqala ukuthuthukela kunguqulo 4.3.19. Njengoba i-quay.io isebenza ku-Red Hat OpenShift Dedicated (OSD), ukubuyekezwa okuvamile bekuyinjwayelo futhi akuzange kubangele izinkinga. Ngaphezu kwalokho, ezinyangeni eziyisithupha ezedlule, siye sathuthukisa amaqoqo e-Quay izikhathi ezimbalwa ngaphandle kokuphazamiseka enkonzweni.

Ngenkathi sizama ukubuyisela insizakalo, abanye onjiniyela baqala ukulungisa iqoqo elisha le-OSD ngenguqulo yangaphambilini yesoftware, ukuze kuthi uma kwenzeka okuthile, bakwazi ukusebenzisa konke okukuyo.

Ukuhlaziya Imbangela

Uphawu oluyinhloko lokwehluleka kwaba ukunqwabelana kwamashumi ezinkulungwane zokuxhunywa kwesizindalwazi, okunikeze isibonelo se-MySQL singasebenzi ngempumelelo. Lokhu kwenza kwaba nzima ukuhlonza inkinga. Sibeke umkhawulo enanini eliphezulu loxhumo oluvela kumakhasimende ukuze sisize ithimba le-SRE lihlole udaba. Asibonanga ithrafikhi engajwayelekile kusizindalwazi: empeleni, izicelo eziningi zifundiwe, futhi ezimbalwa kuphela ezibhaliwe.

Siphinde sazama ukuhlonza iphethini ku-traffic yesizindalwazi engase ibangele lokhu kuqubuka. Nokho, asikwazanga ukuthola amaphethini kulogi. Ngenkathi silinde iqoqo elisha eline-OSD 4.3.18 ukuthi lilungile, saqhubeka nokuzama ukwethula ama-quay.io pods. Ngaso sonke isikhathi uma iqoqo lifinyelela umthamo ogcwele, isizindalwazi sasiba yiqhwa. Lokhu kwakusho ukuthi kwakudingeka ukuqalisa kabusha isenzakalo se-RDS ngaphezu kwawo wonke ama-quay.io pod.

Kusihlwa, sazinzisa isevisi kumodi yokufunda kuphela futhi sakhubaza imisebenzi eminingi engabalulekile ngangokunokwenzeka (isibonelo, ukuqoqwa kukadoti wendawo yegama) ukuze kwehliswe umthwalo kusizindalwazi. Ukubanda kumisiwe kodwa isizathu asizange sitholakale. Iqoqo elisha le-OSD lalilungile, futhi sathutha isevisi, saxhuma ithrafikhi futhi saqhubeka nokuqapha.

I-Quay.io isebenze ngokuzinza eqoqweni elisha le-OSD, ngakho-ke sibuyele kulogi egciniwe, kodwa asikwazanga ukuthola ukuhlobana okungachaza ukuvinjwa. Onjiniyela be-OpenShift basebenze nathi ukuze baqonde ukuthi izinguquko ku-Red Hat OpenShift 4.3.19 zingadala izinkinga nge-Quay. Nokho, akukho lutho olutholakele, futhi Kwakungenakwenzeka ukukhiqiza kabusha inkinga ezimweni zaselabhorethri.

Ukwehluleka kwesibili

Ngomhla ka-May 28, ngaphambi nje kwasemini ye-EDT, i-quay.io yaphahlazeka futhi ngophawu olufanayo: isizindalwazi savinjwa. Futhi saphinde saphonsa yonke imizamo yethu ophenyweni. Okokuqala, kwakudingeka ukubuyisela isevisi. Nokho kulokhu ukuqalisa kabusha i-RDS nokuqalisa kabusha ama-quay.io pods akwenzanga lutho: enye i-avalanche yokuxhumana idlule isisekelo. Kodwa kungani?

I-Quay ibhalwe nge-Python futhi i-pod ngayinye isebenza njengesiqukathi esisodwa se-monolithic. Isikhathi sokusebenza sesiqukathi senza imisebenzi eminingi efanayo ngesikhathi esisodwa. Sisebenzisa umtapo wolwazi gevent ngaphansi gunicorn ukucubungula izicelo zewebhu. Lapho isicelo singena ku-Quay (nge-API yethu, noma nge-Docker's API), sinikezwa isisebenzi se-gevent. Ngokuvamile lesi sisebenzi kufanele sithinte isizindalwazi. Ngemuva kokwehluleka kokuqala, sithole ukuthi abasebenzi be-gevent babexhuma kusizindalwazi besebenzisa izilungiselelo ezizenzakalelayo.

Uma kubhekwa inani elibalulekile lama-Quay pods nezinkulungwane zezicelo ezingenayo ngomzuzwana, inani elikhulu lokuxhunywa kwedathabhesi lingasibekela isibonelo se-MySQL. Ngenxa yokuqapha, bekwaziwa ukuthi i-Quay icubungula isilinganiso sezicelo eziyizinkulungwane ezi-5 ngomzuzwana. Inani lokuxhunywa kusizindalwazi lalicishe lifane. Izinkulungwane ezi-5 zokuxhuma bezingaphakathi kwamandla esibonelo sethu se-RDS (okungeke kushiwo mayelana ngamashumi ezinkulungwane). Ngesizathu esithile kube nama-spikes angalindelekile enanini loxhumo, nokho, asizange sikuqaphele noma yikuphi ukuhlobana nezicelo ezingenayo.

Kulokhu besizimisele ukuthola nokuqeda umthombo wenkinga, futhi singagcini ngokuqalisa kabusha. Ku-Quay codebase kwenziwa izinguquko ukuze kukhawulwe inani lokuxhunywa kusizindalwazi somsebenzi ngamunye gevent. Le nombolo yaba ipharamitha ekucushweni: kwaba nokwenzeka ukuyishintsha empukaneni ngaphandle kokwakha isithombe sesitsha esisha. Ukuthola ukuthi kungakanani ukuxhumana okungasingathwa ngendlela engokoqobo, senze izivivinyo ezimbalwa endaweni yesiteji, sibeka amanani ahlukene ukuze sibone ukuthi lokhu kuzozithinta kanjani izimo zokuhlola umthwalo. Ngenxa yalokho, kwatholakala ukuthi I-Quay iqala ukuphonsa amaphutha angu-502 lapho inani lokuxhuma lidlula izinkulungwane eziyi-10.

Ngokushesha safaka le nguqulo entsha ekukhiqizweni futhi saqala ukuqapha uhlelo lokuxhuma kusizindalwazi. Esikhathini esidlule, isisekelo sasivalelwa phansi ngemva kwemizuzu engaba ngu-20. Ngemva kwemizuzu engu-30 engenazinkinga saba nethemba, futhi ngemva kwehora saba nokuzethemba. Sibuyisele ithrafikhi kusayithi futhi saqala ukuhlaziywa kwe-postmortem.

Njengoba ukwazile ukweqa inkinga eholela ekuvinjweni, asikazitholi izizathu zayo zangempela. Kuqinisekisiwe ukuthi ayihlobene nanoma yiziphi izinguquko ku-OpenShift 4.3.19, njengoba kwenzeka okufanayo kunguqulo 4.3.18, eyayisebenza ngaphambili ne-Quay ngaphandle kwezinkinga.

Ngokusobala kwakukhona enye into eyayicashe kuleli qoqo.

Isifundo Esiningiliziwe

I-Quay.io isebenzise izilungiselelo ezizenzakalelayo ukuze ixhume kusizindalwazi iminyaka eyisithupha ngaphandle kwezinkinga. Yini eshintshile? Kuyacaca ukuthi sonke lesi sikhathi ithrafikhi ku-quay.io ibilokhu ikhula kancane. Esimweni sethu, kubonakale sengathi inani elithile lifinyelelwe, elisebenze njengesicupho se-avalanche yokuxhuma. Siqhubekile nokutadisha izingodo zesizindalwazi ngemva kokuhluleka kwesibili, kodwa asizange sithole amaphethini noma ubudlelwano obusobala.

Okwamanje, ithimba le-SRE belisebenzela ukuthuthukisa ukubonakala kwesicelo sika-Quay kanye nempilo yesevisi iyonke. Amamethrikhi amasha namadeshibhodi asetshenzisiwe, okubonisa ukuthi yiziphi izingxenye ze-Quay ezidingeka kakhulu kumakhasimende.

I-Quay.io yasebenza kahle kwaze kwaba uJuni 9. Namuhla ekuseni (EDT) siphinde sabona ukwanda okukhulu kwenani lokuxhunywa kwedathabhesi. Kulokhu besingekho isikhathi sokuphumula, njengoba ipharamitha entsha ikhawulele inombolo yabo futhi ayizange ibavumele ukuthi badlule i-MySQL throughput. Nokho, cishe isigamu sehora, abasebenzisi abaningi baphawule ukusebenza kancane kwe-quay.io. Siqoqe ngokushesha yonke idatha engenzeka sisebenzisa amathuluzi okuqapha angeziwe. Kusenjalo kwavela iphethini.

Ngaphambi nje kokwanda kokuxhumeka, inani elikhulu lezicelo lenziwe ku-API Yokubhaliswa Kwezinhlelo Zokusebenza. I-App Registry isici esaziwa kancane se-quay.io. Ikuvumela ukuthi ugcine izinto ezifana namashadi e-Helm neziqukathi ezinemethadatha ecebile. Iningi labasebenzisi be-quay.io alisebenzi ngalesi sici, kodwa i-OpenShift yesigqoko esibomvu iyasisebenzisa. I-OperatorHub njengengxenye ye-OpenShift igcina wonke ama-opharetha Kubhalisi Lohlelo Lokusebenza. Laba opharetha bakha isisekelo se-OpenShift yomthwalo wemvelo womthwalo wokusebenza kanye nemodeli yokusebenza egxile kozakwethu (ukusebenza kosuku lwesi-2).

Iqoqo ngalinye le-OpenShift 4 lisebenzisa ama-opharetha asuka ku-OperatorHub eyakhelwe ngaphakathi ukuze ashicilele ikhathalogi yama-opharetha atholakalayo ukuze afakwe futhi anikeze izibuyekezo kulabo asebevele bafakiwe. Ngokuthandwa okukhulayo kwe-OpenShift 4, isibalo samaqoqo kuyo emhlabeni jikelele senyukile. Ngayinye yalezi qoqo ilanda okuqukethwe kuka-opharetha ukuze iqalise i-OperatorHub eyakhelwe ngaphakathi, isebenzisa Isibhalisi Sohlelo Lokusebenza ngaphakathi kwe-quay.io njengesingemuva. Ekusesheni kwethu umthombo wenkinga, siphuthelwe iqiniso lokuthi njengoba i-OpenShift ikhula kancane kancane ekudumeni, umthwalo komunye wemisebenzi engavamile ukusetshenziswa ye-quay.io nawo wanda..

Senze ukuhlaziya okuthile kwethrafikhi yesicelo sesibhalisi sohlelo lokusebenza futhi sabheka ikhodi yokubhalisa. Ngokushesha, ukushiyeka kwembulwa, ngenxa yokuthi imibuzo ku-database ayizange yakheke kahle. Lapho umthwalo usuphansi, abazange babangele noma iyiphi inkinga, kodwa lapho umthwalo ukhula, baba umthombo wezinkinga. I-App Registry ivele yaba neziphetho ezimbili eziyinkinga ezingazange zisabele kahle ekukhuphuleni umthwalo: eyokuqala inikeze uhlu lwawo wonke amaphakheji endaweni, eyesibili ibuyise wonke ama-blob ephakheji.

Ukuqedwa kwezinkinga

Evikini elilandelayo sichithe sithuthukisa ikhodi Yesibhalisi Sohlelo Lokusebenza uqobo kanye nendawo yayo. Ngokusobala imibuzo ye-SQL engasebenzi kahle iphinde yasetshenziswa futhi izingcingo zokuyalelwa ezingadingekile zaqedwa tar (yayiqhutshwa njalo lapho ama-blobs ebuyiselwa), ukugcinwa kwesikhashana kwakungezwa noma nini lapho kungenzeka khona. Sibe sesihlola ukusebenza okubanzi futhi saqhathanisa isivinini Sokubhaliswa Kwezinhlelo Zokusebenza ngaphambi nangemuva kwezinguquko.

Izicelo ze-API ngaphambilini ezithathe kufika kuhhafu weminithi manje seziqediwe ngama-millisecond. Ngesonto elilandelayo sithumele izinguquko ekukhiqizeni, futhi kusukela lapho i-quay.io ibisebenza ngokuzinzile. Phakathi nalesi sikhathi, kube nokukhuphuka okubukhali okumbalwa kwethrafikhi endaweni yokugcina yesibhalisi sohlelo lokusebenza, kodwa ukuthuthukiswa kwenze kwanqanda ukuphela kwesizindalwazi.

Yini esiyifundile?

Kuyacaca ukuthi noma iyiphi isevisi izama ukugwema isikhathi sokuphumula. Esimeni sethu, sikholwa ukuthi ukuphela kwakamuva kusize ukwenza i-quay.io ibe ngcono. Sifunde izifundo ezimbalwa ezibalulekile esingathanda ukwabelana ngazo:

  1. Idatha mayelana nokuthi ubani osebenzisa isevisi yakho nokuthi kanjani ayilokothi ibe ngaphezulu. Ngenxa yokuthi i-Quay β€œisanda kusebenza,” asizange kudingeke ukuthi sichithe isikhathi sithuthukisa ithrafikhi futhi silawula ukulayisha. Konke lokhu kudale umuzwa wokuvikeleka okungamanga ukuthi isevisi ingakhula unomphela.
  2. Lapho inkonzo yehla, ukuyibuyisela futhi isebenze kuyinto ehamba phambili.. Ngenxa yokuthi i-Quay yaqhubeka ihlushwa isizindalwazi esikhiyiwe ngesikhathi sokucisha kokuqala, izinqubo zethu ezivamile azizange zibe nomphumela owawuhlosiwe futhi asikwazanga ukubuyisela isevisi ngokuzisebenzisa. Lokhu kwaholela esimweni lapho kudingeke ukuba kusetshenziswe isikhathi sokuhlaziya nokuqoqwa kolwazi ngethemba lokuthola umnyombo - esikhundleni sokugxila kuyo yonke imizamo yokubuyisela ukusebenza.
  3. Linganisa umthelela wesici ngasinye sesevisi. Amaklayenti awavamile ukusebenzisa Isibhalisi Sohlelo Lokusebenza, ngakho bekungeyona into ehamba phambili eqenjini lethu. Uma ezinye izici zomkhiqizo zisetshenziswa kancane, iziphazamisi zazo azivamile ukuvela, futhi onjiniyela bayayeka ukuqapha ikhodi. Kulula ukuba yisisulu sombono oyiphutha wokuthi yile ndlela okufanele kube ngayoβ€”kuze kube yilapho kungazelelwe lowo msebenzi uzithola usenkabeni yesigameko esikhulu.

Yini okulandelayo?

Umsebenzi wokuqinisekisa ukuzinza kwesevisi awupheli futhi sihlala siwuthuthukisa. Njengoba amanani ethrafikhi eqhubeka nokukhula ku-quay.io, siyaqaphela ukuthi sinesibopho sokwenza konke esingakwenza ukuze siphile ngokuvumelana nokusethemba kwamakhasimende ethu. Ngakho-ke, okwamanje sisebenzela le misebenzi elandelayo:

  1. Sebenzisa ama-replicas esizindalwazi sokufunda kuphela ukuze usize isevisi isingathe ithrafikhi efanele uma kunezinkinga ngesenzakalo esiyinhloko se-RDS.
  2. Ibuyekeza isibonelo se-RDS. Inguqulo yamanje ngokwayo akuyona inkinga. Kunalokho, sifuna ukususa umkhondo wamanga (esiwulandele ngesikhathi sokwehluleka); Ukugcina isofthiwe isesikhathini kuzosusa enye into uma kwenzeka kunqamuka esikhathini esizayo.
  3. Ukulondoloza okwesikhashana okungeziwe kulo lonke iqoqo. Siyaqhubeka nokubheka izindawo lapho ukugcinwa kwesikhashana kunganciphisa khona umthwalo kusizindalwazi.
  4. Ukwengeza i-firewall yesicelo sewebhu (i-WAF) ukuze ubone ukuthi ubani oxhuma ku-quay.io nokuthi kungani.
  5. Kusukela ekukhishweni okulandelayo, amaqoqo e-Red Hat OpenShift azoshiya Isibhalisi Sohlelo Lokusebenza ukuze avune Amakhathalogi Omsebenzi asekelwe ezithombeni zesiqukathi ezitholakala ku-quay.io.
  6. Ukushintshwa kwesikhathi eside Kwesibhalisi Sohlelo Lokusebenza kungase kube usekelo lwezicaciso ze-artifact ze-Open Container Initiative (OCI). Okwamanje isetshenziswa njengomsebenzi we-Quay womdabu futhi izotholakala kubasebenzisi uma ukucaciswa ngokwako kuqedwa.

Konke lokhu okungenhla kuyingxenye yokutshalwa kwezimali okuqhubekayo kwe-Red Hat ku-quay.io njengoba sisuka eqenjini elincane "lesitayela sokuqalisa" siya endaweni yesikhulumi esivuthiwe esiqhutshwa yi-SRE. Siyazi ukuthi amakhasimende ethu amaningi athembele ku-quay.io emsebenzini wawo wansuku zonke (okuhlanganisa ne-Red Hat!) futhi sizama ukubeka obala ngangokunokwenzeka mayelana nokucisha kwakamuva nemizamo eqhubekayo yokuthuthukisa.

I-PS evela kumhumushi

Funda futhi kubhulogi yethu:

Source: www.habr.com

Engeza amazwana