Igama lam ndinguAnton Baderin. Ndisebenza kwiZiko leTekhnoloji ePhakamileyo kwaye ndenza ulawulo lwenkqubo. Kwinyanga edlulileyo, inkomfa yethu yenkampani iphelile, apho sabelana ngamava ethu aqokelelweyo kunye noluntu lwe-IT yesixeko sethu. Ndathetha ngokubeka iliso kwizicelo zewebhu. Izinto eziphathekayo zenzelwe inqanaba eliphantsi okanye eliphakathi, ongazange akhe le nkqubo ukususela ekuqaleni.
Isiseko esisisiseko sayo nayiphi na inkqubo yokubeka iliso kukusombulula iingxaki zoshishino. Ukubeka iliso ngenjongo yokubeka iliso akunamdla nakubani na. Lifuna ntoni ishishini? Ukuze yonke into isebenze ngokukhawuleza kwaye ngaphandle kweempazamo. Amashishini afuna ukuqhubela phambili, ukuze thina ngokwethu sichonge iingxaki kwinkonzo kwaye sizilungise ngokukhawuleza. Ezi, eneneni, ziingxaki endizisombulule kunyaka ophelileyo kwiprojekthi yomnye wabathengi bethu.
Malunga neprojekthi
Le projekthi yenye yeenkqubo ezinkulu zokunyaniseka kweli lizwe. Sinceda amatyathanga okuthengisa anyuse amaxesha okuthengisa ngokusebenzisa izixhobo ezahlukeneyo zokuthengisa ezifana namakhadi ebhonasi. Lilonke, le projekthi ibandakanya izicelo ezili-14 ezisebenza kwiiseva ezilishumi.
Ngexesha lenkqubo yodliwano-ndlebe, ndaqaphela ngokuphindaphindiweyo ukuba ii-admins azisoloko zisondela ngokufanelekileyo ukubeka iliso kwizicelo zewebhu: ezininzi zisagxila kwiimetriki zenkqubo yokusebenza kwaye ngamanye amaxesha zibeke iliso kwiinkonzo.
Kwimeko yam, inkqubo yokubeka iliso yomthengi ngaphambili yayisekelwe kwi-Icinga. Ayizange isombulule ezi ngxaki zingasentla nangayiphi na indlela. Rhoqo umxhasi ngokwakhe usixelela ngeengxaki, kwaye amaxesha amaninzi, besingenalo ulwazi olwaneleyo lokufikelela emazantsi esizathu.
Ukongezelela, kwakukho ukuqonda okucacileyo kokungabi nto kophuhliso lwalo olongezelelweyo. Ndicinga ukuba aba baqhelene no Icinga bazondiqonda. Ke, sigqibe kwelokuba siyiyile ngokupheleleyo inkqubo yokubeka iliso yesicelo sewebhu yeprojekthi.
Prometheus
Sikhethe iPrometheus ngokusekwe kwizalathisi ezintathu eziphambili:
- Inani elikhulu leemethrikhi ezikhoyo. Kwimeko yethu kukho amawaka angama-60 kubo. Ngokuqinisekileyo, kuyafaneleka ukuqaphela ukuba asisebenzisi uninzi lwazo (mhlawumbi malunga ne-95%). Kwelinye icala, zonke zinexabiso eliphantsi. Kithina, oku kolunye olugqithisileyo xa kuthelekiswa ne-Icinga eyayisetyenziswa ngaphambili. Kuyo, ukongeza i-metrics kwakuyintlungu ethile: ekhoyo yayibiza (jonga nje ikhowudi yomthombo wayo nayiphi na iplagin). Nayiphi na iplagin yayisisikripthi kwiBash okanye kwiPython, ukuqaliswa kwayo kuyabiza ngokwezinto ezisetyenziswayo.
- Le nkqubo idla ubuncwane obuncinci ngokwentelekiso. I-600 MB ye-RAM, i-15% yondoqo omnye kunye nee-IOPS ezimbalwa zanele kuzo zonke iimethrikhi zethu. Ewe kunjalo, kuya kufuneka uqhube abathengisi beemetrics ngaphandle, kodwa zonke zibhalwe kwi-Go kwaye azilambanga kakhulu. Andiqondi ukuba kwiinyani zanamhlanje le yingxaki.
- Inika amandla okufudukela eKubernetes. Ukuqwalasela izicwangciso zomthengi, ukhetho lucacile.
I-ELK
Ngaphambili, asizange siqokelele okanye siqhube iilog. Iintsilelo zicacile kumntu wonke. Sakhetha i-ELK kuba sele sinamava kule nkqubo. Sigcina kuphela iilogi zesicelo apho. Iikhrayitheriya zokukhetha eziphambili zaziyi-full-text search kunye nesantya sayo.
Π‘lickhouse
Ekuqaleni, ukhetho lwawela kwi-InfluxDB. Siye sabona isidingo sokuqokelela iilogi ze-Nginx, izibalo ezivela kwi-pg_stat_statements, kunye nokugcina idatha yembali ye-Prometheus. Asizange siyithande i-Influx kuba ngamaxesha athile yaqala ukusebenzisa inani elikhulu lememori kwaye yawa. Ukongeza, bendifuna ukwenza iqela imibuzo nge-remote_addr, kodwa ukwahlulahlula kule DBMS kungokumathegi kuphela. Iithegi ziyabiza (inkumbulo), inani labo lilinganiselwe ngokwemiqathango.
Saqalisa ukukhangela kwakhona. Into efunekayo yayiyi-database yokuhlalutya kunye nokusetyenziswa okuncinci kwemithombo, ngokukhethekileyo kunye noxinzelelo lwedatha kwidiski.
I-Clickhouse ihlangabezana nazo zonke ezi migaqo, kwaye asizange sizisole ngokukhetha kwethu. Asibhali naziphi na izixa ezingaqhelekanga zedatha kuyo (inani lofakelo limalunga namawaka amahlanu kuphela ngomzuzu).
NewRelic
INewRelic ibisoloko inathi ngokwembali kuba ibilukhetho lomthengi. Siyisebenzisa njenge-APM.
Zabbix
Sisebenzisa i-Zabbix kuphela ukubeka iliso kwiBhokisi emnyama yee-APIs ezahlukeneyo.
Ukuchaza iNdlela yokuBeka iliso
Besifuna ukuwohloza umsebenzi kwaye ngokwenza njalo silungelelanise indlela yokubeka iliso.
Ukwenza oku, ndohlule inkqubo yethu ngokwala manqanaba alandelayo:
- hardware kunye neVMS;
- inkqubo yokwenza;
- iinkonzo zenkqubo, istaki sesoftware;
- isicelo;
- ingqiqo yoshishino.
Kutheni le ndlela iluncedo:
- siyazi ukuba ngubani onoxanduva lomsebenzi kwinqanaba ngalinye kwaye, ngokusekelwe koku, sinokuthumela izilumkiso;
- singasebenzisa ulwakhiwo xa sicinezela izilumkiso-iyakuba ngumnqa ukuthumela isilumkiso malunga nokungafumaneki kwesiseko sedata xa umatshini wenyani uwonke ungekho.
Ekubeni umsebenzi wethu kukuchonga ukuphulwa komsebenzi wenkqubo, kufuneka kwinqanaba ngalinye sigxininise isethi ethile yeemetriki ezifanelekile ukunikela ingqalelo xa ubhala imithetho yokulumkisa. Okulandelayo, masiye kumanqanaba "VMS", "Inkqubo yokusebenza" kunye "neenkonzo zenkqubo, istaki sesoftware".
Oomatshini benyani
Ukusingathwa kusinika iprosesa, idiski, imemori kunye nenethiwekhi. Kwaye saba neengxaki kwezi zimbini zokuqala. Ngoko ke, imilinganiselo:
Ixesha elibiweyo le-CPU - xa uthenga umatshini we-virtual kwi-Amazon (t2.micro, umzekelo), kufuneka uqonde ukuba awubelwanga i-core core processor, kodwa kuphela i-quota yexesha layo. Kwaye xa uyigqibile, iprosesa iya kususwa kuwe.
Le metric ikuvumela ukuba ulandelele amaxesha anje kwaye wenze izigqibo. Umzekelo, ngaba kuyimfuneko ukuthatha irhafu etyebileyo okanye usasaze ukusetyenzwa kwemisebenzi yangasemva kunye nezicelo ze-API kwiiseva ezahlukeneyo?
IOPS + CPU iowait ixesha - ngenxa yesizathu esithile, abaninzi abasingathi bamafu bona ngokungaboneleli nge-IOPS eyaneleyo. Ngaphezu koko, ishedyuli ene-IOPS ephantsi ayikho ingxabano kubo. Ke ngoko, kufanelekile ukuqokelela i-CPU iowait. Ngolu luhlu lweegrafu - kunye ne-IOPS ephantsi kunye ne-I / O ephezulu yokulinda - unokukwazi ukuthetha nokusingatha kunye nokusombulula ingxaki.
kwinkqubo yokusebenza
Imilinganiselo yenkqubo yokusebenza:
- inani lememori ekhoyo kwi-%;
- tshintsha umsebenzi wosetyenziso: vmstat swapin, swapout;
- inani leenodi ezikhoyo kunye nendawo ekhululekileyo kwindlela yefayile kwi %
- umthwalo ophakathi;
- inani loqhagamshelo kwi-tw state;
- contrack ukugcwala kwetafile;
- Umgangatho womnatha unokujongwa usebenzisa i-ss utility, i-iproute2 iphakheji - fumana isalathisi sokuxhunywa kwe-RTT kwimveliso yayo kwaye idibanise nge-dest port.
Kwakhona kwinqanaba lenkqubo yokusebenza sinequmrhu elinjalo njengeenkqubo. Kubalulekile ukuchonga kwinkqubo isethi yeenkqubo ezidlala indima ebalulekileyo ekusebenzeni kwayo. Ukuba, umzekelo, unama-pgpools amaninzi, ngoko kufuneka uqokelele ulwazi ngalunye kubo.
Iseti yeemetrics imi ngolu hlobo lulandelayo:
- ICPU;
- imemori ikakhulu ihlala;
- IO - ngokukhethekileyo kwi-IOPS;
- FileFd - vula kwaye umda;
- ukusilela okukhulu kwephepha - ngale ndlela unokuqonda ukuba yeyiphi inkqubo etshintshwayo.
Sibeka lonke esweni kwi-Docker, kwaye sisebenzisa uMcebisi ukuqokelela idatha yeemetrics. Kwabanye oomatshini sisebenzisa inkqubo-ngaphandle.
Iinkonzo zenkqubo, istaki sesoftware
Isicelo ngasinye sineenkcukacha zaso, kwaye kunzima ukukhetha uluhlu oluthile lweemetrics.
Iseti yendalo yonke yile:
- ixabiso lesicelo;
- inani leempazamo;
- ukubambezeleka;
- ukuhluza.
Eyona mizekelo ibalaseleyo yokubeka iliso kweli nqanaba yiNginx kunye nePostgreSQL.
Eyona nkonzo ilayishiweyo kwinkqubo yethu yidatabase. Kwixesha elidlulileyo, besihlala sineengxaki ekucingeni ukuba idatabase yenza ntoni.
Sabona umthwalo ophezulu kwiidiski, kodwa iilogi ezicothayo azizange zibonise nantoni na. Siyisombulule le ngxaki sisebenzisa i-pg_stat_statements, umbono oqokelela iinkcukacha-manani.
Yiyo yonke into efunwayo admin.
Sakha iigrafu zomsebenzi wokufunda nokubhala izicelo:
Yonke into ilula kwaye icacile, isicelo ngasinye sinombala waso.
Umzekelo obambekayo ngokulinganayo zizigodo zeNginx. Ayimangalisi into yokuba abantu abambalwa bazicazulule okanye bazikhankanye kuluhlu lwezinto ekufuneka zibe nazo. Ifomathi eqhelekileyo ayinalwazi kakhulu kwaye ifuna ukwandiswa.
Mna ngokwam ndongeze isicelo_ixesha, upstream_response_time, body_bytes_sent, request_length, request_id. Siceba ixesha lokuphendula kunye nenani leempazamo:
Sakha iigrafu zexesha lokuphendula kunye nenani leempazamo. Khumbula? Ngaba ndithethe ngemisebenzi yeshishini? Ukuya ngokukhawuleza nangaphandle kweempazamo? Sele siyigubungele le miba ngeetshathi ezimbini. Kwaye ungafowunela abalawuli abasemsebenzini usebenzisa bona.
Kodwa enye ingxaki isele - ukuqinisekisa ukupheliswa ngokukhawuleza kwezizathu zesiganeko.
Isisombululo sesiganeko
Yonke inkqubo ukusuka ekuchongeni ukuya ekusombululeni ingxaki inokwahlulwa ibe ngamanyathelo amaninzi:
- ukuchonga ingxaki;
- isaziso kumlawuli womsebenzi;
- impendulo kwisiganeko;
- ukupheliswa koonobangela.
Kubalulekile ukuba oku kufuneka sikwenze ngokukhawuleza. Kwaye ukuba kwizigaba zokuchonga ingxaki kunye nokuthumela isaziso asikwazi ukufumana ixesha elininzi - imizuzu emibini iya kuchithwa kuyo nayiphi na imeko, emva koko yintsimi engavulwanga yokuphucula.
Masicinge nje ukuba kwakhala umnxeba wegosa elijongene nomsebenzi. Uza kwenza ntoni? Khangela iimpendulo kwimibuzo - yintoni eyaphukileyo, yaphuka phi, indlela yokusabela? Nantsi indlela esiyiphendula ngayo le mibuzo:
Sifaka nje yonke le ngcaciso kwisicatshulwa sesaziso, siyinike ikhonkco kwiphepha le-wiki elichaza indlela yokuphendula le ngxaki, indlela yokuyicombulula kunye nokunyuka kwayo.
Andizange ndithethe nto malunga noluhlu lwesicelo kunye nengqiqo yezoshishino. Ngelishwa, usetyenziso lwethu alukayiphumezi ingqokelela yeemetrics. Umthombo kuphela walo naluphi na ulwazi oluvela kula manqanaba ziilogi.
Amanqaku amabini.
Okokuqala, bhala iilogi ezicwangcisiweyo. Akukho mfuneko yokubandakanya umxholo kwisicatshulwa somyalezo. Oku kwenza kube nzima ukuhlanganisa kunye nokuhlalutya. I-Logstash ithatha ixesha elide ukuyilungisa yonke le nto.
Okwesibini, sebenzisa amanqanaba obunzima ngokuchanekileyo. Ulwimi ngalunye lunomgangatho walo. Ngokomntu, ndohlula amanqanaba amane:
- akukho mpazamo;
- impazamo yecala lomxhasi;
- impazamo isecaleni lethu, asilahlekelwa yimali, asithwali mngcipheko;
- Impazamo ikwicala lethu, silahlekelwa yimali.
Makhe ndishwankathele. Kufuneka uzame ukwakha esweni ngokusekelwe logic ishishini. Zama ukubeka esweni isicelo ngokwaso kwaye usebenze ngeemetrics ezifana nenani lokuthengisa, inani lokubhaliswa kwabasebenzisi abatsha, inani labasebenzisi abasebenzayo ngoku, njalo njalo.
Ukuba ishishini lakho lonke liqhosha elinye kwisikhangeli, kufuneka ujonge ukuba liyacofa kwaye lisebenza ngokufanelekileyo. Konke okunye akunamsebenzi.
Ukuba awunayo le nto, ungazama ukuyibamba kwiilog zesicelo, iilogi zeNginx, njalo njalo, njengoko senzayo. Kufuneka ube kufutshane nesicelo kangangoko.
Iimetriki zenkqubo yokusebenza zibalulekile ngokuqinisekileyo, kodwa ishishini alinamdla kubo, asihlawulwanga.
umthombo: www.habr.com