Igama lami ngingu-Anton Baderin. Ngisebenza e-High Technology Centre futhi ngenza i-system administration. Enyangeni edlule, ingqungquthela yethu yezinkampani yaphela, lapho sabelana ngolwazi esiluqongelele nomphakathi we-IT wedolobha lethu. Ngikhulume ngokuqapha izinhlelo zokusebenza zewebhu. Okubalulekile kwakuhloselwe izinga elincane noma eliphakathi, abangazange bakhe le nqubo kusukela ekuqaleni.
Isisekelo esiyisisekelo sanoma yiluphi uhlelo lokuqapha ukuxazulula izinkinga zebhizinisi. Ukuqapha ngenjongo yokuqapha akuthakazelisi muntu. Lifunani ibhizinisi? Ukuze konke kusebenze ngokushesha futhi ngaphandle kwamaphutha. Amabhizinisi afuna ukusukumela phezulu, ukuze thina ngokwethu sihlonze izinkinga kusevisi futhi sizilungise ngokushesha okukhulu. Lezi, empeleni, yizinkinga engizixazulule unyaka wonke odlule kuphrojekthi yelinye lamakhasimende ethu.
Mayelana nephrojekthi
Lo msebenzi ungolunye lwezinhlelo ezinkulu zokwethembeka ezweni. Sisiza amaketanga okuthengisa akhuphule imvamisa yokuthengisa ngamathuluzi ahlukahlukene okuthengisa njengamakhadi ebhonasi. Sekukonke, iphrojekthi ihlanganisa izinhlelo zokusebenza eziyi-14 ezisebenza kumaseva ayishumi.
Phakathi nenqubo yenhlolokhono, ngiphawule ngokuphindaphindiwe ukuthi abalawuli abasondeli njalo ukuqapha izinhlelo zokusebenza zewebhu ngendlela efanele: abaningi basagxile kumamethrikhi esistimu yokusebenza futhi ngezikhathi ezithile baqapha amasevisi.
Endabeni yami, uhlelo lokuqapha ikhasimende ngaphambilini lwalusekelwe ku-Icinga. Ayizange ixazulule izinkinga ezingenhla nganoma iyiphi indlela. Ngokuvamile iklayenti ngokwalo lalisazisa ngezinkinga, futhi ezikhathini eziningi, sasimane singenayo idatha eyanele ukuze sifinyelele ekugcineni isizathu.
Ngaphezu kwalokho, kwakukhona ukuqonda okucacile kobuze bokuthuthukiswa kwayo okuqhubekayo. Ngicabanga ukuthi abajwayelene ne-Icinga bazongiqonda. Ngakho-ke, sinqume ukuklama kabusha ngokuphelele uhlelo lokuqapha uhlelo lokusebenza lwewebhu lwephrojekthi.
Prometheus
Sikhethe i-Prometheus ngokusekelwe ezinkomba ezintathu eziyinhloko:
- Inani elikhulu lamamethrikhi atholakalayo. Esimeni sethu kukhona izinkulungwane ezingama-60 zazo. Kunjalo, kuyaphawuleka ukuthi asisebenzisi iningi labo (mhlawumbe mayelana 95%). Ngakolunye uhlangothi, zonke zishibhile uma kuqhathaniswa. Kithina, lokhu okunye okwedlulele uma kuqhathaniswa ne-Icinga esetshenziswe ngaphambilini. Kuyo, ukungeza amamethrikhi kwakuwubuhlungu obuthile: lezi ezikhona zazibiza (bheka nje ikhodi yomthombo yanoma iyiphi i-plugin). Noma iyiphi i-plugin bekuyisikripthi ku-Bash noma i-Python, ukwethulwa kwayo okubiza kakhulu ngokwezinsiza ezisetshenzisiwe.
- Lolu hlelo lusebenzisa inani elincane lezinsiza. U-600 MB we-RAM, u-15% wengqikithi eyodwa kanye nama-IOPS ambalwa anele kuwo wonke ama-metrics ethu. Yebo, kufanele usebenzise abathumeli bamamethrikhi, kodwa bonke babhalwe kokuthi Go futhi abalambi kakhulu amandla. Angicabangi ukuthi emaqiniso esimanje lokhu kuyinkinga.
- Inikeza ikhono lokuthuthela e-Kubernetes. Ngokucabangela izinhlelo zekhasimende, ukukhetha kusobala.
Elk
Ngaphambilini, asizange siqoqe noma sicubungule amalogi. Amaphutha asobala kuwo wonke umuntu. Sikhethe i-ELK ngoba besesivele sinesipiliyoni ngalolu hlelo. Sigcina kuphela izingodo zohlelo lokusebenza lapho. Imibandela yokukhetha eyinhloko bekuwukusesha umbhalo ogcwele kanye nesivinini sakhona.
Сlickhouse
Ekuqaleni, ukukhetha kuwele ku-InfluxDB. Sibone isidingo sokuqoqa izingodo ze-Nginx, izibalo ezivela ku-pg_stat_statements, futhi sigcine idatha yomlando ye-Prometheus. Asikuthandanga Ukuthutheleka Kwabantu ngoba ngezikhathi ezithile kwaqala ukudla inani elikhulu lenkumbulo futhi kwaphahlazeka. Ngaphezu kwalokho, bengifuna ukuqoqa imibuzo nge-remote_addr, kodwa ukuqoqa kule DBMS kungomaka kuphela. Omaka bayabiza (inkumbulo), inombolo yabo inomkhawulo ngokwemibandela.
Siqale ukusesha futhi. Okwakudingeka kwakuyisizindalwazi sokuhlaziya esinokusetshenziswa okuncane kwezinsiza, okungcono ngokucindezelwa kwedatha kudiski.
I-Clickhouse ihlangabezana nazo zonke lezi zindlela, futhi asikaze sizisole ngokukhetha kwethu. Asibhali noma yimaphi amanani angajwayelekile edatha kuyo (inani lokufakiwe lingaba yizinkulungwane ezinhlanu kuphela ngomzuzu).
I-NewRelic
I-NewRelic ibikade inathi ngokomlando ngoba bekuyisinqumo sekhasimende. Siyisebenzisa njenge-APM.
Zabbix
Sisebenzisa i-Zabbix ngokukhethekile ukuze sigade i-Black Box yama-API ahlukahlukene.
Ukuchaza Indlela Yokuqapha
Besifuna ukuwohloza umsebenzi futhi ngaleyo ndlela sihlele indlela yokuqapha.
Ukwenza lokhu, ngihlukanise isistimu yethu ngamaleveli alandelayo:
- hardware kanye VMS;
- isistimu esebenzayo;
- amasevisi wesistimu, isitaki sesofthiwe;
- isicelo;
- logic yebhizinisi.
Kungani le ndlela ifaneleka:
- siyazi ukuthi ubani onesibopho somsebenzi wezinga ngalinye futhi, ngokusekelwe kulokhu, singathumela izexwayiso;
- singasebenzisa isakhiwo lapho sicindezela izexwayiso - kungaba yinqaba ukuthumela isaziso mayelana nokungatholakali kwesizindalwazi lapho umshini obonakalayo uwonke ungatholakali.
Njengoba umsebenzi wethu uwukubona ukwephulwa kokusebenza kwesistimu, kufanele kuleveli ngayinye sigqamise isethi ethile yamamethrikhi okufanele inakwe lapho sibhala imithetho yokuxwayisa. Okulandelayo, ake sidlule kumazinga "VMS", "Isistimu yokusebenza" kanye "Nezinsizakalo zesistimu, isitaki sesoftware".
Imishini ebonakalayo
Ukusingatha kusabele iprosesa, idiski, inkumbulo kanye nenethiwekhi. Futhi saba nezinkinga ngababili bokuqala. Ngakho, ama-metrics:
Isikhathi esibiwe se-CPU - uma uthenga umshini obonakalayo ku-Amazon (t2.micro, isibonelo), kufanele uqonde ukuthi awubelwanga i-processor core yonke, kodwa isabelo sesikhathi saso kuphela. Futhi uma uyiqeda, iphrosesa izosuswa kuwe.
Le metric ikuvumela ukuthi ulandelele izikhathi ezinjalo futhi wenze izinqumo. Isibonelo, ingabe kuyadingeka ukuthatha intela ekhuluphele noma ukusabalalisa ukucutshungulwa kwemisebenzi yangemuva nezicelo ze-API kumaseva ahlukene?
IOPS + CPU iowait time - ngesizathu esithile, abasingathi abaningi bamafu bona ngokunganikezi i-IOPS eyanele. Ngaphezu kwalokho, ishejuli ene-IOPS ephansi ayiyona ingxabano kubo. Ngakho-ke, kufanelekile ukuqoqa i-CPU iowait. Ngala magrafu amabili - ane-IOPS ephansi kanye nokulinda kwe-I/O ephezulu - usungakwazi kakade ukukhuluma nomsingathi futhi uxazulule inkinga.
operating system
Amamethrikhi esistimu yokusebenza:
- inani lememori etholakalayo ku-%;
- shintsha umsebenzi wokusetshenziswa: vmstat swapin, swapout;
- inombolo yama-inode atholakalayo nesikhala samahhala ohlelweni lwefayela ku-%
- isilinganiso somthwalo;
- inombolo yokuxhumana esimweni se-tw;
- contrack ukugcwala kwetafula;
- Izinga lenethiwekhi lingagadwa kusetshenziswa insiza ye-ss, iphakheji ye-iproute2 - thola inkomba yokuxhumeka kwe-RTT kusuka ekuphumeni kwayo bese uyihlanganisa nge-dest port.
Futhi ezingeni lesistimu yokusebenza sinebhizinisi elinjalo njengezinqubo. Kubalulekile ukukhomba ohlelweni isethi yezinqubo ezidlala indima ebalulekile ekusebenzeni kwayo. Uma, ngokwesibonelo, unama-pgpools amaningana, udinga ukuqoqa imininingwane ngayinye yazo.
Isethi yamamethrikhi imi kanje:
- Ama-CPU;
- inkumbulo ngokuyinhloko ihlala;
- IO - okungcono ku-IOPS;
- FileFd - vula futhi ukhawule;
- ukwehluleka okubalulekile kwekhasi - ngale ndlela ungaqonda ukuthi iyiphi inqubo eshintshwayo.
Sifaka konke ukuqapha ku-Docker, futhi sisebenzisa i-Advisor ukuze siqoqe idatha yamamethrikhi. Kweminye imishini sisebenzisa i-process-exporter.
Amasevisi esistimu, isitaki sesofthiwe
Uhlelo lokusebenza ngalunye lunokucaciswa kwalo, futhi kunzima ukuhlukanisa isethi ethile yamamethrikhi.
Isethi yendawo yonke ithi:
- isilinganiso sesicelo;
- inani lamaphutha;
- ukubambezeleka;
- saturation
Izibonelo zethu eziphawuleka kakhulu zokuqapha kuleli zinga i-Nginx ne-PostgreSQL.
Isevisi elayishwe kakhulu ohlelweni lwethu isizindalwazi. Esikhathini esedlule, sasivame ukuba nenkinga yokuthola ukuthi isizindalwazi senzani.
Sibone umthwalo ophezulu kumadiski, kodwa izingodo ezihamba kancane azizange zibonise lutho ngempela. Sixazulule le nkinga sisebenzisa i-pg_stat_statements, umbono oqoqa izibalo zemibuzo.
Yilokho kuphela okudingayo admin.
Sakha amagrafu omsebenzi wokufunda nokubhala izicelo:
Konke kulula futhi kucacile, isicelo ngasinye sinombala waso.
Isibonelo esimangalisa ngokulinganayo amalogi we-Nginx. Akumangazi ukuthi bambalwa abantu abawahlaziyayo noma abakhulume ngawo ohlwini lwezinto okufanele ube nazo. Ifomethi evamile ayinalo ulwazi kakhulu futhi idinga ukunwetshwa.
Mina ngokwami, ngengeze isicelo_isikhathi, isikhathi_sokuphendula_i-upstream, i-body_bytes_sent, request_length, request_id. Sihlela isikhathi sokuphendula nenombolo yamaphutha:
Sakha amagrafu esikhathi sokuphendula kanye nenani lamaphutha. Uyakhumbula? Ngike ngakhuluma ngezinjongo zebhizinisi? Ukuze ngokushesha futhi ngaphandle kwamaphutha? Sesike sazihlanganisa lezi zindaba ngamashadi amabili. Futhi usungakwazi kakade ukushayela abalawuli abasemsebenzini usebenzisa bona.
Kodwa enye inkinga isasele - ukuqinisekisa ukuqedwa ngokushesha kwezimbangela zesigameko.
Ukulungiswa kwesigameko
Yonke inqubo ukusuka ekuhlonzeni kuye ekuxazululeni inkinga ingahlukaniswa ngezinyathelo ezimbalwa:
- ukuhlonza inkinga;
- isaziso kumphathi womsebenzi;
- impendulo ngesigameko;
- ukuqedwa kwezinkinga.
Kubalulekile ukuthi lokhu sikwenze ngokushesha ngangokunokwenzeka. Futhi uma ezigabeni zokuhlonza inkinga nokuthumela isaziso asikwazi ukuzuza isikhathi esiningi - kuzosetshenziswa imizuzu emibili kuzo noma kunjalo, bese kuthi okulandelayo kuyinkambu engalinyiwe ukuze kuthuthukiswe.
Ake sicabange nje kwakhala ucingo lwesikhulu esisebenza emsebenzini. Uzokwenzani? Bheka izimpendulo zemibuzo - yini ephukile, yephuka kuphi, ukuthi usabele kanjani? Nansi indlela esiphendula ngayo le mibuzo:
Simane nje sifake lonke lolu lwazi embhalweni wesaziso, sinikeze isixhumanisi sekhasi le-wiki elichaza indlela yokuphendula le nkinga, ukuyixazulula nokuyandisa.
Angikasho lutho mayelana nesendlalelo sesicelo kanye nengqondo yebhizinisi. Ngeshwa, izinhlelo zethu zokusebenza azikasebenzisi ukuqoqwa kwamamethrikhi. Umthombo kuphela wanoma yiluphi ulwazi olusuka kulawa maleveli amalogi.
Amaphuzu ambalwa.
Okokuqala, bhala izingodo ezihlelekile. Asikho isidingo sokufaka umongo embhalweni womlayezo. Lokhu kwenza kube nzima ukuhlanganisa nokuhlaziya. I-Logstash ithatha isikhathi eside ukwenza konke lokhu kujwayelekile.
Okwesibili, sebenzisa amazinga obunzima ngendlela efanele. Ulimi ngalunye lunendinganiso yalo. Ngokwami, ngihlukanisa amazinga amane:
- alikho iphutha;
- iphutha lecala leklayenti;
- iphutha lisohlangothini lwethu, asilahlekelwa yimali, asithwali izingozi;
- Iphutha lingakithi, silahlekelwa yimali.
Ake ngifingqe. Udinga ukuzama ukwakha ukuqapha ngokusekelwe kumqondo webhizinisi. Zama ukuqapha uhlelo lokusebenza ngokwalo futhi usebenze ngamamethrikhi afana nenombolo yokuthengisa, inombolo yokubhaliswa kwabasebenzisi abasha, inombolo yabasebenzisi abasebenzayo njengamanje, njalo njalo.
Uma ibhizinisi lakho lonke liyinkinobho eyodwa kusiphequluli, udinga ukuqapha ukuthi liyachofoza futhi lisebenza kahle yini. Konke okunye akusho lutho.
Uma ungenakho lokhu, ungazama ukukuthola ezingodweni zohlelo lokusebenza, izingodo ze-Nginx, nokunye, njengoba senzile. Kufanele ube seduze nesicelo ngangokunokwenzeka.
Amamethrikhi esistimu yokusebenza abalulekile vele, kodwa ibhizinisi alinantshisekelo kuwo, asikhokhelwa ngawo.
Source: www.habr.com