Ini ndinonzi Anton Baderin. Ini ndinoshanda kuHigh Technology Center uye ndinoita system management. Mwedzi wapfuura, musangano wedu wekambani wakapera, kwatakagovana ruzivo rwedu rwakaunganidzirwa nenharaunda yeIT yeguta redu. Ndakataura nezve monitoring web applications. Izvo zvinyorwa zvakaitirwa junior kana wepakati level, uyo asina kuvaka iyi maitiro kubva pakutanga.
Ibwe repakona riri pasi pechero monitoring system iri kugadzirisa matambudziko ebhizinesi. Kuongorora nekuda kwekutarisa hakuna zvakunofadza kune chero munhu. Bhizinesi rinodei? Saka kuti zvinhu zvose zvishande nokukurumidza uye pasina zvikanganiso. Mabhizinesi anoda kuve anoshanda, kuitira kuti isu pachedu tione matambudziko mubasa uye tigadzirise nekukurumidza sezvinobvira. Aya, chokwadi, ndiwo matambudziko andakagadzirisa gore rapfuura rese pane purojekiti yemumwe wevatengi vedu.
Nezve purojekiti
Chirongwa ichi ndechimwe chezvirongwa zvakakura zvekuvimbika munyika. Isu tinobatsira cheni dzekutengesa kuwedzera kuwanda kwekutengesa kuburikidza neakasiyana maturusi ekushambadzira akadai semakadhi ebhonasi. Pakazara, chirongwa ichi chinosanganisira gumi nemana zvikumbiro zvinomhanya pamaseva gumi.
Munguva yekubvunzurudza, ndakadzokorora kuona kuti maadmins haagari achisvika pakutarisisa mawebhusaiti nenzira kwayo: mazhinji anoramba akatarisana neanoshanda system metrics uye dzimwe nguva anotarisisa masevhisi.
Mune yangu kesi, sisitimu yekutarisa yemutengi yaive yakavakirwa paIcinga. Hachina kugadzirisa matambudziko ari pamusoro apa neimwe nzira. Kazhinji mutengi pachake akatiudza nezvematambudziko, uye kazhinji pane kwete, isu takanga tisina data rakakwana kuti tisvike pasi pechikonzero.
Mukuwedzera, pakanga pane kunzwisisa kwakajeka kwekusava nematuro kwekuwedzera kwayo. Ndinofunga vanojairana naIcinga vachandinzwisisa. Saka, takafunga kugadzirisa zvachose iyo webhu application yekutarisa system yeprojekiti.
Prometheus
Isu takasarudza Prometheus zvichibva pane matatu makuru zviratidzo:
- Nhamba huru yemametrics aripo. Kwatiri kune 60 zviuru zvavo. Ehe, zvakakosha kuziva kuti isu hatishandise ruzhinji rwacho (zvichida nezve 95%). Ukuwo, ose akachipa. Kwatiri, iyi ndiyo imwe yakanyanyisa ichienzaniswa neyakamboshandiswa Icinga. Mariri, kuwedzera ma metrics kwaive kurwadza kwakanyanya: izvo zvaivepo zvaive zvichidhura (ingo tarisa iyo sosi kodhi yechero plugin). Chero plugin yaive script muBash kana Python, kutangwa kwayo kunodhura maererano nezviwanikwa zvakadyiwa.
- Iyi sisitimu inodya zvishoma zviwanikwa. 600 MB ye RAM, 15% yeimwe musimboti uye akati wandei maIOPS anokwana kune edu ese metrics. Ehe, iwe unofanirwa kumhanyisa metrics ekunze, asi ese akanyorwa muGo uye haanawo nzara yesimba zvakanyanya. Handifungi kuti mumamiriro ezvinhu emazuva ano iyi idambudziko.
- Inopa kugona kutamira kuKubernetes. Tichifunga nezvezvirongwa zvemutengi, sarudzo iri pachena.
ELK:
Kare, hatina kuunganidza kana kugadzirisa matanda. Izvo zvinokanganisa zviri pachena kumunhu wese. Isu takasarudza ELK nekuti takanga tatova neruzivo nehurongwa uhu. Isu tinongochengeta matanda ekushandisa ipapo. Mamiriro makuru ekusarudza aive akazara-mavara ekutsvaga uye kumhanya kwayo.
Π‘lickhouse
Pakutanga, sarudzo yakawira paInfluxDB. Takaona kukosha kwekuunganidza matanda eNginx, nhamba kubva pg_stat_statements, uye kuchengeta Prometheus nhoroondo data. Isu hatina kufarira Influx nekuti nguva nenguva yaitanga kupedza ndangariro zhinji nekurovera. Pamusoro pezvo, ndaida kuunganidza mibvunzo neremote_addr, asi kuisa mapoka muDBMS iyi chete nematagi. Mategi anodhura (memory), nhamba yavo inoganhurwa.
Takatanga kutsvaga zvekare. Chaidiwa yaive yekuongorora dhatabhesi ine zvishoma zviwanikwa zvekushandisa, zviri nani nekumanikidza data pa diski.
Clickhouse inosangana nemaitiro ese aya, uye hatina kumbodemba sarudzo yedu. Isu hatinyore chero huwandu hunoshamisa hwe data mairi (nhamba yekuisa inongosvika zviuru zvishanu paminiti).
NewRelic
NewRelic yakagara nesu nekuti yaive sarudzo yemutengi. Isu tinoishandisa seAPM.
Zabbix
Isu tinoshandisa Zabbix chete kutarisa iyo Black Bhokisi reakasiyana APIs.
Kutsanangura Nzira Yekutarisa
Isu taida kukanganisa basa uye nekudaro kuronga nzira yekutarisa.
Kuti ndiite izvi, ndakakamura sisitimu yedu mumatanho anotevera:
- hardware uye VMS;
- operating system;
- masevhisi ehurongwa, software stack;
- application;
- business logic.
Nei nzira iyi iri nyore:
- tinoziva kuti ndiani ane mutoro webasa rechikamu chimwe nechimwe uye, zvichibva pane izvi, tinogona kutumira zviziviso;
- isu tinogona kushandisa chimiro kana tichidzvanya chenjedzo - zvingave zvinoshamisa kutumira yambiro nezvekusavapo kwedatabase kana muchina chaiwo wese usingawanikwe.
Sezvo basa redu riri rekuona kutyorwa mukushanda kwehurongwa, isu padanho rega rega tinofanirwa kuratidza imwe seti yemameta akakodzera kutarisisa kana tichinyora mitemo yekuzivisa. Tevere, ngatiendei nematanho "VMS", "Operating system" uye "System services, software stack".
Virtual michina
Kubata kunotipa isu processor, dhisiki, ndangariro uye network. Uye takava nezvinetso nevaviri vokutanga. Saka, metrics:
CPU yakabiwa nguva - kana iwe uchitenga chaiwo muchina paAmazon (t2.micro, semuenzaniso), iwe unofanirwa kunzwisisa kuti hauna kupihwa iyo yese processor core, asi chete quota yenguva yayo. Uye kana iwe waipedza, processor inotorwa kubva kwauri.
Iyi metric inokutendera kuti utarise nguva dzakadai uye kuita sarudzo. Semuenzaniso, zvakakosha here kutora mutero wakakora kana kugovera kugadzirisa kwemashure mabasa uye zvikumbiro zveAPI kumaseva akasiyana?
IOPS + CPU iowait nguva - nekuda kwechimwe chikonzero, akawanda makore anotambira anotadza nekusapa IOPS yakakwana. Uyezve, purogiramu ine yakaderera IOPS haisi nharo kwavari. Naizvozvo, zvakakodzera kuunganidza CPU iowait. Neiyi magirafu maviri - ane yakaderera IOPS uye yakakwira I / O kumirira - unogona kutotaura neanotambira uye kugadzirisa dambudziko.
uchishandisa maitiro
Operating system metrics:
- huwandu hwekuyeuka huripo mu%;
- chinjanisa kushandiswa kwekuita: vmstat swapin, swapout;
- nhamba yeinode iripo uye nzvimbo yemahara pane faira system mu%
- avhareji mutoro;
- nhamba yekubatanidza mutw state;
- contrack tafura yakazara;
- Hunhu hwetiweki hunogona kutariswa uchishandisa ss utility, iyo iproute2 package - tora chiratidzo cheRTT yekubatanidza kubva pakubuda kwayo uye iite boka nedest port.
Zvakare padanho rekushandisa sisitimu tine chakadai semaitiro. Zvakakosha kuziva muhurongwa seti yemaitiro anoita basa rinokosha mukushanda kwayo. Kana, semuenzaniso, uine akati wandei pgpools, saka iwe unofanirwa kuunganidza ruzivo rweumwe neumwe wavo.
Seti yemametrics ndeaya anotevera:
- CPUs;
- ndangariro inonyanya kugara;
- IO - zviri nani muIOPS;
- FileFd - vhura uye muganho;
- kwakakosha peji kukundikana - nenzira iyi unogona kunzwisisa kuti ndeipi maitiro ari kuchinjwa.
Isu tinotumira ese ekutarisa muDocker, uye isu tinoshandisa Advisor kuunganidza metrics data. Pane mimwe michina tinoshandisa process-exporter.
Sistimu masevhisi, software stack
Chishandiso chega chega chine chayo chakati, uye zvakaoma kudoma imwe seti yemametrics.
Universal set ndeiyi:
- chiyero chekukumbira;
- nhamba yezvikanganiso;
- latency;
- saturation
Yedu inokatyamadza mienzaniso yekutarisa padanho iri Nginx uye PostgreSQL.
Iyo yakanyanya kurodha sevhisi muhurongwa hwedu ndeye database. Munguva yakapfuura, taigara tichinetseka kuziva kuti dhatabhesi raiitei.
Takaona mutoro wakakwirira pa disks, asi matanda anononoka haana kunyatsoratidza chero chinhu. Takagadzirisa dambudziko iri tichishandisa pg_stat_statements, maonero anounganidza nhamba dzemibvunzo.
Ndizvo zvese zvinoda admin.
Isu tinovaka magirafu echiitwa chekuverenga nekunyora zvikumbiro:
Zvose zviri nyore uye zvakajeka, chikumbiro chimwe nechimwe chine ruvara rwayo.
Muenzaniso wakaenzana unokatyamadza ndeye Nginx matanda. Hazvishamise kuti vanhu vashoma vanoatsanangudza kana kutaura nezvawo murondedzero yezvinofanira-kuva nazvo. Iyo yakajairwa fomati haina ruzivo rwakanyanya uye inoda kuwedzerwa.
Ini pachangu, ndawedzera request_time, upstream_response_time, body_bytes_sent, request_length, request_id. Isu tinoronga nguva yekupindura uye nhamba yezvikanganiso:
Isu tinovaka magirafu enguva yekupindura uye nhamba yezvikanganiso. Rangarira? Ndataura nezvemabasa ebhizinesi here? Kukurumidza uye pasina zvikanganiso? Takatovhara nyaya idzi nemachati maviri. Uye iwe unogona kutofonera vatariri vari pabasa uchivashandisa.
Asi rimwe dambudziko rinoramba riripo - kuve nechokwadi chekukurumidza kubviswa kwezvikonzero zvechiitiko chacho.
Chiitiko kugadzirisa
Maitiro ese kubva pakuziva kusvika pakugadzirisa dambudziko anogona kukamurwa kuita akati wandei matanho:
- kuziva dambudziko;
- chiziviso kumutungamiri webasa;
- mhinduro kune chiitiko;
- kubviswa kwezvikonzero.
Zvakakosha kuti tiite izvi nekukurumidza sezvinobvira. Uye kana pamatanho ekuona dambudziko uye kutumira chiziviso isu hatigone kuwana nguva yakawanda - maminetsi maviri achashandiswa pazviri chero zvakadaro, zvino inotevera ingori munda usina kurima wekuvandudza.
Tombofungidzira kuti nhare yemukuru webasa yakarira. Achaita sei? Tsvaga mhinduro dzemibvunzo - chii chakaputsika, chakaputsika kupi, kuti ungaita sei? Heano maitiro atinopindura mibvunzo iyi:
Isu tinongo sanganisira ruzivo rwese urwu muzvinyorwa zvechiziviso, tipe chinongedzo kune peji rewiki rinotsanangura maitiro ekuita dambudziko iri, magadzirisiro aro uye kuwedzera.
Ini handisati ndataura chero chinhu nezve iyo application layer uye bhizinesi pfungwa. Nehurombo, maapplication edu haasati ashandisa metrics kuunganidzwa. Iko chete kunobva chero ruzivo kubva kumatanho aya matanda.
Zvibodzwa zviviri.
Kutanga, nyora matanda akarongwa. Hapana chikonzero chekuisa mamiriro muzvinyorwa zvemeseji. Izvi zvinoita kuti zvive zvakaoma kuunganidza uye kuongorora. Logstash inotora nguva yakareba kugadzirisa zvese izvi.
Chechipiri, shandisa mazinga ekuomarara nemazvo. Mutauro wega wega une mwero wawo. Ini pachangu, ndinosiyanisa mazinga mana:
- hapana kukanganisa;
- mutengi side kukanganisa;
- kukanganisa kuri kudivi redu, hatirasikirwe nemari, hatitakuri njodzi;
- Kukanganisa kuri kudivi redu, tinorasikirwa nemari.
Rega ndipfupise. Iwe unofanirwa kuyedza kuvaka yekutarisa zvichienderana nebhizinesi logic. Edza kutarisa iyo application pachayo uye kushanda nemametric akadai sehuwandu hwekutengesa, nhamba yevatsva vashandisi kunyoreswa, nhamba yevashandisi vari kushanda, zvichingodaro.
Kana bhizinesi rako rese riri bhatani rimwe mubrowser, unofanirwa kutarisa kuti rinodzvanya uye richishanda nemazvo. Zvimwe zvese hazvina basa.
Kana iwe usina izvi, unogona kuedza kubata nazvo mumatanda ekunyorera, matanda eNginx, zvichingodaro, sezvatakaita. Iwe unofanirwa kunge uri pedyo nechikumbiro sezvinobvira.
Operating system metrics akakosha, asi bhizinesi harifarire mazviri, isu hatibhadharwe ivo.
Source: www.habr.com