Nhasi, kuwedzera kune monolithic kodhi, chirongwa chedu chinosanganisira akawanda e microservices. Chimwe nechimwe chazvo chinoda kuongororwa. Kuita izvi pachiyero chakadaro uchishandisa DevOps mainjiniya zvinonetsa. Isu takagadzira yekutarisa sisitimu inoshanda sevhisi yevagadziri. Ivo vanogona kunyora vakazvimiririra metrics mune yekutarisa sisitimu, vashandise, kuvaka dashboards zvichibva pavari, uye nekuisa zviziviso kwavari izvo zvinozokonzereswa kana pachikumbaridzo hunhu hwasvika. Kune mainjiniya eDevOps, zvivakwa chete uye zvinyorwa.
Ichi chinyorwa chinyorwa chekutaura kwangu nekwedu paRIT++. Vanhu vazhinji vakatikumbira kuti tiite zvinyorwa zvemishumo kubva ipapo. Dai wanga uri kumusangano kana kuona vhidhiyo, hapana chitsva chauchawana. Uye vamwe vose - kugamuchirwa kukati. Ini ndichakuudza kuti takasvika sei kune yakadaro system, kuti inoshanda sei uye kuti tinoronga sei kuigadzirisa.
Zvakapfuura: zvirongwa uye zvirongwa
Takasvika sei pahurongwa hwekutarisa huripo? Kuti upindure mubvunzo uyu, unofanirwa kuenda ku2015. Izvi ndizvo zvazvakaita panguva iyoyo:
Taive nemanode angangoita makumi maviri nemana aive nebasa rekutarisa. Iko kune yakazara paki yemakorona akasiyana, zvinyorwa, madhimoni anotarisisa chimwe chinhu, kutumira mameseji, uye kuita mabasa. Takafunga kuti kana tichienda mberi, hurongwa hwakadaro hwaizova hushoma. Hapana chikonzero chekuigadzira: yakanyanya kuoma.
Isu takasarudza kusarudza izvo zvekutarisa izvo zvatichachengeta nekuvandudza, uye izvo zvatichasiya. Paive ne19. Magraphite chete, aggregator uye Grafana sedashboard yakasara. Asi gadziriro itsva icharatidzika sei? Sezvizvi:
Isu tine metrics ekuchengetedza: aya magraphite, ayo achange akavakirwa nekukurumidza SSD madhiraivha, aya mamwe aggregator emametrics. Zvinotevera - Grafana yekuratidza madhibhodhi uye Moira yekuzivisa. Taidawo kugadzira hurongwa hwekutsvaga anomalies.
Standard: Monitoring 2.0
Izvi ndizvo zvakaitwa nehurongwa muna 2015. Asi taifanira kugadzirira kwete zvivakoti uye sevhisi pachayo, asiwo zvinyorwa zvayo. Isu takagadzira mwero wemakambani isu pachedu, watinodaidza kuti monitoring 2.0. Ndezvipi zvaidiwa kuhurongwa?
- kuwanikwa nguva dzose;
- metrics kuchengetedza nguva = masekonzi gumi;
- kuchengetwa kwakarongeka kwema metrics uye dashboards;
- SLA > 99,99%
- kuunganidzwa kwezviitiko metrics kuburikidza neUDP (!).
Isu taida UDP nekuti isu tine kuyerera kukuru kwetraffic uye zviitiko zvinogadzira metrics. Kana iwe ukanyora zvese mugraphite kamwechete, chengetedzo inodonha. Isu takasarudza zvakare yekutanga-chikamu prefixes kune ese metrics.
Imwe neimwe yezvivakamberi ine chimwe chinhu. Kune metrics emaseva, network, midziyo, zviwanikwa, maapplication, zvichingodaro. Kusefa kwakajeka, kwakasimba, kwakadhindwa kwaitwa, kwatinotambira mametrics ekutanga uye nekungodonhedza mamwe. Aya ndiwo marongero atakaita sisitimu iyi muna 2015. Chii chiripo nhasi?
Ikozvino: dhayagiramu yekudyidzana kwezvikamu zvekutarisa
Chekutanga pane zvese, isu tinotarisisa maapplication: yedu PHP kodhi, maapplication uye microservices - muchidimbu, zvese zvinonyorwa nevagadziri vedu. Zvese zvikumbiro zvinotumira metrics kuburikidza neUDP kune iyo Brubeck aggregator (statsd, yakanyorwazve muC). Yakazove yakakurumidza muongororo dzekugadzira. Uye inotumira iyo yakatounganidzwa metrics kuGraphite kuburikidza neTCP.
Iine mhando yemetrics inonzi timers. Ichi chinhu chiri nyore kwazvo. Semuenzaniso, kune yega yega mushandisi yekubatanidza kune sevhisi, unotumira metric ine nguva yekupindura kuBrubeck. Mamiriyoni emhinduro dzakauya, asi muunganidzi akadzoka chete gumi metrics. Iwe une nhamba yevanhu vakauya, iyo yakawanda, shoma uye yepakati yekupindura nguva, iyo yepakati uye 10 percentiles. Ipapo iyo data inoendeswa kuGraphite uye tinoiona yese ichirarama.
Isu tine zvakare aggregation yemametric pane Hardware, software, system metrics uye yedu yekare Munin yekutarisa system (yakashanda kwatiri kusvika 2015). Isu tinounganidza zvese izvi kuburikidza neC daemon CollectD (ine boka rese remapulagi akasiyana akavakirwa mukati mayo, inogona kuvhota zvese zviwanikwa zveiyo host system iyo yakaiswa, ingotsanangura mukugadzirisa kwainonyorera data) uye nyora iyo data kuGraphite kuburikidza nayo. Inotsigirawo python plugins uye shell scripts, saka iwe unogona kunyora yako yega tsika mhinduro: CollectD ichaunganidza iyi data kubva kune yemunharaunda kana kure host host (kutora Curl) uye kuitumira kuGraphite.
Tinobva tatumira ese metrics atakaunganidza kuCarbon-c-relay. Iyi ndiyo Carbon Relay solution kubva kuGraphite, yakagadziridzwa muC. Iyi i router inounganidza ma metrics ose atinotumira kubva kune aggregator edu uye anoaendesa kune nodes. Zvakare padanho rekufambisa, inotarisa chokwadi chemetrics. Chekutanga, dzinofanirwa kuenderana ne prefix scheme yandakaratidza pakutanga uye, chechipiri, inoshanda kune graphite. Zvikasadaro vachadonha.
Carbon-c-relay inobva yatumira metrics kuGraphite cluster. Isu tinoshandisa Carbon-cache, yakanyorwa patsva muGo, seyakanyanya kuchengetedza metrics. Enda-kabhoni, nekuda kwekuwanda kwayo, inodarika kure Carbon-cache. Iyo inogamuchira data uye inoinyora kumadhisiki uchishandisa iyo whisper package (yakajairwa, yakanyorwa mupython). Kuti tiverenge data kubva mudura redu, tinoshandisa Graphite API. Inokurumidza kupfuura yakajairwa Graphite WEB. Chii chinoitika kune data inotevera?
Vanoenda kuGrafana. Isu tinoshandisa masango edu egraphite seyo huru yedata, uye isu tine Grafana sewebhu interface yekuratidzira metrics uye kuvaka dashboard. Kune yega yega masevhisi avo, vanogadzira vanogadzira yavo dashboard. Ipapo ivo vanovaka magirafu akavakirwa paari, ayo anoratidza metrics yavanonyora kubva mukushandisa kwavo. Pamusoro peGrafana, tinewo SLAM. Iri idhimoni repython rinoverenga SLA zvichienderana nedata kubva kugraphite. Sezvandambotaura, isu tine akati wandei ma microservices, imwe neimwe ine yayo yainoda. Tichishandisa SLAM, tinoenda kune zvinyorwa tozvienzanisa nezviri muGraphite uye toenzanisa kuti zvinodiwa zvinoenderana sei nekuwanikwa kwesevhisi yedu.
Ngatienderere mberi: kunyevera. Yakarongwa uchishandisa hurongwa hwakasimba - Moira. Yakazvimiririra nekuti ine yayo Graphite pasi pehodhi. Yakagadzirwa nevakomana kubva kuSKB "Kontur", yakanyorwa mupython uye Go, yakavhurika sosi. Moira anogamuchira kuyerera kwakafanana kunoenda mumagraphite. Kana nekuda kwechimwe chikonzero chengetedzo yako ikafa, yambiro yako icharamba ichishanda.
Isu takaisa Moira muKubernetes; inoshandisa sumbu reRedis maseva sedatabase huru. Mhedzisiro yacho yaive system yekushivirira kukanganisa. Inofananidza rwizi rwemetrics nerunyorwa rwezvinokonzeresa: kana pasina kutaurwa mairi, zvino inodonhedza metric. Saka inokwanisa kugaya gigabytes yemametrics paminiti.
Isu takabatanidzawo LDAP yekambani kwairi, nerubatsiro rwekuti mushandisi wega wega wehurongwa hwekambani anogona kugadzira zviziviso zvavo zvichibva pane zviripo (kana kuti zvichangogadzirwa) zvinokonzeresa. Sezvo Moira iine Graphite, inotsigira ese ayo maficha. Saka unotanga watora mutsara wokopa muGrafana. Ona kuti iyo data inoratidzwa sei pamagirafu. Uye wobva watora mutsetse mumwechete woukopa muMoira. Iwe unoisungirira nemiganhu uye uwane yambiro pane inobuda. Kuti uite zvese izvi, haudi chero ruzivo rwakananga. Moira anogona kungwarira kuburikidza neSMS, email, Jira, Slack... Inotsigirawo kuitwa kwetsika zvinyorwa. Kana chinokonzeresa chikaitika kwaari, uye akanyoreswa kune tsika script kana bhinari, anoimhanyisa uye anotumira JSON ku stdin kune iyi bhinary. Nekuda kweizvozvo, chirongwa chako chinofanirwa kuchiparura. Zvauchaita neJSON iyi zviri kwauri. Kana uchida, tumira kuTeregiramu, kana uchida, vhura mabasa muJira, ita chero.
Isu tinoshandisawo yedu budiriro yekuzivisa - Imagotag. Isu takagadzirisa mapaneru, ayo anowanzo shandiswa pamagetsi emitengo tags muzvitoro, kuti aenderane nezvatinoda. Takaunza matrigger kubva kuMoira kwairi. Inoratidza kuti vari mumamiriro akaita sei uye kuti zvakaitika rini. Vamwe vevakomana vebudiriro vakasiya zviziviso muSlack uye email vachifarira pani iyi.
Zvakanaka, sezvo isu tiri kambani inofambira mberi, takatarisawo Kubernetes mune ino system. Isu takaisanganisira muhurongwa tichishandisa Heapster, iyo yatakaisa musumbu, inounganidza data uye inotumira kuGraphite. Nekuda kweizvozvo, dhizaini rinotaridzika seizvi:
Monitoring Zvikamu
Heino runyoro rwezvinongedzo kune zvikamu zvatakashandisa pabasa iri. Vese vari open source.
Graphite:
- go-carbon:
- zevezera:
- graphite-api:
Carbon-c-relay:
Brubeck:
Zvakaunganidzwa:
Moira:
Grafana:
Heapster:
Π‘ΡΠ°ΡΠΈΡΡΠΈΠΊΠ°
Uye heano mamwe manhamba ekuti sisitimu inoshanda sei kwatiri.
Aggregator (brubeck)
Nhamba yemetrics: ~300/sec
Nguva yekutumira metrics kuGraphite: 30 sec
Sevha yekushandisa zviwanikwa: ~ 6% CPU (tiri kutaura nezvesevha yakazara-yakazara); ~ 1Gb RAM; ~ 3 Mbps LAN
Graphite (go-carbon)
Nhamba yemetrics: ~ 1 / min
Metrics yekuvandudza nguva: 30 sec
Metrics kuchengetedza chirongwa: 30sec 35d, 5min 90d, 10min 365d (inokupa kunzwisisa kwezvinoitika kune sevhisi kwenguva yakareba)
Server resource kushandiswa: ~ 10% CPU; ~ 20Gb RAM; ~30 Mbps LAN
Flexibility
Isu paAvito tinokoshesa kuchinjika mubasa redu rekutarisa. Nei akanyatsoita seizvi? Chekutanga, zvikamu zvaro zvinochinjika: zvese zvikamu pachazvo neshanduro dzazvo. Chechipiri, kutsigira. Sezvo purojekiti yese yakavhurika sosi, unogona kugadzirisa kodhi iwe pachako, shandura, uye shandisa mabasa asiri kuwanikwa kunze kwebhokisi. Zvakawanda zvakajairika stacks anoshandiswa, kunyanya Go nePython, saka izvi zvinoitwa zviri nyore.
Heino muenzaniso wedambudziko chairo. Metric muGraphite ifaira. Rine zita. Zita refaira = metric name. Uye kune nzira yekusvika ikoko. Mazita emafaira muLinux anogumira kune 255 mavara. Uye isu tine (se "vatengi vemukati") vakomana kubva kudhipatimendi re database. Vanotiudza: "Tinoda kutarisa mibvunzo yedu yeSQL. Uye havasi 255 mavara, asi 8 MB imwe neimwe. Tinoda kuvaratidza muGrafana, ona maparamita echikumbiro ichi, uye kunyange zvirinani, tinoda kuona pamusoro pezvikumbiro zvakadaro. Zvichave zvakanaka kana zvichiratidzwa munguva chaiyo. Zvingava zvakanaka kuvaisa muchenjedzo.β
Muenzaniso SQL query inotorwa semuenzaniso kubva
Isu tinogadzira sevha yeRedis uye tinoshandisa Collectd plugins yedu, inoenda kuPostgres uye inotora data rese kubva ipapo, kutumira metrics kuGraphite. Asi isu tinotsiva zita remetric nemahashi. Isu panguva imwe chete tinotumira hashi imwechete kuRedis sekiyi, uye iyo yese SQL mubvunzo semutengo. Chatinofanira kuita ndechekuti Grafana aende kuRedis otora ruzivo urwu. Tiri kuvhura iyo Graphite API nekuti... iyi ndiyo huru interface yekubatana kwezvinhu zvose zvekuongorora negraphite, uye tinopinda basa idzva ikoko rinonzi aliasByHash () - kubva kuGrafana tinowana zita remetric, uye tinorishandisa mukukumbira Redis sekiyi, mu mhinduro tinowana kukosha kwekiyi, inova yedu "SQL query" " Nokudaro, takaratidza muGrafana chiratidziro chemubvunzo weSQL, iyo mupfungwa yakanga isingabviri kuratidza ipapo, pamwe chete nenhamba pairi (kufona, mitsara, total_time, ...).
Migumisiro
Kuwanika Yedu yekutarisa sevhisi inowanikwa 24/7 kubva kune chero application uye chero kodhi. Kana iwe uchikwanisa kuwana nzvimbo dzekuchengetedza, unogona kunyora data kune sevhisi. Mutauro hauna kukosha, zvisarudzo hazvina kukosha. Iwe unongoda kuziva kuvhura socket, isa metric ipapo uye kuvhara socket.
Kuvimbika Zvese zvikamu zvinoshivirira kukanganisa uye zvinobata mitoro yedu zvakanaka.
Yakaderera chipingamupinyi kupinda. Kuti ushandise iyi sisitimu, haufanirwe kudzidza mitauro yekuronga uye mibvunzo muGrafana. Ingovhura application yako, isa socket mairi inotumira metrics kuGraphite, ivhare, vhura Grafana, gadzira madhibhodhi ipapo uye tarisa maitiro emametrics ako, kugamuchira zviziviso kuburikidza neMoira.
Kuzvimirira. Iwe unogona kuita zvese izvi iwe pachako, pasina rubatsiro rweDevOps mainjiniya. Uye izvi zvakanakira, nekuti iwe unogona kutarisa chirongwa chako izvozvi, haufanirwe kubvunza chero munhu - kutanga basa kana kuita shanduko.
Tiri kuvavarira chii?
Zvese zvakanyorwa pazasi hazvisi zvekungofungira pfungwa, asi chimwe chinhu chakanangana nacho icho kana matanho ekutanga atorwa.
- Anomaly detector. Tinoda kugadzira sevhisi inoenda kune yedu Graphite storages uye tarisa metric yega yega tichishandisa akasiyana algorithms. Patova nema algorithms atinoda kuona, pane data, tinoziva kushanda nayo.
- Metadata. Tine masevhisi akawanda, anochinja nekufamba kwenguva, sezvakangoita vanhu vanoshanda navo. Kugara uchichengeta zvinyorwa pamaoko haisi sarudzo. Ndosaka tave kuisa metadata mumamicroservices edu. Inoti ndiani akaigadzira, mitauro yainodyidzana nayo, SLA zvinodiwa, kupi uye kunaani zviziviso zvinofanirwa kutumirwa. Kana uchitumira sevhisi, data yese yesangano inogadzirwa yakazvimirira. Nekuda kweizvozvo, iwe unowana maviri ekubatanidza - imwe yekukonzeresa, imwe kune dashboards muGrafana.
- Monitoring mudzimba dzese. Isu tinotenda kuti vese vanogadzira vanofanirwa kushandisa system yakadai. Muchiitiko ichi, iwe unogara uchinzwisisa kuti traffic yako iripi, chii chinoitika kwairi, kwainowira, uko kusasimba kwayo. Kana, semuenzaniso, chimwe chinhu chikauya uye chikakanganisa sevhisi yako, ipapo iwe uchadzidza nezvazvo kwete panguva yekufona kubva kune maneja, asi kubva payambiro, uye iwe unogona kuvhura nekukurumidza matanda uye kuona zvakaitika ipapo.
- Kuita kwepamusoro. Basa redu riri kuramba richikura, uye nhasi rinoita nezve 2 metric values ββpaminiti. Gore rakapfuura, nhamba iyi yaiva 000 000. Uye kukura kunoenderera mberi, uye izvi zvinoreva kuti mushure menguva yakati Graphite (zevezera) ichatanga kutakura zvakanyanya disk subsystem. Sezvandambotaura, iyi yekutarisa sisitimu ndeyepasirese nekuda kwekuchinjana kwezvikamu. Mumwe munhu anochengetedza uye anogara achiwedzera zvivakwa zvavo zvakanangana neGraphite, asi isu takasarudza kuenda neimwe nzira: shandisa sechipo chemetrics edu. Shanduko iyi inenge yapera, uye munguva pfupi iri kutevera ndichakuudza zvakadzama kuti izvi zvakaitwa sei: ndeapi matambudziko aivepo uye kuti akakundwa sei, mafambiro akafamba sei, ini ndichatsanangura zvikamu zvakasarudzwa sekusunga uye magadzirirwo azvo.
Ndinokutendai nekuteerera kwenyu! Bvunza mibvunzo yako pamusoro wenyaya, ini ndichaedza kupindura pano kana mune zvinotevera zvinyorwa. Zvichida mumwe munhu ane ruzivo rwekuvaka yakafanana yekutarisa sisitimu kana kushandura kuClickhouse mune yakafanana mamiriro - igovera mune zvakataurwa.
Source: www.habr.com