Saka iwe unounganidza metrics. Sezvatiri. Isu tinounganidza zvakare metrics. Zvechokwadi, zvakakosha kune bhizinesi. Nhasi tichataura nezve yekutanga link yekutarisa system yedu - statsd-inoenderana aggregation server. , nei takainyora uye nei takasiya brubeck.
Kubva muzvinyorwa zvedu zvakapfuura (, ) unogona kuona kuti kusvika imwe nguva takaunganidza mamakisi tichishandisa . Yakanyorwa muC. Kubva pane imwe kodhi yekuona, iri nyore sepulagi (izvi zvakakosha kana iwe uchida kugovera) uye, zvinonyanya kukosha, inobata mavhoriyamu edu emamiriyoni maviri emamita pasekondi (MPS) pakakwirira. pasina matambudziko. Zvinyorwa zvinoti rutsigiro rwemamiriyoni mana eMPS ane asterisk. Izvi zvinoreva kuti iwe uchawana iyo yakataurwa nhamba kana iwe ukagadzirisa network nenzira kwayo paLinux. (Hatizive kuti maMPS mangani aunogona kuwana kana ukasiya network sezvairi). Pasinei zvapo neizvi, takava nokunyunyuta kwakakomba kunoverengeka pamusoro pebrubeck.
Mhosva 1. Github, muvambi wepurojekiti, akamira kuitsigira: kuburitsa zvigamba nekugadzirisa, kugamuchira yedu uye (kwete yedu chete) PR. Mumwedzi mishoma yapfuura (kune imwe nzvimbo kubva munaFebruary-March 2018), basa rakatangazve, asi zvisati zvaitika pakanga pane makore anenge maviri ekudzikama kwakakwana. Pamusoro pezvo, chirongwa ichi chiri kugadzirwa , iyo inogona kuva chipingamupinyi chakakomba kuunzwa kwezvinhu zvitsva.
Mhosva 2. Kururama kwekuverenga. Brubeck inounganidza huwandu hwe65536 kukosha kwekuunganidza. Kwatiri, kune mamwe ma metrics, panguva yekuunganidza (30 masekondi), akawanda akawanda maitiro anogona kusvika (1 panhongonya). Nekuda kweiyi sampling, iyo yakanyanya uye yakaderera kukosha inoita seisina basa. Semuenzaniso, seizvi:
Sezvazvaive
Zvazvaifanira kunge zvakaita
Nechikonzero chimwe chete, mari inowanzoverengwa zvisiri izvo. Wedzera pano tsikidzi ine 32-bit float kufashukira, iyo inowanzotumira sevha kune segfault kana inogamuchira inoita seisina mhosva metric, uye zvese zvinova zvakanaka. Iyo bug, nenzira, haina kugadziriswa.
Uye, pakupedzisira, Mhosva X. Panguva yekunyora, takagadzirira kuzvipa kune ese gumi nemana akawanda kana mashoma ekushanda statsd maitirwo atakwanisa kuwana. Ngatimbofungidzira kuti zvimwe zvivakwa zvakura zvekuti kugamuchira mamirioni mana MPS hazvichakwane. Kana kunyangwe iyo isati yakura, asi ma metrics atove akakosha kwauri zvekuti kunyangwe ipfupi, 14-4 maminetsi anonyura mumachati anogona kutove akatsoropodza uye kukonzera kubhuroka kwekushungurudzika kusingakundiki pakati pemaneja. Sezvo kurapa kuora mwoyo riri basa risingaongi, mhinduro dzounyanzvi dzinodiwa.
Chekutanga, kukanganisa kushivirira, kuitira kuti dambudziko rakangoerekana pavhavha rirege kukonzera psychiatric zombie apocalypse muhofisi. Chechipiri, kuyera kukwanisa kugamuchira anopfuura mamirioni mana MPS, pasina kuchera zvakadzika mukati meLinux network stack uye zvakadzikama kukura "muhupamhi" kune saizi inodiwa.
Sezvo taive nenzvimbo yekuyera, takasarudza kutanga nekushivirira mhosva. "ABOUT! Mhosva kushivirira! Zviri nyore, tinogona kuzviita, βtakafunga ndokutangisa maseva maviri, tichisimudza kopi yebrubeck pane imwe neimwe. Kuti tiite izvi, taifanira kukopa traffic nemametrics kune ese maseva uye kunyange kunyorera izvi . Isu takagadzirisa dambudziko rekushivirira mhosva neizvi, asi ... kwete zvakanyanya. Pakutanga zvinhu zvese zvaiita sezvikuru: brubeck imwe neimwe inounganidza vhezheni yayo yekuunganidza, inonyora data kuGraphite kamwe chete mumasekonzi makumi matatu, ichinyora nguva yekare (izvi zvinoitwa padivi reGraphite). Kana imwe sevha ikangoerekana yakundikana, isu tinogara tine yechipiri ine kopi yayo yedata rakaunganidzwa. Asi heino dambudziko: kana sevha ikatadza, "saw" inoonekwa pamagirafu. Izvi zvinokonzerwa nekuti brubeck's 30-second intervals haina kuwiriraniswa, uye panguva yekuparara imwe yacho haina kunyorwa. Kana yechipiri server inotanga, chinhu chimwe chete chinoitika. Zvinoshivirira, asi ndinoda zviri nani! Dambudziko re scalability harina kuperawo. Mametric ese achiri "kubhururuka" kune imwe sevha, uye saka isu tinogumira kune imwecheteyo 30-2 miriyoni MPS, zvichienderana netiweki level.
Kana iwe uchifunga zvishoma nezvedambudziko uye panguva imwechete uchichera chando nefoshoro, ipapo iyo inotevera pfungwa yakajeka inogona kuuya mupfungwa: iwe unoda statsd inogona kushanda mune yakagoverwa mode. Ndiko kuti, imwe inoshandisa kuwiriranisa pakati penodhi munguva uye metrics. "Ehe, mhinduro yakadai ingangove iripo," takataura ndokuenda kuGoogleβ¦. Uye havana chavakawana. Mushure mekupfuura nepakati pezvinyorwa zveakasiyana statsd ( kubva muna Zvita 11.12.2017, XNUMX), hapana chatakawana. Sezviri pachena, havagadzirisi kana vashandisi vemhinduro idzi havasati vasangana neakawanda metrics, zvikasadaro vaizouya nechimwe chinhu.
Uye takabva tarangarira nezve "toyi" statsd - bioyino, iyo yakanyorwa kuJust for Fun hackathon (zita repurojekiti rakagadzirwa nerunyoro isati yatanga hackathon) uye takaona kuti isu taida nekukurumidza statsd yedu. Yechii?
- nekuti kune mashoma ma statsd clones munyika,
- nekuti zvinokwanisika kupa chinodiwa kana padhuze kune yaunoda kukanganisa kushivirira uye scalability (kusanganisira synchronizing aggregated metrics pakati pemaseva uye kugadzirisa dambudziko rekutumira kupokana),
- nekuti zvinokwanisika kuverenga metrics zvakanyatsonaka kupfuura zvinoita brubeck,
- nekuti iwe unogona kuunganidza huwandu hwakadzama iwe pachako, izvo brubeck haana kupa kwatiri,
- nokuti ndaiva nemukana wekuronga zvangu hyperperformance distributedscalelabplication, iyo isingazodzokorore zvachose chivakwa cheimwe hyperfor yakafanana ... zvakanaka, ndizvozvo.
Chii chokunyora pamusoro pacho? Zvechokwadi, muRust. Sei?
- nekuti paitove neprototype solution,
- nekuti munyori wechinyorwa aitoziva Rust panguva iyoyo uye aida kunyora chimwe chinhu mairi kuti chigadzirwe nemukana wekuchiisa pachena,
- nekuti mitauro ine GC haina kukodzera kwatiri nekuda kwechimiro chetraffic yakagamuchirwa (inenge nguva chaiyo) uye kumbomira kweGC hazvigamuchirwi,
- nekuti iwe unoda yakanyanya kuita inofananidzwa neC
- nekuti Rust inotipa isingatyi concurrency, uye dai takatanga kuinyora muC/C ++, tingadai takaisa muhutete hwakanyanya, buffer mafashama, mamiriro erudzi uye mamwe mazwi anotyisa kupfuura brubeck.
Pakaitawo kupopotedzana naRust. Iyo kambani yakanga isina ruzivo rwekugadzira mapurojekiti muRust, uye ikozvino isu hatisi kuronga kuishandisa muchirongwa chikuru. Naizvozvo, pakanga paine kutya kwakakomba kuti hapana chaizobuda, asi takasarudza kutora mukana ndokuedza.
Nguva yakafamba...
Pakupedzisira, mushure mekuedza kunoverengeka kwakakundikana, shanduro yekutanga yekushanda yakagadzirira. Chii chaitika? Izvi ndizvo zvakaitika.
Imwe neimwe node inogamuchira yayo yega seti yemametric uye inoaunganidza, uye haiunganidze metrics emhando idzo apo seti yavo izere inodiwa pakuunganidzwa kwekupedzisira. Manodhi akabatanidzwa kune mumwe nemumwe neimwe rudzi rwekugovera lock protocol, iyo inokubvumira kuti usarudze pakati pavo imwe chete (pano takachema) iyo yakakodzera kutumira metrics kune Mukuru. Dambudziko iri parizvino riri kugadziriswa ne , asi mune ramangwana vavariro dzemunyori dzinowedzera kusvika Raft, uko akanyanya kufanirwa achave, chokwadi, kuve kubvumirana mutungamiri node. Pamusoro pekubvumirana, node kazhinji (kamwe pasekondi nekukasira) vanotumira kuvavakidzani vavo zvikamu zve pre-aggregated metrics zvavakakwanisa kuunganidza musekondi iyoyo. Zvinoitika kuti kuyera uye kushivirira kukanganisa kunochengetedzwa - imwe neimwe node ichakabata yakazara seti yemametrics, asi metrics inotumirwa yatove yakaunganidzwa, kuburikidza neTCP uye yakavharirwa mubhinari protocol, saka mutengo wekudzokorora wakaderedzwa zvakanyanya kana uchienzaniswa neUDP. Pasinei nehuwandu hwakakura hwema metrics anouya, kuunganidza kunoda ndangariro shoma uye kunyange shoma CPU. Kune edu akanyanya kudzvanya mertics, aya angori mashoma mashoma emakumi emamegabytes edata. Seimwe bhonasi yekuwedzera, isu hatiwane zvisina kufanira data inonyorazve muGraphite, sezvaive zvakaita burbeck.
UDP mapaketi ane metrics haana kuenzana pakati pemanodhi panetiweki michina kuburikidza neyakapusa Round Robin. Ehe, iyo network hardware haina kupatsanura zviri mukati memapakiti uye nekudaro inogona kudhonza zvakanyanya kupfuura 4M mapaketi pasekondi, tisingataure metrics nezvainoziva nezvayo zvachose. Kana isu tikafunga kuti metrics haauye imwe panguva mupakiti yega yega, saka isu hatifanoona chero matambudziko ekuita munzvimbo ino. Kana sevha ikaparara, network network inokurumidza (mukati me1-2 seconds) inoona ichi chokwadi uye inobvisa server yakaputsika kubva pakutenderera. Nekuda kweizvi, passive (kureva, asiri-mutungamiri) node anogona kubatidzwa nekudzimwa zvisingaite pasina kuona kudhirowa pamachati. Iyo yakawanda yatinorasikirwa chikamu chema metrics akauya sekondi yekupedzisira. Kurasikirwa nekukasira / kudzima / kuchinjika kwemutungamiri kucharamba kuchigadzira diki anomaly (iyo 30 yechipiri nguva ichiri kunze kwekuenderana), asi kana paine kutaurirana pakati pemanodhi, matambudziko aya anogona kudzikiswa, semuenzaniso, nekutumira mapeji ekuyananisa. .
Zvishoma nezvechimiro chemukati. Iko kushandiswa, hongu, kwakaverengerwa, asi iyo yekushongedza dhizaini yakasiyana neiyo inoshandiswa mubrubeck. Iyo tambo mu brubeck yakafanana - imwe neimwe yadzo ine basa rekuunganidza ruzivo uye kuunganidzwa. Mu bioyino, vashandi vakakamurwa kuita mapoka maviri: avo vane mutoro wetiweki uye avo vane basa rekuunganidza. Ichi chikamu chinokutendera iwe kuti uwedzere kuchinjika kubata application zvichienderana nerudzi rwemetrics: painodiwa kuunganidzwa kwakakura, unogona kuwedzera aggregator, uko kune yakawanda network traffic, unogona kuwedzera huwandu hwetiweki inoyerera. Parizvino, pamaseva edu tinoshanda mu8 network uye 4 aggregation inoyerera.
Chikamu chekuverenga (chine basa rekuunganidza) chikamu chinofinha. Mabhafa akazadzwa netiweki anoyerera anogovaniswa pakati pekuyerera kwekuverenga, uko anozopatsanurwa uye akaunganidzwa. Pakukumbira, metrics inopihwa yekutumira kune dzimwe node. Zvese izvi, kusanganisira kutumira data pakati pemanodhi uye kushanda neConsul, inoitwa asynchronously, ichimhanya pahurongwa. .
Matambudziko mazhinji panguva yekusimudzira akakonzerwa nechikamu chetiweki chine mutoro wekugamuchira metrics. Chinangwa chikuru chekuparadzanisa network inoyerera muzvikamu zvakasiyana chaive chishuwo chekudzikisa nguva inoshandiswa nekuyerera kwete kuverenga data kubva pane socket. Sarudzo dzinoshandisa asynchronous UDP uye yenguva dzose recvmsg yakakurumidza kunyangarika: yekutanga inoshandisa yakawandisa mushandisi-nzvimbo CPU yekugadzirisa chiitiko, yechipiri inoda yakawandisa mamiriro ekuchinja. Saka ikozvino inoshandiswa nema buffers makuru (uye mabuffers, gentlemen officer, hapana kwauri!). Tsigiro yenguva dzose UDP yakachengeterwa mwenje makesi uko recvmmsg isingadiwi. Mune multimessage modhi, zvinokwanisika kuita chinhu chikuru: iyo yakawanda yenguva, iyo network tambo inomhanyisa OS mutsara - inoverenga data kubva pachisokisi uye ichiendesa kune iyo mushandisi buffer, ichingopota ichichinja kupa iyo yakazara buffer aggregators. Mutsetse mune socket hauunganidzire, huwandu hweakadonhedza mapaketi haakure.
taura pfungwa
Mune zvigadziriso zvekutanga, saizi yebhafa inoiswa kuti ive yakakura. Kana iwe ukaerekana wafunga kuyedza sevha iwe pachako, unogona kusangana nenyaya yekuti mushure mekutumira nhamba diki yemetrics, ivo havasvike muGraphite, vachisara mune network stream buffer. Kuti ushande nehuwandu hudiki hwemetrics, iwe unofanirwa kuseta bufsize uye basa-mutsetse-saizi kune madiki kukosha mune config.
Pakupedzisira, mamwe machati evanoda machati.
Nhamba dzehuwandu hwema metrics anouya kune yega sevha: anopfuura mamirioni maviri MPS.
Kudzima imwe yemanodhi uye kugovera zvakare ma metric anouya.
Statistics pamametrics anobuda: node imwe chete inogara ichitumira - the raid boss.
Nhamba dzekushanda kwenode imwe neimwe, uchifunga nezvekukanganisa mune dzakasiyana siyana system module.
Tsanangudzo yeanouya metrics (metric mazita akavanzwa).
Tiri kuronga kuitei nezvose izvi zvinotevera? Zvechokwadi, nyora kodhi, damn ...! Iyo purojekiti yakatanga kurongwa kuve yakavhurika-sosi uye icharamba yakadaro muhupenyu hwayo hwese. Zvirongwa zvedu zvepakarepo zvinosanganisira kuchinja kune yedu vhezheni yeRaft, kushandura peer protocol kune inotakurika, kuunza humwe huwandu hwemukati, mhando nyowani dzemetrics, bug kugadzirisa uye kumwe kuvandudzwa.
Zvechokwadi, munhu wose anogamuchirwa kuti abatsire mukugadzirwa kweprojekti: gadzira PR, Nyaya, kana zvichibvira isu tichapindura, kuvandudza, nezvimwe.
Zvichitaurwa, ndizvo chete vanhu, tengai nzou dzedu!
Source: www.habr.com