Imwe monitoring system

16 modem, 4 cellular operators = Inobuda kumhanya 933.45 Mbit/s

Nhanganyaya

Mhoro! Ichi chinyorwa chiri pamusoro pemanyorero atakaita a new monitoring system pachedu. Iyo inosiyana kubva kune iripo mukukwanisa kwayo kuwana yakakwirira-frequency synchronous metrics uye yakaderera kwazvo zviwanikwa kushandiswa. Chiyero chekuvhota chinogona kusvika 0.1 milliseconds nekunyatso kuwiriranisa pakati pemetrics yegumi nanoseconds. Ese mabhinari mafaira anotora 10 megabytes.

Nezve purojekiti

Tine chigadzirwa chakasarudzika. Isu tinogadzira mhinduro yakazara yekupfupikisa mabudiro uye kukanganisa kushivirira kwematanho ekufambisa data. Apa ndipo kana paine akati wandei chiteshi, ngatiti Operator1 (40Mbit/s) + Operator2 (30Mbit/s)+ Chimwe chinhu (5 Mbit/s), mhedzisiro yacho imwe yakagadzikana uye inokurumidza chiteshi, kumhanya kwayo kunenge kwakafanana. izvi: (40+ 30+5)x0.92=75×0.92=69 Mbit/s.

Mhinduro dzakadai dziri kudiwa uko kugona kwechero chiteshi hakuna kukwana. Semuenzaniso, zvekufambisa, vhidhiyo yekutarisa masisitimu uye chaiyo-nguva yevhidhiyo kutenderera, kutepfenyura kwehupenyu terevhizheni neredhiyo nhepfenyuro, chero nzvimbo dzepasi peguta apo pakati pevafambisi venhare pane vamiriri veBig Four uye kumhanya pane imwe modem / chiteshi hakuna kukwana. .
Kune imwe neimwe yenzvimbo idzi, isu tinogadzira mutsara wakasiyana wemidziyo, asi software yavo chikamu chakangofanana uye yemhando yepamusoro yekutarisa sisitimu ndeimwe yemamodule ayo makuru, pasina kurongeka kwacho kwaisazogoneka chigadzirwa.

Mukufamba kwemakore akati wandei, takakwanisa kugadzira multi-level, inokurumidza, muchinjika-chikuva uye isina kuremerwa yekutarisa system. Izvi ndizvo zvatinoda kugovana nenharaunda yedu inoremekedzwa.

Kugadzirwa kwedambudziko

Iyo yekutarisa sisitimu inopa metrics emakirasi maviri akasiyana akasiyana: real-time metrics uye mamwe ese. Iyo monitoring system yaingove nezvinodiwa zvinotevera:

1. Yakakwira-frequency synchronous yekuwana chaiyo-nguva metrics uye kutamiswa kwavo kubva kune yekutaurirana manejimendi system pasina kunonoka.
   Yakakwira frequency uye kuwiriranisa kwemametric akasiyana hakuna kukosha chete, kwakakosha pakuongorora iyo entropy yematanho ekufambisa data. Kana mune imwe chiteshi chekufambisa data avhareji yekunonoka imakumi matatu emamilliseconds, ipapo kukanganisa mukuyananisa pakati peasara metrics yemillisecond imwe chete kunotungamira mukuderedzwa kwekumhanya kweiyo chiteshi chiteshi neinosvika 30%. Kana tikakanganisa nguva ne5 millisecond pamatanho mana, kukurumidza kuderedzwa kunogona kudonha nyore kusvika makumi matatu%. Uye zvakare, entropy mumachani inoshanduka nekukurumidza, saka kana tikaiyera isingasviki kamwe chete yega 1 milliseconds, pamatanho anokurumidza nekunonoka kudiki tinowana yakanyanya kukurumidza kuderedzwa. Ehe, kurongeka kwakadaro hakudiwi kune ese metrics uye kwete mumamiriro ese. Kana kunonoka muchiteshi kuri 4 milliseconds, uye isu tichishanda neakadaro, ipapo kukanganisa kwe30 millisecond kunenge kusingaonekwe. Zvakare, kune hupenyu hwekutsigira masisitimu metrics, isu tine yakakwana yekuvhota uye kuwiriranisa mazinga emasekonzi maviri, asi iyo yekutarisa sisitimu pachayo inofanirwa kukwanisa kushanda neakanyanya-akakwirira mareti ekuvhota uye ekupedzisira-chaiyo kuwiriranisa kwemametrics.
2. Zvishoma zvekushandisa zviwanikwa uye imwe stack.
   Mudziyo wekupedzisira unogona kunge une simba pa-board complex unokwanisa kuongorora mamiriro ezvinhu mumugwagwa kana kuita biometric kurekodhwa kwevanhu, kana komputa ine saizi yemichindwe yebhodhi inopfekwa nemusoja pasi penhumbi dzake dzokurwa nadzo kuendesa vhidhiyo mukati. nguva chaiyo mumamiriro asina kunaka ekukurukurirana. Pasinei nemhando dzakasiyana dzezvivakwa uye simba rekombuta, isu tinoda kuve neiyo software stack.
3. Umbrella architecture
   Metrics inofanirwa kuunganidzwa uye kuunganidzwa pamudziyo wekupedzisira, kuchengetwa munharaunda, uye kuratidzwa munguva chaiyo uye zvekare. Kana paine chinongedzo, endesa data kune yepakati monitoring system. Kana pasina kubatana, mutsara wekutumira unofanirwa kuunganidza uye kusadya RAM.
4. API yekubatanidza mutengi yekutarisa sisitimu, nekuti hapana anoda akawanda ekutarisa masisitimu. Mutengi anofanirwa kuunganidza data kubva kune chero madivayiri uye netiweki mune imwechete yekutarisa.

Chii chaitika

Kuti urege kuremedza iyo yatove inonakidza kuverenga kwenguva refu, ini handisi kuzopa mienzaniso uye zviyero zveese masystem ekutarisa. Izvi zvichatungamira kune imwe nyaya. Ndichangotaura kuti hatina kukwanisa kuwana hurongwa hwekutarisa hunokwanisa kutora mametric maviri panguva imwe chete nekanganiso isingasviki 1 millisecond uye inoshanda zvakaenzana zvese paArM architecture ine 64 MB ye RAM uye pa x86_64 architecture ine 32. GB ye RAM. Nokudaro, takasarudza kunyora zvedu, izvo zvinogona kuita izvi zvose. Hezvino izvo zvatakawana:

Kupfupikisa kubuda kwematanho matatu kune akasiyana network topology

Kuona mamwe akakosha metrics

akitekicha

Isu tinoshandisa Golang semutauro mukuru wekuronga, zvese pamudziyo uye munzvimbo yedata. Yakarerutsa zvakanyanya hupenyu nekuita kwayo multitasking uye kugona kuwana imwe yakabatana yakabatana inogoneka bhinari faira kune yega sevhisi. Nekuda kweizvozvo, isu tinochengetedza zvakanyanya muzviwanikwa, nzira uye traffic yekuendesa sevhisi kupedzisa zvishandiso, nguva yekuvandudza uye kodhi debugging.

Iyo sisitimu inoshandiswa zvinoenderana neiyo classic modular musimboti uye ine akati wandei ma subsystems:

1. Metrics kunyoresa.
   Imwe neimwe metric inoshumirwa neshinda yayo uye inowiriraniswa mumatanho. Isu takakwanisa kuwana kuwiriranisa kurongeka kweanosvika gumi nanoseconds.
2. Metrics kuchengetedza
   Isu takanga tichisarudza pakati pekunyora yedu yekuchengetera nguva yakatevedzana kana kushandisa chimwe chinhu chaive chiripo. Dhatabhesi rinodiwa pakutarisisa data inoenderana nekuona kunotevera.Kureva kuti, haina data yekunonoka muchiteshi chesekondi 0.5 ega ega kana kuverenga kukanganisa munetiweki yekutakura, asi pane kukurumidza pane yega yega 500 milliseconds. Pamusoro pezvakakwirira zvinodikanwa zvekuyambuka-platform uye yakaderera zviwanikwa zvekushandisa, zvakakosha zvakanyanya kuti isu tikwanise kugadzirisa. data ndiko kwainochengetwa. Izvi zvinochengetedza hukuru hwemakomputa zviwanikwa. Isu tanga tichishandisa iyo Tarantool DBMS muchirongwa ichi kubvira 2016 uye kusvika pari zvino hatisi kuona inotsiviwa nayo pakatarisana. Flexible, ine yakakwana yekushandisa zviwanikwa, kupfuura yakakwana tekinoroji rutsigiro. Tarantool inoshandisawo GIS module. Ehe, haina simba sePostGIS, asi yakakwana kumabasa edu ekuchengetedza mamwe metric ane hukama nenzvimbo (anoenderana nekutakura).
3. Kuona metrics
   Zvese zviri nyore pano. Isu tinotora data kubva mudura toiratidza ingave munguva chaiyo kana kudzoka.
4. Kuwiriranisa data neiyo central monitoring system.
   Iyo yepakati yekutarisa sisitimu inogamuchira data kubva kune ese maturusi, inoichengeta ine nhoroondo yakatarwa uye inotumira kune iyo Mutengi yekutarisa sisitimu kuburikidza neAPI. Kusiyana neakare ekutarisa masisitimu, umo "musoro" unofamba uchitenderedza uye kuunganidza data, isu tine yakatarisana chirongwa. Iyo michina pachayo inotumira data kana paine kubatana. Iyi ipfungwa yakakosha, nekuti inobvumidza iwe kuti ugamuchire data kubva kune mudziyo kwenguva idzo nguva iyo yanga isiri kuwanikwa uye kwete kurodha machaneti uye zviwanikwa apo mudziyo usipo. Isu tinoshandisa Influx yekutarisa server seyepakati yekutarisa system. Kusiyana nemaanalogue ayo, inogona kuunza data retrospective (kureva kuti, ine chitambi chenguva chakasiyana kubva panguva yakagamuchirwa metrics). Mametrics akaunganidzwa anoonekwa naGrafana, akagadziridzwa nefaira. Iyi yakajairwa stack yakasarudzwawo nekuti ine yakagadzirira-yakagadzirwa API yekubatanidza nechero chero mutengi wekutarisa system.
5. Data synchronization ine yepakati mudziyo maneja system.
   Iyo dhizaini manejimendi sisitimu inoshandisa Zero Touch Provisioning (updating firmware, configuration, nezvimwewo) uye, kusiyana nehurongwa hwekutarisa, inogamuchira matambudziko chete pachigadzirwa. Izvi zvinokonzeresa kushanda kwe-on-board hardware watchdog masevhisi uye ese metrics ehupenyu hwekutsigira masisitimu: CPU uye SSD tembiricha, CPU mutoro, yemahara nzvimbo uye SMART hutano pamadisiki. Iyo subsystem yekuchengetedza inovakwa zvakare paTarantool. Izvi zvinotipa kukurumidza kukuru mukuunganidza nguva dzakatevedzana muzviuru zvemidziyo, uye zvakare inogadzirisa zvizere nyaya yekuwiriranisa data nemidziyo iyi. Tarantool ine yakanakisa queuing uye yakavimbiswa kuendesa system. Isu tine chinhu chakakosha ichi kunze kwebhokisi, zvakanaka!

Network manejimendi system

Chii chinotevera

Parizvino, yedu isina kusimba chinongedzo ndiyo yepakati yekutarisa system. Inoitwa 99.9% pane yakajairwa stack uye ine huwandu hwekuipa:

1. InfluxDB inorasa data kana simba rarasika. Sezvo mutemo, Mutengi anokurumidza kuunganidza zvose zvinobva kune zvigadzirwa uye dhatabhesi pachayo haina data yakakura kudarika maminitsi mashanu, asi mune ramangwana izvi zvinogona kuva marwadzo.
2. Grafana ine akati wandei matambudziko nekuunganidza data uye kuwiriranisa kwekuratidzira kwayo. Dambudziko rinowanzoitika nderokuti dhatabhesi rine nguva yakatevedzana ine nguva ye2 masekonzi kutanga kubva, taura, 00:00:00, uye Grafana anotanga kuratidza data mukuunganidza kubva +1 sekondi. Somugumisiro, mushandisi anoona girafu yekutamba.
3. Yakawandisa huwandu hwekodhi yekubatanidza API neyechitatu-bato rekutarisa masisitimu. Inogona kuitwa zvakanyanya compact uye zvechokwadi kunyorwazve muGo)

Ini ndinofunga imi mese makaona zvakakwana zvakaita Grafana uye munoziva matambudziko ayo pasina ini, saka ini handingaremedze iyo post nemifananidzo.

mhedziso

Ini nemaune handina kutsanangura ruzivo rwehunyanzvi, asi ndakatsanangura chete dhizaini yekutanga yegadziriro ino. Chekutanga, kutsanangura zvizere sisitimu, chimwe chinyorwa chichadikanwa. Chechipiri, havasi vose vachafarira izvi. Nyora mune zvakataurwa kuti ndeupi ruzivo rwehunyanzvi rwaungade kuziva.

Kana paine chero munhu ane mibvunzo inopfuura chiyero chechinyorwa ichi, unogona kunyora kwandiri pa a.rodin @ qedr.com

Source: www.habr.com