HighLoad++, Mikhail Makurov, Maxim Chernetsov (Intersvyaz): Zabbix, 100kNVPS pane imwe server

Musangano unotevera weHighLoad++ uchaitwa muna Kubvumbi 6 na7, 2020 muSt. Petersburg Details nematikiti batanidzo. HighLoad ++ Moscow 2018. Horo "Moscow". Mbudzi 9, 15:00. Izvi uye mharidzo.

HighLoad++, Mikhail Makurov, Maxim Chernetsov (Intersvyaz): Zabbix, 100kNVPS pane imwe server

* Kutarisisa - online uye analytics.
* Yekutanga zvipimo zveZABBIX papuratifomu.
* Mhinduro yekuyera analytics kuchengetedza.
* Kugonesa kweZABBIX server.
* UI optimization.
* Ziva kushandisa sisitimu pasi pemitoro inodarika 40k NVPS.
* Mhedziso pfupi.

Mikhail Makurov (pano - MM): - Mhoroi mose!

Maxim Chernetsov (pano - MCH): - Masikati akanaka!

MM: – Rega ndizivise Maxim. Max injinjiniya ane tarenda, akanakisa networker wandinoziva. Maxim inobatanidzwa mune network uye masevhisi, kukura kwavo uye kushanda.

HighLoad++, Mikhail Makurov, Maxim Chernetsov (Intersvyaz): Zabbix, 100kNVPS pane imwe server

MCH: - Uye ndinoda kukuudza nezvaMikhail. Mikhail mugadziri weC. Akanyora akati wandei akakwira-mutoro traffic kugadzirisa mhinduro kukambani yedu. Isu tinogara uye tinoshanda muUrals, muguta revarume vakaoma Chelyabinsk, mukambani yeIntersvyaz. Kambani yedu inopa Internet uye tambo dzeterevhizheni masevhisi kuvanhu miriyoni imwe mumaguta gumi nematanhatu.

MM: - Uye zvakakodzera kutaura kuti Intersvyaz inopfuura kungopa, ikambani yeIT. Mazhinji emhinduro dzedu anogadzirwa nedhipatimendi redu reIT.

A: kubva kumaseva anogadzirisa traffic kuenda kunzvimbo yekufona uye mobile application. Idhipatimendi reIT parizvino rine vanhu vangangosvika makumi masere vane hunyanzvi hwakasiyana.

Nezve Zabbix uye mavakirwo ayo

MCH: - Uye zvino ndichaedza kuseta rekodhi rangu uye nditaure muminiti imwe chete kuti Zabbix chii (inozonzi "Zabbix").

Zabbix inozvimisa senge bhizinesi-chikamu kunze-kwe-the-bhokisi chekutarisa system. Iine akawanda maficha anoita kuti hupenyu huve nyore: mitemo yekukwira kwepamusoro, API yekubatanidza, kuita mapoka uye auto-kuonekwa kwevatenzi uye metrics. Zabbix ine anonzi maturusi ekuyera - proxies. Zabbix ndeye yakavhurika sosi system.

Muchidimbu nezvezvivakwa. Tinogona kutaura kuti ine zvikamu zvitatu:

HighLoad++, Mikhail Makurov, Maxim Chernetsov (Intersvyaz): Zabbix, 100kNVPS pane imwe server

  • Server. Yakanyorwa muna C. Nekuomesesa kugadzirisa uye kuendesa ruzivo pakati peshinda. Kugadziriswa kwese kunoitika mairi: kubva pakugamuchira kusvika pakuchengetedza kune dhatabhesi.
  • Yese data inochengetwa mudhatabhesi. Zabbix inotsigira MySQL, PostreSQL uye Oracle.
  • Iyo webhu interface yakanyorwa muPHP. Pane akawanda masisitimu inouya neApache server, asi inoshanda zvakanyanya mukubatana ne nginx + php.

Nhasi tinoda kutaura nyaya imwe kubva muhupenyu hwekambani yedu ine chekuita neZabbix...

Nyaya kubva muupenyu hwekambani yeIntersvyaz. Chii chatinacho uye chii chatinoda?

HighLoad++, Mikhail Makurov, Maxim Chernetsov (Intersvyaz): Zabbix, 100kNVPS pane imwe server
5 kana 6 mwedzi yapfuura. Rimwe zuva ndabva kubasa...

MCH: - Misha, mhoro! Ndinofara kuti ndakwanisa kukubata - pane hurukuro. Takava nematambudziko zvakare nekutarisa. Munguva yenjodzi huru, zvinhu zvese zvainonoka uye pakanga pasina ruzivo nezve mamiriro etiweki. Sezvineiwo, aka hakasi kekutanga izvi zvichiitika. Ndinoda rubatsiro rwenyu. Ngatiite kuti kutarisa kwedu kushande chero mamiriro ezvinhu!

MM: - Asi ngatitangei synchronize. Ndave nemakore mashoma ndisina kutarisa ikoko. Sekurangarira kwandinorangarira, takasiya Nagios ndokuchinjira kuZabbix anenge makore masere apfuura. Uye ikozvino tinoita kunge tine 8 maseva ane simba uye anenge gumi nemaviri proxies. Pane chandiri kuvhiringa here?

MCH: - Zvinenge. 15 maseva, mamwe acho ari chaiwo muchina. Chinonyanya kukosha ndechekuti haitiponese panguva yatinoda zvakanyanya. Senge tsaona - maseva anononoka uye hapana chaunoona. Takaedza kukwenenzvera kugadziridzwa, asi izvi hazvina kupa iyo yakakwana yekuwedzera kwekuita.

MM: - Zviri pachena. Wakatarisa chimwe chinhu, wanga watochera chimwe chinhu kubva kune diagnostics?

MCH: - Chinhu chekutanga chaunofanirwa kubata nacho dhatabhesi. MySQL inogara yakarodha, ichichengeta metrics matsva, uye Zabbix painotanga kugadzira boka rezviitiko, dhatabhesi inopinda mukuwandisa kwemaawa mashoma. Ini ndakatokuudza nezve optimization yekumisikidza, asi chaizvo gore rino vakagadziridza Hardware: maseva ane anopfuura zana gigabytes yekurangarira uye disk arrays paSSD RAIDs - hapana chikonzero chekuikura iyo mutsara mukufamba kwenguva. Tinoita sei?

MM: - Zviri pachena. Kazhinji, MySQL ndeye LTP dhatabhesi. Sezviri pachena, haichakodzeri kuchengetedza dura rema metrics ehukuru hwedu. Ngatizvionei.

MCH: - Ngatiitei!

Kubatanidzwa kweZabbix uye Clickhouse semhedzisiro yehackathon

Mushure mechinguva takagamuchira data inonakidza:

HighLoad++, Mikhail Makurov, Maxim Chernetsov (Intersvyaz): Zabbix, 100kNVPS pane imwe server

Yakawanda yenzvimbo mudhatabhesi yedu yaive yakagarwa nemetrics archive uye isingasviki 1% yakashandiswa kugadzirisa, matemplate uye marongero. Panguva iyoyo, tanga tichishandisa iyo Big data mhinduro yakavakirwa paClickhouse kweanopfuura gore. Mafambiro aaive aita aive pachena kwatiri. Pachirimo chedu Hackathon, ndakanyora kubatanidzwa kweZabbix neClickhouse ye server uye frontend. Panguva iyoyo, Zabbix yainge yatove nerutsigiro rweElasticSearch, uye takafunga kuvaenzanisa.

HighLoad++, Mikhail Makurov, Maxim Chernetsov (Intersvyaz): Zabbix, 100kNVPS pane imwe server

Kuenzanisa kweClickhouse uye Elasticsearch

MM: -Kuenzanisa, isu takagadzira mutoro wakafanana sewakapihwa neSevha yeZabbix uye takatarisa kuti masisitimu aizoita sei. Isu takanyora data mumabhechi emitsara ye1000, tichishandisa CURL. Isu takafungidzira pachine nguva kuti Clickhouse yaizonyanya kushanda kune iyo mutoro profile iyo Zabbix inoita. Mibairo yakatopfuura zvataitarisira:

HighLoad++, Mikhail Makurov, Maxim Chernetsov (Intersvyaz): Zabbix, 100kNVPS pane imwe server

Pasi pemamiriro akafanana ekuyedzwa, Clickhouse akanyora katatu data rakawanda. Panguva imwecheteyo, masisitimu ese ari maviri akadya zvakanyanya (zvishoma zviwanikwa) pakuverenga data. Asi Elastics yaida huwandu hukuru hwe processor pakurekodha:

HighLoad++, Mikhail Makurov, Maxim Chernetsov (Intersvyaz): Zabbix, 100kNVPS pane imwe server

Pakazara, Clickhouse yaive yakakwira zvakanyanya kune Elastix maererano nekushandisa processor uye kumhanya. Panguva imwecheteyo, nekuda kwekudzvanywa kwedata, Clickhouse inoshandisa ka11 kashoma pane hard drive uye inoita kanenge ka30 mashoma mashoma disk mashandiro:

HighLoad++, Mikhail Makurov, Maxim Chernetsov (Intersvyaz): Zabbix, 100kNVPS pane imwe server

MCH: - Ehe, basa reClickhouse rine disk subsystem rinoitwa zvakanyanya. Iwe unogona kushandisa hombe SATA dhisiki kune dhatabhesi uye kuwana kumhanya kwekunyora kwemazana ezviuru zvemitsara pasekondi. Iyo yekunze-ye-bhokisi system inotsigira sharding, kudzokorora, uye iri nyore kwazvo kugadzirisa. Isu tinogutsikana nekushandiswa kwayo mukati megore rose.

Kuti uwedzere zviwanikwa, unogona kuisa Clickhouse padivi peyako huru dhatabhesi uye nekudaro chengetedza yakawanda yeCPU nguva uye disk mashandiro. Isu takafambisa dura remetrics kune iripo Clickhouse masumbu:

HighLoad++, Mikhail Makurov, Maxim Chernetsov (Intersvyaz): Zabbix, 100kNVPS pane imwe server

Isu takasunungura dhatabhesi huru yeMySQL zvakanyanya zvekuti taigona kuisanganisa pamushini mumwe neZabbix server uye kusiya sevha yakatsaurirwa yeMySQL.

Kuvhota kunoshanda sei muZabbix?

4 mwedzi yapfuura

MM: - Zvakanaka, tinogona kukanganwa nezvematambudziko ane hwaro?

MCH: - Ndizvozvo zvechokwadi! Rimwe dambudziko ratinofanira kugadzirisa nderekunonoka kuunganidza data. Iye zvino ese edu gumi nemashanu maseva eproxy akaremerwa neSNMP uye maitiro ekuvhota. Uye hapana nzira kunze kwekuisa maseva matsva uye matsva.

MM: - Hukuru. Asi chekutanga, tiudze kuti kuvhota kunoshanda sei muZabbix?

MCH: -Muchidimbu, kune makumi maviri emhando dzemetrics uye gumi nemaviri nzira dzekuawana. Zabbix inogona kuunganidza data ingave mu "chikumbiro-mhinduro" modhi, kana kumirira data nyowani kuburikidza ne "Trapper Interface".

HighLoad++, Mikhail Makurov, Maxim Chernetsov (Intersvyaz): Zabbix, 100kNVPS pane imwe server

Zvakakosha kuziva kuti muZabbix yepakutanga iyi nzira (Trapper) ndiyo inokurumidza.

Kune maseva eproxy ekugovera mutoro:

HighLoad++, Mikhail Makurov, Maxim Chernetsov (Intersvyaz): Zabbix, 100kNVPS pane imwe server

MaProxies anogona kuita mabasa akafanana ekuunganidza sevhavha yeZabbix, achigamuchira mabasa kubva kwairi uye nekutumira akaunganidzwa metrics kuburikidza neTrapper interface. Iyi ndiyo nzira inokurudzirwa zviri pamutemo yekugovera mutoro. MaProxies anobatsirawo pakutarisa zvivakwa zviri kure zvinoshanda kuburikidza neNAT kana inononoka chiteshi:

HighLoad++, Mikhail Makurov, Maxim Chernetsov (Intersvyaz): Zabbix, 100kNVPS pane imwe server

MM: - Zvese zviri pachena nemavakirwo. Tinofanira kutarisa kune masources...

Mazuva mashoma gare gare

Iyo nyaya yekuti nmap fping yakakunda sei

MM: "Ndinofunga ndakachera chimwe chinhu."

MCH: - Ndiudze!

MM: - Ndakaona kuti kana ndichitarisa kuwanikwa, Zabbix inotarisa huwandu hwe128 mauto panguva. Ndakaedza kuwedzera nhamba iyi kusvika ku500 uye kubvisa nguva yepakati-packet muping yavo (ping) - izvi zvakapetwa kaviri kuita. Asi ndinoda nhamba huru.

MCH: - Mukuita kwangu, dzimwe nguva ndinofanira kutarisa kuwanikwa kwezviuru zvemauto, uye handisati ndamboona chero chinhu nekukurumidza kupfuura nmap yeizvi. Ndine chokwadi kuti iyi ndiyo nzira inokurumidza. Ngatiedzei! Isu tinofanirwa kuwedzera zvakanyanya huwandu hwevaenzi pane imwe iteration.

MM: - Tarisa kupfuura mazana mashanu? 600?

MCH: - Kanenge zviuru zviviri.

MM: - OK. Chinhu chakanyanya kukosha chandaida kutaura ndechekuti ndakaona kuti kuvhota kwakawanda muZabbix kunoitwa nenzira yakafanana. Isu tinofanirwa kuishandura kuita asynchronous mode. Ipapo isu tinokwanisa kuwedzera zvinoshamisa huwandu hwemetrics inounganidzwa nevavhoti, kunyanya kana isu tikawedzera huwandu hwemetrics per iteration.

MCH: - Hukuru! Uye rini?

MM: – Semazuva ose, nezuro.

MCH: -Isu takafananidza ese mavhezheni efping uye nmap:

HighLoad++, Mikhail Makurov, Maxim Chernetsov (Intersvyaz): Zabbix, 100kNVPS pane imwe server

Panhamba huru yevaenzi, nmap yaitarisirwa kunge yakapetwa kashanu inoshanda. Sezvo nmap inongotarisa kuwanikwa uye nguva yekupindura, takafambisa kuverenga kwekurasikirwa kune zvinokonzeresa uye zvakanyanya kuderedza kuwanikwa kwekutarisa nguva. Isu takawana iyo yakakwana nhamba yevaenzi ve nmap kuve vakatenderedza zviuru zvina pairi iteration. Nmap yakatibvumira kudzikisa mutengo weCPU wekuwanikwa cheki nekatatu uye kuderedza nguva kubva pamasekonzi zana kusvika gumi.

Polling optimization

MM: “Takabva tatanga kuita mapollers. Isu tainyanya kufarira kuona SNMP uye vamiririri. MuZabbix, kuvhota kunoitwa synchronously uye matanho akakosha akatorwa kuwedzera kushanda kwehurongwa. Mune synchronous mode, kusawanikwa kwemugamuchiri kunokonzera kukanganisa kukuru kwekuvhota. Kune hurongwa hwese hwehurumende, kune maitiro anokosha - anonzi asingasviki mavhoti, anoshanda chete nevasingasviki mauto:

HighLoad++, Mikhail Makurov, Maxim Chernetsov (Intersvyaz): Zabbix, 100kNVPS pane imwe server

Iyi itsanangudzo inoratidza matrix ehurumende, kuoma kwese kwehurongwa hwekushandura hunodiwa kuitira kuti hurongwa hurambe huchishanda. Mukuwedzera, synchronous polling pachayo inononoka:

HighLoad++, Mikhail Makurov, Maxim Chernetsov (Intersvyaz): Zabbix, 100kNVPS pane imwe server

Ndosaka zviuru zvevavhoti hova pamakumi evamiriri vakatadza kuunganidza huwandu hunodiwa hwe data kwatiri. Kuitwa kweasynchronous kwakagadzirisa kwete matambudziko chete nehuwandu hwetambo, asi zvakare yakarerutsa hurongwa hwehurumende yevasingawanikwi, nekuti kune chero nhamba yakatariswa mune imwe yekuvhota iteration, yakanyanya kumirira nguva yaive 1 timeout:

HighLoad++, Mikhail Makurov, Maxim Chernetsov (Intersvyaz): Zabbix, 100kNVPS pane imwe server

Pamusoro pezvo, takagadzirisa nekuvandudza hurongwa hwekuvhota hweSNMP zvikumbiro. Icho chokwadi ndechekuti vanhu vazhinji havagone kupindura kune akawanda SNMP zvikumbiro panguva imwe chete. Naizvozvo, isu takagadzira iyo yakasanganiswa modhi, apo kuvhota kweSNMP yemuiti mumwe chete kunoitwa asynchronously:

HighLoad++, Mikhail Makurov, Maxim Chernetsov (Intersvyaz): Zabbix, 100kNVPS pane imwe server

Izvi zvinoitirwa paki yese yevaenzi. Iyi modhi haina kunonoka kupfuura yakanyatso asynchronous, sezvo kuvhota zana nehafu yeSNMP kukosha kuchiri kukurumidza kupfuura 1 nguva yekubuda.

Zviyedzo zvedu zvakaratidza kuti iyo yakakwana nhamba yezvikumbiro mune imwe iteration ingangoita zviuru zvisere neSNMP polling. Pakazara, shanduko yeasynchronous mode yakatibvumira kuti tikurumidze kuita sarudzo ne8 times, mazana akati wandei.

MCH: -Kugadziriswa kwekuvhota kwakaratidza kuti isu hatigone kungobvisa ese ma proxies, asi zvakare kuderedza nguva dzekutarisa kwakawanda, uye ma proxies haazodiwi senzira yekugovera mutoro.

Inenge mwedzi mitatu yapfuura

Shandura zvivakwa - wedzera mutoro!

MM: - Zvakanaka, Max, inguva yekugadzira? Ndinoda sevha ine simba uye injiniya akanaka.

MCH: - Zvakanaka, ngatizvironge. Inguva yakakwira yekusimuka kubva pakufa kwe5 zviuru zvemetrics pasekondi.

Mangwanani mushure mekusimudzira

MCH: - Misha, takazvigadzirisa pachedu, asi mangwanani takadzokera kumashure ... Fungidzira kuti imhanho ipi yatakakwanisa kuwana?

MM: - 20 zviuru zvakanyanya.

MCH: - Hongu, 25! Sezvineiwo, tiri right patakatanga.

MM: - Sei? Wakamboongorora here?

MCH: - Hongu chokwadi! Heino, semuenzaniso, inonakidza pamusoro:

HighLoad++, Mikhail Makurov, Maxim Chernetsov (Intersvyaz): Zabbix, 100kNVPS pane imwe server

MM: - Ngatitarisei. Ndiri kuona kuti takaedza nhamba huru yetambo dzekuvhota:

HighLoad++, Mikhail Makurov, Maxim Chernetsov (Intersvyaz): Zabbix, 100kNVPS pane imwe server

Asi panguva imwecheteyo ivo havana kukwanisa kudzokorodza sisitimu kunyange nehafu:

HighLoad++, Mikhail Makurov, Maxim Chernetsov (Intersvyaz): Zabbix, 100kNVPS pane imwe server

Uye kuita kwese kudiki, kungangoita zviuru zvina metrics pasekondi:

HighLoad++, Mikhail Makurov, Maxim Chernetsov (Intersvyaz): Zabbix, 100kNVPS pane imwe server

Pane chimwe chinhu here?

MCH: - Ehe, mutsara wemumwe wevavhoti:

HighLoad++, Mikhail Makurov, Maxim Chernetsov (Intersvyaz): Zabbix, 100kNVPS pane imwe server

MM: - Pano iwe unogona kuona zvakajeka kuti nzira yekuvhota yakamirira "semaphores". Aya ndiwo makiya:

HighLoad++, Mikhail Makurov, Maxim Chernetsov (Intersvyaz): Zabbix, 100kNVPS pane imwe server

MCH: - Hazvina kujeka.

MM: - Tarisai, izvi zvakafanana nemamiriro ezvinhu apo boka reshinda riri kuedza kushanda nezviwanikwa izvo chete zvinogona kushanda nazvo panguva. Ipapo zvese zvavanogona kuita kugovera iyi sosi nekufamba kwenguva:

HighLoad++, Mikhail Makurov, Maxim Chernetsov (Intersvyaz): Zabbix, 100kNVPS pane imwe server

Uye kuita kwese kwekushanda nechinhu chakadaro kunogumira nekumhanya kweimwe musimboti:

HighLoad++, Mikhail Makurov, Maxim Chernetsov (Intersvyaz): Zabbix, 100kNVPS pane imwe server

Pane nzira mbiri dzekugadzirisa dambudziko iri.

Simudzira iyo Hardware yemuchina, chinja kune anokurumidza cores:

HighLoad++, Mikhail Makurov, Maxim Chernetsov (Intersvyaz): Zabbix, 100kNVPS pane imwe server

Kana kuti shandura dhizaini uye panguva imwechete shandura mutoro:

HighLoad++, Mikhail Makurov, Maxim Chernetsov (Intersvyaz): Zabbix, 100kNVPS pane imwe server

MCH: - Nenzira, pamushini wekuyedza isu tichashandisa mashoma macores pane ekurwa, asi iwo ari 1,5 times nekukurumidza mufrequency pamusimboti!

MM: - Zvakajeka? Iwe unofanirwa kutarisa pane server kodhi.

Nzira yedata muZabbix server

MCH: -Kuti tizvione, takatanga kuongorora kuti data rinofambiswa sei mukati meseva yeZabbix:

HighLoad++, Mikhail Makurov, Maxim Chernetsov (Intersvyaz): Zabbix, 100kNVPS pane imwe server

Mufananidzo wakanaka, handiti? Ngatifambei nayo nhanho nhanho kuti zvinyatsojeka kana kushoma. Pane shinda uye masevhisi ane basa rekuunganidza data:

HighLoad++, Mikhail Makurov, Maxim Chernetsov (Intersvyaz): Zabbix, 100kNVPS pane imwe server

Ivo vanoendesa akaunganidzwa metrics kuburikidza nesoketi kune Preprocessor maneja, kwavanochengetwa mumutsara:

HighLoad++, Mikhail Makurov, Maxim Chernetsov (Intersvyaz): Zabbix, 100kNVPS pane imwe server

Iyo "preprocessor maneja" inoendesa data kuvashandi vayo, iyo inoita preprocessing mirairo uye inovadzosera kuburikidza nesocket imwechete:

HighLoad++, Mikhail Makurov, Maxim Chernetsov (Intersvyaz): Zabbix, 100kNVPS pane imwe server

Mushure meizvi, maneja wepreprocessor anoachengeta munhoroondo cache:

HighLoad++, Mikhail Makurov, Maxim Chernetsov (Intersvyaz): Zabbix, 100kNVPS pane imwe server

Kubva ipapo ivo vanotorwa nevanonyura nhoroondo, vanoita basa rakawanda: semuenzaniso, kuverenga zvinokonzeresa, kuzadza kukosha cache uye, zvakanyanya kukosha, kuchengetedza metric munhoroondo yekuchengetedza. Kazhinji, maitiro acho akaoma uye anovhiringidza zvikuru.

HighLoad++, Mikhail Makurov, Maxim Chernetsov (Intersvyaz): Zabbix, 100kNVPS pane imwe server

MM: - Chinhu chekutanga chatakaona ndechekuti tambo zhinji dzinokwikwidza kune iyo inonzi "configuration cache" (nzvimbo yekuyeuka inochengeterwa ese masevha). Mathreads ane basa rekuunganidza data anoita kunyanya kuvharira kwakawanda:

HighLoad++, Mikhail Makurov, Maxim Chernetsov (Intersvyaz): Zabbix, 100kNVPS pane imwe server

...sezvo magadzirirwo anochengetedza kwete metrics chete neayo paramita, asiwo mitsetse kubva iyo vanovhota vanotora ruzivo rwezvekuita zvinotevera. Kana paine vakawanda vanovhota uye mumwe achivharira gadziriso, vamwe vanomirira zvikumbiro:

HighLoad++, Mikhail Makurov, Maxim Chernetsov (Intersvyaz): Zabbix, 100kNVPS pane imwe server

Vavhoti havafanire kupokana

HighLoad++, Mikhail Makurov, Maxim Chernetsov (Intersvyaz): Zabbix, 100kNVPS pane imwe server

Naizvozvo, chinhu chekutanga chatakaita kwaive kupatsanura mutsara kuita zvikamu zvina uye kubvumira vanovhota kuti vavhare mitsetse iyi, zvikamu izvi panguva imwe chete, pasi pemamiriro akachengeteka:

HighLoad++, Mikhail Makurov, Maxim Chernetsov (Intersvyaz): Zabbix, 100kNVPS pane imwe server

Izvi zvakabvisa makwikwi ekugadzirisa cache, uye kukurumidza kwevavhoti kwakawedzera zvakanyanya. Asi takazosangana nenyaya yekuti preprocessor maneja akatanga kuunganidza mutsara wemabasa:

HighLoad++, Mikhail Makurov, Maxim Chernetsov (Intersvyaz): Zabbix, 100kNVPS pane imwe server

Preprocessor maneja anofanira kukwanisa kuisa pamberi

Izvi zvakaitika mumamiriro ezvinhu apo akashaya kuita. Ipapo chaaigona kuita kuunganidza zvikumbiro kubva kumatanho ekuunganidza data uye nekuwedzera buffer yavo kudzamara yapedza ndangariro yese nekupunzika:

HighLoad++, Mikhail Makurov, Maxim Chernetsov (Intersvyaz): Zabbix, 100kNVPS pane imwe server

Kuti tigadzirise dambudziko iri, takawedzera socket yechipiri yaive yakatsaurirwa kune vashandi:

HighLoad++, Mikhail Makurov, Maxim Chernetsov (Intersvyaz): Zabbix, 100kNVPS pane imwe server

Saka, preprocessor maneja aive nemukana wekukoshesa basa rayo uye, kana buffer ikakura, basa nderekudzikisa kubviswa, zvichipa vashandi mukana wekutora iyi buffer:

HighLoad++, Mikhail Makurov, Maxim Chernetsov (Intersvyaz): Zabbix, 100kNVPS pane imwe server

Ipapo takaona kuti chimwe chezvikonzero nei kuderera kwacho kwaiva vashandi pachavo, sezvo vakanga vachikwikwidza kuti vawane chokushandisa chakanga chisingakoshi zvachose pabasa ravo. Isu takanyora dambudziko iri se-bug-kugadzirisa, uye rakatogadziriswa mushanduro itsva dzeZabbix:

HighLoad++, Mikhail Makurov, Maxim Chernetsov (Intersvyaz): Zabbix, 100kNVPS pane imwe server

Isu tinowedzera nhamba yezvigadziko - tinowana mhedzisiro

Kupfuurirazve, iyo preprocessor maneja pachayo yakava bhodhoro, sezvo iri shinda imwe. Yakazorora pakumhanya kwepakati, ichipa kumhanya kwakanyanya kweanosvika zviuru makumi manomwe metrics pasekondi:

HighLoad++, Mikhail Makurov, Maxim Chernetsov (Intersvyaz): Zabbix, 100kNVPS pane imwe server

Naizvozvo, takaita ina, nemaseti mana ezvigadziko, vashandi:

HighLoad++, Mikhail Makurov, Maxim Chernetsov (Intersvyaz): Zabbix, 100kNVPS pane imwe server

Uye izvi zvakatibvumira kuti tiwedzere kumhanya kusvika kunosvika zviuru zana nemakumi matatu emametrics:

HighLoad++, Mikhail Makurov, Maxim Chernetsov (Intersvyaz): Zabbix, 100kNVPS pane imwe server

Iyo isiri-mutsara wekukura inotsanangurwa nenyaya yekuti kukwikwidza kwenhoroondo cache yakaonekwa. 4 preprocessor mamaneja uye nhoroondo sinkers vakakwikwidzana nazvo. Panguva ino, takanga tichigamuchira angangoita zviuru zana nemakumi matatu zvemetrics pasekondi pamushini wekuyedza, tichishandisa ingangoita 130% ye processor:

HighLoad++, Mikhail Makurov, Maxim Chernetsov (Intersvyaz): Zabbix, 100kNVPS pane imwe server

Inenge mwedzi 2,5 yapfuura

Kuramba kubva kusnmp-community kwakawedzera maNVP nekamwe nehafu

MM: - Max, ini ndinoda mota nyowani yekuyedza! Hatichakodzeri mune yazvino.

MCH: - Chii chaunacho zvino?

MM: - Ikozvino - 130k NVPs uye sherufu-yakagadzirira processor.

MCH: - Wow! Kutonhora! Mira, ndine mibvunzo miviri. Zvinoenderana nekuverenga kwangu, zvatinoda zvakatenderedza 15-20 zviuru metrics pasekondi. Nei tichida zvimwe?

MM: "Ndinoda kupedza basa." Ndinoda kuona kuti tinogona sei kudzvanya kubva muhurongwa huno.

MCH: - Asi…

MM: "Asi hazvina basa kune bhizinesi."

MCH: - Zviri pachena. Uye mubvunzo wechipiri: tinogona kutsigira zvatinazvo isu pachedu, pasina rubatsiro rwemugadziri?

MM: - Handifungi. Kuchinja kuti cache yekumisikidza inoshanda sei idambudziko. Inobata shanduko mune dzakawanda shinda uye yakaoma chaizvo kuchengetedza. Zvingangodaro, zvichange zvakaoma zvikuru kuichengeta.

MCH: "Ipapo tinoda imwe nzira."

MM: - Pane sarudzo yakadaro. Isu tinogona kushandura kune anokurumidza cores, tichisiya iyo itsva yekuvhara system. Ticharamba tichiwana kuita kwe60-80 zviuru zvemetrics. Panguva imwecheteyo, tinogona kusiya mamwe ese ekodhi. Clickhouse uye asynchronous polling ichashanda. Uye zvichava nyore kuchengetedza.

MCH: - Zvinoshamisa! Ndinoti timire pano.

Mushure mekugadzirisa chikamu cheseva, takazokwanisa kuburitsa kodhi nyowani mukugadzira. Isu takasiya dzimwe shanduko tichifarira kuchinjira kumuchina une kukurumidza cores uye kuderedza nhamba yekuchinja kwekodhi. Isu takarerutsawo magadziriso uye takabvisa macros muzvinhu zve data pazvinogoneka, sezvavanounza kumwe kukiya.

HighLoad++, Mikhail Makurov, Maxim Chernetsov (Intersvyaz): Zabbix, 100kNVPS pane imwe server

Semuyenzaniso, kusiya snmp-community macro, iyo inowanzo wanikwa muzvinyorwa nemienzaniso, kwatiri kwaita kuti zvikwanisike kuenderera mberi nekumhanyisa NVPs kanenge ka1,5.

Mushure memazuva maviri mukugadzirwa

Kubvisa nhoroondo yezvakaitika pop-ups

MCH: - Misha, tanga tichishandisa system kwemazuva maviri, uye zvese zvinoshanda. Asi chete kana zvese zvashanda! Isu takanga taronga basa nekuchinjisa chikamu chakakura kwazvo chetiweki, uye takatarisa zvakare nemaoko edu kuti chii chakakwira uye chii chisina.

MM: - Hazvigoneki! Takaongorora zvese ka10. Sevha inobata kunyange kusawanikwa kwetiweki kwakazara ipapo ipapo.

MCH: - Hongu, ndinonzwisisa zvose: server, database, top, austat, logs - zvinhu zvose zvinokurumidza ... Asi isu tinotarisa pawebhu web interface, uye pane processor "musherufu" pane sevha uye izvi:

HighLoad++, Mikhail Makurov, Maxim Chernetsov (Intersvyaz): Zabbix, 100kNVPS pane imwe server

MM: - Zviri pachena. Ngatitarise dandemutande. Takaona kuti mumamiriro ezvinhu apo pakanga paine nhamba huru yezviitiko zvinoshanda, majeti mazhinji epamoyo akatanga kushanda zvishoma nezvishoma:

HighLoad++, Mikhail Makurov, Maxim Chernetsov (Intersvyaz): Zabbix, 100kNVPS pane imwe server

Chikonzero cheichi chaive chizvarwa chezvakaitika nhoroondo pop-ups inogadzirwa kune chimwe nechimwe chinhu chiri murondedzero. Naizvozvo, takasiya chizvarwa chemahwindo aya (akataura mitsara ye5 mukodhi), uye izvi zvakagadzirisa matambudziko edu.

Nguva yekurodha yemajeti, kunyangwe isingawanikwe zvachose, yakaderedzwa kubva pamaminetsi akati wandei kusvika kune inogamuchirwa 10-15 masekonzi kwatiri, uye nhoroondo inogona kuramba ichionekwa nekudzvanya panguva:

HighLoad++, Mikhail Makurov, Maxim Chernetsov (Intersvyaz): Zabbix, 100kNVPS pane imwe server

Mushure mebasa. 2 months ago

MCH: - Misha, uri kuenda here? Tinofanira kutaura.

MM: - Handina kuda. Chimwe chinhu neZabbix zvakare?

MCH: - Kwete, zorora! Ndaingoda kutaura: zvose zvinoshanda, ndinokutendai! Ndine doro.

Zabbix inoshanda

Zabbix ndeye yakasarudzika uye yakapfuma sisitimu uye basa. Inogona kushandiswa kune zvidiki kuisirwa kunze kwebhokisi, asi sezvo zvinodiwa zvinokura, zvinofanirwa kuvandudzwa. Kuchengeta dura rakakura remetrics, shandisa chengetedzo yakakodzera:

  • unogona kushandisa akavakirwa-mukati maturusi muchimiro chekubatanidza neElasticsearch kana kurodha nhoroondo kune mameseji mafaera (anowanikwa kubva mushanduro 4);
  • Unogona kutora mukana wechiitiko chedu uye kubatanidzwa neClickhouse.

Kuti uwedzere zvinoshamisa kumhanya kwekuunganidza metrics, unganidza uchishandisa asynchronous nzira uye uzvitumire kuburikidza neye trapper interface kune Zabbix server; kana iwe unogona kushandisa chigamba kuita Zabbix pollers asynchronous.

Zabbix yakanyorwa muC uye inoshanda chaizvo. Kugadzirisa akati wandei mabhodhoro ekuvaka anotendera iwe kuti uwedzere kuwedzera kuita kwayo uye, mune yedu ruzivo, tora anopfuura zviuru zana metrics pamushini we-single-processor.

HighLoad++, Mikhail Makurov, Maxim Chernetsov (Intersvyaz): Zabbix, 100kNVPS pane imwe server

Iyo imwechete Zabbix chigamba

MM: - Ndinoda kuwedzera mapoinzi akati wandei. Chirevo chese chazvino, bvunzo dzese, nhamba dzakapihwa kumisikidzwa yatinoshandisa. Tave kutora zvinosvika zviuru makumi maviri zvemetrics pasekondi kubva mairi. Kana uri kuedza kunzwisisa kana izvi zvichakuitira iwe, unogona kuenzanisa. Izvo zvakakurukurwa nhasi zvakatumirwa paGitHub nenzira yechigamba: github.com/miklert/zabbix

HighLoad++, Mikhail Makurov, Maxim Chernetsov (Intersvyaz): Zabbix, 100kNVPS pane imwe server

Chigamba chinosanganisira:

  • kubatanidzwa kwakazara neClickhouse (zvese Zabbix server uye frontend);
  • kugadzirisa matambudziko ane preprocessor maneja;
  • asynchronous polling.

Iyo chigamba inoenderana neese vhezheni 4, kusanganisira lts. Zvingangodaro, neshanduko shoma ichashanda pane vhezheni 3.4.

Ndinokutendai uchicherechedze.

Mibvunzo yako

Mubvunzo kubva kuvateereri (pano - A): - Masikati akanaka! Ndapota ndiudze, une zvirongwa zvekudyidzana kwakanyanya neboka reZabbix kana navo newe, kuitira kuti ichi chisiri chigamba, asi maitiro akajairika eZabbix?

MM: - Ehe, isu zvirokwazvo tichaita dzimwe shanduko. Chimwe chinhu chichaitika, chimwe chinhu chicharamba chiri muchigamba.

A: - Ndatenda zvikuru nemushumo wakanakisa! Ndokumbira undiudze, mushure mekushandisa chigamba, rutsigiro kubva kuZabbix rucharamba uye kuti ungaenderera sei nekuvandudza kune epamusoro shanduro? Zvinozoita here kugadzirisa Zabbix mushure mechikwata chako ku4.2, 5.0?

MM: - Handigoni kutaura chero chinhu pamusoro pekutsigirwa. Dai ndanga ndiri Zabbix technical support, ndingadai ndati kwete, nekuti iyi kodhi yemumwe munhu. Kana iri 4.2 codebase, chinzvimbo chedu ndechekuti: "Tichafamba nekufamba kwenguva, uye tichazvigadzirisa pachedu pane inotevera vhezheni." Naizvozvo, kwenguva yakati tichange tichitumira chigamba cheshanduro dzakagadziridzwa. Ini ndakatotaura mumushumo: nhamba yekuchinja neshanduro ichiri diki. Ndinofunga kuti kuchinja kubva ku3.4 kusvika ku4 kwakatitorera anenge maminitsi 15. Chimwe chinhu chakachinja ipapo, asi kwete kunyanya kukosha.

A: - Saka iwe unoronga kutsigira chigamba chako uye iwe unogona kuchimisa zvakachengeteka mukugadzira uye kugamuchira zvigadziriso neimwe nzira mune ramangwana?

MM: - Tinoikurudzira zvakasimba. Izvi zvinogadzirisa matambudziko mazhinji kwatiri.

MCH: - Zvakare, ndinoda kukwevera pfungwa kune chokwadi chekuti shanduko dzisina basa nekuvaka uye hadzinei nekuvhara kana mitsetse iri modular, iri mumamodule akasiyana. Kunyangwe nekuchinja kudiki iwe unogona kuzvichengeta zviri nyore.

MM: - Kana iwe uchifarira ruzivo, ipapo "Clickhouse" inoshandisa iyo inonzi nhoroondo raibhurari. Yakasunungurwa - ikopi yerutsigiro rweElastics, ndiko kuti, inogadziriswa. Polling inongochinja vavhoti. Tinotenda kuti izvi zvichashanda kwenguva yakareba.

A: - Kutenda kwazvo. Ndiudze, pane here zvinyorwa zvekuchinja kwakaitwa?

HighLoad++, Mikhail Makurov, Maxim Chernetsov (Intersvyaz): Zabbix, 100kNVPS pane imwe server

MM: - Zvinyorwa chigamba. Zviripachena, nekuunzwa kweClickhouse, nekuunzwa kwemhando nyowani dzevavhoti, sarudzo nyowani dzekugadzirisa dzinomuka. Iyo link kubva kune yekupedzisira siraidhi ine tsananguro pfupi yemashandisirwo ayo.

Nezve kutsiva fping nenmap

A: - Wakazoita sei izvi pakupedzisira? Unogona here kupa chaiyo mienzaniso: une strappers uye yekunze script? Chii chinoguma nekutarisa nhamba yakakura kudaro yevaenzi nekukurumidza? Unovaita sei vaenzi ava? Isu tinoda kuvadyisa kunmap neimwe nzira, kuvatora kubva kune imwe nzvimbo, kuvaisa mukati, kumhanya chimwe chinhu? ..

MM: - Kutonhora. Mubvunzo wakarurama! Nyaya ndeiyi. Isu takagadzirisa raibhurari (ICMP ping, chikamu cheZabbix) che ICMP cheki, iyo inoratidza nhamba yemapaketi - imwe (1), uye kodhi inoedza kushandisa nmap. Ndiko kuti, iyi ibasa remukati reZabbix, iro rava basa remukati repinger. Saizvozvo, hapana kuwiriranisa kana kushandiswa kwetrapper kunodiwa. Izvi zvakaitwa nemaune kuitira kuti usiye hurongwa hwakasimba uye usafanirwa kutarisana nekuenderana kwezvirongwa zviviri zve database: chii chekutarisa, kuisa kuburikidza nemuvhoti, uye kurodha kwedu kwakaputsika here?.. Izvi zviri nyore zvikuru.

A: -Inoshanda kune ma proxies zvakare?

MM: - Hongu, asi isu hatina kutarisa. Kodhi yekuvhota yakafanana mune zvese Zabbix uye sevha. Inofanira kushanda. Rega ndisimbise zvakare: mashandiro ehurongwa zvekuti hatidi proxy.

MCH: - Mhinduro chaiyo kumubvunzo ndeyekuti: "Nei uchida proxy ine system yakadai?" Chete nekuda kweNAT kana kutarisa kuburikidza neimwe mhando yeinononoka chiteshi ...

A: - Uye iwe unoshandisa Zabbix seallertor, kana ndichinzwisisa nemazvo. Kana kuti iva nemifananidzo yako (pane iyo archive layer) yakaendeswa kune imwe system, yakadai seGrafana? Kana kuti hausi kushandisa basa iri?

MM: - Ini ndichasimbisa zvakare: isu takawana kubatanidzwa kwakazara. Tiri kudurura nhoroondo muClickhouse, asi panguva imwe chete isu takashandura php kumberi. Iyo Php kumberi inoenda kuClickhouse uye inoita ese magiraidhi kubva ipapo. Panguva imwecheteyo, kuva vakatendeseka, isu tine chikamu chinovaka data mune mamwe graphic kuratidza masisitimu kubva kune imwechete Clickhouse, kubva kune imwechete Zabbix data.

MCH: - Mu "Grafan" zvakare.

Sarudzo dzakaitwa sei pamusoro pekugovewa kwezvinhu?

A: - Govera zvishoma zvekicheni yako yemukati. Sarudzo yakaitwa sei kuti zvaive zvakafanira kugovera zviwanikwa zvekugadzirisa zvakanyanya chigadzirwa? Izvi, kazhinji, dzimwe njodzi. Uye ndapota ndiudze, mumamiriro ezvinhu ekuti iwe uchatsigira shanduro itsva: sarudzo iyi inoruramisa sei kubva pakuona kwekutonga?

MM: - Sezviri pachena, isu hatina kuudza mutambo wenhoroondo zvakanaka. Takazviwana tiri mumamiriro ezvinhu apo chimwe chinhu chaifanira kuitwa, uye takaenda nezvikwata zviviri zvakafanana:

  • Imwe yaive yekutangisa hurongwa hwekutarisa uchishandisa nzira nyowani: kutarisa sevhisi, yakajairwa seti yeakavhurika sosi mhinduro dzatinosanganisa tobva taedza kushandura maitiro ebhizinesi kuitira kushanda neiyo nyowani yekutarisisa system.
  • Panguva imwe cheteyo, takava nemugadziri wepurogiramu ane chido aiita izvi (pamusoro pake). Zvakaitika kuti akakunda.

A: - Uye ndeipi saizi yechikwata?

MCH: - Ari pamberi pako.

A: - Saka, senguva dzose, iwe unoda chido?

MM: - Handizivi chinonzi chido.

A: - Muchiitiko ichi, sezviri pachena, iwe. Ndinokutendai zvikuru, munoshamisa.

MM: - Ndatenda.

Nezve zvigamba zveZabbix

A: - Kune sisitimu inoshandisa proxies (semuenzaniso, mune mamwe masisitimu akagoverwa), zvinokwanisika here kuchinjika uye chigamba, toti, vavhoti, proxies uye chikamu preprocessor yeZabbix pachayo; nekudyidzana kwavo? Zvinogoneka here kukwidziridza budiriro iripo kune system ine akawanda proxies?

MM: -Ndinoziva kuti sevha yeZabbix inounganidzwa uchishandisa proxy (iyo kodhi inounganidzwa uye inowanikwa). Hatina kuyedza izvi mukugadzira. Ini handina chokwadi nezve izvi, asi ndinofunga preprocessor maneja haashandiswe muproxy. Basa remumiriri nderekutora seti yemametrics kubva kuZabbix, kuisanganisa (inorekodhawo magadzirirwo, dhatabhesi yenzvimbo) uye woidzosera kuZabbix server. Iyo sevha pachayo inozoita preprocessing kana yaigamuchira.

Kufarira muproxies kunonzwisisika. Tichazviongorora. Iyi inyaya inofadza.

A: -Pfungwa yaive iyi: kana uchikwanisa kupeta vavhoti, unogona kuvapfekedza pane proxy uye kupeta kupindirana neseva, uye gadzirisa preprocessor yezvinangwa izvi chete paseva.

MM: - Ndinofunga zviri nyore. Iwe unotora iyo kodhi, shandisa chigamba, wozochigadzirisa nenzira yaunoda - unganidza proxy maseva (semuenzaniso, neODBC) uye govera iyo yakavharwa kodhi mumasisitimu ese. Pazvinenge zvakakodzera - unganidza proxy, pazvinenge zvakakodzera - sevha.

A: - Zvingangoitika, haufanire kubatanidza iyo proxy kutapurirana kune sevha zvakare?

MCH: - Kwete, zviri mureza.

MM: - Muchokwadi, imwe yemazano haina kunzwika. Isu tagara takachengetedza kuenzana pakati pekuputika kwemafungiro nehuwandu hwekuchinja uye nyore kwekutsigirwa.

Dzimwe ads 🙂

Ndinokutendai nekugara nesu. Unoda zvinyorwa zvedu here? Unoda kuona zvimwe zvinonakidza zvemukati? Titsigire nekuisa odha kana kukurudzira kushamwari, Cloud VPS yevagadziri kubva kumadhora 4.99, yakasarudzika analogue yekupinda-level maseva, iyo yakagadzirwa nesu kuti iwe: Chokwadi chese nezveVPS (KVM) E5-2697 v3 (6 Cores) 10GB DDR4 480GB SSD 1Gbps kubva pamadhora makumi maviri kana kugovera sevha? (inowanikwa neRAID1 uye RAID10, kusvika ku24 cores uye kusvika ku40GB DDR4).

Dell R730xd 2 nguva yakachipa muEquinix Tier IV data center muAmsterdam? Chete pano 2 x Intel TetraDeca-Core Xeon 2x E5-2697v3 2.6GHz 14C 64GB DDR4 4x960GB SSD 1Gbps 100 TV kubva $199 muNetherlands! Dell R420 - 2x E5-2430 2.2Ghz 6C 128GB DDR3 2x960GB SSD 1Gbps 100TB - kubva pamadhora makumi mapfumbamwe nemapfumbamwe! Verenga nezve Nzira yekuvaka Infrastructure Corp. kirasi nekushandiswa kweDell R730xd E5-2650 v4 maseva anokosha 9000 euros penny?

Source: www.habr.com

Voeg