Lub rooj sib tham HighLoad++ tom ntej no yuav muaj nyob rau lub Plaub Hlis 6 thiab 7, 2020 hauv St. Petersburg Cov Lus Qhia thiab daim pib . HighLoad++ Moscow 2018. Hall "Moscow". Peb 9, 15:00. Cov no thiab .
* Saib xyuas - online thiab tshuaj xyuas.
* Cov kev txwv tseem ceeb ntawm ZABBIX platform.
* Kev daws teeb meem rau scaling analytics cia.
* Kev ua kom zoo ntawm ZABBIX server.
* UI optimization.
* Kev paub txog kev ua haujlwm ntawm lub kaw lus hauv qab ntawm ntau dua 40k NVPS.
* Cov lus xaus.
Mikhail Makurov (Ntawm no - MM): - Nyob zoo txhua tus!
Maxim Chernetsov (ntawm no - MCH): - Nyob zoo tav su!
MM: - Cia kuv qhia Maxim. Max yog tus kws tshaj lij engineers, tus tshaj plaws networker kuv paub. Maxim koom nrog hauv tes hauj lwm thiab kev pabcuam, lawv txoj kev loj hlob thiab kev ua haujlwm.
MCH: - Thiab kuv xav qhia koj txog Mikhail. Mikhail yog tus tsim tawm C. Nws tau sau ob peb lub tsheb thauj khoom siab ua haujlwm rau peb lub tuam txhab. Peb nyob thiab ua hauj lwm nyob rau hauv lub Urals, nyob rau hauv lub nroog ntawm cov txiv neej tawv Chelyabinsk, nyob rau hauv lub tuam txhab Intersvyaz. Peb lub tuam txhab yog ib qho chaw muab kev pabcuam hauv Internet thiab cable TV rau ib lab tus tib neeg hauv 16 lub nroog.
MM: - Thiab nws tsim nyog hais tias Intersvyaz yog ntau tshaj li qhov chaw muab kev pabcuam, nws yog lub tuam txhab IT. Feem ntau ntawm peb cov kev daws teeb meem yog tsim los ntawm peb lub tuam txhab IT.
A: los ntawm cov servers ua cov tsheb khiav mus rau qhov chaw hu xov tooj thiab daim ntawv thov mobile. Lub tuam tsev IT tam sim no muaj txog 80 tus neeg uas muaj peev xwm sib txawv heev.
Hais txog Zabbix thiab nws cov architecture
MCH: -Thiab tam sim no kuv yuav sim teeb tsa tus kheej cov ntaub ntawv thiab hais hauv ib feeb seb Zabbix yog dab tsi (tom qab no hu ua "Zabbix").
Zabbix tso nws tus kheej ua lag luam-theem tawm-ntawm-lub-box saib xyuas qhov system. Nws muaj ntau yam nta uas ua rau lub neej yooj yim dua: cov kev cai nce qib siab, API rau kev sib koom ua ke, pab pawg thiab nws pib-nrhiav cov tswv thiab metrics. Zabbix muaj lub npe hu ua scaling tools - proxies. Zabbix yog qhov qhib qhov system.
Luv luv txog architecture. Peb tuaj yeem hais tias nws muaj peb yam:
- Server. Sau hauv C. Nrog es complex ua thiab hloov cov ntaub ntawv ntawm threads. Tag nrho cov txheej txheem yuav siv sij hawm qhov chaw nyob rau hauv nws: los ntawm tau txais mus rau txuag rau lub database.
- Tag nrho cov ntaub ntawv yog khaws cia rau hauv lub database. Zabbix txhawb MySQL, PostreSQL thiab Oracle.
- Lub web interface yog sau hauv PHP. Ntawm cov tshuab feem ntau nws los nrog Apache server, tab sis ua haujlwm tau zoo dua nrog nginx + php.
Niaj hnub no peb xav qhia ib zaj dab neeg ntawm lub neej ntawm peb lub tuam txhab ntsig txog Zabbix ...
Ib zaj dab neeg los ntawm lub neej ntawm lub tuam txhab Intersvyaz. Peb muaj dab tsi thiab peb xav tau dab tsi?
5 los yog 6 lub hlis dhau los. Ib hnub tom qab ua haujlwm ...
MCH: - Misha, nyob zoo! Kuv zoo siab tias kuv tau ntes koj - muaj kev sib tham. Peb rov muaj teeb meem nrog kev saib xyuas. Thaum muaj xwm txheej loj, txhua yam tau qeeb thiab tsis muaj cov ntaub ntawv hais txog lub xeev ntawm lub network. Hmoov tsis zoo, qhov no tsis yog thawj zaug qhov no tau tshwm sim. Kuv xav tau koj kev pab. Cia peb ua haujlwm saib xyuas hauv txhua qhov xwm txheej!
MM: - Tab sis cia peb synchronize ua ntej. Kuv tsis tau saib muaj ob peb xyoos. Raws li kuv nco qab, peb tau tso tseg Nagios thiab hloov mus rau Zabbix txog 8 xyoo dhau los. Thiab tam sim no peb zoo li muaj 6 cov servers muaj zog thiab txog kaum ob tus neeg sawv cev. Kuv puas paub dab tsi?
MCH: - Yuav luag. 15 servers, qee qhov yog cov tshuab virtual. Qhov tseem ceeb tshaj plaws yog tias nws tsis txuag peb lub sijhawm thaum peb xav tau nws tshaj plaws. Zoo li kev sib tsoo - cov servers qeeb thiab koj tsis tuaj yeem pom dab tsi. Peb tau sim optimize lub configuration, tab sis qhov no tsis muab qhov zoo tshaj plaws kev ua tau zoo nce.
MM: - Nws yog qhov tseeb. Koj puas tau saib ib yam dab tsi, koj puas tau khawb ib yam dab tsi los ntawm kev kuaj mob?
MCH: - Thawj qhov uas koj yuav tsum tau nrog yog lub database. MySQL tau thauj khoom tas li, khaws cov kev ntsuas tshiab, thiab thaum Zabbix pib tsim ib pawg ntawm cov xwm txheej, cov ntaub ntawv nkag mus rau hauv overdrive rau ob peb teev. Kuv twb tau hais rau koj txog kev ua kom zoo rau kev teeb tsa, tab sis qhov tseeb xyoo no lawv hloov kho kho vajtse: cov servers muaj ntau tshaj li ib puas gigabytes ntawm lub cim xeeb thiab disk arrays ntawm SSD RAIDs - tsis muaj qhov taw tes rau kev loj hlob nws linearly nyob rau hauv lub sij hawm ntev. Peb ua li cas?
MM: - Nws yog qhov tseeb. Feem ntau, MySQL yog LTP database. Thaj, nws tsis tsim nyog rau khaws cia ib qho archive ntawm metrics ntawm peb qhov loj. Cia peb xav txog nws.
MCH: - Cia peb!
Kev koom ua ke ntawm Zabbix thiab Clickhouse raws li qhov tshwm sim ntawm hackathon
Tom qab qee lub sijhawm peb tau txais cov ntaub ntawv nthuav dav:
Feem ntau ntawm qhov chaw hauv peb cov ntaub ntawv tau nyob los ntawm metrics archive thiab tsawg dua 1% yog siv rau configuration, templates thiab chaw. Los ntawm lub sijhawm ntawd, peb tau ua haujlwm cov ntaub ntawv loj raws li Clickhouse rau ntau tshaj ib xyoos. Cov kev taw qhia ntawm kev txav yog pom tseeb rau peb. Ntawm peb lub caij nplooj ntoos hlav Hackathon, kuv tau sau qhov kev koom ua ke ntawm Zabbix nrog Clickhouse rau lub server thiab frontend. Lub sijhawm ntawd, Zabbix twb muaj kev txhawb nqa rau ElasticSearch, thiab peb tau txiav txim siab los sib piv lawv.
Kev sib piv ntawm Clickhouse thiab Elasticsearch
MM: - Rau kev sib piv, peb tsim tib lub thauj khoom raws li Zabbix server muab thiab saib seb cov tshuab yuav coj li cas. Peb tau sau cov ntaub ntawv hauv batch ntawm 1000 kab, siv CURL. Peb xav ua ntej tias Clickhouse yuav ua tau zoo dua rau cov ntaub ntawv thauj khoom uas Zabbix ua. Cov txiaj ntsig tau tshaj peb qhov kev cia siab:
Raws li tib yam kev sim, Clickhouse tau sau peb zaug ntxiv cov ntaub ntawv. Tib lub sijhawm, ob lub tshuab siv tau zoo heev (me me ntawm cov peev txheej) thaum nyeem cov ntaub ntawv. Tab sis Elastics xav tau ntau tus processor thaum kaw:
Nyob rau hauv tag nrho, Clickhouse yog ho superior rau Elastix nyob rau hauv cov nqe lus ntawm processor noj thiab ceev. Nyob rau tib lub sijhawm, vim cov ntaub ntawv compression, Clickhouse siv 11 zaug tsawg dua ntawm lub hard drive thiab ua haujlwm kwv yees li 30 npaug tsawg dua disk:
MCH: - Yog lawm, Clickhouse txoj haujlwm nrog lub disk subsystem yog siv tau zoo heev. Koj tuaj yeem siv cov SATA loj loj rau cov ntaub ntawv thiab tau txais kev sau nrawm ntawm ntau pua txhiab kab hauv ib ob. Cov txheej txheem tawm ntawm lub thawv txhawb nqa sharding, replication, thiab yooj yim heev rau configure. Peb txaus siab rau nws siv thoob plaws hauv lub xyoo.
Txhawm rau txhim kho cov peev txheej, koj tuaj yeem nruab Clickhouse ib sab ntawm koj cov ntaub ntawv tseem ceeb uas twb muaj lawm thiab yog li txuag tau ntau lub sijhawm CPU thiab kev ua haujlwm disk. Peb tau hloov cov ntaub ntawv khaws tseg ntawm kev ntsuas mus rau pawg Clickhouse uas twb muaj lawm:
Peb tau tshem lub ntsiab MySQL database ntau heev uas peb tuaj yeem muab tso rau ntawm ib lub tshuab nrog Zabbix server thiab tso tseg cov neeg rau zaub mov rau MySQL.
Kev xaiv tsa ua haujlwm li cas hauv Zabbix?
4 lub hlis dhau los
MM: - Zoo, peb tuaj yeem tsis nco qab txog cov teeb meem nrog lub hauv paus?
MCH: - Qhov ntawd yog qhov tseeb! Lwm qhov teeb meem peb yuav tsum tau daws yog qeeb cov ntaub ntawv sau. Tam sim no tag nrho peb 15 lub npe servers dhau los nrog SNMP thiab cov txheej txheem xaiv tsa. Thiab tsis muaj txoj hauv kev tshwj tsis yog rau nruab cov servers tshiab thiab tshiab.
MM: - Zoo heev. Tab sis ua ntej, qhia peb tias kev xaiv tsa ua haujlwm hauv Zabbix li cas?
MCH: - Hauv luv luv, muaj 20 hom kev ntsuas thiab ntau txoj hauv kev kom tau txais lawv. Zabbix tuaj yeem sau cov ntaub ntawv nyob rau hauv "thov- teb" hom, lossis tos cov ntaub ntawv tshiab los ntawm "Trapper Interface".
Nws yog tsim nyog sau cia tias nyob rau hauv thawj Zabbix hom no (Trapper) yog qhov ceev tshaj plaws.
Muaj cov npe servers rau kev xa khoom xa tawm:
Proxies tuaj yeem ua tau tib yam kev sau ua haujlwm raws li Zabbix server, tau txais cov haujlwm los ntawm nws thiab xa cov metrics sau los ntawm Trapper interface. Nov yog txoj hauv kev pom zoo los faib cov khoom thauj. Proxies kuj tseem muaj txiaj ntsig zoo rau kev saib xyuas cov chaw taws teeb tswj kev ua haujlwm ntawm NAT lossis qeeb channel:
MM: - Txhua yam yog meej nrog architecture. Peb yuav tsum saib cov peev txheej ...
Ob peb hnub tom qab
Zaj dab neeg ntawm yuav ua li cas nmap fping yeej
MM: "Kuv xav tias kuv khawb ib yam dab tsi."
MCH: - Qhia kuv!
MM: - Kuv pom tias thaum kuaj xyuas qhov muaj, Zabbix txheeb xyuas qhov siab tshaj plaws ntawm 128 tus tswv ntawm ib lub sijhawm. Kuv sim nce tus lej no mus rau 500 thiab tshem tawm cov pob ntawv sib cuam tshuam hauv lawv cov ping (ping) - qhov no ua rau qhov ua tau zoo dua ob npaug. Tab sis kuv xav tau cov lej loj dua.
MCH: - Hauv kuv qhov kev coj ua, qee zaum kuv yuav tsum kuaj xyuas ntau txhiab tus tswv, thiab kuv tsis tau pom dab tsi sai dua nmap rau qhov no. Kuv paub tseeb tias qhov no yog txoj kev ceev tshaj plaws. Cia peb sim! Peb yuav tsum tau nce tus naj npawb ntawm hosts ib iteration.
MM: - Tshawb xyuas ntau tshaj tsib puas? 600?
MCH: - Tsawg kawg yog ob peb txhiab.
MM: - OK. Qhov tseem ceeb tshaj plaws uas kuv xav hais yog tias kuv pom tias feem ntau kev xaiv tsa hauv Zabbix yog ua tiav synchronously. Peb yeej yuav tsum tau hloov nws mus rau asynchronous hom. Tom qab ntawd peb tuaj yeem nce tus naj npawb ntawm cov metrics sau los ntawm cov neeg pov npav, tshwj xeeb tshaj yog tias peb nce tus naj npawb ntawm cov metrics ib iteration.
MCH: - Zoo heev! Thiab thaum twg?
MM: - Raws li niaj zaus, nag hmo.
MCH: - Peb piv ob qho tib si ntawm fping thiab nmap:
Ntawm ntau tus tswv, nmap tau xav tias yuav ua tau zoo dua li tsib zaug. Txij li thaum nmap tsuas yog kuaj xyuas qhov muaj thiab lub sijhawm teb, peb tau txav qhov kev suav ntawm kev poob mus rau qhov tshwm sim thiab txo qis qhov muaj nyob rau lub sijhawm. Peb pom tus naj npawb zoo ntawm cov tswv rau nmap kom nyob ib ncig ntawm 4 txhiab ib iteration. Nmap tau tso cai rau peb txo tus nqi CPU ntawm kev kuaj xyuas muaj peb zaug thiab txo lub sijhawm ntawm 120 vib nas this mus rau 10.
Kev xaiv tsa optimization
MM: “Tom qab ntawd peb pib ua cov neeg xaiv tsa. Peb feem ntau txaus siab rau SNMP tshawb pom thiab cov neeg sawv cev. Hauv Zabbix, kev xaiv tsa tau ua tiav synchronously thiab cov kev ntsuas tshwj xeeb tau raug coj los ua kom muaj txiaj ntsig ntawm lub kaw lus. Hauv hom synchronous, tus tswv tsev tsis muaj nyob ua rau muaj kev cuam tshuam loj rau kev xaiv tsa. Muaj tag nrho cov txheej txheem ntawm lub xeev, muaj cov txheej txheem tshwj xeeb - lub npe hu ua cov neeg pov npav tsis tuaj yeem, uas ua haujlwm tsuas yog nrog cov tswv tsis tuaj yeem:
Qhov no yog ib qho kev tawm tswv yim uas qhia tau hais tias lub xeev matrix, tag nrho cov complexity ntawm cov txheej txheem ntawm kev hloov uas yuav tsum tau nyob rau hauv thiaj li yuav nyob twj ywm zoo. Tsis tas li ntawd, synchronous polling nws tus kheej yog qeeb heev:
Tias yog vim li cas ntau txhiab tus neeg xaiv tsa kwj ntawm ntau tus neeg sawv cev tsis tuaj yeem sau cov ntaub ntawv xav tau rau peb. Kev siv asynchronous daws tsis tau tsuas yog cov teeb meem nrog cov xov tooj xwb, tab sis kuj tseem ua kom yooj yim rau lub xeev system ntawm cov tswv tsis muaj, vim tias rau ib tus lej txheeb xyuas hauv ib qho kev xaiv tsa, qhov siab tshaj plaws tos lub sijhawm yog 1 lub sijhawm:
Tsis tas li ntawd, peb tau hloov kho thiab txhim kho qhov kev xaiv tsa rau SNMP thov. Qhov tseeb yog tias feem ntau cov neeg tsis tuaj yeem teb rau ntau qhov kev thov SNMP tib lub sijhawm. Yog li ntawd, peb tau ua hom hybrid, thaum SNMP kev xaiv tsa ntawm tib tus tswv tsev tau ua tiav asynchronously:
Qhov no yog ua tiav rau tag nrho pob ntawm cov tswv. Hom no yog qhov kawg tsis qeeb tshaj li ib qho asynchronous kiag li, txij li kev xaiv tsa ib thiab ib nrab puas SNMP qhov tseem ceeb tseem ceev dua 1 lub sijhawm.
Peb qhov kev sim tau pom tias qhov zoo tshaj plaws ntawm kev thov hauv ib qho iteration yog kwv yees li 8 txhiab nrog SNMP kev xaiv tsa. Nyob rau hauv tag nrho, kev hloov mus rau asynchronous hom tau tso cai rau peb kom ceev cov kev xaiv tsa los ntawm 200 zaug, ntau pua zaus.
MCH: - Cov txiaj ntsig kev xaiv tsa tau pom tau tias peb tuaj yeem tsis tsuas yog tshem tawm tag nrho cov neeg sawv cev, tab sis kuj txo cov ntu rau ntau qhov kev kuaj xyuas, thiab cov neeg sawv cev yuav tsis xav tau ntxiv lawm raws li txoj hauv kev los faib cov khoom thauj.
Txog peb lub hlis dhau los
Hloov cov architecture - nce lub load!
MM: - Zoo, Max, nws puas yog lub sijhawm kom tau txais txiaj ntsig? Kuv xav tau lub server muaj zog thiab tus engineer zoo.
MCH: - Okay, cia peb npaj nws. Nws yog lub sijhawm siab kom txav los ntawm qhov chaw tuag ntawm 5 txhiab metrics ib ob.
Sawv ntxov tom qab kev txhim kho
MCH: - Misha, peb hloov kho peb tus kheej, tab sis thaum sawv ntxov peb dov rov qab ... Kwv yees li cas ceev peb tswj kom ua tiav?
MM: - 20 txhiab qhov siab tshaj plaws.
MCH: - Yog, 25! Hmoov tsis zoo, peb nyob qhov twg peb pib.
MM: - Vim li cas? Koj puas tau khiav ib qho kev kuaj mob?
MCH: - Yog, tseeb! Ntawm no, piv txwv li, yog ib qho nthuav saum toj kawg nkaus:
MM: - Cia peb saib. Kuv pom tias peb tau sim ntau qhov kev xaiv tsa xov tooj:
Tab sis tib lub sij hawm lawv tsis tuaj yeem rov ua dua lub system txawm tias ib nrab:
Thiab tag nrho kev ua tau zoo yog me me, txog 4 txhiab metrics ib ob:
Puas muaj lwm yam?
MCH: - Yog, kab ntawm ib qho ntawm cov neeg xaiv tsa:
MM: - Ntawm no koj tuaj yeem pom meej tias cov txheej txheem xaiv tsa tab tom tos "semaphores". Cov no yog cov locks:
MCH: - Tsis paub meej.
MM: - Saib, qhov no zoo ib yam li qhov xwm txheej uas ib pawg ntawm cov xov tau sim ua haujlwm nrog cov peev txheej uas tsuas yog ib tus tuaj yeem ua haujlwm nrog ib lub sijhawm. Tom qab ntawd txhua yam lawv tuaj yeem ua tau yog qhia cov peev txheej hauv lub sijhawm:
Thiab tag nrho cov kev ua tau zoo ntawm kev ua haujlwm nrog xws li cov peev txheej yog txwv los ntawm kev ceev ntawm ib qho tseem ceeb:
Muaj ob txoj hauv kev los daws qhov teeb meem no.
Txhim kho qhov kho vajtse ntawm lub tshuab, hloov mus rau cov cores sai dua:
Los yog hloov lub architecture thiab tib lub sij hawm hloov lub load:
MCH: - Los ntawm txoj kev, ntawm lub tshuab kuaj peb yuav siv cov cores tsawg dua li ntawm kev sib ntaus sib tua, tab sis lawv yog 1,5 npaug sai dua nyob rau hauv ntau zaus!
MM: - Ntshiab? Koj yuav tsum tau saib ntawm server code.
Cov ntaub ntawv kab hauv Zabbix server
MCH: - Txhawm rau txheeb xyuas nws, peb pib txheeb xyuas seb cov ntaub ntawv pauv mus li cas hauv Zabbix server:
Daim duab txias, txoj cai? Cia peb mus dhau nws ib kauj ruam los ua kom pom tseeb ntau dua lossis tsawg dua. Muaj cov xov thiab cov kev pab cuam lub luag haujlwm rau kev sau cov ntaub ntawv:
Lawv xa cov metrics uas tau sau los ntawm lub qhov (socket) mus rau tus thawj tswj hwm Preprocessor, uas lawv tau txais kev cawmdim hauv ib kab:
Tus "preprocessor manager" xa cov ntaub ntawv mus rau nws cov neeg ua haujlwm, uas ua tiav cov lus qhia ua ntej thiab xa rov qab los ntawm tib lub qhov (socket):
Tom qab ntawd, tus thawj tswj hwm preprocessor khaws cia rau hauv keeb kwm cache:
Los ntawm qhov ntawd lawv raug coj los ntawm keeb kwm sinkers, uas ua haujlwm ntau heev: piv txwv li, suav cov txiaj ntsig, sau cov nqi cache thiab, qhov tseem ceeb tshaj, txuag cov ntsuas hauv keeb kwm cia. Feem ntau, cov txheej txheem yog complex thiab tsis meej pem heev.
MM: - Thawj qhov uas peb pom yog tias feem ntau cov xov sib tw rau qhov hu ua "configuration cache" (qhov chaw nco uas txhua tus neeg rau zaub mov configurations khaws cia). Cov xov muaj lub luag haujlwm rau kev sau cov ntaub ntawv tshwj xeeb tshaj yog ntau qhov thaiv:
... txij li thaum lub khw muag khoom configuration tsis tsuas yog metrics nrog lawv cov tsis, tab sis kuj queues los ntawm cov neeg xaiv tsa coj cov ntaub ntawv hais txog yuav ua li cas ntxiv. Thaum muaj ntau tus neeg xaiv tsa thiab ib qho thaiv qhov kev teeb tsa, lwm tus tos rau kev thov:
Cov neeg xaiv tsa yuav tsum tsis txhob sib cav
Yog li ntawd, thawj qhov peb tau ua yog faib cov kab hauv 4 qhov chaw thiab tso cai rau cov neeg xaiv tsa los thaiv cov kab no, cov khoom no tib lub sijhawm, nyob rau hauv kev nyab xeeb:
Qhov no tshem tawm kev sib tw rau lub configuration cache, thiab qhov ceev ntawm pollers tau nce ntau. Tab sis tom qab ntawd peb tau ntsib qhov tseeb tias tus thawj tswj hwm ua ntej pib sau ib kab ntawm txoj haujlwm:
Preprocessor tus thawj coj yuav tsum muaj peev xwm ua ntej
Qhov no tshwm sim thaum nws tsis muaj kev ua haujlwm. Tom qab ntawd txhua yam nws tuaj yeem ua tau yog sau cov lus thov los ntawm cov txheej txheem sau cov ntaub ntawv thiab ntxiv lawv qhov tsis tau txog thaum nws noj tag nrho cov cim xeeb thiab poob:
Txhawm rau daws qhov teeb meem no, peb ntxiv lub qhov (socket) thib ob uas tau mob siab rau cov neeg ua haujlwm tshwj xeeb:
Yog li, tus thawj tswj hwm preprocessor muaj lub sijhawm los saib xyuas qhov tseem ceeb ntawm nws txoj haujlwm thiab, yog tias qhov tsis tuaj yeem loj hlob, txoj haujlwm yog kom txo qis kev tshem tawm, muab sijhawm rau cov neeg ua haujlwm los ua qhov tsis zoo no:
Tom qab ntawd peb pom tias ib qho ntawm cov laj thawj rau kev qeeb qeeb yog cov neeg ua haujlwm lawv tus kheej, vim lawv tau sib tw rau cov peev txheej uas tsis tseem ceeb rau lawv txoj haujlwm. Peb tau sau cov teeb meem no raws li kab laum-kho, thiab nws twb tau raug daws nyob rau hauv cov tshiab versions ntawm Zabbix:
Peb nce tus naj npawb ntawm qhov (socket) - peb tau txais qhov tshwm sim
Tsis tas li ntawd, tus thawj tswj hwm tus thawj tswj hwm nws tus kheej tau dhau los ua lub cev tsis muaj zog, vim nws yog ib qho xov. Nws so ntawm qhov ceev ceev, muab qhov siab tshaj plaws ntawm 70 txhiab metrics ib ob:
Yog li ntawd, peb ua plaub, nrog plaub lub qhov (sockets), cov neeg ua haujlwm:
Thiab qhov no tau tso cai rau peb nce qhov nrawm mus rau kwv yees li 130 txhiab metrics:
Qhov tsis yog-linearity ntawm kev loj hlob yog piav qhia los ntawm qhov tseeb tias kev sib tw rau keeb kwm cache tau tshwm sim. 4 preprocessor managers thiab keeb kwm sinkers sib tw rau nws. Lub sijhawm no, peb tau txais kwv yees li 130 txhiab metrics ib ob ntawm lub tshuab kuaj, siv nws li ntawm 95% ntawm cov processor:
Kwv yees li 2,5 lub hlis dhau los
Kev tsis lees paub los ntawm snmp-zej zog nce NVPs los ntawm ib thiab ib nrab zaug
MM: - Max, Kuv xav tau lub tsheb sim tshiab! Peb tsis haum rau qhov tam sim no.
MCH: - Koj muaj dab tsi tam sim no?
MM: - Tam sim no - 130k NVPs thiab lub txee-npaj processor.
MCH: - Wow! Txias! Tos kuv muaj ob lo lus nug. Raws li kuv cov kev suav, peb xav tau yog nyob ib ncig ntawm 15-20 txhiab metrics ib ob. Vim li cas peb thiaj xav tau ntau dua?
MM: "Kuv xav ua kom tiav txoj haujlwm." Kuv xav pom ntau npaum li cas peb tuaj yeem nyem tawm ntawm qhov system no.
MCH: -Tab sis…
MM: "Tab sis nws tsis muaj txiaj ntsig rau kev lag luam."
MCH: - Nws yog qhov tseeb. Thiab lo lus nug thib ob: peb puas tuaj yeem txhawb nqa yam peb muaj tam sim no ntawm peb tus kheej, yam tsis muaj kev pab los ntawm tus tsim tawm?
MM: - Kuv tsis xav. Hloov li cas lub configuration cache ua haujlwm yog ib qho teeb meem. Nws cuam tshuam cov kev hloov hauv feem ntau cov xov thiab yog qhov nyuaj heev los tswj. Feem ntau, nws yuav nyuaj heev los tswj nws.
MCH: "Tom qab ntawd peb xav tau qee yam kev xaiv."
MM: - Muaj qhov kev xaiv zoo li no. Peb tuaj yeem hloov mus rau cov cores ceev, thaum tso qhov kev kaw haujlwm tshiab. Peb tseem yuav tau txais kev ua haujlwm ntawm 60-80 txhiab metrics. Nyob rau tib lub sijhawm, peb tuaj yeem tso tag nrho cov lej seem. Clickhouse thiab asynchronous pov npav yuav ua haujlwm. Thiab nws yuav yooj yim los tswj.
MCH: - Zoo kawg! Kuv xav kom peb nres ntawm no.
Tom qab optimizing tus neeg rau zaub mov sab, peb thaum kawg muaj peev xwm tshaj tawm cov cai tshiab rau hauv ntau lawm. Peb tau tso tseg qee qhov kev hloov pauv hauv kev pom zoo ntawm kev hloov mus rau lub tshuab nrog ceev cores thiab txo qis cov lej hloov pauv. Peb kuj tau ua kom yooj yim rau kev teeb tsa thiab tshem tawm cov macros hauv cov ntaub ntawv cov khoom yog qhov ua tau, raws li lawv qhia txog kev kaw lus ntxiv.
Piv txwv li, tso tseg snmp-community macro, uas feem ntau pom nyob rau hauv cov ntaub ntawv thiab cov piv txwv, nyob rau hauv peb cov ntaub ntawv ua rau nws muaj peev xwm mus ntxiv ceev NVPs los ntawm txog 1,5 zaug.
Tom qab ob hnub hauv kev tsim khoom
Tshem tawm cov xwm txheej keeb kwm pop-ups
MCH: - Misha, peb tau siv lub kaw lus tau ob hnub, thiab txhua yam ua haujlwm. Tab sis tsuas yog thaum txhua yam ua haujlwm! Peb tau npaj ua haujlwm nrog kev hloov pauv ntawm ntu loj ntawm lub network, thiab peb rov kuaj xyuas nrog peb txhais tes dab tsi tau nce thiab dab tsi tsis tau.
MM: - Ua tsis tau! Peb kuaj txhua yam 10 zaug. Tus neeg rau zaub mov ua haujlwm txawm tias ua tiav network tsis muaj tam sim.
MCH: - Yog, kuv nkag siab txhua yam: server, database, sab saum toj, austat, cav - txhua yam yog nrawm ... Tab sis peb saib ntawm lub vev xaib interface, thiab muaj cov txheej txheem "hauv txee" ntawm lub server thiab qhov no:
MM: - Nws yog qhov tseeb. Wb saib lub vev xaib. Peb pom tias nyob rau hauv ib qho xwm txheej uas muaj ntau qhov xwm txheej tshwm sim, feem ntau cov widgets nyob tau pib ua haujlwm qeeb heev:
Yog vim li cas rau qhov no yog tiam ntawm qhov xwm txheej keeb kwm pop-ups uas yog tsim rau txhua yam khoom hauv daim ntawv teev npe. Yog li ntawd, peb tso tseg tiam ntawm cov qhov rais no (hais tawm 5 kab hauv cov cai), thiab qhov no daws peb cov teeb meem.
Lub sijhawm thauj khoom rau widgets, txawm tias thaum tsis muaj tag nrho, tau raug txo los ntawm ob peb feeb mus rau qhov tau txais 10-15 vib nas this rau peb, thiab cov keeb kwm tseem tuaj yeem pom los ntawm txhaj rau lub sijhawm:
Tom qab ua haujlwm. 2 lub hlis dhau los
MCH: - Misha, koj tawm mus? Peb yuav tsum tham.
MM: - Kuv tsis tau npaj siab yuav. Ib yam dab tsi nrog Zabbix dua?
MCH: - Tsis yog, so! Kuv tsuas yog xav hais tias: txhua yam ua haujlwm, ua tsaug! Kuv tau npias.
Zabbix muaj txiaj ntsig
Zabbix yog ib tug ncaj ncees universal thiab nplua nuj system thiab muaj nuj nqi. Nws tuaj yeem siv rau kev teeb tsa me me tawm ntawm lub thawv, tab sis raws li kev xav tau loj hlob, nws yuav tsum tau kho kom zoo. Txhawm rau khaws cov ntaub ntawv loj ntawm cov ntsuas, siv qhov chaw cia kom haum:
- koj tuaj yeem siv cov cuab yeej built-in hauv daim ntawv ntawm kev koom ua ke nrog Elasticsearch lossis upload keeb kwm rau cov ntawv nyeem (muaj los ntawm version XNUMX);
- Koj tuaj yeem coj kom zoo dua ntawm peb cov kev paub dhau los thiab kev koom ua ke nrog Clickhouse.
Txhawm rau ua kom nrawm nrawm ntawm kev sau cov ntsuas, sau lawv siv txoj hauv kev asynchronous thiab xa lawv los ntawm kev cuam tshuam rau tus neeg rau zaub mov Zabbix; lossis koj tuaj yeem siv thaj ua rau Zabbix pollers asynchronous.
Zabbix tau sau hauv C thiab ua tau zoo heev. Kev daws teeb meem ntau qhov kev tsim vaj tsev tso cai rau koj ntxiv nws qhov kev ua tau zoo thiab, hauv peb qhov kev paub, tau txais ntau tshaj 100 txhiab metrics ntawm ib lub tshuab ua haujlwm ib leeg.
Tib Zabbix thaj
MM: - Kuv xav ntxiv ob peb lub ntsiab lus. Tag nrho cov ntawv tshaj tawm tam sim no, txhua qhov kev xeem, cov lej raug muab rau kev teeb tsa uas peb siv. Tam sim no peb tab tom noj kwv yees li 20 txhiab metrics ib ob los ntawm nws. Yog tias koj tab tom sim nkag siab seb qhov no puas yuav ua haujlwm rau koj, koj tuaj yeem sib piv. Dab tsi tau tham hnub no yog tshaj tawm ntawm GitHub hauv daim ntawv ntawm thaj:
Patch muaj xws li:
- tag nrho kev koom ua ke nrog Clickhouse (ob leeg Zabbix server thiab frontend);
- daws teeb meem nrog tus thawj coj ua ntej;
- asynchronous kev xaiv tsa.
Lub thaj yog sib xws nrog tag nrho cov version 4, suav nrog lts. Feem ntau yuav, nrog kev hloov me me nws yuav ua haujlwm ntawm version 3.4.
Ua tsaug koj rau koj mloog.
Lus nug
Cov lus nug los ntawm cov neeg tuaj saib (tom qab no – A): – Nyob zoo tav su! Thov qhia rau kuv, koj puas muaj kev npaj rau kev sib cuam tshuam nrog pab pawg Zabbix lossis nrog lawv nrog koj, yog li ntawd qhov no tsis yog thaj, tab sis tus cwj pwm ntawm Zabbix?
MM: - Yog, peb yuav twv yuav raug hu ua qee qhov kev hloov pauv. Ib yam dab tsi yuav tshwm sim, ib yam dab tsi yuav nyob twj ywm hauv thaj.
A: - Ua tsaug ntau rau daim ntawv qhia zoo heev! Thov qhia rau kuv, tom qab siv thaj ua rau thaj, kev txhawb nqa los ntawm Zabbix yuav nyob twj ywm thiab yuav ua li cas txuas ntxiv hloov kho mus rau qib siab dua? Nws puas tuaj yeem hloov kho Zabbix tom qab koj thaj rau 4.2, 5.0?
MM: - Kuv tsis tuaj yeem hais dab tsi txog kev txhawb nqa. Yog tias kuv yog Zabbix kev txhawb nqa, Kuv yuav zaum hais tias tsis yog, vim qhov no yog lwm tus neeg txoj cai. Raws li rau 4.2 codebase, peb txoj haujlwm yog: "Peb yuav txav mus nrog lub sijhawm, thiab peb yuav hloov kho peb tus kheej ntawm cov ntawv tom ntej." Yog li ntawd, rau qee lub sijhawm peb yuav tshaj tawm ib thaj rau cov hloov tshiab. Kuv twb tau hais nyob rau hauv tsab ntawv ceeb toom: tus naj npawb ntawm cov kev hloov pauv nrog versions tseem tsawg heev. Kuv xav tias qhov kev hloov ntawm 3.4 mus rau 4 tau coj peb mus txog 15 feeb. Ib yam dab tsi hloov nyob ntawd, tab sis tsis tseem ceeb heev.
A: - Yog li koj npaj los txhawb koj thaj thiab koj tuaj yeem txhim kho nws hauv kev tsim khoom thiab tau txais kev hloov tshiab hauv qee txoj hauv kev yav tom ntej?
MM: - Peb xav kom nws. Qhov no daws teeb meem ntau rau peb.
MCH: - Ib zaug ntxiv, kuv xav ua tib zoo mloog rau qhov tseeb tias cov kev hloov pauv uas tsis cuam tshuam txog kev tsim vaj tsev thiab tsis muaj kev txhawj xeeb txog kev thaiv lossis cov kab ke yog modular, lawv nyob hauv cov modules cais. Txawm tias muaj kev hloov me me koj tuaj yeem tswj tau lawv yooj yim heev.
MM: - Yog tias koj txaus siab rau cov ntsiab lus, ces "Clickhouse" siv lub npe hu ua tsev qiv ntawv keeb kwm. Nws yog untied - nws yog ib daim qauv ntawm Elastics kev txhawb nqa, uas yog, nws yog configurable. Kev xaiv tsa tsuas yog hloov cov neeg xaiv tsa. Peb ntseeg tias qhov no yuav ua haujlwm ntev.
A: - Ua tsaug ntau. Qhia rau kuv, puas muaj cov ntaub ntawv ntawm kev hloov pauv?
MM: - Cov ntaub ntawv yog ib thaj. Pom tseeb, nrog kev taw qhia ntawm Clickhouse, nrog rau kev qhia txog hom tshiab ntawm cov neeg xaiv tsa, cov kev xaiv teeb tsa tshiab tshwm sim. Qhov txuas los ntawm tus swb kawg muaj cov lus piav qhia luv luv ntawm kev siv nws li cas.
Txog kev hloov fping nrog nmap
A: – Koj ua li cas thaum kawg ua qhov no? Koj tuaj yeem muab cov piv txwv tshwj xeeb: koj puas muaj cov hlua khi thiab cov ntawv sau sab nraud? Yuav ua li cas mus xyuas xws li ib tug loj tus naj npawb ntawm hosts sai sai? Koj ua li cas kuv cov tswv tsev no? Peb puas yuav tsum tau pub lawv rau nmap qee qhov, tau txais los ntawm qhov chaw, muab tso rau hauv, khiav ib yam dab tsi?..
MM: - Txias. Ib lo lus nug tseeb heev! Lub ntsiab lus yog qhov no. Peb hloov kho lub tsev qiv ntawv (ICMP ping, ib feem ntawm Zabbix) rau ICMP cov tshev, uas qhia txog cov pob ntawv - ib qho (1), thiab tus lej sim siv nmap. Ntawd yog, qhov no yog kev ua haujlwm sab hauv ntawm Zabbix, uas tau dhau los ua haujlwm sab hauv ntawm pinger. Yog li ntawd, tsis muaj synchronization los yog siv ib tug trapper yog yuav tsum tau. Qhov no tau ua kom txhob txwm tshaj tawm kom tawm hauv lub cev tsis zoo thiab tsis tas yuav cuam tshuam nrog kev sib koom ua ke ntawm ob lub chaw khaws ntaub ntawv: dab tsi los kuaj xyuas, upload los ntawm cov neeg xaiv tsa, thiab peb cov upload puas lawm?.. Qhov no yog qhov yooj yim dua.
A: - Nws puas ua haujlwm rau tus neeg sawv cev thiab?
MM: - Yog, tab sis peb tsis tau kuaj. Cov lej pov npav yog tib yam hauv Zabbix thiab server. Yuav tsum ua haujlwm. Cia kuv hais ib zaug ntxiv: qhov kev ua tau zoo ntawm qhov system yog qhov uas peb tsis xav tau tus neeg sawv cev.
MCH: - Cov lus teb raug rau cov lus nug yog: "Vim li cas koj thiaj li xav tau tus neeg sawv cev nrog cov kab ke no?" Tsuas yog vim NAT los yog saib xyuas los ntawm qee yam qeeb channel ...
A: - Thiab koj siv Zabbix ua tus allertor, yog tias kuv nkag siab raug. Los yog muaj koj cov duab (qhov twg cov txheej txheem archive yog) hloov mus rau lwm qhov system, xws li Grafana? Los yog koj tsis siv no functionality?
MM: - Kuv yuav hais ntxiv ib zaug ntxiv: peb tau ua tiav kev koom ua ke. Peb tab tom nchuav keeb kwm mus rau Clickhouse, tab sis tib lub sijhawm peb tau hloov php frontend. Php frontend mus rau Clickhouse thiab ua tag nrho cov duab los ntawm qhov ntawd. Nyob rau tib lub sijhawm, kom ua siab ncaj, peb muaj ib feem uas tsim cov ntaub ntawv hauv lwm cov duab kos duab los ntawm tib lub Clickhouse, los ntawm tib cov ntaub ntawv Zabbix.
MCH: - Hauv "Grafan" ib yam nkaus.
Cov kev txiav txim siab li cas txog kev faib cov peev txheej?
A: - Qhia me ntsis ntawm koj chav ua noj sab hauv. Qhov kev txiav txim siab tau ua li cas thiaj li tsim nyog los faib cov peev txheej rau kev ua haujlwm loj ntawm cov khoom? Cov no yog, feem ntau, tej yam txaus ntshai. Thiab thov qhia rau kuv, nyob rau hauv cov ntsiab lus ntawm qhov tseeb hais tias koj yuav txhawb cov tshiab versions: yuav ua li cas qhov kev txiav txim no yog vim li cas los ntawm ib tug tswj point ntawm view?
MM: - Thaj, peb tsis tau qhia zaj dab neeg ntawm keeb kwm zoo heev. Peb pom peb tus kheej nyob rau hauv ib qho xwm txheej uas yuav tsum tau ua, thiab peb tseem ceeb tau mus nrog ob pawg sib npaug:
- Ib qho tau tshaj tawm cov kev soj ntsuam uas siv cov txheej txheem tshiab: kev saib xyuas raws li kev pabcuam, cov txheej txheem qhib cov kev daws teeb meem uas peb muab sib xyaw ua ke thiab tom qab ntawd sim hloov cov txheej txheem kev lag luam txhawm rau ua haujlwm nrog cov kev soj ntsuam tshiab.
- Nyob rau tib lub sijhawm, peb muaj ib tus neeg ua haujlwm txaus siab uas tau ua qhov no (txog nws tus kheej). Nws thiaj li tshwm sim tias nws yeej.
A: - Thiab qhov loj ntawm pab neeg yog dab tsi?
MCH: - Nws nyob ntawm koj xub ntiag.
A: - Yog li, raws li ib txwm muaj, koj xav tau kev mob siab rau?
MM: - Kuv tsis paub tias tus mob siab rau yog dab tsi.
A: - Hauv qhov no, pom tseeb, koj. Ua tsaug ntau, koj zoo heev.
MM: - Ua tsaug.
Txog thaj ua rau thaj rau Zabbix
A: - Rau lub kaw lus uas siv tus neeg sawv cev (piv txwv li, hauv qee qhov kev faib tawm), nws puas tuaj yeem hloov kho thiab kho thaj chaw, hais tias, cov neeg xaiv tsa, tus neeg sawv cev thiab ib feem ntawm tus txheej txheem ua ntej ntawm Zabbix nws tus kheej; thiab lawv txoj kev sib raug zoo? Puas yog nws muaj peev xwm txhim kho cov kev tsim kho uas twb muaj lawm rau lub kaw lus nrog ntau tus neeg sawv cev?
MM: - Kuv paub tias Zabbix neeg rau zaub mov tau sib sau ua ke siv lub npe (cov cai tau muab tso ua ke thiab tau txais). Peb tsis tau sim qhov no hauv kev tsim khoom. Kuv tsis paub meej txog qhov no, tab sis kuv xav tias tus thawj tswj hwm preprocessor tsis siv hauv lub npe. Lub luag haujlwm ntawm tus neeg sawv cev yog coj cov txheej txheem ntsuas los ntawm Zabbix, sib koom ua ke (nws kuj sau cov kev teeb tsa, cov ntaub ntawv hauv zos) thiab muab nws rov qab rau Zabbix server. Lub server nws tus kheej yuav tom qab ntawd ua qhov preprocessing thaum nws tau txais nws.
Qhov kev txaus siab ntawm tus neeg sawv cev yog nkag siab. Peb mam li xyuas nws. Qhov no yog ib lub ntsiab lus nthuav.
A: - Lub tswv yim yog qhov no: yog tias koj tuaj yeem kho cov neeg xaiv tsa, koj tuaj yeem kho lawv ntawm lub npe thiab kho qhov kev cuam tshuam nrog cov neeg rau zaub mov, thiab hloov kho cov txheej txheem ua ntej rau cov hom phiaj no tsuas yog ntawm lub server.
MM: - Kuv xav tias nws yooj yim dua. Koj nqa cov cai, siv thaj ua rau thaj, tom qab ntawd teeb tsa nws txoj hauv kev koj xav tau - sau npe servers (piv txwv li, nrog ODBC) thiab faib cov lej patched hla cov kab ke. Thaum tsim nyog - sau npe, qhov tsim nyog - server.
A: - Feem ntau yuav, koj yuav tsis tau patch lub npe xa mus rau lub server ntxiv?
MCH: - Tsis yog, nws yog tus qauv.
MM: - Qhov tseeb, ib qho ntawm cov tswv yim tsis suab. Peb ib txwm tswj hwm qhov sib npaug ntawm qhov tawg ntawm cov tswv yim thiab qhov hloov pauv thiab yooj yim ntawm kev txhawb nqa.
Ib co ads 🙂
Ua tsaug uas koj tau nyob nrog peb. Koj puas nyiam peb cov ntawv? Xav pom cov ntsiab lus nthuav ntxiv? Txhawb nqa peb los ntawm kev tso ib qho kev txiav txim lossis qhia rau cov phooj ywg, , ib qho tshwj xeeb analogue ntawm nkag-theem servers, uas tau tsim los ntawm peb rau koj: (muaj nrog RAID1 thiab RAID10, mus txog 24 cores thiab mus txog 40GB DDR4).
Dell R730xd 2x pheej yig dua hauv Equinix Tier IV data center hauv Amsterdam? Tsuas yog nyob ntawm no hauv Netherlands! Dell R420 - 2x E5-2430 2.2Ghz 6C 128GB DDR3 2x960GB SSD 1Gbps 100TB - los ntawm $ 99! Nyeem txog
Tau qhov twg los: www.hab.com