Peb yuav saib yuav ua li cas Zabbix ua haujlwm nrog TimescaleDB database ua tus backend. Peb mam li qhia koj yuav pib li cas los ntawm kos thiab yuav ua li cas tsiv teb tsaws los ntawm PostgreSQL. Peb kuj tseem yuav muab kev sib piv kev ua tau zoo ntawm ob qho kev teeb tsa.
HighLoad++ Siberia 2019. Tomsk Hall. Peb 24, 16:00. Cov no thiab . Lub rooj sib tham HighLoad++ tom ntej no yuav muaj rau lub Plaub Hlis 6 thiab 7, 2020 hauv St. Petersburg. Paub meej thiab daim pib .
Andrey Gushchin (txuas ntxiv mus - AG): - Kuv yog ZABBIX technical support engineer (tom qab no hu ua "Zabbix"), tus kws qhia. Kuv tau ua haujlwm hauv kev txhawb nqa rau ntau tshaj 6 xyoo thiab tau muaj kev paub ncaj qha nrog kev ua haujlwm. Hnub no kuv yuav tham txog qhov ua tau zoo uas TimescaleDB tuaj yeem muab tau thaum piv nrog PostgreSQL 10 li niaj zaus. Tsis tas li ntawd, qee qhov kev qhia txog kev ua haujlwm feem ntau.
Cov kev sib tw ua tau zoo tshaj plaws: los ntawm kev sau cov ntaub ntawv mus rau kev ntxuav cov ntaub ntawv
Yuav pib nrog, muaj qee yam kev sib tw ua haujlwm uas txhua qhov kev saib xyuas tau ntsib. Thawj qhov kev sib tw tsim khoom yog kev sau thiab ua cov ntaub ntawv sai.
Kev saib xyuas zoo yuav tsum tau sai, raws sijhawm tau txais tag nrho cov ntaub ntawv, ua raws li cov lus qhia, uas yog, ua raws li qee cov txheej txheem (qhov no sib txawv hauv cov tshuab sib txawv) thiab khaws cia rau hauv cov ntaub ntawv txhawm rau siv cov ntaub ntawv no hauv yav tom ntej.
Qhov kev sib tw ua haujlwm thib ob yog keeb kwm khaws cia. Khaws rau hauv cov ntaub ntawv feem ntau thiab muaj kev nkag tau sai thiab yooj yim rau cov kev ntsuas no uas tau sau nyob rau lub sijhawm. Qhov tseem ceeb tshaj plaws yog tias cov ntaub ntawv no yog qhov yooj yim kom tau txais, siv nws hauv cov ntawv ceeb toom, cov duab kos, ua rau, hauv qee qhov pib qhov tseem ceeb, rau kev ceeb toom, thiab lwm yam.
Qhov kev sib tw thib peb yog keeb kwm tshem tawm, uas yog, thaum koj mus txog qhov chaw koj tsis tas yuav khaws cov ncauj lus kom ntxaws uas tau sau ntau dua 5 xyoo (txawm lub hlis lossis ob hlis). Qee lub network nodes raug tshem tawm, lossis qee tus tswv tsev, cov kev ntsuas tsis xav tau ntxiv lawm vim tias lawv twb dhau los lawm thiab tsis tau sau ntxiv lawm. Tag nrho cov no yuav tsum tau muab ntxuav kom huv si kom koj cov ntaub ntawv tsis loj dhau. Feem ntau, kev tshem tawm keeb kwm feem ntau yog qhov kev sim loj rau kev khaws cia - nws feem ntau muaj kev cuam tshuam zoo heev rau kev ua haujlwm.
Yuav daws teeb meem caching li cas?
Tam sim no kuv yuav tham tshwj xeeb txog Zabbix. Hauv Zabbix, thawj thiab thib ob hu tau daws siv caching.
Kev sau thiab ua cov ntaub ntawv - Peb siv RAM khaws tag nrho cov ntaub ntawv no. Cov ntaub ntawv no tam sim no yuav tau tham ntau yam ntxiv.
Tsis tas li ntawm sab database muaj qee qhov caching rau kev xaiv tseem ceeb - rau cov duab thiab lwm yam.
Caching ntawm sab ntawm Zabbix server nws tus kheej: peb muaj ConfigurationCache, ValueCache, HistoryCache, TrendsCache. Nws yog dab tsi?
ConfigurationCache yog lub ntsiab cache nyob rau hauv uas peb khaws metrics, hosts, cov ntaub ntawv khoom, triggers; txhua yam koj xav tau ua ntej ua ntej, sau cov ntaub ntawv, los ntawm cov tswv los sau, nrog dab tsi zaus. Tag nrho cov no yog khaws cia hauv ConfigurationCache kom tsis txhob mus rau hauv cov ntaub ntawv thiab tsim cov lus nug tsis tsim nyog. Tom qab tus neeg rau zaub mov pib, peb hloov kho qhov cache no (tsim nws) thiab hloov kho nws ib ntus (nyob ntawm qhov teeb tsa kev teeb tsa).
Caching hauv Zabbix. Cov ntaub ntawv khaws tseg
Ntawm no daim duab yog loj heev:
Lub ntsiab tseem ceeb hauv cov tswv yim yog cov neeg sau khoom no:
Cov no yog cov txheej txheem sib dhos lawv tus kheej, ntau yam "pollers" uas yog lub luag haujlwm rau ntau hom kev sib dhos. Lawv sau cov ntaub ntawv ntawm icmp, ipmi, thiab ntau yam kev cai thiab hloov tag nrho mus rau preprocessing.
PreProcessing HistoryCache
Tsis tas li ntawd, yog tias peb tau suav cov ntaub ntawv ntsiab lus (cov uas paub txog Zabbix paub), uas yog, suav, cov ntaub ntawv sib sau ua ke, peb coj lawv ncaj qha los ntawm ValueCache. Kuv mam li qhia koj seb nws tau sau li cas tom qab. Tag nrho cov neeg sau no siv ConfigurationCache kom tau txais lawv txoj haujlwm thiab tom qab ntawd xa lawv mus rau qhov ua ntej.
Preprocessing kuj tseem siv ConfigurationCache kom tau txais cov kauj ruam ua ntej thiab ua cov ntaub ntawv no ntau txoj hauv kev. Pib los ntawm version 4.2, peb tau hloov nws mus rau lub npe. Qhov no yooj yim heev, vim hais tias preprocessing nws tus kheej yog ib qho nyuaj ua hauj lwm. Thiab yog tias koj muaj Zabbix loj heev, nrog ntau cov ntaub ntawv hais txog thiab kev sau ntau zaus, qhov no ua rau kev ua haujlwm yooj yim heev.
Raws li, tom qab peb tau ua tiav cov ntaub ntawv no hauv qee txoj kev siv preprocessing, peb khaws nws hauv HistoryCache txhawm rau ua nws ntxiv. Qhov no xaus kev sau cov ntaub ntawv. Peb txav mus rau qhov txheej txheem tseem ceeb.
Keeb kwm syncer ua haujlwm
Cov txheej txheem tseem ceeb hauv Zabbix (vim nws yog monolithic architecture) yog keeb kwm syncer. Qhov no yog cov txheej txheem tseem ceeb uas cuam tshuam nrog kev ua haujlwm atomic ntawm txhua cov ntaub ntawv, uas yog, txhua tus nqi:
- tus nqi los (nws yuav siv nws los ntawm HistoryCache);
- tshawb xyuas hauv Configuration syncer: seb puas muaj kev cuam tshuam rau kev suav - suav lawv;
yog tias muaj - tsim cov xwm txheej, tsim kev nce ntxiv txhawm rau tsim kev ceeb toom, yog tias tsim nyog raws li kev teeb tsa; - cov ntaub ntawv ua rau kev ua haujlwm tom ntej, sib sau ua ke; yog tias koj sib sau ua ke dhau ib teev dhau los thiab lwm yam, tus nqi no nco qab los ntawm ValueCache kom tsis txhob mus rau lub rooj keeb kwm; Yog li, ValueCache tau sau nrog cov ntaub ntawv tsim nyog uas tsim nyog los suav cov cim, suav cov ntsiab lus, thiab lwm yam;
- ces keeb kwm syncer sau tag nrho cov ntaub ntawv mus rau lub database;
- cov ntaub ntawv sau lawv rau disk - qhov no yog qhov uas cov txheej txheem ua tiav.
Database. Caching
Nyob rau sab database, thaum koj xav saib cov duab lossis qee cov lus ceeb toom ntawm cov xwm txheej, muaj ntau yam caches. Tab sis nyob rau hauv tsab ntawv ceeb toom no kuv yuav tsis tham txog lawv.
Rau MySQL muaj Innodb_buffer_pool, thiab ib pawg ntawm cov caches sib txawv uas tuaj yeem teeb tsa.
Tab sis cov no yog cov tseem ceeb:
- shared_buffers;
- effect_cache_size;
- shared_pool.
Rau tag nrho cov databases, kuv tau hais tias muaj qee yam caches uas tso cai rau koj khaws cia hauv RAM cov ntaub ntawv uas feem ntau xav tau rau cov lus nug. Lawv muaj lawv tus kheej technologies rau qhov no.
Hais txog Database Performance
Raws li, muaj kev sib tw ib puag ncig, uas yog, Zabbix server khaws cov ntaub ntawv thiab sau nws. Thaum rov pib dua, nws kuj nyeem los ntawm keeb kwm kom sau ValueCache thiab lwm yam. Ntawm no koj tuaj yeem muaj cov ntawv sau thiab cov ntawv ceeb toom uas siv Zabbix API, uas yog tsim los ntawm lub vev xaib interface. Zabbix API nkag mus rau hauv cov ntaub ntawv thiab tau txais cov ntaub ntawv tsim nyog kom tau txais cov duab kos, cov ntawv ceeb toom, lossis qee cov npe ntawm cov xwm txheej, teeb meem tsis ntev los no.
Tsis tas li ntawd ib qho kev daws teeb meem nrov heev yog Grafana, uas peb cov neeg siv siv. Muaj peev xwm nkag ncaj qha rau hauv ob qho tib si los ntawm Zabbix API thiab los ntawm cov ntaub ntawv. Nws kuj tsim ib qho kev sib tw rau kev tau txais cov ntaub ntawv: ib qho kev sib tw, zoo dua ntawm cov ntaub ntawv yuav tsum tau ua raws li kev xa tawm sai ntawm cov txiaj ntsig thiab kev sim.
Clearing keeb kwm. Zabbix muaj Tus Saib Xyuas Tsev
Qhov thib peb hu uas yog siv nyob rau hauv Zabbix yog clearing keeb kwm siv Housekeeper. Tus neeg saib xyuas hauv tsev ua raws tag nrho cov kev teeb tsa, uas yog, peb cov ntaub ntawv hais qhia tias yuav khaws ntev npaum li cas (hauv hnub), ntev npaum li cas los khaws cov qauv, thiab cov kev hloov pauv hloov.
Kuv tsis tau tham txog TrendCache, uas peb suav cov yoov: cov ntaub ntawv tuaj txog, peb suav nws rau ib teev (feem ntau cov no yog cov lej rau teev dhau los), tus nqi nruab nrab / tsawg kawg thiab peb sau nws ib teev hauv ib teev. lub rooj ntawm cov kev hloov pauv hloov ("Tshuaj"). "Housekeeper" pib thiab tshem tawm cov ntaub ntawv los ntawm cov ntaub ntawv siv cov kev xaiv tsis tu ncua, uas tsis tas yuav ua haujlwm zoo.
Yuav ua li cas to taub tias nws tsis muaj txiaj ntsig? Koj tuaj yeem pom cov duab hauv qab no ntawm cov duab ua haujlwm ntawm cov txheej txheem sab hauv:
Koj keeb kwm syncer tsis tu ncua (daim duab liab). Thiab cov duab "liab" uas mus rau saum. Qhov no yog "Tus Saib Xyuas Tsev" uas pib thiab tos cov ntaub ntawv tshem tawm tag nrho cov kab uas nws tau teev tseg.
Cia peb coj qee yam ID: koj yuav tsum rho tawm 5 txhiab kawg; tau kawg, los ntawm indexes. Tab sis feem ntau cov ntaub ntawv loj heev - cov ntaub ntawv tseem nyeem nws los ntawm disk thiab muab tso rau hauv lub cache, thiab qhov no yog qhov haujlwm kim heev rau cov ntaub ntawv. Nyob ntawm nws qhov loj me, qhov no tuaj yeem ua rau qee yam teeb meem kev ua haujlwm.
Koj tuaj yeem lov tes taw Tus Neeg Saib Xyuas Tsev hauv txoj hauv kev yooj yim - peb muaj lub vev xaib paub. Kev teeb tsa hauv Thawj Tswj Hwm (kev teeb tsa rau "Tus Saib Xyuas Tsev") peb lov tes taw kev saib xyuas hauv tsev rau sab hauv keeb kwm thiab cov qauv. Yog li ntawd, tus tswv tsev tsis tswj qhov no ntxiv lawm:
Koj tuaj yeem ua dab tsi ntxiv? Koj muab nws tso tseg, koj cov duab tau ua tiav lawm ... Dab tsi ntxiv teeb meem tuaj yeem tshwm sim hauv qhov no? Dab tsi tuaj yeem pab tau?
Partitioning (zeeg)
Feem ntau qhov no tau teeb tsa nyob rau hauv ib txoj kev sib txawv ntawm txhua qhov kev sib raug zoo database kuv tau teev tseg. MySQL muaj nws tus kheej thev naus laus zis. Tab sis tag nrho lawv zoo sib xws thaum nws los txog rau PostgreSQL 10 thiab MySQL. Tau kawg, muaj ntau qhov sib txawv ntawm qhov sib txawv ntawm nws txhua qhov kev siv thiab yuav ua li cas nws cuam tshuam rau kev ua haujlwm. Tab sis feem ntau, kev tsim cov kev faib tshiab feem ntau kuj ua rau qee yam teeb meem.
Nyob ntawm koj qhov teeb tsa (ntau npaum li cas cov ntaub ntawv koj tsim hauv ib hnub), lawv feem ntau teeb tsa qhov tsawg kawg nkaus - qhov no yog 1 hnub / batch, thiab rau "kev nyiam", kev hloov pauv hloov - 1 hli / batch tshiab. Qhov no tuaj yeem hloov pauv yog tias koj muaj kev teeb tsa loj heev.
Cia peb hais tam sim ntawd txog qhov loj ntawm kev teeb tsa: mus txog 5 txhiab tus nqi tshiab ib ob (hu ua nvps) - qhov no yuav suav tias yog "kev teeb tsa" me me. Nruab nrab - los ntawm 5 mus rau 25 txhiab tus nqi ib ob. Txhua yam uas yog saum toj no yog twb loj thiab loj heev installations uas yuav tsum tau ceev faj heev configuration ntawm database.
Hauv kev teeb tsa loj heev, 1 hnub yuav tsis zoo. Kuv tus kheej tau pom kev faib tawm ntawm MySQL ntawm 40 gigabytes ib hnub (thiab yuav muaj ntau dua). Qhov no yog cov ntaub ntawv loj heev, uas tuaj yeem ua rau qee qhov teeb meem. Nws yuav tsum tau txo.
Vim li cas koj thiaj xav tau partitioning?
Qhov Partitioning muab, Kuv xav tias txhua tus paub, yog kev faib cov lus. Feem ntau cov no yog cov ntaub ntawv cais ntawm disk thiab ncua kev thov. Nws xaiv ib tug muab faib ntau optimally yog hais tias nws yog ib feem ntawm ib txwm muab faib.
Rau Zabbix, tshwj xeeb, nws yog siv los ntawm ntau yam, los ntawm ntau yam, uas yog, peb siv lub sijhawm (tus lej tsis tu ncua, lub sijhawm txij li thaum pib ntawm lub sijhawm). Koj qhia qhov pib ntawm hnub / kawg ntawm hnub, thiab qhov no yog qhov muab faib. Raws li, yog tias koj tab tom thov cov ntaub ntawv uas muaj hnub nyoog ob hnub, txhua yam rov qab los ntawm cov ntaub ntawv sai dua, vim tias koj tsuas yog yuav tsum tau thauj ib cov ntaub ntawv rau hauv cache thiab xa rov qab (tsis yog lub rooj loj).
Ntau cov ntaub ntawv kuj tseem ua kom nrawm nrawm (ntxig rau hauv ib lub rooj me nyuam). Kuv tabtom hais abstractly rau tam sim no, tab sis qhov no kuj ua tau. Kev faib feem ntau pab tau.
Elasticsearch rau NoSQL
Tsis ntev los no, hauv 3.4, peb tau siv qhov kev daws teeb meem NoSQL. Ntxiv qhov muaj peev xwm sau ntawv hauv Elasticsearch. Koj tuaj yeem sau qee yam: koj xaiv - sau tus lej lossis qee yam cim; peb muaj hlua cov ntawv, koj tuaj yeem sau cov cav rau Elasticsearch... Raws li, lub vev xaib interface tseem yuav nkag mus rau Elasticsearch. Qhov no ua haujlwm zoo hauv qee kis, tab sis tam sim no nws tuaj yeem siv tau.
TimescaleDB. Hypertables
Rau 4.4.2 peb tau xyuam xim rau ib yam zoo li TimescaleDB. Nws yog dab tsi? Qhov no yog qhov txuas ntxiv rau PostgreSQL, uas yog, nws muaj ib txwm muaj PostgreSQL interface. Ntxiv rau, qhov txuas ntxiv no tso cai rau koj ua haujlwm tau zoo dua nrog cov ntaub ntawv teev sijhawm thiab muaj kev faib tsis siv neeg. Nws zoo li cas:
Qhov no yog hypertable - muaj xws li lub tswv yim hauv Timescale. Qhov no yog hypertable uas koj tsim, thiab nws muaj chunks. Chunks yog partitions, cov no yog cov rooj me nyuam, yog tias kuv tsis yuam kev. Nws ua tau zoo heev.
TimescaleDB thiab PostgreSQL
Raws li TimescaleDB cov tuam txhab paub tseeb, lawv siv cov txheej txheem kom raug ntau dua rau kev ua cov lus nug, tshwj xeeb hauv cov ntawv ntxig, uas tso cai rau kwv yees li qhov ua tau zoo nrog qhov loj ntawm cov ntaub ntawv ntxig. Ntawd yog, tom qab 200 lab kab ntawm Postgres, ib txwm pib sag ntau heev thiab poob qhov kev ua tau zoo rau xoom, thaum Timescale tso cai rau koj los ntxig ntxig cov ntaub ntawv zoo li ua tau nrog cov ntaub ntawv ntau npaum li cas.
Yuav ua li cas rau nruab TimescaleDB? Nws yog qhov yooj yim!
Nws nyob hauv cov ntaub ntawv, nws tau piav qhia - koj tuaj yeem nruab nws los ntawm pob khoom rau ib qho twg ... Nws nyob ntawm cov pob khoom Postgres. Yuav muab tso ua ke manually. Nws thiaj li tshwm sim uas kuv yuav tsum tau sau rau lub database.
Ntawm Zabbix peb tsuas qhib Extention. Kuv xav tias cov uas siv Extention hauv Postgres... Koj tsuas yog qhib Extention, tsim nws rau Zabbix database uas koj siv.
Thiab cov kauj ruam kawg ...
TimescaleDB. Kev tsiv teb tsaws ntawm cov ntxhuav keeb kwm
Koj yuav tsum tsim ib lub rooj sib tham. Nws muaj qhov tshwj xeeb ua haujlwm rau qhov no - Tsim hypertable. Nyob rau hauv nws, thawj parameter yog lub rooj uas yuav tsum tau nyob rau hauv no database (rau uas koj yuav tsum tau tsim ib tug hypertable).
Qhov chaw uas yuav tsim, thiab chunk_time_interval (qhov no yog lub caij nyoog ntawm chunks (partitions uas yuav tsum tau siv). 86 yog ib hnub.
Migrate_data parameter: Yog tias koj ntxig rau qhov tseeb, qhov no yuav hloov pauv tag nrho cov ntaub ntawv tam sim no mus rau qhov tsim ua ntej.
Kuv tau siv migrate_data kuv tus kheej - nws yuav siv sij hawm ncaj ncees, nyob ntawm seb koj cov ntaub ntawv loj npaum li cas. Kuv muaj tshaj ib terabyte - nws siv sijhawm li ib teev los tsim. Qee zaum, thaum kuaj, kuv tshem tawm cov ntaub ntawv keeb kwm rau cov ntawv nyeem (history_text) thiab hlua (history_str) kom tsis txhob hloov lawv - lawv tsis txaus siab rau kuv.
Thiab peb ua qhov hloov tshiab kawg hauv peb db_extention: peb nruab timescaledb kom cov ntaub ntawv thiab, tshwj xeeb, peb Zabbix nkag siab tias muaj db_extention. Nws qhib nws thiab siv qhov tseeb syntax thiab cov lus nug rau cov ntaub ntawv, siv cov "tus yam ntxwv" uas tsim nyog rau TimescaleDB.
Server configuration
Kuv siv ob lub servers. Thawj tus neeg rau zaub mov yog lub tshuab virtual me me, 20 processors, 16 gigabyte ntawm RAM. Kuv teeb tsa Postgres 10.8 ntawm nws:
Lub operating system yog Debian, cov ntaub ntawv kaw lus yog xfs. Kuv tau teeb tsa tsawg heev los siv cov ntaub ntawv tshwj xeeb no, rho tawm dab tsi Zabbix nws tus kheej yuav siv. Hauv tib lub tshuab muaj Zabbix server, PostgreSQL thiab cov neeg ua haujlwm thauj khoom.
Kuv tau siv 50 tus neeg ua haujlwm nquag uas siv LoadableModule kom sai sai tsim cov txiaj ntsig sib txawv. Lawv yog cov uas tsim cov hlua, cov lej, thiab lwm yam. Kuv sau cov ntaub ntawv nrog ntau cov ntaub ntawv. Thaum xub thawj, qhov kev teeb tsa muaj 5 txhiab cov ntaub ntawv hauv ib tus tswv tsev, thiab kwv yees li ntawm txhua cov ntaub ntawv muaj ib qho kev tshwm sim - txhawm rau ua qhov teeb meem tiag tiag. Qee lub sij hawm koj tseem xav tau ntau tshaj ib qho kev siv los siv.
Kuv tswj hwm lub sijhawm hloov tshiab thiab kev thauj khoom nws tus kheej los ntawm tsis tsuas yog siv 50 tus neeg sawv cev (ntxiv ntxiv), tab sis kuj siv cov ntaub ntawv dynamic thiab txo qhov hloov tshiab rau 4 vib nas this.
Kev xeem ua haujlwm. PostgreSQL: 36 NVPs
Thawj qhov kev tso tawm, thawj qhov teeb tsa kuv tau muaj nyob rau ntawm PostreSQL 10 ntshiab ntawm cov khoom siv no (35 txhiab tus nqi ib ob). Feem ntau, raws li koj tuaj yeem pom ntawm lub vijtsam, ntxig cov ntaub ntawv siv feem ntawm ib pliag - txhua yam zoo thiab nrawm, SSD drives (200 gigabytes). Qhov tsuas yog qhov 20 GB ua tiav sai heev.
Yuav muaj ntau cov duab zoo li no yav tom ntej. Qhov no yog tus qauv Zabbix server kev ua haujlwm dashboard.
Thawj daim duab yog tus naj npawb ntawm cov nqi ib ob (xiav, sab laug sab laug), 35 txhiab tus nqi hauv qhov no. Qhov no (qhov chaw saum toj kawg nkaus) yog qhov chaw thau khoom ntawm cov txheej txheem tsim, thiab qhov no (sab saum toj sab xis) yog qhov chaw thau khoom ntawm cov txheej txheem sab hauv: keeb kwm syncers thiab tus neeg saib xyuas tsev, uas ntawm no (hauv qab nruab nrab) tau ua haujlwm rau qee lub sijhawm.
Daim duab no (hauv qab nruab nrab) qhia txog kev siv ValueCache - pes tsawg tus ValueCache hits rau qhov tshwm sim (ob peb txhiab tus nqi ib ob). Lwm daim duab tseem ceeb yog qhov thib plaub (hauv qab sab laug), uas qhia txog kev siv HistoryCache, uas kuv tau tham txog, uas yog qhov tsis zoo ua ntej tso rau hauv cov ntaub ntawv.
Kev xeem ua haujlwm. PostgreSQL: 50 NVPs
Tom ntej no, kuv nce lub load rau 50 txhiab tus nqi ib ob ntawm tib lub kho vajtse. Thaum thauj khoom los ntawm Tus Saib Xyuas Tsev, 10 txhiab tus nqi raug kaw hauv 2-3 vib nas this nrog kev suav. Dab tsi, qhov tseeb, yog qhia hauv cov screenshot hauv qab no:
"Tus neeg saib xyuas tsev" twb pib cuam tshuam nrog kev ua haujlwm, tab sis feem ntau, cov ntaub ntawv ntawm cov dab neeg hauv keeb kwm-sinker trappers tseem nyob rau theem 60% (thib peb daim duab, sab xis). HistoryCache twb pib ua kom nquag plias thaum tus neeg saib xyuas tsev khiav (hauv qab sab laug). Nws yog kwv yees li ib nrab ntawm gigabyte, 20% tag nrho.
Kev xeem ua haujlwm. PostgreSQL: 80 NVPs
Tom qab ntawd kuv tau nce nws mus rau 80 txhiab tus nqi ib ob:
Nws yog kwv yees li 400 txhiab cov ntaub ntawv ntsiab lus, 280 txhiab ua rau. Qhov ntxig, raws li koj tuaj yeem pom, nyob rau hauv cov nqe lus ntawm lub load ntawm keeb kwm sinkers (muaj 30 ntawm lawv) twb heev heev. Tom qab ntawd kuv tau nce ntau qhov tsis sib xws: keeb kwm sinkers, cache ... Ntawm cov khoom siv no, cov khoom thauj ntawm keeb kwm sinkers pib nce mus rau qhov siab tshaj plaws, yuav luag "ntawm lub txee" - raws li, HistoryCache tau mus rau hauv qhov hnyav heev:
Txhua lub sijhawm no kuv tau saib xyuas tag nrho cov kab ke tsis ua haujlwm (yuav ua li cas cov processor siv, RAM) thiab pom tias kev siv disk yog qhov siab tshaj plaws - Kuv ua tiav qhov siab tshaj plaws ntawm qhov disk ntawm cov khoom siv no, ntawm lub tshuab virtual no. "Postgres" pib pov tseg cov ntaub ntawv tseem ceeb heev ntawm qhov kev siv, thiab lub disk tsis muaj sijhawm los sau, nyeem ...
Kuv coj lwm tus neeg rau zaub mov uas twb muaj 48 processors thiab 128 gigabytes ntawm RAM:
Kuv kuj "tuned" nws - ntsia Keeb kwm syncer (60 daim) thiab ua tiav qhov kev ua tau zoo. Qhov tseeb, peb tsis yog "ntawm lub txee," tab sis qhov no yog qhov txwv ntawm kev tsim khoom, qhov twg nws yog qhov tsim nyog los ua ib yam dab tsi txog nws.
Kev xeem ua haujlwm. TimescaleDB: 80 NVPs
Kuv lub luag haujlwm tseem ceeb yog siv TimescaleDB. Txhua daim duab qhia txog qhov poob:
Cov kev ua tsis tiav no yog qhov tseeb cov ntaub ntawv tsiv teb tsaws. Tom qab ntawd, hauv Zabbix server, qhov chaw thauj khoom ntawm keeb kwm sinkers, raws li koj tuaj yeem pom, hloov ntau heev. Nws tso cai rau koj ntxig cov ntaub ntawv yuav luag 3 zaug sai dua thiab siv HistoryCache tsawg dua - raws li, koj yuav muaj cov ntaub ntawv xa tuaj raws sijhawm. Ib zaug ntxiv, 80 txhiab tus nqi ib ob yog tus nqi siab (ntawm chav kawm, tsis yog rau Yandex). Zuag qhia tag nrho qhov no yog qhov teeb tsa loj loj, nrog rau ib tus neeg rau zaub mov.
PostgreSQL kev xeem ua haujlwm: 120 txhiab NVPs
Tom ntej no, kuv tau nce tus nqi ntawm cov ntaub ntawv hais txog ib nrab lab thiab tau txais tus nqi suav ntawm 125 txhiab ib ob:
Thiab kuv tau txais cov duab no:
Hauv txoj cai, qhov no yog kev teeb tsa ua haujlwm, nws tuaj yeem ua haujlwm ntev heev. Tab sis txij li thaum kuv tsuas muaj 1,5 terabyte disk, kuv siv nws li ob peb hnub. Qhov tseem ceeb tshaj plaws yog tias tib lub sijhawm tshiab partitions tau tsim nyob rau hauv TimescaleDB, thiab qhov no yog tag nrho unnoticed rau kev ua tau zoo, uas tsis tuaj yeem hais txog MySQL.
Feem ntau, cov partitions yog tsim thaum hmo ntuj, vim hais tias qhov no feem ntau thaiv kev ntxig thiab ua haujlwm nrog cov ntxhuav thiab tuaj yeem ua rau degradation ntawm cov kev pabcuam. Hauv qhov no tsis yog qhov xwm txheej! Lub luag haujlwm tseem ceeb yog kuaj lub peev xwm ntawm TimescaleDB. Qhov tshwm sim yog cov duab hauv qab no: 120 txhiab tus nqi ib ob.
Tseem muaj cov piv txwv hauv zej zog:
Tus neeg kuj tau tig rau TimescaleDB thiab cov load ntawm kev siv io.weight poob ntawm lub processor; thiab kev siv cov txheej txheem sab hauv kuj tau txo qis vim suav nrog TimescaleDB. Ntxiv mus, cov no yog cov pancake zoo tib yam, uas yog, lub tshuab virtual zoo tib yam ntawm cov disks zoo tib yam (tsis yog SSDs)!
Rau qee qhov kev teeb tsa me me uas txwv los ntawm kev ua haujlwm disk, TimescaleDB, hauv kuv lub tswv yim, yog ib qho kev daws teeb meem zoo heev. Nws yuav tso cai rau koj ua haujlwm txuas ntxiv ua ntej tsiv mus rau cov khoom siv ceev dua rau cov ntaub ntawv.
Kuv caw nej txhua tus los ntawm peb cov xwm txheej: Kev Sib Tham hauv Moscow, Summit hauv Riga. Siv peb cov channel - Telegram, rooj sab laj, IRC. Yog tias koj muaj lus nug, tuaj rau peb lub rooj, peb tuaj yeem tham txog txhua yam.
Cov Lus Qhuab Qhia
Cov lus nug los ntawm cov neeg tuaj saib (tom qab no - A): - Yog tias TimescaleDB yooj yim rau kev teeb tsa, thiab nws muab qhov kev txhawb nqa zoo li no, tej zaum qhov no yuav tsum tau siv los ua qhov kev coj ua zoo tshaj plaws rau kev teeb tsa Zabbix nrog Postgres? Thiab puas muaj qhov tsis zoo thiab qhov tsis zoo ntawm qhov kev daws teeb meem no, lossis tom qab tag nrho, yog tias kuv txiav txim siab ua Zabbix rau kuv tus kheej, Kuv tuaj yeem yooj yim nqa Postgres, nruab Timescale tam sim ntawd, siv nws thiab tsis xav txog tej teeb meem?
AG: - Yog lawm, kuv xav hais tias qhov no yog qhov kev pom zoo: siv Postgres tam sim ntawd nrog TimescaleDB txuas ntxiv. Raws li kuv twb tau hais lawm, ntau qhov kev tshuaj xyuas zoo, txawm hais tias qhov "feature" no yog kev sim. Tab sis qhov tseeb kev sim qhia tias qhov no yog ib qho kev daws teeb meem zoo (nrog TimescaleDB) thiab kuv xav tias nws yuav hloov zuj zus! Peb tab tom saib xyuas qhov txuas ntxiv no txhim kho li cas thiab yuav hloov pauv raws li qhov tsim nyog.
Txawm tias thaum lub sij hawm txoj kev loj hlob, peb cia siab rau ib qho ntawm lawv cov paub zoo "tus yam ntxwv": nws muaj peev xwm ua hauj lwm nrog chunks txawv me ntsis. Tab sis tom qab ntawd lawv txiav nws tawm hauv qhov kev tso tawm tom ntej, thiab peb yuav tsum tsis txhob tso siab rau txoj cai no. Kuv xav kom siv qhov kev daws teeb meem no rau ntau qhov teeb tsa. Yog tias koj siv MySQL... Rau nruab nrab kev teeb tsa, txhua qhov kev daws teeb meem ua haujlwm zoo.
A: - Ntawm daim duab kawg ntawm lub zej zog, muaj ib daim duab nrog "Tus Saib Xyuas Tsev":
Nws txuas ntxiv ua haujlwm. Tus Kws Saib Xyuas Tsev ua li cas nrog TimescaleDB?
AG: - Tam sim no kuv tsis tuaj yeem hais meej - Kuv yuav saib cov cai thiab qhia koj kom ntxaws ntxiv. Nws siv TimescaleDB cov lus nug tsis txhob rho tawm cov chunks, tab sis rau qee qhov sib sau ua ke. Kuv tseem tsis tau npaj los teb cov lus nug no. Peb mam li paub ntau ntxiv ntawm qhov sawv ntsug hnub no lossis tag kis.
A: - Kuv muaj lus nug zoo sib xws - txog kev ua haujlwm ntawm kev tshem tawm hauv Timescale.
A (cov lus teb los ntawm cov neeg tuaj saib): - Thaum koj rho tawm cov ntaub ntawv los ntawm lub rooj, yog tias koj ua nws ntawm kev tshem tawm, ces koj yuav tsum mus dhau lub rooj - tshem tawm, ntxuav, kos txhua yam rau lub tshuab nqus tsev yav tom ntej. Hauv Timescale, txij li koj muaj chunks, koj tuaj yeem poob. Hais lus ntxhib, koj tsuas qhia cov ntaub ntawv uas nyob hauv cov ntaub ntawv loj: "Rho tawm!"
Timescale tsuas to taub tias tej chunk no tsis muaj lawm. Thiab txij li nws tau koom ua ke rau hauv cov lus nug npaj, nws siv hooks los ntes koj cov xwm txheej hauv kev xaiv lossis lwm yam haujlwm thiab tam sim ntawd nkag siab tias cov chunk no tsis muaj lawm - "Kuv yuav tsis mus lawm!" (cov ntaub ntawv tsis muaj). Yog tag nrho! Ntawd yog, lub rooj scan yog hloov los ntawm kev rho tawm cov ntaub ntawv binary, yog li nws nrawm.
A: - Peb twb tau kov lub ntsiab lus tsis yog SQL. Raws li kuv nkag siab, Zabbix tsis tshua xav hloov cov ntaub ntawv, thiab tag nrho cov no yog qee yam zoo li lub cav. Puas yog nws tuaj yeem siv cov ntaub ntawv tshwj xeeb uas tsis tuaj yeem hloov pauv lawv cov ntaub ntawv, tab sis tib lub sijhawm txuag, khaws, thiab faib sai dua - Clickhouse, piv txwv li, ib yam dab tsi Kafka-zoo li?... Kafka kuj yog lub cav! Puas yog nws ua tau rau qee yam kev koom ua ke rau lawv?
AG: - Kev tshem tawm tuaj yeem ua tiav. Peb muaj ib qho "feature" txij li version 3.4: koj tuaj yeem sau tag nrho cov ntaub ntawv keeb kwm, txheej xwm, txhua yam rau cov ntaub ntawv; thiab tom qab ntawd xa nws mus rau lwm cov ntaub ntawv siv qee tus tuav. Qhov tseeb, ntau tus neeg rov ua haujlwm thiab sau ncaj qha rau hauv cov ntaub ntawv. Thaum ya, keeb kwm sinkers sau tag nrho cov no rau hauv cov ntaub ntawv, tig cov ntaub ntawv no, thiab lwm yam, thiab koj tuaj yeem hloov qhov no mus rau Clickhouse. Kuv tsis tuaj yeem hais txog cov phiaj xwm, tab sis tej zaum ntxiv kev txhawb nqa rau NoSQL cov kev daws teeb meem (xws li Clickhouse) yuav txuas ntxiv mus.
A: - Feem ntau, nws hloov tawm tias koj tuaj yeem tshem tawm tag nrho cov postgres?
AG: - Tau kawg, qhov nyuaj tshaj plaws hauv Zabbix yog cov ntxhuav keeb kwm, uas tsim cov teeb meem feem ntau, thiab cov xwm txheej. Nyob rau hauv cov ntaub ntawv no, yog tias koj tsis khaws cov xwm txheej rau lub sijhawm ntev thiab khaws cov keeb kwm nrog cov qauv hauv qee qhov chaw khaws cia, feem ntau, kuv xav tias, yuav tsis muaj teeb meem.
A: - Koj puas tuaj yeem kwv yees ntau npaum li cas txhua yam yuav ua haujlwm yog tias koj hloov mus rau Clickhouse, piv txwv li?
AG: - Kuv tsis tau sim nws. Kuv xav tias tsawg kawg tib tus lej tuaj yeem ua tiav yooj yim, muab qhov Clickhouse muaj nws tus kheej interface, tab sis kuv tsis tuaj yeem hais meej. Nws yog qhov zoo dua los sim. Nws tag nrho yog nyob ntawm kev teeb tsa: pes tsawg tus tswv koj muaj, thiab lwm yam. Ntxig yog ib yam, tab sis koj kuj yuav tsum tau muab cov ntaub ntawv no - Grafana lossis lwm yam.
A: - Yog li peb tab tom tham txog kev sib ntaus sib tua, thiab tsis yog hais txog qhov zoo ntawm cov ntaub ntawv ceev ceev no?
AG: - Kuv xav tias thaum peb sib koom ua ke, yuav muaj kev sim siab dua.
A: - Qhov zoo qub RRD mus qhov twg? Dab tsi ua rau koj hloov mus rau SQL databases? Thaum pib, tag nrho cov kev ntsuas tau sau rau ntawm RRD.
AG: - Zabbix muaj RRD, tej zaum nyob rau hauv ib tug ancient version. Muaj ib txwm muaj SQL databases - ib txoj hauv kev classic. Txoj hauv kev classic yog MySQL, PostgreSQL (lawv tau muaj nyob rau lub sijhawm ntev). Peb yuav luag tsis tau siv ib qho kev sib txuas rau SQL thiab RRD databases.
Ib co ads π
Ua tsaug uas koj tau nyob nrog peb. Koj puas nyiam peb cov ntawv? Xav pom cov ntsiab lus nthuav ntxiv? Txhawb nqa peb los ntawm kev tso ib qho kev txiav txim lossis qhia rau cov phooj ywg, , ib qho tshwj xeeb analogue ntawm nkag-theem servers, uas tau tsim los ntawm peb rau koj: (muaj nrog RAID1 thiab RAID10, mus txog 24 cores thiab mus txog 40GB DDR4).
Dell R730xd 2x pheej yig dua hauv Equinix Tier IV data center hauv Amsterdam? Tsuas yog nyob ntawm no hauv Netherlands! Dell R420 - 2x E5-2430 2.2Ghz 6C 128GB DDR3 2x960GB SSD 1Gbps 100TB - los ntawm $ 99! Nyeem txog
Tau qhov twg los: www.hab.com