Kuchita kwakukulu komanso kugawa kwawo: Zabbix yokhala ndi chithandizo cha TimescaleDB
Zabbix ndi njira yowunikira. Monga machitidwe ena aliwonse, ikukumana ndi mavuto atatu akuluakulu a machitidwe onse owunikira: kusonkhanitsa ndi kukonza deta, kusunga mbiri, ndi kuyeretsa.
Magawo olandila, kukonza ndi kujambula deta amatenga nthawi. Osati zambiri, koma kwa dongosolo lalikulu izi zingayambitse kuchedwa kwakukulu. Vuto losungirako ndi vuto la kupeza deta. Amagwiritsidwa ntchito popanga malipoti, macheke ndi zoyambitsa. Kuchedwa kwa data kumakhudzanso magwiridwe antchito. Pamene nkhokwe zikukula, deta yosafunika iyenera kuchotsedwa. Kuchotsa ndi ntchito yovuta yomwe imadyanso zinthu zina.
Mavuto akuchedwa panthawi yosonkhanitsa ndi kusungirako ku Zabbix amathetsedwa ndi caching: mitundu ingapo ya cache, caching mu database. Kuti athetse vuto lachitatu, caching siyoyenera, kotero Zabbix adagwiritsa ntchito TimescaleDB. Iye adzakuuzani inu za izo Andrey Gushchin - injiniya wothandizira luso Zabbix SIA. Andrey wakhala akuthandizira Zabbix kwa zaka zoposa 6 ndipo ali ndi chidziwitso chachindunji ndi ntchito.
Kodi TimescaleDB imagwira ntchito bwanji, ingapereke ntchito yotani poyerekeza ndi PostgreSQL yokhazikika? Kodi Zabbix imagwira ntchito yanji pa database ya TimescaleDB? Kodi mungayambire bwanji kuchokera pachiyambi komanso momwe mungasamukire kuchokera ku PostgreSQL ndi kasinthidwe kati komwe kamagwira ntchito bwino? Za zonsezi pansi pa odulidwa.
Zovuta Zazantchito
Dongosolo lililonse loyang'anira limayang'anizana ndi zovuta zinazake. Ndilankhula za zitatu mwa izo: kusonkhanitsa deta ndi kukonza, kusunga, ndi kuchotsa mbiri.
Kusonkhanitsa deta mwachangu ndi kukonza. Dongosolo loyang'anira bwino liyenera kulandira mwachangu deta yonse ndikuyikonza molingana ndi mawu oyambitsa - molingana ndi momwe imakhalira. Pambuyo pokonza, dongosololi liyeneranso kusunga mwachangu deta iyi mu database kuti igwiritsidwe ntchito mtsogolo.
Kusungirako mbiri. Dongosolo labwino loyang'anira liyenera kusungitsa mbiri mu nkhokwe ndikupereka mwayi wosavuta kumeta. Mbiri ikufunika kuti igwiritsidwe ntchito mu malipoti, ma graph, zoyambitsa, zoyambira, ndi zinthu zowerengera za chenjezo.
Kuyeretsa deta yakale ndi vuto lalikulu lomwe limakhudza kwambiri magwiridwe antchito a database.
Kusungidwa mu Zabbix
Ku Zabbix, kuyimba koyamba ndi kwachiwiri kumathetsedwa pogwiritsa ntchito caching. RAM imagwiritsidwa ntchito kusonkhanitsa ndi kukonza deta. Posungira - mbiri mu zoyambitsa, ma graph ndi zinthu zowerengeka za data. Pambali ya database pali zosungira zina zosankhidwa, mwachitsanzo, ma graph.
Kusunga kumbali ya seva ya Zabbix palokha ndi:
ConfigurationCache;
ValueCache;
HistoryCache;
TrendsCache.
Tiyeni tiwalingalire mwatsatanetsatane.
ConfigurationCache
Umu ndiye posungira wamkulu momwe timasungira ma metrics, zosungira, zinthu za data, zoyambitsa - chilichonse chomwe timafunikira pa PreProcessing komanso kusonkhanitsa deta.
Zonsezi zimasungidwa mu ConfigurationCache kuti musapange mafunso osafunikira mu database. Seva ikayamba, timasintha cache iyi, kupanga ndikusintha masinthidwe nthawi ndi nthawi.
Kusonkhanitsa deta
Chithunzicho ndi chachikulu ndithu, koma chinthu chachikulu mmenemo ndi osankha. Awa ndi "oponya voti" osiyanasiyana - njira zosonkhanitsira. Iwo ali ndi udindo wa mitundu yosiyanasiyana ya msonkhano: amasonkhanitsa deta kudzera pa SNMP, IPMI, ndikusamutsira zonse ku PreProcessing.
Zosonkhanitsa zimafotokozedwa mu lalanje.
Zabbix yawerengera zinthu zophatikiza zomwe zimafunikira kuti macheke awonjezeke. Ngati tili nawo, timawatengera deta mwachindunji kuchokera ku ValueCache.
Pambuyo pokonza deta pogwiritsa ntchito PreProcessing, timayisunga mu HistoryCache kuti tiyikonze. Izi zimathetsa kusonkhanitsa deta ndipo timapita ku njira yayikulu mu Zabbix - mbiri syncer, chifukwa ndi zomangamanga za monolithic.
Chidziwitso: PreProcessing ndi ntchito yovuta kwambiri. Ndi v 4.2 yasunthidwa kukhala proxy. Ngati muli ndi Zabbix yaikulu kwambiri yokhala ndi zinthu zambiri za deta ndi kusonkhanitsa pafupipafupi, ndiye kuti izi zimapangitsa kuti ntchitoyi ikhale yosavuta.
ValueCache, mbiri & mayendedwe posungira
Kuphatikizika kwa mbiri ndiyo njira yayikulu yomwe imasinthira ma atomu chilichonse cha data, ndiye kuti, mtengo uliwonse.
Kuphatikizika kwa mbiri kumatenga zinthu kuchokera ku HistoryCache ndikuyang'ana Kusintha kwa kukhalapo kwa zoyambitsa kuwerengera. Ngati alipo, amawerengera.
Kuphatikizika kwa mbiri kumapanga chochitika, kukwera kuti apange zidziwitso ngati zikufunika ndi kasinthidwe, ndi zolemba. Ngati pali zoyambitsa kukonzanso kotsatira, ndiye kuti zimasunga mtengo uwu mu ValueCache kuti musapeze mbiri yakale. Umu ndi momwe ValueCache imadzazidwa ndi deta yomwe ndiyofunikira kuwerengera zoyambitsa ndi zowerengera.
Mbiri syncer imalemba zonse ku database, ndipo imalemba ku disk. Kukonzekera kumathera apa.
Kusunga mu database
Pambali ya database pali zosungira zosiyanasiyana mukafuna kuwona ma graph kapena malipoti a zochitika:
Innodb_buffer_pool pa MySQL mbali;
shared_buffers pa PostgreSQL mbali;
effective_cache_size kumbali ya Oracle;
shared_pool ku DB2 mbali.
Palinso ma cache ena ambiri, koma awa ndi omwe amafunikira ma database onse. Amakulolani kuti musunge deta mu RAM yomwe imafunikira nthawi zambiri pamafunso. Iwo ali ndi matekinoloje awo pa izi.
Kuchita kwa database ndikofunikira
Seva ya Zabbix nthawi zonse imasonkhanitsa deta ndikulemba. Mukayambiranso, imawerengeranso mbiri yakale kuti mudzaze ValueCache. Amagwiritsa ntchito zolemba ndi malipoti Zabbix API, yomwe imamangidwa pa intaneti. Zabbix API imalowa m'malo osungirako zinthu zakale ndikupeza zofunikira pazithunzi, malipoti, mndandanda wa zochitika ndi nkhani zaposachedwa.
Kwa mawonekedwe - grafana. Iyi ndi njira yotchuka pakati pa ogwiritsa ntchito athu. Ikhoza kutumiza mwachindunji zopempha kudzera mu Zabbix API ndi ku database, ndikupanga mpikisano wina wolandira deta. Chifukwa chake, kukonza bwino komanso koyenera kwa database kumafunika kuti zigwirizane ndi kutulutsa mwachangu kwazotsatira ndi kuyesa.
Wolemba nyumba
Vuto lachitatu mu Zabbix ndikuchotsa mbiri yakale pogwiritsa ntchito Wosunga Nyumba. Imatsatira makonda onse - zinthu za data zikuwonetsa kutalika kosungirako zosinthika (mayendedwe) m'masiku.
Timawerengera TrendsCache pa ntchentche. Deta ikafika, timayiphatikiza kwa ola limodzi ndikuyilemba m'matebulo akusintha kwamayendedwe.
Chithunzi chofiira chikuwonetsa kuti syncer ya Mbiri imakhala yotanganidwa nthawi zonse. Grafu ya lalanje pamwamba ndi Woyang'anira Nyumba, yomwe imayenda mosalekeza. Amadikirira kuti database ichotse mizere yonse yomwe adafotokoza.
Ndi liti pamene muyenera kuletsa Wosunga Nyumba? Mwachitsanzo, pali "Item ID" ndipo muyenera kuchotsa mizere yomaliza 5 zikwi mkati mwa nthawi inayake. Zachidziwikire, izi zimachitika ndi index. Koma nthawi zambiri deta imakhala yayikulu kwambiri, ndipo nkhokwe imawerengedwabe kuchokera ku disk ndikuyiyika mu cache. Izi nthawi zonse zimakhala zodula kwambiri pa database ndipo, kutengera kukula kwa nkhokwe, zitha kubweretsa zovuta.
Woyang'anira nyumba ndi wosavuta kuyimitsa. Mu mawonekedwe a Webusayiti pali makonda mu "Administration general" kwa Woyang'anira Nyumba. Timayimitsa Kusunga Zanyumba kwa mbiri yakale yamkati ndipo sikumayendetsanso.
Woyang'anira nyumba adazimitsidwa, ma graph adatulutsidwa - ndi mavuto ati omwe angakhalepo pankhaniyi ndipo ndi chiyani chomwe chingathandize kuthana ndi vuto lachitatu?
Kugawa - kugawa kapena kugawa
Nthawi zambiri, kugawa kumakonzedwa mwanjira yosiyana pa database iliyonse yaubale yomwe ndalemba. Aliyense ali ndi teknoloji yake, koma ndizofanana kwambiri. Kupanga gawo latsopano nthawi zambiri kumabweretsa zovuta zina.
Nthawi zambiri, magawo amakonzedwa kutengera "kukhazikitsa" - kuchuluka kwa data yomwe imapangidwa tsiku limodzi. Monga lamulo, Kugawa kumaperekedwa tsiku limodzi, izi ndizochepa. Kwa machitidwe a batch yatsopano - 1 mwezi.
Makhalidwe angasinthe ngati "kukhazikitsa" kuli kwakukulu kwambiri. Ngati "kukhazikitsa" kwakung'ono kumafika ku 5 nvps (zatsopano pa sekondi iliyonse), sing'anga imodzi imachokera ku 000 mpaka 5, ndiye kuti yaikulu imakhala pamwamba pa 000 nvps. Awa ndi makhazikitsidwe akulu komanso akulu kwambiri omwe amafunikira kusanja koyenera kwa database.
Pakuyika kwakukulu, nthawi ya tsiku limodzi singakhale yabwino. Ndawona magawo a MySQL a 40 GB kapena kupitilira apo patsiku. Izi ndizochuluka kwambiri zomwe zingayambitse mavuto ndipo ziyenera kuchepetsedwa.
Kodi Partitioning imapereka chiyani?
Matebulo ogawa. Nthawi zambiri awa amakhala osiyana owona pa litayamba. Dongosolo lamafunso limasankha gawo limodzi bwino kwambiri. Nthawi zambiri kugawa kumagwiritsidwa ntchito mosiyanasiyana - izi ndizowonanso za Zabbix. Timagwiritsa ntchito "timestamp" pamenepo - nthawi kuyambira chiyambi cha nthawi. Izi ndi manambala wamba kwa ife. Inu mumayika chiyambi ndi mapeto a tsiku - uku ndi kugawa.
Imafulumizitsa kwambiri kubweza detaSELECT - amagwiritsa ntchito gawo limodzi kapena angapo, osati tebulo lonse. Ngati mukupeza deta yomwe ili ndi masiku awiri, imatengedwa kuchokera ku database mofulumira chifukwa mumangofunika kuyika fayilo imodzi mu cache ndikuyibwezera, osati tebulo lalikulu.
Nthawi zambiri ma database ambiri amafulumizitsanso INSERT - zolowetsa mu tebulo la ana.
Nthawi
Kwa v 4.2, tidatembenukira ku TimescaleDB. Uku ndikuwonjezera kwa PostgreSQL yokhala ndi mawonekedwe achilengedwe. Kuwonjezako kumagwira ntchito bwino ndi data yotsatizana nthawi, osataya phindu lazosunga zolumikizana. TimescaleDB imagawanizanso zokha.
TimescaleDB ili ndi lingaliro hypertable (hypertable) zomwe mumapanga. Lili ndi zidutswa - magawo. Chunks amangoyendetsedwa ndi zidutswa za hypertable zomwe sizikhudza zidutswa zina. Chigawo chilichonse chimakhala ndi nthawi yake.
Ntchitoyi ili ndi magawo atatu. Choyamba - tebulo mu database, zomwe muyenera kupanga hypertable. Chachiwiri - munda, malinga ndi zomwe muyenera kupanga chunk_time_interval - kagawo kakang'ono ka magawo ogawa kuti agwiritsidwe ntchito. Kwa ine, nthawiyo ndi tsiku limodzi - 86.
Gawo lomaliza - UPDATE: at db_extension kuika timescaledbkotero kuti database imvetsetse kuti kukulitsa uku kulipo. Zabbix imayiyambitsa ndikugwiritsa ntchito molondola mawu ndi mafunso ku database - zomwe ndizofunikira pa TimescaleDB.
Kukonzekera kwa Hardware
Ndinagwiritsa ntchito ma seva awiri. Choyamba - Makina a VMware. Ndi yaying'ono kwambiri: 20 Intel® Xeon® CPU E5-2630 v 4 @ 2.20GHz mapurosesa, 16 GB RAM ndi 200 GB SSD.
Ndinayika PostgreSQL 10.8 pa iyo ndi Debian 10.8-1.pgdg90+1 OS ndi xfs file system. Ndidakonza chilichonse kuti ndigwiritse ntchito nkhokweyi, kusiya zomwe Zabbix izigwiritsa ntchito.
Pa makina omwewo panali seva ya Zabbix, PostgreSQL ndi katundu wothandizira. Ndinali ndi othandizira 50 omwe anali kugwiritsa ntchito LoadableModulekutulutsa mwachangu zotsatira zosiyanasiyana: manambala, zingwe. Ndinadzaza nkhokwe ndi deta yambiri.
Poyamba kasinthidwe munali 5 zinthu deta pa host host. Pafupifupi chinthu chilichonse chinali ndi choyambitsa kuti chifanane ndi kukhazikitsa kwenikweni. Nthawi zina panali zoyambitsa zingapo. Kwa node imodzi ya netiweki panali 3-000 zoyambitsa.
Nthawi Yosinthira Zinthu − Masekondi 4 mpaka 7. Ndidayendetsa katunduyo osagwiritsa ntchito othandizira 50 okha, koma ndikuwonjezera zina. Komanso, pogwiritsa ntchito zinthu za data, ndidasinthiratu katunduyo ndikuchepetsa nthawi yosinthira kukhala 4 s.
PostgreSQL. 35 nvps
Kuthamanga kwanga koyamba pa hardware iyi kunali pa PostgreSQL yoyera - 35 zikwi zikwi pamphindi. Monga mukuonera, kuyika deta kumatenga tizigawo ting'onoting'ono - zonse ndi zabwino komanso zachangu. Chokhacho ndi chakuti 200 GB SSD disk imadzaza mwamsanga.
Iyi ndi dashboard yokhazikika ya seva ya Zabbix.
Grafu yoyamba ya buluu ndi nambala yamtengo pa sekondi iliyonse. Grafu yachiwiri kumanja ndikutsitsa njira zomanga. Chachitatu ndikutsegula njira zomangira zamkati: zolumikizira mbiri yakale ndi Woyang'anira Nyumba, zomwe zakhala zikuyenda pano kwa nthawi yayitali.
Chithunzi chachinayi chikuwonetsa kugwiritsa ntchito HistoryCache. Uwu ndi mtundu wa buffer musanalowe mu database. Grafu yachisanu yobiriwira ikuwonetsa kugwiritsidwa ntchito kwa ValueCache, ndiye kuti, ndi zingati za ValueCache zomwe zimagunda zoyambitsa - izi ndizinthu masauzande angapo pamphindikati.
Mukatsitsa kuchokera kwa Woyang'anira Nyumba, kuyika ma 10 zikwizikwi kunatenga masekondi 2-3.
Woyang'anira nyumba wayamba kale kusokoneza ntchito.
Chithunzi chachitatu chikuwonetsa kuti, nthawi zambiri, katundu wa otchera ndi ogwirizanitsa mbiri akadali pa 60%. Mu graph yachinayi, HistoryCache yayamba kale kudzaza mwachangu panthawi ya Ogwira Ntchito Panyumba. Ndi 20% yodzaza, yomwe ili pafupi ndi 0,5 GB.
PostgreSQL. 80 nvps
Kenako ndidawonjezera katunduyo kukhala 80 zikwizikwi pamphindikati. Izi ndi pafupifupi 400 zikwi deta zinthu ndi 280 zikwi zoyambitsa.
Mtengo wotsitsa wa ma synchers a mbiri yakale ndiwokwera kale.
Ndidawonjezeranso magawo osiyanasiyana: zolumikizira mbiri, ma cache.
Pa hardware yanga, kutsitsa kwa ma syncers a mbiriyakale kunakwera kwambiri. HistoryCache idadzaza mwachangu ndi data - zomwe zidakonzedwa zidasonkhanitsidwa mu buffer.
Nthawi yonseyi ndidawona momwe purosesa, RAM ndi magawo ena amagwiritsidwira ntchito, ndipo ndidapeza kuti kugwiritsa ntchito disk kunali pamlingo wake.
Ndapindula ntchito kuthekera kwakukulu kwa disk pa hardware iyi ndi pa makina enieni awa. Ndi mphamvu yotereyi, PostgreSQL idayamba kutsitsa deta mwachangu, ndipo disk inalibenso nthawi yolemba ndi kuwerenga.
Seva yachiwiri
Ndinatenga seva ina, yomwe inali kale ndi mapurosesa a 48 ndi 128 GB ya RAM. Ndidazikonza - ndikuziyika ku syncer ya mbiri yakale, ndikuchita bwino.
Ndipotu, awa ndi kale malire a zokolola zomwe ziyenera kuchitika.
NthawiDB. 80 nvps
Ntchito yanga yayikulu ndikuyesa kuthekera kwa TimescaleDB motsutsana ndi Zabbix katundu. 80 zikwi zamtengo wapatali pamphindikati ndizochuluka, kuchuluka kwa ma metrics (kupatula Yandex, kumene) ndi "kukhazikitsa" kwakukulu.
Pali kuviika mu graph iliyonse - uku ndiko kusamuka kwa data. Pambuyo pakulephera kwa seva ya Zabbix, mbiri yotsitsa ya syncer ya mbiri yakale idasintha kwambiri - idatsika katatu.
Kenaka ndinawonjezera chiwerengero cha zinthu za data ku 500 zikwi. Ntchito yaikulu inali kuyesa mphamvu za TimescaleDB - ndinalandira mtengo wowerengeka wa 125 zikwi zikwi pamphindi.
Ichi ndi "kukhazikitsa" ntchito yomwe ingagwire ntchito kwa nthawi yayitali. Koma popeza disk yanga inali 1,5 TB yokha, ndinaidzaza m'masiku angapo.
Chofunika kwambiri ndikuti nthawi yomweyo magawo atsopano a TimescaleDB adapangidwa.
Izi ndizosadziwikiratu pakuchita. Pamene magawo amapangidwa mu MySQL, mwachitsanzo, chirichonse chimakhala chosiyana. Izi nthawi zambiri zimachitika usiku chifukwa zimalepheretsa kuyika, kugwira ntchito ndi matebulo ndipo zimatha kuyambitsa kuwonongeka kwa ntchito. Izi sizili choncho ndi TimescaleDB.
Mwachitsanzo, ndikuwonetsa graph imodzi kuchokera kwa ambiri ammudzi. Pachithunzichi, TimescaleDB yayatsidwa, chifukwa chomwe katundu wogwiritsa ntchito io.weight pa purosesa watsika. Kugwiritsa ntchito zinthu zamkati zamkati kunachepanso. Komanso, awa ndi makina wamba wamba pa zikondamoyo wamba, osati SSD.
anapezazo
TimescaleDB ndi njira yabwino yothetsera "kukhazikitsa" kwazing'ono, zomwe zimakhudza magwiridwe a disk. Zikuthandizani kuti mupitilize kugwira ntchito bwino mpaka database itasamutsidwa kupita ku hardware mwachangu momwe mungathere.
TimescaleDB ndiyosavuta kukonza, imapereka zopindulitsa, imagwira ntchito bwino ndi Zabbix ndi ili ndi zabwino kuposa PostgreSQL.
Ngati mugwiritsa ntchito PostgreSQL ndipo simukukonzekera kusintha, ndikupangira gwiritsani ntchito PostgreSQL ndi kuwonjezera kwa TimescaleDB molumikizana ndi Zabbix. Njira iyi imagwira ntchito bwino mpaka "kukhazikitsa" sing'anga.
Tikamanena kuti “ntchito zapamwamba” tikutanthauza Kuthamanga Kwambiri ++. Simukhala ndi nthawi yayitali kuti muphunzire zaukadaulo ndi machitidwe omwe amathandizira kuti ntchito zithandizire ogwiritsa ntchito mamiliyoni ambiri. Mndandanda malipoti kwa Novembala 7 ndi 8 tapanga kale, koma apa misonkhano zambiri zitha kuperekedwa.
Lembani ku wathu Kalatayi и uthengawo, m’mene timavumbula mbali za msonkhano umene ukubwerawo, ndi kupeza mmene mungapindulire nawo.