Sizobheka ukuthi i-Zabbix isebenza kanjani ne-TimescaleDB database njenge-backend. Sizokukhombisa ukuthi ungaqala kanjani ekuqaleni nokuthi ungafuduka kanjani usuka ku-PostgreSQL. Futhi sizohlinzeka ngokuhlolwa kokuqhathanisa kokusebenza kokucushwa okubili.
I-HighLoad++ Siberia 2019. Ihholo laseTomsk. Juni 24, 16:00. Lezi kanye . Ingqungquthela elandelayo ye-HighLoad++ izoba ngo-April 6 no-7, 2020 e-St. Petersburg. Imininingwane namathikithi .
U-Andrey Gushchin (okulandelayo – AG): – Ngingunjiniyela wosekelo lwe-ZABBIX (okuzobizwa kamuva ngokuthi “Zabbix”), umqeqeshi. Ngisebenze ekusekelweni kwezobuchwepheshe iminyaka engaphezu kwe-6 futhi ngibe nolwazi oluqondile ngokusebenza. Namuhla ngizokhuluma ngokusebenza okungahlinzekwa yi-TimescaleDB uma kuqhathaniswa ne-PostgreSQL 10 evamile. Futhi, ingxenye ethile yesethulo mayelana nendlela esebenza ngayo ngokujwayelekile.
Izinselelo eziphezulu zokukhiqiza: kusuka ekuqoqweni kwedatha kuya ekuhlanzweni kwedatha
Okokuqala, kunezinselelo ezithile zokusebenza zonke izinhlelo zokuqapha ezibhekana nazo. Inselele yokuqala yokukhiqiza ukuqoqa nokucubungula idatha ngokushesha.
Uhlelo oluhle lokuqapha kufanele ngokushesha, ngesikhathi esifanele lwamukele yonke idatha, luyicubungule ngokwendlela yokuqalisa, okungukuthi, luyicubungule ngokuya ngemibandela ethile (lokhu kwehlukile ezinhlelweni ezihlukene) futhi luyigcine kusizindalwazi ukuze kusetshenziswe le datha ku esizayo.
Inselele yesibili yokusebenza isitoreji somlando. Gcina kusizindalwazi njalo futhi ube nokufinyelela okusheshayo nokulula kulawa mamethrikhi aqoqwe ngesikhathi esithile. Into ebaluleke kakhulu ukuthi le datha ilula ukuyithola, iyisebenzise emibikweni, emagrafu, ekucupheni, kwamanye amanani e-threshold, izexwayiso, njll.
Inselele yesithathu yokusebenza ukusula umlando, okungukuthi, uma ufika endaweni lapho ungadingi ukugcina noma yimaphi amamethrikhi anemininingwane aqoqwe phakathi neminyaka emi-5 (ngisho nezinyanga noma izinyanga ezimbili). Amanye amanodi enethiwekhi asusiwe, noma abanye abasingathi, amamethrikhi awasadingeki ngenxa yokuthi asephelelwe isikhathi futhi awasaqoqwa. Konke lokhu kudinga ukuhlanzwa ukuze isizindalwazi sakho singakhuli sibe sikhulu kakhulu. Ngokuvamile, ukususa umlando ngokuvamile kuwukuhlola okubucayi kwesitoreji - ngokuvamile kuba nomthelela onamandla kakhulu ekusebenzeni.
Indlela yokuxazulula izinkinga ze-caching?
Manje ngizokhuluma ngokuqondile nge-Zabbix. Ku-Zabbix, izingcingo zokuqala nezesibili zixazululwa kusetshenziswa ukulondoloza isikhashana.
Ukuqoqwa nokucubungula idatha - Sisebenzisa i-RAM ukuze sigcine yonke le datha. Le datha manje kuzoxoxwa ngayo kabanzi.
Futhi ngasohlangothini lwedathabheyisi kukhona ukugcinwa kwesikhashana kokukhetha okuyinhloko - kwamagrafu nezinye izinto.
Ukulondoloza isikhashana eceleni kweseva ye-Zabbix ngokwayo: sine-ConfigurationCache, ValueCache, HistoryCache, TrendsCache. Yini?
I-ConfigurationCache iyinqolobane eyinhloko lapho sigcina khona amamethrikhi, abasingathi, izinto zedatha, izibangeli; yonke into oyidingayo ukuze ucubungule ukucubungula ngaphambilini, ukuqoqa idatha, okuzoqoqwa kubo abasingathi, ngasiphi imvamisa. Konke lokhu kugcinwa ku-ConfigurationCache ukuze ungayi ku-database futhi udale imibuzo engadingekile. Ngemuva kokuthi iseva iqalile, sibuyekeza le cache (siyidale) futhi siyibuyekeze ngezikhathi ezithile (kuye ngokuthi izilungiselelo zokucushwa).
Igcina isikhashana ku-Zabbix. Ukuqoqwa kwedatha
Lapha umdwebo mkhulu impela:
Okubalulekile ohlelweni yilaba baqoqi:
Lezi yizinqubo zokuhlangana ngokwazo, "abavoti" abahlukahlukene ababhekele izinhlobo ezahlukene zemihlangano. Baqoqa idatha nge-icmp, ipmi, nezivumelwano ezihlukahlukene futhi badlulisele konke ekucutshungulweni kwangaphambili.
Icubungula NgaphambiliInqolobane Yomlando
Futhi, uma sibale ama-elementi edatha (labo abajwayelene ne-Zabbix bayazi), okungukuthi, ama-elementi edatha abaliwe, ahlanganisayo, sizithatha ngokuqondile ku-ValueCache. Ngizokutshela ukuthi igcwaliswa kanjani kamuva. Bonke laba baqoqi basebenzisa i-ConfigurationCache ukuze bathole imisebenzi yabo bese beyidlulisela ekuyicubunguleni ngaphambili.
Ukucubungula ngaphambilini kusebenzisa i-ConfigurationCache ukuze kutholwe izinyathelo zokucubungula kusengaphambili futhi kucubungule le datha ngezindlela ezihlukahlukene. Kusukela kunguqulo 4.2, siyihambise kummeleli. Lokhu kulula kakhulu, ngoba ukucubungula kuqala kuwumsebenzi onzima kakhulu. Futhi uma une-Zabbix enkulu kakhulu, enenani elikhulu lezinto zedatha kanye nemvamisa yeqoqo eliphezulu, khona-ke lokhu kwenza umsebenzi ube lula kakhulu.
Ngokufanelekile, ngemva kokuba sesicubungule le datha ngandlela thile sisebenzisa ukucubungula ngaphambilini, siyilondoloza ku-HistoryCache ukuze siyicubungule ngokuqhubekayo. Lokhu kuphetha ukuqoqwa kwedatha. Siqhubekela enqubweni eyinhloko.
Umsebenzi wokuvumelanisa umlando
Inqubo eyinhloko ku-Zabbix (njengoba iwukwakhiwa kwe-monolithic) i-History syncer. Lena inqubo eyinhloko ephathelene ngokuqondile nokucutshungulwa kwe-athomu yento ngayinye yedatha, okungukuthi, inani ngalinye:
- inani liza (lithatha ku-HistoryCache);
- ihlola Ukuvumelanisa Ukucushwa: noma ngabe zikhona yini izibangeli zokubala - ibala;
uma kukhona - kudala imicimbi, kudala ukukhuphuka ukuze kudale isixwayiso, uma kunesidingo ngokuya ngokucushwa; - amarekhodi okubangela ukucubungula okulandelayo, ukuhlanganisa; uma uhlanganisa ihora lokugcina njalo njalo, leli nani likhunjulwa yi-ValueCache ukuze ungayi etafuleni lomlando; Ngakho, i-ValueCache igcwaliswa ngedatha edingekayo edingekayo ukuze kubalwe izibangeli, izakhi ezibaliwe, njll.;
- bese kuba Umhlaziyi Womlando ubhala yonke idatha kusizindalwazi;
- i-database iwabhala kudiski - yilapho inqubo yokucubungula iphela khona.
Isizindalwazi. Ukulondoloza inqolobane
Ohlangothini lwesizindalwazi, uma ufuna ukubuka amagrafu noma eminye imibiko ngemicimbi, kunezinqolobane ezahlukahlukene. Kodwa kulo mbiko ngeke ngikhulume ngawo.
Ku-MySQL kukhona i-Innodb_buffer_pool, kanye nenqwaba yama-cache ahlukene nawo angalungiselelwa.
Kodwa lezi yizona eziyinhloko:
- amabhafa_abiwe;
- usayizi_wenqolobane_osebenzayo;
- iphuli_ehlanganyelwe.
Kuwo wonke ama-database, ngithe kukhona izinqolobane ezithile ezikuvumela ukuthi ugcine ku-RAM idatha evame ukudingeka emibuzweni. Banobuchwepheshe babo balokhu.
Mayelana Nokusebenza Kwesizindalwazi
Ngakho-ke, kunendawo yokuncintisana, okungukuthi, iseva ye-Zabbix iqoqa idatha futhi iyirekhode. Uma iqalwa kabusha, iphinda ifunde emlandweni ukuze igcwalise i-ValueCache nokunye. Lapha ungaba nemibhalo nemibiko esebenzisa i-Zabbix API, eyakhelwe ku-interface yewebhu. I-Zabbix API ingena kusizindalwazi bese ithola idatha edingekayo ukuze ithole amagrafu, imibiko, noma uhlobo oluthile lohlu lwezenzakalo, izinkinga zakamuva.
Futhi isixazululo esidume kakhulu sokubuka i-Grafana, esisetshenziswa abasebenzisi bethu. Iyakwazi ukungena ngokuqondile kokubili nge-Zabbix API kanye nesizindalwazi. Kuphinde kwakha umncintiswano othile wokuthola idatha: ukushuna okungcono, okungcono kwesizindalwazi kuyadingeka ukuze kuhambisane nokulethwa okusheshayo kwemiphumela nokuhlolwa.
Isula umlando. U-Zabbix unoMgcini Wendlu
Ucingo lwesithathu olusetshenziswa ku-Zabbix ukusula umlando kusetshenziswa u-Housekeeper. Umgcini wasendlini ulandela zonke izilungiselelo, okungukuthi, ama-elementi ethu edatha akhombisa ukuthi kumele kugcinwe isikhathi esingakanani (ngezinsuku), isikhathi esingakanani ukugcina amathrendi, kanye namandla oshintsho.
Angizange ngikhulume nge-TrendCache, esiyibala ngokuhamba kwesikhathi: idatha iyafika, siyihlanganisa ihora elilodwa (ikakhulukazi lezi izinombolo zehora lokugcina), inani liyisilinganiso/lincane futhi siliqopha kanye ngehora ithebula le-dynamics yezinguquko (“Amathrendi”) . "Umgcini wendlu" uqala futhi asuse idatha kusizindalwazi esebenzisa ukukhetha okuvamile, okungaphumeleli ngaso sonke isikhathi.
Indlela yokuqonda ukuthi ayisebenzi? Ungabona isithombe esilandelayo kumagrafu okusebenza ezinqubo zangaphakathi:
Isivumelanisi somlando wakho sihlala simatasa (igrafu ebomvu). Futhi igrafu "ebomvu" ehamba phezulu. Lona "Unomuzi" oqala futhi alinde ukuthi isizindalwazi sisuse yonke imigqa esiyishilo.
Ake sithathe i-ID Yento ethile: udinga ukususa izinkulungwane ezi-5 zokugcina; Yebo, ngezinkomba. Kodwa ngokuvamile isethi yedatha inkulu impela - i-database isayifunda isuka kudiski futhi iyifaka kunqolobane, futhi lokhu kuwumsebenzi obiza kakhulu kusizindalwazi. Ngokuya ngosayizi wayo, lokhu kungaholela ezinkingeni ezithile zokusebenza.
Ungakwazi ukukhubaza umgcini wendlu ngendlela elula - sinokuxhumana okujwayelekile kwewebhu. Izilungiselelo Ezijwayelekile Zokuphatha (izilungiselelo zokuthi “Umninindlu”) sikhubaza ukugcinwa kwendlu kwangaphakathi kumlando wangaphakathi namathrendi. Ngakho-ke, u-Housekeeper akasakwazi ukulawula lokhu:
Yini ongayenza ngokulandelayo? Uyicimile, amagrafu akho asemile... Yiziphi ezinye izinkinga ezingavela kulesi simo? Yini engasiza?
Ukwahlukanisa (izigaba)
Ngokuvamile lokhu kulungiselelwa ngendlela ehlukile kusizindalwazi ngasinye sobudlelwano engisibalile. I-MySQL inobuchwepheshe bayo. Kodwa sezizonke ziyefana kakhulu uma kukhulunywa nge-PostgreSQL 10 kanye ne-MySQL. Kunjalo, kunomehluko omningi wangaphakathi kokuthi konke kwenziwa kanjani nokuthi konke kukuthinta kanjani ukusebenza. Kodwa ngokuvamile, ukwakhiwa kokwahlukanisa okusha kuvame ukuholela ezinkingeni ezithile.
Ngokuya ngokusetha kwakho (ingakanani idatha oyidalayo ngosuku olulodwa), ngokuvamile babeka ubuncane - lolu usuku olu-1 / inqwaba, futhi "okuthrendayo", amandla okushintsha - inyanga engu-1 / inqwaba entsha. Lokhu kungase kushintshe uma unokusethwa okukhulu kakhulu.
Ake sithi ngokushesha mayelana nosayizi wokusetha: kufika kumanani amasha ayizinkulungwane ezingu-5 ngomzuzwana (okuthiwa i-nvps) - lokhu kuzobhekwa "njengokusetha" okuncane. Isilinganiso - kusuka kumanani ayizinkulungwane ezi-5 kuye kwezingama-25 ngomzuzwana. Konke okungenhla sekuvele kukhulu futhi kukhulu kakhulu ukufakwa okudinga ukumiswa ngokucophelela kwesizindalwazi.
Ekufakeni okukhulu kakhulu, usuku olungu-1 lungase lungabi lulungile. Mina ngokwami ngizibonile izingxenye ze-MySQL zamagigabhayithi angu-40 ngosuku (futhi kungase kube nokunye). Leli inani elikhulu kakhulu ledatha, elingaholela ezinkingeni ezithile. Idinga ukuncishiswa.
Kungani udinga ukwahlukanisa?
Lokho okuhlinzekwa yi-Partitioning, ngicabanga ukuthi wonke umuntu uyazi, ukwahlukanisa itafula. Ngokuvamile lawa amafayela ahlukene kudiski kanye nezicelo ze-span. Ikhetha ingxenye eyodwa ngokufanele kakhulu uma iyingxenye yokuhlukanisa okuvamile.
Ku-Zabbix, ikakhulukazi, isetshenziswa ububanzi, ngobubanzi, okungukuthi, sisebenzisa isitembu sesikhathi (inombolo evamile, isikhathi kusukela ekuqaleni kwenkathi). Ucacisa ukuqala kosuku/ukuphela kosuku, futhi lokhu ukwahlukanisa. Ngakho-ke, uma ucela idatha enezinsuku ezimbili ubudala, yonke into ibuyiselwa ku-database ngokushesha, ngoba udinga kuphela ukulayisha ifayela elilodwa kunqolobane bese uyibuyisela (kunokuba itafula elikhulu).
Imininingo egciniwe eminingi ibuye isheshise ukufakwa (ukufaka etafuleni elilodwa lengane). Ngikhuluma ngokusobala okwamanje, kodwa nalokhu kungenzeka. Ukuhlukanisa ngokuvamile kuyasiza.
I-Elasticsearch ye-NoSQL
Muva nje, ngo-3.4, sisebenzise isisombululo se-NoSQL. Kwengezwe ikhono lokubhala ku-Elasticsearch. Ungabhala izinhlobo ezithile: ukhetha - noma bhala izinombolo noma ezinye izimpawu; sinombhalo weyunithi yezinhlamvu, ungabhala amalogi ku-Elasticsearch... Ngokufanelekile, isixhumi esibonakalayo sewebhu sizophinda sifinyelele i-Elasticsearch. Lokhu kusebenza kahle kwezinye izimo, kodwa okwamanje kungasetshenziswa.
I-TimescaleDB. Ama-hypertables
Ku-4.4.2 sinake into eyodwa efana ne-TimescaleDB. Yini? Lesi isandiso se-PostgreSQL, okungukuthi, ine-interface yendabuko ye-PostgreSQL. Futhi, lesi sandiso sikuvumela ukuthi usebenze ngokuphumelelayo kakhulu ngedatha ye-timeseries futhi ube nokuhlukanisa okuzenzakalelayo. Indlela ebukeka ngayo:
Lokhu ku-hypertable - kunomqondo onjalo ku-Timescale. Lesi yi-hypertable oyidalayo, futhi iqukethe izingcezu. Ama-Chunks angama-partitions, lawa amatafula ezingane, uma ngingaphosisi. Iyasebenza ngempela.
I-TimescaleDB ne-PostgreSQL
Njengoba abakhiqizi be-TimescaleDB beqinisekisa, basebenzisa i-algorithm elungile kakhudlwana ukuze bacubungule imibuzo, ikakhulukazi okufakiwe, okubavumela ukuthi babe nokusebenza cishe okungaguquki ngosayizi okhulayo wokufakwa kwedathasethi. Okusho ukuthi, ngemva kwemigqa eyizigidi ezingu-200 ye-Postgres, evamile iqala ukuwohloka kakhulu futhi ilahlekelwe ukusebenza ngokoqobo ibe nguziro, kuyilapho i-Timescale ikuvumela ukuthi ufake okufakwayo ngempumelelo ngangokunokwenzeka nganoma yiliphi inani ledatha.
Ungayifaka kanjani i-TimescaleDB? Kulula!
Ikumadokhumenti, ichaziwe - ungayifaka kusuka kumaphakeji wanoma yikuphi... Kuncike kumaphakheji asemthethweni we-Postgres. Ingahlanganiswa ngesandla. Kwenzeka ukuthi kufanele ngihlanganisele database.
Ku-Zabbix simane senze kusebenze Isandiso. Ngicabanga ukuthi labo abasebenzisa Isandiso kuma-Postgres... Umane wenze kusebenze Isandiso, usidale kusizindalwazi se-Zabbix osisebenzisayo.
Futhi isinyathelo sokugcina...
I-TimescaleDB. Ukufuduka kwamathebula omlando
Udinga ukudala i-hypertable. Kunomsebenzi okhethekile walokhu - Dala i-hypertable. Kuyo, ipharamitha yokuqala itafula elidingekayo kule database (odinga ukudala i-hypertable).
Inkambu okufanele kudalwe ngayo, kanye ne-chunk_time_interval (lesi isikhawu sezingcezu (izingxenye ezidinga ukusetshenziswa). 86 usuku olulodwa.
Migrate_data parameter: Uma ufaka kuqiniso, lokhu kuzothutha yonke idatha yamanje iziqephu ezidalwe kusengaphambili.
Ngisebenzise i-migrate_data ngokwami - kuthatha isikhathi esifanelekile, kuye ngokuthi inkulu kangakanani isizindalwazi sakho. Ngibe ne-terabyte engaphezu kwe-terabyte - kuthathe isikhathi esingaphezu kwehora ukuyidala. Kwezinye izimo, phakathi nokuhlolwa, ngisuse idatha yomlando yombhalo (history_text) kanye neyunithi yezinhlamvu (history_str) ukuze ngingazidluliseli - bezingenantshisekelo ngempela kimi.
Futhi senza isibuyekezo sokugcina ku-db_extention yethu: sifaka i-timescaledb ukuze isizindalwazi futhi, ikakhulukazi, i-Zabbix yethu iqonde ukuthi kukhona i-db_extention. Uyenza isebenze futhi asebenzise i-syntax efanele kanye nemibuzo kusizindalwazi, esebenzisa lezo “zici” ezidingekayo ku-TimescaleDB.
Ukucushwa kweseva
Ngisebenzise amaseva amabili. Iseva yokuqala ingumshini omncane obonakalayo, amaprosesa angama-20, amagigabhayithi ayi-16 e-RAM. Ngilungiselele i-Postgres 10.8 kuyo:
Isistimu yokusebenza yayiyi-Debian, isistimu yefayela kwakuyi-xfs. Ngenze izilungiselelo ezincane ukusebenzisa le database ethile, ngisuse lokho i-Zabbix ngokwayo ezokusebenzisa. Emshinini ofanayo kwakukhona iseva ye-Zabbix, i-PostgreSQL nama-agent okulayisha.
Ngisebenzise ama-ejenti asebenzayo angu-50 asebenzisa i-LoadableModule ukuze ngikhiqize ngokushesha imiphumela ehlukene. Yibo abakhiqiza izintambo, izinombolo, nokunye. Ngigcwalise isizindalwazi ngedatha eningi. Ekuqaleni, ukulungiselelwa kwakuqukethe izici zedatha eziyizinkulungwane ezingu-5 umsingathi ngamunye, futhi cishe ingxenye ngayinye yedatha ibiqukethe isibangeli - ukuze lokhu kube ukusetha kwangempela. Ngezinye izikhathi udinga ngisho ne-trigger engaphezu kweyodwa ongayisebenzisa.
Ngilawule isikhawu sokubuyekeza kanye nomthwalo ngokwawo ngokungasebenzisi ama-ejenti angu-50 kuphela (ngengeza abengeziwe), kodwa futhi ngokusebenzisa ama-elementi edatha ashukumisayo kanye nokunciphisa isikhawu sokubuyekeza sibe amasekhondi angu-4.
Ukuhlolwa kokusebenza. I-PostgreSQL: Izinkulungwane ezingama-36 ze-NVP
Ukwethulwa kokuqala, ukusetha kokuqala enganginakho kwakuku-PostreSQL 10 ehlanzekile kule hardware (amanani ayizinkulungwane ezingama-35 ngomzuzwana). Ngokuvamile, njengoba ubona esikrinini, ukufaka idatha kuthatha izingxenyana zomzuzwana - konke kuhle futhi kuyashesha, ama-SSD ashayela (200 gigabytes). Okuwukuphela kwento ukuthi i-20 GB igcwala ngokushesha okukhulu.
Azoba maningi impela amagrafu anjalo esikhathini esizayo. Lena ideshibhodi evamile yokusebenza kweseva ye-Zabbix.
Igrafu yokuqala inani lamanani ngomzuzwana (oluhlaza okwesibhakabhaka, phezulu kwesokunxele), amanani ayizinkulungwane ezingama-35 kuleli cala. Lesi (isikhungo esiphezulu) ukulayishwa kwezinqubo zokwakha, futhi lokhu (phezulu kwesokudla) ukulayishwa kwezinqubo zangaphakathi: ama-syncers omlando nomgcini wendlu, osekunesikhathi eside esebenza lapha (ngezansi).
Le grafu (isikhungo esingezansi) ibonisa ukusetshenziswa kwe-ValueCache - mangaki amahithi e-ValueCache ezibangeli (amanani ayizinkulungwane ezimbalwa ngomzuzwana). Enye igrafu ebalulekile ingeyesine (ngezansi kwesokunxele), ebonisa ukusetshenziswa kwe-HistoryCache, engikhulume ngayo, okuyi-buffer ngaphambi kokufaka ku-database.
Ukuhlolwa kokusebenza. I-PostgreSQL: Izinkulungwane ezingama-50 ze-NVP
Okulandelayo, nganyusa umthwalo waba amanani ayizinkulungwane ezingama-50 ngomzuzwana kuhadiwe efanayo. Lapho ilayishwa ngu-Housekeeper, amanani ayizinkulungwane eziyi-10 aqoshwa ngemizuzwana emi-2-3 ngesibalo. Yini, empeleni, eboniswa esithombeni-skrini esilandelayo:
"Umgcini wezindlu" usevele eqala ukuphazamisa umsebenzi, kodwa ngokuvamile, umthwalo kubacuphi bomlando-sinki usesezingeni lika-60% (igrafu yesithathu, kwesokudla phezulu). I-HistoryCache isivele iqala ukugcwalisa ngokusebenzayo ngenkathi u-Housekeeper esebenza (ngezansi kwesokunxele). Cishe yayiyingxenye yegigabhayithi, igcwele ngo-20%.
Ukuhlolwa kokusebenza. I-PostgreSQL: Izinkulungwane ezingama-80 ze-NVP
Ngabe sengiyinyusa yaba ngamanani ayizinkulungwane ezingama-80 ngomzuzwana:
Kwakuyizici zedatha ezingaba yizinkulungwane ezingama-400, izibangeli eziyizinkulungwane ezingama-280. Okufakiwe, njengoba ubona, ngokomthwalo wamasinki omlando (kwakukhona angama-30 awo) kwase kuphezulu impela. Ngabe sengikhulisa amapharamitha ahlukahlukene: amasinki omlando, i-cache... Kule hardware, umthwalo kumasinki omlando waqala ukwanda wafinyelela phezulu, cishe “eshalofini” - ngokufanele, i-HistoryCache ingene emthwalweni omkhulu kakhulu:
Ngaso sonke lesi sikhathi ngiqaphe yonke imingcele yesistimu (indlela iprosesa esetshenziswa ngayo, i-RAM) futhi ngathola ukuthi ukusetshenziswa kwediski kwakuphezulu - ngithole umthamo omkhulu wale diski kule hardware, kulo mshini obonakalayo. "Ama-Postgres" aqala ukulahla idatha ngamandla anjalo, futhi idiski yayingasenaso isikhathi sokubhala, ukufunda ...
Ngithathe enye iseva eyayivele inamaphrosesa angu-48 namagigabhayithi angu-128 e-RAM:
Ngiphinde “ngayishuna” - ngifake isivumelanisi somlando (izingcezu ezingama-60) futhi ngathola ukusebenza okwamukelekayo. Eqinisweni, asikho "eshalofini," kodwa lokhu cishe umkhawulo wokukhiqiza, lapho sekudingekile kakade ukwenza okuthile ngakho.
Ukuhlolwa kokusebenza. I-TimescaleDB: Izinkulungwane ezingama-80 ze-NVP
Umsebenzi wami omkhulu bekuwukusebenzisa i-TimescaleDB. Igrafu ngayinye ibonisa ukucwiliswa:
Lokhu kwehluleka kuwukuthuthwa kwedatha ngokuqondile. Ngemuva kwalokho, kuseva ye-Zabbix, iphrofayili yokulayisha yama-sinkers omlando, njengoba ubona, yashintsha kakhulu. Ikuvumela ukuthi ufake idatha ngokushesha izikhathi ezi-3 futhi usebenzise i-HistoryCache encane - ngokufanele, uzolethwa idatha ngesikhathi. Futhi, amanani ayizinkulungwane ezingama-80 ngomzuzwana yinani eliphakeme kakhulu (yebo, hhayi le-Yandex). Sekukonke lokhu ukusethwa okukhulu, okuneseva eyodwa.
Ukuhlolwa kokusebenza kwe-PostgreSQL: ama-NVP ayizinkulungwane eziyi-120
Okulandelayo, ngakhuphula inani lenani lezinto zedatha laya kungxenye yesigidi futhi ngathola inani elibaliwe lezinkulungwane eziyi-125 ngomzuzwana:
Futhi ngithole lawa magrafu:
Eqinisweni, lokhu kuyisethaphu yokusebenza, ingasebenza isikhathi eside impela. Kodwa njengoba nganginediski engu-1,5 terabyte kuphela, ngayisebenzisa ezinsukwini ezimbalwa. Okubaluleke kakhulu ukuthi ngesikhathi esifanayo ukuhlukaniswa okusha kudalwe ku-TimescaleDB, futhi lokhu kwakungabonakali ngokuphelele ekusebenzeni, okungenakushiwo nge-MySQL.
Ngokuvamile, ama-partitions adalwa ebusuku, ngoba lokhu ngokuvamile kuvimbela ukufakwa nokusebenza ngamatafula futhi kungaholela ekulimazeni isevisi. Kulokhu akunjalo! Umsebenzi omkhulu bekuwukuhlola amandla e-TimescaleDB. Umphumela waba inombolo elandelayo: 120 ayizinkulungwane amanani ngomzuzwana.
Kukhona nezibonelo emphakathini:
Umuntu uphinde wavula i-TimescaleDB futhi umthwalo wokusebenzisa i-io.weight wehla kuphrosesa; kanye nokusetshenziswa kwezakhi zenqubo yangaphakathi nakho kwehlile ngenxa yokufakwa kwe-TimescaleDB. Ngaphezu kwalokho, lawa ama-pancake disks ajwayelekile, okungukuthi, umshini ojwayelekile ojwayelekile kumadiski ajwayelekile (hhayi ama-SSD)!
Kokunye ukusetha okuncane okukhawulelwe ukusebenza kwediski, i-TimescaleDB, ngombono wami, iyisixazululo esihle kakhulu. Izokuvumela ukuthi uqhubeke nokusebenza ngaphambi kokuthuthela ku-hardware esheshayo yesizindalwazi.
Nginimema nonke emicimbini yethu: Ingqungquthela eMoscow, iNgqungquthela eRiga. Sebenzisa iziteshi zethu - iTelegram, inkundla, i-IRC. Uma unemibuzo, woza etafuleni lethu, singakhuluma ngakho konke.
Imibuzo Yezithameli
Umbuzo ovela ezithamelini (ngemuva kwalokhu - A): - Uma i-TimescaleDB ilula kakhulu ukuyilungisa, futhi inikeza ukuthuthukiswa okunjalo kokusebenza, mhlawumbe lokhu kufanele kusetshenziswe njengendlela engcono kakhulu yokumisa i-Zabbix nge-Postgres? Futhi ingabe zikhona izingibe kanye nokubi kwalesi sixazululo, noma ngemva kwakho konke, uma nginqume ukuzenzela i-Zabbix, ngingathatha kalula i-Postgres, ngifake i-Timescale lapho ngaso leso sikhathi, ngiyisebenzise futhi ngingacabangi nganoma yiziphi izinkinga?
AG: - Yebo, ngingasho ukuthi lesi isincomo esihle: sebenzisa i-Postgres ngokushesha ngesandiso se-TimescaleDB. Njengoba ngike ngasho, ukubuyekezwa okuningi okuhle, naphezu kokuthi lesi "sici" siwukuhlola. Kepha empeleni ukuhlolwa kukhombisa ukuthi lesi yisixazululo esihle (nge-TimescaleDB) futhi ngicabanga ukuthi sizovela! Siqapha ukuthi lesi sandiso sithuthuka kanjani futhi sizokwenza izinguquko njengoba kudingeka.
Ngisho nangesikhathi sokuthuthukiswa, sithembele kwesinye se "zici" zabo ezaziwa kakhulu: kwakungenzeka ukusebenza ngezingcezu kancane ngokuhlukile. Kodwa babe sebeyinqamula ekukhululweni okulandelayo, futhi kwadingeka siyeke ukuthembela kule khodi. Ngingancoma ukusebenzisa lesi sixazululo ekusetheni okuningi. Uma usebenzisa i-MySQL... Ngokusetha okumaphakathi, noma yisiphi isisombululo sisebenza kahle.
A: - Emagrafu okugcina avela emphakathini, bekukhona igrafu ene-"Housekeeper":
Waqhubeka nokusebenza. Wenzani u-Housekeeper nge-TimescaleDB?
AG: - Manje angikwazi ukusho ngokuqinisekile - ngizobheka ikhodi futhi ngikutshele ngokuningiliziwe. Isebenzisa imibuzo ye-TimescaleDB ukuze ingasusi izingxenye, kodwa ngandlela thize ukuze ihlanganise. Angikakalungeli ukuphendula lo mbuzo wezobuchwepheshe. Sizothola okuningi esitendini namuhla noma kusasa.
A: - Nginombuzo ofanayo - mayelana nokusebenza komsebenzi wokususa ku-Timescale.
A (impendulo evela ezithamelini): – Lapho ususa idatha etafuleni, uma uyenza ngokususa, khona-ke udinga ukudlula etafuleni - susa, hlanza, umaka yonke into ukuze ungabi namuntu esikhathini esizayo. Nge-Timescale, njengoba unama-chunks, ungawa. Uma sikhuluma nje, uvele utshele ifayela elikwidatha enkulu ukuthi: “Susa!”
I-Timescale imane iqonde ukuthi i-chunk enjalo ayisekho. Futhi njengoba ihlanganiswe nomhleli wemibuzo, isebenzisa izingwegwe ukuze ibambe izimo zakho lapho ukhetha noma eminye imisebenzi futhi iyaqonda ngokushesha ukuthi lesi siqephu asisekho - "Ngeke ngisaya lapho!" (idatha ayitholakali). Yilokho kuphela! Okusho ukuthi, ukuskena kwetafula kuthathelwa indawo ukususwa kwefayela kanambambili, ngakho kuyashesha.
A: - Sesike sathinta isihloko sokungeyona i-SQL. Ngokwazi kwami, i-Zabbix ayidingi ngempela ukuguqula idatha, futhi konke lokhu kufana nelogi. Kungenzeka yini ukusebenzisa i-database ekhethekile engakwazi ukushintsha idatha yabo, kodwa ngesikhathi esifanayo ulondoloze, uqongelele, futhi usabalalise ngokushesha okukhulu - Clickhouse, isibonelo, into efana ne-Kafka?.. I-Kafka nayo iyilogi! Kungenzeka ngandlela thize ukuzihlanganisa?
AG: - Ukukhipha kungenziwa. "Sinesici" esithile kusukela kunguqulo 3.4: ungabhala wonke amafayela omlando, imicimbi, yonke into kumafayela; bese uyithumela kunoma iyiphi enye isizindalwazi usebenzisa isibambi esithile. Eqinisweni, abantu abaningi basebenza kabusha futhi babhale ngqo kusizindalwazi. Ophapheme, abacwili bomlando babhala konke lokhu kumafayela, bazungezisa lawa mafayela, njalonjalo, futhi ungadlulisela lokhu ku-Clickhouse. Angikwazi ukusho mayelana nezinhlelo, kodwa mhlawumbe ukusekelwa okwengeziwe kwezixazululo ze-NoSQL (ezifana ne-Clickhouse) kuzoqhubeka.
A: - Ngokuvamile, kuvela ukuthi ungakwazi ukuqeda ngokuphelele ama-postgres?
AG: - Vele, ingxenye enzima kakhulu ku-Zabbix amatafula omlando, adala izinkinga eziningi, nemicimbi. Kulokhu, uma ungagcini imicimbi isikhathi eside futhi ugcine umlando ngezitayela kwezinye isitoreji esisheshayo, khona-ke ngokuvamile, ngicabanga, ngeke kube nezinkinga.
A: - Ungakwazi yini ukulinganisa ukuthi konke kuzosebenza ngokushesha kangakanani uma ushintshela ku-Clickhouse, isibonelo?
AG: - Angizange ngiyihlole. Ngicabanga ukuthi okungenani izinombolo ezifanayo zingafinyelelwa kalula, njengoba i-Clickhouse ine-interface yayo, kodwa angikwazi ukusho ngokuqinisekile. Kungcono ukuhlola. Konke kuncike ekucushweni: bangaki abasingathi onawo, njalonjalo. Ukufaka kuyinto eyodwa, kodwa futhi udinga ukubuyisa le datha - i-Grafana noma enye into.
A: - Ngakho-ke sikhuluma ngokulwa okulinganayo, hhayi ngenzuzo enkulu yalezi zingosi zolwazi ezisheshayo?
AG: - Ngicabanga ukuthi uma sihlanganisa, kuzoba nezivivinyo ezinembe kakhudlwana.
A: - Yashonaphi i-RRD endala? Yini ekwenze washintshela kusizindalwazi se-SQL? Ekuqaleni, wonke ama-metrics aqoqwe ku-RRD.
AG: - I-Zabbix yayine-RRD, mhlawumbe enguqulweni yakudala kakhulu. Bekuhlala kunemininingwane yolwazi ye-SQL - indlela yakudala. Indlela yakudala iMySQL, PostgreSQL (senesikhathi eside ikhona). Cishe asikaze sisebenzise isixhumi esibonakalayo esivamile kusizindalwazi se-SQL ne-RRD.
Ezinye izikhangiso 🙂
Siyabonga ngokuhlala nathi. Uyazithanda izindatshana zethu? Ufuna ukubona okuqukethwe okuthakaselayo okwengeziwe? Sisekele ngokufaka i-oda noma ngokuncoma kubangani, , i-analogue ehlukile yamaseva ezinga lokungena, esungulwe yithi ngenxa yakho: (itholakala nge-RAID1 kanye ne-RAID10, kufika kuma-cores angu-24 kuze kufike ku-40GB DDR4).
I-Dell R730xd 2x ishibhile esikhungweni sedatha se-Equinix Tier IV e-Amsterdam? Lapha kuphela eNetherlands! I-Dell R420 - 2x E5-2430 2.2Ghz 6C 128GB DDR3 2x960GB SSD 1Gbps 100TB - isuka ku-$99! Funda mayelana
Source: www.habr.com