Zabbix inzira yekutarisa. Kufanana nechero imwe system, inotarisana nematambudziko matatu makuru eese masystem ekutarisa: kuunganidza nekugadzirisa data, kuchengetedza nhoroondo, nekuichenesa.
Matanho ekugamuchira, kugadzirisa uye kurekodha data anotora nguva. Kwete zvakawanda, asi kune yakakura system izvi zvinogona kukonzera kunonoka kukuru. Dambudziko rekuchengetedza inyaya yekuwana data. Iyo inoshandiswa kune mishumo, cheki uye zvinokonzeresa. Latencies mukuwana data zvakare inokanganisa mashandiro. Kana dhatabhesi richikura, data isina basa inofanirwa kubviswa. Kubvisa ibasa rakaoma rinodyawo zvimwe zviwanikwa.
Matambudziko ekunonoka panguva yekuunganidza uye kuchengetedza muZabbix anogadziriswa ne caching: akati wandei marudzi e cache, caching mudhatabhesi. Kugadzirisa dambudziko rechitatu, caching haina kukodzera, saka Zabbix yakashandisa TimescaleDB. Achakuudza nezvazvo Andrey Gushchin - technical support engineer . Andrey anga achitsigira Zabbix kwemakore anopfuura 6 uye ane ruzivo rwakananga nekuita.
TimescaleDB inoshanda sei, inokwanisa kupa kuita sei kana ichienzaniswa neyakajairika PostgreSQL? Ibasa ripi rinoitwa naZabbix kune TimescaleDB dhatabhesi? Maitiro ekutanga kubva pakatanga uye sei kutama kubva kuPostgreSQL uye ndeipi gadziriso ine kuita kurinani? Pamusoro peizvi zvose pasi pekucheka.
Matambudziko Ekubudirira
Yese yekutarisa sisitimu inotarisana nematambudziko ekuita. Ini ndichataura pamusoro pezvitatu zvacho: kuunganidza data uye kugadzirisa, kuchengetedza, uye kubvisa nhoroondo.
Kukurumidza kuunganidza data uye kugadzirisa. Iyo yakanaka yekutarisa sisitimu inofanirwa kukurumidza kugamuchira data rese uye kuigadzirisa zvinoenderana neinokonzeresa mataurirwo - zvinoenderana nemaitiro ayo. Mushure mekugadzirisa, sisitimu inofanirwawo nekukurumidza kuchengetedza iyi data mudhatabhesi kuti ishandiswe gare gare.
Kuchengeta nhoroondo. Iyo yakanaka yekutarisa sisitimu inofanirwa kuchengetedza nhoroondo mudhatabhesi uye inopa nyore kuwana metrics. Nhoroondo inodiwa kuti ishandiswe mumishumo, magirafu, zvinokonzeresa, zvikumbaridzo, uye zvakaverengerwa chenjedzo data zvinhu.
Kubvisa nhoroondo. Dzimwe nguva kunouya zuva rausingade kuchengetedza metrics. Sei uchida data yakaunganidzwa makore mashanu apfuura, mwedzi kana maviri: dzimwe node dzakadzimwa, mamwe mauto kana metrics haachadiwi nekuti akasakara uye haasisiri kuunganidzwa. Iyo yakanaka yekutarisa sisitimu inofanirwa kuchengetedza nhoroondo data uye kuidzima nguva nenguva kuitira kuti dhatabhesi isakure.
Kuchenesa dhata rekare inyaya yakakomba iyo inokanganisa zvakanyanya kuita kwedatabase.
Kuchengeta muZabbix
MuZabbix, yekutanga uye yechipiri mafoni anogadziriswa uchishandisa caching. RAM inoshandiswa kuunganidza uye kugadzirisa data. Yekuchengetedza - nhoroondo mune zvinokonzeresa, magirafu uye akaverengerwa data zvinhu. Padivi redhatabhesi kune imwe caching yezvakasarudzika sarudzo, semuenzaniso, magirafu.
Kuchengetera padivi reZabbix server pachayo ndeiyi:
- ConfigurationCache;
- ValueCache;
- HistoryCache;
- TrendsCache.
Funga nezvazvo zvakanyanya.
ConfigurationCache
Iyi ndiyo cache huru yatinochengeta metrics, mauto, data zvinhu, zvinokonzeresa - zvese zvatinoda kuPreProcessing uye yekuunganidza data.
Zvese izvi zvakachengetwa muConfigurationCache kuitira kuti usagadzire zvisina basa mudhatabhesi. Mushure mekunge sevha yatanga, tinogadziridza iyi cache, kugadzira uye nguva nenguva kugadzirisa zvigadziriso.
Data collection
Iyo dhizaini yakakura kwazvo, asi chinhu chikuru mairi pickers. Aya akasiyana-siyana "vavhoti" - magungano maitiro. Ivo vane basa remhando dzakasiyana dzegungano: vanounganidza data kuburikidza neSNMP, IPMI, uye voiendesa yese kuPreProcessing.
Vaunganidzi vanoratidzwa neorenji.
Zabbix yakaverenga zvinhu zvakaunganidzwa zvinodikanwa kuunganidza cheki. Kana tiinazvo, tinovatorera data zvakananga kubva kuValueCache.
PreProcessing HistoryCache
Vese vaunganidzi vanoshandisa ConfigurationCache kugamuchira mabasa. Vanobva vaaendesa kuPreProcessing.
PreProcessing inoshandisa ConfigurationCache kugamuchira PreProcessing matanho. Inobata iyi data nenzira dzakasiyana.
Mushure mekugadzirisa iyo data tichishandisa PreProcessing, tinoichengeta muHistoryCache kuti igadziriswe. Izvi zvinopedza kuunganidzwa kwedata uye isu tinoenda kune iyo huru maitiro muZabbix - nhoroondo syncer, sezvo iri monolithic architecture.
Cherechedza: PreProcessing ibasa rakaoma. Kubva v 4.2 yakaendeswa kune proxy. Kana iwe uine Zabbix yakakura kwazvo ine nhamba huru yezvinhu zve data uye frequency yekuunganidza, zvino izvi zvinoita kuti basa rive nyore.
ValueCache, nhoroondo & maitiro cache
Nhoroondo syncer ndiyo nzira huru iyo inogadzirisa atomu yega yega data data, kureva kukosha kwega kwega.
Nhoroondo syncer inotora kukosha kubva kuNhoroondoCache uye inotarisa Configuration yekuvapo kwezvinokonzeresa zvekuverenga. Kana iripo, inoverengera.
Nhoroondo syncer inogadzira chiitiko, kukwira kugadzira zviziviso kana zvichidikanwa nekumisikidza, uye marekodhi. Kana paine zvinokonzeresa kunotevera kugadzirisa, saka inochengeta kukosha uku muValueCache kuitira kuti usasvike patafura yenhoroondo. Iyi ndiyo nzira iyo ValueCache inozadzwa nayo nedata inodiwa kuverenga zvinokonzeresa uye zvakaverengerwa zvinhu.
Nhoroondo syncer inonyora data rese kune dhatabhesi, uye inonyora kune dhisiki. Iyo yekugadzirisa inopera pano.
Kuchengetedza mu database
Padivi re database kune akasiyana cache kana iwe uchida kuona magirafu kana mishumo pane zviitiko:
Innodb_buffer_poolkudivi reMySQL;shared_bufferskudivi rePostgreSQL;effective_cache_sizeparutivi rweOracle;shared_poolkudivi reDB2.
Kune mamwe akawanda cache, asi aya ndiwo makuru kune ese dhatabhesi. Ivo vanokutendera kuti uchengete data mu RAM inowanzo kudiwa pamibvunzo. Vane matekinoroji avo eizvi.
Database performance yakakosha
Sevha yeZabbix inogara ichiunganidza data uye inonyora. Kana yatangwazve, inoverengawo kubva munhoroondo kuzadza iyo ValueCache. Inoshandisa zvinyorwa nemishumo Purogiramu inonzi Zabbix, iyo yakavakirwa paWebhu interface. Zabbix API inowana iyo dhatabhesi uye inotora iyo inodiwa data yemagirafu, mishumo, rondedzero yezviitiko uye nyaya dzichangoburwa.
Zvekuona - grafana. Iyi ndiyo mhinduro yakakurumbira pakati pevashandisi vedu. Inogona kutumira zvakananga zvikumbiro kuburikidza neZabbix API uye kune database, uye inogadzira imwe makwikwi ekugamuchira data. Naizvozvo, kukwenenzvera uye kurinani kwedhatabhesi kunodiwa kuti ienderane nekukasira kuburitsa kwemhedzisiro uye kuyedzwa.
Muridzi wemba
Dambudziko rechitatu rekuita muZabbix ndeyekubvisa nhoroondo uchishandisa Housekeeper. Inoteedzera ese magadzirirwo - iyo data data inoratidza kuti inguva yakadii kuchengetedza dynamics yekuchinja (maitiro) mumazuva.
Isu tinoverenga TrendsCache pane nhunzi. Kana iyo data yasvika, tinoiunganidza kweawa imwe chete uye tinoinyora mumatafura emasimba ekuchinja kwemaitiro.
Muchengeti wemba anotanga uye anodzima ruzivo kubva mudhatabhesi achishandisa yakajairwa "select". Izvi hazvisi nguva dzose zvinobudirira, sezvinogona kuonekwa kubva pamagirafu ekuita kwemaitiro emukati.
Girafu dzvuku rinoratidza kuti Nhoroondo syncer inogara yakabatikana. Girafu reorenji kumusoro ndeye Housekeeper, iyo inogara ichimhanya. Anomirira dhatabhesi kuti abvise mitsetse yese yaakataura.
Ndepapi paunofanira kudzima Housekeeper? Semuenzaniso, pane "Item ID" uye iwe unofanirwa kudzima yekupedzisira zviuru zvishanu mitsara mukati meimwe nguva. Zvechokwadi, izvi zvinoitika nendekisi. Asi kazhinji iyo dataset yakakura kwazvo, uye dhatabhesi ichiri kuverenga kubva kudhisiki uye inoiisa mu cache. Izvi zvinogara zvichidhura zvakanyanya kune dhatabhesi uye, zvichienderana nehukuru hwedhatabhesi, zvinogona kutungamirira kumatambudziko ekuita.
Muchengeti wemba ari nyore kudzima. MuWeb interface pane marongero mu "Administration general" yeMuchengeti wemba. Isu tinodzima mukati meKuchengeta Imba yemukati maitiro enhoroondo uye haichazvigone.
Muchengeti wemba akavharwa, magirafu akaiswa kunze - ndeapi matambudziko angave mune iyi kesi uye chii chingabatsira kugadzirisa dambudziko rechitatu rekuita?
Kupatsanura - kupatsanura kana kupatsanura
Kazhinji, kugovera kunogadziriswa nenzira yakasiyana pane imwe neimwe yehukama database yandakanyora. Imwe neimwe ine tekinoroji yayo, asi yakafanana mune zvese. Kugadzira chikamu chitsva kazhinji kunotungamira kune mamwe matambudziko.
Kazhinji, zvikamu zvinogadziriswa zvinoenderana ne "seta" - huwandu hwe data inogadzirwa muzuva rimwe. Sezvo mutemo, Partitioning inopihwa muzuva rimwe, iyi ndiyo shoma. Kune maitiro ebhechi nyowani - 1 mwedzi.
Izvo zvakakosha zvinogona kuchinja kana "setup" yakakura kwazvo. Kana "setup" diki inosvika 5 nvps (zvinyowani pasekondi), yepakati kubva pa000 kusvika 5, saka yakakura iri pamusoro pe000 nvps. Aya akakura uye akakura kwazvo masisitimu anoda kunyatso gadziridzwa dhatabhesi.
Pakuisa kwakakura kwazvo, nguva yezuva rimwe inogona kunge isiri yakakwana. Ndakaona MySQL partitions ye40 GB kana kupfuura pazuva. Iyi yakawanda kwazvo data inogona kukonzera matambudziko uye inoda kuderedzwa.
Partitioning inopa chii?
Partitioning matafura. Kazhinji aya mafaira akaparadzana pane diski. Chirongwa chemubvunzo chinosarudza chikamu chimwe zvakanyanya zvakanyanya. Kazhinji kupatsanura kunoshandiswa nemhando - izvi ndezvechokwadi kune Zabbix. Isu tinoshandisa "timestamp" ipapo - nguva kubva pakutanga kwenguva. Idzi nhamba dzakajairika kwatiri. Iwe unoisa kutanga uye kupera kwezuva - ichi chikamu.
Kukurumidza kubvisa - DELETE. Imwe faira/subtable inosarudzwa, pane kusarudzwa kwemitsara yekudzima.
Zvinonyanya kukurumidza kutora data SELECT - inoshandisa imwe kana kupfuura zvikamu, pane tafura yese. Kana iwe ukawana data ine mazuva maviri ekare, inotorwa kubva kudhatabhesi nekukurumidza nekuti iwe unongoda kurodha faira rimwe mu cache uye woridzosera, kwete tafura hombe.
Kazhinji dhatabhesi dzakawanda dzinokwidziridzwawo INSERT - kuiswa mutafura yemwana.
TimescaleDB
Yev 4.2, isu takaisa pfungwa dzedu kuTimescaleDB. Uku ndiko kuwedzera kwePostgreSQL ine chimiro chekuzvarwa. Iyo yekuwedzera inoshanda nemazvo nenguva yakatevedzana data, pasina kurasikirwa nemabhenefiti ehukama dhatabhesi. TimescaleDB zvakare inopatsanura otomatiki.
TimescaleDB ine pfungwa hypertable (hypertable) yaunogadzira. Rine chunks - partitions. Chunks anodzorwa otomatiki hypertable zvimedu zvisingakanganise zvimwe zvimedu. Chikamu chimwe nechimwe chine nguva yacho.
TimescaleDB vs PostgreSQL
TimescaleDB inoshanda chaizvo. Vagadziri veawedzero vanoti vanoshandisa yakanyatso bvunzo kugadzirisa algorithm, kunyanya inserts . Sezvo saizi yekuisa dhata inokura, iyo algorithm inochengetedza inogara ichiita.
Mushure memitsara yemamiriyoni mazana maviri, PostgreSQL inowanzotanga kunyura zvakanyanya uye inorasikirwa nekushanda ku 200. TimescaleDB inokubvumira kuti uise zvakanaka "kuisa" kune chero huwandu hwe data.
Kuiswa
Kuisa TimescaleDB zviri nyore kune chero package. IN zvese zvinotsanangurwa zvakadzama - zvinoenderana neiyo official PostgreSQL mapakeji. TimescaleDB inogona zvakare kuvakwa uye kuunganidzwa nemaoko.
Kune dhatabhesi yeZabbix isu tinongomutsa kuwedzera:
echo "CREATE EXTENSION IF NOT EXISTS timescaledb CASCADE;" | sudo -u postgres psql zabbix
Iwe shandisa extension uye gadzira iyo Zabbix dhatabhesi. Nhanho yekupedzisira ndeyekugadzira hypertable.
Kutamisa matafura enhoroondo kuenda kuTimescaleDB
Pane basa rinokosha reizvi create_hypertable:
SELECT create_hypertable(‘history’, ‘clock’, chunk_time_interval => 86400, migrate_data => true);
SELECT create_hypertable(‘history_unit’, ‘clock’, chunk_time_interval => 86400, migrate_data => true);
SELECT create_hypertable(‘history_log’, ‘clock’, chunk_time_interval => 86400, migrate_data => true);
SELECT create_hypertable(‘history_text’, ‘clock’, chunk_time_interval => 86400, migrate_data => true);
SELECT create_hypertable(‘history_str’, ‘clock’, chunk_time_interval => 86400, migrate_data => true);
SELECT create_hypertable(‘trends’, ‘clock’, chunk_time_interval => 86400, migrate_data => true);
SELECT create_hypertable(‘trends_unit’, ‘clock’, chunk_time_interval => 86400, migrate_data => true);
UPDATE config SET db_extension=’timescaledb’, hk_history_global=1, hk_trends_global=1
Basa racho rine maparameter matatu. Chekutanga - tafura mu database, iyo yaunoda kugadzira hypertable. Chepiri - munda, maererano nezvaunoda kugadzira chunk_time_interval - kupindirana kwezvikamu zvekuparadzanisa zvichashandiswa. Mune mhaka yangu, nguva yacho izuva rimwe - 86.
Chechitatu parameter - migrate_data. Kana ukaisa true, ipapo data yese yazvino inotamirwa kune dzakafanogadzirwa chunks. Ndakazvishandisa ini migrate_data. Ndaiva neinenge TB, izvo zvakatora inopfuura awa. Kunyangwe mune zvimwe zviitiko, panguva yekuyedzwa, ndakadzima nhoroondo data yemhando dzemhando dzaive dzisingadiwe kuchengetedza, kuti ndirege kuvaendesa.
Danho rekupedzisira - UPDATE: mukati db_extension put timescaledbkuitira kuti dhatabhesi rinzwisise kuti iyi yekuwedzera iripo. Zabbix inoimutsa uye inoshandisa nenzira kwayo syntax uye mibvunzo kune dhatabhesi - izvo zvinhu zvinodiwa kuTimescaleDB.
Hardware configuration
Ndakashandisa maseva maviri. Chekutanga - VMware muchina. Idiki chaizvo: 20 Intel® Xeon® CPU E5-2630 v 4 @ 2.20GHz processors, 16 GB RAM uye 200 GB SSD.
Ndakaisa PostgreSQL 10.8 pairi neDebian 10.8-1.pgdg90+1 OS uye xfs faira system. Ini ndakagadzirisa zvese zvishoma kuti ndishandise iyi dhatabhesi, kubvisa iyo Zabbix pachayo ichashandisa.
Pamuchina mumwechete paive neZabbix server, PostgreSQL uye load agents. Ndakanga ndiine 50 maagent agents aishandisa LoadableModulekukurumidza kuburitsa mhinduro dzakasiyana: nhamba, tambo. Ndakazadza dhatabhesi nedata rakawanda.
Pakutanga iyo configuration yaivemo 5 zvinhu data per host. Chinenge chinhu chose chaive chine chinokonzeresa kuti chifanane nekuisa chaiko. Mune zvimwe zviitiko paive nekanopfuura imwe trigger. Kune imwe network node yaivepo 3-000 zvinokonzeresa.
Data Item Update Interval − 4-7 masekondi. Ini ndakagadzirisa mutoro pachawo nekushandisa kwete chete 50 vamiririri, asi kuwedzera zvimwe. Zvakare, ndichishandisa data data, ndakagadzirisa mutoro uye ndikadzikisa nguva yekuvandudza kusvika ku4 s.
PostgreSQL. 35 nvps
Kumhanya kwangu kwekutanga pane iyi Hardware yaive paPostgreSQL yakachena - zviuru makumi matatu neshanu zvakakosha pasekondi. Sezvauri kuona, kuisa data kunotora zvikamu zvesekondi - zvese zvakanaka uye nekukurumidza. Chinhu chete ndechekuti 35 GB SSD disk inozadza nekukurumidza.
Iyi ndiyo yakajairwa Zabbix server performance dashboard.
Girafu yekutanga yebhuruu ndiyo nhamba yemakoshero pasekondi. Yechipiri girafu kurudyi ndiko kurodha kwemaitiro ekuvaka. Yetatu iri kurodha maitiro ekuvaka emukati: nhoroondo masyncers uye Housekeeper, anga achimhanya pano kwenguva yakati rebei.
Yechina girafu inoratidza HistoryCache kushandiswa. Iyi imhando yebuffer isati yapinza mudhatabhesi. Girafu yechishanu yegirini inoratidza kushandiswa kweValueCache, ndiko kuti, vangani ValueCache inorova yezvinokonzeresa - izvi zviuru zvakati wandei pasekondi.
PostgreSQL. 50 nvps
Ipapo ndakawedzera mutoro kusvika kumakumi mashanu ezviuru zvakakosha pasekondi pane imwechete hardware.
Pakurodha kubva kuMuchengeti Wemba, kuisa zviuru gumi zvakakosha kwakatora 10-2 masekondi.
Muchengeti wemba ari kutotanga kuvhiringidza basa.
Girafu yechitatu inoratidza kuti, kazhinji, mutoro kune vatepi uye nhoroondo synchers ichiri pa60%. Mugirafu rechina, HistoryCache yave kutotanga kuzadza zvakanyanya panguva yekushanda kweMuchengeti wemba. Iyo 20% yakazara, iyo inenge 0,5 GB.
PostgreSQL. 80 nvps
Ipapo ndakawedzera mutoro kusvika ku80 zviuru zvetsika pasekondi. Izvi zvingangoita zviuru mazana mana zve data zvinhu uye 400 zviuru zvinokonzeresa.
Mutengo wekurodha wemakumi matatu enhoroondo synchers watokwira zvakanyanya.
Ini zvakare ndakawedzera akasiyana ma paramita: nhoroondo masyncers, cache.
Pane yangu Hardware, kurodha kwenhoroondo masyncers akawedzera kusvika pakakwirira. NhoroondoCache yakakurumidza kuzadzwa nedata - data rekugadzirisa rakanga raungana mubuffer.
Nguva yese iyi ndakatarisa mashandisirwo akaitwa processor, RAM uye mamwe masisitimu ma paramita, uye ndakaona kuti kushandiswa kwedisiki kwaive pakakwirira.
Ndawana kushandiswa yakanyanya disk kugona pane iyi hardware uye pamushini uyu chaiwo. Nekusimba kwakadaro, PostgreSQL yakatanga kumhanyisa data zvakanyanya, uye dhisiki rakanga risisina nguva yekunyora nekuverenga.
Sevha yechipiri
Ndakatora imwe sevha, iyo yaitova ne 48 processors uye 128 GB ye RAM. Ndakaigadzirisa - kuiisa kune makumi matanhatu nhoroondo syncer, uye ndikawana kuita kunogamuchirika.
Muchokwadi, uyu watova muganhu wekubudirira uko chimwe chinhu chinoda kuitwa.
TimescaleDB. 80 nvps
Basa rangu guru nderekuyedza kugona kwe TimescaleDB kurwisa Zabbix mutoro. 80 zviuru zvakakosha pasekondi yakawanda, kuwanda kwekuunganidza metrics (kunze kweYandex, hongu) uye yakakura kwazvo "setup".
Pane kunyura mune yega girafu - uku ndiko chaizvo kutama kwedata. Mushure mekutadza muSevha yeZabbix, iyo yekurodha nhoroondo yesyncer yakachinja zvakanyanya - yakadonha katatu.
TimescaleDB inokutendera kuti uise data kanenge katatu nekukurumidza uye shandisa shoma HistoryCache.
Saizvozvowo, iwe uchagamuchira data munguva yakakodzera.
TimescaleDB. 120 nvps
Ipapo ndakawedzera nhamba yezvinhu zve data kusvika kune zviuru 500. Basa guru raiva rekuedza kukwanisa kweTimescaleDB - ndakagamuchira kukosha kwakaverengwa kwe125 zviuru zvakakosha pasekondi.
Iyi ibasa re "setup" rinogona kushanda kwenguva yakareba. Asi sezvo dhisiki rangu raingova 1,5 TB, ndakarizadza mumazuva mashoma.
Chinhu chinonyanya kukosha ndechekuti panguva imwe chete zvikamu zvitsva zveTimescaleDB zvakagadzirwa.
Izvi hazvionekwe zvachose pakuita. Kana zvikamu zvagadzirwa muMySQL, semuenzaniso, zvese zvakasiyana. Izvi zvinowanzoitika husiku nekuti zvinovharira kuisirwa zvakajairika, kushanda nematafura uye zvinogona kugadzira kuderedzwa kwesevhisi. Izvi hazvisizvo neTimescaleDB.
Semuenzaniso, ini ndicharatidza girafu rimwe kubva kune vakawanda munharaunda. Mumufananidzo, TimescaleDB inogoneswa, nekuda kwekuti mutoro wekushandisa io.weight pane processor wadonha. Kushandiswa kwezvinhu zvemukati maitiro zvakare kwakadzikira. Uyezve, uyu ndiwo wakajairika muchina pane akajairwa pancake disks, kwete SSD.
zvakawanikwa
TimescaleDB mhinduro yakanaka kune diki "setup", iyo inokanganisa dhisiki kuita. Ichakubvumidza kuti uenderere mberi uchishanda zvakanaka kudzamara dhatabhesi yaendeswa kune Hardware nekukurumidza sezvinobvira.
TimescaleDB iri nyore kugadzirisa, inopa maitiro ekuwana, inoshanda zvakanaka neZabbix uye ine zvakanakira pamusoro pePostgreSQL.
Kana iwe ukashandisa PostgreSQL uye usingaronge kuichinja, ini ndinokurudzira shandisa PostgreSQL neTimescaleDB yekuwedzera pamwe chete neZabbix. Iyi mhinduro inoshanda zvinobudirira kusvika kune yepakati "setup".
Kana tichiti "high performance" tinoreva . Iwe hauzotore nguva yakareba kumirira kuti udzidze nezve matekinoroji uye maitiro anogonesa masevhisi kushandira mamirioni evashandisi. List yaNovember 7 na8 isu takatonyora, asi pano zvakawanda zvinogona kutaurwa.
Nyorera kune yedu и , umo tinoburitsa zvinhu zvemusangano uri kuuya, uye toona kuti tingawana sei zvakawanda kubva mairi.
Source: www.habr.com