Zabbix tsarin kulawa ne. Kamar kowane tsarin, yana fuskantar manyan matsaloli guda uku na duk tsarin sa ido: tattarawa da sarrafa bayanai, adana tarihi, da tsaftace su.
Matakan karɓa, sarrafawa da rikodin bayanai suna ɗaukar lokaci. Ba yawa, amma ga babban tsarin wannan zai iya haifar da babban jinkiri. Matsalar ajiya matsala ce ta samun bayanai. Ana amfani da su don rahotanni, dubawa da kuma jawowa. Latencies a cikin samun damar bayanai kuma yana tasiri aiki. Lokacin da bayanan bayanai suka girma, dole ne a share bayanan da basu da mahimmanci. Cire aiki ne mai wahala wanda kuma yana cinye wasu albarkatu.
Matsalolin jinkiri yayin tattarawa da ajiya a cikin Zabbix ana magance su ta hanyar caching: nau'ikan cache da yawa, caching a cikin bayanan. Don magance matsala ta uku, caching bai dace ba, don haka Zabbix yayi amfani da TimecaleDB. Zai ba ku labarin Andrey Gushchin - injiniya goyon bayan fasaha . Andrey yana tallafawa Zabbix fiye da shekaru 6 kuma yana da kwarewa kai tsaye tare da aiki.
Yaya TimescaleDB ke aiki, wane aiki zai iya bayarwa idan aka kwatanta da PostgreSQL na yau da kullun? Wace rawa Zabbix ke takawa don bayanan TimescaleDB? Yadda za a fara daga karce da yadda ake yin ƙaura daga PostgreSQL kuma wane tsari ya fi dacewa? Game da duk wannan a ƙarƙashin yanke.
Kalubalen Haɓakawa
Kowane tsarin sa ido yana fuskantar takamaiman ƙalubalen ayyuka. Zan yi magana game da uku daga cikinsu: tattarawa da sarrafa bayanai, adanawa, da share tarihi.
Tarin bayanai da sauri da sarrafawa. Kyakkyawan tsarin kulawa yakamata ya karɓi duk bayanan da sauri kuma ya sarrafa shi bisa ga maganganun faɗakarwa - gwargwadon ka'idodinsa. Bayan sarrafawa, tsarin dole ne kuma yayi sauri adana wannan bayanan a cikin ma'ajin bayanai don amfani daga baya.
Adana tarihi. Kyakkyawan tsarin sa ido ya kamata ya adana tarihi a cikin ma'ajin bayanai kuma ya samar da sauƙi ga ma'auni. Ana buƙatar tarihi don a yi amfani da shi a cikin rahotanni, jadawalai, masu faɗakarwa, kofa, da abubuwan bayanan faɗakarwa.
Share tarihi. Wani lokaci akwai ranar da ba kwa buƙatar adana awo. Me yasa kuke buƙatar bayanan da aka tattara shekaru 5 da suka gabata, wata ɗaya ko biyu: an share wasu nodes, wasu runduna ko awoyi ba a buƙatar su saboda sun tsufa kuma ba a tattara su. Kyakkyawan tsarin sa ido ya kamata ya adana bayanan tarihi kuma yana share su lokaci zuwa lokaci don kada bayanan su girma.
Tsaftace bayanan da ba su da tushe wani lamari ne mai mahimmanci wanda ke tasiri sosai ga aikin bayanai.
Caching in Zabbix
A cikin Zabbix, ana warware kiran farko da na biyu ta amfani da caching. Ana amfani da RAM don tattarawa da sarrafa bayanai. Don ajiya - tarihi a cikin abubuwan jawo, jadawalai da abubuwan da aka ƙididdige su. A gefen bayanan bayanai akwai wasu caching don zaɓi na asali, misali, jadawalai.
Caching a gefen sabar Zabbix kanta shine:
- Kanfigareshan Cache;
- ValueCache;
- Tarihin Cache;
- TrendsCache.
Bari mu bincika su daki daki.
Kanfigareshan Cache
Wannan shine babban cache inda muke adana awo, runduna, abubuwan bayanai, abubuwan jan hankali - duk abin da muke buƙata don Gabatarwa da tattara bayanai.
Ana adana duk waɗannan a cikin ConfigurationCache don kar a haifar da tambayoyin da ba dole ba a cikin bayanan. Bayan uwar garken ya fara, muna sabunta wannan cache, ƙirƙira da sabunta saitunan lokaci-lokaci.
Tarin bayanai
Tsarin yana da girma sosai, amma babban abu a ciki shine masu tarawa. Waɗannan su ne daban-daban "pollers" - taro tafiyar matakai. Suna da alhakin nau'ikan taro daban-daban: suna tattara bayanai ta SNMP, IPMI, kuma suna tura su duka zuwa PreProcessing.
An zayyana masu tarawa da orange.
Zabbix ya ƙididdige abubuwan tarawa waɗanda ake buƙata don tara cak. Idan muna da su, muna debo musu bayanan kai tsaye daga ValueCache.
PreProcessing HistoryCache
Duk masu tarawa suna amfani da ConfigurationCache don karɓar ayyuka. Sannan su canza su zuwa PreProcessing.
PreProcessing yana amfani da ConfigurationCache don karɓar matakan Gabatarwa. Yana aiwatar da wannan bayanan ta hanyoyi daban-daban.
Bayan sarrafa bayanan ta amfani da PreProcessing, mun adana shi a cikin HistoryCache don sarrafawa. Wannan ya ƙare tarin bayanai kuma mun matsa zuwa babban tsari a Zabbix - tarihin daidaitawa, tunda tsarin gine-gine ne na monolithic.
Lura: PreProcessing aiki ne mai wahala. Tare da v 4.2 an motsa shi zuwa wakili. Idan kana da babban Zabbix mai girma tare da adadi mai yawa na abubuwan bayanai da mita tarin, to wannan yana sa aikin ya fi sauƙi.
ValueCache, tarihi & cache abubuwan da ke faruwa
Syncer tarihi shine babban tsari wanda ke sarrafa kowane ɓangaren bayanai ta atomatik, wato kowane ƙima.
Mai daidaita tarihi yana ɗaukar ƙima daga HistoryCache kuma yana bincika Kanfigareshan don kasancewar abubuwan da ke haifar da ƙididdiga. Idan akwai, yana lissafta.
Daidaitaccen tarihi yana haifar da aukuwa, haɓakawa don ƙirƙirar faɗakarwa idan an buƙata ta hanyar daidaitawa, da rikodi. Idan akwai abubuwan da ke haifar da aiki na gaba, to yana adana wannan ƙimar a cikin ValueCache don kar a shiga teburin tarihi. Wannan shine yadda ValueCache ke cike da bayanan da suka wajaba don ƙididdige abubuwan da ke haifar da ƙididdigewa.
Tarihin daidaitawa yana rubuta duk bayanai zuwa ma'ajin bayanai, kuma yana rubutawa zuwa faifai. Tsarin sarrafawa ya ƙare a nan.
Caching a cikin database
A gefen ma'ajin bayanai akwai caches daban-daban lokacin da kake son duba hotuna ko rahotanni kan abubuwan da suka faru:
Innodb_buffer_poola gefen MySQL;shared_buffersa gefen PostgreSQL;effective_cache_sizea gefen Oracle;shared_poola gefen DB2.
Akwai wasu caches da yawa, amma waɗannan su ne manyan na duk bayanan bayanai. Suna ba ku damar adana bayanai a cikin RAM waɗanda galibi ake buƙata don tambayoyi. Suna da nasu fasahar don wannan.
Ayyukan tushen bayanai yana da mahimmanci
Sabar Zabbix koyaushe tana tattara bayanai tana rubuta ta. Lokacin da aka sake kunna shi, yana kuma karantawa daga tarihi don cike ValueCache. Yana amfani da rubutun da rahotanni Zabbix API, wanda aka gina akan hanyar sadarwa ta Yanar Gizo. Zabbix API yana shiga cikin bayanan bayanai kuma yana maido da mahimman bayanai don jadawalai, rahotanni, jerin abubuwan da suka faru da sabbin batutuwa.
Don gani - Grafana. Wannan sanannen bayani ne a tsakanin masu amfani da mu. Yana iya aika buƙatun kai tsaye ta hanyar Zabbix API da zuwa rumbun adana bayanai, kuma ya ƙirƙiri wata gasa don karɓar bayanai. Don haka, ana buƙatar mafi kyawun daidaita ma'ajin bayanai don dacewa da saurin isar da sakamako da gwaji.
Maigida
Kalubalen wasan kwaikwayon na uku a cikin Zabbix shine share tarihi ta amfani da mai kula da gida. Yana biye da duk saitunan - abubuwan bayanan suna nuna tsawon lokacin da za a adana canje-canje na canje-canje (trends) a cikin kwanaki.
Muna lissafin TrendsCache akan tashi. Lokacin da bayanan ya zo, muna tara shi har tsawon sa'a ɗaya kuma mu yi rikodin shi a cikin teburi don haɓakar canje-canjen yanayi.
Mai kula da gida yana farawa kuma yana share bayanai daga rumbun adana bayanai ta amfani da “zaɓi” na yau da kullun. Wannan ba koyaushe yake tasiri ba, kamar yadda ake iya gani daga jadawali na ayyukan ciki.
Hoton ja yana nuna cewa Mai daidaita Tarihi yana aiki koyaushe. Hoton lemu a saman shine Mai tsaron gida, wanda koyaushe yana gudana. Yana jiran ma'aunin bayanai ya goge duk layuka da ya kayyade.
Yaushe ya kamata ku kashe ma'aikacin gida? Misali, akwai "ID na abu" kuma kuna buƙatar share layuka dubu 5 na ƙarshe a cikin takamaiman lokaci. Tabbas, wannan yana faruwa ta hanyar index. Amma yawanci bayanan suna da girma sosai, kuma bayanan har yanzu suna karantawa daga faifai kuma suna sanya shi cikin ma'ajin. Wannan ko da yaushe aiki ne mai tsada sosai ga ma'ajiyar bayanai kuma, dangane da girman ma'ajin bayanai, na iya haifar da matsalolin aiki.
Mai aikin gida yana da sauƙin kashewa. A cikin mahallin Yanar Gizo akwai saiti a cikin "Gudanarwa gabaɗaya" don Mai Kula da Gida. Muna hana Kiwon Gida na ciki don tarihin abubuwan da ke faruwa a ciki kuma baya sarrafa shi.
An kashe ma'aikacin gida, an daidaita jadawali - wadanne matsaloli za a iya samu a wannan yanayin kuma menene zai taimaka wajen magance ƙalubalen aiki na uku?
Rarraba - raba ko raba
Yawanci, ana saita rabuwa ta wata hanya dabam akan kowace rumbun adana bayanai da na lissafa. Kowannensu yana da nasa fasahar, amma suna kama da gaba ɗaya. Ƙirƙirar sabon bangare sau da yawa yana haifar da wasu matsaloli.
Yawanci, ana saita ɓangarori dangane da “saitin” - adadin bayanan da aka ƙirƙira a rana ɗaya. A matsayinka na mai mulki, ana bayar da Rarraba a cikin rana ɗaya, wannan shine mafi ƙarancin. Don yanayin sabon tsari - wata 1.
Ƙimar na iya canzawa idan "saitin" yana da girma sosai. Idan karamin "saitin" ya kai 5 nvps (sabbin dabi'u a sakan daya), matsakaicin shine daga 000 zuwa 5, to babban yana sama da 000 nvps. Waɗannan manyan kayan aiki ne masu girma kuma masu girman gaske waɗanda ke buƙatar tsayayyen tsarin bayanai.
A kan manyan kayan aiki, tsawon yini ɗaya bazai yi kyau ba. Na ga sassan MySQL na 40 GB ko fiye a kowace rana. Wannan adadi ne mai yawan gaske wanda zai iya haifar da matsala kuma yana buƙatar ragewa.
Menene Partitioning ke bayarwa?
Tebur masu rarrabawa. Yawancin lokaci waɗannan fayiloli ne daban akan faifai. Shirin tambayar yana zabar bangare ɗaya mafi kyawu. Yawancin lokaci ana amfani da rarrabuwa ta kewayon - wannan kuma gaskiya ne ga Zabbix. Muna amfani da "timestamp" a can - lokaci tun farkon zamanin. Waɗannan lambobi ne na yau da kullun a gare mu. Kun saita farkon rana da ƙarshen rana - wannan bangare ne.
Saurin cirewa - DELETE. An zaɓi fayil/subtable ɗaya, maimakon zaɓin layuka don gogewa.
Mahimmanci yana hanzarta dawo da bayanai SELECT - yana amfani da bangare ɗaya ko fiye, maimakon duka tebur. Idan kun sami damar shiga bayanan da suka cika kwanaki biyu, ana samun su da sauri daga ma'ajin bayanai saboda kuna buƙatar loda fayil ɗaya kawai a cikin cache ɗin ku mayar da su, ba babban tebur ba.
Yawancin lokaci ma ana haɓaka rumbun adana bayanai da yawa INSERT - shigarwa a cikin tebur na yaro.
LokaciDB
Don v 4.2, mun mai da hankalinmu ga TimecaleDB. Wannan haɓakawa ne don PostgreSQL tare da ƙirar ƙasa. Tsawaita yana aiki yadda ya kamata tare da bayanan jeri na lokaci, ba tare da rasa fa'idodin bayanan bayanai na alaƙa ba. TimecaleDB kuma yana rarraba ta atomatik.
TimecaleDB yana da ra'ayi hypertable (hypertable) wanda kuke halittawa. Ya ƙunshi dunkulewa - partitions. Ana sarrafa chunks ta atomatik gaɓoɓin ɓarna masu ƙarfi waɗanda ba su shafar sauran guntu. Kowane chunk yana da nasa kewayon lokaci.
TimecaleDB vs PostgreSQL
TimecaleDB yana aiki da kyau sosai. Masu kera na'urorin haɓaka suna da'awar cewa suna amfani da ingantaccen tsarin sarrafa tambaya, musamman inserts . Yayin da girman saka bayanan bayanai ke girma, algorithm ɗin yana kula da aiki akai-akai.
Bayan layuka miliyan 200, PostgreSQL yakan fara raguwa sosai kuma ya rasa aikin zuwa 0. TimecaleDB yana ba ku damar saka “saka” da kyau don kowane adadin bayanai.
saitin
Shigar da TimescaleDB yana da sauƙi ga kowane fakiti. IN An bayyana komai dalla-dalla - ya dogara da fakitin PostgreSQL na hukuma. TimecaleDB kuma ana iya ginawa da kuma haɗa shi da hannu.
Don bayanan Zabbix kawai muna kunna haɓakawa kawai:
echo "CREATE EXTENSION IF NOT EXISTS timescaledb CASCADE;" | sudo -u postgres psql zabbix
Kuna kunna extension kuma ƙirƙira shi don Zabbix database. Mataki na ƙarshe shine ƙirƙirar hypertable.
Taswirar ƙaura zuwa TimecaleDB
Akwai aiki na musamman don wannan create_hypertable:
SELECT create_hypertable(‘history’, ‘clock’, chunk_time_interval => 86400, migrate_data => true);
SELECT create_hypertable(‘history_unit’, ‘clock’, chunk_time_interval => 86400, migrate_data => true);
SELECT create_hypertable(‘history_log’, ‘clock’, chunk_time_interval => 86400, migrate_data => true);
SELECT create_hypertable(‘history_text’, ‘clock’, chunk_time_interval => 86400, migrate_data => true);
SELECT create_hypertable(‘history_str’, ‘clock’, chunk_time_interval => 86400, migrate_data => true);
SELECT create_hypertable(‘trends’, ‘clock’, chunk_time_interval => 86400, migrate_data => true);
SELECT create_hypertable(‘trends_unit’, ‘clock’, chunk_time_interval => 86400, migrate_data => true);
UPDATE config SET db_extension=’timescaledb’, hk_history_global=1, hk_trends_global=1
Aikin yana da sigogi uku. Na farko - tebur a cikin database, wanda kuke buƙatar ƙirƙirar hypertable. Na biyu - filin, bisa ga abin da kuke buƙatar ƙirƙirar chunk_time_interval - tazara na yanki chunks don amfani. A cikin yanayina, tazarar rana ɗaya ce - 86.
Siga ta uku - migrate_data. Idan kun saita true, sannan duk bayanan yanzu ana canjawa wuri zuwa guntun da aka riga aka halicce su. Na yi amfani da shi da kaina migrate_data. Ina da kusan TB 1, wanda ya ɗauki sama da awa ɗaya. Ko da a wasu lokuta, yayin gwaji, na share bayanan tarihi na nau'ikan halayen da ba a buƙata don ajiya ba, don kada in canza su.
Mataki na ƙarshe - UPDATE: a db_extension saka timescaledbta yadda ma’adanar bayanai ta fahimci cewa akwai wannan kari. Zabbix yana kunna shi kuma yana amfani da daidaitattun kalmomi da tambayoyi zuwa ga bayanai - waɗancan fasalulluka waɗanda suka wajaba don TimescaleDB.
Hardware sanyi
Na yi amfani da sabobin biyu. Na farko - VMware inji. Yana da ƙananan ƙananan: 20 Intel® Xeon® CPU E5-2630 v 4 @ 2.20GHz masu sarrafawa, 16 GB na RAM da 200 GB SSD.
Na shigar da PostgreSQL 10.8 akansa tare da Debian 10.8-1.pgdg90+1 OS da xfs tsarin fayil. Na saita komai kadan don amfani da wannan takamaiman bayanai, ban da abin da Zabbix da kanta za ta yi amfani da shi.
A kan wannan na'ura akwai uwar garken Zabbix, PostgreSQL da kaya wakilai. Ina da wakilai 50 masu aiki da suke amfani da su LoadableModuledon samar da sakamako daban-daban da sauri: lambobi, kirtani. Na cika ma'ajiyar bayanai da bayanai masu yawa.
Da farko saitin ya ƙunshi abubuwa 5 bayanai kowane mai masaukin baki. Kusan kowane nau'in yana ɗauke da jan hankali don yin kama da na gaske. A wasu lokuta an sami jan hankali fiye da ɗaya. Ga kumburin hanyar sadarwa ɗaya akwai 3-000 masu tayar da hankali.
Tazarar Sabunta Abun Bayanai - 4-7 seconds. Na tsara nauyin kanta ta hanyar amfani da ba kawai 50 wakilai ba, amma ƙara ƙarin. Hakanan, ta yin amfani da abubuwan bayanai, na daidaita nauyin da ƙarfi kuma na rage tazarar sabuntawa zuwa 4 s.
PostgreSQL. 35 nvps
Gudu na na farko akan wannan kayan masarufi ya kasance akan tsarkakakken PostgreSQL - ƙimar 35 dubu a sakan daya. Kamar yadda kake gani, shigar da bayanai yana ɗaukar ɓangarorin daƙiƙa - komai yana da kyau da sauri. Abinda kawai shine 200 GB SSD faifan ya cika da sauri.
Wannan daidaitaccen dashboard ɗin uwar garken Zabbix ne.
Hoton shuɗi na farko shine adadin ƙimar kowane daƙiƙa. Jaridu na biyu a hannun dama shine lodin ayyukan gini. Na uku yana loda ayyukan ginin ciki: masu daidaita tarihi da mai kula da gida, wanda ke gudana a nan na ɗan lokaci.
Hoto na hudu yana nuna amfani da HistoryCache. Wannan wani nau'i ne na buffer kafin sakawa cikin bayanan. Koren jadawali na biyar yana nuna yadda ake amfani da ValueCache, wato, yawan ValueCache nawa ya buge don jawo - wannan shine ƙimar dubu da yawa a sakan daya.
PostgreSQL. 50 nvps
Sa'an nan na ƙara kaya zuwa 50 dubu dabi'u da biyu a kan wannan hardware.
Lokacin lodawa daga Mai Kula da Gida, shigar da ƙimar 10 ya ɗauki 2-3 seconds.
Tuni ma'aikacin gida ya fara tsoma baki tare da aiki.
Hoto na uku ya nuna cewa, gabaɗaya, nauyin da ke kan masu tarko da masu haɗin gwiwar tarihi har yanzu yana kan 60%. A cikin jadawali na huɗu, HistoryCache ya riga ya fara cikawa sosai yayin aikin Ma'aikata. Ya cika 20%, wanda kusan 0,5 GB ne.
PostgreSQL. 80 nvps
Sa'an nan na ƙara kaya zuwa 80 dubu dabi'u da biyu. Wannan shine kusan abubuwan bayanai 400 da abubuwan jawo 280 dubu.
Kudin ɗorawa na synchers tarihi talatin ya riga ya yi yawa sosai.
Na kuma ƙara sigogi daban-daban: masu daidaita tarihi, caches.
A kan kayan aikina, ɗorawa na syncers tarihi ya ƙaru zuwa matsakaicin. HistoryCache da sauri cike da bayanai - bayanai don sarrafawa sun taru a cikin buffer.
Duk wannan lokacin na lura da yadda ake amfani da na'ura mai sarrafa kwamfuta, RAM da sauran sigogin tsarin, kuma na gano cewa amfani da faifai yana kan iyakarsa.
Na yi nasarar amfani iyakar iyawar faifai akan wannan kayan masarufi da kuma na'urar kama-da-wane. Tare da irin wannan ƙarfin, PostgreSQL ya fara zubar da bayanai sosai, kuma faifan ba ya da lokacin rubutu da karantawa.
Sabar ta biyu
Na ɗauki wani uwar garken, wanda ya riga yana da masu sarrafawa 48 da 128 GB na RAM. Na kunna shi - saita shi zuwa tarihin daidaitawa guda 60, kuma na sami aiki mai karɓuwa.
A gaskiya ma, wannan ya riga ya kasance iyakar yawan aiki inda ake buƙatar yin wani abu.
TimecaleDB. 80 nvps
Babban aikina shine gwada iyawar TimecaleDB akan nauyin Zabbix. 80 dabi'u a sakan daya yana da yawa, yawan adadin ma'auni (sai dai Yandex, ba shakka) da babban "saitin".
Akwai tsomawa a kowane jadawali - wannan shine ainihin ƙauran bayanai. Bayan gazawar a cikin uwar garken Zabbix, bayanin martaba na ɗorawa na tarihin daidaitawa ya canza da yawa - ya faɗi sau uku.
TimecaleDB yana ba ku damar saka bayanai kusan sau 3 cikin sauri kuma ku yi amfani da ƙasa da HistoryCache.
A sakamakon haka, za ku sami bayanai a kan lokaci.
TimecaleDB. 120 nvps
Sa'an nan na ƙara yawan adadin bayanai zuwa dubu 500. Babban aikin shi ne don gwada iyawar TimecaleDB - Na karɓi ƙididdiga na ƙimar 125 dubu a sakan daya.
Wannan "saitin" mai aiki ne wanda zai iya aiki na dogon lokaci. Amma da yake faifan faifai na TB 1,5 ne kawai, na cika shi cikin kwanaki biyu.
Abu mafi mahimmanci shine cewa a lokaci guda an ƙirƙiri sabbin ɓangarori na TimecaleDB.
Wannan gaba ɗaya ba a san shi ba don aiki. Lokacin da aka ƙirƙiri ɓangarori a cikin MySQL, alal misali, komai ya bambanta. Wannan yawanci yana faruwa da dare saboda yana toshe shigarwa gabaɗaya, aiki tare da tebur kuma yana iya haifar da lalacewar sabis. Wannan ba shine lamarin TimecaleDB ba.
A matsayin misali, zan nuna hoto ɗaya daga yawancin jama'a. A cikin hoton, an kunna TimecaleDB, godiya ga wanda nauyin amfani da io.weight akan mai sarrafawa ya ragu. Amfani da abubuwan sarrafawa na ciki kuma ya ragu. Haka kuma, wannan na'ura ce ta yau da kullun akan faifan pancake na yau da kullun, ba SSD ba.
binciken
TimecaleDB shine mafita mai kyau don ƙananan "saitin", wanda ke tasiri aikin diski. Zai ba ka damar ci gaba da aiki da kyau har sai an yi ƙaura zuwa kayan aikin da sauri da sauri.
TimecaleDB yana da sauƙin daidaitawa, yana ba da nasarorin aiki, yana aiki da kyau tare da Zabbix da yana da fa'ida akan PostgreSQL.
Idan kuna amfani da PostgreSQL kuma ba ku shirya canza shi ba, ina ba da shawarar Yi amfani da PostgreSQL tare da tsawo na TimecaleDB tare da Zabbix. Wannan bayani yana aiki yadda ya kamata har zuwa matsakaici "saitin".
Idan muka ce "babban aiki" muna nufin . Ba za ku daɗe da jira don koyo game da fasahohi da ayyuka waɗanda ke ba da damar sabis don hidimar miliyoyin masu amfani ba. Jerin ga Nuwamba 7 da 8 mun riga mun tattara, amma a nan ana iya ba da shawara.
Kuyi subscribing din mu и , wanda a cikinsa ne muka bayyana abubuwan da taron na gaba zai kasance, da kuma gano yadda za mu ci gajiyar shi.
source: www.habr.com