Kugawidwa kotseka pogwiritsa ntchito Redis

Pa Habr!

Lero tikubweretserani chidwi chanu kumasuliridwa kwa nkhani yovuta yokhudza kukhazikitsidwa kwa kutseka kogawa pogwiritsa ntchito Redis ndikukupemphani kuti mulankhule za chiyembekezo cha Redis ngati mutu. Kuwunika kwa algorithm ya Redlock yomwe ikufunsidwa kuchokera kwa Martin Kleppmann, wolemba bukuli "Mapulogalamu Olemetsa Kwambiri", kupereka apa.

Kutsekera kogawa ndi chinthu chofunikira kwambiri chomwe chimagwiritsidwa ntchito m'malo ambiri momwe njira zosiyanasiyana ziyenera kugwirira ntchito pazogawana zomwe zimagawidwa mokhazikika.

Pali malaibulale angapo ndi zolemba kunja uko zomwe zimalongosola momwe mungagwiritsire ntchito DLM (Distributed Lock Manager) pogwiritsa ntchito Redis, koma laibulale iliyonse imatenga njira yosiyana ndipo zitsimikizo zomwe amapereka ndizochepa kwambiri poyerekeza ndi zomwe zingatheke ndi mapangidwe apamwamba kwambiri.

M'nkhaniyi tiyesera kufotokoza ndondomeko yovomerezeka yovomerezeka yomwe imasonyeza momwe mungagwiritsire ntchito kutsekedwa kogawidwa pogwiritsa ntchito Redis. Tilankhula za algorithm yotchedwa Redlock, imagwiritsa ntchito woyang'anira loko wogawidwa ndipo, m'malingaliro athu, ndondomekoyi ndi yotetezeka kusiyana ndi momwe zimakhalira nthawi imodzi. Tikukhulupirira kuti anthu ammudzi azisanthula, kupereka ndemanga, ndikuzigwiritsa ntchito ngati poyambira ntchito zovuta kapena zina.

Kukhazikitsa

Tisanapitirire ku kufotokozera kwa algorithm, timapereka maulalo angapo okonzekera okonzeka. Zitha kugwiritsidwa ntchito pofotokozera.

• Redlock-rb (kukwaniritsa kwa Ruby). Palinso foloko Redlock-rb, yomwe imawonjezera phukusi (mwala wamtengo wapatali) kuti mugawike mosavuta, osati zokhazo.
• Redlock-py (Python kukhazikitsa).
• Aioredlock (kukwaniritsidwa kwa Asyncio Python).
• Redlock-php (kukhazikitsa kwa PHP).
• PHPRedisMutex (kukhazikitsa kwina kwa PHP)
• cheprasov/php-redis-lock (laibulale ya PHP yamaloko)
• Redsync (kukhazikitsa kwa Go).
• Redisson (kukhazikitsa kwa Java).
• Redis ::DistLock (kukwaniritsa kwa Perl).
• Redlock-cpp (kukhazikitsa kwa C ++).
• Redlock-cs (kukwaniritsa kwa C#/.NET).
• RedLock.net (kukwaniritsidwa kwa C#/.NET). Ndi chithandizo cha async ndi loko yowonjezera.
• ScarletLock (kukhazikitsa C# .NET yokhala ndi sitolo yosinthika ya data)
• Redlock4Net (kukwaniritsidwa kwa C# .NET)
• node-redlock (kukhazikitsa kwa NodeJS). Imaphatikizanso thandizo lokulitsa maloko.

Zitsimikizo za Chitetezo ndi Kupezeka

Tipanga chitsanzo cha kapangidwe kathu ndi zinthu zitatu zokha zomwe tikuganiza kuti zimapereka zitsimikizo zochepa zomwe zimafunikira kuti tigwiritse ntchito bwino zotsekera zogawidwa.

1. Katundu wachitetezo: Kupatula onse. Nthawi iliyonse, kasitomala mmodzi yekha akhoza kugwira loko.
2. Katundu A kupezeka: Palibe zotsekera. Nthawi zonse zimakhala zotheka kupeza loko, ngakhale kasitomala yemwe adatseka gweroyo alephera kapena kugwera pagawo lina la disk.
3. Katundu B: Kulekerera Zolakwa. Bola ma node ambiri a Redis akuyenda, makasitomala amatha kupeza ndikumasula maloko.

Chifukwa kukhazikitsa kutengera kulephera kuchira sikokwanira pankhaniyi
Kuti timvetsetse zomwe titi tichite, tiyeni tiwunike momwe zinthu zilili pano ndi malaibulale otsekera omwe amagawidwa kwambiri kutengera Redis.

Njira yosavuta yotsekera chida pogwiritsa ntchito Redis ndikupanga makiyi pachitsanzo. Nthawi zambiri, fungulo limapangidwa ndi moyo wocheperako, izi zimatheka pogwiritsa ntchito zomwe zimaperekedwa ku Redis, posakhalitsa funguloli limatulutsidwa (katundu 2 pamndandanda wathu). Pamene kasitomala akufunika kumasula gwero, amachotsa kiyi.

Poyang'ana koyamba, yankho ili limagwira ntchito bwino, koma pali vuto: zomanga zathu zimapanga vuto limodzi lolephera. Kodi chimachitika ndi chiyani ngati chitsanzo cha Redis chikalephera? Tiwonjeze kapolo ndiye! Ndipo tidzaigwiritsa ntchito ngati wowonetsa sakupezeka. Tsoka ilo, njira iyi siyotheka. Pochita izi, sitingathe kugwiritsa ntchito bwino zinthu zomwe timafunikira kuti titeteze chitetezo, chifukwa kubwereza mu Redis ndikofanana.

Mwachiwonekere, mtundu woterewu umachitika:

1. Client A amapeza loko pa master.
2. Mbuyeyo akulephera kulemba makiyi asanasamutsidwe kwa kapoloyo.
3. Wotsatira amakwezedwa kukhala mtsogoleri.
4. Makasitomala B amapeza loko pachinthu chomwe A adachitseka kale. KUSINTHA KWA CHITENDERO!

Nthawi zina zimakhala zachilendo kuti muzochitika zapadera, monga kulephera, makasitomala ambiri amatha kugwira loko nthawi imodzi. Zikatero, njira yobwerezabwereza ingagwiritsidwe ntchito. Apo ayi, tikupangira yankho lomwe likufotokozedwa m'nkhaniyi.

Kukhazikitsa koyenera ndi chitsanzo chimodzi

Tisanayese kuthana ndi zophophonya za kasinthidwe kachinthu chimodzi komwe tafotokoza pamwambapa, tiyeni timvetsetse momwe tingagwirire bwino vuto losavutali, popeza yankho ili liri lovomerezeka m'mapulogalamu omwe mtundu wamtundu umavomerezeka nthawi ndi nthawi, komanso chifukwa kutsekereza pa chitsanzo chimodzi chimagwira ntchito ngati maziko omwe amagwiritsidwa ntchito mu algorithm yogawidwa yomwe yafotokozedwa apa.

Kuti mupeze loko, chitani izi:

SET resource_name my_random_value NX PX 30000

Lamuloli limayika kiyi pokhapokha ngati palibe (NX njira), yokhala ndi nthawi yovomerezeka ya 30000 milliseconds (PX option). Kiyi yakhazikitsidwa kuti "myrandomvalue" Mtengowu uyenera kukhala wapadera pamakasitomala onse komanso zopempha zonse za loko.
Kwenikweni, mtengo wachisawawa umagwiritsidwa ntchito kumasula loko mosamala, ndi script ikuwuza Redis: chotsani fungulo ngati liripo ndipo mtengo wosungidwa momwemo ndi womwe unkayembekezeredwa. Izi zimatheka pogwiritsa ntchito Lua script:

if redis.call("get",KEYS[1]) == ARGV[1] then
    return redis.call("del",KEYS[1])
else
    return 0
end

Izi ndizofunikira kuti loko yosungidwa ndi kasitomala wina isachotsedwe. Mwachitsanzo, kasitomala atha kupeza loko, kenako nkuyamba kutsekedwa ntchito yomwe imatha nthawi yayitali kuposa loko yoyamba (kuti kiyiyo ikhale ndi nthawi yotha), kenako ndikuchotsa loko yomwe kasitomala wina adayika.
Kugwiritsa ntchito DEL yosavuta ndikosayenera chifukwa kasitomala amatha kuchotsa loko yosungidwa ndi kasitomala wina. Mosiyana ndi zimenezi, pogwiritsira ntchito script pamwambapa, loko iliyonse "imasaina" ndi chingwe chosasinthika, kotero kuti kasitomala yekha amene adayiyikapo akhoza kuchotsa.

Kodi chingwechi chizikhala chiyani? Ndikuganiza kuti iyenera kukhala 20 byte kuchokera ku / dev / urandom, koma mutha kupeza njira zotsika mtengo zopangira chingwecho kukhala chosiyana mokwanira ndi zolinga zanu. Mwachitsanzo, zingakhale bwino kubzala RC4 ndi /dev/urandom ndiyeno kupanga mtsinje wachinyengo kuchokera pamenepo. Yankho losavuta limaphatikizapo kuphatikiza kwa unix nthawi muzosintha za microsecond kuphatikiza ID ya kasitomala; sizili zotetezeka, koma mwina zimagwira ntchito nthawi zambiri.

Nthawi yomwe timagwiritsa ntchito ngati muyeso wa moyo wa kiyiyo imatchedwa "lock lifetime". Mtengowu ndi nthawi yomwe lokoyo isanatulutsidwe yokha komanso kuchuluka kwa nthawi yomwe kasitomala amafunikira kuti amalize ntchitoyo kasitomala wina asanayambe kutseka chinthucho, popanda kuphwanya zitsimikizo zosiyanitsidwa. Chitsimikizochi chimangokhala pawindo linalake la nthawi, lomwe limayamba kuyambira pomwe loko idagulidwa.

Chifukwa chake takambirana njira yabwino yopezera ndikumasula loko. Dongosolo (ngati tikukamba za dongosolo lomwe silinagawidwe lomwe lili ndi chochitika chimodzi komanso nthawi zonse) ndi lotetezeka. Tiyeni tiwonjezere lingaliro ili ku kachitidwe kogawidwa, komwe tilibe zitsimikizo zotere.

Redlock algorithm

Mtundu wogawidwa wa algorithm umaganiza kuti tili ndi N Redis masters. Ma node awa ndi odziyimira pawokha, kotero sitigwiritsa ntchito kubwerezabwereza kapena njira ina iliyonse yolumikizirana. Tafotokoza kale momwe mungapezere ndi kumasula loko motetezeka panthawi imodzi. Timangoganiza kuti algorithm, mukamagwira ntchito limodzi, idzagwiritsa ntchito njira iyi. M'zitsanzo zathu timayika N mpaka 5, yomwe ndi mtengo wololera. Chifukwa chake, tidzafunika kugwiritsa ntchito 5 Redis masters pamakompyuta osiyanasiyana kapena makina enieni kuti tiwonetsetse kuti amachita mosadalira wina ndi mnzake.

Kuti apeze loko, kasitomala amachita izi:

1. Imapeza nthawi yomwe ilipo mu ma milliseconds.
2. Sequentially kuyesa kupeza loko pazochitika zonse za N, pogwiritsa ntchito dzina lofunikira lomwelo komanso zikhalidwe mwachisawawa nthawi zonse. Mu Gawo 2, pamene kasitomala akhazikitsa loko pang'onopang'ono, kasitomala amagwiritsa ntchito kuchedwa kuti ayipeze yomwe ili yochepa kwambiri poyerekeza ndi nthawi yomwe loko imatulutsidwa. Mwachitsanzo, ngati nthawi yotsekereza ndi masekondi 10, ndiye kuti kuchedwa kungakhale pakati pa ~ 5-50 milliseconds. Izi zimathetsa vuto lomwe kasitomala atha kukhala wotsekedwa kwa nthawi yayitali akuyesera kuti afikire node yolephera ya Redis: ngati chitsanzocho sichikupezeka, ndiye timayesetsa kulumikizana ndi chochitika china posachedwa.
3. Kuti atenge loko, kasitomala amawerengera nthawi yayitali bwanji; Kuti tichite izi, imachotsa ku nthawi yeniyeni mtengo wanthawi yeniyeni yomwe idapezedwa mu gawo 1. Ngati ndipo pokhapokha ngati kasitomala adatha kupeza loko nthawi zambiri (osachepera 3), komanso nthawi yonse yomwe idatenga kupeza loko, kuchepera nthawi yotsekera, loko imatengedwa kuti yapezedwa.
4. Ngati loko yapezedwa, nthawi yokhomayo imatengedwa kukhala nthawi yotsekera yoyambira kuchotsera nthawi yomwe yadutsa yowerengedwa mu gawo 3.
5. Ngati kasitomala alephera kupeza loko pazifukwa zina (mwina sanathe kutseka zochitika za N/2+1, kapena nthawi yotsekera inali yolakwika), ndiye kuti ayesa kumasula nthawi zonse (ngakhale zomwe akuganiza kuti sangatseke. ).

Kodi algorithm ndi asynchronous?

Ma aligorivimuwa amachokera ku lingaliro lakuti, ngakhale kuti palibe wotchi yolumikizidwa yomwe njira zonse zingagwire ntchito, nthawi ya m'deralo mu ndondomeko iliyonse imayendabe pafupifupi mayendedwe omwewo, ndipo cholakwikacho ndi chaching'ono poyerekeza ndi nthawi yonse yomwe lokoko kumayendera. zomasulidwa zokha. Lingaliro ili ndi lofanana kwambiri ndi momwe zimakhalira pamakompyuta wamba: kompyuta iliyonse imakhala ndi wotchi yakumaloko, ndipo nthawi zambiri titha kudalira kuti kusiyana kwa nthawi pakati pa makompyuta osiyanasiyana kumakhala kochepa.

Pakadali pano, tiyenera kupanga mosamalitsa lamulo lathu lopatula: kusagwirizana kumatsimikizika pokhapokha ngati kasitomala akugwira loko akutuluka panthawi yomwe loko kuli kovomerezeka (mtengo uwu wapezeka mu gawo 3), kuchotseratu nthawi yochulukirapo (zochepa zochepa). milliseconds kuti abwezere kusiyana kwa nthawi pakati pa njira).

Nkhani yosangalatsa yotsatirayi ikufotokoza zambiri za machitidwe otere omwe amafunikira kugwirizanitsa kwa nthawi: Kubwereketsa: njira yabwino yololera zolakwika pakugawa mafayilo osasinthika.

Yesaninso kulephera

Wofuna chithandizo akalephera kupeza loko, ayenera kuyesanso pambuyo pochedwa; izi zimachitika kuti aletse-synchronize makasitomala angapo kuyesera kupeza loko pa chinthu chomwecho nthawi imodzi (zomwe zingayambitse "kugawanika-ubongo" momwe mulibe opambana). Kuphatikiza apo, kasitomala akamayesa kupeza loko nthawi zambiri za Redis, kumachepetsa zenera momwe kugawanika kwaubongo kumatha kuchitika (ndiponso kufunikira koyesanso). Chifukwa chake, moyenera, kasitomala ayese kutumiza malamulo a SET ku zochitika za N nthawi imodzi pogwiritsa ntchito kuchulukitsa.

Ndikoyenera kutsindika apa momwe kulili kofunika kwa makasitomala omwe amalephera kupeza maloko ambiri kuti amasule maloko (ochepa) omwe adapeza, kuti asadikire kuti chinsinsi chithe ntchito yotsekera isanapezekenso. (ngakhale ngati kugawanika kwa ma netiweki kumachitika, ndipo kasitomala amasiya kulumikizana ndi zochitika za Redis, ndiye kuti muyenera kulipira chilango chopezeka podikirira kuti kiyi ithe).

Tulutsani loko

Kutulutsa loko ndi ntchito yosavuta yomwe imangofunika kutsegulira zochitika zonse, mosasamala kanthu kuti kasitomala akuwoneka kuti watseka bwino chochitika china.

Malingaliro a Chitetezo

Kodi algorithm ndi yotetezeka? Tiyeni tiyese kulingalira zomwe zimachitika muzochitika zosiyanasiyana.

Poyamba, tiyeni tiyerekeze kuti kasitomala adatha kupeza loko nthawi zambiri. Chitsanzo chilichonse chizikhala ndi kiyi yokhala ndi moyo wofanana kwa onse. Komabe, makiyi aliwonsewa adayikidwa pa nthawi yosiyana, kotero amatha nthawi zosiyanasiyana. Koma, ngati kiyi yoyamba idayikidwa pa nthawi yoyipa kuposa T1 (nthawi yomwe timasankha tisanakumane ndi seva yoyamba), ndipo kiyi yomaliza idayikidwa pa nthawi yoyipa kuposa T2 (nthawi yomwe yankho lidalandiridwa. kuchokera pa seva yomaliza), ndiye tili ndi chidaliro kuti kiyi yoyamba mu seti yomwe yatha ipulumuka osachepera MIN_VALIDITY=TTL-(T2-T1)-CLOCK_DRIFT. Makiyi ena onse adzatha nthawi ina, kotero titha kutsimikiza kuti makiyi onse adzakhala ovomerezeka nthawi imodzi kwanthawiyi.

Panthawi yomwe makiyi ambiri amakhalabe ovomerezeka, kasitomala wina sangathe kupeza loko, popeza ntchito za N/2+1 SET NX sizingapambane ngati makiyi a N/2+1 alipo kale. Choncho, pamene loko yapezedwa, n'zosatheka kupeza kachiwiri pa nthawi yomweyo (izi zikhoza kuphwanya mutual exclusion katundu).
Komabe, tikufuna kuwonetsetsa kuti makasitomala angapo omwe akuyesera kupeza loko nthawi imodzi sangathe kuchita bwino nthawi imodzi.

Ngati kasitomala watseka nthawi zambiri kwanthawi yayitali kapena kupitilira nthawi yotseka, imawona kuti loko ndi kolakwika ndikutsegula. Chifukwa chake, tiyenera kungoganizira momwe kasitomala adatha kuletsa nthawi zambiri munthawi yochepa kuposa tsiku lomaliza. Pankhaniyi, ponena za mkangano womwe uli pamwambapa, panthawiyi MIN_VALIDITY palibe kasitomala amene angathe kutenganso loko. Choncho, makasitomala ambiri adzatha kutseka zochitika za N / 2 + 1 nthawi imodzi (yomwe imatha kumapeto kwa gawo la 2) pokhapokha nthawi yotseka ambiri inali yaikulu kuposa nthawi ya TTL, yomwe imapangitsa kuti loko ikhale yosavomerezeka.

Kodi mungapereke umboni wotsimikizira zachitetezo, kuwonetsa ma aligorivimu omwe alipo, kapena kupeza cholakwika pamwambapa?

Kufikirako

Kupezeka kwadongosolo kumatengera mikhalidwe itatu yayikulu:

1. Kumasula maloko (makiyi akatha ntchito): Makiyi apezekanso kuti agwiritsidwe ntchito pamaloko.
2. Mfundo yakuti makasitomala nthawi zambiri amathandizana wina ndi mzake pochotsa maloko pamene loko yofunidwayo sikunapezeke, kapena yapezedwa ndipo ntchitoyo yatha; kotero ndizotheka kuti sitiyenera kudikirira kuti makiyi atha kuti titengenso loko.
3. Mfundo yoti kasitomala akafuna kuyesanso kupeza loko, imadikirira nthawi yayitali kuposa nthawi yofunikira kuti apeze maloko ambiri. Izi zimachepetsa kuthekera kwa kugawanikana kwaubongo komwe kumachitika popikisana pazachuma.

Komabe, pali chilango chopezeka chofanana ndi TTL ya magawo a maukonde, kotero ngati pali magawo ogwirizana, chilangocho chikhoza kukhala chosatha. Izi zimachitika nthawi iliyonse kasitomala akapeza loko ndikukankhira gawo lina asanatulutse.

M'malo mwake, kupatsidwa magawo osawerengeka a netiweki, makina amatha kukhala osapezeka kwa nthawi yayitali.

Performance, failover ndi fsync

Anthu ambiri amagwiritsa ntchito Redis chifukwa amafunikira magwiridwe antchito apamwamba a loko potengera kuchedwa komwe kumafunikira kuti apeze ndikumasula maloko, komanso kuchuluka kwa zogula / zotulutsidwa zomwe zitha kumalizidwa pamphindikati. Kuti mukwaniritse izi, pali njira yolumikizirana ndi ma seva a N Redis kuti muchepetse latency. Iyi ndi njira yochulukitsa (kapena "kuchulukitsa kwa munthu wosauka", pomwe soketi imayikidwa munjira yosatsekereza, kutumiza malamulo onse, ndikuwerenga malamulo pambuyo pake, poganiza kuti nthawi yobwereranso pakati pa kasitomala ndi nthawi iliyonse ndi yofanana) .

Komabe, tiyeneranso kuganizira zoganizira zomwe zimagwirizanitsidwa ndi kusungidwa kwa deta kwa nthawi yaitali ngati timayesetsa kupanga chitsanzo ndi kuchira kodalirika kuchokera ku zolephera.

Kwenikweni, kuti timveketse nkhaniyi, tiyeni tiyerekeze kuti timakonza Redis popanda kusungidwa kwanthawi yayitali konse. Wothandizira amatha kuletsa zochitika 3 mwa 5. Chimodzi mwazochitika zomwe kasitomala adatha kuletsa zimayambiranso, ndipo pakadali pano pali zochitika za 3 kachiwiri pazomwezo, zomwe tingathe kuletsa, ndipo kasitomala wina akhoza, kuletsa zomwe zayambikanso, kuphwanya chitetezo chomwe chilipo. amangotengera maloko okha.

Ngati mutsegula deta patsogolo (AOF), zinthu zidzasintha pang'ono. Mwachitsanzo, mutha kulimbikitsa seva potumiza lamulo la SHUTDOWN ndikuyiyambitsanso. Popeza kuti ntchito zotha ntchito ku Redis zimayendetsedwa mwachidwi kotero kuti nthawi imapitilira kuyenda ngakhale seva itazimitsidwa, zofunikira zathu zonse zili bwino. Izi ndizabwinobwino bola ngati kutseka kwabwinoko kutsimikizike. Zoyenera kuchita ngati magetsi azimitsidwa? Ngati Redis imakonzedwa mwachisawawa, ndi fsync synchronizing pa disk sekondi iliyonse, ndiye kuti ndizotheka kuti titayambiranso sitidzakhala ndi kiyi yathu. Mwachidziwitso, ngati tikufuna kutsimikizira chitetezo cha loko nthawi iliyonse tikayambiranso, tiyenera kuyatsa fsync=always m'makonzedwe a kusungirako deta kwa nthawi yaitali. Izi zidzaphatu magwiridwe antchito, mpaka pamlingo wa machitidwe a CP omwe mwamwambo amagwiritsidwa ntchito pokhazikitsa zotsekera zogawidwa.

Koma zinthu zili bwino kuposa mmene zimaonekera poyamba. M'malo mwake, chitetezo cha algorithm chimasungidwa chifukwa chiwonetserochi chikayambikanso chikalephera, sichitenga nawo gawo pazotseka zilizonse zomwe zikugwira ntchito pano.

Kuti titsimikizire izi, tikungofunika kuwonetsetsa kuti pakalephera chitsanzocho sichipezeka kwa nthawi yayitali kuposa kuchuluka kwa TTL komwe timagwiritsa ntchito. Mwanjira iyi tidzadikirira mpaka tsiku lotha ntchito ndikutulutsa zokha makiyi onse omwe anali akugwira ntchito panthawi yakulephera.

Pogwiritsa ntchito kuyambiranso kochedwa, ndizotheka kupeza chitetezo ngakhale pakalibe kulimbikira kwanthawi yayitali mu Redis. Dziwani, komabe, kuti izi zitha kubweretsa chindapusa chophwanya mwayi wopezeka. Mwachitsanzo, ngati nthawi zambiri zalephera, dongosololi silidzapezeka padziko lonse lapansi ku TTL (ndipo palibe chothandizira chomwe chingatsekedwe panthawiyi).

Timawonjezera kupezeka kwa algorithm: timakulitsa kutsekereza

Ngati ntchito yochitidwa ndi makasitomala imakhala ndi masitepe ang'onoang'ono, ndizotheka kuchepetsa nthawi yotsekera ndikukhazikitsa njira yowonjezera zotsekera. M'malo mwake, ngati kasitomala ali wotanganidwa ndi makompyuta ndipo mtengo wake umatsika kwambiri, mutha kutumiza zolemba za Lua nthawi zonse zomwe zimakulitsa TTL ya fungulo ngati fungulo likadalipo ndipo mtengo wake ukadali mtengo wachisawawa womwe umapezeka. loko anapezedwa.

Wofuna chithandizo akuyenera kungoganizira loko yoti atengedwenso ngati atseka nthawi zambiri mkati mwa nthawi yovomerezeka.

Zowona, mwaukadaulo ma aligorivimu sasintha, chifukwa chake kuchuluka kwa zoyeserera mobwerezabwereza kuti mupeze maloko kuyenera kukhala kochepa, apo ayi zopezeka zidzaphwanyidwa.

Source: www.habr.com