Momwe kufufuza kwa Yandex.Market kumagwirira ntchito ndi zomwe zimachitika ngati imodzi mwa ma seva ikulephera

Moni, dzina langa ndine Evgeniy. Ndimagwira ntchito mu Yandex.Market search infrastructure. Ndikufuna kuuza gulu la Habr za khitchini yamkati ya Msika - ndipo ndili ndi zambiri zoti ndinene. Choyamba, momwe kusaka kwa Msika kumagwirira ntchito, njira ndi mamangidwe ake. Kodi timatani ndi zochitika zadzidzidzi: chimachitika ndi chiyani ngati seva imodzi ikutsika? Nanga bwanji ngati pali ma seva oterowo 100?

Muphunziranso momwe timagwirira ntchito zatsopano pagulu la maseva nthawi imodzi. Ndi momwe timayesera ntchito zovuta mwachindunji pakupanga, osayambitsa zovuta kwa ogwiritsa ntchito. Mwambiri, momwe Kusaka Kwamsika kumagwirira ntchito kuti aliyense akhale ndi nthawi yabwino.

Pang'ono za ife: vuto lomwe timathetsa

Mukalemba mawu, fufuzani malonda ndi magawo, kapena yerekezerani mitengo m'masitolo osiyanasiyana, zopempha zonse zimatumizidwa ku ntchito yosakira. Kusaka ndiye ntchito yayikulu kwambiri pa Msika.

Timakonza zopempha zonse zakusaka: kuchokera kumasamba a market.yandex.ru, beru.ru, Supercheck service, Yandex.Advisor, mafoni a m'manja. Timaphatikizanso zotsatsa pazotsatira za yandex.ru.

Ndi ntchito yosakira sindikutanthauza kungofufuza kokha, komanso nkhokwe yokhala ndi zonse zomwe zimaperekedwa pa Msika. Mulingo wake ndi uwu: zopempha zopitilira biliyoni zimakonzedwa patsiku. Ndipo zonse ziyenera kugwira ntchito mwachangu, popanda zosokoneza ndipo nthawi zonse zimatulutsa zotsatira zomwe mukufuna.

Kodi ndi chiyani: Zomangamanga zamsika

Ndifotokoza mwachidule kamangidwe kamakono ka Msika. Ikhoza kufotokozedwa momveka bwino ndi chithunzi chomwe chili pansipa:

Tinene kuti sitolo yothandizana nayo imabwera kwa ife. Akuti ndikufuna kugulitsa chidole: mphaka woyipa uyu wokhala ndi squeaker. Ndi mphaka wina wokwiya wopanda chokwinira. Ndi mphaka chabe. Ndiye sitolo iyenera kukonzekera zopereka zomwe Msika umasaka. Sitolo imapanga xml yapadera yokhala ndi zotsatsa ndipo imalumikizana ndi njira yopita ku xml iyi kudzera mu mawonekedwe ogwirizana. Cholozeracho chimatsitsa xml iyi nthawi ndi nthawi, imayang'ana zolakwika ndikusunga zonse munkhokwe yayikulu.

Pali zambiri zosungidwa za xml. Cholozera chofufuzira chimapangidwa kuchokera munkhokwe iyi. Mndandandawu umasungidwa mumtundu wamkati. Pambuyo popanga index, ntchito ya Layout imayiyika ku ma seva.

Chotsatira chake, mphaka wokwiya ndi squeaker amawonekera mu database, ndipo ndondomeko ya paka ikuwonekera pa seva.

Ndikuuzani momwe timasakasaka mphaka mu gawo lazomangamanga.

Kamangidwe kakusaka kwa msika

Tikukhala m'dziko la microservices: pempho lililonse lomwe likubwera market.yandex.ru zimayambitsa ma subqueries ambiri, ndipo mautumiki ambiri amakhudzidwa pakukonza kwawo. Chithunzichi chikuwonetsa zochepa chabe:

Chosavuta pempho pokonza dongosolo

Utumiki uliwonse uli ndi chinthu chodabwitsa - chowerengera chake chomwe chili ndi dzina lapadera:

The balancer imatipatsa ife kusinthasintha kwakukulu pakuwongolera ntchito: mungathe, mwachitsanzo, kuzimitsa ma seva, omwe nthawi zambiri amafunika kuti asinthe. Wolinganiza amawona kuti seva palibe ndipo imangotumiza zopempha ku ma seva ena kapena malo a data. Mukawonjezera kapena kuchotsa seva, katunduyo amagawidwanso pakati pa ma seva.

Dzina lapadera la balancer silidalira pa data center. Utumiki A ukapempha ku B, ndiye kuti mwachisawawa chowerengera B chimalozera pempholo kumalo omwe alipo. Ngati ntchitoyo siyikupezeka kapena palibe malo omwe alipo panopa, pempholi limatumizidwa ku malo ena a deta.

FQDN imodzi yamalo onse opangira data imalola kuti ntchito A ikhale yosiyana ndi malo. Pempho lake la utumiki B lidzakonzedwa nthawi zonse. Kupatulapo ndizochitika pamene ntchitoyo ili m'malo onse a deta.

Koma sizinthu zonse zomwe zimakhala zabwino kwambiri ndi balancer iyi: tili ndi gawo lina lapakati. Chotsaliracho chikhoza kukhala chosakhazikika, ndipo vutoli limathetsedwa ndi ma seva owonjezera. Palinso kuchedwa kwina pakati pa mautumiki A ndi B. Koma muzochita ndi zosakwana 1 ms ndipo kwa mautumiki ambiri izi sizovuta.

Kuchita ndi Zosayembekezeka: Kusakatula kwa Service Bancing ndi Kukhazikika

Tangoganizani kuti pali kugwa: muyenera kupeza mphaka ndi squeaker, koma seva ikugwa. Kapena ma seva 100. Kutuluka bwanji? Kodi tidzasiya wosuta wopanda mphaka?

Mkhalidwewu ndi wowopsa, koma ndife okonzeka. Ndikuuzani mwadongosolo.

Zofufuza zili m'malo angapo a data:

Popanga, timaphatikizapo kuthekera kotseka malo amodzi a data. Moyo uli wodzaza ndi zodabwitsa - mwachitsanzo, wofukula akhoza kudula chingwe chapansi (inde, izo zinachitika). Kuchuluka kwa malo otsala a data kuyenera kukhala kokwanira kupirira kuchuluka kwamphamvu.

Tiyeni tilingalire limodzi data center. Chilichonse cha data chili ndi njira yofananira yochitira:

Balancer imodzi imakhala ndi ma seva atatu akuthupi. Kuperewera uku kumapangidwira kudalirika. Mabalancer amayenda pa HAProx.

Tidasankha HAProx chifukwa cha magwiridwe ake apamwamba, zofunikira zochepa komanso magwiridwe antchito ambiri. Mapulogalamu athu osaka amayendera mkati mwa seva iliyonse.

Kuthekera kwa seva imodzi kulephera ndi kochepa. Koma ngati muli ndi ma seva ambiri, mwayi woti osachepera umodzi utsike ukuwonjezeka.

Izi ndi zomwe zimachitika zenizeni: ma seva akuwonongeka. Chifukwa chake, ndikofunikira kuyang'anira nthawi zonse momwe ma seva onse alili. Seva ikasiya kuyankha, imachotsedwa pamayendedwe. Pachifukwa ichi, HAProxy ili ndi cheke chokhazikika chaumoyo. Imapita ku maseva onse kamodzi sekondi imodzi ndi pempho la HTTP "/ping".

Chinthu china cha HAProxy: cheke-chothandizira chimakulolani kuti muyike ma seva onse mofanana. Kuti tichite izi, HAProxy imagwirizanitsa ndi ma seva onse, ndipo amabwezera kulemera kwawo malinga ndi katundu wamakono kuchokera ku 1 mpaka 100. Kulemera kwake kumawerengedwa malinga ndi chiwerengero cha zopempha pamzere wokonzekera ndi katundu pa purosesa.

Tsopano za kupeza mphaka. Kusaka kumabweretsa zopempha monga: /search?text=ngry+mpaka. Kuti kusaka kukhale kofulumira, mndandanda wonse wa amphaka uyenera kulowa mu RAM. Ngakhale kuwerenga kuchokera ku SSD sikofulumira mokwanira.

Kamodzi pa nthawi, malo osungirako zinthu anali ochepa, ndipo RAM ya seva imodzi inali yokwanira. Pamene maziko operekedwawo akukula, chirichonse sichinalinso mu RAM iyi, ndipo deta inagawidwa m'magawo awiri: shard 1 ndi shard 2.

Koma izi zimachitika nthawi zonse: njira iliyonse, ngakhale yabwino, imayambitsa mavuto ena.

Balancer adapitabe ku seva iliyonse. Koma pamakina omwe pempholo lidabwera, panali theka la index yokha. Zina zonse zinali pa maseva ena. Chifukwa chake, seva idayenera kupita ku makina oyandikana nawo. Pambuyo polandira deta kuchokera ku maseva onse awiri, zotsatirazo zinaphatikizidwa ndikusinthidwanso.

Popeza kuti balancer imagawira zopempha mofanana, ma seva onse adagwira ntchito yokonzanso, osati kungotumiza deta.

Vutoli lidachitika ngati seva yoyandikana nayo sinapezeke. Yankho lake linali kutchula ma seva angapo omwe ali ndi zofunikira zosiyanasiyana monga seva "yoyandikana nayo". Choyamba, pempholi linatumizidwa kwa ma seva omwe ali muzitsulo zamakono. Ngati palibe yankho, pempholo linatumizidwa ku ma seva onse mu data center iyi. Ndipo potsiriza, pempho linapita kumalo ena a deta.
Pamene chiwerengero cha malingaliro chinakula, deta inagawidwa m'magawo anayi. Koma amenewa sanali malire.

Pakalipano, kasinthidwe ka shards eyiti imagwiritsidwa ntchito. Kuphatikiza apo, kuti musunge kukumbukira kochulukirapo, cholozeracho chidagawidwa kukhala gawo lofufuzira (lomwe limagwiritsidwa ntchito posaka) ndi gawo lachidule (lomwe silikuphatikizidwa pakufufuza).

Seva imodzi ili ndi chidziwitso cha shard imodzi yokha. Chifukwa chake, kuti mufufuze mndandanda wathunthu, muyenera kusaka ma seva asanu ndi atatu okhala ndi ma shards osiyanasiyana.

Ma seva amagawidwa m'magulu. Gulu lililonse lili ndi mainjini osakira asanu ndi atatu ndi seva imodzi yazachidule.

Seva ya snippet imayendetsa nkhokwe yamtengo wapatali yokhala ndi deta yosasunthika. Amafunika kuti apereke zikalata, mwachitsanzo, kufotokoza za mphaka ndi squeaker. Deta imasamutsidwa mwapadera ku seva yosiyana kuti isatsegule kukumbukira kwa maseva osakira.

Popeza ma ID a zikalata ndi apadera mkati mwa index imodzi yokha, zinthu zitha kuchitika pomwe mulibe zolemba m'mawu. Chabwino, kapena kuti pa ID imodzi padzakhala zosiyana. Choncho, kuti kufufuzako kugwire ntchito ndi kubwezeredwa, pankafunika kusasinthasintha pakati pa gulu lonse. Ndikufotokozerani pansipa momwe timayang'anira kusasinthika.

Kusaka komweko kumapangidwa motere: pempho losaka likhoza kubwera ku ma seva asanu ndi atatu. Tiyerekeze kuti adabwera ku seva 1. Seva iyi imayendetsa mikangano yonse ndikumvetsetsa zomwe ndi momwe angayang'anire. Malingana ndi pempho lomwe likubwera, seva ikhoza kupempha zowonjezera ku mautumiki akunja kuti mudziwe zofunikira. Pempho limodzi likhoza kutsatiridwa ndi zopempha khumi ku mautumiki akunja.

Pambuyo posonkhanitsa zofunikira, kusaka kumayamba mu database yopereka. Kuti muchite izi, ma subqueries amapangidwa ku ma seva onse asanu ndi atatu pagulu.

Mayankho akalandiridwa, zotsatira zake zimaphatikizidwa. Pamapeto pake, mafunso enanso angapo ku seva ya snippet angafunike kuti apange zotsatira.

Mafunso osakira mgululi amawoneka motere: /shard1?text=ngry+mpaka. Kuphatikiza apo, ma subqueries a mawonekedwe amapangidwa nthawi zonse pakati pa ma seva onse mkati mwa gulu kamodzi sekondi imodzi: /mkhalidwe.

Funsani /mkhalidwe imazindikira vuto pomwe seva palibe.

Imawongoleranso kuti mtundu wa injini zosakira ndi mtundu wa index ndizofanana pa maseva onse, apo ayi padzakhala data yosagwirizana mkati mwa gululo.

Ngakhale kuti seva imodzi ya snippet imayendetsa zopempha kuchokera kumainjini asanu ndi atatu osakira, purosesa yake imakhala yopepuka kwambiri. Chifukwa chake, tsopano tikusamutsa deta yachidule ku ntchito ina.

Kusamutsa deta, tinayambitsa makiyi onse a zolemba. Tsopano ndizosatheka kuti zinthu zibwere kuchokera ku chikalata china pogwiritsa ntchito kiyi imodzi.

Koma kusintha kwa zomangamanga zina sikunathe. Tsopano tikufuna kuchotsa seva yodzipatulira ya snippet. Kenako chokani pagulu lamagulu onse. Izi zidzatithandiza kuti tipitirize kukula mosavuta. Bhonasi yowonjezera ndiyo kupulumutsa kwakukulu kwachitsulo.

Ndipo tsopano ku nkhani zowopsya zokhala ndi mapeto osangalatsa. Tiyeni tikambirane zochitika zingapo za kusapezeka kwa seva.

Chinachake choyipa chachitika: seva imodzi palibe

Tinene seva imodzi palibe. Kenako ma seva otsala mugululi akhoza kupitiliza kuyankha, koma zotsatira zakusaka sizikhala zokwanira.

Pogwiritsa ntchito cheke /mkhalidwe ma seva oyandikana nawo amamvetsetsa kuti imodzi palibe. Chifukwa chake, kuti mukhalebe amphumphu, ma seva onse omwe ali mgulu pa pempho lililonse /ping amayamba kuyankha balancer kuti nawonso sakupezeka. Zikuwonekeratu kuti ma seva onse omwe ali mgululi adamwalira (zomwe sizowona). Ichi ndiye cholepheretsa chachikulu cha dongosolo lathu lamagulu - ndichifukwa chake tikufuna kuchokako.

Zopempha zomwe zalephera ndi zolakwika zimatsutsidwa ndi balancer pa ma seva ena.
Balancer imasiyanso kutumiza kuchuluka kwa ogwiritsa ntchito ku maseva akufa, koma akupitiliza kuyang'ana momwe alili.

Seva ikapezeka, imayamba kuyankha /ping. Mayankho anthawi zonse ku ma pings ochokera ku maseva akufa ayamba kufika, owerengera amayamba kutumiza kuchuluka kwa ogwiritsa ntchito kumeneko. Ntchito yamagulu yabwezeretsedwa, mwachangu.

Choyipa kwambiri: ma seva ambiri sapezeka

Gawo lalikulu la ma seva mu data center amadulidwa. Zoyenera kuchita, kuthamangira kuti? Wolinganiza amabwera kudzapulumutsa kachiwiri. Balancer iliyonse imasunga kukumbukira kuchuluka kwa ma seva amoyo. Imawerengera nthawi zonse kuchuluka kwa magalimoto omwe malo apakati a data amatha kuwongolera.

Pamene ma seva ambiri mu data center akutsika, balancer amazindikira kuti deta iyi sichitha kuyendetsa magalimoto onse.

Ndiye kuchuluka kwa magalimoto kumayamba kugawidwa mwachisawawa kumalo ena a data. Zonse zimagwira ntchito, aliyense ali wokondwa.

Momwe timachitira: kufalitsa zofalitsa

Tsopano tiyeni tikambirane mmene timafalitsira kusintha kwa utumiki. Apa tatenga njira yochepetsera njira: kutulutsa kutulutsidwa kwatsopano kumakhala kongochitika zokha.
Zosintha zina zikasonkhanitsidwa pulojekitiyi, kumasulidwa kwatsopano kumangopangidwa zokha ndipo kumanga kwake kumayamba.

Kenaka ntchitoyi imayendetsedwa ku kuyesa, kumene kukhazikika kwa ntchito kumafufuzidwa.

Nthawi yomweyo, kuyesa kwa magwiridwe antchito kumayambika. Izi zimayendetsedwa ndi ntchito yapadera. Sindilankhula za izi tsopano - kufotokozera kwake kuli koyenera ndi nkhani ina.

Ngati kusindikizidwa pakuyesa kukuyenda bwino, kufalitsa kutulutsidwa mu prestable kumangoyambika. Prestable ndi gulu lapadera lomwe anthu ambiri amagwiritsa ntchito. Ngati ibweza cholakwika, wolinganiza amafunsiranso kuti apange.

Mu prestable, nthawi zoyankhira zimayesedwa ndikufaniziridwa ndi zomwe zidatulutsidwa m'mbuyomu. Ngati zonse zili bwino, ndiye kuti munthu amalumikizana: amayang'ana ma graph ndi zotsatira za kuyezetsa katundu ndikuyamba kutulutsa.

Zabwino zonse zimapita kwa wogwiritsa ntchito: kuyesa kwa A/B

Sizidziwika nthawi zonse ngati kusintha kwa ntchito kumabweretsa phindu lenileni. Kuti muone kufunika kwa kusintha, anthu adabwera ndi kuyesa kwa A / B. Ndikuuzani pang'ono momwe zimagwirira ntchito pakusaka kwa Yandex.Market.

Zonse zimayamba ndikuwonjezera gawo latsopano la CGI lomwe limathandizira magwiridwe antchito atsopano. Lolani parameter yathu ikhale: msika_zatsopano_functionality=1. Kenako mu code timathandiza izi ngati mbendera ilipo:

If (cgi.experiments.market_new_functionality) {
// enable new functionality
}

Zatsopano zikugwira ntchito pakupanga.

Kuti mugwiritse ntchito kuyesa kwa A/B, pali ntchito yodzipatulira yomwe imapereka zambiri zafotokozedwa apa. Kuyesera kumapangidwa mu utumiki. Gawo lamagalimoto limayikidwa, mwachitsanzo, 15%. Maperesenti sanakhazikitsidwe a mafunso, koma kwa ogwiritsa ntchito. Kutalika kwa kuyesera kumasonyezedwanso, mwachitsanzo, sabata.

Zoyeserera zingapo zitha kuyendetsedwa nthawi imodzi. Muzokonda mungatchule ngati mphambano ndi zoyesera zina zingatheke.

Zotsatira zake, ntchitoyi imangowonjezera mkangano msika_zatsopano_functionality=1 mpaka 15% ya ogwiritsa ntchito. Imawerengetseranso ma metric omwe asankhidwa. Kuyeserako kukatsirizika, openda amayang'ana zotsatira ndikupeza mfundo. Kutengera zomwe zapezedwa, chigamulo chimapangidwa kuti chikhazikike pakupanga kapena kukonzanso.

Deft hand la msika: kuyesa pakupanga

Nthawi zambiri zimachitika kuti muyenera kuyesa magwiridwe antchito atsopano pakupanga, koma simukudziwa momwe zingakhalire mu "nkhondo" yolemetsa.

Pali yankho: mbendera mu magawo a CGI angagwiritsidwe ntchito osati kuyesa kwa A / B kokha, komanso kuyesa ntchito zatsopano.

Tinapanga chida chomwe chimakulolani kuti musinthe masinthidwe nthawi yomweyo pamaseva masauzande ambiri osayika ntchito pachiwopsezo. Imatchedwa Stop Tap. Lingaliro loyambirira linali loti mutha kuletsa mwachangu magwiridwe antchito popanda masanjidwe. Kenako chidacho chinakula ndikukhala chovuta kwambiri.

Chiwonetsero chamayendedwe a service chikuwonetsedwa pansipa:

Mitengo ya mbendera imayikidwa kudzera pa API. Ntchito yoyang'anira imasunga izi mu database. Ma seva onse amapita ku database kamodzi pamasekondi khumi aliwonse, tulutsani mitengo ya mbendera ndikugwiritsa ntchito izi pazopempha zilizonse.

Mu Stop tap mutha kukhazikitsa mitundu iwiri yamakhalidwe:

1) Mawu okhazikika. Ikani pamene chimodzi mwazofunikira chiri chowona. Mwachitsanzo:

{
	"condition":"IS_DC1",
	"value":"3",
}, 
{
	"condition": "CLUSTER==2 and IS_BERU", 
	"value": "4!" 
}

Mtengo "3" udzagwiritsidwa ntchito pamene pempho lidzakonzedwa pamalo a DC1. Ndipo mtengo ndi "4" pamene pempho likukonzedwa pa gulu lachiwiri la beru.ru site.

2) Mfundo zopanda malire. Lemberani mwachisawawa ngati palibe zomwe zakwaniritsidwa. Mwachitsanzo:

mtengo, mtengo!

Ngati mtengo utha ndi mawu okweza, amapatsidwa patsogolo kwambiri.

CGI parameter parser imapanga URL. Kenako imagwiritsa ntchito mfundo zochokera ku Stop Tap.

Mfundo zomwe zili ndi zofunika izi zimagwiritsidwa ntchito:

1. Ndi kufunikira kowonjezereka kuchokera ku Stop Tap (chizindikiro chofuula).
2. Mtengo wa pempho.
3. Mtengo wofikira kuchokera ku Stop tap.
4. Mtengo wofikira pamakhodi.

Pali mbendera zambiri zomwe zimawonetsedwa pazotsatira - ndizokwanira pazonse zomwe tikudziwa:

• Data center.
• Chilengedwe: kupanga, kuyesa, mthunzi.
• Malo: msika, beru.
• Nambala ya Cluster.

Ndi chida ichi, mutha kuthandizira magwiridwe antchito pagulu linalake la ma seva (mwachitsanzo, pamalo amodzi okha a data) ndikuyesa magwiridwe antchito awa popanda chiopsezo chautumiki wonse. Ngakhale mutalakwitsa kwambiri kwinakwake, chirichonse chinayamba kugwa ndipo malo onse a deta adatsika, owerengera adzatumiza zopempha kumalo ena a deta. Ogwiritsa ntchito mapeto sazindikira kalikonse.

Ngati muwona vuto, mutha kubweza mbendera nthawi yomweyo ku mtengo wake wakale ndipo zosintha zidzabwezanso.

Utumikiwu ulinso ndi zovuta zake: opanga amaukonda kwambiri ndipo nthawi zambiri amayesa kukankhira zosintha zonse mu Stop Tap. Tikuyesera kulimbana ndi kugwiritsidwa ntchito molakwa.

Njira ya Stop Tap imagwira ntchito bwino mukakhala kale ndi code yokhazikika yokonzeka kutulutsidwa kuti ipangidwe. Panthawi imodzimodziyo, mumakayikirabe, ndipo mukufuna kuyang'ana kachidindo mu "kumenyana".

Komabe, Stop Tap siyoyenera kuyesa panthawi ya chitukuko. Pali gulu lapadera la omanga lotchedwa "shadow cluster".

Kuyesa Kwachinsinsi: Shadow Cluster

Zopempha kuchokera kugulu limodzi zimabwerezedwa kumagulu amithunzi. Koma balancer imanyalanyaza kwathunthu mayankho ochokera kugululi. Chithunzi cha ntchito yake chikuwonetsedwa pansipa.

Timapeza gulu loyesera lomwe lili muzochitika zenizeni za "kumenyana". Magalimoto achizolowezi amapita kumeneko. Ma hardware m'magulu onsewa ndi ofanana, choncho ntchito ndi zolakwika zikhoza kufananizidwa.

Ndipo popeza balancer imanyalanyaza kwathunthu mayankho, ogwiritsa ntchito mapeto sadzawona mayankho kuchokera kugulu lamthunzi. Choncho, sizowopsya kulakwitsa.

anapezazo

Ndiye, tinapanga bwanji Kusaka kwa Msika?

Kuti zonse ziyende bwino, timalekanitsa magwiridwe antchito kukhala mautumiki osiyanasiyana. Mwanjira iyi titha kukulitsa zigawo zomwe tikufuna ndikupangitsa kuti zigawozo zikhale zosavuta. Ndikosavuta kugawa gawo lapadera ku gulu lina ndikugawana maudindo ogwirira ntchito. Ndipo kupulumutsa kwakukulu mu chitsulo ndi njira iyi ndikowonjezera koonekeratu.

Gulu la mthunzi limatithandizanso: titha kupanga mautumiki, kuwayesa panthawiyi komanso osasokoneza wogwiritsa ntchito.

Chabwino, kuyesa mu kupanga, ndithudi. Mukufuna kusintha masinthidwe pa masauzande masauzande a maseva? Zosavuta, gwiritsani ntchito Stop Tap. Mwanjira iyi mutha kutulutsa nthawi yomweyo yankho lokonzekera lokonzekera ndikubwezeretsanso ku mtundu wokhazikika ngati mavuto abuka.

Ndikukhulupirira kuti nditha kuwonetsa momwe timapangira Msika mwachangu komanso mokhazikika ndizomwe zikukula nthawi zonse. Momwe timathetsera mavuto a seva, kuthana ndi zopempha zambiri, kukonza kusinthasintha kwautumiki ndikuchita izi popanda kusokoneza ntchito.

Source: www.habr.com