Ichi ndi kupitiriza kwa nkhani yayitali yokhudzana ndi njira yathu yaminga yopanga dongosolo lamphamvu, lolemera kwambiri lomwe limatsimikizira kugwira ntchito kwa Kusinthana. Gawo loyamba lili pano:
Cholakwika chodabwitsa
Pambuyo poyesedwa kangapo, njira yatsopano yogulitsira ndi kuyeretsa idayamba kugwira ntchito, ndipo tidakumana ndi cholakwika chomwe titha kulemba nkhani yachinsinsi.
Atangoyambitsa pa seva yayikulu, imodzi mwazochitazo idakonzedwa ndi cholakwika. Komabe, zonse zinali bwino pa seva yosunga zobwezeretsera. Zinapezeka kuti ntchito yosavuta ya masamu yowerengera exponent pa seva yayikulu idapereka zotsatira zoyipa kuchokera pamkangano weniweni! Tidapitiliza kafukufuku wathu, ndipo mu kaundula wa SSE2 tidapeza kusiyana pang'ono, komwe kumapangitsa kuzungulira pogwira ntchito ndi manambala oyandama.
Tinalemba chida choyesera chosavuta kuti tiwerengere exponent ndi seti yozungulira. Zinapezeka kuti mu mtundu wa RedHat Linux womwe tidagwiritsa ntchito, panali cholakwika pogwira ntchito ndi masamu pomwe chopanda cholakwika chinayikidwa. Tinafotokozera izi kwa RedHat, patapita kanthawi tinalandira chigamba kuchokera kwa iwo ndikuchitulutsa. Cholakwikacho sichinachitikenso, koma sizinadziwike kuti pang'ono izi zidachokera kuti? Ntchitoyi inali ndi udindo wake fesetround kuchokera ku chinenero cha C. Tinasanthula mosamalitsa kachidindo kathu pofufuza zomwe tikuganiza kuti ndi zolakwika: tinayang'ana zochitika zonse zomwe zingatheke; kuyang'ana ntchito zonse zomwe zimagwiritsa ntchito kuzungulira; adayesa kutulutsa gawo lolephera; amagwiritsa ntchito ma compilers osiyanasiyana ndi zosankha zosiyanasiyana; Kusanthula kosasunthika komanso kwamphamvu kunagwiritsidwa ntchito.
Choyambitsa cholakwika sichinapezeke.
Kenako adayamba kuyang'ana zida: adayesa kuyesa kwa ma processor; fufuzani RAM; Tidayesanso zokayikitsa za cholakwika chambiri mu selo limodzi. Palibe phindu.
Pamapeto pake, tinakhazikika pa chiphunzitso chochokera ku dziko la sayansi yamphamvu kwambiri: tinthu tating'ono tamphamvu kwambiri tinawulukira kumalo athu a data, kuboola khoma lamilandu, kugunda purosesa ndikupangitsa kuti latch yoyambitsayo isamamatire pamenepo. Chiphunzitso chopanda pakechi chinatchedwa "neutrino." Ngati muli kutali ndi particle physics: neutrinos pafupifupi samalumikizana ndi dziko lakunja, ndipo ndithudi sangathe kukhudza ntchito ya purosesa.
Popeza sikunali kotheka kupeza chifukwa cha kulephera, seva "yokhumudwitsa" inachotsedwa ntchito pokhapokha.
Patapita nthawi, tinayamba kukonza zosunga zobwezeretsera zotentha: tinayambitsa zomwe zimatchedwa "malo ofunda" (ofunda) - asynchronous replicas. Analandira mtsinje wa zochitika zomwe zikhoza kupezeka m'malo osiyanasiyana a deta, koma kutentha sikunagwirizane ndi ma seva ena.
Chifukwa chiyani izi zidachitika? Ngati seva yosunga zobwezeretsera ikulephera, ndiye kuti kutentha kumangiriridwa pa seva yayikulu kumakhala kusungirako kwatsopano. Izi ndizo, pambuyo polephera, dongosololi silikhala ndi seva imodzi yayikulu mpaka kumapeto kwa gawo la malonda.
Ndipo pamene mtundu watsopano wa makinawo unayesedwa ndikuyamba kugwira ntchito, cholakwika chozungulira chinachitikanso. Komanso, ndi kuchuluka kwa ma seva ofunda, cholakwikacho chinayamba kuwonekera nthawi zambiri. Panthawi imodzimodziyo, wogulitsa analibe chilichonse chosonyeza, popeza panalibe umboni weniweni.
Pakuwunika kotsatiraku, kunabuka chiphunzitso chakuti vutoli likhoza kukhala lokhudzana ndi OS. Tinalemba pulogalamu yosavuta yomwe imayitana ntchito muzitsulo zopanda malire fesetround, amakumbukira momwe zinthu zilili panopa ndikuzifufuza kupyolera mu tulo, ndipo izi zimachitika m'magulu ambiri opikisana. Titasankha magawo ogona komanso kuchuluka kwa ulusi, tidayamba kubweza kulephera pang'ono pambuyo pa mphindi 5 zogwiritsa ntchito. Komabe, chithandizo cha Red Hat sichinathe kuzipanganso. Kuyesedwa kwa ma seva athu ena kwawonetsa kuti okhawo omwe ali ndi mapurosesa ena ndi omwe angatengeke cholakwika. Nthawi yomweyo, kusinthira ku kernel yatsopano kunathetsa vutoli. Pamapeto pake, tinangosintha OS, ndipo chifukwa chenicheni cha cholakwikacho sichinadziwike.
Ndipo mwadzidzidzi chaka chatha nkhani idasindikizidwa pa HabrΓ© "" Zomwe zafotokozedwa mmenemo zinali zofanana kwambiri ndi zathu, koma wolembayo adapitiliza kufufuza ndikuyika chiphunzitso chakuti cholakwikacho chinali mu microcode. Ndipo ma kernels a Linux akasinthidwa, opanga amakonzanso ma microcode.
Kupititsa patsogolo kachitidwe
Ngakhale tidachotsa cholakwikacho, nkhaniyi idatikakamiza kuti tiganizirenso kamangidwe kadongosolo. Ndipotu, sitinatetezedwe ku kubwerezabwereza kwa nsikidzi.
Mfundo zotsatirazi zidapanga maziko owongoleranso kachitidwe kakusungitsa malo:
- Simungakhulupirire aliyense. Ma seva mwina sangagwire bwino ntchito.
- Ambiri kusungitsa.
- Kuonetsetsa mgwirizano. Monga chowonjezera chomveka pakusungitsa ambiri.
- Zolephera kawiri ndizotheka.
- Mphamvu. Chiwembu chatsopano choyimirira chotentha sichiyenera kukhala choyipa kuposa choyambirira. Kugulitsa kuyenera kuchitika mosadodometsedwa mpaka seva yomaliza.
- Kuwonjezeka pang'ono kwa latency. Nthawi iliyonse yopuma imakhala ndi ndalama zambiri.
- Kulumikizana kwakung'ono kwa netiweki kuti musunge latency yotsika momwe mungathere.
- Kusankha master seva yatsopano mumasekondi.
Palibe njira zomwe zilipo pamsika zomwe zidatikwanira, ndipo protocol ya Raft idakali yakhanda, kotero tidapanga yankho lathu.
Networking
Kuphatikiza pa dongosolo losungitsa malo, tinayamba kusinthiratu kulumikizana kwa maukonde. Dongosolo laling'ono la I/O linali ndi njira zambiri, zomwe zidakhudza kwambiri jitter ndi latency. Ndi mazana a njira zogwirira ntchito zolumikizira za TCP, tinakakamizika kusinthana pakati pawo, ndipo pamlingo wa microsecond iyi ndi ntchito yowononga nthawi. Koma choyipa kwambiri ndichakuti njira ikalandira paketi kuti ikasinthidwe, idatumiza pamzere umodzi wa SystemV ndikudikirira chochitika kuchokera pamzere wina wa SystemV. Komabe, pakakhala ma node ambiri, kubwera kwa paketi yatsopano ya TCP mu ndondomeko imodzi ndi kulandila deta pamzere wina kumayimira zochitika ziwiri zotsutsana za OS. Pankhaniyi, ngati palibe mapurosesa akuthupi omwe amapezeka pa ntchito zonse ziwiri, imodzi idzakonzedwa, ndipo yachiwiri idzayikidwa pamzere wodikira. Nβzosatheka kuneneratu zotsatira zake.
M'mikhalidwe yotereyi, kuwongolera kofunikira kwadongosolo kumatha kugwiritsidwa ntchito, koma izi zidzafuna kugwiritsa ntchito mafoni amtundu wazinthu. Zotsatira zake, tidasinthira ku ulusi umodzi pogwiritsa ntchito epoll yachikale, izi zidakulitsa liwiro ndikuchepetsa nthawi yochitira zinthu. Tinachotsanso njira zoyankhulirana zapaintaneti komanso kulumikizana kudzera pa SystemV, tidachepetsa kwambiri mafoni am'dongosolo ndikuyamba kuwongolera zomwe zimafunikira kwambiri. Pa gawo laling'ono la I / O lokha, zinali zotheka kupulumutsa pafupifupi 8-17 ma microseconds, kutengera zomwe zikuchitika. Chiwembu chokhala ndi ulusi umodzichi chakhala chikugwiritsidwa ntchito mosasinthika kuyambira pamenepo; ulusi umodzi wa epoll wokhala ndi malire ndiwokwanira kulumikiza maulumikizidwe onse.
Transaction Processing
Kuchulukirachulukira pamakina athu kumafunikira kukweza pafupifupi zigawo zake zonse. Koma, mwatsoka, kuyimirira pakukula kwa liwiro la wotchi ya purosesa m'zaka zaposachedwa sikunapangitsenso kuti zitheke kukulitsa njira. Chifukwa chake, tidaganiza zogawa njira ya Injini m'magawo atatu, otanganidwa kwambiri ndi njira yowunikira zoopsa, yomwe imayesa kupezeka kwa ndalama muakaunti ndikupanga zochitikazo zokha. Koma ndalama zikhoza kukhala mu ndalama zosiyana, ndipo kunali koyenera kudziwa pazifukwa zomwe zopempha ziyenera kugawidwa.
Yankho lomveka ndikuligawa ndi ndalama: seva imodzi imagulitsa madola, ina mu mapaundi, ndipo yachitatu mu euro. Koma ngati, ndi ndondomeko yotereyi, zochitika ziwiri zimatumizidwa kukagula ndalama zosiyana, ndiye kuti vuto la desynchronization la chikwama lidzabuka. Koma kulunzanitsa ndi kovuta komanso okwera mtengo. Chifukwa chake, zingakhale zolondola kugawa padera ndi ma wallet komanso mosiyana ndi zida. Mwa njira, kusinthanitsa kwamayiko akumadzulo kulibe ntchito yoyang'ana zoopsa monga momwe timachitira, choncho nthawi zambiri izi zimachitika popanda intaneti. Tinafunika kukhazikitsa zotsimikizira pa intaneti.
Tiyeni tifotokoze ndi chitsanzo. Wogulitsa akufuna kugula $ 30, ndipo pempho likupita ku kutsimikiziridwa kwa malonda: timayang'ana ngati wogulitsa uyu amaloledwa ku malonda awa komanso ngati ali ndi ufulu wofunikira. Ngati zonse zili bwino, pempholo limapita ku dongosolo lotsimikizira zoopsa, i.e. kuyang'ana kuchuluka kwa ndalama kuti atsirize malonda. Pali cholembedwa kuti ndalama zofunika panopa oletsedwa. Pempholo limatumizidwa ku dongosolo la malonda, lomwe limavomereza kapena kutsutsa malondawo. Tiyerekeze kuti ntchitoyo ikuvomerezedwa - ndiye kuti dongosolo lotsimikizira zoopsa likuwonetsa kuti ndalamazo sizikutsekedwa, ndipo ma ruble amasanduka madola.
Nthawi zambiri, dongosolo loyang'anira zoopsa lili ndi ma aligorivimu ovuta ndipo limachita mawerengedwe ochulukirachulukira, ndipo samangoyang'ana "malire aakaunti", momwe zingawonekere poyang'ana koyamba.
Pamene tidayamba kugawa ndondomeko ya Injini m'magulu, tinakumana ndi vuto: code yomwe inalipo panthawiyo imagwiritsa ntchito deta yofanana pazigawo zovomerezeka ndi zotsimikizira, zomwe zinkafunika kulemberanso ma code onse. Zotsatira zake, tidabwereka njira yopangira malangizo kuchokera kwa mapurosesa amakono: chilichonse chimagawika m'magawo ang'onoang'ono ndipo zochita zingapo zimachitika mofananira mkombero umodzi.
Pambuyo pakusintha pang'ono kwa kachidindo, tidapanga payipi yolumikizirana yofananira, momwe malondawo adagawidwa m'magawo 4 a payipi: kulumikizana kwa netiweki, kutsimikizira, kuphedwa ndi kufalitsa zotsatira.
Tiyeni tione chitsanzo. Tili ndi machitidwe awiri opangira, serial ndi parallel. Ntchito yoyamba ikufika ndikutumizidwa kuti itsimikizidwe mu machitidwe onse awiri. Kugulitsa kwachiwiri kumafika nthawi yomweyo: mu dongosolo lofanana limatengedwa nthawi yomweyo kukagwira ntchito, ndipo muzotsatira zotsatizana zimayikidwa pamzere wodikira kuti ntchito yoyamba ipitirire pakalipano. Ndiko kuti, mwayi waukulu wokonza mapaipi ndikuti timakonza mzere wothamanga mwachangu.
Umu ndi momwe tidapangira dongosolo la ASTS +.
Zowona, sizinthu zonse zomwe zimakhala zosalala kwambiri ndi ma conveyor. Tiyerekeze kuti tili ndi zochitika zomwe zimakhudza kusanja kwa data mumalonda oyandikana nawo; izi ndizochitika zosinthana. Kugulitsa koteroko sikungachitike mwapaipi chifukwa kungakhudze ena. Izi zimatchedwa ngozi ya data, ndipo zochitika zoterezi zimangokonzedwa mosiyana: pamene zochitika "zofulumira" pamzere zidzatha, payipi imayima, dongosolo limagwira ntchito "lochedwa", ndiyeno limayambitsa payipi kachiwiri. Mwamwayi, kuchuluka kwa zochitika zotere pakuyenda konseko ndi kochepa kwambiri, kotero kuti payipi imayima kawirikawiri kotero kuti sizimakhudza ntchito yonse.
Kenako tinayamba kuthetsa vuto la kulunzanitsa ulusi atatu wakupha. Chotsatira chake chinali kachitidwe kozikidwa pa ring buffer yokhala ndi ma cell osasunthika. Mu dongosolo ili, chirichonse chimadalira pa liwiro la kukonza; deta siinakopedwe.
- Mapaketi onse a netiweki omwe akubwera amalowa gawo logawa.
- Timawayika mumndandanda ndikuyika chizindikiro kuti alipo pagawo #1.
- Ntchito yachiwiri yafika, ikupezekanso pa siteji No.
- Ulusi woyamba wokonza umawona zochitika zomwe zilipo, kuzikonza, ndikuzisunthira ku gawo lotsatira la ulusi wachiwiri wokonza.
- Kenako imayendetsa ntchito yoyamba ndikuyika mbendera yofananira
deleted- tsopano ikupezeka kuti igwiritsidwe ntchito mwatsopano.
Mzere wonse umakonzedwa motere.
Kukonza gawo lililonse kumatenga mayunitsi kapena ma microseconds. Ndipo ngati tigwiritsa ntchito njira zolumikizirana za OS, ndiye kuti tidzataya nthawi yochulukirapo pakulumikizana komweko. Ndicho chifukwa chake tinayamba kugwiritsa ntchito spinlock. Komabe, iyi ndi mawonekedwe oyipa kwambiri munthawi yeniyeni, ndipo RedHat sichimalimbikitsa kuchita izi, chifukwa chake timayika spinlock ya 100 ms, kenako ndikusintha ku semaphore mode kuti tichotse kuthekera kwa kufa.
Zotsatira zake, tapeza magwiridwe antchito pafupifupi 8 miliyoni pa sekondi iliyonse. Ndipo kwenikweni miyezi iwiri pambuyo pake za LMAX Disruptor tidawona kufotokozera kwa dera lomwe lili ndi magwiridwe antchito omwewo.
Tsopano pakhoza kukhala mitundu ingapo ya kuphedwa pa gawo limodzi. Zochita zonse zidakonzedwa m'modzim'modzi, monga momwe adalandirira. Zotsatira zake, ntchito yapamwamba idakwera kuchokera pa 18 mpaka 50 pa sekondi iliyonse.
Kusinthana koyang'anira ngozi
Palibe malire a ungwiro, ndipo posakhalitsa tinayambanso kukonzanso zamakono: mkati mwa ASTS +, tinayamba kusuntha machitidwe oyendetsa zoopsa ndi kuthetsa ntchito kukhala zigawo zodziimira. Tinapanga zomangira zamakono zosinthika komanso mtundu watsopano wowopsa, ndipo tidayesa kugwiritsa ntchito kalasiyo kulikonse komwe kungathekere. fixed_point mmalo mwa double.
Koma vuto linabuka nthawi yomweyo: momwe mungagwirizanitse malingaliro onse abizinesi omwe akhala akugwira ntchito kwa zaka zambiri ndikusamutsira ku dongosolo latsopano? Zotsatira zake, mtundu woyamba wa mawonekedwe a dongosolo latsopanolo unayenera kusiyidwa. Mtundu wachiwiri, womwe ukugwira ntchito pakupanga, umachokera ku code yomweyi, yomwe imagwira ntchito muzogulitsa ndi zoopsa. Pachitukuko, chinthu chovuta kwambiri kuchita chinali git merge pakati pamitundu iwiri. Mnzathu Evgeniy Mazurenok anachita opaleshoniyi mlungu uliwonse ndipo nthawi iliyonse ankatukwana kwa nthawi yaitali.
Posankha dongosolo latsopano, nthawi yomweyo tinayenera kuthetsa vuto la kuyanjana. Posankha basi ya data, kunali koyenera kuonetsetsa kuti jitter yokhazikika komanso latency yochepa. Netiweki ya InfiniBand RDMA inali yoyenera kwambiri pa izi: pafupifupi nthawi yokonza ndi 4 nthawi zochepa kuposa ma 10 G Ethernet network. Koma chomwe chidatikopa kwambiri chinali kusiyana kwa ma percentiles - 99 ndi 99,9.
Inde, InfiniBand ili ndi zovuta zake. Choyamba, API yosiyana - ibverbs m'malo mwa sockets. Kachiwiri, palibe njira zopezera mauthenga otseguka. Tidayesa kupanga tokha, koma zidakhala zovuta kwambiri, kotero tidasankha njira yamalonda - Confinity Low Latency Messaging (omwe kale anali IBM MQ LLM).
Kenako ntchito yogawa bwino dongosolo lachiwopsezo idawuka. Ngati mungochotsa Injini Yowopsa ndipo osapanga node yapakatikati, ndiye kuti zotuluka kuchokera kuzinthu ziwiri zitha kusakanikirana.
Mayankho omwe amatchedwa Ultra Low Latency ali ndi njira yoyitanitsanso: zochitika kuchokera kuzinthu ziwiri zitha kukonzedwa mwanjira yofunikira mutalandira; izi zimakhazikitsidwa pogwiritsa ntchito njira ina yosinthira zidziwitso za dongosolo. Koma sitikugwiritsabe ntchito njira iyi: imasokoneza njira yonseyi, ndipo pamayankho angapo sizimathandizidwa konse. Kuonjezera apo, ntchito iliyonse iyenera kupatsidwa zizindikiro zofananira, ndipo mu ndondomeko yathu makinawa ndi ovuta kwambiri kuti agwiritse ntchito molondola. Chifukwa chake, tidagwiritsa ntchito dongosolo lachikale lokhala ndi meseji broker, ndiye kuti, ndi dispatcher yomwe imagawa mauthenga pakati pa Risk Engine.
Vuto lachiwiri linali lokhudzana ndi mwayi wa kasitomala: ngati pali Zipata Zowopsa zingapo, kasitomala ayenera kulumikizana ndi aliyense wa iwo, ndipo izi zidzafuna kusintha kwa kasitomala wosanjikiza. Tinkafuna kuti tichoke pa izi pakadali pano, kotero mapangidwe a Risk Gateway omwe alipo tsopano amayang'anira njira yonse ya data. Izi zimachepetsa kwambiri kupititsa patsogolo, koma zimathandizira kwambiri kuphatikiza kwadongosolo.
Kubwereza
Dongosolo lathu siliyenera kukhala ndi vuto limodzi, ndiye kuti, zigawo zonse ziyenera kubwerezedwa, kuphatikiza wotumizira uthenga. Tinathetsa vutoli pogwiritsa ntchito dongosolo la CLLM: liri ndi gulu la RCMS momwe otumizira awiri amatha kugwira ntchito mumtundu wa master-slave, ndipo pamene wina alephera, dongosololi limasinthira ku lina.
Kugwira ntchito ndi malo osungirako deta
InfiniBand imakonzedwa kuti igwire ntchito ngati malo ochezera a m'deralo, ndiko kuti, kugwirizanitsa zipangizo zopangira rack, ndipo InfiniBand network sichikhoza kuikidwa pakati pa malo awiri omwe amagawidwa m'madera. Chifukwa chake, tidakhazikitsa mlatho / dispatcher, yomwe imalumikizana ndi kusungirako uthenga kudzera pamanetiweki wamba a Ethernet ndikutumiza zonse ku netiweki yachiwiri ya IB. Pamene tikufunika kusamuka kuchokera ku data center, tikhoza kusankha malo oti tigwire nawo ntchito panopa.
Zotsatira
Zonse zomwe zili pamwambazi sizinachitike nthawi imodzi; zinatengera kangapo kuti apange zomangamanga zatsopano. Tidapanga chithunzicho m'mwezi umodzi, koma zidatenga zaka zopitilira ziwiri kuti zitheke. Tinayesetsa kukwaniritsa mgwirizano wabwino kwambiri pakati pa kuonjezera nthawi yogwiritsira ntchito ndikuwonjezera kudalirika kwadongosolo.
Popeza kuti dongosololi linasinthidwa kwambiri, tinagwiritsa ntchito kubwezeretsa deta kuchokera kuzinthu ziwiri zodziimira. Ngati sitolo ya uthenga sikugwira ntchito bwino pazifukwa zina, mutha kutenga chipikacho kuchokera ku gwero lachiwiri - kuchokera ku Risk Engine. Mfundo imeneyi imawonedwa mu dongosolo lonse.
Mwa zina, tidatha kusunga API yamakasitomala kuti ma broker kapena wina aliyense asafune kukonzanso kwakukulu pakumanga kwatsopano. Tinayenera kusintha mawonekedwe ena, koma panalibe chifukwa chosinthira kwambiri mawonekedwe ogwiritsira ntchito.
Tidatcha mtundu waposachedwa wa nsanja yathu Rebus - ngati chidule chazinthu ziwiri zowoneka bwino pamapangidwe, Risk Engine ndi BUS.
Poyamba, tinkafuna kugawira gawo lokhalokha, koma zotsatira zake zinali njira yogawa kwambiri. Makasitomala tsopano atha kulumikizana ndi Trade Gateway, Clearing Gateway, kapena zonse ziwiri.
Zomwe tidapeza pamapeto pake:
Anachepetsa mlingo wa latency. Pokhala ndi ndalama zochepa, dongosololi limagwira ntchito mofanana ndi lapitalo, koma nthawi yomweyo limatha kupirira katundu wapamwamba kwambiri.
Kuchita kwapamwamba kudakwera kuchoka pa 50 zikwi kufika ku 180 zikwi zogulitsa pamphindikati. Kuwonjezeka kwina kumalepheretsedwa ndi njira yokhayo yofananira.
Pali njira ziwiri zopititsira patsogolo: kufananiza kufananitsa ndikusintha momwe zimagwirira ntchito ndi Gateway. Tsopano Gateways onse amagwira ntchito molingana ndi dongosolo lobwerezabwereza, lomwe, pansi pa katundu wotere, limasiya kugwira ntchito bwino.
Pomaliza, nditha kupereka upangiri kwa iwo omwe akumaliza mabizinesi:
- Khalani okonzekera zoyipa nthawi zonse. Mavuto nthawi zonse amabwera mosayembekezereka.
- Nthawi zambiri zimakhala zosatheka kukonzanso zomangamanga. Makamaka ngati mukufuna kukwaniritsa kudalirika kwakukulu pazizindikiro zingapo. Ma node ochulukira, m'pamenenso zithandizo zambiri zimafunikira kuti zithandizire.
- Mayankho onse achikhalidwe ndi eni ake adzafuna zowonjezera zowonjezera pakufufuza, chithandizo ndi kukonza.
- Osazengereza kuthetsa nkhani za kudalirika kwa dongosolo ndi kuchira pambuyo polephera; ziganizireni poyambira kupanga.
Source: www.habr.com