Kuyesa pa Prod: Kutumiza kwa Canary

Mbalameyi ndi mbalame yaing'ono yomwe imayimba nthawi zonse. Mbalamezi zimakhudzidwa kwambiri ndi methane ndi carbon monoxide. Ngakhale kuchokera ku mpweya wochepa kwambiri mumlengalenga, amataya chikumbumtima kapena kufa. Ofukula golide ndi oyendetsa migodi anatenga mbalame kupita ku mgodi: pamene canaries ikuimba, mukhoza kugwira ntchito, ngati muli chete - pali mpweya mu mgodi ndipo ndi nthawi yoti muchoke. Ogwira ntchito m’migodiwo anapereka nsembe ya mbalame yaing’ono kuti ituluke m’migodi yamoyo.

Mchitidwe wofananawo wapezeka mu IT. Mwachitsanzo, mu ntchito yokhazikika yotumiza mtundu watsopano wa ntchito kapena kugwiritsa ntchito kupanga ndi kuyesa zisanachitike. Malo oyeserera amatha kukhala okwera mtengo kwambiri, zoyeserera zokha sizimaphimba chilichonse chomwe mukufuna, komanso kusayesa komanso kudzipereka ndikowopsa. Apa ndipamene njira ya Canary Deployment imabwera bwino, pomwe magalimoto ena enieni opanga amalunjika ku mtundu watsopano. Njirayi imathandiza mosamala onani mtundu watsopano za kupanga, kupereka nsembe pang'ono pazifukwa zazikulu. Zambiri za momwe njirayo imagwirira ntchito, zomwe zili zothandiza komanso momwe mungagwiritsire ntchito, zidzatiuza Andrey Markelov (Andrey_V_Markelov), pa chitsanzo cha kukhazikitsa mu kampani Infobip.

Andrey Markelov - Lead Software Engineer ku Infobip, wakhala akupanga mapulogalamu a Java pankhani yazachuma ndi matelefoni kwa zaka 11. Amapanga zinthu za Open Source, amatenga nawo mbali mu Atlassian Community ndikulemba mapulagini azinthu za Atlassian. Mlaliki wa Prometheus, Docker ndi Redis.

Zambiri pa Infobip

Ndi nsanja yapadziko lonse lapansi yolumikizirana yomwe imalola mabanki, ogulitsa, ogulitsa pa intaneti ndi makampani oyendetsa magalimoto kutumiza mauthenga kwa makasitomala awo kudzera pa SMS, kukankha, makalata ndi mauthenga amawu. Mu bizinesi yotereyi, kukhazikika ndi kudalirika ndizofunikira kuti makasitomala alandire mauthenga pa nthawi yake.

Infobip IT zomangamanga mu manambala:

• Ma data 15 padziko lonse lapansi;
• 500 ntchito zapadera zomwe zikugwira ntchito;
• 2500 zochitika za mautumiki, zomwe ziri zambiri kuposa malamulo;
• 4,5 TB magalimoto pamwezi;
• manambala amafoni 4,5 biliyoni;

Bizinesi ikukula, komanso kuchuluka kwa zotulutsidwa. Timawononga ndalama 60 zotulutsidwa patsikuchifukwa makasitomala amafuna zambiri ndi mphamvu. Koma izi ndizovuta - pali mautumiki ambiri, koma malamulo ochepa. Muyenera kulemba msanga code yomwe iyenera kugwira ntchito popanga popanda zolakwika.

Zomasulidwa

Kutulutsidwa kwanthawi zonse kumapita motere. Mwachitsanzo, pali ntchito A, B, C, D ndi E, aliyense wa iwo amapangidwa ndi gulu osiyana.

Panthawi ina, gulu la utumiki A lisankha kutumiza mtundu watsopano, koma magulu a mautumiki B, C, D ndi E sakudziwa za izo. Pali njira ziwiri za momwe gulu lautumiki A lidzachitire.

Agwira kumasulidwa kowonjezereka: choyamba sinthani mtundu umodzi, kenako wachiwiri.

Koma pali njira yachiwiri: lamulo adzapeza zina zowonjezera ndi makina, tumizani mtundu watsopano, ndiyeno sinthani rauta, ndipo mtunduwo uyamba kugwira ntchito yopanga.

Mulimonsemo, nthawi zonse pamakhala mavuto pambuyo potumiza, ngakhale mtunduwo utayesedwa. Mukhoza kuyesa pamanja, mukhoza kuchita izo zokha, simungathe kuyesa - mavuto adzauka mulimonse. Njira yosavuta komanso yolondola kwambiri yowathetsera ndikubwerera ku mtundu womwe ukugwira ntchito. Pokhapokha mungathe kuthana ndi zowonongeka, ndi zomwe zimayambitsa ndikuzikonza.

Ndiye tikufuna chiyani?

Sitifunikira mavuto. Ngati makasitomala aziwapeza mwachangu kuposa ife, zingawononge mbiri yawo. Choncho tiyenera kupeza mavuto mofulumira kuposa makasitomala. Kugwira ntchito mwachangu, timachepetsa kuwonongeka.

Pa nthawi yomweyo, tikufuna kufulumizitsa kutumizakotero kuti zichitike mofulumira, mosavuta, mwachibadwa komanso popanda kukakamizidwa ndi gulu. Mainjiniya, mainjiniya a DevOps ndi opanga mapulogalamu ayenera kutetezedwa - kutulutsidwa kwa mtundu watsopano ndikodetsa nkhawa. Gulu sichitha, timayesetsa kugwiritsa ntchito mwanzeru chuma cha anthu.

Nkhani zotumizidwa

Kuchuluka kwamakasitomala sikudziwika. Sizingatheke kuneneratu pamene magalimoto a kasitomala adzakhala otsika kwambiri. Sitikudziwa komwe kapena nthawi yomwe makasitomala ayambire kampeni yawo - mwina usikuuno ku India, mawa ku Hong Kong. Poganizira kusiyana kwakukulu kwa nthawi, kutumiza ngakhale pa 2 koloko sikutsimikizira kuti makasitomala sadzakhudzidwa.

Mavuto othandizira. Atumiki ndi opereka chithandizo ndi othandizana nawo. Nthawi zina amakhala ndi kuwonongeka komwe kumayambitsa zolakwika pakutumiza kwatsopano.

Magulu Ogawidwa. Magulu omwe amapanga mbali ya kasitomala ndi backend ali m'madera osiyanasiyana a nthawi. Chifukwa cha ichi, nthawi zambiri sangagwirizane pakati pawo.

Malo a data sangathe kubwerezedwa pa siteji. Pali ma rack 200 pamalo amodzi a data - simungathe kubwereza izi mu sandbox.

Nthawi zopumazosavomerezeka! Tili ndi Bajeti Yolakwika pamene tikugwira ntchito 99,99% ya nthawiyo, mwachitsanzo, ndipo peresenti yotsalayo ndi "malire olakwika". Kukwaniritsa kudalirika kwa 100% sikungatheke, koma ndikofunikira kuyang'anira nthawi zonse madontho ndi nthawi yopuma.

Classic zothetsera

Lembani code popanda zolakwika. Pamene ndinali wachinyamata wopanga mapulogalamu, mameneja anandiyandikira ndi pempho loti nditulutse popanda nsikidzi, koma izi sizingatheke nthawi zonse.

Lembani mayeso. Mayesero amagwira ntchito, koma nthawi zina osati momwe bizinesi ikufunira. Kupeza ndalama si ntchito ya mayeso.

Yesani pa siteji. Kwa zaka 3,5 za ntchito yanga ku Infobip, sindinawonepo kuti momwe gawoli likuyendera limagwirizana pang'ono ndi kupanga.

Tinayeseranso kupanga lingaliro ili: poyamba tinali ndi siteji, kenako kupanga chisanadze, ndiyeno kupanga chisanadze. Koma izi sizinathandizenso - iwo sanagwirizane ngakhale mu mphamvu. Ndi siteji, titha kutsimikizira magwiridwe antchito, koma sitikudziwa momwe ingagwire ntchito modzaza.

Kutulutsidwa kumapangidwa ndi amene adayambitsa. Izi ndizochitika zabwino: ngakhale wina atasintha dzina la ndemangayo, amawonjezera nthawi yomweyo pakupanga. Izi zimathandiza kukulitsa udindo komanso osaiwala za kusintha komwe kunachitika.

Palinso zovuta zina. Ndizosautsa kwa wopanga mapulogalamu kuti awononge nthawi yambiri akuyang'ana chilichonse.

Zotulutsidwa Zogwirizana. Izi nthawi zambiri zimaperekedwa ndi oyang'anira: "Tiyeni tivomereze kuti tsiku lililonse mudzayesa ndikuwonjezera zatsopano." Sichigwira ntchito: nthawi zonse pamakhala lamulo lodikirira wina aliyense, kapena mosemphanitsa.

Mayeso a utsi

Njira ina yothetsera mavuto athu otumizidwa. Ganizirani momwe mayeso a utsi amagwirira ntchito muchitsanzo cham'mbuyomu, pamene gulu A likufuna kutumiza mtundu watsopano.

Choyamba, gulu limagwiritsa ntchito chitsanzo chimodzi pakupanga. Mauthenga ku chitsanzo kuchokera ku nthabwala amatsanzira magalimoto enienikuti mufanane ndi kuchuluka kwa magalimoto tsiku ndi tsiku. Ngati zonse zili bwino, gululo limasintha mtundu watsopano kukhala wogwiritsa ntchito.

Njira yachiwiri ndikutumiza ndi chitsulo chowonjezera. Gululo limayesa kuti lipangidwe, kenako limasintha, ndipo zonse zimagwira ntchito.

Kuipa kwa mayeso a utsi:

• Mayesero sangadaliridwe. Mungapeze kuti magalimoto ofanana ndi opanga? Mutha kugwiritsa ntchito dzulo kapena sabata yapitayo, koma sizimafanana nthawi zonse.
• Zovuta kusamalira. Muyenera kusunga maakaunti oyeserera, kuwakonzanso nthawi zonse musanatumizidwe, zolemba zogwira zikatumizidwa kumalo osungirako. Izi ndizovuta kuposa kulemba mayeso mu sandbox yanu.

Bonasi yokhayo apa ndi ntchito akhoza kuyesedwa.

Kutulutsa kwa Canary

Chifukwa cha kuchepa kwa mayeso a utsi, tinayamba kugwiritsa ntchito zotulutsa za canary.

Mchitidwe wofanana ndi momwe anthu ogwira ntchito m'migodi amagwiritsira ntchito canaries kuti asonyeze mlingo wa mpweya womwe umalowa mu IT. Timalola ena enieni kupanga magalimoto kwa Baibulo latsopanopoyesera kukwaniritsa Mgwirizano wa Utumiki wa Utumiki (SLA). SLA ndi "ufulu wathu wolakwitsa", zomwe tingagwiritse ntchito kamodzi pachaka (kapena kwa nthawi ina). Ngati zonse zikuyenda bwino, tidzawonjezera magalimoto ambiri. Ngati sichoncho, tidzabwezeranso zomasulira zam'mbuyomu.

Kukhazikitsa ndi ma nuances

Kodi tinagwiritsa ntchito bwanji zolemba za canary? Mwachitsanzo, gulu lamakasitomala limatumiza mauthenga kudzera muutumiki wathu.

Kutumiza kumapita motere: timachotsa node imodzi pansi pa balancer (1), kusintha mtundu (2) ndikulola magalimoto ena padera (3).

Kawirikawiri, aliyense mu gululo adzakhala osangalala, ngakhale wogwiritsa ntchito mmodzi sakusangalala. Ngati zonse zili bwino, timasintha mitundu yonse.

Ndikuwonetsa mwadongosolo momwe zimawonekera ma microservices nthawi zambiri.

Pali Service Discovery ndi ntchito zina ziwiri: S1N1 ndi S2. Ntchito yoyamba (S1N1) imadziwitsa Service Discovery ikayamba, ndipo Service Discovery imakumbukira. Utumiki wachiwiri wokhala ndi mfundo ziwiri (S2N1 ndi S2N2) umadziwitsanso Service Discovery ikayamba.

Utumiki wachiwiri woyamba umagwira ntchito ngati seva. Woyamba amafunsa Service Discovery kuti adziwe zambiri za ma seva ake, ndipo akalandira, amawayang'ana ndi kuwayang'ana ("cheke chaumoyo"). Akayang'ana, amawatumizira mameseji.

Pamene wina akufuna kuyika mtundu watsopano wa utumiki wachiwiri, amauza Service Discovery kuti node yachiwiri idzakhala canary node: kuchepa kwa magalimoto kudzatumizidwa kwa izo, chifukwa kutumizidwa kudzachitika tsopano. Timachotsa node ya canary kuchokera pansi pa balancer ndipo ntchito yoyamba situmiza magalimoto kwa iyo.

Timasintha mtunduwo ndipo Service Discovery ikudziwa kuti node yachiwiri tsopano ndi canary - mutha kuyipatsa zochepa (5%). Ngati zonse zili bwino, timasintha mtunduwo, kubweza katunduyo ndikugwira ntchito.

Kuti tichite izi, tifunika:

• kusanja;
• kuyang'anirachifukwa ndikofunikira kudziwa zomwe wogwiritsa ntchito aliyense amayembekezera komanso momwe ntchito zathu zimagwirira ntchito mwatsatanetsatane;
• kusanthula mtundukumvetsetsa momwe Baibulo latsopanoli lidzagwirira ntchito popanga;
• zochita zokha - timalemba ndondomeko yotumizira (paipi yotumizira).

Kusamala

Ichi ndi chinthu choyamba chimene tiyenera kuganizira. Pali njira ziwiri zofananira.

Njira yosavuta pamene node imodzi nthawi zonse imakhala canary. Node iyi nthawi zonse imalandira magalimoto ochepa ndipo timayamba kutumizidwa kuchokera pamenepo. Pakakhala zovuta, tidzafanizira ntchito yake isanatumizidwe komanso panthawi yake. Mwachitsanzo, ngati pali zolakwika zina 2, ndiye kuti kuwonongeka kwawonjezeka 2 nthawi.

Node ya Canary imayikidwa panthawi yotumiza. Kutumiza kukatha ndipo timachotsa malo a canary node kuchokera pamenepo, kuchuluka kwa magalimoto kudzabwezeretsedwa. Pokhala ndi magalimoto ochepa, tipeza kugawidwa koyenera.

Kuwunikira

Mwala wapangodya wa kutulutsidwa kwa canary. Tiyenera kumvetsetsa chifukwa chomwe tikuchitira izi komanso ma metric omwe tikufuna kusonkhanitsa.

Zitsanzo za ma metric omwe timatolera kuzinthu zathu.

• Chiwerengero cha zolakwika, zomwe zimalembedwa ku zipika. Ichi ndi chizindikiro chodziwikiratu kuti zonse zikuyenda momwe ziyenera kukhalira. Kawirikawiri, iyi ndi metric yabwino.
• Nthawi Yochita Mafunso (kuchedwa). Aliyense amayang'anira metric iyi chifukwa aliyense amafuna kugwira ntchito mwachangu.
• Kukula kwa mzere (kupitilira).
• Chiwerengero cha mayankho opambana pa sekondi iliyonse.
• Nthawi yochitira 95% ya zopempha zonse.
• Miyezo yamabizinesi: ndi ndalama zingati zomwe bizinesi imapanga mu nthawi yoperekedwa kapena kusokoneza kwa ogwiritsa ntchito. Ma metric awa a mtundu wathu watsopano akhoza kukhala wofunikira kwambiri kuposa omwe adawonjezedwa ndi mainjiniya.

Zitsanzo za ma metrics m'makina otchuka kwambiri owunikira.

Kauntala. Izi ndi zina zowonjezera mtengo, mwachitsanzo, kuchuluka kwa zolakwika. Metric iyi ndiyosavuta kutanthauzira ndikuwerenga tchati: dzulo panali zolakwika 2, ndipo lero 500, zomwe zikutanthauza kuti china chake chalakwika.

Kuchuluka kwa zolakwika pamphindi kapena pamphindi imodzi ndiye chizindikiro chofunikira kwambiri chomwe chitha kuwerengedwa pogwiritsa ntchito Counter. Deta iyi imapereka chithunzi chomveka bwino cha momwe dongosolo limagwirira ntchito patali. Ganizirani chitsanzo cha graph ya chiwerengero cha zolakwika pa sekondi imodzi kwa mitundu iwiri ya machitidwe opanga.

Panali zolakwika zochepa mu mtundu woyamba, mwina kafukufukuyo sanagwire ntchito. Mu Baibulo lachiwiri, chirichonse chiri choipa kwambiri. Titha kunena motsimikiza kuti pali zovuta, chifukwa chake tiyenera kubweza bukuli.

Gauge. Ma metrics ndi ofanana ndi Counter, koma timalemba zinthu zomwe zitha kuwonjezeka kapena kutsika. Mwachitsanzo, nthawi yofunsa mafunso kapena kukula kwa mzere.

Grafu ikuwonetsa chitsanzo cha latency. Grafu ikuwonetsa kuti matembenuzidwewo ndi ofanana, mutha kugwira nawo ntchito. Koma ngati muyang'anitsitsa, mukhoza kuona momwe mtengo umasinthira. Ngati nthawi yofunsa mafunso ikuwonjezeka pamene ogwiritsa ntchito akuwonjezeredwa, ndiye kuti nthawi yomweyo pali mavuto - izi sizinali choncho kale.

Chidule. Chimodzi mwa zizindikiro zofunika kwambiri pa bizinesi ndi percentiles. Metric ikuwonetsa izi 95% ya milandu dongosolo lathu limagwira ntchito momwe ife tikufunira. Titha kuvomereza ngati pali mavuto kwinakwake, chifukwa timamvetsetsa momwe zinthu zilili, momwe zonse zilili zabwino kapena zoyipa.

Zida

Mtengo wa ELK. Mutha kugwiritsa ntchito canary pogwiritsa ntchito Elasticsearch - timalemba zolakwika pazochitikazo. Ndi kuyimba kosavuta kwa API, mutha kupeza kuchuluka kwa zolakwika nthawi iliyonse ndikuyerekeza ndi magawo akale: GET /applg/_cunt?q=level:errr.

Prometheus. Adadziwonetsa bwino ku Infobip. Zimakuthandizani kuti mugwiritse ntchito ma metrics amitundu yosiyanasiyana chifukwa zilembo zimagwiritsidwa ntchito.

Titha kugwiritsa ntchito level, instance, service, phatikizani iwo mu dongosolo limodzi. Ndi chithandizo offset mukhoza kuwona, mwachitsanzo, mtengo wamtengo wapatali sabata yapitayo ndi lamulo limodzi lokha GET /api/v1/query?query={query}kumene {query}:

rate(logback_appender_total{ 
    level="error",  
    instance=~"$instance" 
}[5m] offset $offset_value)

Kusanthula Baibulo

Pali njira zingapo zosinthira.

Onani ma metric a canary node okha. Chimodzi mwazosankha zosavuta: tumizani mtundu watsopano ndikuphunzira ntchito yokhayo. Koma ngati injiniya pa nthawi ino akuyamba kuphunzira zipika, nthawi zonse mantha reloading masamba, ndiye yankho si wosiyana ndi ena.

Node ya Canary imafananizidwa ndi mfundo ina iliyonse. Uku ndikufanizira ndi zochitika zina zomwe zimayenderana ndi anthu ambiri. Mwachitsanzo, ngati zinthu zikuipiraipira ndi magalimoto ang'onoang'ono, kapena osachita bwino kuposa zochitika zenizeni, ndiye kuti pali cholakwika.

Node ya Canary ikufaniziridwa yokha m'mbuyomu. Node zoperekedwa ku canary zitha kufananizidwa ndi mbiri yakale. Mwachitsanzo, ngati zonse zinali bwino sabata yapitayi, ndiye kuti tikhoza kuyang'ana pa deta iyi kuti timvetse momwe zinthu zilili panopa.

Zodzichitira

Tikufuna kumasula mainjiniya kuti asayerekezere pamanja, chifukwa chake ndikofunikira kukhazikitsa makina. Njira yotumizira nthawi zambiri imawoneka motere:

• timayamba;
• chotsani node pansi pa balancer;
• kupanga canary node;
• tsegulani balancer ndi kuchuluka kwa magalimoto;
• yerekezerani.

Panthawi imeneyi, timakhazikitsa kuyerekeza kodziwikiratu. Momwe zimawonekera komanso chifukwa chake zili bwino kuposa kutsimikizira pambuyo potumizidwa, tiyeni tiwone chitsanzo kuchokera kwa Jenkins.

Ili ndiye payipi yopita ku Groovy.

while (System.currentTimeMillis() < endCanaryTs) {
    def isOk = compare(srv, canary, time, base, offset, metrics)
    if (isOk) {
        sleep DEFAULT SLEEP
    }   else {
        echo "Canary failed, need to revert"  
        return false
    }
}

Pano mu loop timayika kuti tidzafanizira node yatsopano kwa ola limodzi. Ngati ndondomeko ya canary siinathe ntchitoyi, timayitcha ntchitoyo. Ananena kuti zonse zili bwino kapena ayi: def isOk = compare(srv, canary, time, base, offset, metrics).

Ngati zonse zili bwino - sleep DEFAULT SLEEP, mwachitsanzo, kwa mphindi imodzi, ndikupitiriza. Ngati sichoncho, tulukani - kutumiza kwalephera.

Kufotokozera kwa metric. Tiyeni tiwone momwe ntchitoyi ingawonekere compare pa chitsanzo cha DSL.

metric(
    'errorCounts',
    'rate(errorCounts{node=~"$canaryInst"}[5m] offset $offset)',
    {   baseValue, canaryValue ->
        if (canaryValue > baseValue * 1.3) return false 
        return true
    }
)

Tiyerekeze kuti tikufanizira kuchuluka kwa zolakwika ndipo tikufuna kudziwa kuchuluka kwa zolakwika pamphindi imodzi kwa mphindi zisanu zapitazi.

Tili ndi mfundo ziwiri: maziko ndi canary node. Mtengo wa canary node ndi womwe ulipo. Choyambira - baseValue ndi mtengo wa node ina iliyonse yopanda canary. Timafanizira zikhalidwe ndi wina ndi mzake molingana ndi chilinganizo, chomwe timayika kutengera zomwe takumana nazo komanso zomwe taziwona. Ngati mtengo canaryValue zoipa, ndiye kutumiza kunalephera, ndipo timabwerera mmbuyo.

N’chifukwa chiyani zonsezi zili zofunika?

Munthu sangayang'ane mazana ndi masauzande a metricsmakamaka kuchita mwamsanga. Kufananitsa kokha kumathandizira kuyang'ana ma metrics onse ndikukudziwitsani mwachangu zamavuto. Nthawi ya chenjezo ndi yofunika kwambiri: ngati chinachake chinachitika mu masekondi a 2 otsiriza, kuwonongeka sikudzakhala kwakukulu ngati kuti zinachitika mphindi 15 zapitazo. Mpaka wina awona vuto, alembe kuti atithandizire, ndikutithandizira kuti tibwerere, mutha kutaya makasitomala.

Ngati ndondomekoyi yadutsa ndipo zonse zili bwino, tidzatumiza ma node ena onse okha. Panthawi imeneyi, mainjiniya sachita chilichonse. Pokhapokha akayambitsa canary ndipamene amasankha zomwe angatenge, nthawi yayitali yofananiza, ndi njira yoti agwiritse ntchito.

Ngati pali zovuta, timangobweza ma canary node, kugwira ntchito pamatembenuzidwe am'mbuyomu ndikukonza zolakwika zomwe tapeza. Ndi ma metrics, ndizosavuta kupeza ndikuwona kuwonongeka kwa mtundu watsopano.

Zopinga

Inde, izi si zophweka kukhazikitsa. Choyamba, muyenera General monitoring system. Mainjiniya ali ndi ma metric awo, othandizira ndi owunika ali ndi ma metric osiyanasiyana, ndipo mabizinesi ali ndi achitatu. Dongosolo wamba ndi chilankhulo chofala chomwe bizinesi ndi chitukuko zimalankhula.

Iyenera kuyesedwa muzochita kukhazikika kwa metric. Kuyang'ana kumakuthandizani kumvetsetsa ndi ma metrics ochepera otani omwe amafunikira kuti atsimikizire mtundu.

Kodi kukwaniritsa izi? Gwiritsani ntchito canary-service osati panthawi yotumiza. Timawonjezera ntchito inayake pamtundu wakale, womwe nthawi iliyonse ukhoza kutenga node iliyonse yodzipatulira, kuchepetsa magalimoto popanda kutumizidwa. Tikayerekeza: timaphunzira zolakwa ndikuyang'ana mzere umenewo pamene tikwaniritsa khalidwe.

Kodi tinapindula bwanji ndi mabuku a canary?

Kuchepetsa kuwonongeka kwa nsikidzi. Zolakwika zambiri zotumizira zimachitika chifukwa cha kusagwirizana kwa data ina kapena zofunika kwambiri. Pali zolakwika zochepa ngati izi, chifukwa titha kuthetsa vutoli mumasekondi oyamba.

Kukhathamiritsa ntchito yamagulu. Oyamba kumene ali ndi "ufulu wolakwitsa": akhoza kupititsa patsogolo kupanga popanda kuopa kulakwitsa, pali njira yowonjezera, kulimbikitsa kugwira ntchito. Ngati aphwanya chinachake, ndiye kuti sichidzakhala chotsutsa, ndipo wolakwayo sadzachotsedwa ntchito.

Kutumiza kwamagetsi. Izi sizilinso ndondomeko yamanja, monga kale, koma yeniyeni yeniyeni. Koma zimatenga nthawi yaitali.

Ma metrics ofunikira. Kampani yonse, kuyambira mabizinesi ndi mainjiniya, imamvetsetsa zomwe zili zofunika kwambiri pazogulitsa zathu, ma metric, mwachitsanzo, kutuluka ndi kulowa kwa ogwiritsa ntchito. Timayendetsa ndondomekoyi: timayesa ma metrics, timayambitsa zatsopano, tiwone momwe zakale zimagwirira ntchito kuti tipange dongosolo lomwe lingapangitse ndalama kuti zikhale zopindulitsa.

Tili ndi machitidwe ambiri abwino komanso machitidwe omwe amatithandiza. Ngakhale zili choncho, timayesetsa kukhala akatswiri ndikuchita bwino ntchito yathu, mosasamala kanthu kuti tili ndi dongosolo lomwe lingatithandize kapena ayi.

Njira zama engineering ndi machitidwe - Cholinga chachikulu cha TechLead Conf. Ngati mwachita bwino panjira yopita kuukadaulo ndipo mwakonzeka kukuuzani zomwe zidakuthandizani pa izi, - funsani lipoti.

Tikukonzekera kutero Tech Lead Conf Juni 8. Timamvetsetsa kuti n'zovuta kupanga zisankho zokhudzana ndi kutenga nawo mbali pamsonkhanowu tsopano. Koma nthawi yomweyo, timakhulupirira kuti kukhala kwaokha si chifukwa choyimitsa kulumikizana ndi akatswiri. Chifukwa chake, mulimonse, tipeza njira yokambilana za ntchito zaukadaulo ndi njira zowathetsera - ngati kuli kofunikira, tidzapita pa intaneti ndikukhazikitsa ma network pamenepo!

Source: www.habr.com