Limodzi mwamavuto omwe ogulitsa mapulogalamu amitundu yambiri nthawi zambiri amakumana nawo ndi kubwereza kwa luso la mainjiniya - omanga, oyesa, ndi oyang'anira zomangamanga - pafupifupi gulu lililonse. Izi zikugwiranso ntchito kwa mainjiniya okwera mtengo - akatswiri pantchito yoyezetsa katundu.
M'malo mochita ntchito zawo zachindunji ndikugwiritsa ntchito luso lawo lapadera kuti apange njira yoyesera katundu, sankhani njira, ma metrics abwino ndikulemba zoyeserera molingana ndi mbiri ya katundu, mainjiniya nthawi zambiri amayenera kuyika zida zoyeserera kuyambira poyambira, kukonza zida zonyamula katundu, ndikuziyika. okha mu machitidwe a CI, kukhazikitsa kuwunika ndi kufalitsa malipoti.
Mutha kupeza mayankho kumavuto amabungwe poyesa omwe timagwiritsa ntchito ku Positive Technologies in . Ndipo mu iyi, ndilankhula za kuthekera kophatikiza mayeso olemetsa mu payipi wamba wa CI pogwiritsa ntchito lingaliro la "kuyesa katundu ngati ntchito" (kuyesa katundu ngati ntchito). Muphunzira momwe ndi zithunzi za docker zomwe zingagwiritsidwe ntchito paipi ya CI; momwe mungalumikizire magwero a katundu ku polojekiti yanu ya CI pogwiritsa ntchito template yomanga; momwe mapaipi owonetsera amawonekera poyesa kuyesa katundu ndikusindikiza zotsatira. Nkhaniyi ikhoza kukhala yothandiza kwa akatswiri oyesa mapulogalamu ndi mainjiniya odzipangira okha ku CI omwe akuganiza za kamangidwe kakatundu wawo.
Chofunika cha lingaliro
Lingaliro la kuyezetsa katundu ngati ntchito limatanthawuza kuthekera kophatikiza zida zonyamula Apache JMeter, Yandex.Tank ndi zomangira zanu munjira yophatikizira mosalekeza. Chiwonetserocho chidzakhala cha GitLab CI, koma mfundo zake ndizofanana ndi machitidwe onse a CI.
Kuyesa katundu ngati ntchito ndi ntchito yapakati yoyesera katundu. Kuyesa kwa katundu kumayendetsedwa m'mayiwe odzipereka, zotsatira zake zimasindikizidwa zokha mu GitLab Pages, Influx DB ndi Grafana kapena m'machitidwe ofotokozera mayeso (TestRail, ReportPortal, etc.). Makina ndi makulitsidwe amakhazikitsidwa mosavuta momwe angathere - powonjezera ndikuyika template yanthawi zonse ya gitlab-ci.yml mu projekiti ya GitLab CI.
Ubwino wa njirayi ndikuti zida zonse za CI, zonyamula katundu, zithunzi za docker za malo onyamula, mapaipi oyesera, ndi malipoti osindikiza zimasungidwa ndi dipatimenti yodziyimira payokha (DevOps engineers), pomwe mainjiniya oyesa katundu amatha kuyang'ana zoyesayesa zawo pakukulitsa mayeso. ndi kusanthula zotsatira zawo, popanda kuthana ndi zovuta za zomangamanga.
Kuti tifotokoze mophweka, tidzaganiza kuti ntchito yomwe tikufuna kapena seva yomwe ikuyesedwa yatumizidwa kale ndikukonzedwa pasadakhale (zolemba zokha mu Python, SaltStack, Ansible, etc. zitha kugwiritsidwa ntchito pa izi). Kenako lingaliro lonse la kuyezetsa katundu ngati ntchito limalowa m'magawo atatu: kukonzekera, kuyesa, kufalitsa malipoti. Zambiri pachithunzichi (zithunzi zonse ndizodina):
Malingaliro oyambira ndi matanthauzidwe pakuyesa katundu
Pochita mayeso olemetsa, timayesetsa kutsatira , gwiritsani ntchito mawu oyenerera ndi miyeso yoyenera. Ndipereka mndandanda waufupi wamalingaliro akulu ndi matanthauzidwe pakuyesa katundu.
Wothandizira katundu - makina enieni omwe ntchitoyo idzayambitsidwe - gwero la katundu (Apache JMeter, Yandex.Tank kapena gawo lodzilemba lokha).
Cholinga cha mayeso (chofuna) - seva kapena pulogalamu yomwe imayikidwa pa seva yomwe imayenera kutumizidwa.
Zoyeserera (zoyeserera) - seti ya masitepe okhala ndi magawo: zochita za ogwiritsa ntchito ndi zomwe akuyembekezera pazochita izi, zopempha zokhazikika pamanetiweki ndi mayankho, kutengera magawo omwe atchulidwa.
Mbiri kapena dongosolo lotsegula (mbiri) - mkati (Ndime 4.2.4, p. 43) katundu mbiri amatanthawuza ma metrics kuti n'zofunika kwambiri kwa mayeso enaake ndi options kusintha magawo katundu pa mayeso. Mutha kuwona zitsanzo zambiri pachithunzichi.
Yesani - script yokhala ndi magawo okonzedweratu.
Mapulani oyesera (mapulani oyesa) - mndandanda wa mayesero ndi mbiri ya katundu.
Testran (matenda) - kubwereza kamodzi koyesa mayeso amodzi ndi zomwe zachitika mokwanira komanso lipoti lolandilidwa.
Pempho la netiweki (pempho) - Pempho la HTTP lotumizidwa kuchokera kwa wothandizira kupita ku chandamale.
Mayankho pa netiweki (mayankho) - Yankho la HTTP lotumizidwa kuchokera ku chandamale kupita kwa wothandizira.
Khodi yoyankhira ya HTTP (mayankhidwe a HTTP) - nambala yoyankhira yokhazikika kuchokera ku seva yofunsira.
Kuchita ndi kuyankha kwathunthu kwa pempho. Kugulitsa kumawerengedwa kuyambira poyambira kutumiza pempho (pempho) mpaka kumaliza kulandira yankho (yankho).
Momwe mukuchitira - ngati zinali zotheka kumaliza kuyankha kopempha. Ngati panali cholakwika chilichonse panjira iyi, ndiye kuti ntchito yonseyo imawonedwa kuti sinapambane.
Nthawi yoyankhira (latency) - nthawi kuyambira kumapeto kwa kutumiza pempho (pempho) mpaka kuyamba kulandira yankho (yankho).
Kwezani ma metric - Makhalidwe a ntchito yodzaza ndi katundu wothandizira amatsimikiziridwa panthawi yoyesa katundu.
Ma metrics oyambira kuyeza magawo a katundu
Zina mwazomwe zimagwiritsidwa ntchito kwambiri komanso zolimbikitsidwa munjira (tsamba 36, ββ52) ma metrics akuwonetsedwa mu tebulo ili m'munsimu. Ma metrics ofanana a wothandizira ndi chandamale amalembedwa pamzere womwewo.
Metrics kwa katundu wothandizira
Ma metrics a dongosolo lomwe mukufuna kapena pulogalamu yomwe ikuyesedwa ikulemedwa
Chiwerengero cha vCPU ndi kukumbukira Ram,
litayamba - "Iron" makhalidwe a katundu wothandizira
CPU, Memory, kugwiritsa ntchito Disk - Mphamvu za CPU, kukumbukira ndi kutsitsa disk
poyesa. Kawirikawiri amayezedwa ngati peresenti ya
kuchuluka komwe kulipo
kutulutsa kwa netiweki (pa katundu wothandizira) - throughput
mawonekedwe a netiweki pa seva,
kumene wothandizira katundu waikidwa.
Nthawi zambiri amayezedwa ma byte pa sekondi imodzi (bps)
kutulutsa kwa netiweki(pa chandamale) - network interface bandwidth
pa seva chandamale. Nthawi zambiri amayezedwa ma byte pa sekondi imodzi (bps)
Ogwiritsa ntchito- kuchuluka kwa ogwiritsa ntchito,
kukhazikitsa zochitika zolemetsa ndi
kutengera zochita zenizeni za ogwiritsa ntchito
Mawonekedwe a ogwiritsa ntchito, Wadutsa/Walephera/Okwana β chiwerengero cha opambana ndi
zolephera za ogwiritsa ntchito
kwa zochitika zolemetsa, komanso chiwerengero chawo chonse.
Nthawi zambiri zimayembekezeredwa kuti ogwiritsa ntchito onse adatha kumaliza
ntchito zanu zonse zafotokozedwa mu mbiri ya katundu.
Cholakwika chilichonse chidzatanthauza kuti wosuta weniweni sangathe
kuthetsa vuto lanu pamene mukugwira ntchito ndi dongosolo
Zopempha pa sekondi iliyonse (mphindi)- kuchuluka kwa zopempha zapaintaneti pamphindi (kapena mphindi).
Chikhalidwe chofunikira cha wothandizira katundu ndi kuchuluka kwa zopempha zomwe zingapangitse.
M'malo mwake, uku ndikutsanzira mwayi wogwiritsa ntchito pulogalamuyi ndi ogwiritsa ntchito
Mayankho pa sekondi iliyonse (mphindi)
- kuchuluka kwa mayankho pa intaneti pamphindikati (kapena mphindi).
Chikhalidwe chofunikira cha ntchito yomwe mukufuna: zingati
kupanga ndi kutumiza mayankho ku mafunso ndi
katundu wothandizira
Mayankhidwe a HTTP- chiwerengero cha ma code osiyanasiyana
kuchokera pa seva yofunsira yomwe idalandiridwa ndi wothandizira katundu.
Mwachitsanzo, 200 OK amatanthauza kuyimba kopambana,
ndi 404 - kuti gwero silinapezeke
Latency (nthawi yoyankha) - nthawi kuchokera kumapeto
kutumiza pempho (pempho) musanayambe kulandira yankho (yankho).
Nthawi zambiri amayezedwa mu milliseconds (ms)
Nthawi yoyankha- nthawi yantchito imodzi yathunthu,
kutsirizitsa kuzungulira kwa pempho-mayankhidwe.
Iyi ndi nthawi kuyambira pachiyambi kutumiza pempho (pempho)
mpaka kumaliza kulandira yankho (yankho).
Nthawi yamalonda imatha kuyezedwa mumasekondi (kapena mphindi)
m'njira zingapo: ganizirani zochepa,
pazipita, pafupifupi ndipo, mwachitsanzo, 90 peresenti.
Zowerengera zochepa komanso zochulukirapo ndizokwera kwambiri
mawonekedwe a machitidwe.
Gawo la makumi asanu ndi anayi ndilomwe limagwiritsidwa ntchito kwambiri,
monga zikuwonetsa ambiri ogwiritsa ntchito,
kugwira ntchito momasuka pakhomo la machitidwe
Zochita pa sekondi iliyonse (mphindi) - chiwerengero chokwanira
zochita pa sekondi iliyonse (mphindi),
ndiko kuti, kuchuluka kwa ntchito kunatha kuvomera ndi
ndondomeko zopempha ndi kupereka mayankho.
M'malo mwake, uku ndiko kutulutsa kwadongosolo
Momwe mukuchitira , Zadutsa / Zalephera / Chiwerengero - chiwerengero
zopambana, zolephereka komanso kuchuluka kwazomwe zachitika.
Kwa ogwiritsa ntchito enieni sizinaphule kanthu
kugulitsako kudzatanthauzadi
kulephera kugwira ntchito ndi dongosolo pansi pa katundu
Load Testing Schematic Diagram
Lingaliro la kuyezetsa katundu ndilosavuta kwambiri ndipo lili ndi magawo atatu, omwe ndanena kale: Konzekerani-Kuyesa-Lipoti, ndiko kuti, kukonzekera zolinga zoyesa ndikuyika magawo a zotengera katundu, ndiyeno kuyesa zolemetsa ndipo, pamapeto pake, kupanga ndi kusindikiza lipoti loyesa.
Ndemanga zadongosolo:
- QA.Tester ndi katswiri pakuyesa katundu,
- Target ndi ntchito yomwe mukufuna kudziwa momwe imagwirira ntchito.
Wopanga magulu, magawo ndi masitepe pachithunzichi
Masitepe ndi masitepe
Chikuchitikandi chiyani
Zomwe zili pakhomo
Zotulutsa zake ndi chiyani
Konzekerani: siteji yokonzekera kuyesa
LoadParameters
Kukhazikitsa ndi kuyambitsa
wosuta
katundu parameters,
kusankha kwa metrics ndi
mayeso dongosolo kukonzekera
(mbiri yodzaza)
Zosankha zamakonda za
katundu wothandizira kuyambitsa
Mayeso dongosolo
Cholinga choyesa
VM
Kutumiza kwamtambo
makina enieni ndi
zofunika makhalidwe
Zokonda pa VM za katundu wothandizira
Zolemba zokha za
Kupanga kwa VM
VM yosinthidwa
mtambo
Tumizani
Kukhazikitsa ndi kukonza OS
chilengedwe cha
ntchito yonyamula katundu
Zokonda zachilengedwe za
katundu wothandizira
Zolemba zokha za
zokonda zachilengedwe
Malo okonzekera:
OS, ntchito ndi ntchito,
zofunika kuntchito
katundu wothandizira
LoadAgents
Kuyika, kasinthidwe ndi parameterization
katundu wothandizira.
Kapena kutsitsa chithunzi cha docker kuchokera
preconfigured load source
Lowetsani chithunzi cha docker
(YAT, JM kapena chimango chodzilemba chokha)
Zokonda
katundu wothandizira
Konzani ndikukonzekera
ntchito katundu wothandizira
Mayeso: siteji yoyeserera zolemetsa. Magwero ndi othandizira omwe amatumizidwa m'madziwe odzipereka a GitLab CI
katundu
Kuyambitsa Katundu Wothandizira
ndi dongosolo loyesera losankhidwa
ndi katundu magawo
Zosankha Zogwiritsa Ntchito
za kuyambika
katundu wothandizira
Mayeso dongosolo
Cholinga choyesa
Zipika za kuphedwa
katundu mayesero
Zolemba zadongosolo
Mphamvu zakusintha kwa ma metric a zolinga ndi katundu wothandizira
Thamanga Agents
Kuphedwa kwa Agent
zolemba zambiri zoyeserera
malinga ndi
katundu mbiri
Katundu Wothandizira
pofuna kuyesa
Mayeso dongosolo
Cholinga choyesa
zipika
Kusonkhanitsa zipika "zaiwisi".
panthawi yoyezetsa katundu:
zizindikiro za ntchito ya wothandizira katundu,
mkhalidwe wa cholinga cha mayeso
ndi VM yoyendetsa wothandizira
Zipika za kuphedwa
katundu mayesero
Zolemba zadongosolo
Miyala
Kusonkhanitsa ma metrics "yaiwisi" poyesa
Mphamvu zakusintha kwa ma metric a zolinga
ndi katundu wothandizira
Lipoti: siteji yokonzekera lipoti la mayeso
Generator
Processing anasonkhanitsidwa
loading system ndi
monitoring system "yaiwisi"
metrics ndi zipika
Kupanga lipoti mu
mawonekedwe owerengeka aumunthu
zotheka ndi maelementi
openda
Zipika za kuphedwa
katundu mayesero
Zolemba zadongosolo
Mphamvu zakusintha kwa ma metrics
chandamale ndi katundu wothandizira
Zolemba "zaiwisi" zokonzedwa
m'njira yoyenera
zokwezedwa kumalo osungirako akunja
Lipoti la static katundu,
zowerengeka ndi anthu
kufalitsa
Kusindikizidwa kwa lipoti
za katundu
kuyesa kunja
utumiki
Zokonzedwa "raw"
zipika m'njira yoyenera
kwa kutsitsa kupita kunja
nkhokwe
Kusungidwa kunja
malipoti osungira pa
katundu, oyenera
zowunikira anthu
Kulumikiza magwero a katundu mu template ya CI
Tiyeni tipitirire ku gawo lothandiza. Ndikufuna kuwonetsa ma projekiti ena pakampani takhazikitsa lingaliro la kuyesa katundu ngati ntchito.
Choyamba, mothandizidwa ndi mainjiniya athu a DevOps, tidapanga gulu lodzipereka la othandizira ku GitLab CI kuti tiyesetse mayeso. Kuti tisawasokoneze mu ma templates ndi ena, monga maiwe ochitira misonkhano, tidawonjezera ma tag kwa othandizira awa, : katundu. Mutha kugwiritsa ntchito ma tag ena aliwonse omveka. Amafunsa Othamanga a GitLab CI.
Momwe mungadziwire mphamvu yofunikira ndi hardware? Makhalidwe a othandizira katundu - chiwerengero chokwanira cha vCPU, RAM ndi Disk - akhoza kuwerengedwa potengera kuti Docker, Python (ya Yandex.Tank), GitLab CI wothandizira, Java (ya Apache JMeter) iyenera kukhala ikuyenda pa wothandizira. . Kwa Java pansi pa JMeter, ikulimbikitsidwanso kugwiritsa ntchito osachepera 512 MB ya RAM ndipo, ngati malire apamwamba, .
Chifukwa chake, kutengera zomwe takumana nazo, timalimbikitsa kugwiritsa ntchito osachepera 4 vCPU, 4 GB RAM, 60 GB SSD kwa othandizira katundu. Mayendedwe a khadi la netiweki amatsimikiziridwa potengera zofunikira za mbiri yonyamula.
Timagwiritsa ntchito magwero awiri - Apache JMeter ndi Yandex.Tank docker zithunzi.
ndi chida chotseguka chochokera ku Yandex choyesa kuyesa katundu. Kapangidwe kake kamene kamakhazikitsidwa ndi Phantom's high-performance asynchronous hit-based HTTP application jenereta. Tanki ili ndi kuwunika kokhazikika kwazinthu za seva yomwe ikuyesedwa kudzera pa protocol ya SSH, imatha kuyimitsa mayesowo pamikhalidwe yodziwika, imatha kuwonetsa zotsatira zonse mu console komanso mawonekedwe a ma graph, mutha kulumikiza ma module anu. kwa izo kukulitsa magwiridwe antchito. Mwa njira, tidagwiritsa ntchito Tank pomwe inali isanakwane. M'nkhani yakuti "Β»mutha kuwerenga nkhani ya momwe tidayesa nayo katundu mu 2013 ndi chimodzi mwazinthu zamakampani athu.
ndi chida chotseguka choyezera katundu kuchokera ku Apache. Itha kugwiritsidwa ntchito mofanana poyesa ma intaneti osasunthika komanso osinthika. JMeter imathandizira ma protocol ambiri ndi njira zolumikizirana ndi mapulogalamu: HTTP, HTTPS (Java, NodeJS, PHP, ASP.NET, etc.), SOAP / REST Webservices, FTP, TCP, LDAP, SMTP(S), POP3( S) ) ndi IMAP(S), nkhokwe kudzera pa JDBC, amatha kulamula zipolopolo ndikugwira ntchito ndi zinthu za Java. JMeter ili ndi IDE yopangira, kukonza zolakwika ndi kukonza mapulani oyesa. Palinso CLI yogwiritsira ntchito mzere wamalamulo pamakina aliwonse ogwiritsira ntchito Java (Linux, Windows, Mac OS X). Chidachi chikhoza kupanga lipoti la mayeso a HTML.
Kuti mugwiritse ntchito mosavuta mkati mwa kampani yathu, kuti oyesawo azitha kusintha ndikuwonjezera chilengedwe, tidapanga zithunzi za docker zonyamula katundu pa GitLab CI ndikufalitsa mkati. . Izi zimapangitsa kuti zikhale zofulumira komanso zosavuta kuzilumikiza m'mapaipi poyesa katundu. Momwe mungapangire docker kukankhira ku registry kudzera pa GitLab CI - onani .
Tatenga fayilo yofunikira iyi ya Yandex.Tank:
Dockerfile
1 | FROM direvius/yandex-tank
2 | ENTRYPOINT [""]Ndipo kwa Apache JMeter iyi:
Dockerfile
1 | FROM vmarrazzo/jmeter
2 | ENTRYPOINT [""]Mutha kuwerenga momwe dongosolo lathu lophatikizira limagwirira ntchito m'nkhaniyi "".
Template ndi njira
Chitsanzo cha template yochitira mayeso a katundu chikupezeka mu polojekitiyi . The Mutha kuwerenga malangizo ogwiritsira ntchito template. Mu template yokha (file ) pali zolemba za zomwe sitepe iliyonse ili nayo.
Tsambali ndi losavuta kwambiri ndipo likuwonetsa magawo atatu a kuyezetsa katundu omwe akufotokozedwa mu chithunzi pamwambapa: kukonzekera, kuyesa, ndi kusindikiza malipoti. Udindo pa izi : Konzekerani, Yesani ndi Lipoti.
- Gawo ziyenera kugwiritsidwa ntchito pokonzeratu zoyeserera kapena kuyang'ana kupezeka kwake. Chilengedwe cha magwero a katundu sichiyenera kukonzedwa, amamangidwa kale ngati zithunzi za docker ndikuyikidwa mu registry ya docker: ingotchulani mtundu womwe mukufuna pagawo la Mayeso. Koma mutha kuwamanganso ndikupanga zithunzi zanu zosinthidwa.
- Gawo amagwiritsidwa ntchito kufotokoza kumene katundu akuchokera, kuyesa kuyesa, ndi kuyesa kwa sitolo. Mutha kusankha gwero lililonse la katundu: Yandex.Tank, Apache JMeter, yanu, kapena zonse palimodzi. Kuti mulepheretse magwero osafunikira, ingoperekani ndemanga kapena kufufuta ntchitoyo. Malo olowera pazochokera katundu:
- Magawo otsegulira a Yandex.Tank afotokozedwa mu .,
- Zoyambira za Apache JMeter zafotokozedwa mufayilo .
Zindikirani: Template yosinthira msonkhano imagwiritsidwa ntchito kukhazikitsa kulumikizana ndi dongosolo la CI ndipo sizitanthauza kuyika malingaliro oyesa momwemo. Kwa mayesero, malo olowera amatchulidwa, kumene bash script yolamulira ilipo. Njira yoyendetsera mayeso, kupanga malipoti, ndi zolemba zoyeserera ziyenera kukhazikitsidwa ndi akatswiri a QA. Muchiwonetsero, pazowonjezera zonse ziwiri, pempho latsamba lalikulu la Yandex limagwiritsidwa ntchito ngati mayeso osavuta. Ma script ndi magawo oyesera ali mu ndandanda .
- Pa siteji muyenera kufotokozera momwe mungafalitsire zotsatira za mayeso zomwe zapezedwa pagawo la Mayeso kumalo osungira akunja, mwachitsanzo, ku GitLab Pages kapena machitidwe apadera ofotokozera. Masamba a GitLab amafuna kuti ./public chikwatu chikhale chopanda kanthu ndipo chimakhala ndi fayilo ya index.html mayeso akamaliza. Mutha kuwerenga za ma nuances a ntchito ya GitLab Pages. .
Zitsanzo zamomwe mungatumizire deta:
- kuchokera ku JMeter kupita ku ,
- kuchokera ku Yandex.Tank kupita ku .
Kuyika malangizo:
- HTML statics mu ,
- kupita ku InfluxDB kenako ku .
Muchitsanzo chowonetsera, mapaipi okhala ndi mayeso olemetsa ndi magwero awiri onyamula (mutha kuletsa zosafunikira) amawoneka motere:
Apache JMeter imatha kupanga lipoti la HTML lokha, kotero ndizopindulitsa kwambiri kuzisunga mu Masamba a GitLab pogwiritsa ntchito zida zokhazikika. Umu ndi momwe lipoti la Apache JMeter limawonekera:
Muchitsanzo chachiwonetsero cha Yandex.Tank, mudzangowona m'gawo la Masamba a GitLab. Pakuyesa, Tank imatha kusunga zotsatira ku database ya InfluxDB, ndipo kuchokera pamenepo zitha kuwonetsedwa, mwachitsanzo, ku Grafana (kusinthidwa kumachitika mufayilo. ). Umu ndi momwe lipoti la Tank limawonekera ku Grafana:
Chidule
M'nkhaniyi, ndinalankhula za lingaliro la "kuyesa katundu ngati ntchito" (kuyesa katundu ngati ntchito). Lingaliro lalikulu ndikugwiritsira ntchito zomangira za maiwe okhazikitsidwa kale a othandizira katundu, zithunzi za docker za malo onyamula katundu, machitidwe operekera malipoti ndi mapaipi omwe amawaphatikiza mu GitLab CI kutengera template yosavuta ya .gitlab-ci.yml (chitsanzo ). Zonsezi zimathandizidwa ndi kagulu kakang'ono ka mainjiniya odzichitira okha ndikusinthidwanso popempha magulu azogulitsa. Ndikukhulupirira kuti izi zidzakuthandizani kukonzekera ndi kukhazikitsa dongosolo lofananalo pakampani yanu. Zikomo chifukwa chomvetsera!
PS Ndikufuna kunena zikomo kwambiri kwa anzanga, Sergey Kurbanov ndi Nikolai Yusev, chifukwa cha chithandizo chaukadaulo pakukhazikitsa lingaliro la kuyezetsa katundu ngati ntchito mu kampani yathu.
wolemba: - Wachiwiri Mutu wa Technologies and Development Processes (DevOps) ku Positive Technologies
Source: www.habr.com