Hayi Habr!
Kweli nqaku singathanda ukuthetha malunga ne-automation yeziseko zenethiwekhi. Umzobo osebenzayo womnatha osebenza kwinkampani enye encinci kodwa enebhongo kakhulu iya kuboniswa. Zonke iimatshisi ezinezixhobo zenethiwekhi yokwenyani azikhethi. Siza kujonga kwimeko eyenzeka kule nethiwekhi, enokuthi ibangele ukuvalwa kweshishini ixesha elide kunye nokulahlekelwa okukhulu kwemali. Isisombululo sale meko sihambelana kakuhle nombono othi "Automation of network infrastructure". Ukusebenzisa izixhobo ezizenzekelayo, siya kubonisa indlela onokuthi uzisombulule ngayo ngempumelelo iingxaki ezinzima ngexesha elifutshane, kwaye siya kubonakalisa ukuba kutheni ezi ngxaki kufuneka zisonjululwe ngolu hlobo kwaye kungekhona ngenye indlela (nge-console).
isikhanyeli
Izixhobo zethu eziphambili ze-automation zi-Ansible (njengesixhobo esizenzekelayo) kunye ne-Git (njengendawo yokugcina iincwadi zokudlala eziAnsible). Ndingathanda ukwenza ngokukhawuleza ugcino ukuba eli ayilonqaku lentshayelelo, apho sithetha ngengqiqo ye-Ansible okanye i-Git, kwaye sichaze izinto ezisisiseko (umzekelo, yintoni i-roletaskimodules, iifayile ze-inventory, ii-variables kwi-Ansible, okanye kwenzeka ntoni xa ungenisa i-git push okanye imiyalelo ye-git commit). Eli bali alikho malunga nokuba ungaziqhelanisa njani ne-Ansible kwaye uqwalasele i-NTP okanye i-SMTP kwisixhobo sakho. Eli libali malunga nendlela onokuthi ukhawuleze kwaye ukhethe ukusombulula ingxaki yenethiwekhi ngaphandle kweempazamo. Kwakhona kucetyiswa ukuba ube nokuqonda kakuhle indlela umsebenzi womnatha osebenza ngayo, ngokukodwa ukuba yintoni iTCP / IP, OSPF, BGP protocol stack. Siza kuphinda sithathe ukhetho lwe-Ansible kunye neGit ngaphandle kwe-equation. Ukuba usafuna ukukhetha isisombululo esithile, sincoma kakhulu ukuba ufunde incwadi ethi "Network Programmability and Automation. Izakhono zeNjineli yeNethiwekhi elandelayo "nguJason Edelman, uScott S. Lowe, kunye noMat Oswalt.
Ngoku ukuya kwinqanaba.
Џџ ѕЃ ° °
Makhe sicinge ngemeko: ngentsimbi yesi-3 ekuseni, ulele ngokukhawuleza kwaye uphupha. Umnxeba wefowni. Umlawuli wezobugcisa ubiza:
- Ewe?
— ###, ####, #####, i-firewall cluster iwile kwaye ayivuki!!!
Uhlikihla amehlo akho, uzama ukuqonda ukuba kwenzeka ntoni kwaye ucinge ukuba oku kunokwenzeka njani. Kwifowuni unokuva inwele entloko yomlawuli ekrazula, kwaye ucela ukuba abuyele kwakhona kuba u-general uyambiza kumgca wesibini.
Isiqingatha seyure kamva, uqokelele amanqaku entshayelelo yokuqala ukusuka kwinguqu yomsebenzi, wavusa wonke umntu onokuvuswa. Ngenxa yoko, umlawuli wezobuchwephesha akazange axoke, yonke into injengoko injalo, iqela eliphambili leentsimbi zomlilo liwile, kwaye akukho zintshukumo zomzimba ezisisiseko ezizisa kuye. Zonke iinkonzo ezinikezelwa yinkampani azisebenzi.
Khetha ingxaki kwincasa yakho, wonke umntu uya kukhumbula into eyahlukileyo. Ngokomzekelo, emva kokuhlaziywa kobusuku ngokungabikho komthwalo onzima, yonke into yasebenza kakuhle, kwaye wonke umntu waya kulala evuya. I-traffic yaqala ukuhamba, kwaye i-interface buffers yaqala ukuphuphuma ngenxa ye-bug kumqhubi wekhadi lenethiwekhi.
UJackie Chan unokuyichaza kakuhle imeko.
Enkosi, Jackie.
Ayiyomeko imnandi kakhulu, akunjalo?
Masikhe siyishiye i-network yethu bhuti ngengcinga zakhe ezibuhlungu kancinci.
Makhe sixubushe ngendlela iziganeko eziya kuhambela phambili ngayo.
Sicebisa ngolu hlobo lulandelayo lwenkcazo-ntetho
- Makhe sijonge umzobo womnatha kwaye sibone ukuba isebenza njani;
- Siza kuchaza indlela esidlulisela ngayo izicwangciso ukusuka kwenye i-router ukuya kwenye usebenzisa i-Ansible;
- Makhe sithethe malunga ne-automation yeziseko ze-IT ngokubanzi.
Umzobo womnatha kunye nenkcazo
Iskimu
Makhe siqwalasele umzobo osengqiqweni wentlangano yethu. Asiyi kukhankanya abavelisi bezixhobo ezithile; ngeenjongo zeli nqaku ayinamsebenzi (Umfundi omameleyo uya kuqikelela ukuba luhlobo luni lwesixhobo esisetyenziswayo). Le yenye yeenzuzo ezilungileyo zokusebenza nge-Ansible; xa useka, asikhathali ngokubanzi ukuba luhlobo luni lwesixhobo. Ukuqonda nje, oku zixhobo ezivela kubathengisi abaziwayo, njengeCisco, iJuniper, i-Check Point, i-Fortinet, i-Palo Alto ... unokutshintsha inketho yakho.
Sinemisebenzi emibini ephambili yokuhambisa itrafikhi:
- Ukuqinisekisa ukupapashwa kweenkonzo zethu, ezizishishini lenkampani;
- Ukubonelela ngonxibelelwano namasebe, iziko ledatha elikude kunye nemibutho yangaphandle (amaqabane kunye nabathengi), kunye nokufikelela kwamasebe kwi-Intanethi ngeofisi ephakathi.
Masiqale ngezinto ezisisiseko:
- Iindlela ezimbini zomda (BRD-01, BRD-02);
- I-Firewall Cluster (FW-CLUSTER);
- Ukutshintsha okungundoqo (L3-CORE);
- I-router eya kuba yinto yokuphila (njengoko sixazulula ingxaki, siya kudlulisela izicwangciso zenethiwekhi ukusuka kwi-FW-CLUSTER ukuya kwi-EMERGENCY) (EMERGENCY);
- Iiswitshi zolawulo lweziseko zonxibelelwano (L2-MGMT);
- Umatshini obonakalayo oneGit kunye neAnsible (VM-AUTOMATION);
- Ilaptop apho uvavanyo kunye nophuhliso lweencwadi zokudlala ze-Ansible (Laptop-Automation) lwenziwa.
Uthungelwano luqwalaselwe ngenkqubo yendlela yokusebenza ye-OSPF enamandla enezi ndawo zilandelayo:
- Ummandla we-0 - indawo equka ii-routers ezinoxanduva lokuhambisa i-traffic kwindawo ye-EXCHANGE;
- Ummandla we-1 - indawo ebandakanya ii-routers ezijongene nokusebenza kweenkonzo zenkampani;
- Ummandla we-2 - indawo ebandakanya ii-routers ezijongene nokulawulwa kwe-traffic traffic;
- Ummandla N – imimandla yothungelwano lwamasebe.
Kwii-router zomda, i-router ebonakalayo (i-VRF-INTERNET) yenziwe, apho i-eBGP imboniselo epheleleyo ifakwe kunye ne-AS ehambelanayo eyabelwe. iBGP iqwalaselwe phakathi kweVRFs. Inkampani ineedilesi ezininzi ezimhlophe ezipapashwa kwezi VRF-INTERNET. Ezinye zeedilesi ezimhlophe zihanjiswa ngokuthe ngqo kwi-FW-CLUSTER (idilesi apho iinkonzo zenkampani zisebenza), ezinye zihanjiswa nge-EXCHANGE zone (iinkonzo zenkampani yangaphakathi ezifuna iidilesi ze-IP zangaphandle, kunye needilesi zangaphandle ze-NAT kwiiofisi). Emva koko, i-traffic iya kwii-routers ezibonakalayo ezenziwe kwi-L3-CORE ngeedilesi ezimhlophe kunye nezingwevu (imimandla yokhuseleko).
Inethiwekhi yoLawulo isebenzisa iiswitshi ezizinikeleyo kwaye imele inethiwekhi ezinikele ngokwasemzimbeni. Uthungelwano lolawulo lukwahlulwe ngokwemimandla yokhuseleko.
I-router EMERGENCY ngokoqobo kwaye ngokufanelekileyo iphinda i-FW-CLUSTER. Lonke ujongano kuyo luvaliwe ngaphandle kwezo zijonga kuthungelwano lolawulo.
Ukuzenzekela kunye nenkcazo yayo
Siye safumanisa ukuba inethiwekhi isebenza njani. Ngoku makhe sijonge inyathelo ngenyathelo kwinto esiya kuyenza ukuhambisa itrafikhi ukusuka kwi-FW-CLUSTER ukuya kwi-EMERGENCY:
- Sikhubaza i-interfaces kwi-core switch (L3-CORE) edibanisa kwi-FW-CLUSTER;
- Sikhubaza i-interfaces kwi-L2-MGMT kernel switch edibanisa kwi-FW-CLUSTER;
- Siqwalasela i-router EMERGENCY (ngokungagqibekanga, zonke ii-interfaces zivaliwe kuyo, ngaphandle kwezo zihambelana ne-L2-MGMT):
- Sivumela ujongano kwi-EMERGENCY;
- Siqwalasela idilesi ye-IP yangaphandle (ye-NAT) eyayikwi-FW-Cluster;
- Senza izicelo ze-gARP ukwenzela ukuba iidilesi ze-poppy kwiitafile ze-arp ze-L3-CORE zitshintshwe ukusuka kwi-FW-Cluster ukuya kwi-EMERGENCY;
- Sibhalisa indlela engagqibekanga njenge-static kwi-BRD-01, BRD-02;
- Yenza imithetho ye-NAT;
- Phakamisela kuMmandla we-OSPF ONGXAMISEKILEYO 1;
- Phakamisela kuMmandla we-OSPF ONGXAMISEKILEYO 2;
- Sitshintsha iindleko zeendlela kwiNdawo 1 ukuya kwi-10;
- Sitshintsha iindleko zendlela engagqibekanga kwiNdawo 1 ukuya kwi-10;
- Sitshintsha iidilesi ze-IP ezinxulumene ne-L2-MGMT (kwezo zazikwi-FW-CLUSTER);
- Senza izicelo ze-gARP ukwenzela ukuba iidilesi ze-poppy kwi-L2-MGMT arp table zitshintshwe ukusuka kwi-FW-CLUSTER ukuya kwi-EMERGENCY.
Kwakhona, sibuyela ekuqulunqweni kokuqala kwengxaki. Ngentsimbi yesithathu ekuseni, uxinzelelo olukhulu, impazamo kulo naliphi na inqanaba kunokukhokelela kwiingxaki ezintsha. Ngaba ukulungele ukuchwetheza imiyalelo nge-CLI? Ewe? Kulungile, ubuncinci hamba uhlambe ubuso bakho, usele ikofu kwaye uqokelele amandla akho.
Bruce, nceda uncede abafana.
Ewe, siyaqhubeka nokuphucula i-automation yethu.
Apha ngezantsi kukho umzobo wendlela incwadi yokudlala esebenza ngayo ngokwemigaqo yeAnsible. Esi sikimu sibonisa le nto siyichazileyo ngasentla, kukuphunyezwa okuthe ngqo kwi-Ansible.
Kweli nqanaba, siye saqonda ukuba yintoni ekufuneka yenziwe, siphuhlise incwadi yokudlala, senze uvavanyo, kwaye ngoku sikulungele ukuyisungula.
Olunye uhlaziyo oluncinci lweengoma. Ukukhululeka kwebali akufanele kukulahlekise. Inkqubo yokubhala iincwadi zokudlala yayingelula kwaye ikhawuleza njengoko isenokubonakala. Uvavanyo lwathatha ixesha elininzi, ukuma okubonakalayo kwenziwa, isisombululo savavanywa amaxesha amaninzi, malunga neemvavanyo ezili-100 zenziwa.
Masiqalise ... Kukho imvakalelo yokuba yonke into yenzeka ngokucothayo, kukho impazamo kwenye indawo, into ayizukusebenza ekugqibeleni. Uvakalelo lokutsiba ngeparachute, kodwa iparachute ayifuni ukuvula kwangoko ... oku kuqhelekile.
Okulandelayo, sifunda umphumo wemisebenzi eyenziwayo ye-Ansible playbook (idilesi ye-IP yatshintshwa ngeenjongo zokufihla):
[xxx@emergency ansible]$ ansible-playbook -i /etc/ansible/inventories/prod_inventory.ini /etc/ansible/playbooks/emergency_on.yml
PLAY [------->Emergency on VCF] ********************************************************
TASK [vcf_junos_emergency_on : Disable PROD interfaces to FW-CLUSTER] *********************
changed: [vcf]
PLAY [------->Emergency on MGMT-CORE] ************************************************
TASK [mgmt_junos_emergency_on : Disable MGMT interfaces to FW-CLUSTER] ******************
changed: [m9-03-sw-03-mgmt-core]
PLAY [------->Emergency on] ****************************************************
TASK [mk_routeros_emergency_on : Enable EXT-INTERNET interface] **************************
changed: [m9-04-r-04]
TASK [mk_routeros_emergency_on : Generate gARP for EXT-INTERNET interface] ****************
changed: [m9-04-r-04]
TASK [mk_routeros_emergency_on : Enable static default route to EXT-INTERNET] ****************
changed: [m9-04-r-04]
TASK [mk_routeros_emergency_on : Change NAT rule to EXT-INTERNET interface] ****************
changed: [m9-04-r-04] => (item=12)
changed: [m9-04-r-04] => (item=14)
changed: [m9-04-r-04] => (item=15)
changed: [m9-04-r-04] => (item=16)
changed: [m9-04-r-04] => (item=17)
TASK [mk_routeros_emergency_on : Enable OSPF Area 1 PROD] ******************************
changed: [m9-04-r-04]
TASK [mk_routeros_emergency_on : Enable OSPF Area 2 MGMT] *****************************
changed: [m9-04-r-04]
TASK [mk_routeros_emergency_on : Change OSPF Area 1 interfaces costs to 10] *****************
changed: [m9-04-r-04] => (item=VLAN-1001)
changed: [m9-04-r-04] => (item=VLAN-1002)
changed: [m9-04-r-04] => (item=VLAN-1003)
changed: [m9-04-r-04] => (item=VLAN-1004)
changed: [m9-04-r-04] => (item=VLAN-1005)
changed: [m9-04-r-04] => (item=VLAN-1006)
changed: [m9-04-r-04] => (item=VLAN-1007)
changed: [m9-04-r-04] => (item=VLAN-1008)
changed: [m9-04-r-04] => (item=VLAN-1009)
changed: [m9-04-r-04] => (item=VLAN-1010)
changed: [m9-04-r-04] => (item=VLAN-1011)
changed: [m9-04-r-04] => (item=VLAN-1012)
changed: [m9-04-r-04] => (item=VLAN-1013)
changed: [m9-04-r-04] => (item=VLAN-1100)
TASK [mk_routeros_emergency_on : Change OSPF area1 default cost for to 10] ******************
changed: [m9-04-r-04]
TASK [mk_routeros_emergency_on : Change MGMT interfaces ip addresses] ********************
changed: [m9-04-r-04] => (item={u'ip': u'х.х.n.254', u'name': u'VLAN-803'})
changed: [m9-04-r-04] => (item={u'ip': u'х.х.n+1.254', u'name': u'VLAN-805'})
changed: [m9-04-r-04] => (item={u'ip': u'х.х.n+2.254', u'name': u'VLAN-807'})
changed: [m9-04-r-04] => (item={u'ip': u'х.х.n+3.254', u'name': u'VLAN-809'})
changed: [m9-04-r-04] => (item={u'ip': u'х.х.n+4.254', u'name': u'VLAN-820'})
changed: [m9-04-r-04] => (item={u'ip': u'х.х.n+5.254', u'name': u'VLAN-822'})
changed: [m9-04-r-04] => (item={u'ip': u'х.х.n+6.254', u'name': u'VLAN-823'})
changed: [m9-04-r-04] => (item={u'ip': u'х.х.n+7.254', u'name': u'VLAN-824'})
changed: [m9-04-r-04] => (item={u'ip': u'х.х.n+8.254', u'name': u'VLAN-850'})
changed: [m9-04-r-04] => (item={u'ip': u'х.х.n+9.254', u'name': u'VLAN-851'})
changed: [m9-04-r-04] => (item={u'ip': u'х.х.n+10.254', u'name': u'VLAN-852'})
changed: [m9-04-r-04] => (item={u'ip': u'х.х.n+11.254', u'name': u'VLAN-853'})
changed: [m9-04-r-04] => (item={u'ip': u'х.х.n+12.254', u'name': u'VLAN-870'})
changed: [m9-04-r-04] => (item={u'ip': u'х.х.n+13.254', u'name': u'VLAN-898'})
changed: [m9-04-r-04] => (item={u'ip': u'х.х.n+14.254', u'name': u'VLAN-899'})
TASK [mk_routeros_emergency_on : Generate gARPs for MGMT interfaces] *********************
changed: [m9-04-r-04] => (item={u'ip': u'х.х.n.254', u'name': u'VLAN-803'})
changed: [m9-04-r-04] => (item={u'ip': u'х.х.n+1.254', u'name': u'VLAN-805'})
changed: [m9-04-r-04] => (item={u'ip': u'х.х.n+2.254', u'name': u'VLAN-807'})
changed: [m9-04-r-04] => (item={u'ip': u'х.х.n+3.254', u'name': u'VLAN-809'})
changed: [m9-04-r-04] => (item={u'ip': u'х.х.n+4.254', u'name': u'VLAN-820'})
changed: [m9-04-r-04] => (item={u'ip': u'х.х.n+5.254', u'name': u'VLAN-822'})
changed: [m9-04-r-04] => (item={u'ip': u'х.х.n+6.254', u'name': u'VLAN-823'})
changed: [m9-04-r-04] => (item={u'ip': u'х.х.n+7.254', u'name': u'VLAN-824'})
changed: [m9-04-r-04] => (item={u'ip': u'х.х.n+8.254', u'name': u'VLAN-850'})
changed: [m9-04-r-04] => (item={u'ip': u'х.х.n+9.254', u'name': u'VLAN-851'})
changed: [m9-04-r-04] => (item={u'ip': u'х.х.n+10.254', u'name': u'VLAN-852'})
changed: [m9-04-r-04] => (item={u'ip': u'х.х.n+11.254', u'name': u'VLAN-853'})
changed: [m9-04-r-04] => (item={u'ip': u'х.х.n+12.254', u'name': u'VLAN-870'})
changed: [m9-04-r-04] => (item={u'ip': u'х.х.n+13.254', u'name': u'VLAN-898'})
changed: [m9-04-r-04] => (item={u'ip': u'х.х.n+14.254', u'name': u'VLAN-899'})
PLAY RECAP ************************************************************************Yenziwe!
Ngapha koko, ayikalungi, ungalibali malunga nokudityaniswa kweeprothokholi zomzila kunye nokulayisha inani elikhulu leendlela kwi-FIB. Asinakuphembelela oku nangayiphi na indlela. Silindile. Iphumelele. Ngoku ilungile.
Kwaye kwidolophana yaseVilabajo (engafuni ukwenza ngokuzenzekelayo ukusekwa kwenethiwekhi) baqhubeka behlamba izitya. UBruce (ngokuqinisekileyo, sele ehluke, kodwa akukho nto ipholileyo) uzama ukuqonda ukuba kuninzi kangakanani ukulungiswa kwakhona kwesixhobo kuya kwenzeka.
Ndingathanda kwakhona ukuhlala kwinqaku elinye elibalulekileyo. Singayibuyisela njani yonke into? Emva kwexesha elithile, siya kubuyisela i-FW-CLUSTER yethu ebomini. Esi sesona sixhobo siphambili, hayi i-backup, inethiwekhi kufuneka iqhube kuyo.
Ngaba uyaziva ukuba iinethiwekhi ziqala ukuvutha? Umlawuli wezobuchwephesha uya kuva iingxoxo eziliwaka ukuba kutheni oku kungenziwa, kutheni oku kunokwenziwa kamva. Ngelishwa, le yindlela inethiwekhi esebenza ngayo ukusuka kwiqela leepatshi, iziqwenga, kunye neentsalela zobunewunewu bayo bangaphambili. Ijika ibe yi-patchwork quilt. Umsebenzi wethu ngokubanzi, kungekhona kule meko, kodwa ngokubanzi, njengeengcali ze-IT, kukuzisa umsebenzi womnatha kwigama elihle lesiNgesi elithi "consistency", lininzi kakhulu, linokuguqulelwa ngokuthi: ukuhambelana. , ukungaguquguquki, ukuqiqa, ukuyondelelana, ulungelelwaniso, ukuthelekisa, ukuyondelelana. Konke ngaye. Kule meko kuphela inethiwekhi iyalawuleka, siqonda ngokucacileyo oko kusebenza kunye nendlela, siqonda ngokucacileyo oko kufuneka kuguqulwe, ukuba kuyimfuneko, siyazi ngokucacileyo ukuba sijonge phi xa kuvela iingxaki. Kwaye kuphela kuthungelwano olunjalo onokuthi wenze amaqhinga afana nala sisanda kuwachaza.
Ngokwenyani, kwalungiswa enye incwadi yokudlala, eyabuyisela useto kwimeko yalo yokuqala. Ingqiqo yokusebenza kwayo iyafana (kubalulekile ukukhumbula ukuba ulandelelwano lwemisebenzi lubaluleke kakhulu), ukuze singalandi inqaku esele lide, sigqibe kwelokuba singaluthumeli uludwe lokwenziwa kwencwadi yokudlala. Emva kokwenza umthambo onjalo, uya kuziva uzolile kwaye uzithembe ngakumbi kwixesha elizayo, ukongeza, naziphi na iintonga oziqokelele apho ziya kuzibonakalisa kwangoko.
Nabani na unokusibhalela kwaye afumane imithombo yayo yonke ikhowudi ebhaliweyo, kunye nazo zonke iipalybooks. Abafowunelwa kwiprofayile.
ezifunyanisiweyo
Ngokombono wethu, iinkqubo ezinokuthi zenziwe ngokuzenzekelayo azikacaciswanga. Ngokusekwe kwinto esiye sadibana nayo kunye noko oogxa bethu baseNtshona baxoxa ngayo, le mixholo ilandelayo iyabonakala ukuza kuthi ga ngoku:
- Ubonelelo ngesixhobo;
- Ukuqokelelwa kwedatha;
- Ingxelo;
- Ukulungisa ingxaki;
- Ukuthotyelwa.
Ukuba kukho umdla, sinokuqhubeka nengxoxo kwesinye sezihloko ezinikiweyo.
Ndingathanda ukuthetha kancinci malunga ne-automation. Kufuneka ibe yintoni ekuqondeni kwethu:
- Inkqubo kufuneka iphile ngaphandle komntu, ngelixa iphuculwa ngumntu. Inkqubo akufuneki ixhomekeke ebantwini;
- Umsebenzi kufuneka ube yingcali. Akukho didi lweengcali zenza imisebenzi yesiqhelo. Kukho iingcali ezizenze ngokuzenzekelayo yonke inkqubo kunye nokusombulula iingxaki ezinzima kuphela;
- Imisebenzi eqhelekileyo yesiqhelo yenziwa ngokuzenzekelayo "ngokucofa iqhosha", akukho zixhobo zichithwayo. Isiphumo semisebenzi enjalo sihlala siqikelelwa kwaye siyaqondakala.
Kwaye ezi ngongoma zifanele zikhokelela ekubeni:
- Ukusekuhleni kweziseko ezingundoqo ze-IT (Umngcipheko ongephi wokusebenza, ukwenziwa kwezinto ngokwala maxesha, ukuphunyezwa. Ukuncipha kwexesha lokuphumla ngonyaka);
- Ikhono lokucwangcisa izibonelelo ze-IT (inkqubo yokucwangcisa umthamo - unokubona ukuba ungakanani udliwayo, unokubona ukuba zininzi kangakanani izixhobo ezifunekayo kwinkqubo enye, kwaye kungekhona ngeeleta kunye nokutyelela kumasebe aphezulu);
- Ukunokwenzeka ukunciphisa inani labasebenzi be-IT.
Ababhali benqaku: Alexander Chelovekov (CCIE RS, CCIE SP) kunye noPavel Kirillov. Sinomdla ekuxoxeni nasekucebiseni izisombululo kwisihloko se-IT ye-automation yeziseko ezizenzekelayo.
umthombo: www.habr.com