ProHoster > Blog > Tsamaiso > Network automation. Nyeoe e tsoang bophelong ba motho

Network automation. Nyeoe e tsoang bophelong ba motho

Hey Habr!

Sehloohong sena re ka rata ho bua ka automation ea lisebelisoa tsa marang-rang. Setšoantšo se sebetsang sa marang-rang se sebetsang k'hamphaning e le 'ngoe e nyenyane empa e le motlotlo haholo e tla hlahisoa. Lipapali tsohle tse nang le lisebelisoa tsa 'nete tsa marang-rang li etsahala ka tšohanyetso. Re tla sheba nyeoe e ileng ea etsahala marang-rang ena, e ka beng e entse hore khoebo e khaotse nako e telele le tahlehelo e tebileng ea lichelete. Tharollo ea nyeoe ena e lumellana hantle haholo le khopolo ea "Automation of network infrastructure". Re sebelisa lisebelisoa tse ikemetseng, re tla bontša kamoo u ka rarollang mathata a rarahaneng ka katleho ka nako e khutšoanyane, 'me re tla nahana ka hore na ke hobane'ng ha mathata ana a lokela ho rarolloa ka tsela ena eseng ka tsela e' ngoe (ka console).

ikgololo

Lisebelisoa tsa rona tsa mantlha tsa ho iketsetsa li-Ansible (e le sesebelisoa sa othomathike) le Git (e le polokelo ea libuka tsa ho bapala tse Ansible). Ke kopa ho etsa pehelo hang-hang hore sena ha se sengoloa sa selelekela, moo re buang ka mohopolo oa Ansible kapa Git, le ho hlalosa lintho tsa mantlha (mohlala, li-roletaskimodules ke eng, lifaele tsa inventory, variables in Ansible, kapa se etsahalang ha o kenya git push kapa git commit commands). Pale ena ha e mabapi le hore na u ka itloaetsa joang le ho lokisa NTP kapa SMTP sesebelisoa sa hau. Ena ke pale ea hore na u ka rarolla bothata ba marang-rang ka potlako joang ntle le liphoso. Hape ho eletsoa ho ba le kutloisiso e ntle ea hore na marang-rang a sebetsa joang, haholo-holo hore na TCP / IP, OSPF, BGP protocol stack ke eng. Re tla boela re ntše khetho ea Ansible le Git ho tsoa masakaneng. Haeba u ntse u hloka ho khetha tharollo e itseng, re khothaletsa haholo ho bala buka "Network Programmability and Automation. Skills for the Next-Generation Network Engineer" ka Jason Edelman, Scott S. Lowe, le Matt Oswalt.

Joale ho ntlha.

Mokhoa oa bothata

A re nahaneng ka boemo: 3 hoseng, u robetse haholo 'me u lora. Mohala. Motsamaisi oa tekheniki o letsetsa:

- Ho joalo?
— ###, ####, #####, sehlopha sa firewall se oele mme ha se tsohe !!!
U pikitla mahlo, u leka ho utloisisa se etsahalang le ho nahana hore na sena se ka etsahala joang. Mohaleng u ka utloa moriri oa hlooho ea motsamaisi o rusolla, 'me o kopa ho khutlela morao hobane molaoli oa kakaretso o mo letsetsa mohala oa bobeli.

Halofo ea hora hamorao, u ile ua bokella lintlha tsa pele tsa selelekela ho tloha mosebetsing, ua tsosa bohle ba neng ba ka tsosoa. Ka lebaka leo, mookameli oa theknoloji ha aa ka a bua leshano, ntho e 'ngoe le e' ngoe e ntse e le joalo, sehlopha se seholo sa li-firewall se oele, 'me ha ho mekhatlo ea motheo ea' mele e mo tlisang maikutlong. Litšebeletso tsohle tseo k'hamphani e fanang ka tsona ha li sebetse.

Khetha bothata ho tatso ea hau, bohle ba tla hopola ntho e fapaneng. Ka mohlala, ka mor'a ho ntlafatsoa ka bosiu bo le bong ha ho se na mojaro o boima, ntho e 'ngoe le e' ngoe e ne e sebetsa hantle, 'me bohle ba robala ba thabile. Sephethephethe se ile sa qala ho phalla, 'me li-buffers tsa li-interface li ile tsa qala ho phalla ka lebaka la phoso ho mokhanni oa karete ea marang-rang.

Jackie Chan a ka hlalosa boemo hantle.

Network automation. Nyeoe e tsoang bophelong ba motho

Kea leboha, Jackie.

Hase boemo bo monate haholo, na ha ho joalo?

Ha re tlohele marang-rang a rona bro ka menahano ea hae e bohloko ka nakoana.

A re tšohleng hore na liketsahalo li tla tsoela pele joang.

Re fana ka tlhahiso ea tatellano e latelang ea nehelano ea boitsebiso

  1. Ha re shebeng setšoantšo sa marang-rang 'me re bone hore na se sebetsa joang;
  2. Re tla hlalosa hore na re fetisetsa litlhophiso joang ho tloha router e 'ngoe ho ea ho e' ngoe re sebelisa Ansible;
  3. Ha re bue ka automation ea lisebelisoa tsa IT ka kakaretso.

Setšoantšo sa marang-rang le tlhaloso

Scheme

Network automation. Nyeoe e tsoang bophelong ba motho

A re hlahlobeng setšoantšo se utloahalang sa mokhatlo oa rōna. Re ke ke ra bolela mabitso a baetsi ba lisebelisoa tse khethehileng; molemong oa sengoloa sena ha ho na taba ('Mali ea hlokolosi o tla hakanya hore na ho sebelisoa lisebelisoa tsa mofuta ofe). Ona ke o mong oa melemo ea ho sebetsa le Ansible; ha re theha, ka kakaretso ha re tsotelle hore na ke sesebelisoa sa mofuta ofe. Ho utloisisa feela, sena ke lisebelisoa tse tsoang ho barekisi ba tsebahalang, joalo ka Cisco, Juniper, Check Point, Fortinet, Palo Alto ... o ka nkela khetho ea hau sebaka.

Re na le mesebetsi e 'meli ea mantlha ea ho tsamaisa sephethephethe:

  1. Netefatsa phatlalatso ea lits'ebeletso tsa rona, e leng khoebo ea k'hamphani;
  2. Fana ka puisano le makala, setsi sa data se hole le mekhatlo ea batho ba boraro (balekane le bareki), hammoho le phihlello ea makala ho Internet ka ofisi e bohareng.

A re qaleng ka lintlha tsa motheo:

  1. Li-routers tse peli tsa moeli (BRD-01, BRD-02);
  2. Sehlopha sa Firewall (FW-CLUSTER);
  3. Phetoho ea mantlha (L3-CORE);
  4. Router e tla fetoha mokhoa oa bophelo (ha re ntse re rarolla bothata, re tla fetisetsa litlhophiso tsa marang-rang ho tloha FW-CLUSTER ho ea ho EMERGENCY) (EMERGENCY);
  5. Liphetoho bakeng sa tsamaiso ea lisebelisoa tsa marang-rang (L2-MGMT);
  6. Mochini oa sebele o nang le Git le Ansible (VM-AUTOMATION);
  7. Laptop eo ho eona ho etsoang liteko le nts'etsopele ea libuka tsa ho bapala bakeng sa Ansible (Laptop-Automation).

Marang-rang a hlophisitsoe ka protocol e matla ea ho tsamaisa OSPF e nang le libaka tse latelang:

  • Sebaka sa 0 - sebaka se kenyelletsang li-routers tse ikarabellang bakeng sa ho tsamaisa sephethephethe sebakeng sa EXCHANGE;
  • Sebaka sa 1 - sebaka se kenyeletsang li-routers tse ikarabellang bakeng sa ts'ebetso ea litšebeletso tsa khampani;
  • Sebaka sa 2 - sebaka se kenyeletsang li-routers tse ikarabellang bakeng sa sephethephethe sa tsamaiso ea litsela;
  • Sebaka sa N - libaka tsa marang-rang a makala.

Ho li-routers tsa moeli, ho etsoa router ea sebele (VRF-INTERNET), moo pono e feletseng ea eBGP e kentsoeng le AS e fanoeng. iBGP e hlophisitsoe lipakeng tsa VRFs. Khampani e na le letamo la liaterese tse tšoeu tse hatisitsoeng ho tsena VRF-INTERNET. Liaterese tse ling tse tšoeu li tsamaisoa ka ho toba ho FW-CLUSTER (liaterese tseo litšebeletso tsa k'hamphani li sebetsang ho tsona), tse ling li fetisetsoa sebakeng sa EXCHANGE (litšebeletso tsa lik'hamphani tsa ka hare tse hlokang liaterese tsa IP tsa ka ntle, le liaterese tsa NAT tse ka ntle bakeng sa liofisi). Ka mor'a moo, sephethephethe se ea ho li-routers tse entsoeng ho L3-CORE tse nang le liaterese tse tšoeu le tse bohlooho (libaka tsa tšireletso).

Marang-rang a Tsamaiso a sebelisa li-switches tse inehetseng mme a emela marang-rang a inehetseng 'meleng. Marang-rang a tsamaiso e boetse e arotsoe ka libaka tsa tšireletso.
Routa ea EMERGENCY ka 'mele le ka mokhoa o utloahalang e etsisa FW-CLUSTER. Likhokahano tsohle tse ho eona li holofetse ntle le tse shebaneng le marang-rang a tsamaiso.

Automation le tlhaloso ea eona

Re fumane hore na marang-rang a sebetsa joang. Joale ha re shebeng mohato ka mohato hore na re tla etsa eng ho fetisetsa sephethephethe ho tloha FW-CLUSTER ho ea ho EMERGENCY:

  1. Re tima li-interfaces ho sesebelisoa sa mantlha (L3-CORE) se se hokahanyang le FW-CLUSTER;
  2. Re tima li-interfaces ho sesebelisoa sa kernel sa L2-MGMT se se hokahanyang le FW-CLUSTER;
  3. Re lokisa router ea EMERGENCY (ka ho sa feleng, li-interfaces tsohle li holofetse ho eona, ntle le tse amanang le L2-MGMT):

  • Re nolofalletsa li-interfaces ho EMERGENCY;
  • Re lokisa aterese ea IP ea kantle (bakeng sa NAT) e neng e le ho FW-Cluster;
  • Re hlahisa likopo tsa gARP e le hore liaterese tsa poppy litafoleng tsa arp tsa L3-CORE li fetoloe ho tloha ho FW-Cluster ho ea ho EMERGENCY;
  • Re ngolisa tsela ea kamehla e le static ho BRD-01, BRD-02;
  • Bopa NAT melao;
  • Phahamisa ho EMERGENCY OSPF Sebaka sa 1;
  • Phahamisa ho EMERGENCY OSPF Sebaka sa 2;
  • Re fetola litšenyehelo tsa litsela sebakeng sa 1 ho ea ho 10;
  • Re fetola litšenyehelo tsa tsela ea kamehla sebakeng sa 1 ho isa ho 10;
  • Re fetola liaterese tsa IP tse amanang le L2-MGMT (ho tse neng li le FW-CLUSTER);
  • Re hlahisa likopo tsa gARP e le hore liaterese tsa poppy litafoleng tsa arp tsa L2-MGMT li fetoloe ho tloha ho FW-CLUSTER ho ea ho EMERGENCY.

Hape, re khutlela tlhahisong ea pele ea bothata. Hora ea boraro hoseng, khatello ea maikutlo e kholo, phoso ka nako leha e le efe e ka lebisa mathateng a macha. U itokiselitse ho thaepa litaelo ka CLI? Ee? Ho lokile, bonyane e-ea u hlatsoe sefahleho, u noe kofi 'me u bokelle matla a hau.
Bruce, ka kopo thusa bashanyana.

Network automation. Nyeoe e tsoang bophelong ba motho

Ho lokile, re ntse re tsoela pele ho ntlafatsa li-automation tsa rona.
Ka tlase ke setšoantšo sa kamoo buka ea ho bapala e sebetsang ka mantsoe a Ansible. Sekema sena se bonts'a seo re se hlalositseng ka holimo, ke ts'ebetsong e itseng feela ho Ansible.
Network automation. Nyeoe e tsoang bophelong ba motho

Mothating ona, re ile ra hlokomela se lokelang ho etsoa, ​​ra theha buka ea ho bapala, ra etsa liteko, 'me joale re se re loketse ho e qala.

Phallo e 'ngoe e nyane ea mantsoe. Bonolo ba pale ha boa lokela ho u khelosa. Mokhoa oa ho ngola libuka tsa ho bapala o ne o se bonolo ebile o potlakile joalokaha o ne o ka bonahala. Teko e ile ea nka nako e telele haholo, ho ile ha etsoa sethala, tharollo e ile ea lekoa hangata, ho ile ha etsoa liteko tse ka bang 100.

Ha re qaleng ... Ho na le maikutlo a hore ntho e 'ngoe le e' ngoe e etsahala butle haholo, ho na le phoso kae-kae, ntho e ke keng ea sebetsa qetellong. Boikutlo ba ho qhoma ka parachute, empa parachute ha e batle ho bula hang-hang ... sena se tloaelehile.

Ka mor'a moo, re bala sephetho sa ts'ebetso e entsoeng ea Ansible playbook (liaterese tsa IP li ile tsa nkeloa sebaka ka sepheo sa ho boloka lekunutu):

[xxx@emergency ansible]$ ansible-playbook -i /etc/ansible/inventories/prod_inventory.ini /etc/ansible/playbooks/emergency_on.yml 

PLAY [------->Emergency on VCF] ********************************************************

TASK [vcf_junos_emergency_on : Disable PROD interfaces to FW-CLUSTER] *********************
changed: [vcf]

PLAY [------->Emergency on MGMT-CORE] ************************************************

TASK [mgmt_junos_emergency_on : Disable MGMT interfaces to FW-CLUSTER] ******************
changed: [m9-03-sw-03-mgmt-core]

PLAY [------->Emergency on] ****************************************************

TASK [mk_routeros_emergency_on : Enable EXT-INTERNET interface] **************************
changed: [m9-04-r-04]

TASK [mk_routeros_emergency_on : Generate gARP for EXT-INTERNET interface] ****************
changed: [m9-04-r-04]

TASK [mk_routeros_emergency_on : Enable static default route to EXT-INTERNET] ****************
changed: [m9-04-r-04]

TASK [mk_routeros_emergency_on : Change NAT rule to EXT-INTERNET interface] ****************
changed: [m9-04-r-04] => (item=12)
changed: [m9-04-r-04] => (item=14)
changed: [m9-04-r-04] => (item=15)
changed: [m9-04-r-04] => (item=16)
changed: [m9-04-r-04] => (item=17)

TASK [mk_routeros_emergency_on : Enable OSPF Area 1 PROD] ******************************
changed: [m9-04-r-04]

TASK [mk_routeros_emergency_on : Enable OSPF Area 2 MGMT] *****************************
changed: [m9-04-r-04]

TASK [mk_routeros_emergency_on : Change OSPF Area 1 interfaces costs to 10] *****************
changed: [m9-04-r-04] => (item=VLAN-1001)
changed: [m9-04-r-04] => (item=VLAN-1002)
changed: [m9-04-r-04] => (item=VLAN-1003)
changed: [m9-04-r-04] => (item=VLAN-1004)
changed: [m9-04-r-04] => (item=VLAN-1005)
changed: [m9-04-r-04] => (item=VLAN-1006)
changed: [m9-04-r-04] => (item=VLAN-1007)
changed: [m9-04-r-04] => (item=VLAN-1008)
changed: [m9-04-r-04] => (item=VLAN-1009)
changed: [m9-04-r-04] => (item=VLAN-1010)
changed: [m9-04-r-04] => (item=VLAN-1011)
changed: [m9-04-r-04] => (item=VLAN-1012)
changed: [m9-04-r-04] => (item=VLAN-1013)
changed: [m9-04-r-04] => (item=VLAN-1100)

TASK [mk_routeros_emergency_on : Change OSPF area1 default cost for to 10] ******************
changed: [m9-04-r-04]

TASK [mk_routeros_emergency_on : Change MGMT interfaces ip addresses] ********************
changed: [m9-04-r-04] => (item={u'ip': u'х.х.n.254', u'name': u'VLAN-803'})
changed: [m9-04-r-04] => (item={u'ip': u'х.х.n+1.254', u'name': u'VLAN-805'})
changed: [m9-04-r-04] => (item={u'ip': u'х.х.n+2.254', u'name': u'VLAN-807'})
changed: [m9-04-r-04] => (item={u'ip': u'х.х.n+3.254', u'name': u'VLAN-809'})
changed: [m9-04-r-04] => (item={u'ip': u'х.х.n+4.254', u'name': u'VLAN-820'})
changed: [m9-04-r-04] => (item={u'ip': u'х.х.n+5.254', u'name': u'VLAN-822'})
changed: [m9-04-r-04] => (item={u'ip': u'х.х.n+6.254', u'name': u'VLAN-823'})
changed: [m9-04-r-04] => (item={u'ip': u'х.х.n+7.254', u'name': u'VLAN-824'})
changed: [m9-04-r-04] => (item={u'ip': u'х.х.n+8.254', u'name': u'VLAN-850'})
changed: [m9-04-r-04] => (item={u'ip': u'х.х.n+9.254', u'name': u'VLAN-851'})
changed: [m9-04-r-04] => (item={u'ip': u'х.х.n+10.254', u'name': u'VLAN-852'})
changed: [m9-04-r-04] => (item={u'ip': u'х.х.n+11.254', u'name': u'VLAN-853'})
changed: [m9-04-r-04] => (item={u'ip': u'х.х.n+12.254', u'name': u'VLAN-870'})
changed: [m9-04-r-04] => (item={u'ip': u'х.х.n+13.254', u'name': u'VLAN-898'})
changed: [m9-04-r-04] => (item={u'ip': u'х.х.n+14.254', u'name': u'VLAN-899'})

TASK [mk_routeros_emergency_on : Generate gARPs for MGMT interfaces] *********************
changed: [m9-04-r-04] => (item={u'ip': u'х.х.n.254', u'name': u'VLAN-803'})
changed: [m9-04-r-04] => (item={u'ip': u'х.х.n+1.254', u'name': u'VLAN-805'})
changed: [m9-04-r-04] => (item={u'ip': u'х.х.n+2.254', u'name': u'VLAN-807'})
changed: [m9-04-r-04] => (item={u'ip': u'х.х.n+3.254', u'name': u'VLAN-809'})
changed: [m9-04-r-04] => (item={u'ip': u'х.х.n+4.254', u'name': u'VLAN-820'})
changed: [m9-04-r-04] => (item={u'ip': u'х.х.n+5.254', u'name': u'VLAN-822'})
changed: [m9-04-r-04] => (item={u'ip': u'х.х.n+6.254', u'name': u'VLAN-823'})
changed: [m9-04-r-04] => (item={u'ip': u'х.х.n+7.254', u'name': u'VLAN-824'})
changed: [m9-04-r-04] => (item={u'ip': u'х.х.n+8.254', u'name': u'VLAN-850'})
changed: [m9-04-r-04] => (item={u'ip': u'х.х.n+9.254', u'name': u'VLAN-851'})
changed: [m9-04-r-04] => (item={u'ip': u'х.х.n+10.254', u'name': u'VLAN-852'})
changed: [m9-04-r-04] => (item={u'ip': u'х.х.n+11.254', u'name': u'VLAN-853'})
changed: [m9-04-r-04] => (item={u'ip': u'х.х.n+12.254', u'name': u'VLAN-870'})
changed: [m9-04-r-04] => (item={u'ip': u'х.х.n+13.254', u'name': u'VLAN-898'})
changed: [m9-04-r-04] => (item={u'ip': u'х.х.n+14.254', u'name': u'VLAN-899'})

PLAY RECAP ************************************************************************

E entsoe!

Ha e le hantle, ha e e-s'o lokele, u se ke ua lebala ka ho kopana ha li-protocol tse matla le ho kenya litsela tse ngata ho FIB. Re ke ke ra susumetsa sena ka tsela efe kapa efe. Re leta. E ile ea sebetsa. Joale e se e lokile.

'Me motseng oa Vilabajo (o sa batleng ho iketsetsa marang-rang) ba tsoela pele ho hlatsoa lijana. Bruce (ke 'nete, e se e fapane, empa e se e pholile) o leka ho utloisisa hore na ho tla etsoa phetoho e kae ka letsoho ea thepa.

Network automation. Nyeoe e tsoang bophelong ba motho

Ke rata hape ho bua ka ntlha e le 'ngoe ea bohlokoa. Re ka khutlisa tsohle joang? Kamora nako e itseng, re tla khutlisetsa FW-CLUSTER ea rona bophelong. Sena ke sesebelisoa sa mantlha, eseng bekapo, marang-rang a tlameha ho sebetsa ho eona.

Na u ikutloa kamoo marang-rang a qalang ho chesa? Motsamaisi oa tekheniki o tla utloa likhang tse sekete hobaneng sena se sa lokela ho etsoa, ​​​​ke hobane'ng ha sena se ka etsoa hamorao. Ka bomalimabe, ena ke tsela eo marang-rang a sebetsang ka eona ho tsoa ho lihlopha tse ngata, likotoana le mesaletsa ea mabothobotho a eona a pele. Ho bonahala eka ke patchwork quilt. Mosebetsi oa rona ka kakaretso, eseng boemong bona bo ikhethileng, empa ka kakaretso, joalo ka litsebi tsa IT, ke ho tlisa mosebetsi oa marang-rang ho lentsoe le letle la Senyesemane "consistency", le na le likarolo tse ngata haholo, le ka fetoleloa e le: ho lumellana. , tsitsipano, moelelo, nyallano, tsamaiso, papiso, momahano. Tsohle di mabapi le yena. Ke boemong bona feela moo marang-rang a ka laoloang, re utloisisa ka ho hlaka hore na ke eng e sebetsang le hore na joang, re utloisisa ka ho hlaka se lokelang ho fetoloa, haeba ho hlokahala, re tseba ka ho hlaka hore na re ka sheba hokae haeba mathata a hlaha. 'Me ke marang-rang a joalo feela moo u ka etsang maqheka a kang ao re sa tsoa a hlalosa.

Ha e le hantle, ho ne ho lokiselitsoe buka e 'ngoe ea ho bapala, e ileng ea khutlisetsa litlhophiso boemong ba tsona ba pele. Taba ea ts'ebetso ea eona e ts'oana (ho bohlokoa ho hopola hore tatellano ea mesebetsi e bohlokoa haholo), e le hore re se ke ra lelefatsa sengoloa se seng se ntse se le telele, re nkile qeto ea ho se behe lethathamo la ts'ebetso ea buka ea ho bapala. Ka mor'a ho ikoetlisa joalo, u tla ikutloa u khobile matšoafo 'me u itšepa haholoanyane nakong e tlang, ho phaella moo, lithupa leha e le life tseo u li bokeletseng moo li tla iponahatsa hang-hang.

Mang kapa mang a ka re ngolla mme a amohela mehloli ea khoutu eohle e ngotsoeng, hammoho le libuka tsohle tsa palybook. Mabitso profaeleng.

fumanoeng ke

Ho ea ka maikutlo a rona, lits'ebetso tse ka iketsetsang li-automated ha li so hlake. Ho ipapisitsoe le seo re kopaneng le sona le seo basebetsi-'moho le rona ba Bophirimela ba buang ka sona, lihlooho tse latelang lia bonahala ho fihlela joale:

  • Ho fana ka lisebelisoa;
  • Pokello ea lintlha;
  • Ho tlaleha;
  • Ho batle phoso;
  • Ho lumellana.

Haeba ho e-na le thahasello, re ka ntšetsa pele puisano ka e ’ngoe ea lihlooho tse fanoeng.

Ke kopa hape ho bua hanyane ka automation. Seo e lokelang ho ba sona kutloisisong ea rona:

  • Tsamaiso e tlameha ho phela ntle le motho, ha e ntse e ntlafatsoa ke motho. Tsamaiso ha ea lokela ho itšetleha ka batho;
  • Ts'ebetso e tlameha ho ba setsebi. Ha ho sehlopha sa litsebi tse etsang mesebetsi e tloaelehileng. Ho na le litsebi tse iketselitseng tloaelo eohle le ho rarolla mathata a rarahaneng feela;
  • Mesebetsi e tloaelehileng e etsoa ka bo eona "ka ho tobetsa konopo", ha ho na lisebelisoa tse senyehang. Litholoana tsa mesebetsi e joalo li lula li lebelloa ebile lia utloahala.

Mme lintlha tsena li lokela ho lebisa ho eng:

  • Ho pepeseha ha lisebelisoa tsa IT (Likotsi tse fokolang tsa ts'ebetso, mekhoa ea morao-rao, ts'ebetsong. Nako e fokolang ka selemo);
  • Bokhoni ba ho rera lisebelisoa tsa IT (Sistimi ea moralo oa bokhoni - u ka bona hore na ho sebelisoa bokae, u ka bona hore na ho hlokahala lisebelisoa tse kae tsamaisong e le 'ngoe, eseng ka mangolo le ho etela mafapha a holimo);
  • Monyetla oa ho fokotsa palo ea basebetsi ba IT.

Bangoli ba sehlooho: Alexander Chelovekov (CCIE RS, CCIE SP) le Pavel Kirillov. Re thahasella ho buisana le ho fana ka litharollo ka taba ea boits'oaro ba lisebelisoa tsa IT.


Source: www.habr.com

Eketsa ka tlhaloso