Muzvinyorwa zviviri zvekutanga, ndakasimudza nyaya ye otomatiki uye ndakadhirowa chimiro chayo, mune yechipiri ndakaita kudzoka munetwork virtualization, senzira yekutanga yekugadzira otomatiki magadzirirwo emasevhisi.
Iye zvino yave nguva yekudhirowa dhayagiramu yemuviri network.
Kana iwe usiri kujairana nekumisikidza data center network, saka ini ndinokurudzira zvakasimba kutanga nazvo .
Nyaya dzese:
Maitiro anotsanangurwa mune ino akateedzana anofanirwa kushanda kune chero rudzi rwetiweki, chero saizi, nechero akasiyana vatengesi (kwete). Zvisinei, hazvibviri kutsanangura muenzaniso wepasi rose wekushandiswa kwemaitiro aya. Naizvozvo, ini ndichatarisa pane yemazuva ano zvivakwa zveDC network: .
Tichaita DCI paMPLS L3VPN.
Iyo Kuwedzeredza network inomhanya pamusoro peiyo yemuviri network kubva kune iyo host (iyi inogona kunge iri OpenStack's VXLAN kana Tungsten Fabric kana chero chimwe chinhu chinoda chete basic IP yekubatanidza kubva kunetiweki).
Muchiitiko ichi, tinowana mamiriro akareruka ekugadzirisa otomatiki, nekuti isu tine michina yakawanda inogadziriswa nenzira imwechete.
Isu tichasarudza spherical DC mune vacuum:
- Imwe dhizaini shanduro kwese kwese.
- Vatengesi vaviri vanogadzira ndege mbiri dzenetwork.
- Imwe DC yakafanana neimwe sepizi mbiri dziri mupodhi.
Zviri mukati
- Physical topology
- Routing
- IP chirongwa
- Laba
- mhedziso
- Useful links
Rega Mupi wedu weSevhisi LAN_DC, semuenzaniso, atore mavhidhiyo ekudzidzisa nezvekupona mumarefiti akamira.
Mune megacities izvi zvakakurumbira zvakanyanya, saka unoda akawanda emuviri michina.
Chekutanga, ini ndichatsanangura network zvakaenzana sezvandinoda kuti ive. Uye ipapo ndichairerutsira iyo lab.
Physical topology
Nzvimbo
LAN_DC ichave ne6 DCs:
- Rashiya (RU):
- Moscow (msk)
- Kazan (kzn)
- Speini (SP):
- Barcelona (bcn)
- Malaga (mlg)
- China (CN):
- Shanghai (sha)
- Xi'an (zvose zviri zviviri)
Mukati meDC (Intra-DC)
Ese maDC ane akafanana emukati ekubatanidza network anoenderana neClos topology.
Ndeapi maClos network iwo uye nei ari akasiyana .
DC yega yega ine machira gumi ane michina, anozoverengerwa kuti A, B, C Etc.
Raki imwe neimwe ine michina makumi matatu. Havazotifariri.
Zvakare mune imwe neimwe rack pane switch kune iyo michina yese yakabatana - izvi ndizvo Pamusoro peiyo Rack switch - ToR kana neimwe nzira, maererano nefekitari yeClos, tichaidaidza Leaf.
General diagram yefekitari.
Tichavadaidza XXX-shizhaYkupi XXX - mabhii matatu chidimbu DC, uye Y - siriyero nhamba. Semuyenzaniso, kzn-shizha11.
Muzvinyorwa zvangu ini ndichazvibvumira kushandisa mazwi Leaf uye ToR pane frivolously semafanana. Zvisinei, tinofanira kuyeuka kuti izvi hazvisizvo.
ToR is switch yakaiswa mune rack iyo michina yakabatana.
Leaf ibasa remudziyo mune network yemuviri kana yekutanga-level switch maererano neCloes topology.
Kureva, Shizha != ToR.
Saka Leaf inogona kuve EndofRaw switch, semuenzaniso.
Zvakadaro, mukati megadziriro yechinyorwa chino tichavabata semafanana.
Imwe neimwe ToR switch inobatanidzwa kune ina yepamusoro-level aggregation switch - Wakapona. Imwe rack muDC yakagoverwa Spines. Tichazvitumidza zvakafanana: XXX-muzongozaY.
Iyo rack imwechete ichange iine network michina yekubatanidza pakati peDC - 2 ma routers ane MPLS pabhodhi. Asi kazhinji, aya akafanana maToR. Ndokunge, kubva pakuona kweSpine switches, yakajairwa ToR nemichina yakabatana kana router yeDCI haina basa zvachose - kungoendesa mberi.
MaToR akakosha akadai anonzi Edge-shizha. Tichavadaidza XXX-kumuchetoY.
Zvichaita seizvi.
Mumufananidzo uri pamusoro, ini ndakaisa mupendero neshizha padanho rimwe chete. Ivo vakatidzidzisa kufunga uplinking (saka izwi) seuplinks. Uye pano zvinozoitika kuti iyo DCI "uplink" inodzokera pasi, iyo kune vamwe zvishoma inotyora yakajairika logic. Panyaya yemataneti makuru, kana nzvimbo dze data dzakakamurwa kuita zvidimbu zvidiki - POD's (Point Of Delivery), simbisa munhu Edge-POD's yeDCI uye kuwana kune ekunze network.
Kuti kuve nyore kuona mune ramangwana, ini ndicharamba ndichidhirowa Edge pamusoro peSpine, isu tichiramba tichifunga kuti hapana hungwaru paSpine uye hapana misiyano kana uchishanda neakajairwa Leaf uye Edge-shizha (kunyangwe panogona kunge paine nuances pano. , asi kazhinji Izvi ndezvechokwadi).
Scheme yefekitori ine Edge-mashizha.
Utatu hweLeaf, Spine uye Edge zvinogadzira Underlay network kana fekitori.
Basa retiweki fekitori (verenga Underlay), sezvatakatsanangura kare mukati , zvakanyanya, zviri nyore - kupa IP yekubatanidza pakati pemichina mukati meiyo imwechete DC uye pakati pavo.
Ndokusaka network ichinzi fekitori, senge, semuenzaniso, fekitori inoshandura mukati memodular network mabhokisi, ayo iwe unogona kuverenga zvakawanda pamusoro. .
Kazhinji, topology yakadaro inonzi fekitari, nokuti jira mukushandura rinoreva jira. Uye zvakaoma kusabvumirana:
Iyo fekitori yakazara L3. Kwete VLAN, hapana Nhepfenyuro - isu tine vakadadisa programmers kuLAN_DC, ivo vanoziva kunyora manyorerwo anogara muL3 paradigm, uye machina chaiwo haadi Live Migration nekuchengetedza IP kero.
Uye zvakare: mhinduro kumubvunzo wekuti nei fekitori uye nei L3 iri mune yakaparadzana .
DCI - Data Center Interconnect (Inter-DC)
DCI ichave yakarongeka tichishandisa Edge-Leaf, ndiko kuti, ndiyo nzvimbo yedu yekubuda kumugwagwa mukuru.
Kuti zvive nyore, isu tinofungidzira kuti maDC akabatana kune mumwe nemumwe nemaitiro akananga.
Ngatisabvisa kubatana kwekunze kubva pakufunga.
Ndinoziva kuti pese pandinobvisa chikamu, ndinorerutsa network. Uye kana isu tichigadzira yedu abstract network, zvese zvichave zvakanaka, asi pane chaiyo pachave nemadondoro.
Ichi ichokwadi. Zvakadaro, poindi yeiyi nhevedzano ndeyekufunga nekushanda pane nzira, kwete kugadzirisa zvine hushingi matambudziko ekufungidzira.
PaEdge-Leafs, iyo underlay inoiswa muVPN uye inofambiswa kuburikidza neMPLS musana (iyo yakafanana yakananga link).
Iri ndiro dhayagiramu yepamusoro-soro yatinowana.
Routing
Kufambisa mukati meDC tichashandisa BGP.
Pahunde yeMPLS OSPF+LDP.
YeDCI, ndiko kuti, kuronga yekubatanidza pasi pevhu - BGP L3VPN pamusoro peMPLS.
General routing scheme
Iko hakuna OSPF kana ISIS (roti yekufambisa inorambidzwa muRussian Federation) pafekitari.
Izvi zvinoreva kuti hapazove nekuzviwanira-kuzviwanira kana kuverenga kwenzira pfupi - bhuku chete (chaizvoizvo otomatiki - tiri kutaura nezve otomatiki pano) kumisikidza protocol, nharaunda uye marongero.
BGP routing scheme mukati meDC
Sei BGP?
Panyaya iyi pane yakatumidzwa zita reFacebook naArista, iyo inotaurira maitiro ekuvaka yakakura kwazvo data centre network uchishandisa BGP. Inoverenga senge manyepo, ini ndinoikurudzira zvakanyanya manheru asina kusimba.
Uye kune zvakare chikamu chose muchinyorwa changu chakatsaurirwa kune ichi. Ndokuendesa kupi uye .
Asi zvakadaro, muchidimbu, hapana IGP yakakodzera kunetiweki yenzvimbo huru dzedata, uko nhamba yemidziyo yetiweki inomhanya muzviuru.
Uye zvakare, kushandisa BGP kwese kwese kunobvumidza iwe kuti usatambise nguva pakutsigira akati wandei maprotocol uye kuwiriranisa pakati pavo.
Ruoko pamwoyo, mufekitari yedu, iyo ine chiyero chepamusoro chemukana haizokura nokukurumidza, OSPF yaizova yakakwana yemaziso. Aya ndiwo chaiwo matambudziko e megascaler uye gore titans. Asi ngatifungei nezve mashoma ekuburitswa atinoda, uye isu tichashandisa BGP, sekupihwa kwaPyotr Lapukhov.
Routing Policies
PaLeaf switch, tinopinza prefixes kubva ku Underlay network interfaces muBGP.
Tichava nemusangano weBGP pakati imwe neimwe a Leaf-Spine pair, umo aya Underlay prefixes achaziviswa pamusoro pe network pano nepapo.
Mukati meimwe nzvimbo yedata, tichagovera zvakatemwa zvatakaunza kunze kuToRe. PaEdge-Leafs tichavaunganidza tovazivisa kune maDC ari kure uye tovatumira pasi kuTORs. Kureva kuti, imwe neimwe ToR inonyatsoziva nzira yekusvika kune imwe ToR mune imwecheteyo DC uye kune yekupinda nzvimbo yekusvika kuToR mune imwe DC.
MuDCI, nzira dzichafambiswa seVPNv4. Kuti uite izvi, paEdge-Leaf, iyo inotarisana nefekitori ichaiswa muVRF, ngatiidaidze UNDERLAY, uye nharaunda ine Spine paEdge-Leaf ichasimuka mukati meVRF, uye pakati peEdge-Leafs muVPNv4- mhuri.
Isu ticharambidza zvakare kuziviswa kwenzira dzakagamuchirwa kubva kune spines kudzokera kwavari.
PaLeaf uye Spine isu hatisi kuzopinza Loopbacks. Isu tinongoda ivo chete kuti vatarise iyo Router ID.
Asi paEdge-Leafs tinoipinza muGlobal BGP. Pakati pemakero eLoopback, Edge-Leafs ichagadzira musangano weBGP muIPv4 VPN-mhuri pamwe chete.
Tichava ne OSPF + LDP musana pakati peEDGE zvishandiso. Zvese zviri munzvimbo imwe chete. Kunyanya nyore kugadzirisa.
Uyu ndiwo mufananidzo une routing.
BGP ASN
Edge-Leaf ASN
PaEdge-Leafs pachave neASN imwe mumaDC ese. Izvo zvakakosha kuti pane iBGP pakati peEdge-Leafs, uye isu hatibatike mumanuances eBGP. Ngaive 65535. Muchokwadi, iyi inogona kunge iri nhamba yeruzhinji AS.
Spine ASN
PaSpine tichava neASN imwe paDC. Ngatitangei pano nenhamba yekutanga kubva kune yakavanzika AS - 64512, 64513 Uye zvichingodaro.
Sei ASN paDC?
Ngatipatsanurei mubvunzo uyu kuva maviri:
- Sei maASN akafanana pane ese maspines eimwe DC?
- Sei akasiyana mumaDC akasiyana?
Sei akafanana maASN pane ese spines eimwe DC?
Izvi ndizvo zvicharatidzika seAS-Path yeUnderlay nzira paEdge-Leaf:
[leafX_ASN, spine_ASN, edge_ASN]
Paunoyedza kuishambadza ichidzokera kuSpine, inoirasa nekuti AS yayo (Spine_AS) yatove mune rondedzero.
Zvisinei, mukati meDC tinogutsikana zvachose kuti nzira dzeUnderlay dzinokwira kuEdge hadzizokwanisi kudzika. Kutaurirana kwese pakati pevaenzi mukati meDC kunofanirwa kuitika mukati meiyo musana level.
Muchiitiko ichi, nzira dzakaunganidzwa dzemamwe maDC anozosvika nyore nyore kuToRs - yavo AS-Path inongove iine ASN 65535 - nhamba yeAS Edge-Leafs, nekuti ndipo pavakasikirwa.
Sei akasiyana mumaDC akasiyana?
Nechepfungwa, isu tingangoda kudhonza Loopback uye mamwe masevhisi chaiwo michina pakati peDCs.
Semuenzaniso, pane muenzi tichamhanya Route Reflector kana (Virtual Network Gateway), iyo ichavhara neTopR kuburikidza neBGP uye kuzivisa loopback yayo, iyo inofanirwa kuwanikwa kubva kune ese maDC.
Saka izvi ndizvo zvichange zvakaita AS-Path yayo:
[VNF_ASN, leafX_DC1_ASN, spine_DC1_ASN, edge_ASN, spine_DC2_ASN, leafY_DC2_ASN]
Uye hapafanirwe kuve neduplicate ASNs chero kupi.
Kureva kuti, Spine_DC1 uye Spine_DC2 dzinofanira kunge dzakasiyana, sezvakangoita leafX_DC1 uye leafY_DC2, zvinova ndizvo chaizvo zvatiri kusvika.
Sezvaungangoziva iwe, kune ma hacks anokutendera kuti ugamuchire nzira dzine duplicate ASNs kunyangwe iyo loop yekudzivirira nzira (allowas-in paCisco). Uye inotova nemashandisirwo ari pamutemo. Asi iri ndiro gwanza rinogona kuitika mukugadzikana kwetiweki. Uye ini pachangu ndakawira mairi kaviri.
Uye kana tikawana mukana wokusashandisa zvinhu zvine ngozi, tichashandisa mukana iwoyo.
Leaf ASN
Tichava nemunhu wega ASN pane imwe neimwe Leaf switch mukati metiweki.
Isu tinoita izvi nekuda kwezvikonzero zvakapihwa pamusoro: AS-Path isina zvishwe, BGP kumisikidzwa isina mabhukimaki.
Kuti nzira dziri pakati peMashizha dzipfuure zvakanaka, iyo AS-Path inofanirwa kutaridzika seizvi:
[leafX_ASN, spine_ASN, leafY_ASN]
apo leafX_ASN uye leafY_ASN zvingava zvakanaka kusiyana.
Izvi zvinodikanwawo kune mamiriro acho nekuziviswa kweVNF loopback pakati peDCs:
[VNF_ASN, leafX_DC1_ASN, spine_DC1_ASN, edge_ASN, spine_DC2_ASN, leafY_DC2_ASN]
Isu tichashandisa 4-byte ASN uye kuigadzira zvichibva paSpine's ASN uye iyo Leaf switch nhamba, kureva, seizvi: Spine_ASN.0000X.
Uyu ndiwo mufananidzo une ASN.
IP chirongwa
Chakanyanya kukosha, isu tinofanirwa kugovera kero yezvibatanidza zvinotevera:
- Underlay network kero pakati peToR nemuchina. Iwo anofanirwa kuve akasiyana mukati metiweki yese kuitira kuti chero muchina ugone kutaurirana nemumwe. Kukwana kukuru 10/8. Kune yega yega rack pane /26 ine chengetedzo. Tichagovera / 19 paDC uye / 17 padunhu.
- Batanidza kero pakati peLeaf/Tor uye Spine.
Ndinoda kuvapa algorithmically, ndiko kuti, vaverenge kubva pamazita emidziyo inoda kubatanidzwa.
Ngazvive... 169.254.0.0/16.
Naizvozvo 169.254.00X.Y/31kupi X - musana nhamba, Y - P2P network /31.
Izvi zvinokutendera kuti utange kusvika ku128 racks, uye anosvika gumi Spines muDC. Batanidza kero inogona (uye ichadzokororwa) kubva kuDC kuenda kuDC. - Isu tinoronga iyo Spine-Edge-Leaf junction pane subnets 169.254.10X.Y/31, papi chaipo X - musana nhamba, Y - P2P network /31.
- Batanidza kero kubva kuEdge-Leaf kuenda kuMPLS musana. Pano mamiriro ezvinhu akati siyanei - iyo nzvimbo iyo zvidimbu zvese zvakabatanidzwa mune imwe pie, saka kushandisa zvakare madhigirii akafanana hazvishande - iwe unofanirwa kusarudza inotevera yemahara subnet. Naizvozvo, ngatitore sechikonzero 192.168.0.0/16 uye tichavhomora vakasununguka mariri.
- Loopback Kero. Isu tinopa iyo yese siyana kwavari 172.16.0.0/12.
- Leaf - / 25 per DC - yakafanana 128 racks. Tichagovera / 23 padunhu.
- Muzongoza - / 28 paDC - kusvika ku16 Spine. Ngatigovei /26 padunhu.
- Edge-Leaf - / 29 paDC - anosvika masere mabhokisi. Ngatigovei /8 padunhu.
Kana isu tisina akakwana akagoverwa masanji muDC (uye hapazove nechero - isu tinoti tiri mahyperscalers), isu tinongosarudza chivharo chinotevera.
Uyu ndiwo mufananidzo une IP kero.
Loopbacks:
Prefix
Basa remudziyo
Nharaunda
ΠΠ¦
172.16.0.0/23
mucheto
172.16.0.0/27
ru
172.16.0.0/29
msk
172.16.0.8/29
kzn
172.16.0.32/27
sp
172.16.0.32/29
bcn
172.16.0.40/29
mlg
172.16.0.64/27
cn
172.16.0.64/29
sha
172.16.0.72/29
zvose zviri zviviri
172.16.2.0/23
muzongoza
172.16.2.0/26
ru
172.16.2.0/28
msk
172.16.2.16/28
kzn
172.16.2.64/26
sp
172.16.2.64/28
bcn
172.16.2.80/28
mlg
172.16.2.128/26
cn
172.16.2.128/28
sha
172.16.2.144/28
zvose zviri zviviri
172.16.8.0/21
mashizha
172.16.8.0/23
ru
172.16.8.0/25
msk
172.16.8.128/25
kzn
172.16.10.0/23
sp
172.16.10.0/25
bcn
172.16.10.128/25
mlg
172.16.12.0/23
cn
172.16.12.0/25
sha
172.16.12.128/25
zvose zviri zviviri
Underlay:
Prefix
Nharaunda
ΠΠ¦
10.0.0.0/17
ru
10.0.0.0/19
msk
10.0.32.0/19
kzn
10.0.128.0/17
sp
10.0.128.0/19
bcn
10.0.160.0/19
mlg
10.1.0.0/17
cn
10.1.0.0/19
sha
10.1.32.0/19
zvose zviri zviviri
Laba
Vatengesi vaviri. Imwe network. ADSM.
Juniper + Arista. Ubuntu. Wakanaka Evha.
Huwandu hwezviwanikwa pane yedu chaiyo server muMirana huchiri hushoma, saka kudzidzira isu tichashandisa network inorerutswa kusvika kumagumo.
Nzvimbo mbiri dze data: Kazan neBarcelona.
- Mitsipa miviri imwe neimwe: Juniper naArista.
- Imwe torus (Leaf) mune imwe neimwe - Juniper naArista, ine imwe chete yakabatana inotambira (ngatitorei isingaremi Cisco IOL yeizvi).
- Imwe Edge-Leaf node imwe neimwe (ikozvino Juniper chete).
- Imwe Cisco switch kuti ivatonge vese.
- Pamusoro pemabhokisi etiweki, muchina wekutonga uri kushanda. Kumhanya Ubuntu.
Iyo ine mukana kune ese maturusi, ichamhanyisa IPAM/DCIM masisitimu, boka rePython zvinyorwa, Ansible uye chero chinhu chatingade.
yezvese zvishandiso zvetiweki, zvatichaedza kubereka tichishandisa otomatiki.
mhedziso
Izvo zvinogamuchirwawo here? Ndinofanira kunyora mhedziso pfupi pasi pechinyorwa chimwe nechimwe here?
Saka takasarudza Kloz network mukati meDC, sezvo isu tinotarisira yakawanda yeEast-West traffic uye tinoda ECMP.
Iyo network yakakamurwa kuita yemuviri (pasi pasi) uye chaiyo (yakavharika). Panguva imwecheteyo, kuvharika kunotanga kubva kumugadziri - nekudaro kurerutsa zvinodiwa kune underlay.
Isu takasarudza BGP senzira yenzira yetiweki network nekuda kweiyo scalability uye kuchinjika kwepolicy.
Tichava nenzvimbo dzakasiyana dzekuronga DCI - Edge-shizha.
Iyo musana ichave ine OSPF + LDP.
DCI ichaitwa yakavakirwa paMPLS L3VPN.
Kune P2P zvinongedzo, isu tichaverenga IP kero algorithmically zvichibva pamazita emudziyo.
Isu tichapa loopbacks zvinoenderana nebasa remidziyo uye nzvimbo yavo sequentially.
Underlay prefixes - chete paLeaf switches zvakatevedzana zvichienderana nenzvimbo yavo.
Ngatifungei kuti izvozvi hatisati tave nemichina yakaiswa.
Naizvozvo, nhanho dzedu dzinotevera dzichava dzekuwedzera kune masisitimu (IPAM, inventory), kuronga kupinda, kugadzira gadziriso uye kuiendesa.
Muchinyorwa chinotevera tichabata neNetbox - inventory uye manejimendi system yeIP nzvimbo muDC.
Ndatenda
- Andrey Glazkov aka @glazgoo yekuongorora uye kugadzirisa
- Alexander Klimenko aka @v00lk yekuongorora uye kugadzirisa
- Artyom Chernobay yeKDPV
Source: www.habr.com