Isu takagadzira data center network dhizaini inobvumira kutumirwa kwemasumbu emakomputa akakura kupfuura zviuru zana maseva ane peak bisection bandwidth inopfuura petabyte imwe pasekondi.
Kubva mushumo waDmitry Afanasyev iwe uchadzidza nezve misimboti yedhizaini nyowani, kuyera topologies, matambudziko anomuka neizvi, sarudzo dzekuagadzirisa, maficha ekufambisa uye kuyera mabasa endege yekufambisa yemidziyo yemazuva ano network mu "densely yakabatana" topology ine nhamba huru yeECMP nzira . Mukuwedzera, Dima akataura muchidimbu nezve sangano rekubatana kwekunze, chidimbu chemuviri, cabling system uye nzira dzekuwedzera kuwedzera simba.
- Masikati akanaka mose! Zita rangu ndiDmitry Afanasyev, ndiri mugadziri wetiweki paYandex uye ndinonyanya kugadzira data center network.
Nyaya yangu ichava pamusoro peiyo yakagadziridzwa network yeYandex data centers. Yakanyanya kushanduka kwedhizaini yataive nayo, asi panguva imwechete kune zvimwe zvitsva zvinhu. Iyi iratidziro yemharidzo nekuti pakanga paine ruzivo rwakawanda rwaifanira kurongedzerwa munguva diki. Isu tinotanga nekusarudza inonzwisisika topology. Ipapo pachava nekutarisisa kwendege yekudzora uye matambudziko ane data ndege scalability, sarudzo yezvichaitika padanho remuviri, uye isu tichatarisa mamwe maficha emidziyo. Ngatimbobatai zvishoma pane zviri kuitika munzvimbo yedata neMPLS, yatakataura nezvayo nguva yapfuura.
Saka, chii chinonzi Yandex maererano nemitoro uye mabasa? Yandex ndiyo yakajairika hyperscaler. Kana tikatarisa vashandisi, isu tinonyanya kugadzirisa zvikumbiro zvevashandisi. Zvakare akasiyana masevhisi ekutepfenyura uye kuendesa data, nekuti isu zvakare tine ekuchengetedza masevhisi. Kana iri padyo nekumashure, ipapo mitoro yezvivakwa uye masevhisi anooneka ipapo, senge akagoverwa kuchengetedza chinhu, kudzokororwa kwedata uye, hongu, mitsetse inoenderera. Imwe yemhando huru dzekuremerwa kwemabasa iMepuReduce uye akafanana masisitimu, kuyerera kwerukova, kudzidza muchina, nezvimwe.
Mamiriro ezvinhu ari pamusoro pazvo zvese izvi zvinoitika sei? Zvekare, isu tiri vakanaka vakajairwa hyperscaler, kunyangwe isu pamwe tiri padyo padyo nediki hyperscaler divi reiyo spectrum. Asi isu tine hunhu hwese. Isu tinoshandisa commodity hardware uye horizontal scaling pese pazvinogoneka. Isu tine yakazara zviwanikwa kubatanidza: isu hatishande nemichina yega, mareki ega, asi anoasanganisa kuita dziva rakakura rezviwanikwa zvinochinjika nemamwe masevhisi ekuwedzera anobata nekuronga uye kugovera, uye kushanda nedziva rese iri.
Saka isu tine inotevera nhanho - iyo inoshanda sisitimu padanho recomputing cluster. Izvo zvakakosha kuti tidzore zvizere tekinoroji stack yatinoshandisa. Isu tinodzora magumo (mauto), network uye software stack.
Tine akati wandei makuru data nzvimbo muRussia uye kunze. Ivo vakabatanidzwa nemusana unoshandisa MPLS tekinoroji. Zvivakwa zvedu zvemukati zvinenge zvakavakirwa paIPv6, asi sezvo isu tichida kushandira kunze traffic iyo ichiri kuuya zvakanyanya pamusoro peIPv4, isu tinofanira neimwe nzira kuendesa zvikumbiro zvinouya pamusoro peIPv4 kumaseva epamberi, uye zvimwe zvishoma kuenda kunze IPv4- Internet - ye semuenzaniso, kune indexing.
Iwo mashoma ekupedzisira ekupedzisira e data centre network madhizaini akashandisa akawanda-layer Clos topology uye ari L3-chete. Takabva kuL2 nguva yadarika ndokutura befu. Chekupedzisira, zvivakwa zvedu zvinosanganisira mazana ezviuru zvemakomputa (server) zviitiko. Hukuru hwemasumbu pane imwe nguva yapfuura hwaive hunosvika zviuru gumi zvemaseva. Izvi zvinonyanya kukonzerwa nemashandisirwo emasisitimu e-cluster-level anoshanda, vanoronga, kugovera zviwanikwa, zvichingodaro.Sezvo kufambira mberi kwakaitika kudivi resoftware yezvivakwa, saizi yekunangwa yave kuita zviuru zana zvemaseva mune imwe komputa cluster, uye Tine basa - kukwanisa kuvaka mafekitori etiweki anobvumira kubatanidza zviwanikwa muboka rakadaro.
Chii chatinoda kubva kune data center network? Chekutanga pane zvese, kune yakawanda yakachipa uye yakaenzana yakagovaniswa bandwidth. Nekuti network ndiyo musana watinogona kubatanidza zviwanikwa. Saizi itsva yakanangwa ndeye 100 zviuru zvemaseva musumbu rimwe.
Isuwo, hongu, tinoda scalable uye yakagadzikana inodzora ndege, nekuti pazvivakwa zvakakura zvakadaro misoro yakawanda inomuka kunyangwe kubva kuzviitiko zvisingaite, uye hatidi kuti ndege yekutonga itiunzirewo misoro. Panguva imwecheteyo, tinoda kuderedza hurumende mairi. Iyo diki iyo mamiriro, zviri nani uye zvakanyanya kugadzikana zvese zvinoshanda, uye zviri nyore kuongorora.
Ehezve, isu tinoda otomatiki, nekuti hazvibviri kubata magadzirirwo akadaro nemaoko, uye zvanga zvisingaite kwenguva yakati. Tinoda rubatsiro rwekushanda zvakanyanya sezvinobvira uye CI / CD tsigiro kusvika pamwero waanogona kupihwa.
Nemasaizi akadai enzvimbo dzedata uye masumbu, basa rekutsigira kuwedzera kwekuisa uye kuwedzera pasina kukanganiswa kwebasa rave rakaoma. Kana pamasumbu ehukuru hwechiuru chemichina, pamwe padyo nezviuru gumi zvemichina, inogona kuramba ichitenderedzwa senge oparesheni imwe - ndiko kuti, tiri kuronga kuwedzera kwezvivakwa, uye zviuru zvinoverengeka zvemichina inowedzerwa sekushanda kumwe chete. zvino sumbu rehukuru hwemazana ezviuru zvemichina harisimuke pakarepo seizvi, rinovakwa nekufamba kwenguva. Uye zvinodikanwa kuti nguva yese iyi izvo zvakatopombwa kunze, zvivakwa zvakaiswa, zvinofanira kuwanikwa.
Uye chimwe chinodiwa chataive nacho uye takasiya: rutsigiro rwemultitenancy, kureva, virtualization kana network segmentation. Iye zvino isu hatidi kuita izvi padanho remachira etiweki, nekuti sharding yaenda kune mauto, uye izvi zvaita kuti kuyera kuve nyore kwatiri. Tinotenda IPv6 nenzvimbo hombe yekero, isu taisada kushandisa kero dzakapetwa mune zvemukati zvivakwa; kero dzese dzaive dzakatosiyana. Uye nekuda kwekuti isu tatora kusefa uye netiweki segmentation kune mauto, isu hatifanire kugadzira chero chaiwo network masangano mu data center network.
Chinhu chakakosha ndechekuti isu hatidi. Kana mamwe mabasa anogona kubviswa kubva kunetiweki, izvi zvinoita kuti hupenyu huve nyore, uye, sekutonga, inowedzera kusarudzwa kwezvinhu zviripo uye software, zvichiita kuti kuongororwa kuve nyore.
Saka, chii chatisingadi, chii chatakakwanisa kurega, kwete nguva dzose nerufaro panguva yazvakaitika, asi nekusununguka kukuru apo nzira yapera?
Chekutanga, kusiya L2. Hatidi L2, kwete chaiyo kana kutevedzerwa. Haina kushandiswa zvakanyanya nekuda kwekuti isu tinodzora application stack. Zvikumbiro zvedu zvinokura zvakadzika, vanoshanda neL3 addressing, havasi kunetsekana kuti imwe nyaya yabuda, vanongoburitsa imwe nyowani, haidi kuti iburitswe pakero yekare, nekuti pane yakaparadzana nhanho yekuwanikwa kwesevhisi uye kutarisa kwemichina iri musumbu. Isu hatisi kupa basa iri kune network. Basa retiweki ndere kuendesa mapaketi kubva panzvimbo A kuenda kunongedzo B.
Isu hatinawo mamiriro ezvinhu apo kero dzinofamba mukati metiweki, uye izvi zvinoda kuongororwa. Mumagadzirirwo mazhinji izvi zvinowanzodiwa kutsigira VM kufamba. Isu hatishandisi mafambiro emagetsi emagetsi mukati memukati meiyo Yandex hombe, uyezve, isu tinotenda kuti kunyangwe izvi zvikaitwa, hazvifanirwe kuitika nerutsigiro rwetiweki. Kana ichida chaizvo kuitwa, inoda kuitwa padanho rekugamuchira, uye kusunda kero dzinogona kutamira kune zvakavharika, kuitira kuti usabate kana kuita shanduko dzakawandisa kune iyo routing system yeiyo underlay pachayo (transport network) .
Imwe tekinoroji yatisingashandisi ndeye multicast. Kana uchida, ndinogona kukuudza zvakadzama kuti sei. Izvi zvinoita kuti hupenyu huve nyore, nekuti kana mumwe munhu akabata nazvo uye akatarisa chaizvo izvo ndege ye multicast inotaridzika, mune zvese kunze kweiyo yakapfava yekumisikidza, uku kurwadza kwemusoro. Uye chii chimwe, zvakaoma kuwana inoshanda-yakasununguka sosi yekushandisa, semuenzaniso.
Chekupedzisira, tinogadzira manetwork edu kuti asanyanya kuchinja. Tinogona kuvimba pamusoro pekuti kuyerera kwezviitiko zvekunze muhurongwa hwekufambisa hudiki.
Ndeapi matambudziko anomuka uye zvirambidzo zvipi zvinofanirwa kuverengerwa patinogadzira data data network? Mutengo, hongu. Scalability, nhanho yatinoda kukura. Kudiwa kwekuwedzera pasina kumisa sevhisi. Bandwidth, kuwanikwa. Kuonekwa kwezviri kuitika pamambure ekutarisa masisitimu, ezvikwata zvinoshanda. Automation rutsigiro - zvakare, zvakanyanya sezvinobvira, sezvo mabasa akasiyana anogona kugadziriswa pamatanho akasiyana, kusanganisira kuiswa kwemamwe zvidimbu. Zvakanaka, kwete [zvichida] zvinoenderana nevatengesi. Kunyangwe munguva dzakasiyana dzenhoroondo, zvichienderana nekuti ndechipi chikamu chaunotarisa, kuzvimiririra uku kwaive nyore kana kuti kwakaoma kuwana. Kana tikatora muchinjiko-chikamu chetiweki mudziyo machipisi, zvino kusvika nguva pfupi yadarika zvaive zvine chirevo kutaura nezve rusununguko kubva kune vatengesi, kana isu taidawo machipisi ane yakakwirira bandwidth.
Ndeipi topology ine musoro yatichashandisa kuvaka network yedu? Iyi ichave yakawanda-level Clos. Kutaura zvazviri, hapana dzimwe nzira chaidzo panguva ino. Uye iyo Clos topology yakanaka chaizvo, kunyangwe ichienzaniswa neakasiyana epamusoro epamusoro ayo ari munzvimbo yekufarira zvedzidzo izvozvi, kana isu tine ma radix machinjo makuru.
Iyo yakawanda-level Clos network yakanyatso kurongeka uye ndezvipi zvinhu zvakasiyana zvinodaidzwa mairi? Pakutanga, mhepo yakasimuka kuti uende kumusoro, uko kune zasi, uko kumabvazuva, uko kumavirira. Manetiweki emhando iyi anowanzo kuvakwa neavo vane yakakura kwazvo kumadokero-kumabvazuva traffic. Kana zviri izvo zvasara zvinhu, kumusoro kune dhizaini dhizaini yakaunganidzwa kubva kudiki switch. Iyi ndiyo pfungwa huru yekudzokororwa kuvaka kweClos network. Isu tinotora zvinhu zvine imwe mhando yeradix tozvibatanidza kuitira kuti izvo zvatinowana zvionekwe sechinochinja chine radix yakakura. Kana iwe uchida zvakatowanda, maitiro anogona kudzokororwa.
Muzviitiko, semuenzaniso, neClos maviri-level, kana zvichikwanisika kujekesa zvakajeka zvikamu zvakamira mumufananidzo wangu, zvinowanzonzi ndege. Dai taizovaka Clos ine mazinga matatu emuzongoza switch (ese asiri muganho kana ToR switch uye ayo anoshandiswa chete pakufambisa), ipapo ndege dzaizotaridzika zvakanyanya kuomarara; maviri-nhanho anotaridzika chaizvo seizvi. Isu tinodaidza bhuroka yeToR kana mashizha switch uye yekutanga nhanho spine switch yakabatana navo Pod. Spine switches yemuzongoza-1 level kumusoro kwePod ndiyo yepamusoro pePod, pamusoro pePod. Maswichi ari pamusoro pefekitari yese ndiyo yepamusoro pefekitori, Pamusoro pemachira.
Ehe, mubvunzo unomuka: Clos network akavakwa kwenguva yakati; iyo pfungwa pachayo inowanzobva kunguva dzechinyakare telephony, TDM network. Zvichida chimwe chinhu chiri nani chakaonekwa, zvichida chimwe chinhu chinogona kuitwa nani? Hongu uye kwete. Zvichireva hongu, mukuita munguva pfupi iri kutevera zvirokwazvo kwete. Nemhaka yokuti kune nhamba yezvinyorwa zvinonakidza, zvimwe zvacho zvinotoshandiswa mukugadzirwa, semuenzaniso, Dragonfly inoshandiswa mukushandiswa kweHPC; Kune zvakare anonakidza topology akadai seXpander, FatClique, Jellyfish. Kana iwe ukatarisa mishumo pamisangano yakaita seSIGCOMM kana NSDI nguva pfupi yadarika, unogona kuwana nhamba yakati wandei yemabasa pane mamwe matopology ane zvivakwa zviri nani (imwe kana imwe) kupfuura Clos.
Asi ese aya matopology ane chinhu chimwe chinonakidza. Iyo inodzivirira kuita kwavo mu data center network, yatiri kuyedza kuvaka pane commodity Hardware uye inodhura mari inonzwisisika. Mune ese aya mamwe matopology, mazhinji ebandwidth zvinosuruvarisa kuti haasvikike kuburikidza nepfupi nzira. Naizvozvo, isu tinobva tarasikirwa nemukana wekushandisa iyo yechinyakare kudzora ndege.
Zvichitaurwa, mhinduro yedambudziko iri inozivikanwa. Aya ndiwo, semuenzaniso, kugadziridzwa kwenzvimbo yekubatanidza uchishandisa k-pfupi nzira, asi, zvakare, hapana mapuroteni akadaro angazoitwa mukugadzira uye anowanikwa zvakanyanya pamidziyo.
Uyezve, sezvo huwandu hwehuwandu husingasvikike nenzira dzakapfupika, isu tinofanirwa kushandura zvinopfuura ndege yekudzora kuti tisarudze ese aya makwara (uye nenzira, izvi zvakanyanya kuwedzera mamiriro mundege yekudzora). Tichiri kufanira kugadzirisa ndege yekuendesa mberi, uye, sekutonga, zvimwe zvinhu zviviri zvekuwedzera zvinodiwa. Uku ndiko kugona kuita sarudzo dzese nezve kutumira kwepaketi imwe-nguva, semuenzaniso, pamugamuchiri. Muchokwadi, iyi ndiyo sosi nzira, dzimwe nguva mumabhuku ekubatanidza network iyi inonzi zvese-kamwe-kamwe kutumira sarudzo. Uye adaptive routing ibasa ratinoda pane network zvinhu, izvo zvinowira pasi, semuenzaniso, kune chokwadi chekuti isu tinosarudza hop inotevera zvichienderana neruzivo nezve mudiki mutoro pamutsara. Semuenzaniso, dzimwe sarudzo dzinogoneka.
Nokudaro, nhungamiro inofadza, asi, nhamo, hatigoni kuishandisa iko zvino.
Zvakanaka, isu takagara paClos inonzwisisika topology. Tichaiyera sei? Ngationei kuti inoshanda sei uye chii chingaitwa.
Munetiweki yeClos kune maviri makuru ma paramita atinogona neimwe nzira kusiyanisa uye kuwana mamwe mhedzisiro: radix yezvinhu uye huwandu hwemazinga mune network. Ndine dhizaini rekuti zvese zvinokanganisa saizi. Zvakanaka, tinobatanidza zvose.
Zvinogona kuonekwa kuti hupamhi hwekupedzisira hweClos network ndiyo chibereko chemazinga ose emusana wekushanduka kweradix yekumaodzanyemba, mangani ma links atinawo pasi, kuti matavi sei. Aya ndiwo maitiro atinoita saizi yetiweki.
Nezve huwandu, kunyanya paToR switch, pane maviri ekuyera sarudzo. Pamwe isu tinogona, tichichengetedza iyo general topology, shandisa zvinongedzo zvinokurumidza, kana isu tinogona kuwedzera dzimwe ndege.
Kana iwe ukatarisa iyo yakawedzera vhezheni yeClos network (pakona yekurudyi) uye wodzokera kumufananidzo uyu neClos network pazasi...
... zvino iyi ndiyo chaiyo topology, asi pane iyi slide inodonha zvakanyanya uye ndege dzefekitori dzakaiswa pamusoro peumwe neumwe. Zvakafanana.
Kuyera network yeClos kunoratidzika sei munhamba? Pano ini ndinopa data pamusoro pehupamhi hwehupamhi hupi hunogona kuwanikwa netiweki, ndeapi huwandu hwema racks, ToR switch kana mashizha anochinja, kana asiri mumaraki, tinogona kuwana zvichienderana nekuti ndeipi radix yekuchinja yatinoshandisa kumusana -level, uye mangani mazinga atinoshandisa.
Heano mangani maracks atinogona kuve nawo, mangani maseva uye angangoita mangani zvese izvi zvinogona kushandisa zvichibva pa20 kW paraki. Nguva pfupi yapfuura ndakataura kuti isu tiri kuvavarira hukuru hwemasumbu hunosvika zviuru zana maseva.
Zvinogona kuonekwa kuti mune iyi yose yakagadzirwa, zvingasarudzwa zviviri nehafu zvinofadza. Iko kune sarudzo ine maviri akaturikidzana e spines uye 64-chiteshi switch, inodonha zvishoma. Kozoita sarudzo dzakanyatsokodzera dze128-port (ine radix 128) spine switch ine mazinga maviri, kana switch ine radix 32 ine mazinga matatu. Uye muzviitiko zvose, uko kune mamwe radix uye mamwe mazamu, unogona kuita network yakakura kwazvo, asi kana iwe ukatarisa kushandiswa kunotarisirwa, kazhinji kune gigawatts. Zvinogoneka kuisa tambo, asi isu hatigone kuwana magetsi akawanda kudaro panzvimbo imwe. Kana iwe ukatarisa nhamba uye neruzhinji data pane data centers, unogona kuwana mashoma data centers ane hukasha hunofungidzirwa hunopfuura 150 MW. Iwo mahombe anowanzo dhata centre makampasi, akati wandei mahombe data centers ari padhuze nepadhuze.
Pane imwe inokosha parameter. Kana iwe ukatarisa kuruboshwe rworuboshwe, inoshandiswa bandwidth yakanyorwa ipapo. Zviri nyore kuona kuti mune Clos network chikamu chakakosha chezviteshi chinoshandiswa kubatanidza switch kune mumwe nemumwe. Inoshandiswa bandwidth, tambo inobatsira, chimwe chinhu chinogona kupihwa kunze, kune maseva. Nomuzvarirwo, ndiri kutaura nezve conditional ports uye chaizvo nezve bhendi. Sezvo mutemo, zvinongedzo mukati metiweki zvinokurumidza kupfuura zvinongedzo kune maseva, asi payuniti yebandwidth, zvakanyanya sezvatinogona kuitumira kune yedu server midziyo, kuchine imwe bandwidth mukati metiweki pachayo. Uye kuwanda kwatinoita mazinga, kunowedzera mutengo chaiwo wekupa uyu mutsetse kune kunze.
Uyezve, kunyange iyi bhendi yekuwedzera haina kunyatsofanana. Nepo mapatani ari mapfupi, tinogona kushandisa chimwe chinhu chakaita seDAC (yakananga batanidza mhangura, kureva, twinax tambo), kana multimode optics, inodhura zvakatowanda kana zvishoma mari inonzwisisika. Patinongoenda kune yakareba spans - sekutonga, aya ndiwo maone mode optics, uye mutengo weiyi yekuwedzera bandwidth inowedzera zvinoonekwa.
Uye zvakare, tichidzokera kune yakapfuura slide, kana tikagadzira network yeClos pasina kunyoreswa, saka zviri nyore kutarisa dhayagiramu, ona kuti network inovakwa sei - tichiwedzera imwe neimwe nhanho yekuchinja kwemusana, tinodzokorora tambo yese yaive pa pasi. Kuwedzera nhanho - pamwe nebhendi rimwechete, nhamba imwechete yezviteshi pane switch sezvaive padanho rekutanga, uye nhamba imwechete yevashanduri. Naizvozvo, zvinodikanwa zvakanyanya kudzikisa huwandu hwemazinga emuzongoza switch.
Kubva pamufananidzo uyu, zviri pachena kuti isu tinoda chaizvo kuvaka pane chimwe chinhu senge switch ine radix ye128.
Pano, musimboti, zvese zvakafanana nezvandabva kutaura; iyi siraidhi yekuzotariswa gare gare.
Ndedzipi sarudzo dziripo dzatingasarudza sema switch akadaro? Idzo nhau dzinonakidza kwatiri kuti ikozvino network dzakadai dzinogona kuvakwa pa-single-chip switch. Uye izvi zvinotonhorera, vane akawanda akanaka maficha. Semuenzaniso, vanenge vasina chimiro chemukati. Izvi zvinoreva kuti vanotsemuka nyore nyore. Vanopwanya nenzira dzakasiyana-siyana, asi sezvineiwo vanoputsika zvachose. Mu modular madivayiri kune nhamba huru yezvikanganiso (zvisingafadzi kwazvo), kana kubva pakuona kwevavakidzani uye ndege yekudzora inoratidzika kunge iri kushanda, asi, semuenzaniso, chikamu chejira chakarasika uye hachishandi. pakukwana. Uye iyo traffic kune iyo yakaenzana zvichienderana nekuti inoshanda zvizere, uye isu tinogona kuwandisa.
Kana, semuenzaniso, matambudziko anomuka neiyo backplane, nekuti mukati memodular mudziyo mune zvakare yakakwira-speed SerDes - yakaoma chaizvo mukati. Zvimwe zviratidzo pakati pezvinhu zvekutumira zvinowiriraniswa kana kuti hazvina kuwiriraniswa. Kazhinji, chero chigadzirwa chinogadzira modular chinosanganisira huwandu hukuru hwezvinhu, sekutonga, ine yakafanana Clos network mukati mayo, asi zvakanyanya kuoma kuongorora. Kazhinji zvakaoma kuti kunyange mutengesi pachake aongorore.
Uye ine huwandu hukuru hwekutadza zviitiko umo mudziyo unodzikisira, asi haubvi kunze kwetopology zvachose. Sezvo network yedu yakakura, kuenzanisa pakati pezvinhu zvakafanana kunoshandiswa zvakanyanya, network inogara yakajairika, ndiko kuti, imwe nzira iyo zvinhu zvese zvakarongeka haina kusiyana neimwe nzira, zvinotibatsira zvakanyanya kuti tingorasikirwa neimwe nzira. midziyo kubva kumusorosoro pane kuguma mumamiriro ezvinhu apo mamwe acho anoita seanoshanda, asi mamwe acho haaiti.
Icho chinotevera chakanaka chimiro che single-chip zvishandiso ndechekuti ivo vanoshanduka zvirinani uye nekukurumidza. Vanowanzoitawo kuve nekugona kurinani. Kana tikatora zvimiro zvakaungana zvakakura zvatinazvo padenderedzwa, ipapo huwandu hwepa rack unit yezviteshi zvekumhanya kwakafanana hunenge hwakapetwa kaviri pane hwema modular zvishandiso. Midziyo yakavakirwa kutenderedza chip imwechete inooneka yakachipa pane modular uye inoshandisa shoma simba.
Asi, chokwadi, izvi zvose nokuda kwechikonzero, panewo zvipingamupinyi. Chekutanga, iyo radix inogara idiki pane iyo modular zvishandiso. Kana tikakwanisa kuwana mudziyo wakavakirwa kutenderedza chip imwe ine 128 ports, saka tinogona kuwana modular imwe ine mazana akati wandei madoko izvozvi pasina matambudziko.
Iyi idiki inooneka saizi yekutumira matafura uye, sekutonga, zvese zvine chekuita nedata ndege scalability. Shallow buffers. Uye, sekutonga, pane zvishoma zvishoma kushanda. Asi zvinozoitika kuti kana iwe uchiziva zvirambidzo izvi uye nekuchenjerera nenguva kuti uzvitenderedze kana kungozvifunga nezvazvo, saka izvi hazvityisi. Icho chokwadi chekuti radix idiki haisisiri dambudziko pamidziyo ine radix ye128 iyo yakazoonekwa nguva pfupi yadarika; tinogona kuvaka muzvikamu zviviri zvemaspines. Asi hazvigoneke kuvaka chero chinhu chidiki pane maviri chinonakidza kwatiri. Neimwe nhanho, masumbu madiki kwazvo anowanikwa. Kunyangwe magadzirirwo edu ekare uye zvatinoda zvichiri kuzvipfuura.
Muchokwadi, kana kamwe kamwe mhinduro iri pane imwe nzvimbo pamucheto, kuchine nzira yekuyera. Kubva yekupedzisira (kana yekutanga), yakaderera nhanho uko maseva akabatana ari ToR switch kana mashizha switch, isu hatifanirwe kubatanidza rack imwe kwavari. Naizvozvo, kana mhinduro ikapfupika nehafu, unogona kufunga nezve kungoshandisa switch ine yakakura radix padanho rezasi uye yekubatanidza, semuenzaniso, maviri kana matatu racks mune imwe switch. Iyi zvakare sarudzo, ine mutengo wayo, asi inoshanda nemazvo uye inogona kuve mhinduro yakanaka kana iwe uchida kusvika kaviri saizi.
Kupfupikisa, tiri kuvaka pamusoro petopology ine mazinga maviri emusana, ane mafekitari masere.
Chii chichaitika kufizikisi? Masvomhu akareruka kwazvo. Kana tine mazinga maviri emusana, saka tine mazinga matatu chete ekuchinja, uye isu tinotarisira kuti pachava nezvikamu zvitatu zvetambo mumambure: kubva kumaseva kusvika kune mashizha emashizha, kumusana 1, kusvika kumusana 2. Zvisarudzo zvatinogona shandisa ndeiyi - idzi twinax, multimode, single mode. Uye pano tinofanira kufunga kuti ndeupi tambo iripo, kuti ichaita marii, hukuru hwemuviri ndehupi, ndezvipi zvipamhi zvatingavhara, uye kuti tichakwidziridza sei.
Maererano nemutengo, zvinhu zvose zvinogona kurongwa. Twinaxes yakachipa zvakanyanya pane inoshanda optics, yakachipa kupfuura multimode transceivers, kana iwe ukaitora nendege kubva kumagumo, yakachipa pane 100-gigabit switch port. Uye, ndapota cherechedzai, inodhura zvishoma pane imwe chete mode optics, nokuti pandege apo imwe nzira inodiwa, munzvimbo dzedata nokuda kwezvikonzero zvakawanda zvine musoro kushandisa CWDM, nepo parallel single mode (PSM) isiri nyore kushanda. ne, mapaketi akakura kwazvo anowanikwa mafibhi, uye kana tikatarisisa matekinoroji aya, tinowana anotevera mutengo wehierarchy.
Chimwezve chinyorwa: zvinosuruvarisa, hazvigoneke kushandisa disassembled zana kusvika 100x4 multimode ports. Nekuda kwemaitiro ekugadzira eSFP25 transceivers, haina kudhura zvakanyanya kupfuura 28 Gbit QSFP28. Uye iyi disassembly ye multimode haishande zvakanyanya.
Chimwe chinogumira ndechekuti nekuda kwehukuru hwemasumbu ekombuta uye huwandu hwemaseva, nzvimbo dzedu dzedata dzinoita kunge dzakakura mumuviri. Izvi zvinoreva kuti kana ndege imwe chete ichafanirwa kuitwa ne singlemod. Zvekare, nekuda kwehukuru hwemuviri hwePods, hazvizogone kumhanyisa spans maviri etwinax (tambo dzemhangura).
Somugumisiro, kana tikagadzirisa mutengo uye tichifunga nezve geometry yekugadzirwa uku, tinowana imwe span ye twinax, imwe span ye multimode uye imwe span ye singlemode tichishandisa CWDM. Izvi zvinofunga nzira dzinogoneka dzekusimudzira.
Izvi ndizvo zvazvinoita munguva pfupi yapfuura, kwatiri kuenda uye izvo zvinogoneka. Zviri pachena, zvirinani, mafambiro akananga ku50-Gigabit SerDes kune ese ari maviri multimode uye singlemode. Uyezve, kana iwe ukatarisa zviri mu-single-mode transceivers iye zvino uye mune ramangwana re400G, kazhinji kunyange kana 50G SerDes ichisvika kubva kudivi remagetsi, 100 Gbps per lane inogona kutoenda kune optics. Naizvozvo, zvinogoneka kuti pane kutamira kumakumi mashanu, pachava neshanduko ku50 Gigabit SerDes uye 100 Gbps pamukoto, nekuti maererano nezvipikirwa zvevazhinji vatengesi, kuwanikwa kwavo kunotarisirwa nekukurumidza. Iyo nguva iyo 100G SerDes yaive yakakurumidza, inoratidzika, haizove yakareba, nekuti makopi ekutanga e50G SerDes ari kubuda munenge gore rinouya. Uye pashure penguva yakati pashure paizvozvo vangangove vakakodzera mari inonzwisisika.
Imwezve nuance nezve kusarudzwa kwefizikisi. Muchidimbu, tinogona kutoshandisa mazana mana kana mazana maviri eGigabit ports tichishandisa 400G SerDes. Asi zvinozoitika kuti izvi hazvina musoro, nekuti, sezvandambotaura, tinoda radix yakakura kwazvo pane switch, mukati mechikonzero, hongu. Tinoda 200. Uye kana tine zvishoma chip simba uye isu tinowedzera yekubatanidza kasi, ipapo radix inodzikira inoderera, hapana zvishamiso.
Uye isu tinokwanisa kuwedzera huwandu hwese tichishandisa ndege, uye hapana yakakosha mari; tinogona kuwedzera huwandu hwendege. Uye kana tikarasikirwa neradix, tichazofanira kuunza imwe nhanho, saka mumamiriro ezvinhu aripo, nehuwandu huripo huripo huripo pane chip, zvinoratidzika kuti inonyanya kushanda kushandisa 100-gigabit ports, nokuti inokubvumira. kuwana radix yakakura.
Mubvunzo unotevera ndewekuti fizikisi yakarongeka sei, asi kubva pakuona kwetambo tambo. Zvinoitika kuti yakarongwa nenzira inosetsa. Cabling pakati peshizha-swichi uye yekutanga-level spines - hapana akawanda malink ipapo, zvese zvakavakwa zvakapusa. Asi kana tikatora ndege imwe chete, chii chinoitika mukati ndechokuti tinoda kubatanidza mitsipa yose yezinga rekutanga nemhando dzose dzechipiri chechipiri.
Uyezve, sekutonga, kune zvimwe zvishuwo zvekuti inofanirwa kutarisa sei mukati me data data. Semuenzaniso, isu taida chaizvo kusanganisa tambo kuita sumbu uye todhonza kuti imwe yepamusoro-density chigamba patch ipinde zvachose mune imwe chigamba patch, kuti pakanga pasina zoo maererano nehurefu. Takakwanisa kugadzirisa dambudziko iri. Kana iwe uchitanga kutarisa inonzwisisika topology, unogona kuona kuti ndege dzakazvimiririra, imwe neimwe ndege inogona kuvakwa pachayo. Asi kana isu tichiwedzera bundling yakadai uye tichida kudhonzera iyo yese chigamba pane patch patch, isu tinofanirwa kusanganisa ndege dzakasiyana mukati mebundle rimwechete uye tosuma chimiro chepakati muchimiro che optical cross-connection kuti tidzorerwe kubva pakuunganidzwa kwadzakaitwa. pane chimwe chikamu, munzira iyo ichaunganidzwa pane imwe chikamu. Nekuda kweizvi, tinowana yakanaka chimiro: ese akaomarara switching haaendi kupfuura racks. Kana iwe uchida kupindira chimwe chinhu zvakasimba, "vhura ndege," sezvainodanwa dzimwe nguva muClos network, iyo yose yakanangidzirwa mukati meraki imwe. Isu hatina zvakanyanya kuparadzaniswa, kusvika kune ega ega zvinongedzo, kushandura pakati pema racks.
Iyi ndiyo nzira iyo inotaridzika kubva pakuona kwesangano rinonzwisisika remagetsi etambo. Mumufananidzo uri kuruboshwe, zvidhinha zvine mavara-akawanda zvinoratidza mabhuroko ekutanga-level spine switch, zvimedu zvisere chimwe nechimwe, uye mabundle mana etambo anouya kubva pazviri, anoenda achinopindirana nemabundle anobva kumabhuroko emusana-2 switch. .
Zvikwere zvidiki zvinoratidza mharadzano. Kumusoro kwekuruboshwe kune kuparara kwemharadzano imwe neimwe yakadai, iyi inotova 512 by 512 port cross-connect module inorongedza tambo kuitira kuti dziuye zvizere mune imwe rack, mune imwe chete spine-2 ndege. Uye kurudyi, scan yemufananidzo uyu yakatsanangurwa zvishoma maererano nePods akati wandei padanho remusana-1, uye kuti yakarongedzerwa sei muchinjiko-yekubatanidza, kuti inosvika sei kumusana-2 level.
Izvi ndizvo zvazvinoita. Iyo isati yanyatso kuunganidzwa musana-2 kumira (kuruboshwe) uye muchinjiko-batanidza kumira. Sezvineiwo, hapana zvakawanda zvekuona ipapo. Ichi chimiro chese chiri kuiswa izvozvi mune imwe yenzvimbo dzedu huru dzedata dziri kuwedzerwa. Iri ibasa riri kuenderera mberi, rinotaridzika zvakanaka, richazadzwa zviri nani.
Mubvunzo wakakosha: isu takasarudza iyo inonzwisisika topology uye takavaka iyo fizikisi. Chii chichaitika kundege inodzora? Zvinonyatsozivikanwa kubva kune ruzivo rwekushanda, kune akati wandei mishumo inobatanidza nyika maprotocol akanaka, zvinonakidza kushanda navo, asi, zvinosuruvarisa, iwo haana kukwira zvakanaka pane yakabatana yakabatana topology. Uye pane chimwe chinhu chikuru chinodzivirira izvi - aya ndiwo mashandiro anoita mafashama mulink state protocol. Kana iwe ukangotora iyo yemafashama algorithm uye wotarisa magadzirirwo etiweki yedu, unogona kuona kuti pachave nefanout yakakura kwazvo pane imwe neimwe nhanho, uye inongofashama ndege yekudzora nekuvandudzwa. Kunyanya, matopology akadai anosanganiswa zvisina kunaka neachinyakare mafashama algorithm mune link state protocol.
Sarudzo ndeye kushandisa BGP. Nzira yekugadzirira nayo zvakarurama inotsanangurwa muRFC 7938 pamusoro pekushandiswa kweBGP munzvimbo huru dze data. Mazano ekutanga akareruka: nhamba shoma yezvivakashure pamunhu mumwe nemumwe uye kazhinji nhamba shoma yezvivakamberi panetiweki, shandisa kuunganidza kana zvichibvira, uye dzvanyirira nzira yekuvhima. Tinoda kungwarira, kudzorwa zvakanyanya kugoverwa kwezvigadziriso, izvo zvinonzi valley yemahara. Isu tinoda kuti zvigadziriso zvishandiswe kamwe chete pavanenge vachipfuura nepanetiweki. Kana dzichibva pazasi, dzinokwira, dzisingapfuunudziki kanopfuura kamwe. Panofanira kunge pasina zigzags. Mazigzags akaipa kwazvo.
Kuti tiite izvi, tinoshandisa dhizaini iri nyore zvakakwana kushandisa ari pasi peBGP masisitimu. Ndiko kuti, isu tinoshandisa eBGP inomhanya pane chinongedzo chenzvimbo, uye masisitimu akazvimirira anopihwa sezvinotevera: inozvimiririra sisitimu paToR, inozvimiririra sisitimu pane yese block yemusana-1 switch yeimwe Pod, uye yakajairika kuzvitonga system paTop yese. of Fabric. Hazvina kuoma kutarisa uye kuona kuti kunyangwe maitiro akajairika eBGP anotipa kugoverwa kwezvigadziriso zvatinoda.
Nomuzvarirwo, kudhirowa uye kuunganidzwa kwekero kunofanirwa kugadzirwa kuitira kuti ienderane nenzira inovakwa nzira, kuitira kuti ive nechokwadi chekugadzikana kwendege yekudzora. L3 kutaura mukutakura kunosungirirwa kune iyo topology, nekuti pasina izvi hazvigoneke kuwana aggregation; pasina izvi, kero dzemunhu dzinopinda muhurongwa hwekufambisa. Uye chimwezve chinhu ndechekuti kuunganidza, zvinosuruvarisa, hakunyatso kusanganiswa neakawanda-nzira, nekuti kana isu tine nzira dzakawanda uye isu tine kuunganidzwa, zvese zvakanaka, kana network yese ine hutano, hapana kukundikana mairi. Nehurombo, kana kutadza kuchionekwa mumambure uye symmetry yetopology yakarasika, tinogona kusvika pane iyo iyo unit yakaziviswa, kubva kwatisingakwanise kuenda mberi kwatinoda kuenda. Naizvozvo, zvakanakisa kuunganidza uko pasina imwe nzira-yakawanda, mune yedu iyi iToR switch.
Muchokwadi, zvinokwanisika kuunganidza, asi nekuchenjerera. Kana isu tichigona kuita inodzorwa disaggregation kana network kutadza kuitika. Asi iri ibasa rakaoma, takatombozvibvunza kuti zvingaite here kuita izvi, kana zvaikwanisika kuwedzera otomatiki, uye michina inogumira yehurumende yaigona kurova BGP kuti iwane maitiro anodiwa. Nehurombo, kugadzirisa makesi emakona hakuna kujeka uye kwakaoma, uye basa iri harina kunyatsogadziriswa nekubatanidza zvekunze zvakabatanidzwa kuBGP.
Basa rinonakidza zvikuru panyaya iyi rakaitwa mukati megadziriro yeRIFT protocol, iyo ichakurukurwa mumushumo unotevera.
Chimwe chinhu chakakosha ndechekuti ndege dzedata dzinokura sei mune dense topology, uko isu tine nhamba huru yedzimwe nzira. Muchiitiko ichi, akati wandei ekuwedzera data zvimiro anoshandiswa: ECMP mapoka, ayo anozotsanangura Inotevera Hop mapoka.
Mune network inowanzoshanda, pasina kukundikana, patinokwira kuClos topology, zvakakwana kushandisa boka rimwe chete, nokuti zvinhu zvose zvisiri zvemunharaunda zvinotsanangurwa nekusavapo, tinogona kukwira. Kana tichibva kumusoro kusvika kuzasi kuenda kumaodzanyemba, ipapo nzira dzese hadzisi ECMP, inzira imwe chete. Zvese zvakanaka. Dambudziko nderekuti, uye kusarudzika kweiyo classic Clos topology ndeyekuti kana tikatarisa Kumusoro kwejira, pane chero chinhu, pane nzira imwe chete kune chero chinhu pazasi. Kana kutadza kukaitika munzira iyi, saka chinhu ichi chiri pamusoro pefekitori chinova chisisashande kune aya prefixes ari kuseri kwegwara rakapwanyika. Asi kune mamwe ose anoshanda, uye isu tinofanirwa kupatsanura mapoka eECMP uye kuunza nyika itsva.
Ko data data scalability inotaridzika sei pamidziyo yemazuva ano? Kana tikaita LPM (yakareba prefix match), zvese zvakanaka, pamusoro pe100k prefixes. Kana isu tichitaura nezveNext Hop mapoka, ipapo zvese zvakaipisisa, 2-4 zviuru. Kana isu tichitaura nezve tafura ine tsananguro yeInotevera Hops (kana adjacencies), saka iyi pane imwe nzvimbo kubva 16k kusvika 64k. Uye izvi zvinogona kuva dambudziko. Uye pano tasvika kune inonakidza digression: chii chakaitika kuMPLS munzvimbo dzedata? Chaizvoizvo, taida kuzviita.
Zvinhu zviviri zvakaitika. Isu takaita micro-segmentation pane mauto; isu hatichadi kuzviita pane network. Yakanga isiri yakanaka kwazvo nerutsigiro kubva kune vakasiyana vatengesi, uye zvakatonyanya nekuita kwakavhurika pamabhokisi machena ane MPLS. Uye MPLS, zvirinani kuita kwayo kwechinyakare, zvinosuwisa, inosanganisa zvakashata neECMP. Uye ndosaka.
Izvi ndizvo zvinotaridzika seECMP yekutumira chimiro cheIP. Nhamba huru ye prefixes inogona kushandisa boka rimwechete uye yakafanana Next Hops block (kana adjacencies, izvi zvinogona kudaidzwa zvakasiyana mumagwaro akasiyana emidziyo yakasiyana). Icho chiripo ndechekuti izvi zvinotsanangurwa sechiteshi chinobuda uye chekunyorazve kero yeMAC kuitira kuti usvike kune chaiyo Inotevera Hop. Kune IP zvese zvinotaridzika zviri nyore, unogona kushandisa nhamba huru kwazvo ye prefixes yeboka rimwechete, yakafanana Inotevera Hops block.
Iyo yekare MPLS dhizaini inoreva kuti, zvichienderana neinobuda interface, iyo label inogona kunyorwazve kune akasiyana hunhu. Naizvozvo, isu tinofanirwa kuchengeta boka uye Inotevera Hops block kune yega yega yekuisa label. Uye izvi, nhamo, hazvina kukwira.
Zviri nyore kuona kuti mukugadzira kwedu taida nezve 4000 ToR switch, iyo yakakura hupamhi yaive 64 ECMP nzira, kana isu tichibva kure nemusana-1 kuenda kumusana-2. Isu tinongopinda mutafura imwe yemapoka eECMP, kana chete chivakashure chimwe chete neToR chichienda, uye isu hatipinde muInotevera Hops tafura zvachose.
Hazvisi zvese zvisina tariro, nekuti zvivakwa seSegment Routing zvinosanganisira zvinyorwa zvepasirese. Pakare, zvaizogoneka kudonhedza ese aya mabhuroki eNext Hops zvakare. Kuti uite izvi, unoda kuvhiyiwa kwemhando yekadhi yemusango: tora chinyorwa uye unyore zvakare kune imwecheteyo pasina kukosha chaiko. Asi zvinosuruvarisa, izvi hazvipo zvakanyanya mukuitwa kuripo.
Uye pakupedzisira, isu tinofanirwa kuunza kunze traffic kune data center. Kuzviita sei? Pakutanga, traffic yakaunzwa muClos network kubva kumusoro. Ndokureva kuti, pakanga paine ma routers akabatana kune ese maturusi ari Pamusoro pejira. Iyi mhinduro inoshanda chaizvo padiki kusvika pakati nepakati. Nehurombo, kuti titumire traffic symmetrically kune network yese nenzira iyi, isu tinofanirwa kusvika panguva imwe chete pazvinhu zvese zvePamusoro pejira, uye kana paine anopfuura zana azvo, zvinoitika kuti isu tinodawo hombe. radix pamucheto routers. Kazhinji, izvi zvinodhura mari, nokuti ma-router edge anonyanya kushanda, zviteshi pavari zvichange zvichidhura, uye dhizaini haina kunaka kwazvo.
Imwe sarudzo ndeyekutanga traffic yakadai kubva pazasi. Zviri nyore kuona kuti iyo Clos topology yakavakwa nenzira yekuti traffic inouya kubva pazasi, kureva kuti, kubva kudivi reToR, inogovaniswa zvakaenzana pakati pemazinga mukati mePamusoro yese yejira mune maviri iterations, kurodha network yese. Naizvozvo, isu tinosuma yakasarudzika mhando yePod, Edge Pod, iyo inopa yekunze yekubatanidza.
Pane imwezve sarudzo. Izvi ndizvo zvinoita Facebook, semuenzaniso. Vanoridaidza kuti Fabric Aggregator kana HGRID. Imwe nhanho yekuwedzera yemusana iri kuunzwa yekubatanidza akawanda data centers. Iyi dhizaini inogoneka kana isu tisina mamwe mabasa kana encapsulation shanduko pamainterfaces. Kana ari mamwe mapoinzi ekubata, zvakaoma. Kazhinji, kune mamwe mabasa uye rudzi rwe membrane inoparadzanisa zvikamu zvakasiyana zve data data. Hapana chikonzero mukugadzira membrane yakadaro yakakura, asi kana ichinyatsodikanwa nokuda kwechimwe chikonzero, zvino zvine musoro kufunga nezve mukana wekuibvisa, ichiita kuti ive yakafara sezvinobvira uye kuiendesa kune mauto. Izvi zvinoitwa, semuenzaniso, nevazhinji vanoita makore. Vane akafukidzwa, vanotanga kubva kune mauto.
Ndeipi mikana yebudiriro yatinoona? Chekutanga, kuvandudza rutsigiro rweCI/CD pombi. Tinoda kubhururuka nenzira yatinoedza nayo uye kuedza nzira yatinobhururuka nayo. Izvi hazvishande zvakanyanya, nekuti zvivakwa zvakakura uye hazvigoneke kuzvidzokorora kuitira bvunzo. Iwe unofanirwa kunzwisisa nzira yekuunza zvinhu zvekuyedza muzvigadzirwa zvekugadzira pasina kuzvidonhedza.
Zvirinani zvirinani uye kutarisa kurinani hakuna kumbonyanya kushata. Mubvunzo wose ndeyekuenzanisa kwekuedza nekudzoka. Kana iwe uchikwanisa kuwedzera nekuedza zvine musoro, zvakanaka kwazvo.
Vhura masisitimu ekushandisa kunetiweki zvishandiso. Zvirinani maprotocol uye ari nani ekufambisa masisitimu, akadai seRIFT. Tsvagiridzo inodiwa zvakare mukushandiswa kwezviri nani congestion control zvirongwa uye pamwe sumo, ingangoita pane dzimwe nzvimbo, yeRDMA rutsigiro mukati mesumbu.
Tichitarisa mberi mune ramangwana, tinoda matopology epamusoro uye pamwe manetwork anoshandisa zvishoma pamusoro. Pazvinhu zvitsva, pachangobva kuburitswa nezve tekinoroji yejira yeHPC Cray Slingshot, iyo yakavakirwa pane commodity Ethernet, asi nesarudzo yekushandisa mapfupi misoro. Somugumisiro, kumusoro kunoderedzwa.
Zvose zvinofanira kuchengetwa zviri nyore sezvinobvira, asi kwete nyore. Complexity muvengi we scalability. Zviri nyore uye zvimiro zvenguva dzose ishamwari dzedu. Kana iwe uchigona kuyera kunze kumwe kunhu, ita. Uye kazhinji, zvakanaka kuve nechekuita mune network tekinoroji izvozvi. Pane zvinhu zvakawanda zvinonakidza zviri kuitika. Ndatenda.
Source: www.habr.com