Moni, choyamba mawu ena. Nthawi zina ndimasilira anzanga omwe amagwira ntchito zakutali - ndizabwino kukhala ndi mwayi wogwira ntchito kuchokera kumalekezero aliwonse a dziko lolumikizidwa ndi intaneti, tchuthi nthawi iliyonse, udindo wama projekiti ndi masiku omalizira, komanso osakhala muofesi kuyambira 8 mpaka 17. udindo ndi maudindo a ntchito zimasiyaniratu kuthekera kwa nthawi yayitali kuchokera ku data center. Komabe, zochitika zosangalatsa monga zomwe zafotokozedwa pansipa zimachitika nthawi ndi nthawi, ndipo ndimamvetsetsa kuti pali malo ochepa pomwe pali kuthekera kopanga mawonekedwe avuto lamkati.
Chodzikanira chaching'ono - panthawi yolemba, mlanduwu sunathetsedwe kwathunthu, koma chifukwa cha liwiro la kuyankha kuchokera kwa ogulitsa, yankho lathunthu lingatenge miyezi, koma ndikufuna kugawana zomwe ndapeza tsopano. Ndikuyembekeza, owerenga okondedwa, mundikhululukire chifukwa chachangu. Koma madzi okwanira - vuto ndi chiyani?
Choyamba, mawu oyambira: pali kampani (komwe ndimagwira ntchito ngati injiniya wamaneti) yomwe imakhala ndi mayankho amakasitomala mumtambo wachinsinsi wa VMWare. Mayankho ambiri atsopano amalumikizana ndi zigawo za VXLAN zomwe zimayendetsedwa ndi NSX-V - sindingayerekeze kuti yankho ili linandipatsa nthawi yochuluka bwanji, mwachidule - zambiri. Ndidakwanitsa kuphunzitsa anzanga pokhazikitsa NSX ESG ndipo mayankho ang'onoang'ono amakasitomala amaperekedwa popanda kutenga nawo mbali. Chidziwitso chofunikira: tili ndi ndege yowongolera yokhala ndi kubwereza kwa unicast. Ma hypervisors amalumikizidwa mochulukirachulukira ndi njira ziwiri zosinthira zamtundu wa Juniper QFX5100 (zosonkhanitsidwa mu Virtual Chassis) ndi njira yozikidwa poyambira ndondomeko yanthawi yanthawi yamadoko - iyi ndi yokwanira.
Mayankho a makasitomala ndi osiyanasiyana kwambiri: kuyambira Windows IIS, komwe zigawo zonse za seva ya pa intaneti zimayikidwa pa makina amodzi, zimafika mpaka zazikulu kwambiri—monga, ma Apache web frontends okhala ndi load-balanced + LB MariaDB mu Galera + ma seva ogawana omwe amagwirizanitsidwa pogwiritsa ntchito GlusterFS. Pafupifupi seva iliyonse imafunika kuyang'aniridwa padera, ndipo si zigawo zonse zomwe zili ndi ma adilesi a anthu onse. Ngati mwakumanapo ndi vutoli ndipo muli ndi yankho labwino kwambiri, ndingayamikire upangiri wanu.
Yankho langa lowunikira limaphatikizapo "kulumikiza" firewall (Fortigate) ku netiweki iliyonse yamakasitomala (+SNAT ndipo, zowonadi, zoletsa zamtundu wamtundu wololedwa) ndikuwunika ma adilesi amkati - motere kugwirizanitsa kwina ndi kuphweka kwa kuwunika ndi zatheka. Kuwunika komweko kumachitika kuchokera kugulu la seva la PRTG. Dongosolo loyang'anira ndi motere:
Ngakhale tinkangogwira ntchito ndi VLAN, zonse zinali zabwinobwino ndipo zimagwira ntchito modziwikiratu, ngati mawotchi. Pambuyo pa kukhazikitsidwa kwa NSX-V ndi VXLAN, tinayang'anizana ndi funso: kodi n'zotheka kupitiriza kuyang'anira njira yakale? Pa nthawi ya funsoli, yankho "lofulumira" linali kuyika NSX ESG ndikulumikiza mawonekedwe a thunthu la VXLAN ku netiweki ya VTEP. Mwachangu m'mawu - popeza kugwiritsa ntchito GUI kukonza maukonde amakasitomala, malamulo a SNAT ndi firewall amatha kugwirizanitsa kasamalidwe mu mawonekedwe amodzi a vSphere, koma m'malingaliro anga ndizovuta kwambiri ndipo, mwa zina, zimalepheretsa zida zothanirana ndi mavuto. Iwo omwe agwiritsa ntchito NSX ESG m'malo mwa "weniweni" firewall, ndikuganiza, avomereza. Ngakhale, mwinamwake, yankho loterolo lingakhale lokhazikika - pambuyo pake, chirichonse chimachitika mkati mwa chimango cha wogulitsa mmodzi.
Njira ina ndiyo kugwiritsa ntchito NSX DLR mumalowedwe a mlatho pakati pa VLAN ndi VXLAN. Apa ndikuganiza kuti zonse zimveka bwino - phindu logwiritsa ntchito VXLAN langotayika - chifukwa pamenepa mukuyenera kugwirizanitsa VLAN ndi kuikapo polojekiti. Mwa njira, pokonza njira yothetsera vutoli, ndinakumana ndi vuto pamene mlatho wa DLR sunatumize mapaketi ku makina enieni omwe anali nawo pamsasa womwewo. Ndikudziwa, ndikudziwa - m'mabuku ndi maupangiri pa NSX-V amanenedwa mwachindunji kuti gulu lapadera liyenera kuperekedwa kwa NSX Edge, koma izi zili m'mabuku ... Njira imodzi kapena imzake, patatha miyezi ingapo ndi thandizo, sitinathetse vutoli. M'malo mwake, ndinamvetsetsa zomwe zikuchitika - gawo la hypervisor kernel lomwe limayang'anira VXLAN encapsulation silinagwiritsidwe ntchito ngati DLR ndi seva yoyang'aniridwa inali pa gulu lomwelo, popeza magalimoto sasiya wolandirayo ndipo, momveka, ayenera kulumikizidwa. ku gawo la VXLAN - encapsulation sikufunika. Ndi chithandizo, tidakhazikika pa mawonekedwe a vdrPort, omwe amalumikizana momveka bwino ndi ma uplinks ndikuchitanso bridging / encapsulation - apa ndipamene kusiyana kwa magalimoto omwe akubwera kunazindikirika, zomwe ndinatenga kuti ndigwire ntchito panopa. Koma monga ananenera, sindinamalize mlandu umenewu, popeza ndinasamutsidwa kukagwira ntchito ina ndipo nthambiyo poyamba inali yachiwembu ndipo panalibe chikhumbo chofuna kuikulitsa. Ngati sindikulakwitsa, vutoli lidawonedwa mumitundu NSX ndi 6.1.4 ndi 6.2.
Kenako - bingo! Fortinet alengeza zakwawo . Osati kungoyang'ana-pa-point kapena VXLAN-over-IPSec, osati pulogalamu ya VLAN-VXLAN bridging - zonsezi zinayamba kukhazikitsidwa mu mtundu 5.4 (ndi zina ), koma thandizo la ndege la unicast. Pokhazikitsa yankho, ndidakumana ndi vuto lina - ma seva omwe amawunikiridwa nthawi ndi nthawi "amasowa" kenako adawonekera pakuwunika, ngakhale makina enieniwo anali amoyo. Chifukwa, monga momwe zinakhalira, chinali chakuti ndinayiwala kuloleza Ping pa mawonekedwe a VXLAN. Panthawi yokonzanso masango, makina owoneka bwino adasunthidwa, ndipo vMotion idamalizidwa ndi Ping kuwonetsa wolandila watsopano wa ESXI komwe makinawo adasamukira. Kupusa kwanga, koma vutoli linasokonezanso kudalira thandizo loperekedwa ndi wopanga - mu nkhani iyi, Fortinet. Sindikunena kuti nkhani iliyonse yokhudzana ndi VXLAN imayamba ndi funso "komwe kuli VLAN-VXLAN softswitch muzokonda zanu?" Nthawi ino ndidalangizidwa kuti ndisinthe MTU - iyi ndi ya Ping, yomwe ndi 32 bytes. Kenako "sewera mozungulira" ndi tcp-send-mss ndi tcp-receive-mss mu ndondomeko - ya VXLAN, yomwe ili mu UDP. Phew, pepani - ikuwira. Nthawi zambiri, ndinathetsa vutoli ndekha.
Pambuyo poyendetsa bwino magalimoto oyesa, adaganiza zogwiritsa ntchito yankho ili. Ndipo popanga zidapezeka kuti patatha tsiku limodzi kapena awiri, chilichonse chomwe chimayang'aniridwa kudzera pa VXLAN pang'onopang'ono chinagwa. Kuyimitsa / kuyambitsa mawonekedwewo kunathandiza, koma kwakanthawi. Pokumbukira kuchedwa kwa chithandizo chomwe chikupangidwa, ndinayamba kuthetsa mavuto kumbali yanga - pambuyo pake, kampani yanga, maukonde anga ndi udindo wanga.
Kupita patsogolo kwa zovuta kuli pansi pa wowononga. Kwa iwo omwe atopa ndi makalata ndi kudzitamandira, dumphani ndikupita ku post-analysis.
Kupititsa patsogolo zovutaZikomo popitiliza kuwerenga - tiyeni tipitilize!
Chifukwa chake, kuyang'anira kumagwira ntchito kwakanthawi, kenako kumagwera kokha. Izi zikutanthauza kuti mwina palibe mavuto ndi ndondomeko za firewall. Komabe, popeza ndinakumana ndi vuto la kupachika machitidwe a machitidwe mu Fortigate versions 5.6+, kotero choyamba timayang'ana "kufufuza kutuluka kwa debug" - monga kuyembekezera, magalimoto amaloledwa ndikusiya mawonekedwe ndipo, monga momwe akuyembekezeredwa, palibe chomwe chimabwera. Kotero ife timakumba mowonjezera pansi pa stack. Tsoka ilo, tidzayenera kubisa ma adilesi, ngakhale atakhala RFC1918, koma ndikuyembekeza kupereka ndondomekoyi ndi kufotokozera kokwanira kuti timvetsetse. Seva mkati mwa VXLAN ili ndi adilesi x.x.x.15, mawonekedwe a Fortigate x.x.x.254, ma adilesi ena onse ndi a netiweki ya VTEP.
Kutumiza bwino kwa mapaketi a VXLAN-encapsulated kumafuna chidziwitso cholondola pamagome angapo. Pokutirana awa ndi ARP ndi OVSDB, pakuyika pansi awa ndi ARP ndi CAM. Pankhani ya Fortigate VXLAN FDB ndi OVSDB. Tiyambire pamenepo:
fortigate (root) #diag sys vxlan fdb list vxlan-LS
mac=00:50:56:8f:3f:5a state=0x0002 flags=0x00 remote_ip=у.у.у.47 port=4789 vni=5008 ifindex=7
Chilichonse apa ndi chophweka - adilesi ya MAC yamakina enieni ayenera kukhala pa VTEP ndi adilesi u.u.u.47. Nditayang'ana zomwe zili ndi zoikamo za gulu la ESXI, ndikupeza kuti MAC ya makina enieni ndi yolondola, komanso adilesi ya VTEP. Ndimayang'ana tebulo la CAM/ARP pa fortik - zonse zimagwirizana ndi makonda a ESXI:
fortigate (root) #get sys arp | grep у.у.у.47
у.у.у.47 0 00:50:56:65:f6:2c dmz
Matebulo ndi olondola ndipo magalimoto akuchoka - mwina vuto silili ndi Fortigate? Ndinalumpha dala kusanthula kwa kusintha kwa magalimoto pa Juniper - zomveka, sitepe yotsatira yothetsera mavuto iyenera kuchitidwa pamenepo, koma maukonde anga ndi osavuta - VLAN imodzi yokha ya VTEP ndipo zigawo zonse zimagwirizanitsidwa mwachindunji. Kuphatikiza apo, ndimakumbukira mlandu wokhala ndi mlatho wa DLR, VDR komanso magalimoto otayika - ndikukankhira gulu la ESXI, ndipo nthawi yomweyo ndikupanga mlandu wa VMWare. Pansi pa MAC "97:6e" ndi ya fortik, vmnic1 ndi mawonekedwe omwe ali ndi VTEP yokhala ndi adilesi ya u.u.u.u.47 snifim mbali zonse "--dir 2":
pktcap-uw --uplink vmnic1 --vni 5008 --mac 90:6c:ac:a9:97:6e --dir 2 -o /tmp/monitor.pcap
Kupita patsogolo - mu kununkhiza ndikuwona pempho la ARP ndi yankho lomwe likubwera. Ndimangopereka yankho la ARP ndipo zonse zili zolondola pamenepo. Sindinazitchule, koma nthawi yonseyi seva yowunikira ikuyang'ana adilesi xxx15 - mayendedwe a ICMP ali kuti? Ndikukumbukira kuti ndili ndi ma uplinks awiri. Pano mukhoza kutsutsa ndi kunena kuti gwero lochokera ku doko ndilofanana (ndondomeko yanga yamagulu), ndiko kuti, uplink womwewo uyenera kusankhidwa ku vNIC yomweyo, koma popeza ndili pa host, kuyang'ana uplink yosiyana si a vuto:
pktcap-uw --uplink vmnic4 --vni 5008 --mac 90:6c:ac:a9:97:6e --dir 2 -o /tmp/monitor.pcap
Zopempha zikuchokera ku Fortigate, koma palibe yankho. Ndiko kuti vuto siliri mu Fortigate. Chabwino, ndizomwezo - ndikuganiza - vuto lomwelonso ndi kusowa kwa magalimoto pa VDR, zidzatenganso miyezi ingapo kuti mlanduwo ukhale wolondola. Patatha masiku angapo, nditazizira, osafuna kupirira, ndinaganiza zokumbanso zofukiza kuti zithandizire kuti ntchitoyi ifulumire. Ndiyeno "mwangozi" maso anga amagwera pa Ethernet underlay encapsulation. Tsar si yeniyeni ndipo adilesi ya MAC ya VTEP sagwirizana ndi IP yake. Ndibwerera ku ziro, kununkhiza, kukumba - ndizowona, sizowona. Ndikupatsani tebulo la ARP pafupi ndi inu kuti likhale losavuta kufanizitsa. Zindikirani kuyika koyamba kwa Ethernet pachithunzi pamwambapa:
fortigate (root) #get sys arp | grep у.у.у.47
у.у.у.47 0 00:50:56:65:f6:2c dmz
fortigate (root) #get sys arp | grep у.у.у.42
у.у.у.42 0 00:50:56:6a:78:86 dmz
Kotero, zomwe timathera ndi kuti titatha kusamuka makina enieni, Fortigate amayesa kutumiza magalimoto ku VTEP kuchokera ku (zolondola) VXLAN FDB, koma amagwiritsa ntchito DST MAC yolakwika ndipo magalimoto akuyembekezeredwa kugwetsedwa ndi mawonekedwe a hypervisor akulandira. Komanso, nthawi imodzi mwa anayi, MAC iyi inali ya hypervisor yoyambirira yomwe makina amasamuka.
Dzulo ndinalandira kalata kuchokera ku Fortinet technical support - bug 615586 inapezeka kwa ine. idzangobwera ndi zosintha za firmware, kapena bwino lotsatira. ChSV imalimbikitsidwanso ndi cholakwika china chomwe ndidapeza mwezi watha, ngakhale nthawi imeneyo mu HTML5 GUI vSphere. Chabwino, dipatimenti yakomweko ya QA ya ogulitsa ...
Ndikosavuta kufotokoza izi:
1 - ndege zowongolera ma multicast mwina sizingakhale ndi vuto lomwe lafotokozedwa - pambuyo pake, ma adilesi a VTEP MAC amachokera ku adilesi ya IP ya gulu lomwe mawonekedwewo adalembetsedwa.
2 - mwina vuto la fortik pakutsitsa magawo pa Network processor (yofanana ndi CEF) - ngati mudutsa paketi iliyonse kudzera mu CPU, matebulo adzagwiritsidwa ntchito okhala ndi zolondola - zosawoneka bwino. Lingaliro ili limathandizidwa ndi chakuti zimathandiza kutseka / kutsegula mawonekedwe kapena kuyembekezera nthawi - kuposa mphindi 5.
3 - kusintha ndondomeko yamagulu, mwachitsanzo kulephera momveka bwino, kapena kuyambitsa LAG sikungathetse vutoli, popeza MAC ya hypervisor gwero inawonedwa kuti "inakanidwa" m'mapaketi otsekedwa.
Kutengera izi, nditha kugawana zomwe ndapeza posachedwa , pomwe imodzi mwa nkhanizi inanena kuti ma firewall a stetfull ndi njira zosungitsira deta zosungidwa ndi ndodo. Chabwino, sindine wodziwa mokwanira mu IT kuti ndinene izi, ndipo pambali pake, sindimagwirizana nthawi yomweyo ndi mawu onse muzolemba zamabulogu. Komabe, chinachake chimandiuza kuti pali chowonadi m'mawu a Ivan.
Zikomo chifukwa chomvetsera! Ndidzakhala wokondwa kuyankha mafunso ndikumva kutsutsidwa kolimbikitsa.
Source: www.habr.com
