Gabatarwa zuwa ɓangaren cibiyar sadarwa na kayan aikin girgije

Gabatarwa zuwa ɓangaren cibiyar sadarwa na kayan aikin girgije

Ƙididdigar girgije tana shiga zurfi da zurfi cikin rayuwarmu kuma tabbas babu mutum ɗaya da bai yi amfani da kowane sabis na girgije aƙalla sau ɗaya ba. Duk da haka, menene ainihin girgije da kuma yadda yake aiki, mutane kaɗan sun sani, har ma a matakin ra'ayi. 5G ya riga ya zama gaskiya kuma kayan aikin sadarwa sun fara motsawa daga ginshiƙan mafita zuwa mafita na girgije, kamar yadda ya yi lokacin da ya tashi daga mafita na kayan aiki gaba ɗaya zuwa "ginshiƙai".

A yau za mu yi magana game da duniyar ciki na kayan aikin girgije, musamman za mu dubi ainihin ɓangaren cibiyar sadarwa.

Menene gajimare? Haɓaka madaidaici iri ɗaya - kallon bayanin martaba?

Fiye da tambaya mai ma'ana. A'a - wannan ba aikin haɓaka ba ne, kodayake ba za a iya yin shi ba tare da shi ba. Bari mu kalli ma’anoni guda biyu:

Cloud Computing (wanda ake kira Cloud) samfuri ne don samar da damar mai amfani mai amfani ga albarkatun kwamfuta da aka rarraba waɗanda dole ne a tura su kuma ƙaddamar da buƙatu tare da mafi ƙarancin jinkiri da ƙarancin farashi ga mai bada sabis.

Ƙwarewa - wannan shine ikon raba mahaɗan jiki guda ɗaya (misali, uwar garken) zuwa nau'ikan kama-da-wane da yawa, don haka ƙara yawan amfani da albarkatu (misali, kuna da sabar 3 da aka loda akan kashi 25-30 cikin ɗari, bayan haɓakawa za ku sami sabar 1 da aka loda. da kashi 80-90). A dabi'a, haɓakawa yana cinye wasu albarkatun - kuna buƙatar ciyar da hypervisor, duk da haka, kamar yadda aikin ya nuna, wasan ya cancanci kyandir. Kyakkyawan misali na haɓakawa shine VMWare, wanda ke shirya injunan kama-da-wane daidai, ko misali KVM, wanda na fi so, amma wannan lamari ne na ɗanɗano.

Muna amfani da haɓakawa ba tare da saninsa ba, har ma da masu amfani da ƙarfe sun riga sun yi amfani da haɓakawa - alal misali, a cikin sabuwar sigar JunOS, ana shigar da tsarin aiki azaman injin kama-da-wane a saman rarraba Linux na ainihi (Wind River 9). Amma gajimare ba shine gajimare ba, amma gajimaren ba zai iya wanzuwa ba tare da sanin yakamata ba.

Ƙwarewa yana ɗaya daga cikin tubalan ginin da aka gina girgije a kansu.

Yin gajimare ta hanyar tattara hypervisors da yawa a cikin yanki ɗaya na L2, ƙara wasu littattafan wasan yaml guda biyu don yin rijistar vlan ta atomatik ta wani nau'i mai yuwuwa, da sanya wani abu kamar tsarin ƙungiyar makaɗa a samansa don ƙirƙirar injuna ta atomatik ba zai yi aiki ba. Zai zama mafi daidai, amma sakamakon Frankenstein ba shine girgijen da muke buƙata ba, kodayake yana iya zama babban mafarki ga wasu. Bugu da ƙari, idan kun ɗauki Opentack iri ɗaya, da gaske har yanzu Frankenstein, amma da kyau, bari mu yi magana game da hakan a yanzu.

Amma na fahimci cewa daga ma'anar da aka gabatar a sama ba a bayyana cikakken abin da za a iya kira gajimare ba.

Don haka, takarda daga NIST (Cibiyar Ka'idoji da Fasaha ta Ƙasa) tana ba da manyan halaye guda 5 waɗanda kayan aikin girgije ya kamata su kasance da su:

Ba da sabis akan buƙata. Dole ne a ba wa mai amfani damar samun damar yin amfani da kayan aikin kwamfuta kyauta da aka ware masa (kamar cibiyoyin sadarwa, faifai masu kama-da-wane, ƙwaƙwalwar ajiya, kayan aikin sarrafawa, da sauransu), kuma dole ne a samar da waɗannan albarkatun ta atomatik - wato, ba tare da tsoma baki daga mai ba da sabis ba.

Samuwar sabis mai faɗi. Dole ne a samar da dama ga albarkatu ta daidaitattun hanyoyin don ba da damar yin amfani da daidaitattun kwamfutoci biyu da abokan ciniki na bakin ciki da na'urorin hannu.

Haɗa albarkatu cikin wuraren waha. Dole ne wuraren tafkunan albarkatu su sami damar samar da albarkatu ga abokan ciniki da yawa a lokaci guda, tabbatar da cewa abokan ciniki sun ware kuma ba su da tasirin tasiri da gasa don albarkatu. Hakanan ana haɗa hanyoyin sadarwa a cikin tafkunan, wanda ke nuna yuwuwar yin amfani da adireshi masu ruɓani. Ruwan ruwa dole ne su iya yin girma akan buƙata. Yin amfani da wuraren waha yana ba da damar samar da matakan da suka dace na haƙuri da kuskuren albarkatu da ƙayyadaddun kayan aiki na zahiri da kama-da-wane - ana ba mai karɓar sabis ɗin kawai tare da saitin albarkatun da ya nema (inda waɗannan albarkatun suke a zahiri, akan nawa ne. sabobin da sauyawa - ba kome ga abokin ciniki). Duk da haka, dole ne mu yi la'akari da cewa dole ne mai bada sabis ya tabbatar da ajiyar waɗannan albarkatun a bayyane.

Saurin daidaitawa zuwa yanayi daban-daban. Dole ne ayyuka su kasance masu sassauƙa - saurin samar da albarkatu, sake rarraba su, ƙarawa ko rage albarkatu a buƙatar abokin ciniki, kuma a ɓangaren abokin ciniki yakamata a sami jin cewa albarkatun girgije ba su da iyaka. Don sauƙin fahimta, alal misali, ba kwa ganin gargaɗin cewa ɓangaren sararin faifan ku a cikin Apple iCloud ya ɓace saboda rumbun kwamfutar da ke kan uwar garken ya karye, kuma direbobi suna rushewa. Bugu da kari, a bangaren ku, yuwuwar wannan sabis ɗin kusan ba su da iyaka - kuna buƙatar TB 2 - babu matsala, kun biya kuma kun karɓi shi. Ana iya ba da irin wannan misali tare da Google.Drive ko Yandex.Disk.

Yiwuwar auna sabis ɗin da aka bayar. Dole ne tsarin girgije ya sarrafa ta atomatik da haɓaka albarkatun da ake cinyewa, kuma waɗannan hanyoyin dole ne su kasance masu gaskiya ga mai amfani da mai bada sabis. Wato, koyaushe kuna iya bincika yawan albarkatun ku da abokan cinikin ku kuke cinyewa.

Yana da daraja la'akari da cewa waɗannan buƙatun sune mafi yawan buƙatun ga girgije na jama'a, don haka ga girgije mai zaman kansa (wato, girgije da aka kaddamar don bukatun cikin gida na kamfanin), waɗannan buƙatun za a iya daidaita su kadan. Koyaya, har yanzu dole ne a yi su, in ba haka ba ba za mu sami duk fa'idodin ƙididdigar girgije ba.

Me yasa muke buƙatar gajimare?

Koyaya, kowace sabuwar fasaha ko data kasance, kowace sabuwar yarjejeniya an ƙirƙira ta don wani abu (da kyau, ban da RIP-ng, ba shakka). Babu wanda ke buƙatar yarjejeniya don kare yarjejeniya (da kyau, sai dai RIP-ng, ba shakka). Yana da ma'ana cewa an ƙirƙiri Cloud don samar da wani nau'in sabis ga mai amfani/abokin ciniki. Dukanmu mun saba da aƙalla sabis na girgije biyu, misali Dropbox ko Google.Docs, kuma na yi imanin yawancin mutane suna amfani da su cikin nasara - alal misali, an rubuta wannan labarin ta amfani da sabis na girgije na Google.Docs. Amma ayyukan girgijen da muka sani kawai ɓangare ne na iyawar girgije-mafi daidai, sabis ne kawai na SaaS. Za mu iya samar da sabis na girgije ta hanyoyi uku: a cikin nau'i na SaaS, PaaS ko IaaS. Wane sabis ɗin da kuke buƙata ya dogara da sha'awar ku da iyawar ku.

Mu kalli kowanne a jere:

Software a matsayin Sabis (SaaS) samfuri ne don samar da cikakken sabis ga abokin ciniki, misali, sabis na imel kamar Yandex.Mail ko Gmail. A cikin wannan ƙirar isar da sabis, kai, a matsayin abokin ciniki, a zahiri ba ku yin komai sai amfani da sabis - wato, ba kwa buƙatar yin tunani game da kafa sabis ɗin, haƙurinsa ko rashin aiki. Babban abu shine kada ku lalata kalmar sirrinku; mai samar da wannan sabis ɗin zai yi muku sauran. Daga ra'ayi na mai bada sabis, yana da cikakken alhakin dukan sabis - daga uwar garken hardware da runduna aiki tsarin zuwa database da software saituna.

Kayan aiki kamar sabis (PaaS) - lokacin amfani da wannan samfurin, mai bada sabis yana bawa abokin ciniki kayan aiki don sabis, misali, bari mu ɗauki sabar gidan yanar gizo. Mai ba da sabis ya ba abokin ciniki tare da uwar garken kama-da-wane (a zahiri, saitin albarkatun, kamar RAM / CPU / Storage / Nets, da sauransu), har ma da shigar da OS da software masu mahimmanci akan wannan uwar garken, duk da haka, daidaitawar duk waɗannan abubuwan ana yin su ta abokin ciniki da kansa kuma don aikin sabis ɗin abokin ciniki ya amsa. Mai ba da sabis, kamar yadda yake a cikin yanayin da ya gabata, yana da alhakin aiwatar da kayan aiki na zahiri, hypervisors, injin kama-da-wane da kansa, wadatar hanyar sadarwar sa, da sauransu, amma sabis ɗin kansa ba ya cikin yankin da ke da alhakin.

Lantarki a matsayin Sabis (IaaS) - wannan tsarin ya riga ya zama mai ban sha'awa, a gaskiya ma, mai ba da sabis yana ba abokin ciniki cikakken kayan aikin haɓaka - wato, wasu saiti (pool) na albarkatun, irin su CPU Cores, RAM, Networks, da dai sauransu. abokin ciniki - abin da abokin ciniki yake so ya yi da waɗannan albarkatun a cikin tafkin da aka keɓe (ƙididdiga) - ba shi da mahimmanci ga mai sayarwa. Ko abokin ciniki yana so ya ƙirƙiri nasa vEPC ko ma ƙirƙirar ƙaramin ma'aikaci da samar da sabis na sadarwa - babu tambaya - yi shi. A cikin irin wannan yanayin, mai ba da sabis yana da alhakin samar da albarkatu, rashin haƙuri da samuwarsu, da kuma OS wanda ke ba su damar haɗa waɗannan albarkatun kuma ya sa su samuwa ga abokin ciniki tare da ikon haɓaka ko rage albarkatun a kowane lokaci. bisa ga bukatar abokin ciniki. Abokin ciniki yana daidaita duk injunan kama-da-wane da sauran tinsel da kansa ta hanyar tashar sabis na kai da na'ura mai kwakwalwa, gami da kafa hanyoyin sadarwa (sai dai cibiyoyin sadarwa na waje).

Menene OpenStack?

A cikin dukkan zaɓuɓɓuka guda uku, mai bada sabis yana buƙatar OS wanda zai ba da damar ƙirƙirar kayan aikin girgije. A gaskiya ma, tare da SaaS, fiye da ɗaya rabo yana da alhakin dukan tarin fasaha - akwai wani yanki wanda ke da alhakin kayan aiki - wato, yana ba da IaaS zuwa wani yanki, wannan sashin yana ba da SaaS ga abokin ciniki. OpenStack yana ɗaya daga cikin tsarin aiki na gajimare wanda ke ba ku damar tattara gungun maɓallai, sabobin da tsarin ajiya a cikin tafkin albarkatu guda ɗaya, raba wannan tafkin gama gari zuwa wuraren zama (masu haya) kuma samar da waɗannan albarkatun ga abokan ciniki akan hanyar sadarwar.

OpenStack tsarin aiki ne na girgije wanda ke ba ku damar sarrafa manyan wuraren tafki na albarkatun kwamfuta, adana bayanai da albarkatun cibiyar sadarwa, ana samarwa da sarrafa su ta hanyar API ta amfani da daidaitattun hanyoyin tabbatarwa.

A takaice dai, wannan wani tsari ne na ayyukan software na kyauta wanda aka tsara don ƙirƙirar ayyukan girgije (na jama'a da masu zaman kansu) - wato, kayan aikin da ke ba ku damar haɗa uwar garke da kuma canza kayan aiki zuwa wani tafkin ruwa guda ɗaya, sarrafa. waɗannan albarkatun, suna ba da matakin da ya dace na haƙuri da kuskure.

A lokacin rubuta wannan abu, tsarin OpenStack yayi kama da haka:
Gabatarwa zuwa ɓangaren cibiyar sadarwa na kayan aikin girgije
Hoton da aka dauka daga openstack.org

Kowane ɗayan abubuwan da aka haɗa a cikin OpenStack yana yin takamaiman aiki. Wannan gine-ginen da aka rarraba yana ba ku damar haɗawa a cikin bayani saitin kayan aikin da kuke buƙata. Koyaya, wasu abubuwan da aka gyara sune tushen tushen kuma cire su zai haifar da rashin aiki cikakke ko wani ɓangare na maganin gaba ɗaya. Ana rarraba waɗannan abubuwan galibi kamar:

  • Gaban - GUI na tushen yanar gizo don sarrafa ayyukan OpenStack
  • Keystone sabis ne na ainihi wanda aka keɓance wanda ke ba da tabbaci da ayyuka na izini don wasu ayyuka, da kuma sarrafa bayanan mai amfani da ayyukansu.
  • Neutron - sabis na cibiyar sadarwa wanda ke ba da haɗin kai tsakanin mu'amala na sabis na OpenStack daban-daban (ciki har da haɗin kai tsakanin VMs da damar su zuwa duniyar waje)
  • Inderan Kotu - yana ba da damar toshe ajiya don injunan kama-da-wane
  • Nova - gudanar da zagayowar rayuwa na injunan kama-da-wane
  • Duba - ma'ajiyar hotuna na injin kama-da-wane da hotuna
  • Swift - yana ba da dama ga abun ajiya
  • Ceilometer - sabis ɗin da ke ba da ikon tattara na'urorin sadarwa da auna abubuwan da ake samu da cinyewa
  • Heat - ƙungiyar kade-kade bisa samfuran ƙirƙira ta atomatik da samar da albarkatu

Ana iya duba cikakken jerin duk ayyukan da manufarsu a nan.

Kowane ɓangaren OpenStack sabis ne wanda ke yin takamaiman aiki kuma yana ba da API don sarrafa wannan aikin da yin hulɗa tare da sauran ayyukan tsarin aiki na girgije don ƙirƙirar kayan haɗin kai. Misali, Nova yana ba da sarrafa albarkatun kwamfuta da API don samun damar daidaita waɗannan albarkatun, Glance yana ba da sarrafa hoto da API don sarrafa su, Cinder yana ba da ajiyar toshewa da API don sarrafa shi, da sauransu. Dukkan ayyuka suna haɗe-haɗe ta hanya mafi kusa.

Koyaya, idan ka duba, duk ayyukan da ke gudana a cikin OpenStack sune wasu nau'ikan injin kama-da-wane (ko kwantena) da aka haɗa da hanyar sadarwa. Tambayar ta taso - me yasa muke buƙatar abubuwa da yawa?

Bari mu shiga cikin algorithm don ƙirƙirar injin kama-da-wane da haɗa shi zuwa hanyar sadarwa da ma'ajiya mai tsayi a cikin Opentack.

  1. Lokacin da ka ƙirƙiri buƙatun don ƙirƙirar na'ura, ya zama buƙata ta hanyar Horizon (Dashboard) ko buƙatu ta hanyar CLI, abu na farko da ya faru shine izinin buƙatar ku akan Keystone - shin zaku iya ƙirƙirar na'ura, shin yana da 'yancin yin amfani da wannan hanyar sadarwa, ya aikata abin daftarin aiki, da sauransu.
  2. Keystone yana tabbatar da buƙatar ku kuma yana samar da alamar tabbatuwa a cikin saƙon amsawa, wanda za a yi amfani da shi gaba. Bayan samun amsa daga Keystone, ana aika buƙatar zuwa Nova (nova api).
  3. Nova-api yana bincika ingancin buƙatarku ta hanyar tuntuɓar Keystone ta amfani da alamar auth da aka ƙirƙira a baya.
  4. Keystone yana aiwatar da tantancewa kuma yana ba da bayanai kan izini da hane-hane bisa wannan alamar tabbatacciyar.
  5. Nova-api yana ƙirƙirar shigarwa don sabon VM a cikin nova-database kuma ya ƙaddamar da buƙatar ƙirƙira na'ura zuwa mai tsara tsarin nova.
  6. Nova-scheduler ya zaɓi mai watsa shiri (kumburin kwamfuta) wanda za a tura VM bisa ƙayyadaddun sigogi, ma'auni da yankuna. An rubuta rikodin wannan da VM ID zuwa nova-database.
  7. Na gaba, nova-scheduler yana tuntuɓar nova-compute tare da buƙatar tura misali. Nova-compute yana tuntuɓar nova-conductor don samun bayanai game da sigogin injin (nova-conductor wani yanki ne na nova wanda ke aiki azaman uwar garken wakili tsakanin nova-database da nova-compute, yana iyakance adadin buƙatun zuwa tushen bayanan nova don guje wa matsaloli tare da bayanan bayanai. rage girman nauyin daidaito).
  8. Nova-conductor yana karɓar bayanin da aka nema daga nova-database kuma ya tura shi zuwa nova-compute.
  9. Na gaba, nova-compute ya kira kallo don samun ID ɗin hoton. Glace yana tabbatar da buƙatar a cikin Keystone kuma ya dawo da bayanin da aka nema.
  10. Nova-ƙididdigar lambobin neutron don samun bayanai game da sigogin cibiyar sadarwa. Kamar kallo, neutron yana tabbatar da buƙatun a cikin Keystone, bayan haka yana ƙirƙirar shigarwa a cikin ma'ajin bayanai (mai gano tashar jiragen ruwa, da dai sauransu), ƙirƙirar buƙatar ƙirƙirar tashar jiragen ruwa, kuma ya dawo da bayanan da ake buƙata zuwa nova-compute.
  11. Nova-compute lambobin sadarwa cinder tare da bukatar kasafta girma ga na'urar kama-da-wane. Mai kama da kallo, cider yana inganta buƙatun a cikin Keystone, yana ƙirƙirar buƙatar ƙirƙira ƙara, kuma yana dawo da bayanin da aka nema.
  12. Nova-ƙididdigar lambobin sadarwa libvirt tare da buƙatar tura injin kama-da-wane tare da takamaiman sigogi.

A zahiri, aiki da alama mai sauƙi na ƙirƙirar injin kama-da-wane mai sauƙi yana juya zuwa irin wannan guguwar kiran API tsakanin abubuwan dandali na girgije. Bugu da ƙari, kamar yadda kuke gani, hatta ayyukan da aka keɓance a baya suma sun ƙunshi ƙananan sassa waɗanda hulɗar ke faruwa. Ƙirƙirar na'ura ƙaramin ɓangaren abin da dandamalin girgije ke ba ku damar yin - akwai sabis da ke da alhakin daidaita zirga-zirga, sabis ɗin da ke da alhakin adana toshewa, sabis ɗin da ke da alhakin DNS, sabis ɗin da ke da alhakin samar da sabar sabar karfe, da sauransu. Girgizawan yana ba ku damar ɗaukar injunan ku kamar garken tumaki (saɓanin haɓakawa). Idan wani abu ya faru da injin ku a cikin yanayi mai kama-da-wane - kuna dawo da shi daga madadin ajiya, da sauransu, amma ana gina aikace-aikacen girgije ta hanyar da injin kama-da-wane ba ya taka muhimmiyar rawa - injin kama-da-wane “ya mutu” - babu matsala. - wani sabon abu kawai an ƙirƙiri abin hawa yana dogara ne akan samfurin kuma, kamar yadda suke faɗa, ƙungiyar ba ta lura da asarar mayaƙin ba. A zahiri, wannan yana ba da kasancewar hanyoyin ƙungiyar kade-kade - ta amfani da samfuran Heat, zaku iya tura aiki mai rikitarwa cikin sauƙi wanda ya ƙunshi dumbin cibiyoyin sadarwa da injunan kama-da-wane.

Yana da mahimmanci a koyaushe a tuna cewa babu kayan aikin girgije ba tare da hanyar sadarwa ba - kowane nau'i ta hanya ɗaya ko wata yana hulɗa tare da wasu abubuwa ta hanyar hanyar sadarwa. Bugu da kari, gajimaren yana da cikakkiyar hanyar sadarwa mara tsaye. A zahiri, cibiyar sadarwar da ke ƙarƙashin ta ma ta fi ko žasa a tsaye - sabbin nodes da masu sauyawa ba a ƙara su kowace rana, amma ɓangaren mai rufi zai iya canzawa koyaushe - za a ƙara ko share sabbin hanyoyin sadarwa, sabbin injunan kama-da-wane za su bayyana kuma tsofaffi za su bayyana. mutu. Kuma kamar yadda kuka tuna daga ma'anar girgijen da aka bayar a farkon labarin, ya kamata a ware albarkatun ga mai amfani ta atomatik kuma tare da ƙarami (ko mafi kyau tukuna, ba tare da) sa baki daga mai bada sabis ba. Wato nau'in samar da albarkatun cibiyar sadarwa wanda yanzu ya wanzu ta hanyar gaba-gaba a cikin nau'in asusunka na sirri da ake samun damar ta hanyar http/https da injiniyan cibiyar sadarwa Vasily a matsayin mai baya ba girgije ba ne, ko da idan Vasily na da hannu takwas.

Neutron, a matsayin sabis na cibiyar sadarwa, yana ba da API don sarrafa ɓangaren cibiyar sadarwa na kayan aikin girgije. Sabis ɗin yana iko kuma yana sarrafa ɓangaren sadarwar Opentack ta hanyar samar da wani yanki mai ƙima mai suna Network-as-a-Service (NaaS). Wato hanyar sadarwa iri ɗaya ce da ake iya aunawa kamar, misali, rumbun kwamfutoci na CPU ko adadin RAM.

Amma kafin mu ci gaba zuwa gine-ginen ɓangaren cibiyar sadarwa na OpenStack, bari mu yi la'akari da yadda wannan cibiyar sadarwa ke aiki a cikin OpenStack da kuma dalilin da ya sa cibiyar sadarwa ta kasance muhimmin bangare na girgije.

Don haka muna da VMs abokin ciniki na RED guda biyu da VMs abokin ciniki na GREEN guda biyu. Bari mu ɗauka cewa waɗannan injunan suna samuwa a kan hypervisors guda biyu ta wannan hanyar:

Gabatarwa zuwa ɓangaren cibiyar sadarwa na kayan aikin girgije

A halin yanzu, wannan shine kawai haɓakar sabar 4 kuma ba wani abu ba ne, tunda ya zuwa yanzu duk abin da muka yi shine ƙaddamar da sabobin 4, sanya su akan sabobin jiki guda biyu. Kuma ya zuwa yanzu ba a ma haɗa su da hanyar sadarwar.

Don yin gajimare, muna buƙatar ƙara abubuwa da yawa. Da farko, muna haɓaka sashin cibiyar sadarwa - muna buƙatar haɗa waɗannan na'urori 4 a cikin nau'i-nau'i, kuma abokan ciniki suna son haɗin L2. Kuna iya amfani da maɓalli kuma saita gangar jikin a cikin hanyarsa kuma ku warware komai ta amfani da gadar Linux ko, don ƙarin masu amfani da ci gaba, openvswitch (zamu dawo wannan daga baya). Amma za a iya samun yawancin cibiyoyin sadarwa, kuma kullum tura L2 ta hanyar sauyawa ba shine mafi kyawun ra'ayi ba - akwai sassa daban-daban, tebur sabis, watanni na jiran aikace-aikacen da za a kammala, makonni na matsala - a cikin duniyar zamani wannan. kusanci ba ya aiki. Kuma da zarar kamfani ya fahimci haka, zai kasance da sauƙin ci gaba. Don haka, tsakanin masu amfani da hypervisors za mu zabi hanyar sadarwa ta L3 wacce injinan mu za su rika sadarwa, kuma a saman wannan hanyar sadarwa ta L3 za mu gina hanyoyin sadarwa na L2 mai rufe fuska inda zirga-zirgar injinan mu za ta rika tafiya. Kuna iya amfani da GRE, Geneve ko VxLAN azaman encapsulation. Bari mu mai da hankali kan na ƙarshe a yanzu, kodayake ba shi da mahimmanci musamman.

Muna buƙatar nemo VTEP a wani wuri (Ina fata kowa ya san kalmomin VxLAN). Tunda muna da hanyar sadarwar L3 da ke zuwa kai tsaye daga sabobin, babu abin da zai hana mu sanya VTEP akan sabobin da kansu, kuma OVS (OpenvSwitch) yana da kyau a yin wannan. A sakamakon haka, mun sami wannan zane:

Gabatarwa zuwa ɓangaren cibiyar sadarwa na kayan aikin girgije

Tunda zirga-zirga tsakanin VMs dole ne a raba, tashoshin jiragen ruwa zuwa injinan kama-da-wane zasu sami lambobin vlan daban-daban. Lambar tambarin tana taka rawa ne kawai a cikin canji mai kama-da-wane, tunda lokacin da aka sanya shi a cikin VxLAN za mu iya cire shi cikin sauƙi, tunda muna da VNI.

Gabatarwa zuwa ɓangaren cibiyar sadarwa na kayan aikin girgije

Yanzu za mu iya ƙirƙira musu injinan mu da cibiyoyin sadarwa na zamani ba tare da wata matsala ba.

Koyaya, menene idan abokin ciniki yana da wata na'ura, amma yana kan hanyar sadarwa daban? Muna buƙatar rooting tsakanin cibiyoyin sadarwa. Za mu dubi wani zaɓi mai sauƙi lokacin da aka yi amfani da hanyar sadarwa ta tsakiya - wato, ana tafiyar da zirga-zirga ta hanyar nodes na cibiyar sadarwa na musamman (da kyau, a matsayin mai mulkin, an haɗa su tare da nodes masu sarrafawa, don haka za mu sami abu ɗaya).

Da alama babu wani abu mai rikitarwa - muna yin ƙirar gada a kan kullin sarrafawa, fitar da zirga-zirga zuwa gare shi kuma daga nan mu bi ta inda muke buƙata. Amma matsalar ita ce abokin ciniki na RED yana so ya yi amfani da hanyar sadarwar 10.0.0.0/24, kuma abokin ciniki na GREEN yana so ya yi amfani da hanyar sadarwar 10.0.0.0/24. Wato, mun fara haɗa wuraren adireshi. Bugu da ƙari, abokan ciniki ba sa son sauran abokan ciniki su sami damar shiga cikin hanyoyin sadarwar su na ciki, wanda ke da ma'ana. Don raba cibiyoyin sadarwa da zirga-zirgar bayanan abokin ciniki, za mu keɓance keɓantaccen sarari suna ga kowane ɗayansu. Namespace a haƙiƙa kwafin tarin cibiyar sadarwar Linux ne, wato, abokan ciniki a cikin sunan sararin samaniya RED sun keɓe gaba ɗaya daga abokan ciniki daga sararin suna GREEN (da kyau, ko dai ana ba da izinin zirga-zirga tsakanin waɗannan cibiyoyin sadarwar abokin ciniki ta hanyar sunan tsoho ko kan kayan sufuri na sama).

Wato muna samun zane mai zuwa:

Gabatarwa zuwa ɓangaren cibiyar sadarwa na kayan aikin girgije

Tunnels L2 suna haɗuwa daga duk nodes ɗin kwamfuta zuwa kumburin sarrafawa. kumburi inda L3 ke dubawa don waɗannan cibiyoyin sadarwa yana samuwa, kowanne a cikin keɓaɓɓen sarari suna don keɓewa.

Duk da haka, mun manta abu mafi mahimmanci. Dole ne injin kama-da-wane ya ba da sabis ga abokin ciniki, wato, dole ne ya kasance yana da aƙalla mahaɗin waje ɗaya wanda za a iya isa gare shi. Wato muna bukatar mu fita cikin duniyar waje. Akwai zaɓuɓɓuka daban-daban a nan. Bari mu yi zaɓi mafi sauƙi. Za mu ƙara cibiyar sadarwa ɗaya zuwa kowane abokin ciniki, wanda zai kasance mai aiki a cikin hanyar sadarwar mai badawa kuma ba zai zo tare da wasu cibiyoyin sadarwa ba. Cibiyoyin sadarwar kuma za su iya haɗuwa da duba VRF daban-daban a gefen cibiyar sadarwar mai bayarwa. Bayanan hanyar sadarwa kuma za su rayu a cikin sunan kowane abokin ciniki. Duk da haka, har yanzu za su fita zuwa duniyar waje ta hanyar haɗin jiki guda ɗaya (ko haɗin gwiwa, wanda ya fi ma'ana). Don raba zirga-zirgar abokin ciniki, zirga-zirgar zirga-zirgar da ke waje za a yiwa alama tare da alamar VLAN da aka keɓe ga abokin ciniki.

A sakamakon haka, mun sami wannan zane:

Gabatarwa zuwa ɓangaren cibiyar sadarwa na kayan aikin girgije

Tambaya mai ma'ana ita ce me yasa ba za a yi ƙofofin ƙofofin a kan nodes ɗin da kansu ba? Wannan ba babbar matsala ba ce, haka ma, idan kun kunna na'ura mai ba da hanya tsakanin hanyoyin sadarwa (DVR), wannan zai yi aiki. A cikin wannan yanayin, muna la'akari da mafi sauƙi zaɓi tare da ƙofa mai tsakiya, wanda aka yi amfani da shi ta tsohuwa a cikin Opentack. Don ayyuka masu girma, za su yi amfani da na'ura mai ba da hanya tsakanin hanyoyin sadarwa da fasahar haɓaka kamar SR-IOV da Passthrough, amma kamar yadda suke faɗa, wannan labari ne daban. Da farko, bari mu yi magana da ainihin ɓangaren, sa'an nan kuma za mu shiga cikin cikakkun bayanai.

A zahiri, tsarinmu ya riga ya yi aiki, amma akwai wasu nuances:

  • Muna buƙatar ko ta yaya kare injinan mu, wato, sanya matattara a kan mahaɗar mu'amala zuwa abokin ciniki.
  • Ba da damar injin kama-da-wane ya sami adireshin IP ta atomatik, don kada ku shiga ciki ta na'urar wasan bidiyo kowane lokaci kuma ku yi rajistar adireshin.

Bari mu fara da kariyar inji. Don wannan zaka iya amfani da banal iptables, me yasa ba.

Wato yanzu topology ɗinmu ya ɗan ƙara rikitarwa:

Gabatarwa zuwa ɓangaren cibiyar sadarwa na kayan aikin girgije

Mu ci gaba. Muna buƙatar ƙara uwar garken DHCP. Mafi kyawun wuri don gano sabar DHCP ga kowane abokin ciniki zai zama kullin sarrafawa da aka riga aka ambata a sama, inda wuraren suna:

Gabatarwa zuwa ɓangaren cibiyar sadarwa na kayan aikin girgije

Duk da haka, akwai ƙananan matsala. Me zai faru idan komai ya sake farawa kuma duk bayanan game da adiresoshin hayar akan DHCP sun ɓace. Yana da ma'ana cewa za a ba da injinan sabbin adireshi, wanda bai dace sosai ba. Akwai hanyoyi guda biyu a nan - ko dai amfani da sunayen yanki kuma ƙara uwar garken DNS ga kowane abokin ciniki, to adireshin ba zai kasance da mahimmanci a gare mu ba (mai kama da sashin cibiyar sadarwa a cikin k8s) - amma akwai matsala tare da cibiyoyin sadarwa na waje, tun da yake. Hakanan ana iya ba da adireshi a cikin su ta hanyar DHCP - kuna buƙatar aiki tare da sabobin DNS a cikin dandamalin girgije da uwar garken DNS na waje, wanda a ganina ba shi da sauƙi, amma yana yiwuwa. Ko kuma zaɓi na biyu shine amfani da metadata - wato, adana bayanai game da adireshin da aka ba na'ura ta yadda uwar garken DHCP ta san adireshin da za ta ba na'ura idan na'urar ta riga ta sami adireshin. Zaɓin na biyu ya fi sauƙi kuma mafi sauƙi, saboda yana ba ku damar adana ƙarin bayani game da mota. Yanzu bari mu ƙara metadata na wakili zuwa zane:

Gabatarwa zuwa ɓangaren cibiyar sadarwa na kayan aikin girgije

Wani batu wanda kuma ya cancanci tattaunawa shine ikon yin amfani da hanyar sadarwa ta waje ta duk abokan ciniki, tun da cibiyoyin sadarwa na waje, idan dole ne su kasance masu inganci a duk hanyar sadarwar, za su kasance da wahala - kuna buƙatar ci gaba da rarrabawa da sarrafa rabon waɗannan cibiyoyin sadarwa. Ikon yin amfani da hanyar sadarwa guda ɗaya da aka saita ta waje don duk abokan ciniki zai zama da amfani sosai lokacin ƙirƙirar girgije na jama'a. Wannan zai sauƙaƙa ƙaddamar da injuna saboda ba sai mun tuntuɓi bayanan adireshi ba kuma mu zaɓi sararin adireshi na musamman don hanyar sadarwar waje ta kowane abokin ciniki. Bugu da ƙari, za mu iya yin rajistar hanyar sadarwa ta waje a gaba kuma a lokacin turawa za mu buƙaci kawai haɗa adiresoshin waje tare da na'urorin abokin ciniki.

Kuma a nan NAT ta zo don taimakonmu - za mu ba da damar abokan ciniki su sami damar shiga duniyar waje ta hanyar tsoho suna ta amfani da fassarar NAT. To, ga karamar matsala. Wannan yana da kyau idan uwar garken abokin ciniki yana aiki azaman abokin ciniki ba azaman uwar garken ba - wato, yana farawa maimakon karɓar haɗin gwiwa. Amma a gare mu zai zama akasin haka. A wannan yanayin, muna buƙatar yin NAT na makoma ta yadda lokacin karɓar zirga-zirga, kullin sarrafawa ya fahimci cewa wannan zirga-zirgar an yi niyya ne don injin kama-da-wane A na abokin ciniki A, wanda ke nufin muna buƙatar yin fassarar NAT daga adireshin waje, misali 100.1.1.1. .10.0.0.1, zuwa adireshin ciki 100. A wannan yanayin, kodayake duk abokan ciniki za su yi amfani da hanyar sadarwa iri ɗaya, keɓancewa na ciki gaba ɗaya ana kiyaye shi. Wato, muna buƙatar yin dNAT da sNAT akan kullin sarrafawa. Ko don amfani da cibiyar sadarwa guda ɗaya tare da adireshi masu iyo ko cibiyoyin sadarwa na waje, ko duka biyu a lokaci ɗaya, ya dogara da abin da kuke son kawowa cikin gajimare. Ba za mu ƙara adireshi masu iyo a cikin zane ba, amma za mu bar cibiyoyin sadarwa na waje da aka riga aka ƙara a baya - kowane abokin ciniki yana da nasa cibiyar sadarwa ta waje (a cikin zane an nuna su a matsayin vlan 200 da XNUMX akan haɗin waje).

A sakamakon haka, mun sami wani bayani mai ban sha'awa kuma a lokaci guda da aka yi la'akari da kyau, wanda yana da wasu sassauƙa amma har yanzu ba a sami hanyoyin jure wa kuskure ba.

Da fari dai, muna da kumburin sarrafawa ɗaya kawai - gazawarsa zai haifar da rushewar dukkan tsarin. Don gyara wannan matsalar, kuna buƙatar yin aƙalla ƙididdiga na nodes 3. Bari mu ƙara wannan a cikin zane:

Gabatarwa zuwa ɓangaren cibiyar sadarwa na kayan aikin girgije

A zahiri, duk nodes suna aiki tare kuma lokacin da kumburi mai aiki ya fita, wani kumburi zai ɗauki nauyinsa.

Matsala ta gaba ita ce faifan injin kama-da-wane. A halin yanzu, an adana su a kan hypervisors da kansu, kuma idan akwai matsaloli tare da hypervisor, mun rasa duk bayanan - kuma kasancewar harin ba zai taimaka a nan ba idan muka rasa faifai, amma duk uwar garke. Don yin wannan, muna buƙatar yin sabis ɗin da zai yi aiki azaman ƙarshen gaba don wani nau'in ajiya. Wane irin ajiya zai kasance ba shi da mahimmanci a gare mu, amma ya kamata ya kare bayanan mu daga gazawar faifai da kumburi, da kuma yiwuwar dukan majalisar ministocin. Akwai da dama zažužžukan a nan - akwai, ba shakka, SAN cibiyoyin sadarwa tare da Fiber Channel, amma bari mu kasance masu gaskiya - FC ya riga ya zama relic na baya - wani analogue na E1 a sufuri - a, na yarda, shi ne har yanzu amfani, amma kawai inda ba zai yiwu ba idan ba tare da shi ba. Don haka, ba zan tura hanyar sadarwar FC da son rai ba a cikin 2020, sanin cewa akwai sauran hanyoyin da suka fi ban sha'awa. Ko da yake ga kowane nasa, ana iya samun waɗanda suka yi imani cewa FC tare da duk iyakokinta shine abin da muke buƙata - ba zan yi jayayya ba, kowa yana da nasa ra'ayi. Koyaya, mafita mafi ban sha'awa a ganina shine amfani da SDS, kamar Ceph.

Ceph yana ba ku damar gina ingantaccen bayani na ajiyar bayanai tare da tarin yuwuwar zaɓuɓɓukan madadin, farawa tare da lambobi tare da bincika daidaito (mai kama da hari 5 ko 6) yana ƙarewa tare da cikakken kwafin bayanai zuwa faifai daban-daban, la'akari da wurin faifai a ciki. sabobin, da sabar a cikin kabad, da dai sauransu.

Don gina Ceph kuna buƙatar ƙarin nodes 3. Hakanan za'a yi hulɗa tare da ma'ajiyar ta hanyar hanyar sadarwa ta hanyar amfani da toshe, abu da sabis na ajiyar fayil. Bari mu ƙara ajiya zuwa tsarin:

Gabatarwa zuwa ɓangaren cibiyar sadarwa na kayan aikin girgije

Lura: Hakanan zaka iya yin nodes ɗin ƙididdigewa na hyperconverged - wannan shine manufar haɗa ayyuka da yawa akan kumburi ɗaya - misali, ajiya + lissafi - ba tare da keɓance nodes na musamman don ajiyar ceph ba. Za mu sami tsari iri ɗaya na haƙuri - tunda SDS zai adana bayanai tare da matakin ajiyar da muka ƙayyade. Koyaya, nodes masu haɗuwa koyaushe suna daidaitawa - tunda kumburin ajiya baya zafi iska kamar yadda ake gani a farkon kallo (tunda babu injunan kama-da-wane akan sa) - yana kashe albarkatun CPU akan hidimar SDS (a zahiri, yana yin duka). Kwafi da dawowa bayan gazawar nodes, diski, da sauransu). Wato, za ku rasa wasu daga cikin ikon kullin lissafin idan kun haɗa shi da ajiya.

Duk wannan kaya yana buƙatar sarrafa ko ta yaya - muna buƙatar wani abu ta hanyar da za mu iya ƙirƙirar na'ura, hanyar sadarwa, na'ura mai ba da hanya tsakanin hanyoyin sadarwa, da sauransu. abokin ciniki zai iya haɗawa zuwa wannan tashar ta hanyar http / https kuma yayi duk abin da yake buƙata (da kyau, kusan).

A sakamakon haka, yanzu muna da tsarin jure kurakurai. Dole ne a sarrafa duk abubuwan da ke cikin wannan kayan aikin ko ta yaya. An bayyana a baya cewa Opentack jerin ayyuka ne, kowannensu yana ba da takamaiman aiki. Kamar yadda muke gani, akwai abubuwa da yawa da ya kamata a daidaita su da sarrafa su. A yau za mu yi magana game da sashin cibiyar sadarwa.

Neutron gine

A cikin OpenStack, Neutron yana da alhakin haɗa tashar jiragen ruwa na inji zuwa cibiyar sadarwar L2 gama gari, tabbatar da zirga-zirgar zirga-zirga tsakanin VMs da ke kan cibiyoyin sadarwar L2 daban-daban, da kuma fitar da waje, samar da ayyuka kamar NAT, Floating IP, DHCP, da sauransu.

A babban matakin, ana iya kwatanta aikin sabis na cibiyar sadarwa (sashe na asali) kamar haka.

Lokacin fara VM, sabis na cibiyar sadarwa:

  1. Ƙirƙirar tashar jiragen ruwa don VM da aka bayar (ko tashoshin jiragen ruwa) kuma yana sanar da sabis na DHCP game da shi;
  2. An ƙirƙiri sabuwar na'urar hanyar sadarwa ta zamani (ta libvirt);
  3. VM yana haɗa zuwa tashar (s) da aka ƙirƙira a mataki na 1;

Abin ban mamaki, aikin Neutron ya dogara ne akan daidaitattun hanyoyin da suka saba da duk wanda ya taɓa nutsewa cikin Linux - wuraren suna, iptables, gadoji na Linux, openvswitch, conntrack, da sauransu.

Ya kamata a fayyace nan da nan cewa Neutron ba mai sarrafa SDN bane.

Neutron ya ƙunshi abubuwan haɗin kai da yawa:

Gabatarwa zuwa ɓangaren cibiyar sadarwa na kayan aikin girgije

Opentack-neutron-uwar garken daemon ne wanda ke aiki tare da buƙatun mai amfani ta hanyar API. Wannan aljani ba ya da hannu wajen yin rajistar duk wata hanyar sadarwa, amma yana ba da bayanan da suka dace don wannan ga plugins ɗin sa, wanda sai ya daidaita abin da ake so. Wakilan Neutron akan nodes na OpenStack suna yin rajista tare da uwar garken Neutron.

Neutron-uwar garken ainihin aikace-aikacen da aka rubuta a cikin Python, wanda ya ƙunshi sassa biyu:

  • Sabis na REST
  • Neutron Plugin (core/sabis)

An tsara sabis ɗin REST don karɓar kiran API daga wasu abubuwan haɗin gwiwa (misali, buƙatar samar da wasu bayanai, da sauransu)

Plugins sune abubuwan haɗin software/modules waɗanda ake kira yayin buƙatun API - wato, siginar sabis yana faruwa ta hanyar su. Plugins sun kasu kashi biyu - sabis da tushen. A matsayinka na mai mulki, plugin ɗin doki yana da alhakin sarrafa sararin adireshi da haɗin L2 tsakanin VMs, kuma plugins ɗin sabis sun riga sun ba da ƙarin ayyuka kamar VPN ko FW.

Ana iya duba lissafin plugins ɗin da ake samu a yau misali a nan

Ana iya samun plugins ɗin sabis da yawa, amma ana iya samun plugin ɗin doki ɗaya kawai.

Opentack-neutron-ml2 shine ma'auni na Opentack tushen plugin. Wannan plugin ɗin yana da tsarin gine-gine na zamani (ba kamar wanda ya riga shi ba) kuma yana daidaita sabis ɗin cibiyar sadarwa ta direbobin da ke da alaƙa da shi. Za mu kalli plugin ɗin kanta kaɗan daga baya, tunda a zahiri yana ba da sassaucin da OpenStack ke da shi a ɓangaren cibiyar sadarwa. Ana iya maye gurbin tushen plugin ɗin (misali, Sadarwar Sadarwar Sadarwa yana yin irin wannan maye).

Sabis na RPC (rabbitmq-uwar garken) - sabis ɗin da ke ba da sarrafa layin layi da hulɗa tare da wasu sabis na OpenStack, da kuma hulɗa tsakanin wakilan sabis na cibiyar sadarwa.

Wakilan hanyar sadarwa - wakilan da ke cikin kowane kumburi, ta hanyar da aka saita ayyukan cibiyar sadarwa.

Akwai nau'ikan wakilai da yawa.

Babban wakili shine L2 wakili. Wadannan jami'ai suna gudana akan kowane ɗayan hypervisors, gami da nodes masu sarrafawa (mafi daidai, akan duk nodes waɗanda ke ba da kowane sabis ga masu haya) kuma babban aikin su shine haɗa na'urori masu ƙima zuwa hanyar sadarwar L2 gama gari, kuma suna haifar da faɗakarwa lokacin da duk wani lamari ya faru ( misali kashe / kunna tashar jiragen ruwa).

Na gaba, ba ƙaramin wakili ba ne L3 wakili. Ta hanyar tsoho, wannan wakili yana aiki ne kawai akan kullin cibiyar sadarwa (sau da yawa ana haɗa kullin cibiyar sadarwa tare da kumburin sarrafawa) kuma yana ba da hanyar zirga-zirga tsakanin hanyoyin sadarwar hayar (duka tsakanin hanyoyin sadarwarta da hanyoyin sadarwar sauran masu haya, kuma ana iya samun dama ga duniyar waje, samar da ita NAT, kazalika da sabis na DHCP). Koyaya, lokacin amfani da DVR (na'ura mai ba da hanya tsakanin hanyoyin sadarwa), buƙatar plugin L3 shima yana bayyana akan nodes ɗin ƙididdigewa.

Wakilin L3 yana amfani da wuraren sunaye na Linux don samarwa kowane ɗan haya saitin cibiyoyin sadarwar sa keɓanta da ayyukan na'urori masu kama da hanya waɗanda ke yin zirga-zirga da samar da sabis na ƙofar don cibiyoyin sadarwa na Layer 2.

database - bayanai na masu gano hanyoyin sadarwa, subnets, tashar jiragen ruwa, wuraren waha, da sauransu.

A zahiri, Neutron yana karɓar buƙatun API daga ƙirƙirar kowane mahallin cibiyar sadarwa, yana tabbatar da buƙatun, kuma ta hanyar RPC (idan yana samun damar wasu plugins ko wakili) ko REST API (idan yana sadarwa a cikin SDN) yana watsawa ga wakilai (ta hanyar plugins) umarnin da ake buƙata don tsara sabis ɗin da ake buƙata.

Yanzu bari mu juya zuwa shigarwa na gwaji (yadda ake tura shi da abin da aka haɗa a ciki, za mu gani daga baya a cikin sashin aiki) kuma mu ga inda kowane bangare yake:

(overcloud) [stack@undercloud ~]$ openstack network agent list  
+--------------------------------------+--------------------+-------------------------------------+-------------------+-------+-------+---------------------------+
| ID                                   | Agent Type         | Host                                | Availability Zone | Alive | State | Binary                    |
+--------------------------------------+--------------------+-------------------------------------+-------------------+-------+-------+---------------------------+
| 10495de9-ba4b-41fe-b30a-b90ec3f8728b | Open vSwitch agent | overcloud-novacompute-1.localdomain | None              | :-)   | UP    | neutron-openvswitch-agent |
| 1515ad4a-5972-46c3-af5f-e5446dff7ac7 | L3 agent           | overcloud-controller-0.localdomain  | nova              | :-)   | UP    | neutron-l3-agent          |
| 322e62ca-1e5a-479e-9a96-4f26d09abdd7 | DHCP agent         | overcloud-controller-0.localdomain  | nova              | :-)   | UP    | neutron-dhcp-agent        |
| 9c1de2f9-bac5-400e-998d-4360f04fc533 | Open vSwitch agent | overcloud-novacompute-0.localdomain | None              | :-)   | UP    | neutron-openvswitch-agent |
| d99c5657-851e-4d3c-bef6-f1e3bb1acfb0 | Open vSwitch agent | overcloud-controller-0.localdomain  | None              | :-)   | UP    | neutron-openvswitch-agent |
| ff85fae6-5543-45fb-a301-19c57b62d836 | Metadata agent     | overcloud-controller-0.localdomain  | None              | :-)   | UP    | neutron-metadata-agent    |
+--------------------------------------+--------------------+-------------------------------------+-------------------+-------+-------+---------------------------+
(overcloud) [stack@undercloud ~]$ 

Gabatarwa zuwa ɓangaren cibiyar sadarwa na kayan aikin girgije

A gaskiya, wannan shine duka tsarin Neutron. Yanzu yana da daraja kashe ɗan lokaci akan plugin ɗin ML2.

Modular Layer 2

Kamar yadda aka ambata a sama, plugin ɗin shine daidaitaccen tushen tushen tushen OpenStack kuma yana da tsarin gine-gine na zamani.

Wanda ya gabaci ML2 plugin yana da tsarin monolithic, wanda bai ba da izini ba, alal misali, ta amfani da cakuda fasahohi da yawa a cikin shigarwa ɗaya. Misali, ba za ku iya amfani da duka openvswitch da linuxbridge a lokaci guda ba - ko dai na farko ko na biyu. Don wannan dalili, an ƙirƙiri plugin ɗin ML2 tare da gine-ginensa.

ML2 yana da abubuwa biyu - nau'ikan direbobi guda biyu: Nau'in direbobi da direbobin injina.

Nau'in direbobi Ƙayyade fasahar da za a yi amfani da su don tsara haɗin yanar gizo, misali VxLAN, VLAN, GRE. A lokaci guda, direba yana ba da damar amfani da fasaha daban-daban. Daidaitaccen fasaha shine VxLAN encapsulation don cibiyoyin sadarwa masu rufi da cibiyoyin sadarwa na waje vlan.

Nau'in direbobi sun haɗa da nau'ikan hanyar sadarwa masu zuwa:

Flat - cibiyar sadarwa ba tare da tagging ba
VLANs - tagged cibiyar sadarwa
Na gida - nau'in cibiyar sadarwa na musamman don shigarwa gabaɗaya (irin waɗannan abubuwan ana buƙatar ko dai don masu haɓakawa ko don horo)
GRE - cibiyar sadarwa mai rufi ta amfani da ramukan GRE
VxLAN - cibiyar sadarwa mai rufi ta amfani da tunnels VxLAN

Direbobin injina ayyana kayan aikin da ke tabbatar da tsarin fasahar da aka ƙayyade a cikin nau'in direba - misali, openvswitch, sr-iov, opendaylight, OVN, da dai sauransu.

Dangane da aiwatar da wannan direban, ko dai wakilan da Neutron ke sarrafawa za a yi amfani da su, ko kuma za a yi amfani da haɗin kai zuwa mai kula da SDN na waje, wanda ke kula da duk batutuwan da suka shafi tsara hanyoyin sadarwa na L2, kewayawa, da sauransu.

Misali: idan muka yi amfani da ML2 tare da OVS, to, an shigar da wakili na L2 akan kowane kumburin kwamfuta wanda ke sarrafa OVS. Koyaya, idan muka yi amfani da, alal misali, OVN ko OpenDayLight, to, ikon OVS yana zuwa ƙarƙashin ikonsu - Neutron, ta hanyar tushen plugin, yana ba da umarni ga mai sarrafawa, kuma ya riga ya aikata abin da aka faɗa.

Bari mu goge kan Buɗe vSwitch

A halin yanzu, ɗayan mahimman abubuwan OpenStack shine Buɗe vSwitch.
Lokacin shigar da OpenStack ba tare da ƙarin SDN mai siyarwa ba kamar Juniper Contrail ko Nokia Nuage, OVS shine babban ɓangaren cibiyar sadarwa na cibiyar sadarwar girgije kuma, tare da iptables, conntrack, wuraren suna, suna ba ku damar tsara cibiyoyin sadarwa masu rufin gidaje masu yawa. A zahiri, ana iya maye gurbin wannan ɓangaren, alal misali, lokacin amfani da mafita na SDN na ɓangare na uku (mai siyarwa).

OVS shine buɗaɗɗen hanyar sauya software wanda aka ƙera don amfani a cikin mahalli masu ƙima azaman mai isar da zirga-zirgar ababen hawa.

A halin yanzu, OVS yana da kyakkyawan aiki, wanda ya haɗa da fasaha kamar QoS, LACP, VLAN, VxLAN, GENEVE, OpenFlow, DPDK, da sauransu.

Lura: Ba a fara ɗaukar OVS azaman canji mai laushi don ayyukan sadarwar da aka ɗora ba kuma an fi ƙera shi don ƙarancin ayyukan IT masu buƙatar bandwidth kamar sabar WEB ko sabar saƙo. Duk da haka, ana ci gaba da haɓaka OVS kuma ayyukan OVS na yanzu sun inganta aikinta da kuma iyawarta, wanda ya ba da damar yin amfani da shi ta hanyar masu amfani da sadarwa tare da ayyuka masu yawa, alal misali, akwai aiwatar da OVS tare da goyon baya ga hanzarin DPDK.

Akwai muhimman abubuwa guda uku na OVS waɗanda kuke buƙatar sani:

  • Kernel module - wani ɓangaren da ke cikin sararin kernel wanda ke tafiyar da zirga-zirga bisa ka'idojin da aka karɓa daga sashin sarrafawa;
  • vSwitch daemon (ovs-vswitchd) wani tsari ne da aka ƙaddamar a cikin sararin mai amfani wanda ke da alhakin tsara tsarin kernel - wato, yana wakiltar ma'anar aikin sauyawa.
  • Sabar Database - bayanai na gida wanda ke kan kowane mai masaukin baki da ke gudana OVS, wanda aka adana tsarin. Masu kula da SDN na iya sadarwa ta wannan tsarin ta amfani da ka'idar OVSDB.

Duk wannan yana tare da saitin kayan aikin bincike da gudanarwa, kamar ovs-vsctl, ovs-appctl, ovs-ofctl, da sauransu.

A halin yanzu, Opentack yana amfani da shi sosai ta hanyar sadarwar sadarwa don ƙaura ayyukan cibiyar sadarwa zuwa gare ta, kamar EPC, SBC, HLR, da sauransu. yawan zirga-zirga (yanzu yawan zirga-zirga ya kai gigabits ɗari da yawa a cikin daƙiƙa guda). A zahiri, tuƙi irin wannan zirga-zirga ta sararin kernel (tunda mai turawa yana can ta tsohuwa) ba shine mafi kyawun ra'ayi ba. Saboda haka, OVS galibi ana tura shi gabaɗaya a cikin sararin mai amfani ta amfani da fasahar haɓaka DPDK don tura zirga-zirga daga NIC zuwa sararin mai amfani da ke kewaye da kwaya.

Lura: don girgijen da aka tura don ayyukan sadarwa, yana yiwuwa a fitar da zirga-zirga daga kullin ƙididdigewa ta hanyar wucewa OVS kai tsaye zuwa kayan aiki na sauyawa. Ana amfani da hanyoyin SR-IOV da Passthrough don wannan dalili.

Ta yaya wannan ke aiki akan shimfidar wuri na gaske?

To, yanzu bari mu matsa zuwa sashin aiki mu ga yadda duk yake aiki a aikace.

Da farko, bari mu ƙaddamar da shigarwa mai sauƙi na Opentack. Tun da ba ni da saitin sabar a hannuna don gwaje-gwaje, za mu haɗa samfurin akan sabar ta jiki ɗaya daga injina. Haka ne, a zahiri, irin wannan bayani bai dace da dalilai na kasuwanci ba, amma don ganin misalin yadda cibiyar sadarwa ke aiki a cikin Openstack, irin wannan shigarwa ya isa ga idanu. Bugu da ƙari, irin wannan shigarwa ya fi ban sha'awa don dalilai na horo - tun da za ku iya kama zirga-zirga, da dai sauransu.

Tun da kawai muna buƙatar ganin ainihin ɓangaren, ba za mu iya amfani da cibiyoyin sadarwa da yawa ba amma tada komai ta amfani da cibiyoyin sadarwa guda biyu kawai, kuma cibiyar sadarwa ta biyu a cikin wannan shimfidar za a yi amfani da ita kawai don samun dama ga uwar garken undercloud da DNS. Ba za mu taɓa hanyoyin sadarwar waje ba a yanzu - wannan jigo ne don babban labarin daban.

Don haka, bari mu fara cikin tsari. Na farko, kadan ka'idar. Za mu shigar da Opentack ta amfani da TripleO (Openstack on Opentack). Mahimmancin TripleO shine mu shigar da Opentack duk-in-one (wato, akan kulli ɗaya), wanda ake kira undercloud, sannan mu yi amfani da damar da aka tura Opentack don shigar da Opentack wanda aka yi niyya don aiki, wanda ake kira overcloud. Undercloud zai yi amfani da ikonsa na asali don sarrafa sabar ta jiki (ƙarfe mara ƙarfi) - aikin Ironic - don samar da hypervisors waɗanda za su yi ayyukan ƙididdigewa, sarrafawa, nodes na ajiya. Wato, ba ma amfani da kowane kayan aikin ɓangare na uku don tura Opentack - muna tura Opentack ta amfani da Opentack. Zai zama mafi bayyane yayin da shigarwa ke ci gaba, don haka ba za mu tsaya a can ba kuma mu ci gaba.

Lura: A cikin wannan labarin, don sauƙaƙa, ban yi amfani da keɓewar cibiyar sadarwa ba don cibiyoyin sadarwa na Opentack na ciki, amma ana tura komai ta amfani da hanyar sadarwa ɗaya kawai. Duk da haka, kasancewar ko rashi na keɓewar cibiyar sadarwa ba ya shafar ainihin aikin mafita - duk abin da zai yi aiki daidai da lokacin amfani da keɓancewa, amma zirga-zirgar zirga-zirga za ta gudana a kan hanyar sadarwa ɗaya. Don shigarwar kasuwanci, a zahiri ya zama dole a yi amfani da keɓewa ta amfani da vlans daban-daban da musaya. Misali, zirga-zirgar sarrafa ma'ajiyar ceph da zirga-zirgar bayanai ita kanta (hanzar da na'ura zuwa faifai, da dai sauransu) lokacin keɓance amfani da subnets daban-daban (Storage Management and Storage) kuma wannan yana ba ku damar sanya maganin ya zama mai jurewa ta hanyar rarraba wannan zirga-zirga, misali. , a cikin tashoshin jiragen ruwa daban-daban, ko amfani da bayanan QoS daban-daban don zirga-zirga daban-daban don kada zirga-zirgar bayanai ta matse zirga-zirgar sigina. A cikin yanayinmu, za su tafi a kan hanyar sadarwa guda ɗaya kuma a gaskiya wannan baya iyakance mu ta kowace hanya.

Lura: Tun da za mu gudanar da injunan kama-da-wane a cikin yanayi mai kama-da-wane dangane da injunan kama-da-wane, da farko muna buƙatar ba da damar ƙwararrun ƙira.

Kuna iya bincika ko an kunna ƙirar ƙira ko a'a kamar haka:


[root@hp-gen9 bormoglotx]# cat /sys/module/kvm_intel/parameters/nested
N
[root@hp-gen9 bormoglotx]# 

Idan kun ga harafin N, to muna ba da goyan baya don ƙirar ƙira bisa ga kowane jagorar da kuka samu akan hanyar sadarwar, misali. irin wannan .

Muna buƙatar haɗa da'ira mai zuwa daga injunan kama-da-wane:

Gabatarwa zuwa ɓangaren cibiyar sadarwa na kayan aikin girgije

A cikin akwati na, don haɗa na'urori masu mahimmanci waɗanda ke cikin ɓangaren shigarwa na gaba (kuma na sami 7 daga cikinsu, amma za ku iya samun ta tare da 4 idan ba ku da albarkatun da yawa), Na yi amfani da OpenvSwitch. Na ƙirƙiri gadar ovs guda ɗaya kuma na haɗa injunan kama-da-wane zuwa gare ta ta ƙungiyoyin tashar jiragen ruwa. Don yin wannan, na ƙirƙiri fayil na xml kamar haka:


[root@hp-gen9 ~]# virsh net-dumpxml ovs-network-1        
<network>
  <name>ovs-network-1</name>
  <uuid>7a2e7de7-fc16-4e00-b1ed-4d190133af67</uuid>
  <forward mode='bridge'/>
  <bridge name='ovs-br1'/>
  <virtualport type='openvswitch'/>
  <portgroup name='trunk-1'>
    <vlan trunk='yes'>
      <tag id='100'/>
      <tag id='101'/>
      <tag id='102'/>
    </vlan>
  </portgroup>
  <portgroup name='access-100'>
    <vlan>
      <tag id='100'/>
    </vlan>
  </portgroup>
  <portgroup name='access-101'>
    <vlan>
      <tag id='101'/>
    </vlan>
  </portgroup>
</network>

An bayyana ƙungiyoyin tashar jiragen ruwa guda uku a nan - damar shiga guda biyu da akwati ɗaya (ana buƙatar na ƙarshe don uwar garken DNS, amma kuna iya yin ba tare da shi ba, ko shigar da shi akan injin mai ɗaukar hoto - duk wanda ya fi dacewa da ku). Bayan haka, ta amfani da wannan samfuri, muna bayyana namu ta hanyar virsh net-define:


virsh net-define ovs-network-1.xml 
virsh net-start ovs-network-1 
virsh net-autostart ovs-network-1 

Yanzu muna gyara saitunan tashar tashar hypervisor:


[root@hp-gen9 ~]# cat /etc/sysconfig/network-scripts/ifcfg-ens1f0   
TYPE=Ethernet
NAME=ens1f0
DEVICE=ens1f0
TYPE=OVSPort
DEVICETYPE=ovs
OVS_BRIDGE=ovs-br1
ONBOOT=yes
OVS_OPTIONS="trunk=100,101,102"
[root@hp-gen9 ~]
[root@hp-gen9 ~]# cat /etc/sysconfig/network-scripts/ifcfg-ovs-br1 
DEVICE=ovs-br1
DEVICETYPE=ovs
TYPE=OVSBridge
BOOTPROTO=static
ONBOOT=yes
IPADDR=192.168.255.200
PREFIX=24
[root@hp-gen9 ~]# 

Lura: a cikin wannan yanayin, adireshin da ke kan tashar ovs-br1 ba zai sami damar shiga ba saboda bashi da alamar vlan. Don gyara wannan, kuna buƙatar bayar da umarnin sudo ovs-vsctl saita tashar ovs-br1 tag=100. Koyaya, bayan sake kunnawa, wannan tag ɗin zai ɓace (idan wani ya san yadda ake sanya shi zama a wurin, zan yi godiya sosai). Amma wannan ba shi da mahimmanci, saboda kawai za mu buƙaci wannan adireshin yayin shigarwa kuma ba za mu buƙaci shi ba lokacin da aka ƙaddamar da Opentack.

Na gaba, mun ƙirƙiri na'urar da ke ƙarƙashin girgije:


virt-install  -n undercloud --description "undercloud"  --os-type=Linux  --os-variant=centos7.0  --ram=8192  --vcpus=8  --disk path=/var/lib/libvirt/images/undercloud.qcow2,bus=virtio,size=40,format=qcow2 --network network:ovs-network-1,model=virtio,portgroup=access-100 --network network:ovs-network-1,model=virtio,portgroup=access-101 --graphics none  --location /var/lib/libvirt/boot/CentOS-7-x86_64-Minimal-2003.iso --extra-args console=ttyS0

Lokacin shigarwa, kuna saita duk sigogin da ake buƙata, kamar sunan injin, kalmomin shiga, masu amfani, sabar ntp, da sauransu, zaku iya daidaita tashoshin jiragen ruwa nan da nan, amma a gare ni, bayan shigarwa, yana da sauƙi don shiga cikin injin ta hanyar. na'urar wasan bidiyo kuma gyara fayilolin da suka dace. Idan kun riga kuna da hoton da aka yi, za ku iya amfani da shi, ko yin abin da na yi - zazzage ƙaramin hoton Centos 7 kuma yi amfani da shi don shigar da VM.

Bayan nasarar shigarwa, ya kamata ka sami injin kama-da-wane wanda zaka iya shigar da undercloud akansa


[root@hp-gen9 bormoglotx]# virsh list
 Id    Name                           State
----------------------------------------------------
 6     dns-server                     running
 62    undercloud                     running

Da farko, shigar da kayan aikin da ake buƙata don tsarin shigarwa:

sudo yum update -y
sudo yum install -y net-tools
sudo yum install -y wget
sudo yum install -y ipmitool

Shigar da Undercloud

Muna ƙirƙirar mai amfani, saita kalmar sirri, ƙara shi zuwa sudoer kuma muna ba shi ikon aiwatar da tushen umarni ta hanyar sudo ba tare da shigar da kalmar sirri ba:


useradd stack
passwd stack

echo “stack ALL=(root) NOPASSWD:ALL” > /etc/sudoers.d/stack
chmod 0440 /etc/sudoers.d/stack

Yanzu mun ƙayyade cikakken sunan undercloud a cikin fayil ɗin runduna:


vi /etc/hosts

127.0.0.1   undercloud.openstack.rnd localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

Bayan haka, muna ƙara ma'ajiyar ajiya kuma mu shigar da software da muke buƙata:


sudo yum install -y https://trunk.rdoproject.org/centos7/current/python2-tripleo-repos-0.0.1-0.20200409224957.8bac392.el7.noarch.rpm
sudo -E tripleo-repos -b queens current
sudo -E tripleo-repos -b queens current ceph
sudo yum install -y python-tripleoclient
sudo yum install -y ceph-ansible

Lura: idan ba ku shirya shigar da ceph ba, to ba kwa buƙatar shigar da umarni masu alaƙa da ceph. Na yi amfani da sakin Queens, amma kuna iya amfani da duk wani wanda kuke so.

Na gaba, kwafi fayil ɗin sanyi na ƙarƙashin Cloud zuwa tari na gidan mai amfani:


cp /usr/share/instack-undercloud/undercloud.conf.sample ~/undercloud.conf

Yanzu muna buƙatar gyara wannan fayil ɗin, daidaita shi zuwa shigarwar mu.

Kuna buƙatar ƙara waɗannan layukan zuwa farkon fayil ɗin:

vi undercloud.conf
[DEFAULT]
undercloud_hostname = undercloud.openstack.rnd
local_ip = 192.168.255.1/24
network_gateway = 192.168.255.1
undercloud_public_host = 192.168.255.2
undercloud_admin_host = 192.168.255.3
undercloud_nameservers = 192.168.255.253
generate_service_certificate = false
local_interface = eth0
local_mtu = 1450
network_cidr = 192.168.255.0/24
masquerade = true
masquerade_network = 192.168.255.0/24
dhcp_start = 192.168.255.11
dhcp_end = 192.168.255.50
inspection_iprange = 192.168.255.51,192.168.255.100
scheduler_max_attempts = 10

Don haka, bari mu shiga cikin saitunan:

undercloud_hostname - cikakken sunan uwar garken undercloud, dole ne ya dace da shigarwa akan uwar garken DNS

local_ip - adireshin ƙaƙƙarfan girgije na gida zuwa samar da hanyar sadarwa

hanyar sadarwa_gateway - adireshin gida guda ɗaya, wanda zai kasance a matsayin ƙofa don samun damar zuwa duniyar waje yayin shigar da nodes na overcloud, kuma ya dace da ip na gida.

undercloud_public_host - Adireshin API na waje, an sanya kowane adireshin kyauta daga hanyar sadarwar samarwa

undercloud_admin_host Adireshin API na ciki, kowane adireshin kyauta daga cibiyar sadarwar samarwa an sanya shi

undercloud_nameservers - uwar garken DNS

samar da_service_certificate - wannan layin yana da mahimmanci a cikin misalin na yanzu, saboda idan ba ku saita shi zuwa ƙarya ba za ku sami kuskure yayin shigarwa, an kwatanta matsalar akan Red Hat bug tracker.

local_interface dubawa a cikin samar da hanyar sadarwa. Za a sake saita wannan haɗin yanar gizon yayin turawar karkashin girgije, don haka kuna buƙatar samun musaya biyu akan undercloud - ɗaya don samun dama gare shi, na biyu don samarwa.

local_mtu - MTU. Tunda muna da dakin gwaje-gwaje kuma ina da MTU na 1500 akan tashar jiragen ruwa na OVS, dole ne a saita shi zuwa 1450 don fakitin da ke cikin VxLAN su wuce.

network_cidr - samar da hanyar sadarwa

m - amfani da NAT don samun damar hanyar sadarwar waje

masquerade_network - cibiyar sadarwa da za a NATed

dhcp_fara - adireshin farawa na wurin ajiyar adireshi daga inda za a sanya adiresoshin zuwa nodes yayin jigilar overcloud

dhcp_ karshen - adireshin ƙarshe na wurin ajiyar adireshin inda za a sanya adiresoshin zuwa nodes yayin jigilar overcloud

dubawa_iprange - tafkin adiresoshin da suka wajaba don dubawa (bai kamata su zo tare da tafkin da ke sama ba)

jadawali_max_kokarin - matsakaicin adadin ƙoƙarin shigar overcloud (dole ne ya fi ko daidai da adadin nodes)

Bayan an bayyana fayil ɗin, zaku iya ba da umarni don tura undercloud:


openstack undercloud install

Hanyar yana ɗaukar daga mintuna 10 zuwa 30 dangane da baƙin ƙarfe. A ƙarshe ya kamata ku ga fitarwa kamar haka:

vi undercloud.conf
2020-08-13 23:13:12,668 INFO: 
#############################################################################
Undercloud install complete.

The file containing this installation's passwords is at
/home/stack/undercloud-passwords.conf.

There is also a stackrc file at /home/stack/stackrc.

These files are needed to interact with the OpenStack services, and should be
secured.

#############################################################################

Wannan fitowar ta ce kun sami nasarar shigar da undercloud kuma yanzu zaku iya duba matsayin undercloud kuma ku ci gaba da shigar da overcloud.

Idan ka kalli fitarwar ifconfig, za ka ga cewa sabon ƙirar gada ya bayyana

[stack@undercloud ~]$ ifconfig
br-ctlplane: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 192.168.255.1  netmask 255.255.255.0  broadcast 192.168.255.255
        inet6 fe80::5054:ff:fe2c:89e  prefixlen 64  scopeid 0x20<link>
        ether 52:54:00:2c:08:9e  txqueuelen 1000  (Ethernet)
        RX packets 14  bytes 1095 (1.0 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 20  bytes 1292 (1.2 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Yanzu za a aiwatar da jigilar overcloud ta wannan hanyar sadarwa.

Daga fitowar da ke ƙasa za ku iya ganin cewa muna da duk ayyuka akan kulli ɗaya:

(undercloud) [stack@undercloud ~]$ openstack host list
+--------------------------+-----------+----------+
| Host Name                | Service   | Zone     |
+--------------------------+-----------+----------+
| undercloud.openstack.rnd | conductor | internal |
| undercloud.openstack.rnd | scheduler | internal |
| undercloud.openstack.rnd | compute   | nova     |
+--------------------------+-----------+----------+

A ƙasa akwai daidaitawar ɓangaren cibiyar sadarwa na ƙarƙashin girgije:


(undercloud) [stack@undercloud ~]$ python -m json.tool /etc/os-net-config/config.json 
{
    "network_config": [
        {
            "addresses": [
                {
                    "ip_netmask": "192.168.255.1/24"
                }
            ],
            "members": [
                {
                    "dns_servers": [
                        "192.168.255.253"
                    ],
                    "mtu": 1450,
                    "name": "eth0",
                    "primary": "true",
                    "type": "interface"
                }
            ],
            "mtu": 1450,
            "name": "br-ctlplane",
            "ovs_extra": [
                "br-set-external-id br-ctlplane bridge-id br-ctlplane"
            ],
            "routes": [],
            "type": "ovs_bridge"
        }
    ]
}
(undercloud) [stack@undercloud ~]$

Overcloud shigarwa

A halin yanzu muna da undercloud ne kawai, kuma ba mu da isassun nodes waɗanda za a taru overcloud. Don haka, da farko, bari mu tura injinan kama-da-wane da muke buƙata. A lokacin aikin, undercloud da kansa zai shigar da OS da software masu mahimmanci akan na'urar overcloud - wato, ba ma buƙatar tura na'urar gaba ɗaya ba, amma kawai ƙirƙira faifai (ko diski) don shi kuma a tantance sigoginsa - wato. , a zahiri, muna samun sabar mara amfani ba tare da shigar da OS akansa ba.

Bari mu je babban fayil tare da faifai na injinan mu kuma mu ƙirƙiri faifai girman da ake buƙata:


cd /var/lib/libvirt/images/
qemu-img create -f qcow2 -o preallocation=metadata control-1.qcow2 60G
qemu-img create -f qcow2 -o preallocation=metadata compute-1.qcow2 60G
qemu-img create -f qcow2 -o preallocation=metadata compute-2.qcow2 60G
qemu-img create -f qcow2 -o preallocation=metadata storage-1.qcow2 160G
qemu-img create -f qcow2 -o preallocation=metadata storage-2.qcow2 160G

Tunda muna aiki azaman tushen, muna buƙatar canza mai waɗannan diski don kada mu sami matsala ta haƙƙoƙin:


[root@hp-gen9 images]# ls -lh
total 5.8G
drwxr-xr-x. 2 qemu qemu 4.0K Aug 13 16:15 backups
-rw-r--r--. 1 root root  61G Aug 14 03:07 compute-1.qcow2
-rw-r--r--. 1 root root  61G Aug 14 03:07 compute-2.qcow2
-rw-r--r--. 1 root root  61G Aug 14 03:07 control-1.qcow2
-rw-------. 1 qemu qemu  41G Aug 14 03:03 dns-server.qcow2
-rw-r--r--. 1 root root 161G Aug 14 03:07 storage-1.qcow2
-rw-r--r--. 1 root root 161G Aug 14 03:07 storage-2.qcow2
-rw-------. 1 qemu qemu  41G Aug 14 03:07 undercloud.qcow2
[root@hp-gen9 images]# 
[root@hp-gen9 images]# 
[root@hp-gen9 images]# chown qemu:qemu /var/lib/libvirt/images/*qcow2
[root@hp-gen9 images]# ls -lh
total 5.8G
drwxr-xr-x. 2 qemu qemu 4.0K Aug 13 16:15 backups
-rw-r--r--. 1 qemu qemu  61G Aug 14 03:07 compute-1.qcow2
-rw-r--r--. 1 qemu qemu  61G Aug 14 03:07 compute-2.qcow2
-rw-r--r--. 1 qemu qemu  61G Aug 14 03:07 control-1.qcow2
-rw-------. 1 qemu qemu  41G Aug 14 03:03 dns-server.qcow2
-rw-r--r--. 1 qemu qemu 161G Aug 14 03:07 storage-1.qcow2
-rw-r--r--. 1 qemu qemu 161G Aug 14 03:07 storage-2.qcow2
-rw-------. 1 qemu qemu  41G Aug 14 03:08 undercloud.qcow2
[root@hp-gen9 images]# 

Lura: idan ba ku shirya shigar da ceph don yin nazarinsa ba, to umarni ba su ƙirƙiri aƙalla nodes 3 tare da aƙalla diski guda biyu ba, amma a cikin samfuri suna nuna cewa za a yi amfani da diski mai kama da vda, vdb, da sauransu.

Mai girma, yanzu muna buƙatar ayyana duk waɗannan injinan:


virt-install --name control-1 --ram 32768 --vcpus 8 --os-variant centos7.0 --disk path=/var/lib/libvirt/images/control-1.qcow2,device=disk,bus=virtio,format=qcow2 --noautoconsole --vnc  --network network:ovs-network-1,model=virtio,portgroup=access-100 --network network:ovs-network-1,model=virtio,portgroup=trunk-1 --dry-run --print-xml > /tmp/control-1.xml  

virt-install --name storage-1 --ram 16384 --vcpus 4 --os-variant centos7.0 --disk path=/var/lib/libvirt/images/storage-1.qcow2,device=disk,bus=virtio,format=qcow2 --noautoconsole --vnc  --network network:ovs-network-1,model=virtio,portgroup=access-100 --dry-run --print-xml > /tmp/storage-1.xml  

virt-install --name storage-2 --ram 16384 --vcpus 4 --os-variant centos7.0 --disk path=/var/lib/libvirt/images/storage-2.qcow2,device=disk,bus=virtio,format=qcow2 --noautoconsole --vnc  --network network:ovs-network-1,model=virtio,portgroup=access-100 --dry-run --print-xml > /tmp/storage-2.xml  

virt-install --name compute-1 --ram 32768 --vcpus 12 --os-variant centos7.0 --disk path=/var/lib/libvirt/images/compute-1.qcow2,device=disk,bus=virtio,format=qcow2 --noautoconsole --vnc  --network network:ovs-network-1,model=virtio,portgroup=access-100 --dry-run --print-xml > /tmp/compute-1.xml  

virt-install --name compute-2 --ram 32768 --vcpus 12 --os-variant centos7.0 --disk path=/var/lib/libvirt/images/compute-2.qcow2,device=disk,bus=virtio,format=qcow2 --noautoconsole --vnc  --network network:ovs-network-1,model=virtio,portgroup=access-100 --dry-run --print-xml > /tmp/compute-2.xml 

A ƙarshe akwai umarni -print-xml> /tmp/storage-1.xml, wanda ke ƙirƙirar fayil xml tare da bayanin kowace na'ura a cikin /tmp/ folder, idan ba ku ƙara ba, ba za ku kasance ba. iya gane injunan kama-da-wane.

Yanzu muna buƙatar ayyana duk waɗannan injina a cikin virsh:


virsh define --file /tmp/control-1.xml
virsh define --file /tmp/compute-1.xml
virsh define --file /tmp/compute-2.xml
virsh define --file /tmp/storage-1.xml
virsh define --file /tmp/storage-2.xml

[root@hp-gen9 ~]# virsh list --all
 Id    Name                           State
----------------------------------------------------
 6     dns-server                     running
 64    undercloud                     running
 -     compute-1                      shut off
 -     compute-2                      shut off
 -     control-1                      shut off
 -     storage-1                      shut off
 -     storage-2                      shut off

[root@hp-gen9 ~]#

Yanzu ƙaramin nuance - tripleO yana amfani da IPMI don sarrafa sabar yayin shigarwa da dubawa.

Introspection shine tsarin bincika kayan aikin don samun sigogin sa waɗanda suka wajaba don ƙarin samar da nodes. Ana gudanar da bincike ta hanyar amfani da baƙin ƙarfe, sabis ɗin da aka tsara don aiki tare da sabar sabar ƙarfe maras tushe.

Amma ga matsalar - yayin da na'urorin IPMI na hardware suna da tashar jiragen ruwa daban (ko tashar jiragen ruwa, amma wannan ba shi da mahimmanci), to, inji mai mahimmanci ba su da irin wannan tashar jiragen ruwa. Anan wani crutch mai suna vbmc ya zo don taimakonmu - kayan aikin da ke ba ku damar yin koyi da tashar tashar IPMI. Wannan nuance ya kamata a kula da shi musamman ga waɗanda suke so su kafa irin wannan dakin gwaje-gwaje a kan ESXI hypervisor - a gaskiya, ban sani ba idan yana da analogue na vbmc, don haka yana da kyau a yi mamaki game da wannan batu kafin ƙaddamar da komai. .

Shigar vbmc:


yum install yum install python2-virtualbmc

Idan OS ɗinku ba zai iya samun fakitin ba, to ƙara ma'ajiyar:

yum install -y https://www.rdoproject.org/repos/rdo-release.rpm

Yanzu mun kafa mai amfani. Komai a nan banal ne har zuwa wulakanci. Yanzu yana da ma'ana cewa babu sabobin a cikin jerin vbmc


[root@hp-gen9 ~]# vbmc list

[root@hp-gen9 ~]# 

Domin su bayyana, dole ne a bayyana su da hannu kamar haka:


[root@hp-gen9 ~]# vbmc add control-1 --port 7001 --username admin --password admin
[root@hp-gen9 ~]# vbmc add storage-1 --port 7002 --username admin --password admin
[root@hp-gen9 ~]# vbmc add storage-2 --port 7003 --username admin --password admin
[root@hp-gen9 ~]# vbmc add compute-1 --port 7004 --username admin --password admin
[root@hp-gen9 ~]# vbmc add compute-2 --port 7005 --username admin --password admin
[root@hp-gen9 ~]#
[root@hp-gen9 ~]# vbmc list
+-------------+--------+---------+------+
| Domain name | Status | Address | Port |
+-------------+--------+---------+------+
| compute-1   | down   | ::      | 7004 |
| compute-2   | down   | ::      | 7005 |
| control-1   | down   | ::      | 7001 |
| storage-1   | down   | ::      | 7002 |
| storage-2   | down   | ::      | 7003 |
+-------------+--------+---------+------+
[root@hp-gen9 ~]#

Ina tsammanin tsarin tsarin umarni a bayyane yake ba tare da bayani ba. Koyaya, a yanzu duk zaman mu yana kan matsayin DOWN. Don su matsa zuwa matsayin UP, kuna buƙatar kunna su:


[root@hp-gen9 ~]# vbmc start control-1
2020-08-14 03:15:57,826.826 13149 INFO VirtualBMC [-] Started vBMC instance for domain control-1
[root@hp-gen9 ~]# vbmc start storage-1 
2020-08-14 03:15:58,316.316 13149 INFO VirtualBMC [-] Started vBMC instance for domain storage-1
[root@hp-gen9 ~]# vbmc start storage-2
2020-08-14 03:15:58,851.851 13149 INFO VirtualBMC [-] Started vBMC instance for domain storage-2
[root@hp-gen9 ~]# vbmc start compute-1
2020-08-14 03:15:59,307.307 13149 INFO VirtualBMC [-] Started vBMC instance for domain compute-1
[root@hp-gen9 ~]# vbmc start compute-2
2020-08-14 03:15:59,712.712 13149 INFO VirtualBMC [-] Started vBMC instance for domain compute-2
[root@hp-gen9 ~]# 
[root@hp-gen9 ~]# 
[root@hp-gen9 ~]# vbmc list
+-------------+---------+---------+------+
| Domain name | Status  | Address | Port |
+-------------+---------+---------+------+
| compute-1   | running | ::      | 7004 |
| compute-2   | running | ::      | 7005 |
| control-1   | running | ::      | 7001 |
| storage-1   | running | ::      | 7002 |
| storage-2   | running | ::      | 7003 |
+-------------+---------+---------+------+
[root@hp-gen9 ~]#

Kuma taɓawa ta ƙarshe - kuna buƙatar gyara ka'idodin Tacewar zaɓi (ko kashe shi gaba ɗaya):


firewall-cmd --zone=public --add-port=7001/udp --permanent
firewall-cmd --zone=public --add-port=7002/udp --permanent
firewall-cmd --zone=public --add-port=7003/udp --permanent
firewall-cmd --zone=public --add-port=7004/udp --permanent
firewall-cmd --zone=public --add-port=7005/udp --permanent
firewall-cmd --reload

Yanzu bari mu je undercloud kuma duba cewa komai yana aiki. Adireshin na'ura mai watsa shiri shine 192.168.255.200, akan undercloud mun kara da buƙatun ipmitool da ake buƙata yayin shirye-shiryen turawa:


[stack@undercloud ~]$ ipmitool -I lanplus -U admin -P admin -H 192.168.255.200 -p 7001 power status          
Chassis Power is off
[stack@undercloud ~]$ ipmitool -I lanplus -U admin -P admin -H 192.168.255.200 -p 7001 power on
Chassis Power Control: Up/On
[stack@undercloud ~]$ 

[root@hp-gen9 ~]# virsh list 
 Id    Name                           State
----------------------------------------------------
 6     dns-server                     running
 64    undercloud                     running
 65    control-1                      running

Kamar yadda kuke gani, mun sami nasarar ƙaddamar da kumburin sarrafawa ta hanyar vbmc. Yanzu bari mu kashe shi mu ci gaba:


[stack@undercloud ~]$ ipmitool -I lanplus -U admin -P admin -H 192.168.255.200 -p 7001 power off
Chassis Power Control: Down/Off
[stack@undercloud ~]$ ipmitool -I lanplus -U admin -P admin -H 192.168.255.200 -p 7001 power status
Chassis Power is off
[stack@undercloud ~]$ 

[root@hp-gen9 ~]# virsh list --all
 Id    Name                           State
----------------------------------------------------
 6     dns-server                     running
 64    undercloud                     running
 -     compute-1                      shut off
 -     compute-2                      shut off
 -     control-1                      shut off
 -     storage-1                      shut off
 -     storage-2                      shut off

[root@hp-gen9 ~]#

Mataki na gaba shine introspection na nodes wanda za'a shigar da overcloud akan su. Don yin wannan, muna buƙatar shirya fayil json tare da bayanin nodes ɗin mu. Lura cewa, sabanin shigarwa akan sabobin da ba su da tushe, fayil ɗin yana nuna tashar tashar da vbmc ke gudana akan kowace na'ura.


[root@hp-gen9 ~]# virsh domiflist --domain control-1 
Interface  Type       Source     Model       MAC
-------------------------------------------------------
-          network    ovs-network-1 virtio      52:54:00:20:a2:2f
-          network    ovs-network-1 virtio      52:54:00:3f:87:9f

[root@hp-gen9 ~]# virsh domiflist --domain compute-1
Interface  Type       Source     Model       MAC
-------------------------------------------------------
-          network    ovs-network-1 virtio      52:54:00:98:e9:d6

[root@hp-gen9 ~]# virsh domiflist --domain compute-2
Interface  Type       Source     Model       MAC
-------------------------------------------------------
-          network    ovs-network-1 virtio      52:54:00:6a:ea:be

[root@hp-gen9 ~]# virsh domiflist --domain storage-1
Interface  Type       Source     Model       MAC
-------------------------------------------------------
-          network    ovs-network-1 virtio      52:54:00:79:0b:cb

[root@hp-gen9 ~]# virsh domiflist --domain storage-2
Interface  Type       Source     Model       MAC
-------------------------------------------------------
-          network    ovs-network-1 virtio      52:54:00:a7:fe:27

Lura: kullin sarrafawa yana da musaya guda biyu, amma a wannan yanayin wannan ba shi da mahimmanci, a cikin wannan shigarwa daya zai ishe mu.

Yanzu muna shirya fayil ɗin json. Muna buƙatar nuna adireshin poppy na tashar jiragen ruwa ta hanyar da za a aiwatar da tanadi, sigogin nodes, ba su sunaye kuma nuna yadda ake zuwa ipmi:


{
    "nodes":[
        {
            "mac":[
                "52:54:00:20:a2:2f"
            ],
            "cpu":"8",
            "memory":"32768",
            "disk":"60",
            "arch":"x86_64",
            "name":"control-1",
            "pm_type":"pxe_ipmitool",
            "pm_user":"admin",
            "pm_password":"admin",
            "pm_addr":"192.168.255.200",
            "pm_port":"7001"
        },
        {
            "mac":[
                "52:54:00:79:0b:cb"
            ],
            "cpu":"4",
            "memory":"16384",
            "disk":"160",
            "arch":"x86_64",
            "name":"storage-1",
            "pm_type":"pxe_ipmitool",
            "pm_user":"admin",
            "pm_password":"admin",
            "pm_addr":"192.168.255.200",
            "pm_port":"7002"
        },
        {
            "mac":[
                "52:54:00:a7:fe:27"
            ],
            "cpu":"4",
            "memory":"16384",
            "disk":"160",
            "arch":"x86_64",
            "name":"storage-2",
            "pm_type":"pxe_ipmitool",
            "pm_user":"admin",
            "pm_password":"admin",
            "pm_addr":"192.168.255.200",
            "pm_port":"7003"
        },
        {
            "mac":[
                "52:54:00:98:e9:d6"
            ],
            "cpu":"12",
            "memory":"32768",
            "disk":"60",
            "arch":"x86_64",
            "name":"compute-1",
            "pm_type":"pxe_ipmitool",
            "pm_user":"admin",
            "pm_password":"admin",
            "pm_addr":"192.168.255.200",
            "pm_port":"7004"
        },
        {
            "mac":[
                "52:54:00:6a:ea:be"
            ],
            "cpu":"12",
            "memory":"32768",
            "disk":"60",
            "arch":"x86_64",
            "name":"compute-2",
            "pm_type":"pxe_ipmitool",
            "pm_user":"admin",
            "pm_password":"admin",
            "pm_addr":"192.168.255.200",
            "pm_port":"7005"
        }
    ]
}

Yanzu muna buƙatar shirya hotuna don ironic. Don yin wannan, zazzage su ta hanyar wget kuma shigar:

(undercloud) [stack@undercloud ~]$ sudo wget https://images.rdoproject.org/queens/delorean/current-tripleo-rdo/overcloud-full.tar --no-check-certificate
(undercloud) [stack@undercloud ~]$ sudo wget https://images.rdoproject.org/queens/delorean/current-tripleo-rdo/ironic-python-agent.tar --no-check-certificate
(undercloud) [stack@undercloud ~]$ ls -lh
total 1.9G
-rw-r--r--. 1 stack stack 447M Aug 14 10:26 ironic-python-agent.tar
-rw-r--r--. 1 stack stack 1.5G Aug 14 10:26 overcloud-full.tar
-rw-------. 1 stack stack  916 Aug 13 23:10 stackrc
-rw-r--r--. 1 stack stack  15K Aug 13 22:50 undercloud.conf
-rw-------. 1 stack stack 2.0K Aug 13 22:50 undercloud-passwords.conf
(undercloud) [stack@undercloud ~]$ mkdir images/
(undercloud) [stack@undercloud ~]$ tar -xpvf ironic-python-agent.tar -C ~/images/
ironic-python-agent.initramfs
ironic-python-agent.kernel
(undercloud) [stack@undercloud ~]$ tar -xpvf overcloud-full.tar -C ~/images/                       
overcloud-full.qcow2
overcloud-full.initrd
overcloud-full.vmlinuz
(undercloud) [stack@undercloud ~]$ 
(undercloud) [stack@undercloud ~]$ ls -lh images/
total 1.9G
-rw-rw-r--. 1 stack stack 441M Aug 12 17:24 ironic-python-agent.initramfs
-rwxr-xr-x. 1 stack stack 6.5M Aug 12 17:24 ironic-python-agent.kernel
-rw-r--r--. 1 stack stack  53M Aug 12 17:14 overcloud-full.initrd
-rw-r--r--. 1 stack stack 1.4G Aug 12 17:18 overcloud-full.qcow2
-rwxr-xr-x. 1 stack stack 6.5M Aug 12 17:14 overcloud-full.vmlinuz
(undercloud) [stack@undercloud ~]$

Ana loda hotuna zuwa undercloud:

(undercloud) [stack@undercloud ~]$ openstack overcloud image upload --image-path ~/images/
Image "overcloud-full-vmlinuz" was uploaded.
+--------------------------------------+------------------------+-------------+---------+--------+
|                  ID                  |          Name          | Disk Format |   Size  | Status |
+--------------------------------------+------------------------+-------------+---------+--------+
| c2553770-3e0f-4750-b46b-138855b5c385 | overcloud-full-vmlinuz |     aki     | 6761064 | active |
+--------------------------------------+------------------------+-------------+---------+--------+
Image "overcloud-full-initrd" was uploaded.
+--------------------------------------+-----------------------+-------------+----------+--------+
|                  ID                  |          Name         | Disk Format |   Size   | Status |
+--------------------------------------+-----------------------+-------------+----------+--------+
| 949984e0-4932-4e71-af43-d67a38c3dc89 | overcloud-full-initrd |     ari     | 55183045 | active |
+--------------------------------------+-----------------------+-------------+----------+--------+
Image "overcloud-full" was uploaded.
+--------------------------------------+----------------+-------------+------------+--------+
|                  ID                  |      Name      | Disk Format |    Size    | Status |
+--------------------------------------+----------------+-------------+------------+--------+
| a2f2096d-c9d7-429a-b866-c7543c02a380 | overcloud-full |    qcow2    | 1487475712 | active |
+--------------------------------------+----------------+-------------+------------+--------+
Image "bm-deploy-kernel" was uploaded.
+--------------------------------------+------------------+-------------+---------+--------+
|                  ID                  |       Name       | Disk Format |   Size  | Status |
+--------------------------------------+------------------+-------------+---------+--------+
| e413aa78-e38f-404c-bbaf-93e582a8e67f | bm-deploy-kernel |     aki     | 6761064 | active |
+--------------------------------------+------------------+-------------+---------+--------+
Image "bm-deploy-ramdisk" was uploaded.
+--------------------------------------+-------------------+-------------+-----------+--------+
|                  ID                  |        Name       | Disk Format |    Size   | Status |
+--------------------------------------+-------------------+-------------+-----------+--------+
| 5cf3aba4-0e50-45d3-929f-27f025dd6ce3 | bm-deploy-ramdisk |     ari     | 461759376 | active |
+--------------------------------------+-------------------+-------------+-----------+--------+
(undercloud) [stack@undercloud ~]$

Dubawa cewa duk hotuna sun loda


(undercloud) [stack@undercloud ~]$  openstack image list
+--------------------------------------+------------------------+--------+
| ID                                   | Name                   | Status |
+--------------------------------------+------------------------+--------+
| e413aa78-e38f-404c-bbaf-93e582a8e67f | bm-deploy-kernel       | active |
| 5cf3aba4-0e50-45d3-929f-27f025dd6ce3 | bm-deploy-ramdisk      | active |
| a2f2096d-c9d7-429a-b866-c7543c02a380 | overcloud-full         | active |
| 949984e0-4932-4e71-af43-d67a38c3dc89 | overcloud-full-initrd  | active |
| c2553770-3e0f-4750-b46b-138855b5c385 | overcloud-full-vmlinuz | active |
+--------------------------------------+------------------------+--------+
(undercloud) [stack@undercloud ~]$

Wani abu guda - kana buƙatar ƙara uwar garken DNS:


(undercloud) [stack@undercloud ~]$ openstack subnet list
+--------------------------------------+-----------------+--------------------------------------+------------------+
| ID                                   | Name            | Network                              | Subnet           |
+--------------------------------------+-----------------+--------------------------------------+------------------+
| f45dea46-4066-42aa-a3c4-6f84b8120cab | ctlplane-subnet | 6ca013dc-41c2-42d8-9d69-542afad53392 | 192.168.255.0/24 |
+--------------------------------------+-----------------+--------------------------------------+------------------+
(undercloud) [stack@undercloud ~]$ openstack subnet show f45dea46-4066-42aa-a3c4-6f84b8120cab
+-------------------+-----------------------------------------------------------+
| Field             | Value                                                     |
+-------------------+-----------------------------------------------------------+
| allocation_pools  | 192.168.255.11-192.168.255.50                             |
| cidr              | 192.168.255.0/24                                          |
| created_at        | 2020-08-13T20:10:37Z                                      |
| description       |                                                           |
| dns_nameservers   |                                                           |
| enable_dhcp       | True                                                      |
| gateway_ip        | 192.168.255.1                                             |
| host_routes       | destination='169.254.169.254/32', gateway='192.168.255.1' |
| id                | f45dea46-4066-42aa-a3c4-6f84b8120cab                      |
| ip_version        | 4                                                         |
| ipv6_address_mode | None                                                      |
| ipv6_ra_mode      | None                                                      |
| name              | ctlplane-subnet                                           |
| network_id        | 6ca013dc-41c2-42d8-9d69-542afad53392                      |
| prefix_length     | None                                                      |
| project_id        | a844ccfcdb2745b198dde3e1b28c40a3                          |
| revision_number   | 0                                                         |
| segment_id        | None                                                      |
| service_types     |                                                           |
| subnetpool_id     | None                                                      |
| tags              |                                                           |
| updated_at        | 2020-08-13T20:10:37Z                                      |
+-------------------+-----------------------------------------------------------+
(undercloud) [stack@undercloud ~]$ 
(undercloud) [stack@undercloud ~]$ neutron subnet-update f45dea46-4066-42aa-a3c4-6f84b8120cab --dns-nameserver 192.168.255.253                                    
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
Updated subnet: f45dea46-4066-42aa-a3c4-6f84b8120cab
(undercloud) [stack@undercloud ~]$

Yanzu za mu iya ba da umarni don introspection:

(undercloud) [stack@undercloud ~]$ openstack overcloud node import --introspect --provide inspection.json 
Started Mistral Workflow tripleo.baremetal.v1.register_or_update. Execution ID: d57456a3-d8ed-479c-9a90-dff7c752d0ec
Waiting for messages on queue 'tripleo' with no timeout.


5 node(s) successfully moved to the "manageable" state.
Successfully registered node UUID b4b2cf4a-b7ca-4095-af13-cc83be21c4f5
Successfully registered node UUID b89a72a3-6bb7-429a-93bc-48393d225838
Successfully registered node UUID 20a16cc0-e0ce-4d88-8f17-eb0ce7b4d69e
Successfully registered node UUID bfc1eb98-a17a-4a70-b0b6-6c0db0eac8e8
Successfully registered node UUID 766ab623-464c-423d-a529-d9afb69d1167
Waiting for introspection to finish...
Started Mistral Workflow tripleo.baremetal.v1.introspect. Execution ID: 6b4d08ae-94c3-4a10-ab63-7634ec198a79
Waiting for messages on queue 'tripleo' with no timeout.
Introspection of node b89a72a3-6bb7-429a-93bc-48393d225838 completed. Status:SUCCESS. Errors:None
Introspection of node 20a16cc0-e0ce-4d88-8f17-eb0ce7b4d69e completed. Status:SUCCESS. Errors:None
Introspection of node bfc1eb98-a17a-4a70-b0b6-6c0db0eac8e8 completed. Status:SUCCESS. Errors:None
Introspection of node 766ab623-464c-423d-a529-d9afb69d1167 completed. Status:SUCCESS. Errors:None
Introspection of node b4b2cf4a-b7ca-4095-af13-cc83be21c4f5 completed. Status:SUCCESS. Errors:None
Successfully introspected 5 node(s).
Started Mistral Workflow tripleo.baremetal.v1.provide. Execution ID: f5594736-edcf-4927-a8a0-2a7bf806a59a
Waiting for messages on queue 'tripleo' with no timeout.
5 node(s) successfully moved to the "available" state.
(undercloud) [stack@undercloud ~]$

Kamar yadda kake gani daga fitarwa, duk abin da aka kammala ba tare da kurakurai ba. Bari mu duba cewa duk nodes suna cikin samuwa:


(undercloud) [stack@undercloud ~]$ openstack baremetal node list
+--------------------------------------+-----------+---------------+-------------+--------------------+-------------+
| UUID                                 | Name      | Instance UUID | Power State | Provisioning State | Maintenance |
+--------------------------------------+-----------+---------------+-------------+--------------------+-------------+
| b4b2cf4a-b7ca-4095-af13-cc83be21c4f5 | control-1 | None          | power off   | available          | False       |
| b89a72a3-6bb7-429a-93bc-48393d225838 | storage-1 | None          | power off   | available          | False       |
| 20a16cc0-e0ce-4d88-8f17-eb0ce7b4d69e | storage-2 | None          | power off   | available          | False       |
| bfc1eb98-a17a-4a70-b0b6-6c0db0eac8e8 | compute-1 | None          | power off   | available          | False       |
| 766ab623-464c-423d-a529-d9afb69d1167 | compute-2 | None          | power off   | available          | False       |
+--------------------------------------+-----------+---------------+-------------+--------------------+-------------+
(undercloud) [stack@undercloud ~]$ 

Idan nodes suna cikin yanayi daban-daban, yawanci ana iya sarrafawa, to, wani abu ya ɓace kuma kuna buƙatar duba log ɗin kuma gano dalilin da yasa hakan ya faru. Ka tuna cewa a cikin wannan yanayin muna amfani da haɓakawa kuma ana iya samun kurakurai masu alaƙa da amfani da injina ko vbmc.

Bayan haka, muna buƙatar nuna ko wane kumburi zai yi aikin - wato, nuna bayanin martaba wanda kumburin zai yi amfani da shi:


(undercloud) [stack@undercloud ~]$ openstack overcloud profiles list
+--------------------------------------+-----------+-----------------+-----------------+-------------------+
| Node UUID                            | Node Name | Provision State | Current Profile | Possible Profiles |
+--------------------------------------+-----------+-----------------+-----------------+-------------------+
| b4b2cf4a-b7ca-4095-af13-cc83be21c4f5 | control-1 | available       | None            |                   |
| b89a72a3-6bb7-429a-93bc-48393d225838 | storage-1 | available       | None            |                   |
| 20a16cc0-e0ce-4d88-8f17-eb0ce7b4d69e | storage-2 | available       | None            |                   |
| bfc1eb98-a17a-4a70-b0b6-6c0db0eac8e8 | compute-1 | available       | None            |                   |
| 766ab623-464c-423d-a529-d9afb69d1167 | compute-2 | available       | None            |                   |
+--------------------------------------+-----------+-----------------+-----------------+-------------------+
(undercloud) [stack@undercloud ~]$ openstack flavor list
+--------------------------------------+---------------+------+------+-----------+-------+-----------+
| ID                                   | Name          |  RAM | Disk | Ephemeral | VCPUs | Is Public |
+--------------------------------------+---------------+------+------+-----------+-------+-----------+
| 168af640-7f40-42c7-91b2-989abc5c5d8f | swift-storage | 4096 |   40 |         0 |     1 | True      |
| 52148d1b-492e-48b4-b5fc-772849dd1b78 | baremetal     | 4096 |   40 |         0 |     1 | True      |
| 56e66542-ae60-416d-863e-0cb192d01b09 | control       | 4096 |   40 |         0 |     1 | True      |
| af6796e1-d0c4-4bfe-898c-532be194f7ac | block-storage | 4096 |   40 |         0 |     1 | True      |
| e4d50fdd-0034-446b-b72c-9da19b16c2df | compute       | 4096 |   40 |         0 |     1 | True      |
| fc2e3acf-7fca-4901-9eee-4a4d6ef0265d | ceph-storage  | 4096 |   40 |         0 |     1 | True      |
+--------------------------------------+---------------+------+------+-----------+-------+-----------+
(undercloud) [stack@undercloud ~]$

Ƙayyade bayanin martaba don kowane kumburi:


openstack baremetal node set --property capabilities='profile:control,boot_option:local' b4b2cf4a-b7ca-4095-af13-cc83be21c4f5
openstack baremetal node set --property capabilities='profile:ceph-storage,boot_option:local' b89a72a3-6bb7-429a-93bc-48393d225838
openstack baremetal node set --property capabilities='profile:ceph-storage,boot_option:local' 20a16cc0-e0ce-4d88-8f17-eb0ce7b4d69e
openstack baremetal node set --property capabilities='profile:compute,boot_option:local' bfc1eb98-a17a-4a70-b0b6-6c0db0eac8e8
openstack baremetal node set --property capabilities='profile:compute,boot_option:local' 766ab623-464c-423d-a529-d9afb69d1167

Bari mu duba cewa mun yi komai daidai:


(undercloud) [stack@undercloud ~]$ openstack overcloud profiles list
+--------------------------------------+-----------+-----------------+-----------------+-------------------+
| Node UUID                            | Node Name | Provision State | Current Profile | Possible Profiles |
+--------------------------------------+-----------+-----------------+-----------------+-------------------+
| b4b2cf4a-b7ca-4095-af13-cc83be21c4f5 | control-1 | available       | control         |                   |
| b89a72a3-6bb7-429a-93bc-48393d225838 | storage-1 | available       | ceph-storage    |                   |
| 20a16cc0-e0ce-4d88-8f17-eb0ce7b4d69e | storage-2 | available       | ceph-storage    |                   |
| bfc1eb98-a17a-4a70-b0b6-6c0db0eac8e8 | compute-1 | available       | compute         |                   |
| 766ab623-464c-423d-a529-d9afb69d1167 | compute-2 | available       | compute         |                   |
+--------------------------------------+-----------+-----------------+-----------------+-------------------+
(undercloud) [stack@undercloud ~]$

Idan komai yayi daidai, muna ba da umarni don tura overcloud:

openstack overcloud deploy --templates --control-scale 1 --compute-scale 2  --ceph-storage-scale 2 --control-flavor control --compute-flavor compute  --ceph-storage-flavor ceph-storage --libvirt-type qemu

A cikin shigarwa na gaske, za a yi amfani da samfuran da aka keɓance ta halitta, a cikin yanayinmu wannan zai rikitar da tsarin sosai, tunda kowane gyara a cikin samfuri dole ne a bayyana shi. Kamar yadda aka rubuta a baya, ko da shigarwa mai sauƙi zai ishe mu mu ga yadda yake aiki.

Lura: madaidaicin --libvirt-type qemu ya zama dole a wannan yanayin, tunda za mu yi amfani da ƙayyadaddun ƙira. In ba haka ba, ba za ku iya tafiyar da injunan kama-da-wane ba.

Yanzu kuna da kusan awa ɗaya, ko wataƙila ƙari (dangane da ƙarfin kayan aikin) kuma kuna iya fatan cewa bayan wannan lokacin zaku ga saƙo mai zuwa:


2020-08-14 08:39:21Z [overcloud]: CREATE_COMPLETE  Stack CREATE completed successfully

 Stack overcloud CREATE_COMPLETE 

Host 192.168.255.21 not found in /home/stack/.ssh/known_hosts
Started Mistral Workflow tripleo.deployment.v1.get_horizon_url. Execution ID: fcb996cd-6a19-482b-b755-2ca0c08069a9
Overcloud Endpoint: http://192.168.255.21:5000/
Overcloud Horizon Dashboard URL: http://192.168.255.21:80/dashboard
Overcloud rc file: /home/stack/overcloudrc
Overcloud Deployed
(undercloud) [stack@undercloud ~]$

Yanzu kuna da kusan cikakkiyar sigar openstack, wanda zaku iya yin nazari, gwaji, da sauransu.

Bari mu duba cewa komai yana aiki yadda ya kamata. A cikin tarihin gidan mai amfani akwai fayiloli guda biyu - stackrc ɗaya (don sarrafa undercloud) da na biyu overcloudrc (don sarrafa overcloud). Dole ne a bayyana waɗannan fayilolin azaman tushen, tunda suna ɗauke da bayanan da ake buƙata don tantancewa.


(undercloud) [stack@undercloud ~]$ openstack server list
+--------------------------------------+-------------------------+--------+-------------------------+----------------+--------------+
| ID                                   | Name                    | Status | Networks                | Image          | Flavor       |
+--------------------------------------+-------------------------+--------+-------------------------+----------------+--------------+
| fd7d36f4-ce87-4b9a-93b0-add2957792de | overcloud-controller-0  | ACTIVE | ctlplane=192.168.255.15 | overcloud-full | control      |
| edc77778-8972-475e-a541-ff40eb944197 | overcloud-novacompute-1 | ACTIVE | ctlplane=192.168.255.26 | overcloud-full | compute      |
| 5448ce01-f05f-47ca-950a-ced14892c0d4 | overcloud-cephstorage-1 | ACTIVE | ctlplane=192.168.255.34 | overcloud-full | ceph-storage |
| ce6d862f-4bdf-4ba3-b711-7217915364d7 | overcloud-novacompute-0 | ACTIVE | ctlplane=192.168.255.19 | overcloud-full | compute      |
| e4507bd5-6f96-4b12-9cc0-6924709da59e | overcloud-cephstorage-0 | ACTIVE | ctlplane=192.168.255.44 | overcloud-full | ceph-storage |
+--------------------------------------+-------------------------+--------+-------------------------+----------------+--------------+
(undercloud) [stack@undercloud ~]$ 


(undercloud) [stack@undercloud ~]$ source overcloudrc 
(overcloud) [stack@undercloud ~]$ 
(overcloud) [stack@undercloud ~]$ openstack project list
+----------------------------------+---------+
| ID                               | Name    |
+----------------------------------+---------+
| 4eed7d0f06544625857d51cd77c5bd4c | admin   |
| ee1c68758bde41eaa9912c81dc67dad8 | service |
+----------------------------------+---------+
(overcloud) [stack@undercloud ~]$ 
(overcloud) [stack@undercloud ~]$ 
(overcloud) [stack@undercloud ~]$ openstack network agent list  
+--------------------------------------+--------------------+-------------------------------------+-------------------+-------+-------+---------------------------+
| ID                                   | Agent Type         | Host                                | Availability Zone | Alive | State | Binary                    |
+--------------------------------------+--------------------+-------------------------------------+-------------------+-------+-------+---------------------------+
| 10495de9-ba4b-41fe-b30a-b90ec3f8728b | Open vSwitch agent | overcloud-novacompute-1.localdomain | None              | :-)   | UP    | neutron-openvswitch-agent |
| 1515ad4a-5972-46c3-af5f-e5446dff7ac7 | L3 agent           | overcloud-controller-0.localdomain  | nova              | :-)   | UP    | neutron-l3-agent          |
| 322e62ca-1e5a-479e-9a96-4f26d09abdd7 | DHCP agent         | overcloud-controller-0.localdomain  | nova              | :-)   | UP    | neutron-dhcp-agent        |
| 9c1de2f9-bac5-400e-998d-4360f04fc533 | Open vSwitch agent | overcloud-novacompute-0.localdomain | None              | :-)   | UP    | neutron-openvswitch-agent |
| d99c5657-851e-4d3c-bef6-f1e3bb1acfb0 | Open vSwitch agent | overcloud-controller-0.localdomain  | None              | :-)   | UP    | neutron-openvswitch-agent |
| ff85fae6-5543-45fb-a301-19c57b62d836 | Metadata agent     | overcloud-controller-0.localdomain  | None              | :-)   | UP    | neutron-metadata-agent    |
+--------------------------------------+--------------------+-------------------------------------+-------------------+-------+-------+---------------------------+
(overcloud) [stack@undercloud ~]$

Shigarwa na har yanzu yana buƙatar ƙaramin taɓawa ɗaya - ƙara hanya akan mai sarrafawa, tunda injin ɗin da nake aiki dashi yana kan hanyar sadarwa daban. Don yin wannan, je zuwa sarrafawa-1 a ƙarƙashin asusun mai kula da zafi kuma yi rajistar hanyar


(undercloud) [stack@undercloud ~]$ ssh [email protected]         
Last login: Fri Aug 14 09:47:40 2020 from 192.168.255.1
[heat-admin@overcloud-controller-0 ~]$ 
[heat-admin@overcloud-controller-0 ~]$ 
[heat-admin@overcloud-controller-0 ~]$ sudo ip route add 10.169.0.0/16 via 192.168.255.254

To, yanzu za ku iya shiga cikin sararin sama. Duk bayanai - adireshi, shiga da kalmar sirri - suna cikin fayil /home/stack/overcloudrc. Zane na ƙarshe yayi kama da haka:

Gabatarwa zuwa ɓangaren cibiyar sadarwa na kayan aikin girgije

Af, a cikin shigarwarmu, an ba da adiresoshin injin ta hanyar DHCP kuma, kamar yadda kuke gani, ana ba da su "a bazuwar". Kuna iya ƙayyadaddun ƙayyadaddun ƙayyadaddun ƙayyadaddun adreshin da ya kamata a haɗa da na'ura yayin turawa, idan kuna buƙata.

Ta yaya zirga-zirga ke gudana tsakanin injunan kama-da-wane?

A cikin wannan labarin za mu dubi zaɓuɓɓuka uku don wucewar zirga-zirga

  • Injin biyu akan hypervisor ɗaya akan hanyar sadarwar L2 ɗaya
  • Injin biyu akan mahaɗan hypervisors daban-daban akan hanyar sadarwar L2 iri ɗaya
  • Injin guda biyu akan cibiyoyin sadarwa daban-daban (ruwan layin hanyar sadarwa)

Abubuwan da ke da damar yin amfani da duniyar waje ta hanyar hanyar sadarwa ta waje, ta amfani da adiresoshin iyo, da kuma rarraba rarrabawa, za mu yi la'akari da lokaci na gaba, don yanzu za mu mayar da hankali kan zirga-zirga na ciki.

Don bincika, bari mu haɗa zane mai zuwa:

Gabatarwa zuwa ɓangaren cibiyar sadarwa na kayan aikin girgije

Mun ƙirƙira injunan kama-da-wane 4 - 3 akan hanyar sadarwar L2 ɗaya - net-1, da ƙari 1 akan hanyar sadarwar-2.

(overcloud) [stack@undercloud ~]$ nova list --tenant 5e18ce8ec9594e00b155485f19895e6c             
+--------------------------------------+------+----------------------------------+--------+------------+-------------+-----------------+
| ID                                   | Name | Tenant ID                        | Status | Task State | Power State | Networks        |
+--------------------------------------+------+----------------------------------+--------+------------+-------------+-----------------+
| f53b37b5-2204-46cc-aef0-dba84bf970c0 | vm-1 | 5e18ce8ec9594e00b155485f19895e6c | ACTIVE | -          | Running     | net-1=10.0.1.85 |
| fc8b6722-0231-49b0-b2fa-041115bef34a | vm-2 | 5e18ce8ec9594e00b155485f19895e6c | ACTIVE | -          | Running     | net-1=10.0.1.88 |
| 3cd74455-b9b7-467a-abe3-bd6ff765c83c | vm-3 | 5e18ce8ec9594e00b155485f19895e6c | ACTIVE | -          | Running     | net-1=10.0.1.90 |
| 7e836338-6772-46b0-9950-f7f06dbe91a8 | vm-4 | 5e18ce8ec9594e00b155485f19895e6c | ACTIVE | -          | Running     | net-2=10.0.2.8  |
+--------------------------------------+------+----------------------------------+--------+------------+-------------+-----------------+
(overcloud) [stack@undercloud ~]$ 

Bari mu ga abin da na'urorin da aka ƙirƙira suke a kai:

(overcloud) [stack@undercloud ~]$ nova show f53b37b5-2204-46cc-aef0-dba84bf970c0 | egrep "hypervisor_hostname|instance_name|hostname"
| OS-EXT-SRV-ATTR:hostname             | vm-1                                                     |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | overcloud-novacompute-0.localdomain                      |
| OS-EXT-SRV-ATTR:instance_name        | instance-00000001                                        |
(overcloud) [stack@undercloud ~]$ nova show fc8b6722-0231-49b0-b2fa-041115bef34a | egrep "hypervisor_hostname|instance_name|hostname"
| OS-EXT-SRV-ATTR:hostname             | vm-2                                                     |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | overcloud-novacompute-1.localdomain                      |
| OS-EXT-SRV-ATTR:instance_name        | instance-00000002                                        |
(overcloud) [stack@undercloud ~]$ nova show 3cd74455-b9b7-467a-abe3-bd6ff765c83c | egrep "hypervisor_hostname|instance_name|hostname"
| OS-EXT-SRV-ATTR:hostname             | vm-3                                                     |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | overcloud-novacompute-0.localdomain                      |
| OS-EXT-SRV-ATTR:instance_name        | instance-00000003                                        |
(overcloud) [stack@undercloud ~]$ nova show 7e836338-6772-46b0-9950-f7f06dbe91a8 | egrep "hypervisor_hostname|instance_name|hostname"
| OS-EXT-SRV-ATTR:hostname             | vm-4                                                     |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | overcloud-novacompute-1.localdomain                      |
| OS-EXT-SRV-ATTR:instance_name        | instance-00000004                                        |

(overcloud) [stack@undercloud ~]$
Injin vm-1 da vm-3 suna kan lissafin-0, inji vm-2 da vm-4 suna kan node compute-1.

Bugu da ƙari, an ƙirƙiri na'ura mai ba da hanya tsakanin hanyoyin sadarwa don ba da damar zirga-zirga tsakanin ƙayyadaddun cibiyoyin sadarwa:

(overcloud) [stack@undercloud ~]$ openstack router list  --project 5e18ce8ec9594e00b155485f19895e6c
+--------------------------------------+----------+--------+-------+-------------+-------+----------------------------------+
| ID                                   | Name     | Status | State | Distributed | HA    | Project                          |
+--------------------------------------+----------+--------+-------+-------------+-------+----------------------------------+
| 0a4d2420-4b9c-46bd-aec1-86a1ef299abe | router-1 | ACTIVE | UP    | False       | False | 5e18ce8ec9594e00b155485f19895e6c |
+--------------------------------------+----------+--------+-------+-------------+-------+----------------------------------+
(overcloud) [stack@undercloud ~]$ 

Mai ba da hanya tsakanin hanyoyin sadarwa yana da tashoshin jiragen ruwa guda biyu, waɗanda ke aiki azaman ƙofofin cibiyoyin sadarwa:

(overcloud) [stack@undercloud ~]$ openstack router show 0a4d2420-4b9c-46bd-aec1-86a1ef299abe | grep interface
| interfaces_info         | [{"subnet_id": "2529ad1a-6b97-49cd-8515-cbdcbe5e3daa", "ip_address": "10.0.1.254", "port_id": "0c52b15f-8fcc-4801-bf52-7dacc72a5201"}, {"subnet_id": "335552dd-b35b-456b-9df0-5aac36a3ca13", "ip_address": "10.0.2.254", "port_id": "92fa49b5-5406-499f-ab8d-ddf28cc1a76c"}] |
(overcloud) [stack@undercloud ~]$ 

Amma kafin mu kalli yadda zirga-zirgar ababen hawa ke gudana, bari mu ga abin da muke da shi a halin yanzu akan kullin sarrafawa (wanda kuma shi ne cibiyar sadarwa) da kuma kan node na lissafi. Bari mu fara da kumburin lissafi.


[heat-admin@overcloud-novacompute-0 ~]$ sudo ovs-vsctl show
[heat-admin@overcloud-novacompute-0 ~]$ sudo sudo ovs-appctl dpif/show
system@ovs-system: hit:3 missed:3
  br-ex:
    br-ex 65534/1: (internal)
    phy-br-ex 1/none: (patch: peer=int-br-ex)
  br-int:
    br-int 65534/2: (internal)
    int-br-ex 1/none: (patch: peer=phy-br-ex)
    patch-tun 2/none: (patch: peer=patch-int)
  br-tun:
    br-tun 65534/3: (internal)
    patch-int 1/none: (patch: peer=patch-tun)
    vxlan-c0a8ff0f 3/4: (vxlan: egress_pkt_mark=0, key=flow, local_ip=192.168.255.19, remote_ip=192.168.255.15)
    vxlan-c0a8ff1a 2/4: (vxlan: egress_pkt_mark=0, key=flow, local_ip=192.168.255.19, remote_ip=192.168.255.26)
[heat-admin@overcloud-novacompute-0 ~]$

A halin yanzu, kumburi yana da gadoji na ovs guda uku - br-int, br-tun, br-ex. Tsakanin su, kamar yadda muke gani, akwai saiti na musaya. Don sauƙin fahimta, bari mu tsara duk waɗannan hanyoyin sadarwa a kan zane kuma mu ga abin da ya faru.

Gabatarwa zuwa ɓangaren cibiyar sadarwa na kayan aikin girgije

Duban adiresoshin da aka tayar da ramukan VxLAN zuwa ga, ana iya ganin cewa an ɗaga rami ɗaya don ƙididdigewa-1 (192.168.255.26), rami na biyu yana kallon sarrafawa-1 (192.168.255.15). Amma abu mafi ban sha'awa shi ne cewa br-ex ba shi da musanya ta jiki, kuma idan ka kalli abin da aka daidaita magudanar ruwa, za ka ga cewa wannan gada kawai za ta iya sauke zirga-zirga a halin yanzu.


[heat-admin@overcloud-novacompute-0 ~]$ ifconfig eth0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 192.168.255.19  netmask 255.255.255.0  broadcast 192.168.255.255
        inet6 fe80::5054:ff:fe6a:eabe  prefixlen 64  scopeid 0x20<link>
        ether 52:54:00:6a:ea:be  txqueuelen 1000  (Ethernet)
        RX packets 2909669  bytes 4608201000 (4.2 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1821057  bytes 349198520 (333.0 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[heat-admin@overcloud-novacompute-0 ~]$ 

Kamar yadda kake gani daga fitarwa, adireshin yana murƙushe kai tsaye zuwa tashar jiragen ruwa na zahiri, kuma ba zuwa gadar gada mai kama-da-wane ba.


[heat-admin@overcloud-novacompute-0 ~]$  sudo ovs-appctl fdb/show br-ex
 port  VLAN  MAC                Age
[heat-admin@overcloud-novacompute-0 ~]$  sudo ovs-ofctl dump-flows br-ex
 cookie=0x9169eae8f7fe5bb2, duration=216686.864s, table=0, n_packets=303, n_bytes=26035, priority=2,in_port="phy-br-ex" actions=drop
 cookie=0x9169eae8f7fe5bb2, duration=216686.887s, table=0, n_packets=0, n_bytes=0, priority=0 actions=NORMAL
[heat-admin@overcloud-novacompute-0 ~]$ 

Bisa ga ka'idar farko, duk abin da ya fito daga tashar phy-br-ex dole ne a watsar da shi.
A haƙiƙa, a halin yanzu babu wani wuri don zirga-zirgar ababen hawa da za su shigo cikin wannan gada sai dai daga wannan ƙa'idar (mai haɗawa tare da br-int), da yin la'akari da faɗuwar, zirga-zirgar BUM ta riga ta shiga cikin gadar.

Wato, zirga-zirga na iya barin wannan kumburin ta hanyar rami na VxLAN kawai kuma ba komai. Koyaya, idan kun kunna DVR, yanayin zai canza, amma zamuyi maganin hakan wani lokaci. Lokacin amfani da keɓewar hanyar sadarwa, misali ta amfani da vlans, ba za ku sami haɗin L3 guda ɗaya a cikin vlan 0 ba, amma musaya masu yawa. Koyaya, zirga-zirgar zirga-zirgar VxLAN za ta bar kullin a cikin hanya ɗaya, amma kuma an sanya shi cikin wani nau'in vlan sadaukarwa.

Mun tsara kullin lissafin, bari mu matsa zuwa kullin sarrafawa.


[heat-admin@overcloud-controller-0 ~]$ sudo ovs-appctl dpif/show
system@ovs-system: hit:930491 missed:825
  br-ex:
    br-ex 65534/1: (internal)
    eth0 1/2: (system)
    phy-br-ex 2/none: (patch: peer=int-br-ex)
  br-int:
    br-int 65534/3: (internal)
    int-br-ex 1/none: (patch: peer=phy-br-ex)
    patch-tun 2/none: (patch: peer=patch-int)
  br-tun:
    br-tun 65534/4: (internal)
    patch-int 1/none: (patch: peer=patch-tun)
    vxlan-c0a8ff13 3/5: (vxlan: egress_pkt_mark=0, key=flow, local_ip=192.168.255.15, remote_ip=192.168.255.19)
    vxlan-c0a8ff1a 2/5: (vxlan: egress_pkt_mark=0, key=flow, local_ip=192.168.255.15, remote_ip=192.168.255.26)
[heat-admin@overcloud-controller-0 ~]$

A zahiri, zamu iya cewa komai iri ɗaya ne, amma adireshin IP ɗin baya kan ƙirar jiki amma akan gadar kama-da-wane. Ana yin haka ne saboda wannan tashar jiragen ruwa ita ce tashar da zirga-zirgar zirga-zirgar jiragen ruwa za ta fita zuwa waje.


[heat-admin@overcloud-controller-0 ~]$ ifconfig br-ex
br-ex: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 192.168.255.15  netmask 255.255.255.0  broadcast 192.168.255.255
        inet6 fe80::5054:ff:fe20:a22f  prefixlen 64  scopeid 0x20<link>
        ether 52:54:00:20:a2:2f  txqueuelen 1000  (Ethernet)
        RX packets 803859  bytes 1732616116 (1.6 GiB)
        RX errors 0  dropped 63  overruns 0  frame 0
        TX packets 808475  bytes 121652156 (116.0 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[heat-admin@overcloud-controller-0 ~]$ 
[heat-admin@overcloud-controller-0 ~]$ sudo ovs-appctl fdb/show br-ex
 port  VLAN  MAC                Age
    3   100  28:c0:da:00:4d:d3   35
    1     0  28:c0:da:00:4d:d3   35
    1     0  52:54:00:98:e9:d6    0
LOCAL     0  52:54:00:20:a2:2f    0
    1     0  52:54:00:2c:08:9e    0
    3   100  52:54:00:20:a2:2f    0
    1     0  52:54:00:6a:ea:be    0
[heat-admin@overcloud-controller-0 ~]$ 

Wannan tashar jiragen ruwa tana daura da gadar br-ex kuma tunda babu vlan tags a kanta, wannan tashar tashar tashar jirgin ruwa ce ta gangar jikin da ake ba da izini ga duk vlans, yanzu zirga-zirgar zirga-zirga tana fita waje ba tare da tag ba, kamar yadda vlan-id 0 ya nuna a ciki. fitarwa a sama.

Gabatarwa zuwa ɓangaren cibiyar sadarwa na kayan aikin girgije

Komai sauran a halin yanzu yana kama da kumburin lissafi - gadoji iri ɗaya, ramuka iri ɗaya zuwa nodes ɗin ƙididdigewa biyu.

Ba za mu yi la'akari da nodes na ajiya a cikin wannan labarin ba, amma don fahimta ya zama dole a ce sashin cibiyar sadarwa na waɗannan nodes banal har zuwa abin kunya. A cikin yanayinmu, akwai tashar jiragen ruwa ta zahiri guda ɗaya (eth0) tare da adireshin IP da aka sanya masa kuma shi ke nan. Babu ramukan VxLAN, gadoji na rami, da sauransu - babu ovs kwata-kwata, tunda babu ma'ana a ciki. Lokacin amfani da keɓewar cibiyar sadarwa, wannan kumburin zai sami musaya guda biyu (tashar jiragen ruwa na zahiri, bodny, ko vlans biyu kawai - ba komai - ya dogara da abin da kuke so) - ɗaya don gudanarwa, na biyu don zirga-zirga (rubutu zuwa diski na VM). , karatu daga faifai, da dai sauransu)

Mun gano abin da muke da shi a kan nodes in babu wani sabis. Yanzu bari mu ƙaddamar da injunan kama-da-wane 4 kuma mu ga yadda tsarin da aka bayyana a sama ya canza - yakamata mu sami tashoshin jiragen ruwa, masu amfani da hanyar sadarwa, da sauransu.

Ya zuwa yanzu hanyar sadarwar mu tana kama da haka:

Gabatarwa zuwa ɓangaren cibiyar sadarwa na kayan aikin girgije

Muna da inji guda biyu akan kowane kumburin kwamfuta. Yin amfani da compute-0 a matsayin misali, bari mu ga yadda aka haɗa komai.


[heat-admin@overcloud-novacompute-0 ~]$ sudo virsh list 
 Id    Name                           State
----------------------------------------------------
 1     instance-00000001              running
 3     instance-00000003              running

[heat-admin@overcloud-novacompute-0 ~]$ 

Injin yana da ƙa'idar kama-da-wane ɗaya kawai - tap95d96a75-a0:

[heat-admin@overcloud-novacompute-0 ~]$ sudo virsh domiflist instance-00000001
Interface  Type       Source     Model       MAC
-------------------------------------------------------
tap95d96a75-a0 bridge     qbr95d96a75-a0 virtio      fa:16:3e:44:98:20

[heat-admin@overcloud-novacompute-0 ~]$ 

Wannan keɓancewa yana kallon gadar Linux:

[heat-admin@overcloud-novacompute-0 ~]$ sudo brctl show
bridge name     bridge id               STP enabled     interfaces
docker0         8000.0242904c92a8       no
qbr5bd37136-47          8000.5e4e05841423       no              qvb5bd37136-47
                                                        tap5bd37136-47
qbr95d96a75-a0          8000.de076cb850f6       no              qvb95d96a75-a0
                                                        tap95d96a75-a0
[heat-admin@overcloud-novacompute-0 ~]$ 

Kamar yadda kuke gani daga fitarwa, akwai musaya guda biyu kawai a cikin gadar - tap95d96a75-a0 da qvb95d96a75-a0.

Anan ya cancanci zama ɗan ɗanɗano akan nau'ikan na'urorin cibiyar sadarwar kama-da-wane a cikin OpenStack:
vtap - ƙirar kama-da-wane da aka haɗe zuwa misali (VM)
qbr - Linux gada
qvb da qvo-vEth biyu da aka haɗa zuwa gadar Linux da Buɗe gadar vSwitch
br-int, br-tun, br-vlan - Buɗe gadoji vSwitch
patch-, int-br-, phy-br- - Buɗe vSwitch facin musaya masu haɗa gadoji
qg, qr, ha, fg, sg - Buɗe vSwitch tashoshin jiragen ruwa waɗanda na'urorin kama-da-wane ke amfani da su don haɗawa da OVS

Kamar yadda kuka fahimta, idan muna da tashar qvb95d96a75-a0 a cikin gada, wanda shine nau'in vEth, to a wani wuri akwai takwaransa, wanda yakamata a kira shi a hankali qvo95d96a75-a0. Bari mu kalli menene tashoshin jiragen ruwa akan OVS.


[heat-admin@overcloud-novacompute-0 ~]$ sudo sudo ovs-appctl dpif/show
system@ovs-system: hit:526 missed:91
  br-ex:
    br-ex 65534/1: (internal)
    phy-br-ex 1/none: (patch: peer=int-br-ex)
  br-int:
    br-int 65534/2: (internal)
    int-br-ex 1/none: (patch: peer=phy-br-ex)
    patch-tun 2/none: (patch: peer=patch-int)
    qvo5bd37136-47 6/6: (system)
    qvo95d96a75-a0 3/5: (system)
  br-tun:
    br-tun 65534/3: (internal)
    patch-int 1/none: (patch: peer=patch-tun)
    vxlan-c0a8ff0f 3/4: (vxlan: egress_pkt_mark=0, key=flow, local_ip=192.168.255.19, remote_ip=192.168.255.15)
    vxlan-c0a8ff1a 2/4: (vxlan: egress_pkt_mark=0, key=flow, local_ip=192.168.255.19, remote_ip=192.168.255.26)
[heat-admin@overcloud-novacompute-0 ~]$ 

Kamar yadda muke iya gani, tashar jiragen ruwa tana cikin br-int. Br-int yana aiki azaman maɓalli wanda ke ƙare tashar jiragen ruwa na inji. Baya ga qvo95d96a75-a0, tashar qvo5bd37136-47 ana iya gani a cikin fitarwa. Wannan ita ce tashar jiragen ruwa zuwa na'ura mai mahimmanci ta biyu. Sakamakon haka, jadawalin mu a yanzu ya yi kama da haka:

Gabatarwa zuwa ɓangaren cibiyar sadarwa na kayan aikin girgije

Tambayar da ya kamata nan da nan sha'awar mai karatu mai hankali - menene gadar Linux tsakanin tashar injin injin da tashar OVS? Gaskiyar ita ce, don kare injin, ana amfani da ƙungiyoyin tsaro, waɗanda ba kome ba ne illa iptables. OVS ba ya aiki tare da iptables, don haka an ƙirƙira wannan “crutch”. Duk da haka, yana zama wanda ba a daina amfani da shi ba - ana maye gurbinsa da sabawa a cikin sababbin sakewa.

Wato, a ƙarshe tsarin ya kasance kamar haka:

Gabatarwa zuwa ɓangaren cibiyar sadarwa na kayan aikin girgije

Injin biyu akan hypervisor ɗaya akan hanyar sadarwar L2 ɗaya

Tunda waɗannan VM guda biyu suna kan hanyar sadarwa ta L2 guda ɗaya kuma akan hypervisor iri ɗaya, zirga-zirgar zirga-zirgar tsakanin su za ta gudana cikin hikima ta cikin gida ta hanyar br-int, tunda duka injin ɗin zasu kasance akan VLAN ɗaya:


[heat-admin@overcloud-novacompute-0 ~]$ sudo virsh domiflist instance-00000001
Interface  Type       Source     Model       MAC
-------------------------------------------------------
tap95d96a75-a0 bridge     qbr95d96a75-a0 virtio      fa:16:3e:44:98:20

[heat-admin@overcloud-novacompute-0 ~]$ 
[heat-admin@overcloud-novacompute-0 ~]$ 
[heat-admin@overcloud-novacompute-0 ~]$ sudo virsh domiflist instance-00000003
Interface  Type       Source     Model       MAC
-------------------------------------------------------
tap5bd37136-47 bridge     qbr5bd37136-47 virtio      fa:16:3e:83:ad:a4

[heat-admin@overcloud-novacompute-0 ~]$ 
[heat-admin@overcloud-novacompute-0 ~]$ sudo ovs-appctl fdb/show br-int 
 port  VLAN  MAC                Age
    6     1  fa:16:3e:83:ad:a4    0
    3     1  fa:16:3e:44:98:20    0
[heat-admin@overcloud-novacompute-0 ~]$ 

Injin biyu akan mahaɗan hypervisors daban-daban akan hanyar sadarwar L2 iri ɗaya

Yanzu bari mu ga yadda zirga-zirgar zirga-zirgar za ta kasance tsakanin injuna guda biyu akan hanyar sadarwar L2 guda ɗaya, amma tana kan masu haɓakawa daban-daban. A gaskiya, babu abin da zai canza da yawa, kawai zirga-zirga tsakanin hypervisors za su bi ta hanyar vxlan. Bari mu kalli misali.

Adireshin injunan kama-da-wane tsakanin waɗanda za mu kalli zirga-zirga:

[heat-admin@overcloud-novacompute-0 ~]$ sudo virsh domiflist instance-00000001
Interface  Type       Source     Model       MAC
-------------------------------------------------------
tap95d96a75-a0 bridge     qbr95d96a75-a0 virtio      fa:16:3e:44:98:20

[heat-admin@overcloud-novacompute-0 ~]$ 


[heat-admin@overcloud-novacompute-1 ~]$ sudo virsh domiflist instance-00000002
Interface  Type       Source     Model       MAC
-------------------------------------------------------
tape7e23f1b-07 bridge     qbre7e23f1b-07 virtio      fa:16:3e:72:ad:53

[heat-admin@overcloud-novacompute-1 ~]$ 

Muna kallon teburin turawa a cikin br-int akan compute-0:

[heat-admin@overcloud-novacompute-0 ~]$  sudo ovs-appctl fdb/show br-int | grep fa:16:3e:72:ad:53
    2     1  fa:16:3e:72:ad:53    1
[heat-admin@overcloud-novacompute-0 ~]

Ya kamata zirga-zirga zuwa tashar jiragen ruwa 2 - bari mu ga irin tashar tashar jiragen ruwa:

[heat-admin@overcloud-novacompute-0 ~]$ sudo ovs-ofctl show br-int | grep addr
 1(int-br-ex): addr:7e:7f:28:1f:bd:54
 2(patch-tun): addr:0a:bd:07:69:58:d9
 3(qvo95d96a75-a0): addr:ea:50:9a:3d:69:58
 6(qvo5bd37136-47): addr:9a:d1:03:50:3d:96
 LOCAL(br-int): addr:1a:0f:53:97:b1:49
[heat-admin@overcloud-novacompute-0 ~]$

Wannan shi ne patch-tun - wato, abin dubawa a cikin br-tun. Bari mu ga abin da ya faru da kunshin akan br-tun:

[heat-admin@overcloud-novacompute-0 ~]$ sudo ovs-ofctl dump-flows br-tun | grep fa:16:3e:72:ad:53
 cookie=0x8759a56536b67a8e, duration=1387.959s, table=20, n_packets=1460, n_bytes=138880, hard_timeout=300, idle_age=0, hard_age=0, priority=1,vlan_tci=0x0001/0x0fff,dl_dst=fa:16:3e:72:ad:53 actions=load:0->NXM_OF_VLAN_TCI[],load:0x16->NXM_NX_TUN_ID[],output:2
[heat-admin@overcloud-novacompute-0 ~]$ 

An shirya fakitin a cikin VxLAN kuma an aika zuwa tashar jiragen ruwa 2. Bari mu ga inda tashar jiragen ruwa 2 ke kaiwa:

[heat-admin@overcloud-novacompute-0 ~]$ sudo ovs-ofctl show br-tun | grep addr   
 1(patch-int): addr:b2:d1:f8:21:96:66
 2(vxlan-c0a8ff1a): addr:be:64:1f:75:78:a7
 3(vxlan-c0a8ff0f): addr:76:6f:b9:3c:3f:1c
 LOCAL(br-tun): addr:a2:5b:6d:4f:94:47
[heat-admin@overcloud-novacompute-0 ~]$

Wannan rami ne na vxlan akan lissafin-1:

[heat-admin@overcloud-novacompute-0 ~]$ sudo ovs-appctl dpif/show | egrep vxlan-c0a8ff1a
    vxlan-c0a8ff1a 2/4: (vxlan: egress_pkt_mark=0, key=flow, local_ip=192.168.255.19, remote_ip=192.168.255.26)
[heat-admin@overcloud-novacompute-0 ~]$

Mu je lissafin-1 mu ga abin da zai biyo baya tare da kunshin:

[heat-admin@overcloud-novacompute-1 ~]$ sudo ovs-appctl fdb/show br-int | egrep fa:16:3e:44:98:20
    2     1  fa:16:3e:44:98:20    1
[heat-admin@overcloud-novacompute-1 ~]$ 

Mac yana cikin tebur isarwa na br-int akan lissafin-1, kuma kamar yadda ake iya gani daga fitarwar da ke sama, ana iya gani ta hanyar tashar jiragen ruwa 2, wacce ita ce tashar jiragen ruwa zuwa br-tun:

[heat-admin@overcloud-novacompute-1 ~]$ sudo ovs-ofctl show br-int | grep addr   
 1(int-br-ex): addr:8a:d7:f9:ad:8c:1d
 2(patch-tun): addr:46:cc:40:bd:20:da
 3(qvoe7e23f1b-07): addr:12:78:2e:34:6a:c7
 4(qvo3210e8ec-c0): addr:7a:5f:59:75:40:85
 LOCAL(br-int): addr:e2:27:b2:ed:14:46

Da kyau, sannan mun ga cewa a cikin br-int akan compute-1 akwai poppy manufa:

[heat-admin@overcloud-novacompute-1 ~]$ sudo ovs-appctl fdb/show br-int | egrep fa:16:3e:72:ad:53
    3     1  fa:16:3e:72:ad:53    0
[heat-admin@overcloud-novacompute-1 ~]$ 

Wato fakitin da aka karɓa zai tashi zuwa tashar jiragen ruwa 3, wanda a bayansa akwai misalin injin kama-da-wane-00000003.

Kyawawan tura Opentack don koyo akan ababen more rayuwa shine cewa zamu iya ɗaukar zirga-zirga cikin sauƙi tsakanin masu haɓakawa mu ga abin da ke faruwa da shi. Wannan shine abin da za mu yi yanzu, gudanar da tcpdump akan tashar vnet zuwa lissafin-0:


[root@hp-gen9 bormoglotx]# tcpdump -vvv -i vnet3
tcpdump: listening on vnet3, link-type EN10MB (Ethernet), capture size 262144 bytes

*****************omitted*******************

04:39:04.583459 IP (tos 0x0, ttl 64, id 16868, offset 0, flags [DF], proto UDP (17), length 134)
    192.168.255.19.39096 > 192.168.255.26.4789: [no cksum] VXLAN, flags [I] (0x08), vni 22
IP (tos 0x0, ttl 64, id 8012, offset 0, flags [DF], proto ICMP (1), length 84)
    10.0.1.85 > 10.0.1.88: ICMP echo request, id 5634, seq 16, length 64
04:39:04.584449 IP (tos 0x0, ttl 64, id 35181, offset 0, flags [DF], proto UDP (17), length 134)
    192.168.255.26.speedtrace-disc > 192.168.255.19.4789: [no cksum] VXLAN, flags [I] (0x08), vni 22
IP (tos 0x0, ttl 64, id 59124, offset 0, flags [none], proto ICMP (1), length 84)
    10.0.1.88 > 10.0.1.85: ICMP echo reply, id 5634, seq 16, length 64
	
*****************omitted*******************

Layi na farko ya nuna cewa Patek daga adireshin 10.0.1.85 yana zuwa adireshin 10.0.1.88 (cibiyar zirga-zirgar ICMP), kuma an nannade shi a cikin fakitin VxLAN tare da vni 22 kuma fakitin yana fitowa daga mai masaukin baki 192.168.255.19 (compute-0) don karbar bakuncin 192.168.255.26 .1 (lissafi-XNUMX). Za mu iya duba cewa VNI ta yi daidai da wanda aka kayyade a ovs.

Mu koma wannan layin ayyuka=load:0->NXM_OF_VLAN_TCI[],load:0x16->NXM_NX_TUN_ID[],fitarwa:2. 0x16 shine vni a tsarin lambar hexadecimal. Mu canza wannan lamba zuwa tsarin na 16:


16 = 6*16^0+1*16^1 = 6+16 = 22

Wato vni yayi daidai da gaskiya.

Layi na biyu yana nuna zirga-zirgar dawowa, da kyau, babu ma'ana a bayyana shi, komai a bayyane yake a can.

Injin guda biyu akan hanyoyin sadarwa daban-daban (gudanar da hanyar sadarwa)

Shari'ar ƙarshe ta yau ita ce zazzagewa tsakanin cibiyoyin sadarwa a cikin aiki ɗaya ta amfani da na'ura mai ba da hanya tsakanin hanyoyin sadarwa. Muna la'akari da shari'ar ba tare da DVR ba (za mu duba shi a cikin wani labarin), don haka hanyar sadarwa yana faruwa a kan kullin cibiyar sadarwa. A cikin yanayinmu, ba a sanya kullin cibiyar sadarwa a cikin wani yanki daban kuma yana kan kullin sarrafawa.

Da farko, bari mu ga cewa routing yana aiki:

$ ping 10.0.2.8
PING 10.0.2.8 (10.0.2.8): 56 data bytes
64 bytes from 10.0.2.8: seq=0 ttl=63 time=7.727 ms
64 bytes from 10.0.2.8: seq=1 ttl=63 time=3.832 ms
^C
--- 10.0.2.8 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 3.832/5.779/7.727 ms

Tun da yake a cikin wannan yanayin dole ne fakitin ya tafi ƙofar kuma a bi shi a can, muna buƙatar gano adireshin poppy na ƙofar, wanda muke kallon teburin ARP a cikin misali:

$ arp
host-10-0-1-254.openstacklocal (10.0.1.254) at fa:16:3e:c4:64:70 [ether]  on eth0
host-10-0-1-1.openstacklocal (10.0.1.1) at fa:16:3e:e6:2c:5c [ether]  on eth0
host-10-0-1-90.openstacklocal (10.0.1.90) at fa:16:3e:83:ad:a4 [ether]  on eth0
host-10-0-1-88.openstacklocal (10.0.1.88) at fa:16:3e:72:ad:53 [ether]  on eth0

Yanzu bari mu ga inda ya kamata a aika da zirga-zirga tare da manufa (10.0.1.254) fa:16:3e:c4:64:70:

[heat-admin@overcloud-novacompute-0 ~]$ sudo ovs-appctl fdb/show br-int | egrep fa:16:3e:c4:64:70
    2     1  fa:16:3e:c4:64:70    0
[heat-admin@overcloud-novacompute-0 ~]$ 

Bari mu kalli inda tashar jiragen ruwa 2 ke kaiwa:

[heat-admin@overcloud-novacompute-0 ~]$ sudo ovs-ofctl show br-int | grep addr
 1(int-br-ex): addr:7e:7f:28:1f:bd:54
 2(patch-tun): addr:0a:bd:07:69:58:d9
 3(qvo95d96a75-a0): addr:ea:50:9a:3d:69:58
 6(qvo5bd37136-47): addr:9a:d1:03:50:3d:96
 LOCAL(br-int): addr:1a:0f:53:97:b1:49
[heat-admin@overcloud-novacompute-0 ~]$ 

Komai yana da ma'ana, zirga-zirga yana zuwa br-tun. Bari mu ga wane rami vxlan za a nannade shi:

[heat-admin@overcloud-novacompute-0 ~]$ sudo ovs-ofctl dump-flows br-tun | grep fa:16:3e:c4:64:70
 cookie=0x8759a56536b67a8e, duration=3514.566s, table=20, n_packets=3368, n_bytes=317072, hard_timeout=300, idle_age=0, hard_age=0, priority=1,vlan_tci=0x0001/0x0fff,dl_dst=fa:16:3e:c4:64:70 actions=load:0->NXM_OF_VLAN_TCI[],load:0x16->NXM_NX_TUN_ID[],output:3
[heat-admin@overcloud-novacompute-0 ~]$ 

Tashar ruwa ta uku ita ce rami vxlan:

[heat-admin@overcloud-controller-0 ~]$ sudo ovs-ofctl show br-tun | grep addr
 1(patch-int): addr:a2:69:00:c5:fa:ba
 2(vxlan-c0a8ff1a): addr:86:f0:ce:d0:e8:ea
 3(vxlan-c0a8ff13): addr:72:aa:73:2c:2e:5b
 LOCAL(br-tun): addr:a6:cb:cd:72:1c:45
[heat-admin@overcloud-controller-0 ~]$ 

Wanda ke kallon kumburin sarrafawa:

[heat-admin@overcloud-controller-0 ~]$ sudo sudo ovs-appctl dpif/show | grep vxlan-c0a8ff1a
    vxlan-c0a8ff1a 2/5: (vxlan: egress_pkt_mark=0, key=flow, local_ip=192.168.255.15, remote_ip=192.168.255.26)
[heat-admin@overcloud-controller-0 ~]$ 

Hanyoyin zirga-zirga sun kai ga kumburin sarrafawa, don haka muna buƙatar zuwa gare ta mu ga yadda za a yi ta hanya.

Kamar yadda kuke tunawa, kullin sarrafawa a ciki yayi kama da kullin lissafi - gadoji guda uku, kawai br-ex yana da tashar jiragen ruwa ta jiki wanda kumburin zai iya aika zirga-zirga a waje. Ƙirƙirar misalai sun canza saitin akan nodes na lissafin - gada Linux, iptables da musaya an ƙara su zuwa nodes. Ƙirƙirar cibiyoyin sadarwa da na'ura mai ba da hanya tsakanin hanyoyin sadarwa kuma sun bar alamar sa akan daidaita kullin sarrafawa.

Don haka, a bayyane yake cewa adireshin MAC ɗin ƙofar dole ne ya kasance a cikin tebur ɗin isar da br-int akan kumburin sarrafawa. Bari mu duba cewa yana can da kuma inda yake kallo:

[heat-admin@overcloud-controller-0 ~]$ sudo ovs-appctl fdb/show br-int | grep fa:16:3e:c4:64:70
    5     1  fa:16:3e:c4:64:70    1
[heat-admin@overcloud-controller-0 ~]$ 
[heat-admin@overcloud-controller-0 ~]$  sudo ovs-ofctl show br-int | grep addr
 1(int-br-ex): addr:2e:58:b6:db:d5:de
 2(patch-tun): addr:06:41:90:f0:9e:56
 3(tapca25a97e-64): addr:fa:16:3e:e6:2c:5c
 4(tap22015e46-0b): addr:fa:16:3e:76:c2:11
 5(qr-0c52b15f-8f): addr:fa:16:3e:c4:64:70
 6(qr-92fa49b5-54): addr:fa:16:3e:80:13:72
 LOCAL(br-int): addr:06:de:5d:ed:44:44
[heat-admin@overcloud-controller-0 ~]$ 

Ana iya ganin Mac daga tashar jiragen ruwa qr-0c52b15f-8f. Idan muka koma cikin jerin tashoshin jiragen ruwa masu kama-da-wane a cikin Openstack, ana amfani da irin wannan nau'in tashar don haɗa na'urori masu kama da juna zuwa OVS. Don zama madaidaici, qr tashar jiragen ruwa ce zuwa ga mai amfani da hanyar sadarwa, wanda aka wakilta azaman filin suna.

Bari mu ga mene ne wuraren suna akan sabar:

[heat-admin@overcloud-controller-0 ~]$ sudo  ip netns
qrouter-0a4d2420-4b9c-46bd-aec1-86a1ef299abe (id: 2)
qdhcp-7d541e74-1c36-4e1d-a7c4-0968c8dbc638 (id: 1)
qdhcp-67a3798c-32c0-4c18-8502-2531247e3cc2 (id: 0)
[heat-admin@overcloud-controller-0 ~]$ 

Kamar kwafi uku. Amma idan aka yi la'akari da sunayen, za ku iya gane manufar kowannensu. Za mu koma ga misalai tare da ID 0 da 1 daga baya, yanzu muna sha'awar sararin suna qrouter-0a4d2420-4b9c-46bd-aec1-86a1ef299abe:


[heat-admin@overcloud-controller-0 ~]$ sudo  ip netns exec qrouter-0a4d2420-4b9c-46bd-aec1-86a1ef299abe ip route
10.0.1.0/24 dev qr-0c52b15f-8f proto kernel scope link src 10.0.1.254 
10.0.2.0/24 dev qr-92fa49b5-54 proto kernel scope link src 10.0.2.254 
[heat-admin@overcloud-controller-0 ~]$ 

Wannan filin suna ya ƙunshi na ciki guda biyu waɗanda muka ƙirƙira a baya. Dukansu tashoshin jiragen ruwa na zahiri an ƙara su zuwa br-int. Bari mu duba Mac address na tashar jiragen ruwa qr-0c52b15f-8f, tun da zirga-zirga, yin hukunci da makõma Mac address, tafi zuwa wannan dubawa.

[heat-admin@overcloud-controller-0 ~]$ sudo  ip netns exec qrouter-0a4d2420-4b9c-46bd-aec1-86a1ef299abe ifconfig qr-0c52b15f-8f
qr-0c52b15f-8f: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 10.0.1.254  netmask 255.255.255.0  broadcast 10.0.1.255
        inet6 fe80::f816:3eff:fec4:6470  prefixlen 64  scopeid 0x20<link>
        ether fa:16:3e:c4:64:70  txqueuelen 1000  (Ethernet)
        RX packets 5356  bytes 427305 (417.2 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 5195  bytes 490603 (479.1 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[heat-admin@overcloud-controller-0 ~]$ 

Wato, a wannan yanayin, duk abin da ke aiki bisa ga ka'idodin daidaitaccen hanya. Tunda zirga-zirgar zirga-zirgar ababen hawa an ƙaddara don mai masaukin baki 10.0.2.8, dole ne ta fita ta hanyar dubawa ta biyu qr-92fa49b5-54 kuma ta shiga rami vxlan zuwa kullin ƙididdigewa:


[heat-admin@overcloud-controller-0 ~]$ sudo  ip netns exec qrouter-0a4d2420-4b9c-46bd-aec1-86a1ef299abe arp
Address                  HWtype  HWaddress           Flags Mask            Iface
10.0.1.88                ether   fa:16:3e:72:ad:53   C                     qr-0c52b15f-8f
10.0.1.90                ether   fa:16:3e:83:ad:a4   C                     qr-0c52b15f-8f
10.0.2.8                 ether   fa:16:3e:6c:ad:9c   C                     qr-92fa49b5-54
10.0.2.42                ether   fa:16:3e:f5:0b:29   C                     qr-92fa49b5-54
10.0.1.85                ether   fa:16:3e:44:98:20   C                     qr-0c52b15f-8f
[heat-admin@overcloud-controller-0 ~]$ 

Komai yana da ma'ana, ba abin mamaki ba. Bari mu ga inda adireshin poppy na rundunar 10.0.2.8 ke bayyane a cikin br-int:

[heat-admin@overcloud-controller-0 ~]$ sudo ovs-appctl fdb/show br-int | grep fa:16:3e:6c:ad:9c
    2     2  fa:16:3e:6c:ad:9c    1
[heat-admin@overcloud-controller-0 ~]$ 
[heat-admin@overcloud-controller-0 ~]$ sudo ovs-ofctl show br-int | grep addr
 1(int-br-ex): addr:2e:58:b6:db:d5:de
 2(patch-tun): addr:06:41:90:f0:9e:56
 3(tapca25a97e-64): addr:fa:16:3e:e6:2c:5c
 4(tap22015e46-0b): addr:fa:16:3e:76:c2:11
 5(qr-0c52b15f-8f): addr:fa:16:3e:c4:64:70
 6(qr-92fa49b5-54): addr:fa:16:3e:80:13:72
 LOCAL(br-int): addr:06:de:5d:ed:44:44
[heat-admin@overcloud-controller-0 ~]$ 

Kamar yadda ake tsammani, zirga-zirga yana zuwa br-tun, bari mu ga ko wane rami ne zirga-zirgar ke zuwa na gaba:

[heat-admin@overcloud-controller-0 ~]$ sudo ovs-ofctl dump-flows br-tun | grep fa:16:3e:6c:ad:9c
 cookie=0x2ab04bf27114410e, duration=5346.829s, table=20, n_packets=5248, n_bytes=498512, hard_timeout=300, idle_age=0, hard_age=0, priority=1,vlan_tci=0x0002/0x0fff,dl_dst=fa:16:3e:6c:ad:9c actions=load:0->NXM_OF_VLAN_TCI[],load:0x63->NXM_NX_TUN_ID[],output:2
[heat-admin@overcloud-controller-0 ~]$
[heat-admin@overcloud-controller-0 ~]$ sudo ovs-ofctl show br-tun | grep addr
 1(patch-int): addr:a2:69:00:c5:fa:ba
 2(vxlan-c0a8ff1a): addr:86:f0:ce:d0:e8:ea
 3(vxlan-c0a8ff13): addr:72:aa:73:2c:2e:5b
 LOCAL(br-tun): addr:a6:cb:cd:72:1c:45
[heat-admin@overcloud-controller-0 ~]$ 
[heat-admin@overcloud-controller-0 ~]$ sudo sudo ovs-appctl dpif/show | grep vxlan-c0a8ff1a
    vxlan-c0a8ff1a 2/5: (vxlan: egress_pkt_mark=0, key=flow, local_ip=192.168.255.15, remote_ip=192.168.255.26)
[heat-admin@overcloud-controller-0 ~]$ 

Motoci suna shiga cikin rami don lissafta-1. Da kyau, akan lissafin-1 komai yana da sauƙi - daga br-tun kunshin yana zuwa br-int kuma daga can zuwa ƙirar injin kama-da-wane:

[heat-admin@overcloud-controller-0 ~]$ sudo sudo ovs-appctl dpif/show | grep vxlan-c0a8ff1a
    vxlan-c0a8ff1a 2/5: (vxlan: egress_pkt_mark=0, key=flow, local_ip=192.168.255.15, remote_ip=192.168.255.26)
[heat-admin@overcloud-controller-0 ~]$ 
[heat-admin@overcloud-novacompute-1 ~]$ sudo ovs-appctl fdb/show br-int | grep fa:16:3e:6c:ad:9c
    4     2  fa:16:3e:6c:ad:9c    1
[heat-admin@overcloud-novacompute-1 ~]$ sudo ovs-ofctl show br-int | grep addr                  
 1(int-br-ex): addr:8a:d7:f9:ad:8c:1d
 2(patch-tun): addr:46:cc:40:bd:20:da
 3(qvoe7e23f1b-07): addr:12:78:2e:34:6a:c7
 4(qvo3210e8ec-c0): addr:7a:5f:59:75:40:85
 LOCAL(br-int): addr:e2:27:b2:ed:14:46
[heat-admin@overcloud-novacompute-1 ~]$ 

Bari mu duba cewa lallai wannan shine madaidaicin dubawa:

[heat-admin@overcloud-novacompute-1 ~]$ brctl show
bridge name     bridge id               STP enabled     interfaces
docker0         8000.02429c001e1c       no
qbr3210e8ec-c0          8000.ea27f45358be       no              qvb3210e8ec-c0
                                                        tap3210e8ec-c0
qbre7e23f1b-07          8000.b26ac0eded8a       no              qvbe7e23f1b-07
                                                        tape7e23f1b-07
[heat-admin@overcloud-novacompute-1 ~]$ 
[heat-admin@overcloud-novacompute-1 ~]$ sudo virsh domiflist instance-00000004
Interface  Type       Source     Model       MAC
-------------------------------------------------------
tap3210e8ec-c0 bridge     qbr3210e8ec-c0 virtio      fa:16:3e:6c:ad:9c

[heat-admin@overcloud-novacompute-1 ~]$

A gaskiya, mun bi ta cikin kunshin. Ina tsammanin kun lura cewa zirga-zirgar zirga-zirgar zirga-zirgar zirga-zirgar zirga-zirgar zirga-zirgar zirga-zirgar zirga-zirgar zirga-zirgar zirga-zirgar zirga-zirgar zirga-zirgar zirga-zirgar zirga-zirgar ababen hawa ta bi ta hanyoyi daban-daban na vxlan kuma ta fita da VNI daban-daban. Bari mu ga wane nau'in VNI ne, bayan haka za mu tattara juji a kan tashar sarrafawa ta kumburi kuma mu tabbatar da cewa zirga-zirgar zirga-zirgar zirga-zirgar ta gudana daidai kamar yadda aka bayyana a sama.
Don haka, rami don ƙididdige-0 yana da ayyuka masu zuwa = lodi: 0->NXM_OF_VLAN_TCI[], lodi: 0x16-> NXM_NX_TUN_ID[], fitarwa:3. Bari mu canza 0x16 zuwa tsarin lambar decimal:


0x16 = 6*16^0+1*16^1 = 6+16 = 22

Ramin don ƙididdige-1 yana da VNI mai zuwa: ayyuka = ​​kaya: 0->NXM_OF_VLAN_TCI[], load: 0x63->NXM_NX_TUN_ID[], fitarwa:2. Bari mu canza 0x63 zuwa tsarin lambar decimal:


0x63 = 3*16^0+6*16^1 = 3+96 = 99

To, yanzu bari mu dubi juji:

[root@hp-gen9 bormoglotx]# tcpdump -vvv -i vnet4 
tcpdump: listening on vnet4, link-type EN10MB (Ethernet), capture size 262144 bytes

*****************omitted*******************

04:35:18.709949 IP (tos 0x0, ttl 64, id 48650, offset 0, flags [DF], proto UDP (17), length 134)
    192.168.255.19.41591 > 192.168.255.15.4789: [no cksum] VXLAN, flags [I] (0x08), vni 22
IP (tos 0x0, ttl 64, id 49042, offset 0, flags [DF], proto ICMP (1), length 84)
    10.0.1.85 > 10.0.2.8: ICMP echo request, id 5378, seq 9, length 64
04:35:18.710159 IP (tos 0x0, ttl 64, id 23360, offset 0, flags [DF], proto UDP (17), length 134)
    192.168.255.15.38983 > 192.168.255.26.4789: [no cksum] VXLAN, flags [I] (0x08), vni 99
IP (tos 0x0, ttl 63, id 49042, offset 0, flags [DF], proto ICMP (1), length 84)
    10.0.1.85 > 10.0.2.8: ICMP echo request, id 5378, seq 9, length 64
04:35:18.711292 IP (tos 0x0, ttl 64, id 43596, offset 0, flags [DF], proto UDP (17), length 134)
    192.168.255.26.42588 > 192.168.255.15.4789: [no cksum] VXLAN, flags [I] (0x08), vni 99
IP (tos 0x0, ttl 64, id 55103, offset 0, flags [none], proto ICMP (1), length 84)
    10.0.2.8 > 10.0.1.85: ICMP echo reply, id 5378, seq 9, length 64
04:35:18.711531 IP (tos 0x0, ttl 64, id 8555, offset 0, flags [DF], proto UDP (17), length 134)
    192.168.255.15.38983 > 192.168.255.19.4789: [no cksum] VXLAN, flags [I] (0x08), vni 22
IP (tos 0x0, ttl 63, id 55103, offset 0, flags [none], proto ICMP (1), length 84)
    10.0.2.8 > 10.0.1.85: ICMP echo reply, id 5378, seq 9, length 64
	
*****************omitted*******************

Fakitin farko shine fakitin vxlan daga mai masaukin baki 192.168.255.19 (compute-0) don karbar bakuncin 192.168.255.15 (control-1) tare da vni 22, a ciki wanda aka kunshi fakitin ICMP daga mai masaukin baki 10.0.1.85 don karbar bakuncin 10.0.2.8. Kamar yadda muka lissafta a sama, vni yayi daidai da abin da muka gani a cikin fitarwa.

Fakiti na biyu shine fakitin vxlan daga mai masaukin baki 192.168.255.15 (control-1) don karbar bakuncin 192.168.255.26 (compute-1) tare da vni 99, a ciki wanda aka kunshi fakitin ICMP daga mai masaukin baki 10.0.1.85 don karbar bakuncin 10.0.2.8. Kamar yadda muka lissafta a sama, vni yayi daidai da abin da muka gani a cikin fitarwa.

Fakiti biyu na gaba sune dawowar zirga-zirga daga 10.0.2.8 ba 10.0.1.85 ba.

Wato, a ƙarshe mun sami tsarin kumburin sarrafawa mai zuwa:

Gabatarwa zuwa ɓangaren cibiyar sadarwa na kayan aikin girgije

Kalli shi ke nan? Mun manta game da wuraren sunaye guda biyu:

[heat-admin@overcloud-controller-0 ~]$ sudo  ip netns
qrouter-0a4d2420-4b9c-46bd-aec1-86a1ef299abe (id: 2)
qdhcp-7d541e74-1c36-4e1d-a7c4-0968c8dbc638 (id: 1)
qdhcp-67a3798c-32c0-4c18-8502-2531247e3cc2 (id: 0)
[heat-admin@overcloud-controller-0 ~]$ 

Kamar yadda muka yi magana game da gine-gine na dandalin girgije, zai yi kyau idan inji sun karbi adireshi ta atomatik daga uwar garken DHCP. Waɗannan sabar DHCP guda biyu ce don hanyoyin sadarwar mu guda biyu 10.0.1.0/24 da 10.0.2.0/24.

Mu duba cewa wannan gaskiya ne. Akwai adireshi ɗaya kawai a cikin wannan filin suna - 10.0.1.1 - adireshin uwar garken DHCP kanta, kuma an haɗa shi cikin br-int:

[heat-admin@overcloud-controller-0 ~]$ sudo ip netns exec qdhcp-67a3798c-32c0-4c18-8502-2531247e3cc2 ifconfig
lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 1  bytes 28 (28.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1  bytes 28 (28.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

tapca25a97e-64: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 10.0.1.1  netmask 255.255.255.0  broadcast 10.0.1.255
        inet6 fe80::f816:3eff:fee6:2c5c  prefixlen 64  scopeid 0x20<link>
        ether fa:16:3e:e6:2c:5c  txqueuelen 1000  (Ethernet)
        RX packets 129  bytes 9372 (9.1 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 49  bytes 6154 (6.0 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Bari mu ga ko matakan da suka ƙunshi qdhcp-67a3798c-32c0-4c18-8502-2531247e3cc2 a cikin sunansu akan kullin sarrafawa:


[heat-admin@overcloud-controller-0 ~]$ ps -aux | egrep qdhcp-7d541e74-1c36-4e1d-a7c4-0968c8dbc638 
root      640420  0.0  0.0   4220   348 ?        Ss   11:31   0:00 dumb-init --single-child -- ip netns exec qdhcp-7d541e74-1c36-4e1d-a7c4-0968c8dbc638 /usr/sbin/dnsmasq -k --no-hosts --no-resolv --pid-file=/var/lib/neutron/dhcp/7d541e74-1c36-4e1d-a7c4-0968c8dbc638/pid --dhcp-hostsfile=/var/lib/neutron/dhcp/7d541e74-1c36-4e1d-a7c4-0968c8dbc638/host --addn-hosts=/var/lib/neutron/dhcp/7d541e74-1c36-4e1d-a7c4-0968c8dbc638/addn_hosts --dhcp-optsfile=/var/lib/neutron/dhcp/7d541e74-1c36-4e1d-a7c4-0968c8dbc638/opts --dhcp-leasefile=/var/lib/neutron/dhcp/7d541e74-1c36-4e1d-a7c4-0968c8dbc638/leases --dhcp-match=set:ipxe,175 --local-service --bind-dynamic --dhcp-range=set:subnet-335552dd-b35b-456b-9df0-5aac36a3ca13,10.0.2.0,static,255.255.255.0,86400s --dhcp-option-force=option:mtu,1450 --dhcp-lease-max=256 --conf-file= --domain=openstacklocal
heat-ad+  951620  0.0  0.0 112944   980 pts/0    S+   18:50   0:00 grep -E --color=auto qdhcp-7d541e74-1c36-4e1d-a7c4-0968c8dbc638
[heat-admin@overcloud-controller-0 ~]$ 

Akwai irin wannan tsari kuma bisa ga bayanin da aka gabatar a cikin fitarwar da ke sama, za mu iya, alal misali, ga abin da muke da shi na haya a halin yanzu:

[heat-admin@overcloud-controller-0 ~]$ cat /var/lib/neutron/dhcp/7d541e74-1c36-4e1d-a7c4-0968c8dbc638/leases
1597492111 fa:16:3e:6c:ad:9c 10.0.2.8 host-10-0-2-8 01:fa:16:3e:6c:ad:9c
1597491115 fa:16:3e:76:c2:11 10.0.2.1 host-10-0-2-1 *
[heat-admin@overcloud-controller-0 ~]$

Sakamakon haka, muna samun saitin ayyuka masu zuwa akan kumburin sarrafawa:

Gabatarwa zuwa ɓangaren cibiyar sadarwa na kayan aikin girgije

To, ku tuna - wannan inji 4 ne kawai, cibiyoyin sadarwa na ciki guda 2 da na'ura mai ba da hanya tsakanin hanyoyin sadarwa guda ɗaya ... Ba mu da cibiyoyin sadarwa na waje a nan yanzu, gungun ayyuka daban-daban, kowannensu yana da nasu hanyoyin sadarwa (roba), kuma muna da. An kashe na'ura mai ba da hanya tsakanin hanyoyin sadarwa, kuma a ƙarshe Bayan haka, akwai kullin sarrafawa ɗaya kawai a cikin benci na gwaji (domin haƙurin kuskure dole ne a sami adadin nodes uku). Yana da ma'ana cewa a cikin kasuwanci duk abin da yake "dan kadan" ya fi rikitarwa, amma a cikin wannan misali mai sauƙi mun fahimci yadda ya kamata ya yi aiki - ko kuna da 3 ko 300 sunaye yana da mahimmanci, amma daga ra'ayi na aikin dukan tsarin, babu abin da zai canza da yawa ... ko da yake ba za ku toshe a wasu SDN mai sayarwa ba. Amma wannan labari ne kwata-kwata.

Ina fata yana da ban sha'awa. Idan kuna da wasu sharhi / ƙari, ko kuma wani wuri na yi ƙarya (Ni ɗan adam ne kuma ra'ayina koyaushe zai kasance mai ma'ana) - rubuta abin da ya kamata a gyara / ƙara - za mu gyara / ƙara komai.

A ƙarshe, Ina so in faɗi 'yan kalmomi game da kwatanta Opentack (duka biyun vanilla da mai siyarwa) tare da maganin gajimare daga VMWare - An yi min wannan tambayar sau da yawa a cikin shekaru biyu da suka gabata, kuma, magana ta gaskiya, Ina riga ya gaji da shi, amma har yanzu. A ra'ayina, yana da matukar wahala a kwatanta wadannan hanyoyin guda biyu, amma muna iya cewa akwai illoli a cikin hanyoyin guda biyu kuma lokacin zabar mafita guda daya kuna buƙatar auna fa'ida da rashin amfani.

Idan OpenStack mafita ce ta al'umma, to VMWare yana da 'yancin yin abin da yake so kawai (karanta - abin da ke da riba a gare shi) kuma wannan yana da ma'ana - saboda kamfani ne na kasuwanci wanda ake amfani da shi don samun kuɗi daga abokan cinikinsa. Amma akwai babba mai kitse, AMMA - zaku iya tashi daga OpenStack, misali daga Nokia, kuma tare da ɗan ƙaramin kashe kuɗi zuwa mafita daga, alal misali, Juniper (Contrail Cloud), amma da alama ba za ku iya sauka daga VMWare ba. . A gare ni, waɗannan mafita guda biyu suna kama da wannan - Opentack (mai siyarwa) ƙaramin keji ne wanda aka saka ku, amma kuna da maɓalli kuma kuna iya barin kowane lokaci. VMWare keji ne na zinari, mai shi yana da makullin kejin kuma zai kashe ku da yawa.

Ba na tallata ko dai samfurin farko ko na biyu ba - kun zaɓi abin da kuke buƙata. Amma idan ina da irin wannan zaɓi, zan zaɓi mafita biyu - VMWare don girgijen IT (ƙananan kaya, sauƙin sarrafawa), OpenStack daga wasu dillalai (Nokia da Juniper suna ba da mafita mai kyau na turnkey) - don girgijen Telecom. Ba zan yi amfani da Opentack don tsantsar IT ba - yana kama da harbin sparrows tare da igwa, amma ban ga wani hani game da amfani da shi ban da sakewa. Koyaya, yin amfani da VMWare a cikin telecom yana kama da ɗaure dakakken dutse a cikin Ford Raptor - yana da kyau daga waje, amma direban dole ne ya yi tafiye-tafiye 10 maimakon ɗaya.

A ra'ayi na, babban hasara na VMWare shine cikakken rufewa - kamfanin ba zai ba ku wani bayani game da yadda yake aiki ba, misali, vSAN ko abin da ke cikin kernel hypervisor - ba shi da riba a gare shi - wato, za ku iya. Kada ku taɓa zama ƙwararre a cikin VMWare - ba tare da tallafin mai siyarwa ba, an hallaka ku (sau da yawa ina saduwa da ƙwararrun VMWare waɗanda ke mamakin tambayoyi marasa mahimmanci). A gare ni, VMWare yana siyan mota tare da murfi a kulle - eh, kuna iya samun ƙwararrun ƙwararrun da za su iya canza bel na lokaci, amma wanda ya sayar muku da wannan maganin kawai zai iya buɗe murfin. Da kaina, ba na son mafita waɗanda ba zan iya shiga ciki ba. Za ku ce mai yiwuwa ba lallai ne ku shiga ƙarƙashin hular ba. Haka ne, wannan yana yiwuwa, amma zan dube ku lokacin da kuke buƙatar tara babban aiki a cikin gajimare daga 20-30 inji mai mahimmanci, cibiyoyin sadarwa 40-50, rabin abin da ke son fita waje, rabi na biyu kuma ya nemi. Hanzarta SR-IOV, in ba haka ba za ku buƙaci ƙarin dozin na waɗannan motoci - in ba haka ba aikin ba zai isa ba.

Akwai wasu ra'ayoyi, don haka kawai ku ne za ku iya yanke shawarar abin da za ku zaɓa kuma, mafi mahimmanci, za ku ɗauki alhakin zaɓinku. Wannan ra'ayina ne kawai - mutumin da ya gani kuma ya taɓa samfura aƙalla 4 - Nokia, Juniper, Red Hat da VMWare. Wato ina da abin da zan kwatanta da shi.

source: www.habr.com

Add a comment