Hello, Habr! Ni Artem Karamyshev, shugaban tawagar gudanarwar tsarin . Mun sami sabbin samfura da yawa da aka ƙaddamar a cikin shekarar da ta gabata. Muna son tabbatar da cewa ayyukan API suna da sauƙin daidaitawa, masu jure wa kuskure, kuma a shirye don haɓaka cikin sauri cikin nauyin mai amfani. Ana aiwatar da dandalinmu akan OpenStack, kuma ina so in gaya muku menene matsalolin haƙuri da kuskuren da muka magance don samun tsarin jurewa kuskure. Ina tsammanin wannan zai zama mai ban sha'awa ga waɗanda suma suka haɓaka samfura akan OpenStack.
Hakuri na kuskure na dandamali ya ƙunshi juriyar abubuwan da ke cikin sa. Don haka sannu a hankali za mu bi duk matakan da muka gano haɗari kuma muka rufe su.
Sigar bidiyo na wannan labarin, tushen asalinsa shine rahoto a taron Uptime Day 4, wanda ya shirya , kuna iya gani .
Resiliency na gine-ginen jiki
Bangaren jama'a na gajimare na MCS yanzu yana dogara ne a cikin cibiyoyin bayanai na Tier III guda biyu, a tsakanin su akwai nasa fiber fiber nasa, wanda aka tanada a matakin jiki ta hanyoyi daban-daban, tare da kayan aiki na 200 Gbit/s. Tier III yana ba da mahimmancin matakin haƙuri na kuskure don kayan aikin jiki.
Ana adana fiber mai duhu a duka matakan jiki da na hankali. Tsarin ajiyar tashar ya kasance mai maimaitawa, matsaloli sun taso, kuma muna ci gaba da inganta sadarwa tsakanin cibiyoyin bayanai.
Misali, ba da dadewa ba, yayin da yake aiki a wata rijiya da ke kusa da daya daga cikin cibiyoyin bayanai, wani mai tonawa ya fasa bututu, kuma a cikin wannan bututun akwai babbar igiya da kuma na’urar daukar hoto. Tashar sadarwarmu mai jurewa da kuskure tare da cibiyar bayanai ta zama mai rauni a wani lokaci, a cikin rijiyar. Saboda haka, mun rasa wani ɓangare na abubuwan more rayuwa. Mun yanke shawara kuma mun ɗauki ayyuka da yawa, gami da shigar da ƙarin na'urorin gani a cikin rijiyar da ke kusa.
A cikin cibiyoyin bayanai akwai wuraren kasancewar masu samar da sadarwa waɗanda muke watsa musu prefixes ta BGP. Ga kowane jagorar hanyar sadarwa, an zaɓi mafi kyawun awo, wanda ke ba da damar samar da abokan ciniki daban-daban tare da mafi kyawun haɗin haɗin gwiwa. Idan sadarwa ta hanyar mai ba da sabis ɗaya ta ragu, za mu sake gina hanyarmu ta hanyar masu samarwa.
Idan mai badawa ya gaza, muna canzawa ta atomatik zuwa na gaba. Idan akwai gazawar daya daga cikin cibiyoyin bayanai, muna da kwafin madubi na ayyukanmu a cibiyar bayanai ta biyu, wacce ke ɗaukar nauyin duka.
Resilience na kayan aikin jiki
Abin da muke amfani da shi don haƙurin kuskure matakin aikace-aikace
An gina sabis ɗin mu akan adadin abubuwan abubuwan buɗewa.
ExaBGP sabis ne da ke aiwatar da ayyuka da yawa ta amfani da ka'idar tuƙi mai ƙarfi ta tushen BGP. Muna amfani da shi sosai don tallata adiresoshin IP ɗin mu waɗanda masu amfani ke samun damar API.
HAProxy babban ma'auni ne mai ɗaukar nauyi wanda ke ba ku damar daidaita ƙa'idodin daidaita zirga-zirgar ababen hawa a matakai daban-daban na ƙirar OSI. Muna amfani da shi don daidaitawa a gaban duk ayyuka: bayanan bayanai, dillalai na saƙo, sabis na API, sabis na yanar gizo, ayyukan mu na ciki - duk abin yana bayan HAProxy.
API aikace-aikace - aikace-aikacen yanar gizo da aka rubuta a cikin Python, wanda mai amfani ke sarrafa kayan aikin sa da sabis ɗin sa.
Aikace-aikacen ma'aikaci (nan gaba kawai ma'aikaci) - a cikin ayyukan OpenStack, wannan kayan aikin daemon ne wanda ke ba ku damar watsa umarnin API zuwa kayan aikin. Misali, ƙirƙirar faifai yana faruwa a cikin ma'aikaci, kuma buƙatar ƙirƙirar tana faruwa a cikin API ɗin aikace-aikacen.
Standard OpenStack Application Architecture
Yawancin ayyukan da aka haɓaka don OpenStack suna ƙoƙarin bin tsari guda ɗaya. Sabis yawanci ya ƙunshi sassa 2: API da ma'aikata (masu zartarwa na baya). A matsayinka na mai mulki, API shine aikace-aikacen WSGI a cikin Python, wanda aka ƙaddamar ko dai azaman tsari mai zaman kansa (daemon), ko ta amfani da sabar gidan yanar gizo na Nginx ko Apache da aka shirya. API ɗin yana aiwatar da buƙatar mai amfani kuma yana ba da ƙarin umarni zuwa aikace-aikacen ma'aikaci don aiwatarwa. Canja wurin yana faruwa ta amfani da dillalin saƙo, yawanci RabbitMQ, sauran ba su da tallafi sosai. Lokacin da saƙonni suka isa dillali, ma'aikata ne ke sarrafa su kuma, idan ya cancanta, mayar da martani.
Wannan tsarin ya ƙunshi keɓancewar wuraren gama gari na gazawa: RabbitMQ da ma'ajin bayanai. Amma RabbitMQ ya keɓe a cikin sabis ɗaya kuma, a ka'idar, na iya zama mutum ɗaya don kowane sabis. Don haka a MCS muna raba waɗannan ayyuka gwargwadon yiwuwa; ga kowane aikin mutum ɗaya muna ƙirƙiri keɓantaccen ma'ajin bayanai, RabbitMQ daban. Wannan hanya tana da kyau saboda idan wani haɗari ya faru a wasu wurare masu rauni, ba duk sabis ɗin ya rushe ba, amma kawai wani ɓangare na shi.
Yawan aikace-aikacen ma'aikaci ba shi da iyaka, don haka API ɗin zai iya sauƙi auna a kwance a bayan ma'auni don ƙara aiki da haƙurin kuskure.
Wasu ayyuka suna buƙatar daidaitawa a cikin sabis ɗin lokacin da hadaddun ayyuka masu rikitarwa suka faru tsakanin APIs da ma'aikata. A wannan yanayin, ana amfani da cibiyar daidaitawa guda ɗaya, tsarin cluster irin su Redis, Memcache, da dai sauransu, wanda ke ba ma'aikaci damar gaya wa wani cewa an ba shi wannan aikin ("don Allah kar a ɗauka"). Muna amfani da etcd. A matsayinka na mai mulki, ma'aikata suna sadarwa tare da bayanan bayanai, rubutawa da karanta bayanai daga can. Muna amfani da mariadb azaman ma'ajin bayanai, wanda ke cikin gungun manyan ma'aikata.
An tsara wannan sabis ɗin na al'ada guda ɗaya ta hanyar da aka yarda da ita don OpenStack. Ana iya la'akari da shi azaman tsarin rufaffiyar, wanda hanyoyin haɓakawa da haƙurin kuskure suke bayyane. Misali, don haƙurin kuskuren API, ya isa ya sanya ma'auni a gabansu. Ana samun ma'aikatan sikelin ta hanyar ƙara yawan su.
Matsakaicin rauni a cikin duka makircin shine RabbitMQ da MariaDB. Tsarin gine-ginen su ya cancanci labarin daban. A cikin wannan labarin ina so in mai da hankali kan haƙurin kuskuren API.
Opentack Application Architecture. Daidaitawa da rashin haƙuri na dandalin girgije
Yin Haƙuri mai ma'aunin HAProxy mai haƙuri ta hanyar amfani da ExaBGP
Don sanya APIs ɗin mu su iya daidaitawa, mai sauri da jurewa kuskure, muna sanya ma'aunin nauyi a gabansu. Mun zaɓi HAProxy. A ganina, yana da duk halayen da suka dace don aikinmu: daidaitawa a matakan OSI da yawa, haɗin gwiwar gudanarwa, sassauci da haɓakawa, babban adadin hanyoyin daidaitawa, goyon baya ga teburin zaman.
Matsala ta farko da ake buƙatar warwarewa ita ce rashin haƙuri na ma'auni. Kawai shigar da ma'auni kuma yana haifar da maƙasudin gazawa: mai daidaitawa ya karye kuma sabis ɗin ya rushe. Don hana hakan faruwa, mun yi amfani da HAProxy tare da ExaBGP.
ExaBGP yana ba ku damar aiwatar da tsarin duba yanayin sabis. Mun yi amfani da wannan hanyar don bincika aikin HAProxy kuma, idan akwai matsaloli, musaki sabis na HAProxy daga BGP.
ExaBGP+HAProxy tsarin
- Muna shigar da software mai mahimmanci, ExaBGP da HAProxy, akan sabobin uku.
- Mun ƙirƙiri madaidaicin madauki akan kowane uwar garken.
- A kan duk sabar guda uku muna sanya adireshin IP iri ɗaya zuwa wannan haɗin gwiwa.
- Ana tallata farar adireshin IP zuwa Intanet ta hanyar ExaBGP.
Ana samun haƙurin kuskure ta hanyar tallata adireshin IP iri ɗaya daga duk sabobin uku. Daga ra'ayi na cibiyar sadarwa, ana samun adireshin iri ɗaya daga hops daban-daban guda uku na gaba. Mai amfani da na'ura mai ba da hanya tsakanin hanyoyin sadarwa yana ganin hanyoyi guda uku, yana zaɓar mafi girman fifikon su bisa la'akari da nasa awo (wannan yawanci zaɓi ɗaya ne), kuma zirga-zirgar yana zuwa ɗaya daga cikin sabobin ne kawai.
Idan akwai matsaloli tare da aikin HAProxy ko gazawar uwar garken, ExaBGP yana dakatar da sanar da hanyar, kuma zirga-zirgar zirga-zirgar ta sauya cikin sauƙi zuwa wani sabar.
Don haka, mun sami haƙurin kuskure na ma'auni.
Haƙurin kuskure na masu daidaita ma'aunin HAProxy
Tsarin ya zama mara kyau: mun koyi yadda ake ajiye HAProxy, amma ba mu koyi yadda ake rarraba kaya a cikin ayyukan ba. Saboda haka, mun fadada wannan makirci kadan: mun matsa zuwa daidaitawa tsakanin farar adiresoshin IP da yawa.
Daidaitawa bisa DNS da BGP
Batun daidaita kaya don HAProxy ɗin mu ya kasance ba a warware shi ba. Koyaya, ana iya magance shi a sauƙaƙe, kamar yadda muka yi a nan.
Don daidaita sabobin uku za ku buƙaci farar adiresoshin IP 3 da tsohuwar DNS mai kyau. Kowane ɗayan waɗannan adireshi an ƙaddara akan madaidaicin madauki na kowane HAProxy kuma ana tallata shi zuwa Intanet.
A cikin OpenStack, don sarrafa albarkatu, ana amfani da kundin adireshin sabis, wanda ke ƙayyadaddun API na ƙarshen wani sabis. A cikin wannan jagorar muna yin rajistar sunan yanki - public.infra.mail.ru, wanda aka warware ta hanyar DNS ta adiresoshin IP daban-daban guda uku. A sakamakon haka, muna samun rarraba kaya tsakanin adireshi uku ta hanyar DNS.
Amma tunda lokacin sanar da farar adiresoshin IP ba mu sarrafa abubuwan fifikon zaɓin uwar garken, wannan bai daidaita ba tukuna. Yawanci, uwar garken guda ɗaya kawai za a zaɓa bisa ga girman adireshin IP, sauran biyun kuma za su kasance marasa aiki saboda ba a ƙayyade ma'auni a cikin BGP ba.
Mun fara aika hanyoyi ta hanyar ExaBGP tare da ma'auni daban-daban. Kowane ma'auni yana tallata duk farar adiresoshin IP guda uku, amma ɗaya daga cikinsu, babban na wannan ma'auni, ana tallata shi tare da mafi ƙarancin awo. Don haka yayin da dukkanin ma'auni guda uku ke aiki, kira zuwa adireshin IP na farko yana zuwa madaidaicin farko, kira zuwa na biyu zuwa na biyu, kuma kira zuwa na uku zuwa na uku.
Me zai faru idan ɗaya daga cikin ma'auni ya faɗi? Idan kowane mai daidaitawa ya gaza, babban adireshinsa har yanzu ana tallata shi daga sauran biyun, kuma ana sake rarraba zirga-zirga a tsakanin su. Don haka, muna ba mai amfani da adiresoshin IP da yawa lokaci guda ta hanyar DNS. Ta hanyar daidaitawa ta hanyar DNS da ma'auni daban-daban, muna samun madaidaicin rarraba kaya a duk ma'auni guda uku. Kuma a lokaci guda ba mu rasa haƙuri ga kuskure.
Daidaita HAProxy dangane da DNS + BGP
Ma'amala tsakanin ExaBGP da HAProxy
Don haka, mun aiwatar da haƙurin kuskure idan uwar garken ya fita, dangane da dakatar da sanarwar hanyoyin. Amma HAProxy na iya rufewa don wasu dalilai fiye da gazawar uwar garken: kurakuran gudanarwa, kasawa a cikin sabis ɗin. Muna so mu cire ma'auni mai karya daga ƙarƙashin kaya a cikin waɗannan lokuta ma, kuma muna buƙatar wata hanya ta daban.
Saboda haka, fadada makircin da ya gabata, mun aiwatar da bugun zuciya tsakanin ExaBGP da HAProxy. Wannan aiwatar da software ne na hulɗar tsakanin ExaBGP da HAProxy, lokacin da ExaBGP ke amfani da rubutun al'ada don bincika matsayin aikace-aikace.
Don yin wannan, kuna buƙatar saita mai duba lafiya a cikin saitin ExaBGP, wanda zai iya duba matsayin HAProxy. A cikin yanayinmu, mun saita bayanan kiwon lafiya a cikin HAProxy, kuma daga gefen ExaBGP muna bincika tare da buƙatun GET mai sauƙi. Idan sanarwar ta daina faruwa, to HAProxy ba zai iya aiki ba kuma babu buƙatar tallata shi.
Binciken Lafiya na HAProxy
Abokan HaProxy: aiki tare
Abu na gaba shine a daidaita zaman. Lokacin aiki ta hanyar ma'auni masu rarraba, yana da wuya a tsara ajiyar bayanai game da zaman abokin ciniki. Amma HAProxy yana ɗaya daga cikin ƙananan ma'auni waɗanda zasu iya yin wannan saboda ayyukan Takwarorinsu - ikon canja wurin teburin zaman tsakanin hanyoyin HAProxy daban-daban.
Akwai hanyoyin daidaitawa daban-daban: masu sauƙi kamar , da kuma tsawaita, lokacin da aka tuna da zaman abokin ciniki, kuma duk lokacin da ya ƙare akan sabar iri ɗaya kamar da. Mun so aiwatar da zaɓi na biyu.
HAProxy yana amfani da tebur-tables don adana zaman abokin ciniki na wannan tsarin. Suna adana ainihin adireshin IP na abokin ciniki, adireshin da aka zaɓa (bayan baya) da wasu bayanan sabis. Yawanci, ana amfani da tebur na sanda don adana tushen-IP + manufa-IP biyu, wanda ke da amfani musamman ga aikace-aikacen da ba za su iya canja wurin mahallin zaman mai amfani ba lokacin canzawa zuwa wani ma'auni, misali, a cikin yanayin daidaitawa na RoundRobin.
Idan an koyar da tebur na sanda don motsawa tsakanin hanyoyin HAProxy daban-daban (tsakanin abin da daidaitawa ke faruwa), masu daidaitawa za su iya yin aiki tare da tebur guda ɗaya na tebur. Wannan zai ba da damar sauya hanyar sadarwar abokin ciniki ba tare da matsala ba idan ɗaya daga cikin ma'auni ya gaza; aiki tare da zaman abokin ciniki zai ci gaba a kan madaidaitan baya waɗanda aka zaɓa a baya.
Don aiki mai kyau, dole ne a warware matsalar tushen adireshin IP na ma'auni daga abin da aka kafa zaman. A cikin yanayinmu, wannan adireshi ne mai ƙarfi akan madaidaicin madauki.
Ana samun daidaitaccen aikin abokan aiki ne kawai a ƙarƙashin wasu yanayi. Wato, lokacin TCP dole ne ya zama babban isa ko sauyawa dole ne ya kasance da sauri isa don zaman TCP ba shi da lokacin ƙarewa. Duk da haka, yana ba da damar sauyawa maras kyau.
A cikin IaaS muna da sabis da aka gina ta amfani da fasaha iri ɗaya. Wannan , wanda ake kira Octavia. Ya dogara ne akan matakai biyu na HAProxy kuma da farko ya haɗa da goyon baya ga takwarorinsu. Sun tabbatar da kansu na kwarai a wannan hidimar.
Hoton da aka tsara yana nuna motsi na tebur na abokan gaba tsakanin lokuta uku na HAProxy, ana ba da shawarar saiti akan yadda za'a iya daidaita wannan:
HAProxy Peers (aiki tare da zama)
Idan kun aiwatar da wannan makirci, dole ne a gwada aikin sa a hankali. Ba gaskiya ba ne cewa zai yi aiki a cikin hanya guda 100% na lokaci. Amma aƙalla ba za ku rasa tebur ɗin sanda ba lokacin da kuke buƙatar tunawa da tushen IP na abokin ciniki.
Iyakance adadin buƙatun lokaci guda daga abokin ciniki ɗaya
Duk wani sabis ɗin da ake samu a bainar jama'a, gami da APIs ɗin mu, na iya zama ƙarƙashin ɗumbin buƙatun. Dalilan su na iya bambanta gaba ɗaya, daga kurakuran masu amfani zuwa harin da aka yi niyya. Muna DDoSed lokaci-lokaci ta adiresoshin IP. Abokan ciniki sukan yi kuskure a cikin rubutun su kuma suna ba mu ƙaramin DDoSs.
Wata hanya ko wata, dole ne a samar da ƙarin kariya. Mahimmin bayani shine iyakance adadin buƙatun API kuma kada a ɓata lokacin sarrafa buƙatun ƙeta na CPU.
Don aiwatar da irin waɗannan hane-hane, muna amfani da iyakokin ƙima, wanda aka tsara bisa tushen HAProxy, ta amfani da teburan sanda iri ɗaya. Saita iyaka abu ne mai sauƙi kuma yana ba ku damar iyakance mai amfani da adadin buƙatun zuwa API. Algorithm yana tuna tushen IP daga abin da ake buƙatar buƙatun kuma yana iyakance adadin buƙatun lokaci guda daga mai amfani ɗaya. Tabbas, mun ƙididdige matsakaicin bayanin martabar lodin API don kowane sabis kuma mun saita iyaka na ≈ 10 wannan ƙimar. Muna ci gaba da sanya ido sosai kan lamarin kuma mu ci gaba da yatsa a bugun jini.
Menene wannan yayi kama a aikace? Muna da abokan ciniki waɗanda ke amfani da APIs masu sarrafa kansu koyaushe. Suna ƙirƙira kusan na'urori masu kama da ɗari biyu zuwa ɗari uku da safe kuma suna goge su da yamma. Don OpenStack, ƙirƙirar injin kama-da-wane, kuma tare da sabis na PaaS, yana buƙatar buƙatun API aƙalla 1000, tunda ma'amala tsakanin sabis kuma yana faruwa ta hanyar API.
Irin wannan canja wurin ayyuka yana haifar da babban kaya mai girman gaske. Mun tantance wannan kaya, muka tattara kololuwar yau da kullun, muka kara su sau goma, kuma wannan ya zama iyakar ƙimar mu. Muna ci gaba da yatsanmu akan bugun jini. Sau da yawa muna ganin bots da na'urar daukar hotan takardu waɗanda ke ƙoƙarin duba mu don ganin ko muna da wasu rubutun CGA da za a iya gudanar da su, muna yanke su sosai.
Yadda ake sabunta lambar lambar ku ba tare da lura da masu amfani ba
Muna kuma aiwatar da haƙurin kuskure a matakin matakan tura lambar. Za a iya samun kurakurai a lokacin fiddawa, amma ana iya rage tasirinsu akan samuwar sabis.
Muna sabunta ayyukanmu koyaushe kuma dole ne mu tabbatar da cewa an sabunta lambar tushe ba tare da cutar da masu amfani ba. Mun gudanar da magance wannan matsala ta amfani da ikon gudanarwa na HAProxy da aiwatar da Ƙaƙwalwar Ƙarfafawa a cikin ayyukanmu.
Don magance wannan matsala, ya zama dole don tabbatar da sarrafa ma'auni da kuma "daidai" rufe sabis:
- A cikin yanayin HAProxy, ana yin sarrafawa ta hanyar fayil ɗin ƙididdiga, wanda shine ainihin soket kuma an bayyana shi a cikin tsarin HAProxy. Kuna iya aika umarni zuwa gare shi ta hanyar stdio. Amma babban kayan aikin mu na sarrafa sanyi yana yiwuwa, don haka yana da tsarin ginanniyar don sarrafa HAProxy. Wanda muke amfani da shi sosai.
- Yawancin ayyukan API da Injin mu suna tallafawa fasahohin rufewa masu kyau: lokacin da suke rufewa, suna jira don kammala aikin na yanzu, ya zama buƙatun http ko wani aikin sabis. Haka abin yake faruwa da ma'aikaci. Ya san duk ayyukan da yake yi kuma yana ƙarewa lokacin da ya sami nasarar kammala komai.
Godiya ga waɗannan maki biyu, amintaccen algorithm don tura mu yayi kama da wannan.
- Mai haɓakawa ya haɗa sabon kunshin lambar (a gare mu wannan shine RPM), gwada shi a cikin yanayin dev, gwada shi a cikin mataki, kuma ya bar shi a cikin ma'ajin mataki.
- Mai haɓakawa ya tsara aikin ƙaddamarwa tare da mafi cikakken bayanin "kayan aikin": sigar sabon kunshin, bayanin sabon aikin da sauran cikakkun bayanai game da ƙaddamarwa idan ya cancanta.
- Mai sarrafa tsarin yana fara sabuntawa. Ya ƙaddamar da littafin wasan kwaikwayo mai yiwuwa, wanda kuma yana yin haka:
- Yana ɗaukar fakiti daga ma'ajiyar mataki kuma yana amfani da shi don ɗaukaka sigar fakitin a cikin ma'ajin samfur.
- Yana tattara jerin abubuwan baya na sabis ɗin da aka sabunta.
- Yana rufe sabis na farko da za a sabunta a cikin HAProxy kuma yana jiran matakansa don gama aiki. Godiya ga kyakkyawan rufewa, muna da tabbacin cewa duk buƙatun abokin ciniki na yanzu za su kammala cikin nasara.
- Bayan API da ma'aikata sun tsaya gaba ɗaya, kuma an kashe HAProxy, an sabunta lambar.
- Ayyukan gudanarwa masu yiwuwa.
- Ga kowane sabis, ana jan wasu “hannu”, waɗanda ke yin gwajin naúrar akan adadin gwaje-gwajen maɓalli da aka riga aka ayyana. Binciken ainihin sabon lambar yana faruwa.
- Idan ba a sami kurakurai a matakin da ya gabata ba, ana kunna ƙarshen baya.
- Mu ci gaba zuwa baya na gaba.
- Bayan an sabunta duk bayanan baya, ana ƙaddamar da gwaje-gwajen aiki. Idan sun ɓace, to, mai haɓaka yana duba kowane sabon aikin da ya ƙirƙira.
Wannan yana kammala turawa.
Zagayowar sabunta sabis
Wannan makirci ba zai yi aiki ba idan ba mu da ka'ida ɗaya. Muna goyon bayan duka tsofaffi da sababbin sigar yaƙi. A gaba, a matakin haɓaka software, an tsara cewa ko da an sami canje-canje a cikin bayanan sabis, ba za su karya lambar da ta gabata ba. Sakamakon haka, a hankali ana sabunta tushen lambar.
ƙarshe
Raba tunanina game da tsarin gine-gine na WEB mai jurewa, Ina so in sake lura da mahimman abubuwansa:
- haƙurin kuskuren jiki;
- Haƙuri na kuskuren hanyar sadarwa (masu daidaitawa, BGP);
- rashin haƙuri na software da aka yi amfani da shi da haɓaka.
Tsayayyen lokaci kowa!
source: www.habr.com
