TL; DR: Incazelo yesakhiwo seseva yeklayenti yesistimu yethu yangaphakathi yokulawula ukucushwa kwenethiwekhi, QControl. Isekelwe kuphrothokholi yokuthutha enezendlalelo ezimbili esebenza nemilayezo egcwele i-gzip ngaphandle kokucindezelwa phakathi kwamaphoyinti okugcina. Amarutha asabalalisiwe namaphoyinti okugcina athola izibuyekezo zokucushwa, futhi iphrothokholi ngokwayo ivumela ukufakwa kwama-relay aphakathi nendawo. Uhlelo lwakhiwe phezu komgomo (“okuzinzile kwakamuva”, kuchazwe ngezansi) futhi isebenzisa ulimi lombuzo lwe-JMESpath kanye nenjini yesifanekiso ye-Jinja ukuze inikeze amafayela okumisa.
I-Qrator Labs isebenzisa inethiwekhi yokunciphisa ukuhlasela esatshalaliswa emhlabeni wonke. Inethiwekhi yethu isebenza kumgomo we-anycast, futhi ama-subnet akhangiswa nge-BGP. Njengoba siyinethiwekhi ye-BGP yanoma iyiphi i-cast etholakala ngokoqobo ezifundeni ezimbalwa zomhlaba, singacubungula futhi sihlunge ithrafikhi engekho emthethweni eduze komnyombo we-inthanethi - ama-opharetha we-Tier-1.
Ngakolunye uhlangothi, ukuba yinethiwekhi esabalaliswa ngokwendawo akulula. Ukuxhumana phakathi kwamaphoyinti okuba khona kwenethiwekhi kubalulekile kumhlinzeki wesevisi yezokuphepha ukuze abe nokucushwa okungaguquki kwawo wonke ama-node enethiwekhi, awabuyekeze ngesikhathi esifanele. Ngakho-ke, ukuze sinikeze izinga eliphezulu kakhulu lesevisi eyinhloko kumthengi, besidinga ukuthola indlela yokuvumelanisa ngokuthembekile idatha yokucushwa kuwo wonke amazwekazi.
Ekuqaleni wayekhona uLizwi. Ngokushesha yaba iphrothokholi yezokuxhumana edinga ukubuyekezwa.
Isisekelo sobukhona be-QControl, futhi ngesikhathi esifanayo isizathu esiyinhloko sokuchitha isikhathi esiningi kanye nezinsiza ekwakheni lolu hlobo lwephrothokholi, isidingo sokuthola umthombo owodwa ogunyaziwe wokucushwa futhi, ekugcineni, ukuvumelanisa amaphuzu ethu okuba khona. ngayo. Isitoreji ngokwaso besingesinye sezidingo ezimbalwa ngesikhathi kuthuthukiswa i-QControl. Ngaphezu kwalokho, siphinde sadinga ukuhlanganiswa nezinsiza ezikhona nezihleliwe ezindaweni lapho kukhona khona (i-POP), izindlela ezihlakaniphile (futhi ezingenziwa ngendlela oyifisayo) zokuqinisekisa idatha, kanye nokulawula ukufinyelela. Ngaphandle kwalokhu, besifuna nokulawula uhlelo olunjalo sisebenzisa imiyalo kunokwenza izinguquko kumafayela. Ngaphambi kwe-QControl, idatha ithunyelwe ezindaweni lapho kukhona khona cishe mathupha. Uma elinye lamaphoyinti okuba khona lalingekho futhi sakhohlwa ukulibuyekeza ngokuhamba kwesikhathi, ukulungiselelwa kuzogcina kungavumelanisiwe futhi kuzodingeka simoshe isikhathi ukuyibuyisela futhi isebenze.
Ngenxa yalokho, siqhamuke nalolu hlelo olulandelayo:
Iseva yokumisa inesibopho sokuqinisekisa nokugcinwa kwedatha; irutha inezindawo ezimbalwa zokugcina ezithola futhi zisakaze izibuyekezo zokumisa kusuka kumakhasimende namathimba okusekela kuya kuseva, futhi ukusuka kuseva kuya ezindaweni lapho ukhona.
Izinga loxhumano lwe-inthanethi lisahluka kakhulu emhlabeni jikelele - ukukhombisa leli phuzu, ake sibheke i-MTR elula esuka ePrague, Czech Republic kuya eSingapore naseHong Kong.
I-MTR isuka ePrague iye eSingapore
Into efanayo naseHong Kong
Ukubambezeleka okuphezulu kusho isivinini esiphansi. Ngaphezu kwalokho, kukhona ukulahleka kwephakethe. Ububanzi besiteshi abuyinxephezeli le nkinga, okufanele ngaso sonke isikhathi kubhekwe lapho kwakhiwa amasistimu ahlukaniselwe amazwe.
Ukucushwa okugcwele kwephuzu lokuba khona inani elibalulekile ledatha okufanele ithunyelwe kubamukeli abaningi ngoxhumo olungathembekile. Ngenhlanhla, nakuba ukucushwa kushintsha njalo, kwenzeka ngokunyuka okuncane.
Idizayini ezinzile yakamuva
Singasho ukuthi ukwakha inethiwekhi esabalalisiwe ngokusekelwe kumgomo wokuvuselela okukhulayo kuyisixazululo esisobala. Kodwa kunezinkinga eziningi nge-diffs. Kudingeka silondoloze wonke umehluko phakathi kwamaphoyinti ezithenjwa, futhi sikwazi ukuwathumela kabusha uma kwenzeka othile egeje ingxenye yedatha. Indawo ngayinye kufanele iwasebenzise ngokulandelana okucaciswe ngokuqinile. Ngokuvamile, esimweni sezindawo eziningana, ukusebenza okunjalo kungathatha isikhathi eside. Umamukeli kufanele futhi akwazi ukucela izingxenye ezingekho futhi, yiqiniso, ingxenye emaphakathi kufanele iphendule esicelweni esinjalo ngendlela efanele ngokuthumela idatha engekho kuphela.
Ngenxa yalokho, sifinyelele isixazululo esithakazelisayo - sinesendlalelo esisodwa kuphela, esilungisiwe, asibize ngokuthi sizinzile, futhi enye kuphela i-diff yayo - yakamuva. Okwakamuva ngakunye kusekelwe esitebeleni esikhiqiziwe sokugcina futhi sanele ukwakha kabusha idatha yokucushwa. Lapho nje esanda kufika lapho iya khona, endala ayisadingeki.
Okusele ukuthumela ukucushwa okusha okuzinzile ngezikhathi ezithile, isibonelo ngoba okwakamuva kube kukhulu kakhulu. Okunye okubalulekile lapha ukuthi sithumela zonke lezi zibuyekezo ngemodi yokusakaza/yokusakaza okuningi, ngaphandle kokukhathazeka ngabamukeli ngabanye kanye nekhono labo lokuhlanganisa izingcezu zedatha. Uma sesiqinisekile ukuthi wonke umuntu unesitebele esifanele, sithumela kuphela ezintsha zakamuva. Ingabe kufanelekile ukucacisa ukuthi lokhu kuyasebenza? Iyasebenza. Isitebele sifakwe kunqolobane kuseva yokucushwa kanye nabamukeli, okwakamuva kwakhiwa njengoba kudingeka.
I-Architecture yezokuthutha ezinamazinga amabili
Kungani sakhe ezokuthutha zethu emazingeni amabili? Impendulo ilula impela - besifuna ukuhlukanisa umzila ukusuka kumqondo wezinga eliphezulu, sithatha ugqozi oluvela kumodeli ye-OSI ngezendlalelo zayo zokuthutha nezokusebenza. Sisebenzise i-Thrift endimeni yephrothokholi yezokuthutha, kanye nefomethi ye-msgpack yokulandelanisa yefomethi yezinga eliphezulu lemilayezo yokulawula. Kungakho umzila (owenza i-multicast/streamcast/relay) ingabheki ngaphakathi kwe-msgpack, ingakukhiqizi noma ikubuyisele okuqukethwe, futhi idlulisela kuphela idatha.
I-Thrift (kusuka kusiNgisi - “thrift”, okuphinyiselwa ngokuthi [θrift]) iwulimi oluchaza isixhumi esibonakalayo olusetshenziselwa ukuchaza nokudala izinsiza zezilimi ezihlukene zokuhlela. Kuwuhlaka lwezingcingo zezinqubo ezikude (RPC). Ihlanganisa ipayipi lesofthiwe nenjini yokukhiqiza ikhodi ukuze kuthuthukiswe izinsiza ezisebenza kahle kakhulu futhi kalula phakathi kwezilimi.
Sikhethe uhlaka lwe-Thrift ngenxa ye-RPC nokusekelwa kwezilimi eziningi. Njengenjwayelo, izingxenye ezilula kwakuyiklayenti neseva. Kodwa-ke, i-router iphenduke inadi elinzima ukuqhekeka, ngokwengxenye ngenxa yokuntuleka kwesixazululo esenziwe ngomumo ngesikhathi sokuthuthuka kwethu.
Kukhona ezinye izinketho, njenge-protobuf / gRPC, nokho, ngenkathi siqala iphrojekthi yethu, i-gRPC yayiyintsha impela futhi asizange silokothe siyigibele.
Yebo, besingakwazi (futhi empeleni kufanele) sizakhele ibhayisikili lethu. Kungaba lula ukwakha iphrothokholi yalokho esikudingayo ngoba ukwakheka kweseva yeklayenti kulula ukukusebenzisa uma kuqhathaniswa nokwakha umzila ku-Thrift. Ngandlela thize, kunokuchema okungokwesiko mayelana nezinqubo ezizibhalela yona kanye nasekusetshenzisweni kwemitapo yolwazi edumile (ngesizathu esihle); ngaphezu kwalokho, phakathi nezingxoxo umbuzo uvela njalo: "Sizokuhambisa kanjani lokhu kwezinye izilimi?" Ngakho sawulahla ngokushesha umqondo webhayisikili.
I-Msgpack ifana ne-JSON, kodwa iyashesha futhi incane. Kuyifomethi yokuhlukanisa idatha kanambambili evumela ukuthi idatha ishintshwe phakathi kwezilimi eziningi.
Ezingeni lokuqala sine-Thrift enolwazi oluncane oludingekayo ukuze umzila udlulise umlayezo. Ezingeni lesibili kunezakhiwo ze-msgpack ezipakishiwe.
Sikhethe i-msgpack ngoba ishesha futhi iminyene kakhulu uma iqhathaniswa ne-JSON. Kodwa okubaluleke nakakhulu, isekela izinhlobo zedatha yangokwezifiso, okusivumela ukuthi sisebenzise izici ezinhle njengokudlulisa okuhamba ngakubili okungaphekiwe noma izinto ezikhethekile ezibonisa ukungabikho kwedatha, obekubalulekile esikimini sethu "esizinzile sakamuva".
I-JMESPath
I-JMESPath iwulimi lombuzo lwe-JSON.
Yile ndlela kanye incazelo esiyithola emibhalweni esemthethweni ye-JMESPath ebukeka ngayo, kodwa empeleni, yenza okungaphezu kwalokho. I-JMESPath ikuvumela ukuthi useshe futhi uhlunge izihlahlana ezingaphansi esakhiweni sesihlahla esinganaki, futhi usebenzise izinguquko kudatha empukaneni. Iphinde ikuvumela ukuthi ungeze izihlungi ezikhethekile nezinqubo zokuguqulwa kwedatha. Nakuba, kunjalo, kudinga umzamo wobuchopho ukuqonda.
Jinja
Kwabanye abathengi, sidinga ukushintsha ukucushwa kube ifayela - ngakho-ke sisebenzisa injini yesifanekiso futhi iJinja iyinketho esobala. Ngosizo lwayo, sikhiqiza ifayela lokumisa elisuka kusifanekiso nedatha etholwe endaweni okuyiwa kuyo.
Ukuze sikhiqize ifayela lokumisa, sidinga isicelo se-JMESPath, isifanekiso sendawo yefayela ku-FS, kanye nesifanekiso se-config ngokwayo. Kungumqondo omuhle futhi kulesi sigaba ukucacisa izimvume zefayela. Konke lokhu kuhlanganiswe ngempumelelo efayeleni elilodwa - ngaphambi kokuqala kwesifanekiso sokumisa, sibeka unhlokweni ngefomethi ye-YAML echaza okunye.
Isibonelo:
---
selector: "[@][[email protected]._meta.version == `42`] | items([0].fft_config || `{}`)"
destination_filename: "fft/{{ match[0] }}.json"
file_mode: 0644
reload_daemons: [fft]
...
{{ dict(match[1]) | json(indent=2, sort_keys=True) }}
Ukuze wenze ifayela lokumisa lesevisi entsha, sengeza kuphela ifayela lesifanekiso esisha. Azikho izinguquko kwikhodi yomthombo noma isofthiwe kumaphoyinti okuba khona okudingekayo.
Yini eshintshile kusukela i-QControl yaba bukhoma? Into yokuqala futhi ebaluleke kakhulu ukulethwa okungaguquki nokuthembekile kwezibuyekezo zokucushwa kuwo wonke ama-node kunethiwekhi. Okwesibili ukuthola ithuluzi elinamandla lokuhlola ukucushwa nokwenza izinguquko kukho yithimba lethu labasekeli, kanye nabathengi besevisi.
Sikwazile ukwenza konke lokhu sisebenzisa uhlelo lokubuyekeza oluzinzile lwakamuva ukuze senze ukuxhumana kube lula phakathi kweseva yokucushwa kanye nabamukeli bokucushwa. Ukusebenzisa iphrothokholi enezendlalelo ezimbili ukusekela indlela ezimele yokuqukethwe yedatha yomzila. Ihlanganise ngempumelelo injini ekhiqiza ukulungiselelwa okusekelwe ku-Jinja kunethiwekhi yokuhlunga esabalalisiwe. Le sistimu isekela izinhlobonhlobo zezindlela zokumisa ze-peripherals yethu esabalalisiwe kanye ne-heterogeneous.
Siyabonga ngosizo lwakho ekubhaleni izinto. , , .
okuthunyelwe.
Source: www.habr.com