🥇FAQ malunga noyilo kunye nokusebenza kwiVKontakte

Imbali yokudalwa kweVKontakte ikwiWikipedia; yaxelelwa nguPavel ngokwakhe. Kubonakala ngathi wonke umntu sele emazi. Malunga nabangaphakathi, ulwakhiwo kunye nolwakhiwo lwesiza kwi-HighLoad++ Pavel undixelele ngo2010. Iiseva ezininzi ziye zavuza ukususela ngoko, ngoko siya kuhlaziya ulwazi: siya kuyicalula, sikhuphe izinto zangaphakathi, silinganise, kwaye sijonge isixhobo seVK kwimbono yobugcisa.

Alexey Akulovich (AterCattus) umphuhlisi we-backend kwiqela le-VKontakte. Umbhalo wale ngxelo yimpendulo edibeneyo kwimibuzo ebuzwa rhoqo malunga nokusebenza kweqonga, iziseko zophuhliso, iiseva kunye nokusebenzisana phakathi kwabo, kodwa hayi malunga nophuhliso, oko kukuthi. malunga nentsimbi. Ngokwahlukileyo, malunga nogcino-lwazi kunye noko i-VK inakho endaweni yoko, malunga nokuqokelela izingodo kunye nokubeka iliso kwiprojekthi yonke. Iinkcukacha phantsi kokusikwa.

Ngaphezulu kweminyaka emine ndijongene nazo zonke iintlobo zemisebenzi enxulumene ne-backend.

Ukulayisha, ukugcina, ukucubungula, ukusasaza imidiya: ividiyo, ukusasazwa okubukhoma, iaudio, iifoto, amaxwebhu.
Iziseko zophuhliso, iqonga, ukujongwa komphuhlisi, iilog, ii-cache zengingqi, i-CDN, iprotocol ye-RPC yobunini.
Ukudityaniswa neenkonzo zangaphandle: izaziso zokutyhala, ukwahlulahlula ikhonkco langaphandle, ukondla kweRSS.
Ukunceda oogxa bakhe ngemibuzo eyahlukeneyo, iimpendulo ezifuna ukuntywila kwikhowudi engaziwayo.

Ngeli xesha, ndibe nesandla kwiinxalenye ezininzi zesayithi. Ndifuna ukwabelana ngala mava.

Uyilo lwezakhiwo ngokubanzi

Yonke into, njengesiqhelo, iqala ngomncedisi okanye iqela labancedisi abamkela izicelo.

Umncedisi wangaphambili

Umncedisi wangaphambili wamkela izicelo nge-HTTPS, RTMP kunye ne-WSS.

HTTPS - ezi zizicelo zeenguqu eziphambili kunye neselula yewebhu yesayithi: vk.com kunye ne-m.vk.com, kunye nabanye abathengi abasemthethweni kunye nabangekho semthethweni be-API yethu: abathengi beselula, abathunywa. Sinolwamkelo I-RTMP-itrafikhi yosasazo lweLive kunye neeseva zangaphambili ezahlukeneyo kunye WSS-Unxibelelwano lwe-API yokuHamba.

Kwi-HTTPS kunye ne-WSS kwiiseva kufanelekile nginx. Kusasazo lwe-RTMP, kutshanje sitshintshele kwisisombululo sethu kive, kodwa ingaphaya kobubanzi bengxelo. Ukunyamezela impazamo, ezi seva zithengisa iidilesi ze-IP eziqhelekileyo kwaye zisebenze ngokwamaqela ukuze ukuba kukho ingxaki kwenye yeeseva, izicelo zomsebenzisi azilahlekanga. Kwi-HTTPS kunye ne-WSS, ezi seva ezifanayo zifihla i-traffic ukuze zithathe inxalenye yomthwalo we-CPU kuwo.

Asiyi kuthetha ngokuqhubekayo malunga ne-WSS kunye ne-RTMP, kodwa kuphela malunga nezicelo eziqhelekileyo ze-HTTPS, ezihlala zidibene neprojekthi yewebhu.

Umva

Emva komphambili kukho iiseva ezihlala ngasemva. Baqhuba izicelo ukuba iseva yangaphambili ifumana kubaxhasi.

Oku iiseva ze-kPHP, apho i-daemon ye-HTTP isebenza khona, kuba i-HTTPS sele ikhutshiwe. I-kPHP yiseva eqhuba iimodeli prefork: iqala inkqubo ebalaseleyo, iqela leenkqubo zomntwana, idlulisela iziseko zokuphulaphula kubo kwaye baqhubekisela phambili izicelo zabo. Kule meko, iinkqubo aziphinda ziqalwe phakathi kwesicelo ngasinye esisuka kumsebenzisi, kodwa ngokulula phinda umisele imeko yazo kwimo yentsusa yexabiso elinguziro-isicelo emva kwesicelo, endaweni yokuphinda uqalise.

Ukuhanjiswa kwemithwalo

Zonke ii-backends zethu azikho ichibi elikhulu loomatshini abanokuqhuba nasiphi na isicelo. Thina bona yahlulahlulwe yangamaqela ahlukeneyo: ngokubanzi, iselula, i-api, ividiyo, isiteji ... Ingxaki kwiqela elihlukeneyo lomatshini aliyi kuchaphazela bonke abanye. Kwimeko yeengxaki ngevidiyo, umsebenzisi omamela umculo akayi kukwazi malunga neengxaki. Yeyiphi i-backend yokuthumela isicelo igqitywe ngunginx ngaphambili ngokoqwalaselo.

Ukuqokelelwa kweemetriki kunye nokulinganisa kwakhona

Ukuqonda ukuba zingaphi iimoto ekufuneka sibe nazo kwiqela ngalinye, thina ungathembeli kwi-QPS. Ii-backends zihlukile, zinezicelo ezahlukeneyo, isicelo ngasinye sinobunzima obuhlukeneyo bokubala i-QPS. Yiyo loo nto thina sisebenza ngengcamango yomthwalo kumncedisi ngokubanzi - kwi-CPU kunye ne-perf.

Sinamawakawaka abancedisi abanjalo. Umncedisi wenyama ngamnye uqhuba iqela le-kPHP ukuphinda asetyenziswe kuzo zonke ii-cores (kuba i-kPHP ingumsonto omnye).

Umncedisi woMxholo

I-CS okanye iseva yesiqulatho yindawo yokugcina. I-CS yiseva egcina iifayile kwaye ikwaqhubela phambili iifayile ezilayishiweyo kunye nazo zonke iintlobo zemisebenzi ehambelanayo yangasemva eyabela umphambili wewebhu kuyo.

Sinamashumi amawaka eeseva ezibonakalayo ezigcina iifayile. Abasebenzisi bayakuthanda ukulayisha iifayile, kwaye siyakuthanda ukugcina nokwabelana ngazo. Ezinye zezi seva zivalwa ngabancedisi abakhethekileyo be-pu/pp.

ipu/pp

Ukuba uvule ithebhu yenethiwekhi kwiVK, ubone i-pu/pp.

Yintoni i-pu/pp? Ukuba sivala iseva enye emva kwenye, kukho iindlela ezimbini zokukhetha ukulayisha kunye nokukhuphela ifayile kwiseva ebivaliwe: ngqo ngokusebenzisa http://cs100500.userapi.com/path okanye ngeseva ephakathi - http://pu.vk.com/c100500/path.

I-Pu ligama lembali lokulayishwa kwefoto, kwaye i-pp yi-proxy yeefoto. Oko kukuthi, iseva enye yeyokufaka iifoto, kwaye enye yeyokufaka. Ngoku ayizizo iifoto kuphela ezilayishiweyo, kodwa igama ligciniwe.

Ezi seva phelisa iiseshoni ze-HTTPSukususa umthwalo weprosesa kwindawo yokugcina. Kwakhona, ekubeni iifayile zomsebenzisi zicutshungulwa kwezi seva, ulwazi oluncinci olugcinwe kwaba matshini, lungcono. Umzekelo, amaqhosha e-HTTPS encryption.

Kuba oomatshini bavalwa ngabanye oomatshini bethu, singakwazi ukubanika ii-IP zangaphandle “ezimhlophe”, kwaye nika "grey". Ngale ndlela sigcine echibini le-IP kwaye siqinisekisiwe ukukhusela oomatshini ekufikeleleni ngaphandle - akukho IP yokungena kuyo.

Ukuqina kwii-IPs ezabelwana ngazo. Ngokumalunga nokunyamezelana kwesiphoso, iskimu sisebenza ngokufanayo - iiseva ezininzi zomzimba zine-IP eqhelekileyo yomzimba, kwaye i-hardware phambi kwabo ikhetha apho ithumela khona isicelo. Ndiza kuthetha ngezinye iindlela kamva.

Inqaku eliphikisanayo kukuba kule meko umxhasi ugcina imidibaniso embalwa. Ukuba kukho i-IP efanayo koomatshini abaninzi - kunye nomninimzi ofanayo: pu.vk.com okanye pp.vk.com, umkhangeli we-browser unomda kwinani lezicelo zangaxeshanye kumncedisi omnye. Kodwa ngexesha le-HTTP / 2 elibiquitous, ndiyakholelwa ukuba le nto ayisasebenzi kangako.

Ukungalungi okucacileyo kwesikimu kukuba kufuneka ukupompa zonke iitrafikhi, eya kwindawo yokugcina, ngokusebenzisa enye iseva. Ekubeni simpompa itrafikhi ngoomatshini, asikwazi ukumpompa itrafikhi enzima, umzekelo, ividiyo, sisebenzisa inkqubo efanayo. Sihambisa ngokuthe ngqo - uqhagamshelo oluthe ngqo olwahlukileyo lokugcinwa okwahlukileyo ngokukodwa kwividiyo. Sisasaza umxholo olula ngeproksi.

Kungekudala sifumene inguqulelo ephuculweyo ye-proxy. Ngoku ndiza kukuxelela ukuba bahluke njani kwizinto eziqhelekileyo kwaye kutheni oku kuyimfuneko.

ilanga

NgoSeptemba 2017, i-Oracle, eyayithenge i-Sun ngaphambili, wagxotha inani elikhulu labasebenzi beLanga. Sinokuthi ngalo mzuzu inkampani iyekile ukubakho. Xa sikhetha igama lenkqubo entsha, abalawuli bethu bagqibe ekubeni bahlawule inkumbulo yale nkampani kwaye bathiywe inkqubo entsha yeLanga. Phakathi kwethu simbiza nje ngokuba “ngamalanga”.

pp waba neengxaki ezimbalwa. I-IP enye kwiqela ngalinye - i-cache engasebenziyo. Iiseva ezininzi zomzimba zabelana ngedilesi ye-IP eqhelekileyo, kwaye akukho ndlela yokulawula ukuba isicelo siya kuya kweyiphi iseva. Ngoko ke, ukuba abasebenzisi abahlukeneyo beza kwifayile efanayo, ngoko ukuba kukho i-cache kula maseva, ifayile iphelela kwi-cache yomncedisi ngamnye. Esi sisicwangciso esingasebenzi kakuhle, kodwa akukho nto inokwenziwa.

Ngenxa yoko - asinako ukwahlula umxholo, kuba asikwazi ukukhetha umncedisi othile kweli qela - bane IP eqhelekileyo. Kwakhona ngenxa yezizathu ezithile zangaphakathi esinazo bekungenzeki ukufaka abancedisi abanjalo kwimimandla. Bema kuphela eSt.

Ngelanga, satshintsha inkqubo yokukhetha. Ngoku sinayo nayiphi na indlela yokuphosa: umzila oguquguqukayo, anycast, i-daemon yokuzijonga. Umncedisi ngamnye une-IP yakhe, kodwa i-subnet eqhelekileyo. Yonke into icwangciswe ngendlela yokuba ukuba umncedisi omnye uyasilela, i-traffic isasazwa kwezinye iiseva zeqela elifanayo ngokuzenzekelayo. Ngoku kuyenzeka ukhethe iseva ethile, akukho caching engafunekiyo, kwaye ukuthembeka akuzange kuchaphazeleke.

Inkxaso yobunzima. Ngoku sinokukwazi ukufaka oomatshini bamandla ahlukeneyo njengoko kufuneka, kwaye kwakhona, xa kukho iingxaki zexeshana, utshintshe izisindo "zelanga" ezisebenzayo ukunciphisa umthwalo kubo, ukuze "baphumle" kwaye baqale ukusebenza kwakhona.

Ukwabelwa ngesazisi somxholo. Into ehlekisayo malunga ne-sharding: sidla ngokuqhekeza umxholo ukuze abasebenzisi abahlukeneyo baye kwifayile efanayo "ngelanga" elifanayo ukuze babe ne-cache eqhelekileyo.

Kutshanje siphehlelele isicelo se-"Clover". Lo ngumbuzo we-intanethi kusasazo oluphilayo, apho umphathi ubuza imibuzo kwaye abasebenzisi baphendule ngexesha langempela, bekhetha iinketho. I-app inengxoxo apho abasebenzisi banokuncokola khona. Ingaqhagamshela ngaxeshanye kusasazo abantu abangaphezu kwe-100 lamawaka. Bonke babhala imiyalezo ethunyelwa kubo bonke abathathi-nxaxheba, kwaye i-avatar iza kunye nomyalezo. Ukuba i-100 lamawaka abantu beza kwi-avatar enye "ngelanga" enye, ngoko ngamanye amaxesha inokuqengqeleka emva kwefu.

Ukuze ukwazi ukumelana nokugqabhuka kwezicelo zefayile efanayo, kungenxa yodidi oluthile lomxholo esivula iskimu esisisidenge esisasaza iifayile kuzo zonke "iilanga" ezikhoyo kummandla.

Ilanga ukusuka ngaphakathi

Reverse proxy kwi-nginx, cache nokuba kwi-RAM okanye kwiidiski ze-Optane/NVMe ezikhawulezayo. Umzekelo: http://sun4-2.userapi.com/c100500/path - ikhonkco "kwilanga", elikummandla wesine, iqela lesibini leseva. Ivala ifayile yendlela, elele ngokwasemzimbeni kumncedisi we-100500.

ikhava

Songeza enye i-node kwisikimu sethu soyilo-imeko ye-caching.

Ngezantsi ngumzobo woyilo indawo yokugcina indawo, kukho malunga nama-20 kubo. Ezi ziindawo apho ii-cache kunye "nelanga" zikhona, ezinokugcina i-traffic ngokwabo.

Oku kukugcinwa komxholo wemultimedia; akukho datha yomsebenzisi igcinwe apha-umculo kuphela, ividiyo, iifoto.

Ukumisela indawo yomsebenzisi, thina siqokelela BGP network prefixes ezibhengezwe kwimimandla. Kwimeko yokubuyela umva, kufuneka kwakhona sicazulule isiseko sedatha ye-geoip ukuba asikwazanga ukufumana i-IP ngezimaphambili. Simisela ummandla nge-IP yomsebenzisi. Kwikhowudi, sinokujonga enye okanye imimandla emininzi yomsebenzisi - ezo ngongoma asondele kakhulu kuzo ngokwejografi.

Isebenza njani?

Sibala ukuthandwa kweefayile ngokwengingqi. Kukho inani le-cache yengingqi apho umsebenzisi ahlala khona, kunye ne-identifier yefayile - sithatha esi sibini kwaye sandise ukulinganisa ngokukhuphela ngalunye.

Kwangelo xesha, iidemon - iinkonzo kwimimandla - amaxesha ngamaxesha ziza kwi-API kwaye zithi: "Ndiyi-cache enjalo, ndinike uluhlu lweefayile ezidumileyo kwingingqi yam engekabikho kum. ” I-API ihambisa iqela leefayile ezihlelwe ngokulinganisa, i-daemon iyazikhuphela, izithathele kwimimandla kwaye ihambise iifayile ukusuka apho. Lo ngumahluko osisiseko phakathi kwe-pu/pp kunye ne-Sun kwii-cache: zinika ifayile ngokwazo ngokukhawuleza, nokuba le fayile ayikho kwi-cache, kwaye i-cache ikhuphela kuqala ifayile kuyo, kwaye iqala ukuyibuyisela.

Kule meko sifumana umxholo kufutshane nabasebenzisi kunye nokusabalalisa umthwalo womnatha. Ngokomzekelo, kuphela kwi-cache yaseMoscow sisasaza ngaphezu kwe-1 Tbit / s ngexesha leeyure eziphakamileyo.

Kodwa kukho iingxaki - abancedisi be-cache abayiyo irabha. Ngomxholo odumileyo, ngamanye amaxesha akukho nethiwekhi yaneleyo yeseva eyahlukileyo. Iiseva zethu ze-cache ziyi-40-50 Gbit / s, kodwa kukho umxholo ovala ngokupheleleyo umjelo onjalo. Siqhubela phambili ekuphumezeni ukugcinwa kwekopi engaphezulu kweyodwa yeefayile ezidumileyo kummandla. Ndiyathemba ukuba sizakuyiphumeza ekupheleni konyaka.

Sajonga ulwakhiwo jikelele.

Abancedisi bangaphambili abamkela izicelo.
Ixhasa izicelo zenkqubo.
Ugcino oluvalwa ziindidi ezimbini zeeproxies.
Ugcino lwengingqi.

Yintoni engekhoyo kulo mzobo? Ewe, oovimba beenkcukacha esigcina kuzo idatha.

Iidatabase okanye iinjini

Asizibizi ngoovimba beenkcukacha, kodwa iinjini-iiNjini, kuba ngokwenyani asinazo ingcombolo ngokwengqiqo eyamkelekileyo.

Lo ngumlinganiselo oyimfuneko. Oku kwenzeka ngenxa yokuba ngo-2008-2009, xa i-VK inokukhula okuqhumayo ekuthandeni, iprojekthi yasebenza ngokupheleleyo kwi-MySQL kunye neMemcache kwaye kwakukho iingxaki. I-MySQL yayithanda ukuphahlazeka kunye nokonakalisa iifayile, emva koko yayingayi kuphinda ibuyele, kwaye iMemcache ngokuthe ngcembe yathotywa ekusebenzeni kwaye kwafuneka iqalwe kwakhona.

Kuyavela ukuba iprojekthi eyandayo yayinokugcinwa okuzingileyo, okonakalisa idatha, kunye ne-cache, ethoba isantya. Kwiimeko ezinjalo, kunzima ukuphuhlisa iprojekthi ekhulayo. Kwagqitywa ekubeni sizame ukubhala ngokutsha izinto ezibalulekileyo iprojekthi eyayijoliswe kuzo kwiibhayisekile zethu.

Isicombululo saba yimpumelelo. Kwakukho ithuba lokwenza oku, kunye nemfuneko egqithisileyo, kuba ezinye iindlela zokulinganisa zazingekho ngelo xesha. Kwakungekho qela logcino-lwazi, i-NoSQL yayingekabikho, kwakukho kuphela i-MySQL, iMemcache, iPostgreSQL-kwaye yiyo leyo.

Ukusebenza jikelele. Uphuhliso lwalukhokelwa liqela lethu labaphuhlisi be-C kwaye yonke into yenziwa ngendlela engaguqukiyo. Nokuba yeyiphi i-injini, zonke zinefomathi yefayile efanayo ebhalelwe kwidiski, iiparamitha zokuqalisa ezifanayo, imiqondiso ecutshungulwayo ngendlela efanayo, kwaye baziphathe ngokufanayo kwimeko yeemeko kunye neengxaki. Ngokukhula kweenjini, kukulungele ukuba abalawuli basebenze inkqubo - akukho zoo kufuneka zigcinwe, kwaye kufuneka baphinde bafunde indlela yokusebenzisa i-database entsha yomntu wesithathu, eyenza ukuba kube lula ukunyusa ngokukhawuleza nangokufanelekileyo. inani labo.

Iintlobo zeenjini

Iqela libhale iinjini ezimbalwa. Nazi ezinye zazo: umhlobo, iingcebiso, umfanekiso, ipdb, iileta, uluhlu, logs, memcached, meowdb, iindaba, nostradamus, ifoto, uluhlu lokudlala, pmemcached, ibhokisi yesanti, ukukhangela, ukugcinwa, ukuthanda, imisebenzi, ...

Kumsebenzi ngamnye ofuna ubume bedatha ethile okanye iinkqubo zezicelo ezingaqhelekanga, iqela C libhala injini entsha. Ngoba kutheni.

Sinenjini eyahlukileyo kukhunjulwe, efana neyoqhelekileyo, kodwa kunye neqela lezinto ezilungileyo, kwaye ezingacothiyo. Hayi iClickHouse, kodwa iyasebenza. Ifumaneka ngokwahlukeneyo pmemcached - yi le ezingisayo memcached, enokuthi igcine idatha kwidiski, ngaphezu koko, ingena kwi-RAM, ukuze ungalahlekelwa yidatha xa uqala kwakhona. Kukho iinjini ezahlukeneyo zemisebenzi yomntu ngamnye: imigca, uluhlu, iiseti - yonke into efunwa yiprojekthi yethu.

Amaqela

Ukusuka kumbono wekhowudi, akukho mfuneko yokucinga ngeenjini okanye i-database njengeenkqubo, amaziko, okanye iimeko. Ikhowudi isebenza ngokukodwa kunye namaqela, kunye namaqela eenjini - uhlobo olunye ngeqela ngalinye. Masithi kukho i-memcached cluster - liqela nje loomatshini.

Ikhowudi ayifuni ukwazi indawo ebonakalayo, ubungakanani, okanye inani leseva kwaphela. Uya kwiqela usebenzisa isazisi esithile.

Ukuze oku kusebenze, kufuneka ungeze enye into ekhoyo phakathi kwekhowudi kunye neenjini - ummeli.

Ummeli we-RPC

Ummeli ibhasi yokudibanisa, apho phantse yonke indawo isebenza. Kwangaxeshanye sinazo akukho nkonzo yafunyanwa - endaweni yoko, kukho uqwalaselo lwale proxy, oyaziyo indawo yawo onke amaqela kunye nazo zonke iishadi zeli qela. Nantsi into eyenziwa ngaba admin.

Abacwangcisi abakhathali nokuba yimalini na, phi kwaye kubiza ntoni - baya nje kwiqela. Oku kusivumela kakhulu. Xa ufumana isicelo, i-proxy iphinda iqondise isicelo, isazi apho-imisela oku ngokwayo.

Kule meko, i-proxy yindawo yokukhusela ekungaphumeleli kwenkonzo. Ukuba enye injini iyacotha okanye iphazamiseke, ngoko ummeleli uyayiqonda le nto kwaye aphendule ngokufanelekileyo kwicala lomxhasi. Oku kukuvumela ukuba ususe ukuphelelwa kwexesha - ikhowudi ayilindi ukuba injini iphendule, kodwa iyaqonda ukuba ayisebenzi kwaye kufuneka iziphathe ngendlela eyahlukileyo. Ikhowudi kufuneka ilungiselelwe into yokuba ii-database azihlali zisebenza.

Ukuphunyezwa okuthe ngqo

Ngamanye amaxesha sisafuna ngenene ukuba nolunye uhlobo lwesisombululo esingenamgangatho njengenjini. Kwangaxeshanye, kwagqitywa ekubeni singasebenzisi i-rpc-proxy yethu esele yenziwe, eyenzelwe ngokukodwa iinjini zethu, kodwa ukwenza i-proxy eyahlukileyo yomsebenzi.

Kwi-MySQL, esisenayo apha naphaya, sisebenzisa i-db-proxy, kunye neClickHouse - Indlu yaseKitten.

Isebenza ngokubanzi ngolu hlobo. Kukho iseva ethile, iqhuba i-kPHP, Hamba, iPython - ngokubanzi, nayiphi na ikhowudi enokusebenzisa iprotocol yethu yeRPC. Ikhowudi iqhuba ekuhlaleni kwi-proxy ye-RPC - iseva nganye apho ikhowudi ikhoyo iqhuba i-proxy yayo yendawo. Ngesicelo, ummeli uyayiqonda indawo emaye kuyo.

Ukuba injini enye ifuna ukuya kwelinye, nokuba ngummelwane, ihamba nge-proxy, kuba ummelwane unokuba kwenye indawo yedatha. I-injini akufanele ithembele ekwazini indawo yayo nayiphi na into ngaphandle kwayo - esi sisisombululo sethu esisemgangathweni. Kodwa, ngokuqinisekileyo, kukho iimeko :)

Umzekelo we-TL-scheme apho zonke iinjini zisebenza khona.

memcache.not_found                                = memcache.Value;
memcache.strvalue	value:string flags:int = memcache.Value;
memcache.addOrIncr key:string flags:int delay:int value:long = memcache.Value;

tasks.task
    fields_mask:#
    flags:int
    tag:%(Vector int)
    data:string
    id:fields_mask.0?long
    retries:fields_mask.1?int
    scheduled_time:fields_mask.2?int
    deadline:fields_mask.3?int
    = tasks.Task;
 
tasks.addTask type_name:string queue_id:%(Vector int) task:%tasks.Task = Long;

Le yiprotocol yokubini, eyona analogu ikufutshane eyi protobuf. I-schema ichaza kwangaphambili iindawo ezikhethiweyo, iintlobo ezinzima - izandiso ze-scalar ezakhelwe ngaphakathi, kunye nemibuzo. Yonke into isebenza ngokwale protocol.

I-RPC ngaphezulu kwe-TL ngaphezulu kwe-TCP/UDP… UDP?

Sine-protocol ye-RPC yokwenza izicelo ze-injini ezisebenza ngaphezulu kwesikimu se-TL. Konke oku kusebenza kuqhagamshelwano lwe-TCP/UDP. I-TCP iyaqondakala, kodwa kutheni sifuna i-UDP rhoqo?

I-UDP iyanceda nqanda ingxaki yenani elikhulu loqhagamshelwano phakathi kwabancedisi. Ukuba umncedisi ngamnye unommeli we-RPC kwaye, ngokubanzi, unokuya kuyo nayiphi na injini, ngoko kukho amashumi amawaka oqhagamshelwano lwe-TCP ngomncedisi ngamnye. Kukho umthwalo, kodwa awunamsebenzi. Kwimeko ye-UDP le ngxaki ayikho.

Akukho kuxhawulana ngesandla kwe-TCP engafunekiyo. Le yinkinga eqhelekileyo: xa injini entsha okanye iseva entsha iqaliswa, ezininzi izixhumanisi ze-TCP zisekwe kanye. Ngezicelo ezincinci ezikhaphukhaphu, umzekelo, ukuhlawula kwe-UDP, lonke unxibelelwano phakathi kwekhowudi kunye nenjini iipakethi ezimbini ze-UDP: enye ibhabhela kwelinye icala, eyesibini kwelinye. Olunye uhambo olujikelezayo - kwaye ikhowudi ifumene impendulo kwi-injini ngaphandle kokuxhawula isandla.

Ewe, konke kusebenza nje ngepesenti encinci kakhulu yokulahleka kwepakethi. Iprotocol inenkxaso yokuthunyelwa kwakhona kunye nokuphuma kwexesha, kodwa ukuba silahlekelwa kakhulu, siya kufumana phantse i-TCP, engeyonzuzo. Asiyiqhubeli i-UDP kwiilwandlekazi.

Sinamawaka eeseva ezinjalo, kwaye iskimu siyafana: ipakethe yeenjini ifakwe kwiseva nganye yomzimba. Ubukhulu becala zinemisonto enye ukuze zibaleke ngokukhawuleza kangangoko ngaphandle kokubhloka, kwaye zahlulwe njengezisombululo ezinomsonto omnye. Kwangaxeshanye, asinanto ithembekileyo kunezi njini, kwaye ingqwalasela eninzi ihlawulwe kugcino lwedatha oluzingisileyo.

Ukugcinwa kwedatha okuzingisileyo

Iinjini zibhala ii-binlogs. I-binlog yifayile ekupheleni apho isiganeko sotshintsho kwimeko okanye idatha yongezwa. Kwizisombululo ezahlukeneyo ibizwa ngokwahlukileyo: ilogi yokubini, UDONGA, AOF, kodwa umgaqo uyafana.

Ukuthintela i-injini ukuba ifunde kwakhona yonke i-binlog iminyaka emininzi xa iphinda iqalisa, ii-injini zibhala i-snapshots - imeko yangoku. Ukuba kuyimfuneko, bafunda kuyo kuqala, baze bagqibe ukufunda kwi-binlog. Zonke ii-binlogs zibhalwe kwifomati yokubini enye - ngokuhambelana ne-TL scheme, ukwenzela ukuba i-admins ikwazi ukulawula ngokulinganayo kunye nezixhobo zabo. Akukho mfuneko enjalo ye-snapshots. Kukho i-header jikelele ebonisa ukuba umfanekiso kabani u-int, umlingo we-injini, kwaye nguwuphi umzimba ongabalulekanga nakubani na. Le yingxaki nge-injini erekhode i-snapshot.

Ndiza kuchaza ngokukhawuleza umgaqo wokusebenza. Kukho iseva apho injini isebenza khona. Uvula i-binlog entsha engenanto yokubhala kwaye ubhale isiganeko sokutshintsha kuyo.

Ngexesha elithile, mhlawumbi uthatha isigqibo sokuthatha i-snapshot ngokwakhe, okanye ufumana umqondiso. Umncedisi wenza ifayile entsha, ibhala imeko yayo yonke kuyo, ifakela ubungakanani be binlog yangoku - offset - ukuya ekupheleni kwefayile, kwaye iqhubeka nokubhala ngokubhekele phaya. I-binlog entsha ayenziwanga.

Ngexesha elithile, xa injini iqalwa kwakhona, kuya kubakho zombini i-binlog kunye ne-snapshot kwidiski. I-injini ifunda yonke i-snapshot kwaye iphakamisa imeko yayo kwindawo ethile.

Ifunda indawo ebiyiyo ngexesha kwenziwa umfanekiso okhawulezayo kunye nobungakanani bebhlog.

Ufunda isiphelo sebhinlog ukufumana imeko yangoku kwaye uqhubeke nokubhala ezinye iziganeko. Esi sikimu esilula; zonke iinjini zethu zisebenza ngokungqinelana nayo.

Ukuphindaphindwa kwedatha

Ngenxa yoko, ukuphindaphindwa kwedatha yethu ingxelo-based — sibhala kwi-binlog hayi naliphi na iphepha elitshintshayo, kodwa oko kukuthi tshintsha izicelo. Ifana kakhulu nento ezayo kwinethiwekhi, iguqulwe kancinane kuphela.

Isikimu esifanayo asisetyenziselwa ukuphindaphinda kuphela, kodwa kwakhona ukwenza i-backups. Sine-injini - inkosi yokubhala ebhalela kwi-binlog. Nakweyiphi na enye indawo apho ii-admins ziseta khona, le bhinlog ikhutshelwe, kwaye yiyo loo nto - sine-backup.

Ukuba kuyimfuneko ukufunda ikopiUkunciphisa umthwalo wokufunda we-CPU, i-injini yokufunda iqaliswe nje, efunda isiphelo se-binlog kwaye iphumeze le miyalelo kwindawo.

I-lag apha incinci kakhulu, kwaye kunokwenzeka ukufumanisa ukuba i-replica ingakanani emva kwenkosi.

Ukwabiwa kwedatha kwiproksi ye-RPC

Kusebenza njani ukwahlulahlula? Ingaba i-proxy iyiqonda njani ukuba yeyiphi i-cluster shard emayithunyelwe kuyo? Ikhowudi ayitsho ukuba: "Thumela iishadi ezili-15!" - hayi, oku kwenziwa ngummeli.

Esona sikimu silula sisiqalo - inombolo yokuqala kwisicelo.

get(photo100_500) => 100 % N.

Lo ngumzekelo weprothokholi yokubhaliweyo okulula kwe-memcached, kodwa, ngokuqinisekileyo, imibuzo inokuba nzima kwaye yakhiwe. Umzekelo uthatha inani lokuqala kumbuzo kunye nentsalela xa yahlulwe ngobungakanani beqela.

Oku kuluncedo xa sifuna ukuba nedatha yendawo yequmrhu elinye. Masithi i-100 ngumsebenzisi okanye i-ID yeqela, kwaye sifuna yonke idatha yequmrhu elinye ibe kwindawo enye kwimibuzo enzima.

Ukuba asikhathali ukuba izicelo zisasazwa njani na kwiqela, kukho enye indlela - ukukrazula isitya sonke.

hash(photo100_500) => 3539886280 % N

Sifumana kwakhona i-hash, intsalela yolwahlulo kunye nenombolo ye-shard.

Zombini ezi zikhetho zisebenza kuphela ukuba silungiselele into yokuba xa sinyusa ubungakanani beqela, siya kulwahlula okanye sandise ngamaxesha amaninzi. Umzekelo, sineeshadi ezili-16, asinako ngokwaneleyo, sifuna ngaphezulu - sinokufumana ngokukhuselekileyo i-32 ngaphandle kwexesha lokuphumla. Ukuba sifuna ukwandisa hayi ii-multiples, kuya kubakho ixesha lokuphumla, kuba asiyi kukwazi ukwahlula ngokuchanekileyo yonke into ngaphandle kokulahlekelwa. Olu khetho luluncedo, kodwa alusoloko.

Ukuba sifuna ukongeza okanye ukususa inani elingenasizathu leeseva, sisebenzisa I-hashing ehambelanayo kwiringi a la Ketama. Kodwa kwangaxeshanye, siphulukana ngokupheleleyo nendawo yedatha; kufuneka sidibanise isicelo kwiqela ukuze iqhekeza ngalinye libuyisele impendulo yalo encinci, kwaye emva koko sidibanise iimpendulo kwi-proxy.

Kukho izicelo ezingqale kakhulu. Ijongeka ngolu hlobo: Iproxy ye-RPC ifumana isicelo, imisela ukuba yeyiphi iqela emayiye kuyo kwaye imisele ishard. Emva koko kukho iinkosi zokubhala, okanye, ukuba i-cluster ine-replica inkxaso, ithumela kwi-replica kwimfuno. Ummeli uyakwenza konke oku.

Izigodo

Sibhala iinkuni ngeendlela ezininzi. Eyona nto icacileyo kwaye ilula bhala iilog kwi-memcache.

ring-buffer: prefix.idx = line

Kukho isimaphambili esiphambili - igama lelog, umgca, kwaye kukho ubungakanani bale log - inani lemigca. Sithatha inani elingaqhelekanga ukusuka ku-0 ukuya kwinani lemigca thabatha 1. Isitshixo kwi-memcache sisimaphambili esidityaniswe neli nani. Sigcina umgca welog kunye nexesha langoku kwixabiso.

Xa kuyimfuneko ukufunda iinkuni, senza Fumana ezininzi zonke izitshixo, zihlelwe ngexesha, kwaye ngaloo ndlela ufumane ushicilelo lwemveliso ngexesha lokwenyani. Iskimu sisetyenziswe xa ufuna ukulungisa into ethile kwimveliso ngexesha langempela, ngaphandle kokuphula nantoni na, ngaphandle kokumisa okanye ukuvumela i-traffic kwabanye oomatshini, kodwa le log ayihlali ixesha elide.

Ukugcinwa okuthembekileyo kwezigodo sinenjini logs-injini. Yiyo kanye le nto yadalwa kwaye isetyenziswa ngokubanzi kwinani elikhulu lamaqela. Elona qela likhulu endilaziyo leevenkile ngama-600 TB eelogs ezipakishiweyo.

I-injini indala kakhulu, kukho amaqela asele eneminyaka eyi-6-7 ubudala. Kukho iingxaki kunye nayo esizama ukuyicombulula, umzekelo, saqala ukusebenzisa ngokusebenzayo iClickHouse ukugcina izingodo.

Ukuqokelela iilog kwiClickHouse

Lo mzobo ubonisa indlela esihamba ngayo kwiinjini zethu.

Kukho ikhowudi eya ekuhlaleni nge-RPC ukuya kwi-RPC-proxy, kwaye iyayiqonda indawo yokuya kwi-injini. Ukuba sifuna ukubhala iilogi kwiClickHouse, kufuneka sitshintshe iinxalenye ezimbini kwesi sikimu:

buyisela enye injini ngeClickHouse;
endaweni yommeli we-RPC, engakwaziyo ukufikelela kwi-ClickHouse, kunye nesisombululo esinokuthi, kwaye nge-RPC.

I-injini ilula - siyibuyisela ngomncedisi okanye iqela labancedisi ngeClickHouse.

Kwaye ukuya kwiClickHouse, siyenzile KittenHouse. Ukuba sisuka ngokuthe ngqo kwi-KittenHouse ukuya kwi-ClickHouse, ayiyi kumelana. Ngaphandle kwezicelo, yongeza ukusuka kuqhagamshelo lwe-HTTP yenani elikhulu loomatshini. Ukuze iskimu sisebenze, kumncedisi nge ClickHouse iproxy yendawo engasemva iyaphakanyiswa, ebhalwe ngendlela enokuthi ikwazi ukumelana nomthamo ofunekayo wokudibanisa. Isenokukhusela idatha ngaphakathi kwayo ngokuthembekileyo.

Ngamanye amaxesha asifuni kuphumeza iskimu se-RPC kwizisombululo ezingezizo ezisemgangathweni, umzekelo, kwinginx. Ke ngoko, iKittenHouse inamandla okufumana iilogi nge-UDP.

Ukuba umthumeli kunye nomamkeli welog basebenza kumatshini omnye, ngoko ke amathuba okuba ulahlekelwe yipakethi ye-UDP ngaphakathi komnini wendawo uphantsi kakhulu. Njengokuchasana phakathi kwesidingo sokuphumeza i-RPC kwisisombululo somntu wesithathu kunye nokuthembeka, sisebenzisa nje ukuthumela i-UDP. Siza kubuyela kwesi sikimu kamva.

Ukubeka iliso

Sineentlobo ezimbini zelogi: ezo ziqokelelwe ngabalawuli kwiiseva zabo kunye nezo zibhalwe ngabaphuhlisi kwikhowudi. Ziyahambelana neentlobo ezimbini zeemethrikhi: inkqubo kunye nemveliso.

Iimetriki zesistim

Isebenza kuzo zonke iiseva zethu INetdata, eqokelela amanani kwaye iwathumele ku Ikhabhoni yegraphite. Ngoko ke, i-ClickHouse isetyenziswe njengenkqubo yokugcina, kwaye kungekhona i-Whisper, umzekelo. Ukuba kuyimfuneko, unokufunda ngokuthe ngqo kwiClickHouse, okanye usebenzise IGrafana kwiimetrics, iigrafu kunye neengxelo. Njengabaphuhlisi, sinofikelelo olwaneleyo kwi-Netdata kunye ne-Grafana.

Iimetriki zemveliso

Ukuze kube lula, sibhale izinto ezininzi. Umzekelo, kukho uluhlu lwemisebenzi eqhelekileyo ekuvumela ukuba ubhale amanani, amanani angafaniyo amaxabiso kwizibalo, ezithunyelwa kwenye indawo phambili.

statlogsCountEvent   ( ‘stat_name’,            $key1, $key2, …)
statlogsUniqueCount ( ‘stat_name’, $uid,    $key1, $key2, …)
statlogsValuetEvent  ( ‘stat_name’, $value, $key1, $key2, …)

$stats = statlogsStatData($params)

Emva koko, sinokusebenzisa izihluzi zokuhlela kunye nokwenza amaqela kwaye senze yonke into esiyifunayo kwizibalo - ukwakha iigrafu, ukuqwalasela i-Watchdogs.

Sibhala kakhulu ezininzi metrics inani leziganeko lisuka kwi-600 billion ukuya kwi-1 trillion ngosuku. Nangona kunjalo, sifuna ukuzigcina ubuncinane iminyaka embalwaukuqonda iintsingiselo kwiimethrikhi. Ukuyidibanisa yingxaki enkulu esingekayisombululi. Ndiza kukuxelela ukuba ibisebenza njani le minyaka imbalwa idlulileyo.

Sinemisebenzi ebhala ezi metrics kwi-memcache yasekhayaukunciphisa inani lamangeno. Kanye ngexesha elifutshane yaziswa ekuhlaleni izibalo-daemon iqokelela zonke iirekhodi. Emva koko, idemon idibanisa iimetrics zibe ngamanqanaba amabini amaseva iinkuni-abaqokeleli, edibanisa izibalo ezivela kwiqela loomatshini bethu ukuze umaleko osemva kwabo ungafi.

Ukuba kuyimfuneko, sinokubhala ngokuthe ngqo kwiilogi-abaqokeleli.

Kodwa ukubhala ukusuka kwikhowudi ngokuthe ngqo kubaqokeleli, ukudlula i-stas-daemom, sisisombululo esibi kakhulu kuba sinyusa umthwalo kumqokeleli. Isisombululo sifanelekile kuphela ukuba ngenxa yesizathu esithile asikwazi ukuphakamisa i-memcache stats-daemon kumatshini, okanye iphazamisekile kwaye siye ngokuthe ngqo.

Okulandelayo, iilog-abaqokeleli badibanisa izibalo zibe meoDB - esi sisiseko sedatha yethu, esinokugcina ii-metrics.

Emva koko sinokwenza ukhetho lwebinary "kufuphi-SQL" kwikhowudi.

Uvavanyo

Ehlotyeni le-2018, sasine-hackathon yangaphakathi, kwaye ingcamango yafika ukuzama ukutshintsha inxalenye ebomvu yomzobo kunye nento enokugcina i-metrics kwi-ClickHouse. Sinelogi kwiClickHouse - kutheni ungazami?

Sasinesicwangciso esibhale iingodo ngeKittenHouse.

Sagqiba yongeza enye "* Indlu" kumzobo, eya kufumana ngokuthe ngqo i-metrics kwifomathi njengoko ikhowudi yethu ibhala nge-UDP. Ize ke le *Ndlu izijike zibe zifakelo, njengezigodo, eziqondwa yiKittenHouse. Unokuhambisa ngokugqibeleleyo ezi zigodo kwiClickHouse, ekufuneka ikwazi ukuzifunda.

Iskimu esine-memcache, i-stats-daemon kunye ne-logs-collectors database ithathelwa indawo le.

Kukho ukuthunyelwa kwekhowudi apha, ebhalwe ekuhlaleni kwi-StatsHouse.
I-StatsHouse ibhala i-UDP metrics, esele iguqulelwe kwi-SQL efakelwayo, kwi-KittenHouse kwiibhetshi.
IKittenHouse ibathumela kwiClickHouse.
Ukuba sifuna ukuzifunda, ngoko sizifunde ngokudlula i-StatsHouse - ngokuthe ngqo kwi-ClickHouse sisebenzisa iSQL eqhelekileyo.

Ngaba isekhona ulingo, kodwa siyayithanda indlela eyenzeka ngayo. Ukuba silungisa iingxaki ngeskimu, mhlawumbi siya kutshintshela kuyo ngokupheleleyo. Ngokomntu, ndiyathemba ukuba kunjalo.

Iskimu ayilondolozi intsimbi. Abancedisi abambalwa bayafuneka, ii-stats-daemons zendawo kunye nabaqokeleli belogi abadingeki, kodwa i-ClickHouse ifuna umncedisi omkhulu kunabo bakwiskimu sangoku. Iiseva ezimbalwa ziyafuneka, kodwa kufuneka zibiza kakhulu kwaye zibe namandla ngakumbi.

Hambisa

Okokuqala, makhe sijonge ukuthunyelwa kwe-PHP. Siyaphuhlisa kwi yiya: sebenzisa GitLab и I-TeamCity ukusasazwa. Amasebe ophuhliso adityaniswa abe yi-master branch, ukusuka kumphathi wovavanyo adityaniswa abe yiqonga, kwaye ukusuka kwinqanaba ukuya kwimveliso.

Ngaphambi kokuthunyelwa, isebe lemveliso yangoku kunye neyokuqala ithathwa, kwaye iifayile ezihlukeneyo zicatshangelwa kuzo - utshintsho: lwenziwe, lucinyiwe, lutshintshiwe. Olu tshintsho lurekhodwa kwi-binlog ye-injini ekhethekileyo ye-copyfast, enokuthi iphinde iphinde iphinde iphinde iphinde iphinde iguqule kwiinqanawa zethu zonke zeseva. Into esetyenziswa apha ayikokukopa ngokuthe ngqo, kodwa ukuphindaphinda intlebendwane, xa umncedisi omnye ethumela utshintsho kubamelwane abasondeleyo, abo kubamelwane babo, njalo njalo. Oku kukuvumela ukuba uhlaziye ikhowudi kumashumi kunye neeyunithi zemizuzwana kuzo zonke iinqanawa. Xa utshintsho lufikelela kwi-replica yendawo, lusebenzisa ezi ziqwenga kuyo inkqubo yefayile yendawo. I-Rollback iphinda iqhutywe ngokweskimu esifanayo.

Sikwasebenzisa i-kPHP kakhulu kwaye inophuhliso lwayo yiya ngokomzobo ongentla. Ukususela oku Umncedisi we-HTTP wokubini, ngoko ke asikwazi ukuvelisa idiff - ukukhululwa kokubini kunobunzima obungamakhulu e-MB. Ngoko ke, kukho enye inketho apha - uguqulelo lubhalelwe binlog copyfast. Ngolwakhiwo ngalunye luyanda, kwaye ngexesha lokubuyisela kwakhona luyanda. Inguqulelo iphindaphindwe kwiiseva. Abakhupheli bendawo babona ukuba inguqulelo entsha ingenile kwi-binlog, kwaye ngokuphindaphinda intlebendwane efanayo bathathela inguqulelo yamva nje yokubini, ngaphandle kokudinwa umncedisi wethu omkhulu, kodwa basasaza ngononophelo umthwalo kuwo wonke umsebenzi womnatha. Yintoni elandelayo ukuqaliswa kwakhona okuthandekayo kuguqulelo olutsha.

Kwiinjini zethu, ezikwayi-binaries, iskimu siyafana kakhulu:

isebe le-git master;
yokubini kwi .deb;
inguqulelo ibhalwe kwi-binlog copyfast;
iphindaphindwe kwiiseva;
umncedisi ukhupha entsha .dep;
dpkg -i;
ukwazisa kwakhona okuthandekayo kuguqulelo olutsha.

Umahluko kukuba i-binary yethu ipakishwe koovimba .deb, kwaye xa bempompa ngaphandle dpkg -i zibekwe kwisixokelelwano. Kutheni i-kPHP ibekwa njengokubini, kwaye iinjini zibekwe njenge-dpkg? Kwenzeka ngolo hlobo. Iyasebenza - ungayichukumisi.

Amakhonkco aluncedo:

Ingxelo ka-Anton Kiryushkin "Umlawuli wenkqubo yeVkontakte. Njani?" kunye neenkcukacha malunga ne-copyfast kunye nokuhleba.
Ingxelo nguYuri Nasretdinov "Indlela i-VK eyifaka ngayo idatha kwi-CLickHouse ukusuka kumashumi amawaka eeseva".
Ingxelo yam "Uyilo lweprojekthi ekhulayo usebenzisa umzekelo weVKontakte", kodwa ngokwembono yophuhliso, hayi ihardware.

U-Alexey Akulovich ungomnye wabo, njengenxalenye yeKomiti yeNkqubo, inceda PHP Russia nge-17 kaMeyi iya kuba sesona siganeko sikhulu kubaphuhlisi be-PHP kumaxesha amva nje. Jonga ukuba yintoni iPC epholileyo esinayo, yintoni izithethi (ezimbini kuzo ziphuhlisa i-PHP engundoqo!) - kubonakala ngathi yinto ongenakuyiphosa xa ubhala i-PHP.

umthombo: www.habr.com

FAQ malunga nezakhiwo kunye nomsebenzi weVKontakte