🥇FAQ mayelana nezakhiwo nokusebenza ku-VKontakte

Umlando wokudalwa kwe-VKontakte uku-Wikipedia; yatshelwa nguPavel ngokwakhe. Kubonakala sengathi wonke umuntu useyamazi. Mayelana nabangaphakathi, ukwakheka kanye nesakhiwo sesayithi ku-HighLoad++ Pavel wangitshela emuva ngo-2010. Amaseva amaningi aputshukile kusukela ngaleso sikhathi, ngakho-ke sizobuyekeza imininingwane: sizoyihlukanisa, sikhiphe okungaphakathi, sikale, bese sibheka idivayisi ye-VK ngokombono wezobuchwepheshe.

Alexey Akulovich (AterCattus) unjiniyela we-backend eqenjini le-VKontakte. Okulotshiweyo kwalo mbiko kuyimpendulo ehlangene yemibuzo evame ukubuzwa mayelana nokusebenza kweplatifomu, ingqalasizinda, amaseva kanye nokusebenzisana phakathi kwabo, kodwa hhayi mayelana nentuthuko, okungukuthi mayelana nensimbi. Ngokwehlukana, mayelana nemininingwane yolwazi kanye nalokho i-VK enakho esikhundleni salokho, mayelana nokuqoqa izingodo nokuqapha yonke iphrojekthi iyonke. Imininingwane ngaphansi kokusikwa.

Sekuyiminyaka engaphezu kwemine ngibhekene nazo zonke izinhlobo zemisebenzi ehlobene ne-backend.

Ukulayisha, ukugcina, ukucubungula, ukusabalalisa imidiya: ividiyo, ukusakaza bukhoma, umsindo, izithombe, imibhalo.
Ingqalasizinda, inkundla, ukuqapha konjiniyela, izingodo, izinqolobane zesifunda, i-CDN, iphrothokholi ye-RPC yokuphathelene.
Ukuhlanganiswa nezinsizakalo zangaphandle: izaziso zohlelo lokusebenza, ukudluliswa kwesixhumanisi sangaphandle, okuphakelayo kwe-RSS.
Ukusiza ozakwabo ngemibuzo ehlukahlukene, izimpendulo ezidinga ukutshuza kukhodi engaziwa.

Ngalesi sikhathi, ngaba nesandla ezingxenyeni eziningi zesayithi. Ngifuna ukwabelana ngalokhu okuhlangenwe nakho.

Izakhiwo ezijwayelekile

Konke, njengokujwayelekile, kuqala ngeseva noma iqembu lamaseva amukela izicelo.

Iseva yangaphambili

Iseva yangaphambili yamukela izicelo nge-HTTPS, i-RTMP ne-WSS.

I-HTTPS - lezi yizicelo zezinguqulo zewebhu eziyinhloko nezamaselula zesayithi: i-vk.com ne-m.vk.com, namanye amaklayenti asemthethweni nangekho emthethweni we-API yethu: amaklayenti eselula, izithunywa. Sinendawo yokwamukela izivakashi I-RTMP-i-traffic yokusakaza bukhoma ngamaseva angaphambili ahlukene kanye I-WSS- Ukuxhumana kwe-Streaming API.

Ku-HTTPS ne-WSS kumaseva kufanelekile nginx. Ngokusakaza kwe-RTMP, sisanda kushintshela kusixazululo sethu kive, kodwa kungaphezu kobubanzi bombiko. Ngokubekezelela amaphutha, lawa maseva akhangisa amakheli e-IP avamile futhi enze ngamaqembu ukuze uma kunenkinga kwenye yamaseva, izicelo zabasebenzisi azilahleki. Ku-HTTPS ne-WSS, la maseva afanayo abethela ithrafikhi ukuze athathe ingxenye yomthwalo we-CPU kuwo.

Ngeke siqhubeke sikhuluma nge-WSS ne-RTMP, kodwa mayelana nezicelo ezijwayelekile ze-HTTPS, ezivame ukuhlotshaniswa nephrojekthi yewebhu.

Emuva emuva

Ngemuva kwengaphambili kuvame ukuba namaseva angemuva. Bacubungula izicelo ezitholwe yiseva yangaphambili kumakhasimende.

Yilokho amaseva we-kPHP, lapho i-daemon ye-HTTP esebenza kuyo, ngoba i-HTTPS isivele isuswe ukubethela. I-kPHP iyiseva esebenza amamodeli we-prefork: iqala inqubo eyinhloko, iqoqo lezinqubo zengane, idlulisela amasokhethi okulalela kubo futhi bacubungule izicelo zabo. Kulokhu, izinqubo aziqalwa kabusha phakathi kwesicelo ngasinye esivela kumsebenzisi, kodwa mane nje usethe kabusha isimo sazo sibe isimo sangempela senani elinguziro - isicelo ngemva kwesicelo, esikhundleni sokuqala kabusha.

Ukusabalalisa ukulayisha

Wonke ama-backends ethu akulona iqoqo elikhulu lemishini engacubungula noma yisiphi isicelo. Thina bona ihlukaniswe ngamaqembu ahlukene: jikelele, iselula, i-api, ividiyo, isiteji... Inkinga eqenjini elihlukile lemishini ngeke ithinte bonke abanye. Uma kunezinkinga ngevidiyo, umsebenzisi olalela umculo ngeke azi ngezinkinga. Iyiphi i-backend yokuthumela isicelo kuyo inqunywa yi-nginx ngaphambili ngokusho kwe-config.

Ukuqoqwa kwemethrikhi nokulinganisa kabusha

Ukuze siqonde ukuthi zingaki izimoto okudingeka sibe nazo eqenjini ngalinye, thina unganciki ku-QPS. Ama-backend ahlukile, anezicelo ezihlukene, isicelo ngasinye sinobunzima obuhlukile bokubala i-QPS. Yingakho thina sisebenza ngomqondo womthwalo kuseva iyonke - ku-CPU kanye ne-perf.

Sinezinkulungwane zamaseva anjalo. Iseva ngayinye ebonakalayo isebenzisa iqembu le-kPHP ukuze igaywe kabusha wonke ama-cores (ngoba i-kPHP iwuchungechunge olulodwa).

Iseva Yokuqukethwe

I-CS noma iseva yokuqukethwe iyisitoreji. I-CS iyiseva egcina amafayela futhi iphinde icubungule amafayela alayishiwe nazo zonke izinhlobo zemisebenzi evumelanayo yangemuva enikezwa indawo ephambili yewebhu.

Sinamashumi ezinkulungwane zamaseva aphathekayo agcina amafayela. Abasebenzisi bayathanda ukulayisha amafayela, futhi siyathanda ukuwagcina nokwabelana ngawo. Amanye alawa maseva avaliwe amaseva akhethekile e-pu/pp.

pu/pp

Uma uvule ithebhu yenethiwekhi ku-VK, ubone i-pu/pp.

Yini i-pu/pp? Uma sivala iseva eyodwa ngemva kwenye, khona-ke kukhona izinketho ezimbili zokulayisha nokulanda ifayela kuseva eyayivaliwe: ngqo ngokusebenzisa http://cs100500.userapi.com/path noma ngeseva ephakathi - http://pu.vk.com/c100500/path.

I-Pu igama elingokomlando lokulayishwa kwesithombe, futhi u-pp ungummeleli wesithombe. Okusho ukuthi, iseva eyodwa ngeyokulayisha izithombe, kanti enye ngeyokulayisha. Manje akuzona kuphela izithombe ezilayishiwe, kodwa igama ligciniwe.

Lezi ziphakeli nqamula izikhathi ze-HTTPSukususa umthwalo weprosesa endaweni yokugcina. Futhi, njengoba amafayela omsebenzisi ecutshungulwa kulawa maseva, ulwazi olungabucayi olugcinwe kule mishini, lungcono. Isibonelo, okhiye bokubethela be-HTTPS.

Njengoba imishini ivaliwe ngeminye imishini yethu, singakwazi ukukhokhela ama-IP angaphandle "amhlophe", futhi nika "grey". Ngale ndlela silondoloze echibini le-IP futhi siqinisekisiwe ukuthi sizovikela imishini ekufinyeleleni ngaphandle - ayikho i-IP yokungena kuyo.

Ukuqina phezu kwama-IP abiwe. Ngokuphathelene nokubekezelelana kwamaphutha, uhlelo lusebenza ngendlela efanayo - amaseva angokwenyama amaningana ane-IP evamile ngokomzimba, futhi i-hardware ephambi kwabo ikhetha ukuthi izosithumela kuphi isicelo. Ngizokhuluma ngezinye izinketho kamuva.

Iphuzu eliphikisanayo ukuthi kuleli cala iklayenti ligcina ukuxhumana okumbalwa. Uma kukhona i-IP efanayo yemishini eminingana - ngomsingathi ofanayo: pu.vk.com noma pp.vk.com, isiphequluli seklayenti sinomkhawulo enanini lezicelo ngesikhathi esisodwa kumsingathi oyedwa. Kepha ngesikhathi se-HTTP/2 etholakala yonke indawo, ngikholwa ukuthi lokhu akusabalulekile kangako.

Ububi obusobala bohlelo ukuthi kufanele ukupompa yonke ithrafikhi, eya endaweni yokugcina, ngesinye iseva. Njengoba simpompa ithrafikhi ngemishini, asikwazi okwamanje ukumpompa ithrafikhi enkulu, isibonelo, ividiyo, sisebenzisa uhlelo olufanayo. Siyidlulisela ngqo - ukuxhumana okuqondile okuhlukile kwezindawo zokugcina ezihlukene eziqondiswe ngqo ividiyo. Sidlulisela okuqukethwe okulula ngommeleli.

Esikhathini esingeside esidlule sithole inguqulo ethuthukisiwe yommeleli. Manje ngizokutshela ukuthi zihluke kanjani kwezijwayelekile nokuthi kungani lokhu kudingekile.

Sun

NgoSepthemba 2017, i-Oracle, eyayithenge i-Sun ngaphambilini, waxosha inqwaba yabasebenzi baseLangeni. Singasho ukuthi okwamanje inkampani ayisekho. Lapho bekhetha igama lesistimu entsha, abalawuli bethu banqume ukuhlonipha inkumbulo yale nkampani futhi baqamba isistimu entsha ethi Sun. Phakathi kwethu simane simbiza ngokuthi “amalanga”.

pp ube nezinkinga ezimbalwa. I-IP eyodwa ngeqembu - inqolobane engasebenzi. Amaseva aphathekayo amaningana abelana ngekheli le-IP elivamile, futhi ayikho indlela yokulawula ukuthi isicelo sizoya kuyiphi iseva. Ngakho-ke, uma abasebenzisi abahlukene beza ngefayela elifanayo, khona-ke uma kukhona inqolobane kulawa maseva, ifayela ligcina kunqolobane yeseva ngayinye. Lolu wuhlelo olungasebenzi kahle, kodwa akukho obekungenziwa.

Ngenxa yalokho - asikwazi ukwabelana ngokuqukethwe, ngoba asikwazi ukukhetha iseva ethile yaleli qembu - bane-IP evamile. Futhi ngenxa yezizathu ezithile zangaphakathi esinazo akukwazanga ukufaka amaseva anjalo ezifundeni. Bama eSt. Petersburg kuphela.

Ngelanga, sashintsha uhlelo lokukhetha. Manje sesinayo umzila wanoma yimuphi ukusakaza: umzila oguquguqukayo, i-anycast, i-daemon yokuzihlola. Iseva ngayinye ine-IP yayo ngayinye, kodwa i-subnet evamile. Konke kulungiselelwa ngendlela yokuthi uma iseva eyodwa ihluleka, ithrafikhi isakazwa kuwo wonke amaseva eqembu elifanayo ngokuzenzakalelayo. Manje kungenzeka ukukhetha iseva ethile, akukho ukulondolozwa kwesikhashana okungafuneki, nokwethembeka akuzange kuthinteke.

Ukwesekwa kwesisindo. Manje singakwazi ukufaka imishini yamandla ahlukene njengoba kudingeka, futhi, uma kunezinkinga zesikhashana, shintsha izisindo "zelanga" ezisebenzayo ukuze zinciphise umthwalo kuzo, ukuze "ziphumule" futhi ziqale ukusebenza futhi.

Ukwabelana nge-id yokuqukethwe. Into ehlekisayo mayelana nokwahlukanisa: ngokuvamile sihlukanisa okuqukethwe ukuze abasebenzisi abahlukene baye kufayela elifanayo “ngelanga” elifanayo ukuze babe nenqolobane evamile.

Sisanda kwethula uhlelo lokusebenza lwe-"Clover". Lona umbuzo we-inthanethi ekusakazeni bukhoma, lapho umsingathi ebuza imibuzo futhi abasebenzisi baphendule ngesikhathi sangempela, bekhetha izinketho. Uhlelo lokusebenza lunengxoxo lapho abasebenzisi bangaxoxa khona. Ingakwazi ukuxhuma ngesikhathi esisodwa ekusakazweni abantu abangaphezu kwezinkulungwane eziyi-100. Bonke babhala imilayezo ethunyelwa kubo bonke abahlanganyeli, futhi i-avatar iza nomlayezo. Uma abantu abayizinkulungwane eziyi-100 beza ku-avatar eyodwa "ngelanga" elilodwa, khona-ke kwesinye isikhathi ingaginqa ngemuva kwefu.

Ukuze umelane nokuqhuma kwezicelo zefayela elifanayo, kungenxa yohlobo oluthile lokuqukethwe lapho sivula khona isikimu esiwubulima esisabalalisa amafayela kuwo wonke "amalanga" atholakalayo esifundeni.

Ilanga ngaphakathi

Buyisela ummeleli ku-nginx, inqolobane kungaba ku-RAM noma kumadiski e-Optane/NVMe asheshayo. Isibonelo: http://sun4-2.userapi.com/c100500/path - isixhumanisi "elanga", esitholakala esifundeni sesine, iqembu lesibili leseva. Ivala ifayela lendlela, elilele ngokoqobo kuseva 100500.

ikhava

Sengeza enye i-node ohlelweni lwethu lwezakhiwo - indawo yokugcina isikhashana.

Ngezansi kunomdwebo wesakhiwo izinqolobane zesifunda, cishe zingama-20. Lezi yizindawo lapho ama-caches kanye "namalanga" atholakala khona, angagcina ithrafikhi ngokwawo.

Lokhu ukugcinwa kwesikhashana kokuqukethwe kwe-multimedia; ayikho idatha yomsebenzisi egcinwe lapha - umculo, ividiyo, izithombe kuphela.

Ukucacisa isifunda somsebenzisi, thina siqoqa iziqalo zenethiwekhi ye-BGP ezimenyezelwe ezifundeni. Esimeni sokubuyela emuva, kufanele futhi sihlaziye isizindalwazi se-geoip uma singakwazanga ukuthola i-IP ngeziqalo. Sinquma isifunda nge-IP yomsebenzisi. Kukhodi, singabheka isifunda esisodwa noma ngaphezulu somsebenzisi - lawo maphuzu asondelene kakhulu nawo ngokwendawo.

Isebenza kanjani?

Sibala ukuduma kwamafayela ngesifunda. Kunenombolo yenqolobane yesifunda lapho umsebenzisi etholakala khona, kanye nesihlonzi sefayela - sithatha lokhu kupheya futhi sikhulise isilinganiso ngokulanda ngakunye.

Ngaso leso sikhathi, amademoni - izinsizakalo ezifundeni - ngezikhathi ezithile eza ku-API futhi athi: "Ngiyinqolobane enjalo, nginike uhlu lwamafayela athandwa kakhulu esifundeni sami angakabi kimi. ” I-API iletha inqwaba yamafayela ahlungwe ngokulinganisa, i-daemon iyawalanda, iwayise ezifundeni futhi ilethe amafayela kusukela lapho. Lona umehluko oyisisekelo phakathi kwe-pu/pp kanye ne-Sun ezinqolobaneni: banikeza ifayela ngokwabo ngokushesha, noma ngabe leli fayela lingekho kunqolobane, futhi inqolobane iqale izidawunilodele yona ifayela, bese iqala ukulibuyisela.

Kulokhu sithola okuqukethwe eduze nabasebenzisi nokusabalalisa umthwalo wenethiwekhi. Isibonelo, kuphela kusuka kunqolobane yaseMoscow sisakaza ngaphezu kwe-1 Tbit / s ngesikhathi samahora aphezulu.

Kodwa kukhona izinkinga - amaseva enqolobane awawona amarabha. Kokuqukethwe okudume kakhulu, kwesinye isikhathi ayikho inethiwekhi eyanele yeseva ehlukile. Amaseva ethu enqolobane angu-40-50 Gbit/s, kodwa kukhona okuqukethwe okuvala ngokuphelele isiteshi esinjalo. Sibheke ekusebenziseni ukugcinwa kwekhophi engaphezu kweyodwa yamafayela adumile esifundeni. Ngethemba ukuthi sizowusebenzisa ekupheleni konyaka.

Sibheke i-architecture evamile.

Amaseva angaphambili amukela izicelo.
Isekela izicelo zenqubo.
Isitoreji esivalwe yizinhlobo ezimbili zama-proxi.
Izinqolobane zesifunda.

Yini engekho kulo mdwebo? Yebo, izizindalwazi lapho sigcina khona idatha.

Imininingwane noma izinjini

Asizibizi ngokugcina imininingwane, kodwa izinjini - Izinjini, ngoba cishe asinazo imininingwane yolwazi ngomqondo owamukelekayo.

Lesi yisinyathelo esidingekayo. Lokhu kwenzeka ngoba ngo-2008-2009, lapho i-VK ikhula kakhulu ekudumeni, iphrojekthi yasebenza ngokuphelele ku-MySQL ne-Memcache futhi kwakukhona izinkinga. I-MySQL yayithanda ukuphahlazeka futhi yonakalise amafayela, ngemva kwalokho yayingeke ilulame, futhi i-Memcache yonakaliswa kancane kancane ekusebenzeni futhi kwadingeka iqalwe kabusha.

Kuvele ukuthi iphrojekthi ethandwa kakhulu inesitoreji esiqhubekayo, sonakalisa idatha, kanye nenqolobane, ehlisa ijubane. Ezimweni ezinjalo, kunzima ukuthuthukisa iphrojekthi ekhulayo. Kwanqunywa ukuba sizame ukubhala kabusha izinto ezibucayi umsebenzi owawugxile kuzo emabhayisikili ethu.

Isixazululo saba yimpumelelo. Kwakunethuba lokwenza lokhu, kanye nesidingo esidlulele, ngoba ezinye izindlela zokukala zazingekho ngaleso sikhathi. Besingekho inqwaba yolwazi, i-NoSQL ibingakatholakali, bekukhona i-MySQL, iMemcache, iPostrgreSQL kuphela - futhi yilokho kuphela.

Ukusebenza kwendawo yonke. Ukuthuthukiswa bekuholwa ithimba lethu labathuthukisi be-C futhi konke kwenziwa ngendlela engaguquki. Kungakhathalekile ukuthi iyini injini, zonke zinefomethi efanayo yefayela ebhalelwe idiski, imingcele efanayo yokuqalisa, amasiginali acutshungulwe ngendlela efanayo, futhi aziphathe cishe ngendlela efanayo uma kunezimo nezinkinga. Ngokukhula kwezinjini, kulula kubaphathi ukuthi basebenzise uhlelo - ayikho i-zoo edinga ukunakekelwa, futhi kufanele bafunde kabusha ukuthi bangasebenzisa kanjani i-database entsha yomuntu wesithathu, okwenze kwaba nokwenzeka ngokushesha futhi ukwandisa kalula inombolo yabo.

Izinhlobo zezinjini

Iqembu libhale izinjini ezimbalwa impela. Nazi ezinye zazo: umngane, amacebo, isithombe, ipdb, izincwadi, uhlu, izingodo, memcached, meowdb, izindaba, nostradamus, isithombe, uhlu lwadlalwayo, pmemcached, sandbox, search, storage, likes, tasks, ...

Kumsebenzi ngamunye odinga isakhiwo sedatha ethile noma ucubungula izicelo ezingavamile, ithimba C libhala injini entsha. Kungani kungenjalo.

Sinenjini ehlukile i-memcached, efana nejwayelekile, kodwa enenqwaba yezinto ezinhle, futhi enganciphisi. Hhayi i-ClickHouse, kodwa futhi iyasebenza. Itholakala ngokwehlukana pmemcached Ingabe i-memcache eqhubekayo, engagcina futhi idatha kudiski, ngaphezu kwalokho, kunokuba ingene ku-RAM, ukuze ingalahlekelwa idatha lapho iqala kabusha. Kukhona izinjini ezihlukahlukene zemisebenzi ngayinye: imigqa, izinhlu, amasethi - konke okudingwa yiphrojekthi yethu.

Amaqoqo

Ngokombono wekhodi, asikho isidingo sokucabanga ngezinjini noma imininingwane yolwazi njengezinqubo, amabhizinisi, noma izimo. Ikhodi isebenza ngokukhethekile ngamaqoqo, namaqembu ezinjini - uhlobo olulodwa ngeqoqo ngalinye. Ake sithi kukhona iqoqo le-memcached - iqoqo lemishini nje.

Ikhodi ayidingi ukwazi indawo ebonakalayo, usayizi, noma inombolo yamaseva nhlobo. Uya kuqoqo esebenzisa isihlonzi esithile.

Ukuze lokhu kusebenze, udinga ukwengeza enye inhlangano etholakala phakathi kwekhodi nezinjini - ummeleli.

Ummeleli we-RPC

Ummeleli ibhasi elixhumayo, lapho cishe yonke indawo isebenza kuyo. Ngesikhathi esifanayo sine akukho ukutholakala kwesevisi - esikhundleni salokho, kukhona ukucushwa kwalo mmeleli, owazi indawo yawo wonke amaqoqo nawo wonke ama-shards aleli qoqo. Nakhu okwenziwa ngabaphathi.

Abahleli abanandaba nhlobo ukuthi malini, kuphi futhi kubizani - bavele baye kuqoqo. Lokhu kusivumela kakhulu. Lapho ethola isicelo, ummeleli uqondisa kabusha isicelo, azi lapho - inquma lokhu ngokwako.

Kulokhu, i-proxy iyindawo yokuvikela ekuhlulekeni kwesevisi. Uma injini yehlisa ijubane noma iphahlazeka, ummeleli uyakuqonda lokhu futhi aphendule ngokufanele ngasohlangothini lweklayenti. Lokhu kukuvumela ukuthi ususe isikhathi sokuvala - ikhodi ayilindi ukuthi injini iphendule, kodwa iyaqonda ukuthi ayisebenzi futhi idinga ukuziphatha ngendlela ehlukile. Ikhodi kufanele ilungiselelwe iqiniso lokuthi imininingwane yolwazi ayisebenzi ngaso sonke isikhathi.

Ukuqaliswa okuqondile

Kwesinye isikhathi sisafuna ngempela ukuba nohlobo oluthile lwesixazululo esingajwayelekile njengenjini. Ngesikhathi esifanayo, kwanqunywa ukuthi singasebenzisi i-rpc-proxy yethu eseyenziwe ngomumo, eyenzelwe ngqo izinjini zethu, kodwa ukwenza ummeleli ohlukile womsebenzi.

Ku-MySQL, esisenayo lapha nalaphaya, sisebenzisa i-db-proxy, kanye ne-ClickHouse - Indlu yekhishi.

Isebenza ngokujwayelekile kanje. Kukhona iseva ethile, isebenzisa i-kPHP, Go, Python - ngokuvamile, noma iyiphi ikhodi engasebenzisa iphrothokholi yethu ye-RPC. Ikhodi isebenza endaweni kummeleli we-RPC - iseva ngayinye lapho ikhodi itholakala khona isebenzisa ummeleli wayo wendawo. Ngesicelo, ummeleli uyaqonda ukuthi kufanele aye kuphi.

Uma injini eyodwa ifuna ukuya kwenye, ngisho noma ingumakhelwane, idlula kummeleli, ngoba umakhelwane angase abe kwesinye isikhungo sedatha. Injini akufanele ithembele ekwazini indawo yanoma yini enye ngaphandle kwayo - lesi yisixazululo sethu esijwayelekile. Kodwa-ke kukhona okuhlukile :)

Isibonelo se-TL-scheme lapho zonke izinjini zisebenza khona.

memcache.not_found                                = memcache.Value;
memcache.strvalue	value:string flags:int = memcache.Value;
memcache.addOrIncr key:string flags:int delay:int value:long = memcache.Value;

tasks.task
    fields_mask:#
    flags:int
    tag:%(Vector int)
    data:string
    id:fields_mask.0?long
    retries:fields_mask.1?int
    scheduled_time:fields_mask.2?int
    deadline:fields_mask.3?int
    = tasks.Task;
 
tasks.addTask type_name:string queue_id:%(Vector int) task:%tasks.Task = Long;

Lena iphrothokholi kanambambili, i-analogue eseduze kakhulu i-protobuf. I-schema ichaza izinkambu ozikhethela zona, izinhlobo eziyinkimbinkimbi - izandiso zama-scalar ezakhelwe ngaphakathi, nemibuzo. Konke kusebenza ngokwale protocol.

I-RPC phezu kwe-TL phezu kwe-TCP/UDP… UDP?

Sinephrothokholi ye-RPC yokwenza izicelo zenjini esebenza phezu kwesikimu se-TL. Konke lokhu kusebenza ngoxhumano lwe-TCP/UDP. I-TCP iyaqondakala, kodwa kungani siyidinga njalo i-UDP?

I-UDP iyasiza gwema inkinga yenani elikhulu lokuxhumana phakathi kwamaseva. Uma iseva ngayinye inommeleli we-RPC futhi, ngokuvamile, ingaya kunoma iyiphi injini, khona-ke kukhona amashumi ezinkulungwane zokuxhunywa kwe-TCP iseva ngayinye. Ukhona umthwalo, kodwa awusizi. Endabeni ye-UDP le nkinga ayikho.

Akukho ukuxhawulana kwe-TCP okungadingekile. Lena inkinga evamile: lapho injini entsha noma iseva entsha yethulwa, ukuxhumana okuningi kwe-TCP kuyasungulwa ngesikhathi esisodwa. Ngezicelo ezincane ezingasindi, isibonelo, ukulayishwa kwe-UDP, konke ukuxhumana phakathi kwekhodi nenjini kunjalo Amaphakethe amabili e-UDP: enye indizela ohlangothini olulodwa, eyesibili kwelinye. Uhambo olulodwa lokuya nokubuya - futhi ikhodi ithole impendulo evela enjinini ngaphandle kokuxhawulana.

Yebo, konke kusebenza nje ngephesenti elincane kakhulu lokulahleka kwephakethe. Iphrothokholi inokusekelwa kokudluliswa kabusha kanye nokuvala isikhathi, kodwa uma silahlekelwa okuningi, sizothola cishe i-TCP, engeyona inzuzo. Asishayeli i-UDP ezilwandle.

Sinezinkulungwane zamaseva anjalo, futhi uhlelo luyafana: iphakethe lezinjini lifakwa kuseva yomzimba ngayinye. Ikakhulukazi inentambo eyodwa ukuze isebenze ngokushesha ngangokunokwenzeka ngaphandle kokuvinjwa, futhi ihlukaniswe njengezixazululo ezinomucu owodwa. Ngesikhathi esifanayo, asinakho okuthembeke ngaphezu kwalezi zinjini, futhi ukunakwa okuningi kukhokhwa ekugcinweni kwedatha okuqhubekayo.

Ukugcinwa kwedatha okuqhubekayo

Izinjini zibhala ama-binlogs. I-binlog yifayela ekugcineni lapho umcimbi woshintsho esimweni noma idatha yengezwe. Ezixazululweni ezahlukene ibizwa ngokuhlukile: log kanambambili, NGENDLELA, I-AOF, kodwa isimiso siyefana.

Ukuvimbela injini ukuthi ifunde kabusha yonke i-binlog iminyaka eminingi lapho iqala kabusha, izinjini ziyabhala izifinyezo - isimo samanje. Uma kunesidingo, bafunda kuyo kuqala, bese beqeda ukufunda ku-binlog. Wonke ama-binlog abhalwe ngendlela efanayo kanambambili - ngokohlelo lwe-TL, ukuze abalawuli bakwazi ukuwaphatha ngokulinganayo ngamathuluzi abo. Asikho isidingo esinjalo sezifinyezo. Kukhona unhlokweni ojwayelekile okhombisa ukuthi isifinyezo sikabani okuyi-int, umlingo wenjini, nokuthi yimuphi umzimba ongabalulekile kunoma ubani. Lena inkinga ngenjini eqophe isifinyezo.

Ngizochaza ngokushesha isimiso sokusebenza. Kukhona iseva lapho injini isebenza khona. Uvula i-binlog entsha engenalutho ukuze abhale futhi abhale umcimbi wokushintsha kuwo.

Ngesinye isikhathi, unquma ukuthatha isifinyezo ngokwakhe, noma uthola isignali. Iseva idala ifayela elisha, ibhale isimo salo sonke kulo, yengeze usayizi wamanje we-binlog - offset - kuze kube sekugcineni kwefayela, bese iqhubeka nokubhala. I-binlog entsha ayidaliwe.

Ngesinye isikhathi, lapho injini iqala kabusha, kuzoba khona kokubili i-binlog kanye nesifinyezo kudiski. Injini isifunda sonke isifinyezo bese iphakamisa isimo sayo endaweni ethile.

Ifunda indawo eyayingesikhathi isifinyezo sidalwa kanye nosayizi we-binlog.

Ifunda isiphetho se-binlog ukuze uthole isimo samanje futhi iqhubeke nokubhala eminye imicimbi. Lolu uhlelo olulula; zonke izinjini zethu zisebenza ngokuvumelana nalo.

Ukuphindaphinda idatha

Ngenxa yalokho, ukuphindaphinda kwedatha kuhlelo lwethu isitatimende-based - sibhala ku-binlog hhayi noma yiliphi ikhasi elishintshayo, kodwa okungukuthi shintsha izicelo. Okufana kakhulu nalokho okuza kunethiwekhi, kulungiswe kancane kuphela.

Uhlelo olufanayo alusetshenziselwa ukuphindaphinda kuphela, kodwa futhi ukudala izipele. Sinenjini - umpetha wokubhala obhalela i-binlog. Kunoma iyiphi enye indawo lapho abalawuli bayimisa khona, le binlog iyakopishwa, futhi yikho - sinesipele.

Uma udinga ukufunda ikhophiUkuze unciphise umthwalo wokufunda we-CPU, injini yokufunda ivele yethulwe, efunda isiphetho se-binlog futhi isebenzise le miyalo endaweni.

I-lag lapha incane kakhulu, futhi kungenzeka ukuthola ukuthi i-replica ishiya kangakanani ngemuva kwenkosi.

Ukwabiwa kwedatha kummeleli we-RPC

Ngabe kusebenza kanjani ukwahlukanisa? Ummeleli uqonda kanjani ukuthi iyiphi i-cluster shard okufanele ithunyelwe kuyo? Ikhodi ayisho ukuthi: "Thumela ama-shards ayi-15!" - cha, lokhu kwenziwa ngummeleli.

Uhlelo olulula kakhulu luyi-firstint - inombolo yokuqala esicelweni.

get(photo100_500) => 100 % N.

Lesi yisibonelo sephrothokholi yombhalo elula ye-memcached, kodwa, kunjalo, imibuzo ingaba yinkimbinkimbi futhi yakheke. Isibonelo sithatha inombolo yokuqala embuzweni kanye nensalela lapho ihlukaniswa ngosayizi weqoqo.

Lokhu kuyasiza uma sifuna ukuba nedatha yendawo yebhizinisi elilodwa. Ake sithi i-100 iwumsebenzisi noma i-ID yeqembu, futhi sifuna yonke idatha yebhizinisi elilodwa ibe kushadi elilodwa ngemibuzo eyinkimbinkimbi.

Uma singenandaba ukuthi izicelo zisakazwa kanjani kuqoqo, kukhona enye inketho - ukugeza yonke inhlama.

hash(photo100_500) => 3539886280 % N

Siphinde sithole i-hashi, ingxenye esele ye-division kanye nenombolo ye-shard.

Zombili lezi zinketho zisebenza kuphela uma silungiselele iqiniso lokuthi uma sikhulisa usayizi weqoqo, sizolihlukanisa noma silikhulise ngokuphindaphindiwe. Isibonelo, sibe namashadi angu-16, asinakho okwanele, sifuna okwengeziwe - singathola ngokuphepha 32 ngaphandle kwesikhathi sokuphumula. Uma sifuna ukwandisa hhayi ukuphindaphinda, kuzoba nesikhathi sokuphumula, ngoba ngeke sikwazi ukuhlukanisa ngokunembile yonke into ngaphandle kokulahlekelwa. Lezi zinketho ziwusizo, kodwa hhayi njalo.

Uma sidinga ukungeza noma ukususa inombolo engafanele yamaseva, sisebenzisa I-hashing engaguquki iringi i-la Ketama. Kodwa ngesikhathi esifanayo, silahlekelwa ngokuphelele indawo yedatha; kufanele sihlanganise isicelo kuqoqo ukuze ucezu ngalunye lubuyisele impendulo yalo encane, bese sihlanganisa izimpendulo kummeleli.

Kunezicelo ezicaciswe kakhulu. Kubukeka kanjena: Ummeleli we-RPC uthola isicelo, unquma ukuthi iyiphi iqoqo okufanele uye kulo futhi inquma i-shard. Bese kuba khona izingcweti zokubhala, noma, uma iqoqo linokwesekwa kwe-replica, lithumela kusifaniso ngokufunwa. Ummeleli wenza konke lokhu.

Izingodo

Sibhala izingodo ngezindlela eziningana. Okusobala kakhulu futhi elula bhala izingodo ku-memcache.

ring-buffer: prefix.idx = line

Kukhona isiqalo esiyinhloko - igama lelogi, umugqa, futhi kukhona ubukhulu balolu logi - inombolo yemigqa. Sithatha inombolo engahleliwe ukusuka ku-0 ukuya enanini lemigqa susa 1. Ukhiye ku-memcache yisiqalo esihlanganiswe nale nombolo engahleliwe. Sigcina umugqa welogi kanye nesikhathi samanje kunani.

Lapho kudingekile ukufunda izingodo, senza Thola okuningi bonke okhiye, bahlungwe ngesikhathi, futhi ngaleyo ndlela bathole ilogu yokukhiqiza ngesikhathi sangempela. Uhlelo lusetshenziswa uma udinga ukulungisa iphutha ekukhiqizeni ngesikhathi sangempela, ngaphandle kokuphula noma yini, ngaphandle kokumisa noma ukuvumela ithrafikhi kweminye imishini, kodwa le log ayihlali isikhathi eside.

Ukuze silondoloze okuthembekile izingodo sinenjini izingodo-injini. Yingakho nje yadalwa futhi isetshenziswa kabanzi enanini elikhulu lamaqoqo. Iqoqo elikhulu kunawo wonke engilaziyo lezitolo ezingu-600 TB zamalogi apakishiwe.

Injini indala kakhulu, kunamaqoqo asevele aneminyaka engu-6-7 ubudala. Kunezinkinga ngakho esizama ukuzixazulula, isibonelo, saqala ukusebenzisa ngenkuthalo i-ClickHouse ukugcina izingodo.

Iqoqa izingodo ku-ClickHouse

Lo mdwebo ukhombisa ukuthi singena kanjani ezinjinini zethu.

Kunekhodi eya endaweni nge-RPC iye kummeleli we-RPC, futhi iyaqonda ukuthi kufanele iye kuphi enjinini. Uma sifuna ukubhala izingodo ku-ClickHouse, sidinga ukushintsha izingxenye ezimbili kulolu hlelo:

buyisela injini ngeClickHouse;
buyisela ummeleli we-RPC, ongakwazi ukufinyelela i-ClickHouse, ngesisombululo esingase, nange-RPC.

Injini ilula - siyishintsha ngeseva noma iqoqo lamaseva ngeClickHouse.

Futhi ukuya ku-ClickHouse, senze I-KittenHouse. Uma sisuka ngqo ku-KittenHouse siye ku-ClickHouse, ngeke kusebenze. Noma ngaphandle kwezicelo, ingeza kusuka ekuxhumekeni kwe-HTTP kwenani elikhulu lemishini. Ukuze uhlelo lusebenze, kuseva ene-ClickHouse ummeleli wendawo ohlanekezelwe uyaphakanyiswa, ebhalwe ngendlela yokuthi ikwazi ukumelana nenani elidingekayo lokuxhumana. Ingaphinda futhi igcine idatha ngaphakathi kwayo ngokwethembekile.

Kwesinye isikhathi asifuni ukusebenzisa uhlelo lwe-RPC kuzixazululo ezingajwayelekile, ngokwesibonelo, ku-nginx. Ngakho-ke, i-KittenHouse inamandla okuthola izingodo nge-UDP.

Uma umthumeli nomamukeli wamalogi besebenza emshinini ofanayo, khona-ke amathuba okulahlekelwa iphakethe le-UDP ngaphakathi komsingathi wendawo aphansi kakhulu. Njengokuvumelana phakathi kwesidingo sokusebenzisa i-RPC kusixazululo nokuthembeka kwenkampani yangaphandle, simane sisebenzisa ukuthumela kwe-UDP. Sizobuyela kulolu hlelo ngokuhamba kwesikhathi.

Ukuqapha

Sinezinhlobo ezimbili zamalogi: lawo aqoqwe abalawuli eziphakelini zabo nalawo abhalwe ngonjiniyela ngekhodi. Zihambisana nezinhlobo ezimbili zamamethrikhi: uhlelo nomkhiqizo.

Amamethrikhi esistimu

Isebenza kuwo wonke amaseva ethu INetdata, eqoqa izibalo bese izithumela ku Ikhabhoni yeGraphite. Ngakho-ke, i-ClickHouse isetshenziswa njengesistimu yokugcina, hhayi i-Whisper, isibonelo. Uma kunesidingo, ungafunda ngokuqondile ku-ClickHouse, noma usebenzise UGrafana amamethrikhi, amagrafu nemibiko. Njengabathuthukisi, sinokufinyelela okwanele ku-Netdata ne-Grafana.

Amamethrikhi omkhiqizo

Ukuze kube lula, sibhale izinto eziningi. Isibonelo, kukhona isethi yemisebenzi evamile ekuvumela ukuthi ubhale Izibalo, amanani we-UniqueCounts kuzibalo, ezithunyelwa kwenye indawo phambili.

statlogsCountEvent   ( ‘stat_name’,            $key1, $key2, …)
statlogsUniqueCount ( ‘stat_name’, $uid,    $key1, $key2, …)
statlogsValuetEvent  ( ‘stat_name’, $value, $key1, $key2, …)

$stats = statlogsStatData($params)

Ngokulandelayo, singasebenzisa izihlungi zokuhlunga nokuhlanganisa futhi senze konke esikufunayo kusuka kuzibalo - ukwakha amagrafu, ukulungisa ama-Watchdogs.

Sibhala kakhulu amamethrikhi amaningi inani lezehlakalo lisuka ku-600 billion kuya ku-1 trillion ngosuku. Nokho, sifuna ukuzigcina okungenani iminyaka embalwaukuqonda amathrendi kumamethrikhi. Ukuhlanganisa konke kuyinkinga enkulu esingakayixazululi. Ngizokutshela ukuthi ibisebenza kanjani kule minyaka embalwa edlule.

Sinemisebenzi ebhala lawa mamethrikhi ku-memcache yendawoukunciphisa inani lokungenela. Kanye ngesikhathi esifushane endaweni yethulwa izibalo-daemon iqoqa wonke amarekhodi. Okulandelayo, idemoni lihlanganisa amamethrikhi libe izendlalelo ezimbili zamaseva izingodo-abaqoqi, ehlanganisa izibalo zenqwaba yemishini yethu ukuze ungqimba olungemuva kwayo lungafi.

Uma kunesidingo, singabhalela ngqo abaqoqi belogi.

Kodwa ukubhala kusuka kukhodi ngokuqondile kuya kubaqoqi, ukweqa i-stas-daemom, kuyisixazululo esingesihle kahle ngoba kwandisa umthwalo kumqoqi. Isixazululo sifaneleka kuphela uma ngesizathu esithile singakwazi ukuphakamisa i-memcache stats-daemon emshinini, noma iphahlazekile futhi saya ngqo.

Okulandelayo, abaqoqi bamalogu bahlanganisa izibalo zibe meoDB - lena database yethu, engagcina futhi amamethrikhi.

Bese singakwazi ukukhetha kanambambili "near-SQL" kusuka kukhodi.

Ukuhlolwa

Ehlobo lika-2018, saba ne-hackathon yangaphakathi, futhi kwavela umqondo wokuzama ukufaka ingxenye ebomvu yomdwebo ngokuthile okungagcina amamethrikhi ku-ClickHouse. Sinezingodo ku-ClickHouse - kungani ungazami?

Sasinohlelo olwalubhala izingodo nge-KittenHouse.

Sanquma engeza enye “*Indlu” kumdwebo, ezothola amamethrikhi ngefomethi njengoba ikhodi yethu iwabhala nge-UDP. Khona-ke le *House iziphendulela zibe izinto ezifakwayo, njengezingodo, eziqondwa yiKittenHouse. Angakwazi ukuletha lezi zingodo ngokuphelele ku-ClickHouse, okufanele ikwazi ukuzifunda.

Uhlelo olune-memcache, i-stats-daemon kanye nesizindalwazi sabaqoqi bamalogu sithathelwa indawo yilesi.

Kukhona ukuthunyelwa okuvela kukhodi lapha, okubhalwe endaweni ku-StatsHouse.
I-StatsHouse ibhala amamethrikhi e-UDP, asevele eguqulelwe ekufakweni kwe-SQL, ku-KittenHouse ngamaqoqo.
I-KittenHouse izithumela ku-ClickHouse.
Uma sifuna ukuzifunda, sizobe sesizifunda ngokudlula i-StatsHouse - ngqo kusuka ku-ClickHouse sisebenzisa i-SQL evamile.

Ngabe namanje ukuhlola, kodwa siyayithanda indlela okwenzeka ngayo. Uma silungisa izinkinga ngohlelo, mhlawumbe sizoshintshela kulo ngokuphelele. Ngokwami, ngithemba kanjalo.

Lolu hlelo ayilondolozi insimbi. Kudingeka amaseva ambalwa, izibalo-daemons zendawo kanye nabaqoqi bamalogu abadingeki, kodwa i-ClickHouse idinga iseva enkulu kunalezo ezisesikimu samanje. Kudingeka amaseva ambalwa, kodwa kufanele abize kakhulu futhi abe namandla kakhulu.

Sebenzisa

Okokuqala, ake sibheke ukuthunyelwa kwe-PHP. Siyathuthuka ku Git: sebenzisa I-GitLab и I-TeamCity ukuthunyelwa. Amagatsha okuthuthukiswa ahlanganiswa abe yi-master branch, ukusuka ku-master ukuze ahlolwe ahlanganiswa abe yisiteji, futhi asuke esiteji aye ekukhiqizeni.

Ngaphambi kokuthunyelwa, igatsha lokukhiqiza lamanje kanye nedlule liyathathwa, futhi amafayela ahlukene acatshangelwa kuwo - izinguquko: zidalwe, zisusiwe, zishintshiwe. Lolu shintsho lurekhodwa ku-binlog yenjini ekhethekile ye-copyfast, engaphindaphinda ngokushesha izinguquko kuwo wonke amaseva ethu. Okusetshenziswe lapha akukopishi ngokuqondile, kodwa ukuphindaphinda inhlebo, lapho iseva eyodwa ithumela izinguquko komakhelwane bayo abaseduze, labo komakhelwane babo, njalonjalo. Lokhu kukuvumela ukuthi ubuyekeze ikhodi ngamashumi namayunithi amasekhondi kuwo wonke umkhumbi. Uma ushintsho lufinyelela emfanekisweni wendawo, lusebenzisa lezi ziqephu kuzo isistimu yefayela yendawo. I-Rollback nayo yenziwa ngokulandela uhlelo olufanayo.

Siphinde sisebenzisa i-kPHP kakhulu futhi nayo inokuthuthukiswa kwayo Git ngokomdwebo ongenhla. Kusukela lokhu Iseva ye-HTTP kanambambili, ngakho-ke asikwazi ukukhiqiza i-diff - kanambambili yokukhululwa inesisindo samakhulu e-MB. Ngakho-ke, kukhona enye inketho lapha - inguqulo ibhalelwe binlog copyfast. Ngokwakhiwa ngakunye kuyanda, futhi ngesikhathi sokuhlehlisa nakho kuyanda. Inguqulo kuphindaphindwe kumaseva. Abakopishi bendawo babona ukuthi inguqulo entsha ingenile ku-binlog, futhi ngokuphindaphinda inhlebo efanayo bazithathela inguqulo yakamuva kanambambili, ngaphandle kokukhathaza iseva yethu eyinhloko, kodwa basakaza umthwalo ngokucophelela kunethiwekhi yonkana. Okulandelayo ukuqaliswa kabusha ngomusa yenguqulo entsha.

Ezinjinini zethu, ezibuye zibe ama-binaries, uhlelo lufana kakhulu:

igatsha le-git master;
kanambambili ku .deb;
inguqulo ibhalwe ku-binlog copyfast;
kuphindaphindwe kumaseva;
iseva ikhipha i-.dep entsha;
dpkg ngi;
ukuqalisa kabusha okuhle kwenguqulo entsha.

Umehluko ukuthi unambambili wethu ufakwe ezinqolobaneni .deb, nalapho bekhipha amanzi dpkg ngi abekwe ohlelweni. Kungani i-kPHP isetshenziswa njenge kanambambili, futhi izinjini zisetshenziswa njenge-dpkg? Kwenzeka kanjalo. Iyasebenza - ungayithinti.

Izixhumanisi eziwusizo:

Umbiko ka-Anton Kiryushkin "Vkontakte system administrator. Kanjani?" ngemininingwane mayelana ne-copyfast nenhlebo.
Umbiko ka-Yuri Nasretdinov "Indlela i-VK efaka ngayo idatha ku-CLickHouse emashumini ezinkulungwane zamaseva".
Umbiko wami "Ukwakhiwa kwephrojekthi ekhulayo usebenzisa isibonelo se-VKontakte", kodwa ngokombono wokuthuthuka, hhayi ihadiwe.

U-Alexey Akulovich ungomunye walabo abasiza, njengengxenye yeKomidi Lohlelo PHP Russia ngoMeyi 17 kuzoba umcimbi omkhulu kunawo wonke wabathuthukisi be-PHP ezikhathini zamuva nje. Bheka ukuthi sinayo i-PC epholile, ini izikhulumi (ezimbili zazo zithuthukisa i-PHP core!) - kubonakala njengento ongeke uyiphuthe uma ubhala i-PHP.

Source: www.habr.com

I-FAQ mayelana nezakhiwo nomsebenzi we-VKontakte