Phinda ubhale i-database yomyalezo weVKontakte ukusuka ekuqaleni kwaye usinde

Abasebenzisi bethu babhalelana imiyalezo ngaphandle kokwazi ukudinwa.

Ininzi kakhulu loo nto. Ukuba uzimisele ukufunda yonke imiyalezo yabo bonke abasebenzisi, kuya kuthatha iminyaka engaphezu kwe-150 lamawaka. Ngaphandle kokuba ungumfundi ohambele phambili ngokufanelekileyo kwaye ungachithi ngaphezu kwesekondi kumyalezo ngamnye.

Ngomthamo onjalo wedatha, kubalulekile ukuba ingqiqo yokugcina kunye nokufikelela kuyo yakhiwe ngokufanelekileyo. Ngaphandle koko, ngexesha elinye elingelilo-elimangalisayo, kunokucaca ukuba yonke into iya kuhamba ngokukhawuleza.

Kuthi, lo mzuzu ufike kunyaka onesiqingatha odlulileyo. Indlela esifike ngayo kule nto kwaye kwenzekani ekugqibeleni - sikuxelela ngokulandelelana.

Imbali yokukhutshwa

Kuphunyezo lokuqala, imiyalezo yeVKontakte yasebenza kwindibaniselwano ye-PHP backend kunye neMySQL. Esi sisisombululo esiqhelekileyo ngokupheleleyo kwiwebhusayithi yabafundi encinci. Nangona kunjalo, le sayithi yakhula ngokungalawulekiyo kwaye yaqala ukufuna ukulungiswa kolwakhiwo lwedatha ngokwayo.

Ekupheleni kwe-2009, i-repository yokuqala ye-injini-injini yabhalwa, kwaye kwi-2010 imiyalezo idluliselwe kuyo.

Kwi-injini yombhalo, imiyalezo igcinwe kuluhlu - uhlobo "lweebhokisi zeposi". Uluhlu ngalunye olunjalo lumiselwa yi-uid - umsebenzisi ongumnini wayo yonke le miyalezo. Umyalezo uneseti yeempawu: isazisi se-interlocutor, umbhalo, izincamatheliso, njalo njalo. Isichongi somyalezo ngaphakathi "kwibhokisi" yi local_id, ayitshintshi kwaye yabelwe ngokulandelelana kwemiyalezo emitsha. "Iibhokisi" zizimeleyo kwaye azihambelani phakathi kwe-injini yokunxibelelana phakathi kwazo kwenzeka kwinqanaba le-PHP. Unokujonga kwisakhiwo sedatha kunye nobuchule be-text-injini ukusuka ngaphakathi apha.

Oku kwakwanele kwimbalelwano phakathi kwabasebenzisi ababini. Qikelela ukuba kwenzeka ntoni emva koko?

NgoMeyi 2011, i-VKontakte yazisa iingxoxo kunye nabathathi-nxaxheba abaninzi-iingxoxo ezininzi. Ukusebenzisana nabo, sikhulise amaqela amabini amatsha - iingxoxo zamalungu kunye namalungu engxoxo. Eyokuqala igcina idatha malunga neengxoxo ngabasebenzisi, okwesibini igcina idatha malunga nabasebenzisi ngeengxoxo. Ukongeza kwizintlu ngokwazo, oku kubandakanya, umzekelo, umsebenzisi omemayo kunye nexesha longezwe kwingxoxo.

"PHP, masithumele umyalezo kwincoko," utshilo umsebenzisi.
"Yiza, {igama lomsebenzisi}," itsho i-PHP.

Kukho izinto ezingeloncedo kwesi sikimu. Ungqamaniso luseluxanduva lwe-PHP. Iingxoxo ezinkulu kunye nabasebenzisi abathumela imiyalezo kubo ngaxeshanye libali eliyingozi. Kuba umzekelo we-text-injini ixhomekeke kwi-uid, abathathi-nxaxheba bencoko banokufumana umyalezo ofanayo ngamaxesha ahlukeneyo. Umntu unokuphila nale nto ukuba inkqubela ibimi ngxi. Kodwa oku akuyi kwenzeka.

Ekupheleni kwe-2015, saqalisa imiyalezo yoluntu, kwaye ekuqaleni kwe-2016, sasungula i-API kubo. Ngokufika kwee-chatbots ezinkulu kuluntu, kwakunokwenzeka ukulibala malunga nokuhanjiswa komthwalo.

I-bot elungileyo ivelisa izigidi ngezigidi zemiyalezo ngosuku-nabona basebenzisi bathetha kakhulu abanakuqhayisa ngale nto. Oku kuthetha ukuba ezinye iimeko ze-injini yetekisi, apho iibhothi ezinjalo zihlala khona, zaqala ukubandezeleka ngokupheleleyo.

Iinjini zomyalezo kwi-2016 ziyi-100 iimeko zengxoxo-amalungu kunye neengxoxo zelungu, kunye ne-8000 text-injini. Baye babanjwa kwiiseva eziliwaka, nganye ine-64 GB yememori. Njengomlinganiselo wokuqala ongxamisekileyo, sandise imemori ngenye i-32 GB. Siqikelele uqikelelo. Ngaphandle kweenguqu ezinkulu, oku bekuya kuba kwanele malunga ngomnye unyaka. Kuya kufuneka ubambe i-hardware okanye wenze isiseko sedatha ngokwaso.

Ngenxa yobume boyilo, kunengqiqo kuphela ukwandisa i-hardware kwii-multiples. Oko kukuthi, ukuphinda kabini inani leemoto - ngokucacileyo, le yindlela ebiza kakhulu. Siza kwandisa.

Ingcamango entsha

Undoqo ongundoqo wendlela entsha yincoko. Incoko inoluhlu lwemiyalezo ehambelana nayo. Umsebenzisi unoluhlu lweencoko.

Ubuncinane obufunekayo ziidathabheyisi ezimbini ezintsha:

• ingxoxo-injini. Le yindawo yokugcina iivektha zengxoxo. Incoko nganye inevector yemiyalezo ehambelana nayo. Umyalezo ngamnye unombhalo kunye nesichongi somyalezo esisodwa ngaphakathi kwincoko - chat_local_id.
• umsebenzisi-injini. Oku kukugcinwa kwabasebenzisi be-vectors - amakhonkco kubasebenzisi. Umsebenzisi ngamnye une-vector ye-peer_id (i-interlocutors - abanye abasebenzisi, i-multi-chat okanye uluntu) kunye ne-vector yemiyalezo. I peer_id nganye inevector yemiyalezo ehambelana nayo. Umyalezo ngamnye une-chat_local_id kunye ne-ID yomyalezo eyahlukileyo yalo msebenzisi- user_local_id.

Amaqela amatsha anxibelelana omnye nomnye usebenzisa i-TCP - oku kuqinisekisa ukuba umyalelo wezicelo awutshintshi. Izicelo ngokwabo kunye neziqinisekiso kubo zirekhodwa kwi hard drive - ukuze sikwazi ukubuyisela imeko emgceni nanini na emva kokusilela okanye ukuqalisa kwakhona injini. Ekubeni i-injini yomsebenzisi kunye ne-injini ye-chat-i-4 ye-shards nganye, umgca wesicelo phakathi kwamaqela uya kuhanjiswa ngokulinganayo (kodwa ngokwenene akukho nanye - kwaye isebenza ngokukhawuleza).

Ukusebenza ngediski kwiinkcukacha zethu kwiimeko ezininzi kusekelwe kwindibaniselwano ye-binary log yotshintsho (binlog), i-snapshots engatshintshiyo kunye nomfanekiso ongaphelelanga kwimemori. Utshintsho ngexesha losuku lubhalwa kwi-binlog, kwaye i-snapshot yemeko yangoku yenziwa ngamaxesha athile. I-snapshot yingqokelela yezakhiwo zedatha ezilungiselelwe iinjongo zethu. Iquka iheader (metaindex yomfanekiso) kunye neseti yeetafile. I-header igcinwe ngokusisigxina kwi-RAM kwaye ibonisa apho ukhangele khona idatha kwi-snapshot. Imetafile nganye ibandakanya idatha enokuthi ifuneke kwiindawo ezikufutshane ngexesha-umzekelo, ezinxulumene nomsebenzisi omnye. Xa ubuza ugcino lwedatha usebenzisa iheader ekhawulezayo, i metafile efunekayo iyafundwa, kwaye emva koko utshintsho lwe binlog lwenzeke emva kokuba umfanekiso okhawulezayo wenziwe uthathelwa ingqalelo. Unokufunda ngakumbi malunga neenzuzo zale ndlela apha.

Ngelo xesha, idatha kwi-hard drive ngokwayo iyatshintsha kanye kuphela ngosuku - ebusuku kakhulu eMoscow, xa umthwalo uncinci. Siyabulela kule nto (ukwazi ukuba isakhiwo kwidiski sihlala sihlala imini yonke), sinokukwazi ukubuyisela ii-vectors kunye nemilinganiselo yobukhulu obusisigxina - kwaye ngenxa yoko, zuza kwimemori.

Ukuthumela umyalezo kwisikimu esitsha kujongeka ngolu hlobo:

1. I-backend ye-PHP iqhagamshelana ne-injini yomsebenzisi ngesicelo sokuthumela umyalezo.
2. i-injini yomsebenzisi i-proxies isicelo kumzekelo we-chat-engine efunekayo, ebuyela kwi-chat-injini ye-chat_local_id - isichongi esisodwa somyalezo omtsha ngaphakathi kwale ngxoxo. I-chat_engine isasaza umyalezo kubo bonke abamkeli kwincoko.
3. i-injini yomsebenzisi ifumana i-chat_local_id kwi-chat-injini kwaye ibuyisela i-user_local_id kwi-PHP - isichongi somyalezo esisodwa kulo msebenzisi. Esi sichongi siyasetyenziswa, umzekelo, ukusebenza ngemiyalezo nge-API.

Kodwa ukongeza ekuthumeleni imiyalezo ngokwenene, kufuneka uphumeze izinto ezininzi ezibalulekileyo:

• Uluhlu, umzekelo, imiyalezo yamva nje oyibonayo xa uvula uluhlu lwencoko. Imiyalezo engekafundwa, imiyalezo enethegi ("Ibalulekile", "Spam", njl. njl.).
• Ukucinezela imiyalezo kwi-injini yengxoxo
• Igcina imiyalezo kwi-injini yomsebenzisi
• Phendla (ngazo zonke iingxoxo kwaye ngaphakathi kwenye ethile).
• Uhlaziyo lwexesha langempela (Longpolling).
• Ukugcina imbali ukuphumeza i-caching kubaxhasi beselula.

Zonke izintlu ezisezantsi zitshintsha ngokukhawuleza izakhiwo. Ukusebenza nabo sisebenzisa Imithi ye-splay. Olu khetho luchazwa yinyaniso yokuba phezulu komthi ngamanye amaxesha sigcina icandelo elipheleleyo lemiyalezo esuka kwi-snapshot - umzekelo, emva kokubuyisela kwakhona ebusuku, umthi uqukethe enye ephezulu, equlethe yonke imiyalezo yoluhlu oluncinci. Umthi we-Splay wenza kube lula ukufaka phakathi kwe-vertex enjalo ngaphandle kokucinga ngokulinganisa. Ukongeza, i-Splay ayigcini idatha engeyomfuneko, egcina imemori.

Imiyalezo ibandakanya isixa esikhulu solwazi, ubukhulu becala isicatshulwa, esiluncedo ukukwazi ukucinezela. Kubalulekile ukuba sikhuphe ngokuchanekileyo umyalezo omnye. Isetyenziselwa ukucinezela imiyalezo Huffman algorithm ngeyethu i-heuristics - umzekelo, siyazi ukuba kwimiyalezo amagama atshintshana "namagama angengawo amagama" - izithuba, iziphumlisi - kwaye sikhumbula ezinye zezinto ezikhethekileyo zokusebenzisa iisimboli kulwimi lwesiRashiya.

Kuba kukho abasebenzisi abambalwa kakhulu kuneencoko, ukugcina izicelo zediski zokufikelela ngokungakhethiyo kwi-injini yengxoxo, sigcina imiyalezo kwi-injini yomsebenzisi.

Uphendlo lomyalezo luphunyezwa njengombuzo we-diagonal ukusuka kwi-injini yomsebenzisi kuzo zonke iimeko ze-injini zencoko eziqulathe iincoko zalo msebenzisi. Iziphumo zidibene kwi-injini yomsebenzisi ngokwayo.

Ewe, zonke iinkcukacha zithathelwe ingqalelo, konke okuseleyo kukutshintshela kwisikimu esitsha - kwaye kukhethwa ngaphandle kokuba abasebenzisi baqaphele.

Ukufuduka kwedatha

Ngoko ke, sine-injini yombhalo egcina imiyalezo ngumsebenzisi, kunye namaqela amabini engxoxo-amalungu kunye neengxoxo zelungu ezigcina idatha malunga namagumbi eengxoxo ezininzi kunye nabasebenzisi kuwo. Indlela yokuhamba ukusuka koku ukuya kwi-injini entsha yomsebenzisi kunye ne-injini yokuxoxa?

amalungu-ingxoxo kwiskim esidala sasetyenziswa ngokuyintloko ukulungiselelwa. Sithumele ngokukhawuleza idatha efunekayo kuyo ukuya kwingxoxo-amalungu, kwaye emva koko ayizange ithathe inxaxheba kwinkqubo yokufuduka.

Umgca wamalungu encoko. Iquka iimeko ze-100, ngelixa i-chat-injini ine-4 lamawaka. Ukudlulisa idatha, kufuneka uyizise ekuthotyelweni - kule nto, amalungu e-chat ahlulwe kwiikopi ezifanayo ze-4 lamawaka, kwaye emva koko ukufundwa kwe-chat-members binlog kwenziwa kwi-injini yengxoxo.

Ngoku i-chat-injini iyazi malunga neengxoxo ezininzi ezivela kumalungu engxoxo, kodwa ayikayazi nto malunga neengxoxo kunye nabanxibelelanisi ababini. Iingxoxo ezinjalo zifumaneka kwi-injini yokubhaliweyo ngokubhekisele kubasebenzisi. Apha sithathe idatha "intloko-on": umzekelo ngamnye we-injini yokuxoxa wabuza yonke imizekelo ye-text-injini ukuba babenengxoxo efunekayo.

Inkulu-injini yencoko iyayazi ukuba zeziphi iincoko ezininzi ezikhoyo kwaye iyayazi ukuba zeziphi iingxoxo ezikhoyo.
Kufuneka udibanise imiyalezo kwiingxoxo ezininzi zengxoxo ukuze uphele noluhlu lwemiyalezo kwincoko nganye. Okokuqala, i-chat-injini ifumana kwakhona kwi-text-injini yonke imiyalezo yabasebenzisi kule ngxoxo. Kwezinye iimeko zininzi kakhulu (ukuya kumakhulu ezigidi), kodwa ngaphandle kokunqaba kakhulu incoko ingena ngokupheleleyo kwi-RAM. Sinemiyalezo engacwangciswanga, nganye kwiikopi ezininzi - emva kwayo yonke loo nto, zonke zitsalwa kwiimeko ezahlukeneyo ze-injini yetekisi ehambelana nabasebenzisi. Injongo kukulungisa imiyalezo kunye nokulahla iikopi ezithatha indawo engeyomfuneko.

Umyalezo ngamnye unesitampu sexesha esiqulethe ixesha othunyelwe ngalo kunye nokubhaliweyo. Sisebenzisa ixesha lokuhlela - sibeka izikhombisi kwimiyalezo emidala yabathathi-nxaxheba be-multichat kwaye sithelekise i-hashes kumbhalo weekopi ezicetywayo, sisiya ekunyuseni isitampu sexesha. Kunengqiqo ukuba iikopi ziya kuba ne-hash kunye nesitampu sexesha elifanayo, kodwa ekusebenzeni oku akusoloko kunjalo. Njengoko ukhumbula, ungqamaniso kwiskimu esidala lwenziwa yi-PHP - kwaye kwiimeko ezinqabileyo, ixesha lokuthumela umyalezo ofanayo lahlukile phakathi kwabasebenzisi abahlukeneyo. Kwezi meko, sizivumele ukuba sihlele isitampu sexesha - ngesiqhelo kwisekondi. Ingxaki yesibini lulandelelwano olwahlukileyo lwemiyalezo yabamkeli abohlukeneyo. Kwiimeko ezinjalo, sivumele ikopi eyongezelelweyo ukuba yenziwe, kunye neendlela ezahlukeneyo zokuodola kubasebenzisi abohlukeneyo.

Emva koku, idatha malunga nemiyalezo kwi-multichat ithunyelwa kwi-injini yomsebenzisi. Kwaye nantsi inqaku elingemnandanga lemiyalezo engenisiweyo. Kumsebenzi oqhelekileyo, imiyalezo eza kwi-injini iodolwa ngokungqongqo ngokwendlela enyukayo nge-user_local_id. Imiyalezo ethathwe ngaphandle esuka kwi-injini endala ukuya kwi-injini yomsebenzisi iphulukene nale mpahla iluncedo. Ngexesha elifanayo, ukwenzela ukuba kube lula ukuvavanywa, kufuneka ukwazi ukufikelela ngokukhawuleza kuzo, ukhangele into ethile kuzo kwaye ungeze ezintsha.

Sisebenzisa ulwakhiwo lwedatha ekhethekileyo ukugcina imiyalezo evela kumazwe angaphandle.

Imele i-vector yobungakanani uphi wonke umntu - ziyahluka kwaye zicwangciswe ngokuhla, kunye nomyalelo okhethekileyo wezinto. Kwicandelo ngalinye elinama-indices izinto ziyahlelwa. Ukukhangela into kwisakhiwo esinjalo kuthatha ixesha ngokusebenzisa uphendlo lokubini. Ukongezwa kwento kuncitshiswa ngaphezulu .

Ke, siye safumanisa indlela yokudlulisa idatha ukusuka kwiinjini ezindala ukuya kwezintsha. Kodwa le nkqubo ithatha iintsuku ezininzi - kwaye akunakwenzeka ukuba abasebenzisi bethu bawuyeke umkhwa wokubhalelana kwezi ntsuku. Ukuze ungalahlekelwa yimiyalezo ngeli xesha, sitshintshela kwisikimu somsebenzi osebenzisa amaqela amadala namatsha.

Idatha ibhalwe kwi-chat-amalungu kunye ne-injini yomsebenzisi (kwaye kungekhona kwi-injini yetekisi, njengoko isebenza ngokuqhelekileyo ngokweskimu esidala). umsebenzisi-injini yokusebenzela isicelo sokuxoxa-injini - kwaye apha ukuziphatha kuxhomekeke ekubeni le ncoko sele idityanisiwe okanye hayi. Ukuba incoko ayikadibanisi, i-injini yokuxoxa ayibhali umyalezo kuyo, kwaye ukusetyenzwa kwayo kwenzeka kuphela kwi-injini yombhalo. Ukuba incoko sele idityanisiwe kwi-chat-injini, ibuyisela i-chat_local_id kwi-injini yomsebenzisi kwaye ithumela umyalezo kubo bonke abamkeli. umsebenzisi-injini proxies yonke idatha ukuya text-injini - ukuze ukuba kukho into eyenzekayo, sinokuhlala umva, ukuba yonke idatha yangoku kwi-injini endala. text-injini ibuyisela user_local_id, leyo umsebenzisi-injini igcina kwaye ibuyisela umva.

Ngenxa yoko, inkqubo yenguqu ibonakala ngolu hlobo: sidibanisa i-injini engenanto kunye ne-chat-injini ye-injini. chat-injini ifunda yonke chat-amalungu binlog, ngoko proxying iqala ngokwenkqubo echazwe ngasentla. Sidlulisela idatha endala kwaye sifumane amaqela amabini alungelelanisiweyo (amadala namatsha). Ekuphela kwento eseleyo kukutshintsha ukufunda ukusuka kwi-injini yombhalo ukuya kwi-injini yomsebenzisi kwaye ukhubaze i-proxying.

Iziphumo

Ngombulelo kwindlela entsha, zonke iimethrikhi zokusebenza kweenjini ziphuculwe kwaye iingxaki zokuhambelana kwedatha ziye zasonjululwa. Ngoku sinokukhawuleza siphumeze amanqaku amatsha kwimiyalezo (kwaye sele siqalisile ukwenza oku - sandise inani eliphezulu labathathi-nxaxheba bengxoxo, siphumeze ukukhangela imiyalezo egqithisiweyo, siqalise imiyalezo ephiniweyo kwaye sinyuse umda kwinani lilonke lemiyalezo ngomsebenzisi ngamnye) .

Utshintsho kwingqiqo lukhulu ngokwenene. Kwaye ndingathanda ukuqaphela ukuba oku akusoloko kuthetha iminyaka epheleleyo yophuhliso liqela elikhulu kunye namawaka emigca yekhowudi. i-chat-injini kunye ne-injini yomsebenzisi kunye nawo onke amabali ongezelelweyo afana ne-Huffman yoxinzelelo lomyalezo, i-Splay imithi kunye nesakhiwo semiyalezo ethunyelwa ngaphandle ingaphantsi kwama-20 amawaka emigca yekhowudi. Kwaye zabhalwa ngabaphuhlisi aba-3 kwiinyanga nje ezili-10 (nangona kunjalo, kufanelekile ukukhumbula ukuba zonke ezintathu umphuhlisi - iintshatsheli zehlabathi kwinkqubo yezemidlalo).

Ngaphezu koko, endaweni yokuphinda kabini inani leeseva, sinciphise inani labo ngesiqingatha - ngoku i-injini yomsebenzisi kunye ne-chat-injini zihlala kumatshini wenyama angama-500, ngelixa iskimu esitsha sinentloko enkulu yomthwalo. Sigcine imali eninzi kwizixhobo - malunga ne-5 yezigidi zeedola + i-750 yeedola ngonyaka kwiindleko zokusebenza.

Sizama ukufumana ezona zisombululo zilungileyo kwezona ngxaki zintsonkothileyo nezinkulu. Sinento eninzi yazo-yiyo loo nto sifuna abaphuhlisi abanetalente kwisebe ledatha. Ukuba uyathanda kwaye uyayazi indlela yokusombulula iingxaki ezinjalo, unolwazi olubalaseleyo lwe-algorithms kunye nezakhiwo zedatha, siyakumema ukuba ujoyine iqela. Qhagamshelana nathi HRngeenkcukacha.

Nokuba eli bali alikho ngawe, nceda uqaphele ukuba sizixabisile iingcebiso. Xelela umhlobo wakho malunga izithuba zomphuhlisi, kwaye ukuba ugqiba ngempumelelo ixesha lokuvavanya, uya kufumana ibhonasi ye-ruble eyi-100 lamawaka.

umthombo: www.habr.com