Nyorazve VKontakte meseji dhatabhesi kubva mukutanga uye urarame

Vashandisi vedu vanonyorerana mameseji vasingazive kuneta.

Izvo zvakanyanya. Kana iwe ukaronga kuverenga meseji ese evashandisi vese, zvinotora anopfuura 150 zviuru zvemakore. Zvichipa kuti iwe uri muverengi wepamberi uye unopedza isingapfuure sekondi pane yega meseji.

Nehuwandu hwakadaro hwe data, zvakakosha kuti pfungwa yekuchengeta uye kuiwana inovakwa zvakanaka. Zvikasadaro, mune imwe nguva isiri-yakanyanya-inoshamisa, zvinogona kuve pachena kuti zvese zvichakurumidza kuipa.

Kwatiri, nguva iyi yakauya gore nehafu yapfuura. Kuti takasvika sei kune izvi uye zvakaitika pakupedzisira - tinokuudza zvakarongeka.

Background

Mukutanga kuita, VKontakte mameseji akashanda pane musanganiswa wePHP backend uye MySQL. Iyi ndiyo yakajairika mhinduro kune diki webhusaiti webhusaiti. Nekudaro, iyi saiti yakakura zvisingaite uye yakatanga kuda optimization ye data zvimiro zvayo.

Pakupera kwa2009, yekutanga text-injini repository yakanyorwa, uye muna 2010 mameseji akaendeswa kwairi.

Muinjini-injini, mameseji akachengetwa mumazita - rudzi rwe "mabhokisi etsamba". Rondedzero yega yega yakadai inotemerwa neuid - mushandisi ane mameseji ese aya. Meseji ine seti yehunhu: interlocutor identifier, zvinyorwa, zvakabatanidzwa, zvichingodaro. Iyo meseji identifier mukati me "bhokisi" ndeye local_id, haimbochinji uye inopihwa sequentially kune mitsva mameseji. Iwo "mabhokisi" akazvimirira uye haana kuwiriraniswa mukati meinjini; kutaurirana pakati pawo kunoitika padanho rePHP. Iwe unogona kutarisa chimiro che data uye kugona kwemavara-injini kubva mukati pano.

Izvi zvaive zvakakwana kutsamba pakati pevashandisi vaviri. Fungidzira kuti chii chakazoitika?

Muna Chivabvu 2011, VKontakte yakaunza nhaurirano nevanoverengeka vatori vechikamu-multi-chat. Kuti tishande navo, takasimudza mapoka maviri matsva - nhengo-chat uye chat-nhengo. Yekutanga inochengeta data nezve chats nevashandisi, yechipiri inochengeta data nezvevashandisi nechats. Pamusoro pezvirongwa pachazvo, izvi zvinosanganisira, semuenzaniso, mushandisi anokoka uye nguva yavakawedzerwa kune chat.

"PHP, ngatitumire meseji kune chat," mushandisi anodaro.
"Huya, {username}," inodaro PHP.

Pane zvinoipira chirongwa ichi. Synchronization ichiri basa rePHP. Huru dzekukurukurirana uye vashandisi vanotumira meseji panguva imwe chete kwavari inyaya ine njodzi. Sezvo chinyorwa-injini chiitiko chinoenderana neuid, vatori vechikamu vanogona kugamuchira meseji yakafanana panguva dzakasiyana. Munhu aigona kurarama neizvi kana kufambira mberi kwakamira. Asi izvozvo hazvizoitiki.

Pakupera kwa2015, takatangisa mameseji enharaunda, uye pakutanga kwa2016, takavamba API yavo. Nekuuya kwechatbots hombe munharaunda, zvaigoneka kukanganwa nezve kunyange kugovera mutoro.

Bot yakanaka inogadzira mamirioni akati wandei pazuva - kunyangwe vashandisi vanonyanya kutaura havagone kuzvirumbidza neizvi. Izvi zvinoreva kuti zvimwe zviitiko zvemavara-injini, iyo mabhoti akadaro aigara, akatanga kutambura zvakanyanya.

Injini dzeMharidzo muna 2016 ndeye 100 zviitiko zvechat-nhengo uye nhengo-chats, uye 8000 text-injini. Ivo vakagadzikwa pane chiuru maseva, imwe neimwe iine 64 GB yekuyeuka. Sekutanga chiyero chekukurumidzira, isu takawedzera ndangariro neimwe 32 GB. Takafungidzira kufanotaura. Pasina kuchinja kukuru, izvi zvingava zvakakwana kwerinenge rimwe gore. Iwe unofanirwa kubata kubata Hardware kana kukwidziridza iwo dhatabhesi ivo pachavo.

Nekuda kwechimiro chezvivakwa, zvine musoro chete kuwedzera Hardware mune akawanda. Kureva, kanenge kaviri nhamba yemotokari - zviri pachena, iyi inzira inodhura. Tichagadzirisa.

Pfungwa itsva

Chinhu chepakati chemaitiro matsva kutaura. Chat ine rondedzero yemameseji anechekuita nayo. Mushandisi ane runyorwa rwechats.

Hushoma hunodiwa madhatabhesi maviri matsva:

• chat-injini. Iyi ichengetedzo yechat vectors. Chat yega yega ine vector yemameseji anechekuita nawo. Meseji yega yega ine mavara uye yakasarudzika meseji identifier mukati mechat - chat_local_id.
• mushandisi-injini. Uku ndiko kuchengetwa kwevashandisi vectors - zvinongedzo kune vashandisi. Wese mushandisi ane vector ye peer_id (interlocutors - vamwe vashandisi, akawanda-chat kana nharaunda) uye vector yemameseji. Peer_id yega yega ine vector yemameseji anechekuita nawo. Meseji yega yega ine chat_local_id uye yakasarudzika meseji ID yemushandisi iyeye - user_local_id.

Masumbu matsva anotaurirana nemumwe achishandisa TCP - izvi zvinovimbisa kuti kurongeka kwezvikumbiro hakuchinje. Izvo zvikumbiro ivo pachavo uye zvisimbiso kwavari zvakanyorwa pane hard drive - saka isu tinokwanisa kudzorera mamiriro emutsetse chero nguva mushure mekutadza kana kutangazve kweinjini. Sezvo mushandisi-injini uye chat-injini iine zviuru zvina shards imwe neimwe, mutsara wekukumbira pakati pemasumbu uchagoverwa zvakaenzana (asi muchokwadi hapana zvachose - uye inoshanda nekukurumidza).

Kushanda ne diski mune yedu dhatabhesi muzviitiko zvakawanda zvinoenderana nekubatanidzwa kwebhinari logi yekuchinja (binlog), static snapshots uye chikamu chikamu mundangariro. Shanduko mukati mezuva dzinonyorerwa kubhinlog, uye mufananidzo wemamiriro azvino unogadzirwa nguva nenguva. Snapshot muunganidzwa we data zvimiro zvakagadziridzwa nekuda kwezvinangwa zvedu. Iyo ine musoro (metaindex yemufananidzo) uye seti yemetafiles. Musoro unochengetwa zvachose mu RAM uye unoratidza kwaunotsvaga data kubva pamufananidzo. Metafile yega yega inosanganisira data ingangodiwa padhuze nenguva-semuenzaniso, ine chekuita nemushandisi mumwechete. Paunobvunza dhatabhesi uchishandisa iyo snapshot musoro, iyo inodiwa metafile inoverengwa, uyezve shanduko mubinlog yakaitika mushure mekunge mufananidzo wagadzirwa unoverengerwa. Iwe unogona kuverenga zvakawanda pamusoro pezvakanakira nzira iyi pano.

Panguva imwecheteyo, iyo data pa hard drive pachayo inoshandura kamwe chete pazuva - pakati pehusiku muMoscow, kana mutoro uri mudiki. Nekuda kweizvi (tichiziva kuti chimiro chiri pa diski chinogara chiripo zuva rese), isu tinokwanisa kutsiva mavheti nearrays ehukuru hwakatarwa - uye nekuda kweizvi, kuwana mundangariro.

Kutumira meseji muchirongwa chitsva chinotaridzika seizvi:

1. Iyo PHP backend inobata mushandisi-injini nechikumbiro chekutumira meseji.
2. mushandisi-injini inomiririra chikumbiro kune yaunoda chat-injini muenzaniso, iyo inodzokera kumushandisi-injini chat_local_id - yakasarudzika inozivisa meseji nyowani mukati meiyi chat. Iyo chat_engine inozotepfenyura meseji kune vese vanogamuchira muchat.
3. mushandisi-injini inogamuchira chat_local_id kubva ku-chat-engine uye inodzorera user_local_id kuPHP - chinozivisa meseji chakasiyana chemushandisi uyu. Ichi chiziviso chinobva chashandiswa, semuenzaniso, kushanda nemameseji kuburikidza neAPI.

Asi pamusoro pekutumira mameseji, unofanirwa kuita zvimwe zvinhu zvakakosha:

• Sublists, semuenzaniso, mameseji achangoburwa aunoona paunovhura rondedzero yekukurukurirana. Mameseji asina kuverenga, mameseji ane ma tag ("Zvakakosha", "Spam", nezvimwewo).
• Kudzvanya mameseji mune chat-injini
• Kuchengetedza mameseji muinjini-mushandisi
• Tsvaga (kuburikidza nemadialog ese uye mukati meimwe chaiyo).
• Real-time update (Longpolling).
• Kuchengetedza nhoroondo yekushandisa caching pane mobile clients.

All sublists ari kukurumidza kuchinja zvimiro. Kushanda navo tinoshandisa Splay miti. Sarudzo iyi inotsanangurwa nenyaya yekuti pamusoro pemuti isu dzimwe nguva tinochengeta chikamu chose chemeseji kubva muchidimbu - semuenzaniso, mushure mehusiku kudzoreredzwa, muti unoumbwa nepamusoro pepamusoro, iyo ine meseji yese ye sublist. Muti weSplay unoita kuti zvive nyore kuisa pakati pevheti yakadaro pasina kufunga nezvekuenzanisa. Mukuwedzera, Splay haichengete data isingakoshi, iyo inotiponesa ndangariro.

Mharidzo dzinosanganisira huwandu hukuru hweruzivo, kunyanya zvinyorwa, izvo zvinobatsira kukwanisa kumanikidza. Izvo zvakakosha kuti tikwanise kuburitsa nenzira kwayo kunyangwe meseji imwe chete. Inoshandiswa kudzvanya mameseji Huffman algorithm nemaitiro edu pachedu - semuenzaniso, tinoziva kuti mumeseji mazwi anochinjana ne "asiri mazwi" - nzvimbo, zviratidzo zvepumisikidzo - uye tinorangarirawo zvimwe zvekushandisa zviratidzo zvemutauro wechiRussian.

Sezvo kune vashoma vashandisi pane chats, kuchengetedza zvisina kujairika-kuwana disk zvikumbiro mu chat-injini, isu cache mameseji mujini-mushandisi.

Kutsvaga meseji kunoitwa semubvunzo we diagonal kubva kune-mushandisi-injini kuenda kune ese einjini yekutaura mamiriro ane hurukuro dzemushandisi uyu. Migumisiro inosanganiswa mumushandisi-injini pachayo.

Zvakanaka, zvese zvese zvakaverengerwa, chasara kuchinja kune chirongwa chitsva - uye zviri nani pasina vashandisi vanozviona.

Kutama kwedata

Saka, isu tine mameseji-injini inochengeta meseji nemushandisi, uye maviri masumbu ekutaura-nhengo uye nhengo-chats inochengeta data nezve akawanda-chat makamuri uye vashandisi vari mazviri. Nzira yekufamba sei kubva pane iyi kuenda kune itsva mushandisi-injini uye yekutaura-injini?

nhengo-chats muchirongwa chekare chaishandiswa kunyanya kugadzirisa. Isu takakurumidza kuendesa iyo inodiwa data kubva kwairi kuenda kunhengo-yekutaura, uyezve haina kuzotora chikamu mukutama.

Mutsara wenhengo dzechat. Inosanganisira 100 zviitiko, nepo chat-injini ine zviuru zvina. Kuti uendese iyo data, unofanirwa kuiunza mukuteerana - nekuda kweizvi, nhengo dzekukurukurirana dzakakamurwa kuita makopi zviuru zvina, uyezve kuverenga kwe-chat-members binlog kwakagoneswa muinjini yekutaura.

Iye zvino chat-injini inoziva nezve-multi-chat kubva kunhengo-dzekutaura, asi haisati yaziva chero chinhu nezvenhaurirano nevaviri vanopindirana. Nhaurirano dzakadaro dziri muzvinyorwa-injini zvichireva vashandisi. Pano takatora iyo data "musoro-pamusoro": imwe neimwe yekutaura-injini muenzaniso yakabvunza ese mameseji-injini zviitiko kana vaine nhaurirano yaidiwa.

Huru - chat-injini inoziva kuti ndezvipi-multi-chat chats iripo uye inoziva kuti ndedzipi nhaurirano dziripo.
Iwe unofanirwa kusanganisa mameseji mu-multi-chat chats kuti iwe upedze nerondedzero yemeseji mune yega chat. Chekutanga, chat-injini inotora kubva kutext-injini meseji yemushandisi kubva pane ino chat. Mune zvimwe zviitiko kune akawanda kwazvo (kusvika kumazana emamiriyoni), asi nekusarudzika kushomeka chat inokodzera zvachose mu RAM. Isu tine mameseji asina kurongeka, imwe neimwe mumakopi akati wandei - mushure mezvose, iwo ese anodhonzwa kubva akasiyana mameseji-injini mamiriro anoenderana nevashandisi. Chinangwa ndechekurongedza mameseji uye kubvisa makopi anotora nzvimbo isingakoshi.

Meseji yega yega ine chidhindo chenguva ine nguva yayakatumirwa uye mameseji. Isu tinoshandisa nguva kuronga - tinoisa anonongedzera kune ekare mameseji evatori vechikamu multichat uye tichienzanisa ma hashes kubva muzvinyorwa zvemakopi akatarwa, tichienda kukuwedzera timestamp. Zvine musoro kuti makopi achave ane hashi yakafanana uye timestamp, asi mukuita izvi hazvisi nguva dzose. Sezvaunorangarira, kuwiriranisa muchirongwa chekare chakaitwa nePHP - uye muzviitiko zvisingawanzo, nguva yekutumira meseji imwechete yakasiyana pakati pevashandisi vakasiyana. Muzviitiko izvi, takazvibvumira kugadzirisa timetamp - kazhinji mukati mesekondi. Dambudziko rechipiri nderekurongeka kwakasiyana kwemeseji kune vakasiyana vanogamuchira. Mumamiriro ezvinhu akadaro, isu takabvumira imwe kopi kuti igadzirwe, iine akasiyana maodha sarudzo dzevashandisi vakasiyana.

Mushure meizvi, data nezve mameseji mumultichat inotumirwa kumushandisi-injini. Uye pano panouya chinhu chisingafadzi chemeseji dzakaunzwa kunze kwenyika. Mukushanda kwakajairika, mameseji anouya kunjini anorairwa zvakanyanya mukukwira kurongeka nemushandisi_local_id. Mameseji akapinzwa kunze kubva muinjini yekare kupinda muinjini yemushandisi akarasikirwa nechinhu ichi chakakosha. Panguva imwecheteyo, kuti zvive nyore kuedza, iwe unofanirwa kukwanisa kukurumidza kuvawana, tsvaga chimwe chinhu mavari uye kuwedzera zvitsva.

Isu tinoshandisa yakakosha data chimiro kuchengetedza mameseji anounzwa kunze kwenyika.

Inomiririra vector yehukuru munhu wese aripi - dzakasiyana uye dzakarongedzerwa mukudzika kurongeka, nehurongwa hunokosha hwezvinhu. Muchikamu chimwe nechimwe chine indices zvinhu zvakarongwa. Kutsvaga chinhu muchimiro chakadaro kunotora nguva kuburikidza binary searches. Kuwedzerwa kwechimwe chinhu kunoderedzwa pamusoro .

Saka, takafunga nzira yekuendesa data kubva kune ekare injini kuenda kune itsva. Asi izvi zvinotora mazuva akati wandei - uye hazvigoneke kuti mumazuva ano vashandisi vedu vasiye tsika yekunyorerana. Kuti urege kurasikirwa nemeseji panguva ino, isu tinochinja kune chirongwa chebasa chinoshandisa ese ekare uye matsva masumbu.

Dhata inonyorerwa kunhengo-nhengo uye mushandisi-injini (uye kwete kune zvinyorwa-injini, senge mukushanda kwakajairwa zvinoenderana nechirongwa chekare). user-engine proxies chikumbiro chekutaura-injini - uye apa maitiro anoenderana nekuti iyi chat yakatobatanidzwa here kana kuti kwete. Kana iyo chat isati yabatanidzwa, iyo-chat-injini hainyore meseji yacho pachayo, uye kugadzirisa kwayo kunoitika chete mune yemavara-injini. Kana iyo chat yakatobatanidzwa kuita chat-injini, inodzosera chat_local_id kune mushandisi-injini uye inotumira meseji kune vese vanoigamuchira. mushandisi-injini proxies data rese kune zvinyorwa-injini - kuitira kuti kana chimwe chinhu chikaitika, tinogona kugara tichidzokera kumashure, tiine data rese razvino muinjini yekare. text-engine inodzosa user_local_id, iyo inochengeterwa mushandisi-injini uye inodzokera kumashure.

Nekuda kweizvozvo, maitiro ekuchinja anotaridzika seizvi: isu tinobatanidza isina mushandisi-injini uye chat-injini masumbu. chat-injini inoverenga iyo yese chat-nhengo binlog, ipapo proxying inotanga zvinoenderana nechirongwa chakatsanangurwa pamusoro. Isu tinotamisa data rekare uye tinowana maviri akawiriraniswa masumbu (ekare uye matsva). Chasara kushandura kuverenga kubva kumavara-injini kuenda kune mushandisi-injini uye kudzima proxying.

Mhinduro

Nekuda kwemaitiro matsva, ese ekuita metrics einjini akagadziridzwa uye matambudziko nekuenderana kwedata akagadziriswa. Iye zvino tinogona kukurumidza kuita zvinhu zvitsva mumeseji (uye tatotanga kuita izvi - isu takawedzera huwandu hwehuwandu hwevatori vechikamu chekutaura, takaita yekutsvaga mameseji anotumirwa, takatangisa mameseji akapinirwa uye kukwidza muganho pahuwandu hwemeseji pamushandisi) .

Shanduko dzepfungwa dzakakura zvechokwadi. Uye ndinoda kuziva kuti izvi hazviwanzoreve makore ese ebudiriro nechikwata chikuru uye zviuru zvemitsara yekodhi. chat-injini uye mushandisi-injini pamwe chete nedzimwe nyaya dzekuwedzera seHuffman yekumanikidza meseji, Splay miti uye chimiro chekutumira mameseji isingasviki zviuru makumi maviri zvemitsara yekodhi. Uye ivo vakanyorwa nevagadziri vatatu mumwedzi gumi chete (zvisinei, zvakakodzera kuyeuka kuti all the vatatu mugadziri - shasha dzenyika mumapurogiramu emitambo).

Uyezve, pachinzvimbo chekupeta kaviri nhamba yemaseva, isu takadzikisa nhamba yavo nehafu - ikozvino mushandisi-injini uye yekutaura-injini inorarama pamachina mazana mashanu emuviri, nepo chirongwa chitsva chine hombe yemusoro yekuremedza. Takachengeta mari yakawanda pamidziyo - inenge 500 miriyoni + $5 zviuru pagore mukushandisa mari.

Tinoedza kuwana mhinduro dzakanakisisa dzezvinetso zvakaoma uye zvakakura. Tine akawanda awo - ndosaka tiri kutsvaga vanogadzira vane tarenda mudhipatimendi re database. Kana iwe uchida uye uchiziva kugadzirisa matambudziko akadai, iva neruzivo rwakanyanya rwealgorithms uye data zvimiro, tinokukoka iwe kuti ubatane nechikwata. Bata yedu HRkuti uwane ruzivo.

Kunyangwe iyi nyaya isiri pamusoro pako, ndapota cherechedza kuti isu tinokoshesa kurudziro. Udza shamwari nezvazvo developer vacancies, uye kana akabudirira kupedza nguva yekuongorora, iwe uchagamuchira bhonasi ye100 zviuru zveRubles.

Source: www.habr.com