Nhoroondo yekusikwa kweVKontakte iri paWikipedia; yakataurwa naPavel pachake. Zvinoita sokuti munhu wose anotomuziva. Nezve vemukati, dhizaini uye chimiro chesaiti paHighLoad ++ Pavel . Masevha mazhinji akadonha kubva ipapo, saka isu tichavandudza ruzivo: isu tinoitsemura, tobvisa mukati, toiyera, uye titarise mudziyo weVK kubva pakuona kwehunyanzvi.
Alexey Akulovich () backend mugadziri muboka reVKontakte. Chinyorwa chemushumo uyu imhinduro yakabatana kumibvunzo inowanzo bvunzwa nezve mashandiro epuratifomu, zvivakwa, maseva uye kudyidzana pakati pavo, asi kwete nezvebudiriro, zvinoti. nezvesimbi. Zvakaparadzana, nezve dhatabhesi uye izvo VK inazvo panzvimbo, nezve kuunganidza matanda uye kutarisa purojekiti yese zvakazara. Tsanangudzo pasi pekucheka.
Kweanopfuura makore mana ndanga ndichibata nemhando dzose dzemabasa ane chekuita nekumashure.
- Kurodha, kuchengetedza, kugadzirisa, kugovera midhiya: vhidhiyo, live kutenderera, odhiyo, mafoto, zvinyorwa.
- Zvivakwa, chikuva, dhizaini yekutarisa, matanda, dunhu cache, CDN, proprietary RPC protocol.
- Kubatanidzwa nemasevhisi ekunze: push notices, yekunze link parsing, RSS feed.
- Kubatsira vaunoshanda navo nemibvunzo yakasiyana siyana, mhinduro dzacho dzinoda kunyura mune isingazivikanwe kodhi.
Munguva iyi, ndaive neruoko muzvikamu zvakawanda zvesaiti. Ndinoda kugovera chiitiko ichi.
General architecture
Zvese, semazuva ese, zvinotanga nesevha kana boka remaseva anobvuma zvikumbiro.
Front server
Iyo yekumberi sevha inogamuchira zvikumbiro kuburikidza neHTTPS, RTMP uye WSS.
HTTPS - izvi zvikumbiro zvemhando huru uye nharembozha dzesaiti: vk.com uye m.vk.com, uye vamwe vari pamutemo uye zvisiri pamutemo vatengi veAPI yedu: nhare vatengi, vatumwa. Tine reception RTMP-traffic yeLive nhepfenyuro ine akasiyana epamberi maseva uye WSS- kubatana kweKutenderera API.
Kune HTTPS neWSS pamaseva zvakakosha nginx. Kune RTMP nhepfenyuro, isu nguva pfupi yadarika takachinjira kune yedu mhinduro kive, asi zviri kunze kwehukuru hwemushumo. Nekutadza kushivirira, maseva aya anoshambadzira akajairika IP kero uye anoita mumapoka kuitira kuti kana paine dambudziko pane imwe yemaseva, zvikumbiro zvevashandisi hazvisi kurasika. Kune HTTPS neWSS, maseva mamwe chete aya encrypt traffic kuitira kuti atore chikamu cheCPU mutoro pavari.
Isu hatizo taura zvimwe nezve WSS neRTMP, asi chete nezve yakajairwa HTTPS zvikumbiro, izvo zvinowanzobatanidzwa newebhu purojekiti.
Dzorera
Seri kumberi kunowanzo kune maseva ekumashure. Vanoita zvikumbiro izvo sevha yepamberi inogamuchira kubva kune vatengi.
ichi kPHP maseva, pane iyo HTTP daemon iri kushanda, nekuti HTTPS yatove decrypted. kPHP iseva inoshanda prefork models: anotanga hunyanzvi maitiro, boka revana maitiro, anopfuudza zvigadziko zvekuteerera kwavari uye ivo vanogadzirisa zvikumbiro zvavo. Muchiitiko ichi, maitiro haatangwazve pakati pechikumbiro chega chega kubva kumushandisi, asi ingo gadzirisa mamiriro avo kune yekutanga zero-kukosha mamiriro - chikumbiro mushure mekukumbira, pane kutangazve.
Kugoverwa kwemitoro
Yese yedu yekumashure haisi dziva rakakura remakina anogona kugadzirisa chero chikumbiro. Isu ivo akakamurwa kuva mapoka akasiyana: general, mobile, api, video, staging... Dambudziko riri paboka rakasiyana remakina harizokanganisa mamwe ese. Kana paine matambudziko nevhidhiyo, mushandisi anoteerera mimhanzi haatomboziva nezvematambudziko. Ndeipi backend yekutumira chikumbiro kune inosarudzwa ne nginx kumberi zvinoenderana neiyo config.
Metric kuunganidza uye kuenzanisa
Kuti tinzwisise kuti ingani mota dzatinofanira kuva nadzo muboka rega rega, isu usavimbe neQPS. Iwo ekumashure akasiyana, ane zvikumbiro zvakasiyana, chikumbiro chega chega chine kuomarara kwakasiyana kwekuverenga QPS. Ndosaka isu isu tinoshanda nepfungwa yekuremerwa pane sevha yakazara - paCPU uye perf.
Tine zviuru zvemaseva akadaro. Sevha yega yega yemuviri inomhanyisa boka rekPHP kudzokorodza ese macores (nekuti kPHP ine shinda imwe chete).
Content Server
CS kana Content Server ichengetedzo. CS iseva inochengeta mafaera uye zvakare inogadziridza mafaera akaiswa uye ese marudzi ese emashure anowirirana mabasa anopihwa iyo huru yewebhu frontend.
Tine makumi ezviuru zvemaseva emuviri anochengeta mafaera. Vashandisi vanofarira kurodha mafaera, uye isu tinoda kuchengeta nekuagovera. Mamwe emaseva aya akavharwa neakakosha pu/pp maseva.
pu/pp
Kana iwe ukavhura iyo network tebhu muVK, wakaona pu/pp.
Chii chinonzi pu/pp? Kana tikavhara imwe sevha mushure meimwe, saka pane mbiri sarudzo dzekuisa uye kurodha faira kune server yakavharwa: zvakananga kuburikidza http://cs100500.userapi.com/path kana kuburikidza nepakati server - http://pu.vk.com/c100500/path.
Pu izita renhoroondo yekurodha mafoto, uye pp ipikicha proxy. Ndiko kuti, imwe sevha ndeyekurodha mafoto, uye imwe ndeyekuisa. Iye zvino haasi mapikicha chete akatakurwa, asi zita rakachengetedzwa.
Aya maseva kumisa zvikamu zveHTTPSkubvisa mutoro we processor kubva mudura. Zvakare, sezvo mafaera emushandisi achigadziriswa pamaseva aya, ruzivo rusina kusimba rwakachengetwa pamakina aya, zviri nani. Semuenzaniso, HTTPS encryption kiyi.
Sezvo michina yakavharwa nemimwe michina yedu, hatigone kuvapa "chena" ekunze IPs, uye. ipa "grey". Nenzira iyi takachengetedza padziva reIP uye takavimbiswa kuchengetedza michina kubva kunze kwekuwana - hapana IP yekupinda mairi.
Resiliency pane yakagovaniswa IPs. Panyaya yekushivirira kukanganisa, chirongwa chinoshanda zvakafanana - akati wandei maseva emuviri ane yakajairika IP yemuviri, uye hardware iri pamberi pavo inosarudza kupi kutumira chikumbiro. Ndichataura dzimwe sarudzo gare gare.
Nyaya inokakavadzana ndeyekuti munyaya iyi mutengi anochengeta mashoma ekubatanidza. Kana paine IP yakafanana yemichina yakawanda - ine muenzi mumwechete: pu.vk.com kana pp.vk.com, mutengi webrowser ane muganhu pahuwandu hwekukumbira panguva imwe chete kune mumwe muenzi. Asi munguva yehubiquitous HTTP/2, ndinotenda kuti izvi hazvichakoshese zvakadaro.
Kusakanganiswa kuri pachena kwechirongwa ndechekuti inofanirwa kupomba traffic yese, iyo inoenda kunzvimbo yekuchengetedza, kuburikidza neimwe sevha. Sezvo isu tichipomba traffic kuburikidza nemichina, isu hatigone kupomba inorema traffic, semuenzaniso, vhidhiyo, tichishandisa iyo yakafanana chirongwa. Isu tinozvifambisa zvakananga - yakaparadzana yakananga yekubatanidza kune yakaparadzana storages yakanangana nevhidhiyo. Isu tinotumira zvinyorwa zvakareruka kuburikidza neproxy.
Munguva pfupi yapfuura takawana yakagadziridzwa vhezheni yeproxy. Zvino ini ndichakuudza kuti vakasiyana sei kubva kune vakajairwa uye nei izvi zvichidikanwa.
zuva
MunaGunyana 2017, Oracle, iyo yakambotenga Zuva, . Tinogona kutaura kuti panguva ino kambani yakaguma kuvapo. Pakusarudza zita rehurongwa hutsva, vatungamiriri vedu vakasarudza kubhadhara mutero kundangariro yekambani iyi uye vakatumidza iyo itsva system Zuva. Pakati pedu tinongomuti "zuva".
pp yaive nematambudziko mashoma. Imwe IP paboka - isingashande cache. Sevha dzinoverengeka dzemuviri dzinogovera yakajairwa IP kero, uye hapana nzira yekudzora iyo server iyo chikumbiro chinoenda kwairi. Naizvozvo, kana vashandisi vakasiyana vachiuya kune imwechete faira, zvino kana paine cache pane aya maseva, iyo faira inoguma mune cache yega yega sevha. Ichi chirongwa chisina kunyatsoshanda, asi hapana chaigona kuitwa.
Naizvozvo - hatigone kugovanisa zviri mukati, nekuti isu hatigone kusarudza chaiyo sevha yeboka iri - vane yakajairika IP. Zvakare nekuda kwezvimwe zvikonzero zvemukati zvatinazvo zvakanga zvisingaiti kuisa maseva akadaro mumatunhu. Vakamira muSt.
Nezuva, takashandura sarudzo yekusarudza. Iye zvino tawana anycast routing: inochinja nzira, anycast, tarisa wega daemon. Sevha yega yega ine yayo yega IP, asi yakajairika subnet. Zvese zvinogadziriswa nenzira yekuti kana sevha imwe ikatadza, traffic inopararira kune mamwe maseva eboka rimwe chete otomatiki. Iye zvino zvinokwanisika kusarudza imwe sevha, hapana redundant caching, uye kuvimbika hakuna kukanganiswa.
Kutsigira uremu. Iye zvino tinogona kukwanisa kuisa michina yemasimba akasiyana-siyana sezvinodiwa, uyewo, kana pane zvinetso zvenguva pfupi, shandura uremu hwe "zuva" rinoshanda kuderedza mutoro pavari, kuitira kuti "vazorore" uye vatange kushanda zvakare.
Kugovaniswa nezvinyorwa id. Chinhu chinosetsa nezve sharding: isu tinowanzo shard zvemukati kuitira kuti vashandisi vakasiyana vaende kune imwechete faira kuburikidza ne "zuva" rimwechete kuti vave neyakafanana cache.
Isu munguva pfupi yapfuura takatanga iyo "Clover" application. Uyu mubvunzo wepamhepo mukutepfenyura mhenyu, uko muenzi anobvunza mibvunzo uye vashandisi vanopindura munguva chaiyo, vachisarudza sarudzo. Iyo app ine chat uko vashandisi vanogona kutaura. Inogona kubatana kune nhepfenyuro panguva imwe chete vanhu vanopfuura 100 zviuru. Vese vanonyora mameseji anotumirwa kune vese vatori vechikamu, uye avatar inouya pamwe neshoko. Kana 100 zviuru zvevanhu vachiuya kune imwe avatar mu "zuva" rimwe chete, saka dzimwe nguva inogona kutenderera shure kwegore.
Kuti utsungirire kuputika kwezvikumbiro zvefaira rimwe chete, ndezveimwe mhando yezvinyorwa zvatinoshandura chirongwa cheupenzi chinoparadzira mafaira kumativi ose anowanikwa "zuva" munharaunda.
Zuva kubva mukati
Reverse proxy pane nginx, cache ingave muRAM kana pakukurumidza Optane/NVMe disks. Muenzaniso: http://sun4-2.userapi.com/c100500/path - chinongedzo kune "zuva", iro riri munharaunda yechina, yechipiri server boka. Iyo inovhara iyo nzira faira, iyo inorara mumuviri pane server 100500.
pabutiro
Isu tinowedzera imwe imwe node kune yedu yekuvaka chirongwa - iyo caching nharaunda.
Pazasi pane dhizaini yekuronga redunhu caches, kune vanenge 20 vavo. Idzi ndidzo nzvimbo dzinowanikwa cache uye "zuva", izvo zvinogona cache traffic kuburikidza ivo pachavo.
Uku ndiko caching yezvinyorwa zvemultimedia; hapana data remushandisi rakachengetwa pano - mimhanzi chete, vhidhiyo, mafoto.
Kuti tione nharaunda yemushandisi, isu isu tinounganidza BGP network prefixes yakaziviswa mumatunhu. Panyaya yekudzokera shure, isu tinofanirwawo kusiyanisa dhatabhesi regeoip kana tisina kuwana IP nemaprefixes. Isu tinosarudza dunhu neIP yemushandisi. Mune kodhi, tinogona kutarisa kune imwe kana anopfuura matunhu emushandisi - iwo mapoinzi kwaari padyo nenzvimbo.
Sei kushanda?
Isu tinoverenga kufarirwa kwemafaira nenzvimbo. Iko kune nhamba yedunhu cache kunowanikwa mushandisi, uye iyo faira identifier - tinotora vaviri ava uye towedzera chiyero nekurodha yega yega.
Panguva imwecheteyo, madhimoni - masevhisi mumatunhu - nguva nenguva anouya kuAPI achiti: "Ini ndiri akadai uye cache, ndipei runyoro rwemafaira anozivikanwa mudunhu mangu asati ave pandiri. β Iyo API inoburitsa boka remafaira akarongwa nechiyero, iyo daemon inoadhawunirodha, inoenda nawo kumatunhu uye inoendesa mafaera kubva ipapo. Uyu ndiwo mutsauko wakakosha pakati pe pu/pp neZuva kubva kumacache: ivo vanopa iyo faira kuburikidza ivo pachavo nekukasira, kunyangwe faira iri risiri mucache, uye cache inotanga kurodha faira kwairi, yobva yatanga kuridzorera.
Munyaya iyi tinowana zvemukati zviri pedyo nevashandisi uye kuparadzira mutoro wetiweki. Semuenzaniso, chete kubva kuMoscow cache tinogovera zvinopfuura 1 Tbit / s panguva dzepamusoro.
Asi pane matambudziko - cache servers haisi rubber. Kune zvakakurumbira zvemukati, dzimwe nguva hapana network yakakwana yeseva yakaparadzana. Yedu cache maseva i40-50 Gbit/s, asi pane zvirimo zvinovhara zvachose chiteshi chakadaro. Tiri kuenda kukushandisa kuchengetedza kweanopfuura kopi imwe yemafaira ane mukurumbira mudunhu. Ndinovimba kuti tichazviita panopera gore.
Takatarisa general architecture.
- Maseva epamberi anobvuma zvikumbiro.
- Inodzosera iyo process zvikumbiro.
- Matura anovharwa nemhando mbiri dzeproxies.
- Regional caches.
Chii chisipo kubva pamufananidzo uyu? Ehe, iyo dhatabhesi yatinochengetera data.
Databases kana injini
Isu tinovadaidza kwete dhatabhesi, asi injini - Injini, nekuti isu hatina dhatabhesi mupfungwa inogamuchirwa.
Ichi chiyero chinodiwa. Izvi zvakaitika nekuti muna 2008-2009, apo VK yaive nekukura kukuru mukuzivikanwa, chirongwa chakashanda zvachose paMySQL neMemcache uye pakanga paine matambudziko. MySQL yaida kuputsa uye kukanganisa mafaira, mushure mezvo yaisazopora, uye Memcache zvishoma nezvishoma yakadzikira mukuita uye yaifanira kutangwazve.
Zvinoitika kuti purojekiti yakawedzera kufarirwa yaive nekuchengetedza kunoramba kuripo, iyo inoshatisa data, uye cache, iyo inononoka. Mumamiriro ezvinhu akadaro, zvakaoma kukudziridza purojekiti inokura. Zvakasarudzwa kuedza kunyorazve zvinhu zvakakosha izvo chirongwa chakanga chakanangana pazviri pamabhasikoro edu pachedu.
Mhinduro yacho yakabudirira. Paiva nemukana wekuita izvi, pamwe chete nezvinodiwa zvakanyanya, nokuti dzimwe nzira dzekuwedzera dzakanga dzisipo panguva iyoyo. Pakanga pasina boka remadatabase, NoSQL yanga isati yavapo, kwaingova neMySQL, Memcache, PostrgreSQL - uye ndizvozvo.
Universal kushanda. Iko kusimukira kwakatungamirwa nechikwata chedu chevagadziri veC uye zvese zvakaitwa nenzira inowirirana. Pasinei neinjini, ivo vese vaive neyakafanana faira fomati yakanyorerwa dhisiki, iwo akafanana ekutanga paramita, akagadziridzwa masaini nenzira imwechete, uye akazvibata zvakangofanana kana kune edge mamiriro nematambudziko. Nekukura kweinjini, zviri nyore kuti vatariri vashande sisitimu - hapana zoo inoda kuchengetedzwa, uye vanofanirwa kudzidzirazve mashandisiro ega yega yega yechitatu-bato dhatabhesi, izvo zvakaita kuti zvikwanise kukurumidza uye zviri nyore kuwedzera. nhamba yavo.
Mhando dzeinjini
Chikwata chakanyora injini shoma. Heano mamwe acho chete: shamwari, mazano, mufananidzo, ipdb, mavara, zvinyorwa, matanda, memcached, meowdb, nhau, nostradamus, mufananidzo, playlists, pmemcached, sandbox, kutsvaga, kuchengetedza, kufarira, mabasa, ...
Pabasa rega rega rinoda chimiro che data kana kuita zvikumbiro zveatypical, timu yeC inonyora injini nyowani. Kuregererei.
Tine injini yakasiyana memcached, iyo yakafanana neyenguva dzose, asi ine boka rezvakanaka, uye iyo isinganonoke. Kwete ClickHouse, asi inoshandawo. Inowanikwa zvakasiyana pmemcached - ichi chi persist memcached, iyo inogonawo kuchengetedza data pa diski, uyezve, kupfuura inokwana mu RAM, kuitira kuti usarasikirwe nedata paunotangazve. Kune akasiyana injini dzemabasa ega ega: mitsetse, rondedzero, seti - zvese izvo purojekiti yedu inoda.
Masumbu
Kubva pakuona kwekodhi, hapana chikonzero chekufunga nezveinjini kana dhatabhesi semaitiro, masangano, kana zviitiko. Iyo kodhi inoshanda chaizvo nemasumbu, ane mapoka einjini - mhando imwe pachikwata. Ngatitii pane memcached cluster - ingori boka remichina.
Iyo kodhi haidi kuziva iyo chaiyo nzvimbo, saizi, kana nhamba yemaseva zvachose. Anoenda kuchikwata achishandisa chimwe chiziviso.
Kuti izvi zvishande, unofanirwa kuwedzera chimwe chinhu chiri pakati pekodhi neinjini - proxy.
RPC proxy
Proxy bhazi rekubatanidza, iyo inenge saiti yese inomhanya. Panguva imwe chete tine hapana kuwanikwa kwesevhisi - pachinzvimbo, pane gadziriso yeiyi proxy, iyo inoziva nzvimbo yemasumbu ese uye shards yese yesumbu iri. Izvi ndozvinoita maadmins.
Mapurogiramu haana hanya zvachose kuti ingani, kupi uye chii chinodhura - vanongoenda kuboka. Izvi zvinotibvumira zvakawanda. Paunenge uchigamuchira chikumbiro, mumiriri anodzoreredza chikumbiro, achiziva kupi-inosarudza izvi pachayo.
Muchiitiko ichi, proxy inzvimbo yekudzivirira kubva pakukundikana kwebasa. Kana imwe injini ikadzikira kana kuparara, ipapo mumiriri anonzwisisa izvi uye anopindura zvinoenderana nedivi remutengi. Izvi zvinokubvumira kuti ubvise nguva - iyo code haimirire kuti injini ipindure, asi inonzwisisa kuti haisi kushanda uye inoda kuzvibata nenzira yakasiyana. Iyo kodhi inofanirwa kugadzirirwa chokwadi chekuti dhatabhesi haashande nguva dzose.
Mashandisirwo chaiwo
Dzimwe nguva isu tichiri kuda chaizvo kuve neimwe mhando yeisina-standard mhinduro seinjini. Panguva imwecheteyo, zvakasarudzwa kusashandisa yedu yakagadzirira-yakagadzirwa rpc-proxy, yakasikwa yakananga injini dzedu, asi kuita puroksi yakasiyana yebasa racho.
YeMySQL, yatichiri nayo pano neapo, tinoshandisa db-proxy, uye yeClickHouse - Kittenhouse.
Inoshanda kazhinji seizvi. Pane imwe sevha, inomhanya kPHP, Go, Python - kazhinji, chero kodhi inogona kushandisa yedu RPC protocol. Iyo kodhi inomhanya munharaunda pane RPC proxy - sevha yega yega panowanikwa kodhi inomhanyisa yayo yemuno proxy. Pakukumbira, mumiriri anonzwisisa kwekuenda.
Kana imwe injini ichida kuenda kune imwe, kunyange kana iri muvakidzani, inopinda nemumiririri, nokuti muvakidzani angave ari mune imwe nzvimbo yedata. Injini haifanire kuvimba nekuziva nzvimbo yechimwe chinhu kunze kwayo - iyi ndiyo mhinduro yedu. Asi chokwadi pane zvisizvo :)
Muenzaniso weTL-scheme zvinoenderana neinjini dzese dzinoshanda.
memcache.not_found = memcache.Value;
memcache.strvalue value:string flags:int = memcache.Value;
memcache.addOrIncr key:string flags:int delay:int value:long = memcache.Value;
tasks.task
fields_mask:#
flags:int
tag:%(Vector int)
data:string
id:fields_mask.0?long
retries:fields_mask.1?int
scheduled_time:fields_mask.2?int
deadline:fields_mask.3?int
= tasks.Task;
tasks.addTask type_name:string queue_id:%(Vector int) task:%tasks.Task = Long;Iyi ibhinari protocol, iyo yepedyo analogue iyo iri protobuf. Iyo schema inotsanangura minda yakasarudzika, yakaoma mhando - edzedzero yeakavakirwa-mukati ma scalar, uye mibvunzo. Zvese zvinoshanda zvinoenderana neiyi protocol.
RPC pamusoro peTL pamusoro peTCP/UDP⦠UDP?
Isu tine RPC protocol yekuita zvikumbiro zveinjini inomhanya pamusoro peiyo TL chirongwa. Izvi zvese zvinoshanda pamusoro peTCP/UDP yekubatanidza. TCP inonzwisisika, asi nei tichida UDP kazhinji?
UDP inobatsira dzivisa dambudziko renhamba huru yekubatanidza pakati pemaseva. Kana sevha yega yega ine RPC proxy uye, kazhinji, inogona kuenda kune chero injini, saka kune makumi ezviuru zveTCP yekubatanidza pane server. Pane mutoro, asi hauna maturo. Panyaya yeUDP dambudziko iri harisipo.
Hapana kusakwana TCP kubata maoko. Iri idambudziko rakajairwa: kana injini nyowani kana sevha nyowani yatangwa, akawanda TCP maratidziro anotangwa kamwechete. Kune zvikumbiro zvidiki zvidiki, semuenzaniso, UDP payload, kutaurirana kwese pakati pekodhi uye injini iri maviri UDP mapaketi: imwe inobhururuka ichienda kune imwe nzira, yechipiri kune imwe. Imwe rwendo rwekutenderera - uye kodhi yakagamuchira mhinduro kubva kune injini pasina kubata ruoko.
Hongu, zvese zvinoshanda chete ine chikamu chidiki kwazvo chekurasikirwa kwepaketi. Iyo protocol ine tsigiro yekudzoreredza uye nguva yekubuda, asi kana tikarasikirwa nezvakawanda, isu tichawana inenge TCP, iyo isinga batsire. Isu hatityairi UDP mhiri kwemakungwa.
Tine zviuru zvemaseva akadaro, uye chirongwa chakafanana: paki yeinjini inoiswa pane yega yega sevha yemuviri. Iwo anonyanya kurongedzerwa-tambo kuti amhanye nekukurumidza sezvinobvira pasina kuvharira, uye anogovaniswa senge-imwe-tambo mhinduro. Panguva imwecheteyo, isu hatina chinhu chakavimbika kupfuura injini idzi, uye kutarisisa kwakawanda kunobhadharwa kune inoenderera kuchengetedza data.
Kuramba kuchengetwa kwedata
Injini dzinonyora mabhii. A binlog ifaira pakupera kwayo iyo chiitiko chekuchinja mamiriro kana data inowedzerwa. Mune mhinduro dzakasiyana inodanwa zvakasiyana: binary log, , , asi nheyo imwe cheteyo.
Kuti udzivise injini kubva pakuverengazve bhinlog yose kwemakore akawanda kana ichitangazve, injini dzinonyora snapshots - ikozvino mamiriro. Kana zvichidiwa, vanotanga kuverenga kubva pairi, uye vozopedzisa kuverenga kubva mubhinlog. Ese mabinlogs akanyorwa mune imwechete bhinari fomati - zvinoenderana neTL chirongwa, kuitira kuti maadmins agone kuvapa zvakaenzana nemidziyo yavo. Iko hakuna kudiwa kwakadai kwemifananidzo. Pane musoro mukuru unoratidza kuti ndeupi snapshot iri int, mashiripiti ejini, uye ndeupi muviri usina kukosha kune chero munhu. Iri idambudziko neinjini yakarekodha snapshot.
Ini ndichakurumidza kutsanangura nheyo yekushanda. Pane sevha iyo injini inoshanda pairi. Anovhura bhinlog nyowani isina chinhu yekunyora uye anonyora chiitiko chekuchinja kwairi.
Pane imwe nguva, anosarudza kutora mufananidzo pachake, kana kuti anogamuchira chiratidzo. Iyo sevha inogadzira faira nyowani, inonyora mamiriro ayo ese mairi, inoisa yazvino binlog saizi - offset - kusvika kumagumo efaira, uye inoenderera mberi nekunyora. A new binlog haina kugadzirwa.
Pane imwe nguva, injini payakatangazve, pachava ne binlog uye snapshot pa diski. Injini inoverenga iyo yose snapshot uye inosimudza mamiriro ayo pane imwe nguva.
Inoverenga nzvimbo yaive panguva yagadzirwa mufananidzo uye saizi yebhinlog.
Inoverenga kupera kwebinlog kuti uwane mamiriro azvino uye anoenderera mberi nekunyora zvimwe zviitiko. Ichi chirongwa chakareruka; injini dzedu dzese dzinoshanda zvinoenderana nazvo.
Data replication
Nekuda kweizvozvo, data replication mune yedu statement-based - tinonyora mubinlog kwete chero peji inoshanduka, asi iyo chinja zvikumbiro. Zvakanyanya kufanana neinouya pamusoro petiweki, zvakangogadziriswa zvishoma.
Iyo imwechete chirongwa chinoshandiswa kwete chete kudzokorora, asiwo kugadzira backups. Tine injini - nyanzvi yekunyora inonyora kune binlog. Mune chero imwe nzvimbo iyo ma admins akaimisa, iyi binlog inoteedzerwa, uye ndizvozvo - isu tine backup.
Kana zvichidikanwa kuverenga replicaKuti uderedze iyo CPU kuverenga mutoro, injini yekuverenga inongotangwa, iyo inoverenga kupera kwebinlog uye inoita iyi mirairo munharaunda.
Iyo lag pano idiki kwazvo, uye zvinokwanisika kuona kuti yakawanda sei iyo replica inosara kumashure kwatenzi.
Kugovana data muRPC proxy
Sharding inoshanda sei? Ko mumiriri anonzwisisa sei kuti cluster shard yekutumira kwairi? Iyo kodhi haina kuti: "Tumira kune gumi nemashanu shards!" - kwete, izvi zvinoitwa nemumiriri.
Iyo yakapfava chirongwa ndeyekutanga - nhamba yekutanga mukukumbira.
get(photo100_500) => 100 % N.
Uyu muenzaniso weiyo yakapfava memcached text protocol, asi, hongu, mibvunzo inogona kuve yakaoma uye yakarongeka. Muenzaniso unotora nhamba yekutanga mumubvunzo uye inosara kana yakamurwa nehukuru hweboka.
Izvi zvinobatsira kana tichida kuve nedata yenzvimbo yechinhu chimwe chete. Ngatitii zana mushandisi kana ID yeboka, uye isu tinoda kuti data rese rechimwe chinhu rive pane imwe shard yemibvunzo yakaoma.
Kana isu tisina basa kuti zvikumbiro zvinopararira sei musumbu, pane imwe sarudzo - kuzadza chikamu chose.
hash(photo100_500) => 3539886280 % N
Isu tinowanawo hashi, iyo yasara yekupatsanurwa uye shard nhamba.
Zvose izvi zvingasarudzwa zvinoshanda chete kana isu takagadzirira chokwadi chokuti patinowedzera ukuru hwesumbu, tichaiparadzanisa kana kuwedzera kakawanda. Semuyenzaniso, takanga tine 16 shards, hatina zvakakwana, tinoda zvimwe - tinogona kuwana 32 zvakachengeteka pasina nguva. Kana tichida kuwedzera kwete kuwanda, pachava nenguva yekuderera, nokuti hatizokwanisi kunyatsoparadzanisa zvinhu zvose pasina kurasikirwa. Idzi sarudzo dzinobatsira, asi kwete nguva dzose.
Kana isu tichida kuwedzera kana kubvisa nhamba inopokana yemaseva, tinoshandisa Inopindirana hashing pachindori a la Ketama. Asi panguva imwecheteyo, isu tinorasikirwa zvachose nenzvimbo yedata; isu tinofanirwa kubatanidza chikumbiro kune cluster kuitira kuti chidimbu chimwe nechimwe chidzose mhinduro yacho diki, uye tozobatanidza mhinduro kune proxy.
Pane zvikumbiro zvakanyanyisa. Zvinoita seizvi: RPC proxy inogamuchira chikumbiro, inosarudza kuti sumbu ripi rekuenda kwairi uye rinosarudza shard. Ipapo kune angave ekunyora masters, kana, kana iyo cluster ine replica rutsigiro, inotumira kune replica pane inodiwa. Iyo proxy inoita zvese izvi.
Logs
Tinonyora matanda nenzira dzakawanda. Iyo yakanyanya kujeka uye iri nyore ndiyo nyora matanda ku memcache.
ring-buffer: prefix.idx = line
Pane chivakashure chakakosha - zita rerogi, mutsara, uye kune saizi yerogi - nhamba yemitsara. Isu tinotora nhamba isina kurongeka kubva pa0 kusvika kunhamba yemitsara minus 1. Kiyi mumemcache ndiyo prefix inobatanidzwa neiyi nhamba isina kurongeka. Isu tinochengetedza mutsara wegi uye nguva yezvino kune kukosha.
Kana zvakakodzera kuverenga matanda, tinoita Multi Get makiyi ese, akarongwa nenguva, uye nekudaro tora gwaro rekugadzira munguva chaiyo. Iyo chirongwa chinoshandiswa kana iwe uchida kugadzirisa chimwe chinhu mukugadzira munguva chaiyo, pasina kutyora chero chinhu, pasina kumira kana kubvumira traffic kune mamwe machina, asi iyi log haigare kwenguva refu.
Nekuchengetedza kwakavimbika kwematanda tine injini logs-injini. Ichi ndicho chikonzero nei yakagadzirwa uye inoshandiswa zvakanyanya muhuwandu hukuru hwemasumbu. Chisumbu chikuru chandinoziva chezvitoro 600 TB yematanda akazara.
Injini yasakara zvikuru, pane masumbu atova nemakore 6-7. Pane matambudziko nazvo atiri kuyedza kugadzirisa, semuenzaniso, takatanga kushingaira kushandisa ClickHouse kuchengeta matanda.
Kuunganidza matanda mu ClickHouse
Mufananidzo uyu unoratidza mafambiro atinoita mumainjini edu.
Pane kodhi inoenda munharaunda kuburikidza neRPC kune iyo RPC-proxy, uye inonzwisisa kwekuenda kunjini. Kana tichida kunyora matanda muClickHouse, tinoda kushandura zvikamu zviviri muchirongwa ichi:
- tsiva imwe injini neClickHouse;
- tsiva iyo RPC proxy, isingakwanise kuwana ClickHouse, neimwe mhinduro inogona, uye kuburikidza neRPC.
Injini iri nyore - tinoitsiva nesevha kana boka remaseva neClickHouse.
Uye kuenda kuClickHouse, takaita KittenHouse. Kana tikaenda zvakananga kubva kuKittenHouse kuenda kuClickHouse, hazvigone. Kunyangwe pasina zvikumbiro, inowedzera kubva kuHTTP yekubatanidza yehuwandu hukuru hwemakina. Kuti chirongwa chishande, pane sevha ine ClickHouse local reverse proxy inosimudzwa, iyo yakanyorwa nenzira yakadai kuti inogona kumira mavhoriyamu anodiwa ekubatanidza. Inogonawo kuvharidzira data mukati mayo zvakavimbika.
Dzimwe nguva isu hatidi kuita iyo RPC chirongwa mune dzisiri-yakajairwa mhinduro, semuenzaniso, mu nginx. Naizvozvo, KittenHouse inokwanisa kugamuchira matanda kuburikidza neUDP.
Kana mutumi uye anogamuchira matanda akashanda pamushini mumwe chete, saka mukana wekurasikirwa neUDP packet mukati memugadziri wenzvimbo wakaderera. Sekuwirirana pakati pekudiwa kwekushandisa RPC mune yechitatu-bato mhinduro uye kuvimbika, isu tinongoshandisa UDP kutumira. Tichazodzokera kuchirongwa ichi gare gare.
Kuongorora
Tine marudzi maviri ematanda: ayo akaunganidzwa nevatungamiriri pamaseva avo uye ayo akanyorwa nevagadziri kubva kukodhi. Iwo anoenderana nemhando mbiri dzemetrics: system uye chigadzirwa.
System metrics
Inoshanda pamaseva edu ese , iyo inounganidza nhamba uye inotumira kune Graphite Carbon. Naizvozvo, ClickHouse inoshandiswa senzira yekuchengetedza, uye kwete Whisper, semuenzaniso. Kana zvichidikanwa, unogona kuverenga zvakananga kubva kuClickHouse, kana kushandisa grafana yemetrics, magirafu uye mishumo. Sevagadziri, tine mukana wakakwana weNetdata neGrafana.
Product metrics
Kuti zvive nyore, takanyora zvinhu zvakawanda. Semuenzaniso, kune seti yeakajairwa mabasa ayo anobvumidza iwe kunyora Maverengero, UniqueCounts makoshero muhuwandu, iyo inotumirwa kumwe kumberi.
statlogsCountEvent ( βstat_nameβ, $key1, $key2, β¦)
statlogsUniqueCount ( βstat_nameβ, $uid, $key1, $key2, β¦)
statlogsValuetEvent ( βstat_nameβ, $value, $key1, $key2, β¦)
$stats = statlogsStatData($params)
Zvadaro, tinogona kushandisa kurongedza uye kuisa mafirita uye kuita zvese zvatinoda kubva kune manhamba - kuvaka magirafu, gadzirisa Watchdogs.
Tinonyora zvakanyanya akawanda metrics nhamba yezviitiko inobva ku600 mabhiriyoni kusvika ku1 trillion pazuva. Zvisinei, tinoda kuvachengeta makore mashomakunzwisisa mafambiro mumametrics. Kuzvibatanidza idambudziko guru ratisati tagadzirisa. Ndichakuudza kuti zvave kushanda sei kwemakore mashoma apfuura.
Tine mabasa anonyora aya metrics kune memcache yemunokuderedza nhamba yezvinyorwa. Kamwe munguva pfupi yakatangwa munharaunda stats-daemon inounganidza zvinyorwa zvese. Tevere, dhimoni rinobatanidza metrics kuita zvidimbu zviviri zvemaseva matanda-vateresi, iyo inounganidza nhamba kubva kune boka remichina yedu kuitira kuti jira riri shure kwavo rirege kufa.
Kana zvichidikanwa, tinogona kunyora zvakananga kune matanda-vateresi.
Asi kunyora kubva kukodhi zvakananga kune vateresi, kupfuura stas-daemom, imhinduro isina kunaka nekuti inowedzera mutoro pamuunganidzi. Mhinduro yacho yakakodzera chete kana nekuda kwechimwe chikonzero isu hatigone kusimudza memcache stats-daemon pamushini, kana yakadonha uye isu takananga.
Tevere, matanda-vateresi vanosanganisa manhamba mukati meowDB - iyi ndiyo dhatabhesi yedu, iyo inogona zvakare kuchengeta metrics.
Ipapo isu tinokwanisa kuita bhinari "pedyo-SQL" sarudzo kubva kukodhi.
Kuedza
Muzhizha ra2018, takave nemukati hackathon, uye pfungwa yakauya yekuyedza kutsiva iyo dzvuku chikamu chedhayagiramu nechimwe chinhu chaigona kuchengeta metrics muClickHouse. Tine matanda paClickHouse - wadii kuzviedza?
Takanga tine chirongwa chakanyora matanda kuburikidza neKittenHouse.
Takasarudza wedzera imwe "* Imba" kudhayagiramu, iyo ichagamuchira chaiwo metrics mufomati sezvo kodhi yedu inoanyora kuburikidza neUDP. Ipapo iyi *Imba inovashandura kuita zvekuisa, sematanda, iyo KittenHouse inonzwisisa. Anokwanisa kunyatsoendesa matanda aya kuClickHouse, iyo inofanirwa kukwanisa kuaverenga.
Chirongwa chine memcache, stats-daemon uye logs-vateresi dhatabhesi inotsiviwa neiyi.
Chirongwa chine memcache, stats-daemon uye logs-vateresi dhatabhesi inotsiviwa neiyi.
- Pane kutumira kubva kukodhi pano, iyo yakanyorwa munharaunda muStatsHouse.
- StatsHouse inonyora UDP metrics, yakatoshandurwa kuita SQL kuisa, kuKittenHouse mumabhechi.
- KittenHouse inovatumira kuClickHouse.
- Kana tichida kuzviverenga, tobva tazviverenga tichipfuura StatsHouse - zvakananga kubva kuClickHouse tichishandisa SQL yenguva dzose.
Zvichiripo here kuyedza, asi isu tinoda mafambiro azvinoita. Kana isu tikagadzirisa matambudziko nechirongwa, saka pamwe isu tichachinja kwairi zvachose. Ini pachangu, ndinovimba zvakadaro.
The scheme haiponesi simbi. Masevha mashoma anodiwa, emunharaunda stats-daemons uye matanda-vaunganidzi hazvidiwe, asi ClickHouse inoda sevha yakakura kupfuura iyo iri muchirongwa chazvino. Masevha mashoma anodiwa, asi anofanirwa kunge achidhura uye ane simba.
Deploy
Kutanga, ngatitarise iyo PHP deployment. Isu tiri kuvandudza mukati Git: kushandiswa GitLab ΠΈ TeamCity yekutumirwa. Mapazi ebudiriro anosanganiswa kuita master bazi, kubva kuna tenzi kuyedza ivo vanobatanidzwa kuita staging, uye kubva padanho kuenda mukugadzira.
Isati yatumirwa, bazi rekugadzira razvino uye rakapfuura rinotorwa, uye mafaera akasiyana anotariswa mavari - shanduko: yakagadzirwa, yakabviswa, yakashandurwa. Shanduko iyi yakanyorwa mubinlog yeakakosha copyfast injini, iyo inogona kukurumidza kudzokorora shanduko kune yedu yese server zvikepe. Chinoshandiswa pano hakusi kukopa zvakananga, asi kudzokorora makuhwa, kana imwe sevha inotumira shanduko kune vavakidzani vayo vepedyo, avo kune vavakidzani vavo, zvichingodaro. Izvi zvinokutendera kuti uvandudze kodhi mumakumi uye mayunitsi emasekonzi mukati mezvikepe zvese. Kana shanduko ichisvika kune replica yenzvimbo, inoshandisa aya mapechi kune yayo local file system. Rollback inoitwawo maererano nechirongwa chimwe chete.
Isu tinotumirawo kPHP yakawanda uye inewo budiriro yayo pa Git maererano nemufananidzo uri pamusoro. Kubva izvi HTTP server binary, saka isu hatigone kuburitsa diff - iyo yekuburitsa binary inorema mazana eMB. Naizvozvo, pane imwe sarudzo pano - iyo vhezheni yakanyorerwa binlog copyfast. Nekuvaka kwega kwega kunowedzera, uye panguva yekudzoka kunowedzerawo. Version inodzokororwa kumaseva. Makopi emunharaunda anoona kuti vhezheni nyowani yapinda mubinlog, uye nekudzokorora kwemakuhwa ivo vanotora iyo yazvino vhezheni yebhinari ivo pachavo, pasina kuneta yedu tenzi server, asi nekungwarira kuparadzira mutoro kune network. Chii chinotevera kutangazve kwakanaka yeshanduro itsva.
Kune edu einjini, ayo ari zvakare akakosha mabhinari, chirongwa ichi chakafanana:
- git master bazi;
- binary mukati .deb;
- shanduro yakanyorerwa binlog copyfast;
- inodzokororwa kumaseva;
- sevha inoburitsa itsva .dep;
- dpkg -i;
- kuvhurazve kune vhezheni itsva.
Musiyano ndewekuti bhinari yedu yakaiswa mudura .deb, uye kana vachipomba kunze dpkg -i zvinoiswa pahurongwa. Nei kPHP ichiiswa sebhinari, uye injini dzichiiswa sedpkg? Zvakaitika saizvozvo. Inoshanda - usaibate.
Useful links:
- Chirevo chaAnton Kiryushkin ine ruzivo nezve copyfast uye makuhwa.
- Chirevo chaYuri Nasretdinov .
- Mharidzo yangu , asi kubva padanho rekusimudzira, kwete hardware.
Alexey Akulovich mumwe weavo, sechikamu cheKomiti yePurogiramu, vanobatsira musi waChivabvu 17 chichava chiitiko chikuru kune vanogadzira PHP munguva pfupi yapfuura. Tarisa uone PC inotonhorera yatinayo, chii (vaviri vavo vari kugadzira PHP musimboti!) - inoita sechinhu chausingakwanise kupotsa kana ukanyora PHP.
Source: www.habr.com