Mafunso okhudza zomangamanga ndi ntchito za VKontakte

Mbiri ya kulengedwa kwa VKontakte ili pa Wikipedia; inanenedwa ndi Pavel mwiniwake. Zikuoneka kuti aliyense akumudziwa kale. Za amkati, mamangidwe ndi kapangidwe ka malowa pa HighLoad ++ Pavel adandiuzanso mu 2010. Ma seva ambiri atayikira kuyambira pamenepo, kotero tidzasintha zambiri: tidzazigawa, kuchotsa zamkati, kuziyeza, ndikuyang'ana chipangizo cha VK kuchokera pamalingaliro aukadaulo.

Alexei Akulovich (AterCattus) backend mapulogalamu mu gulu VKontakte. Zolemba za lipotili ndi yankho lophatikizana la mafunso omwe amafunsidwa kawirikawiri okhudza momwe nsanja ikugwirira ntchito, zomangamanga, ma seva ndi kuyanjana pakati pawo, koma osati za chitukuko, chomwe ndi za chitsulo. Payokha, za nkhokwe ndi zomwe VK ili nazo m'malo mwake, za kutolera zipika ndikuyang'anira polojekiti yonse yonse. Tsatanetsatane pansi pa odulidwa.

Kwa zaka zoposa zinayi ndakhala ndikugwira ntchito zamtundu uliwonse zokhudzana ndi backend.

• Kukweza, kusunga, kukonza, kugawa media: kanema, kukhamukira pompopompo, zomvera, zithunzi, zolemba.
• Zomangamanga, nsanja, kuwunika kwa mapulogalamu, matabwa, ma cache am'madera, CDN, proprietary RPC protocol.
• Kuphatikiza ndi ntchito zakunja: zidziwitso zokankhira, kuyika ulalo wakunja, RSS feed.
• Kuthandizira anzawo ndi mafunso osiyanasiyana, mayankho omwe amafunikira kulowa mu code yosadziwika.

Panthawi imeneyi, ndinali ndi dzanja m'zigawo zambiri za webusaitiyi. Ndikufuna kugawana nawo izi.

General zomangamanga

Chilichonse, monga mwachizolowezi, chimayamba ndi seva kapena gulu la ma seva omwe amavomereza zopempha.

Seva yakutsogolo

Seva yakutsogolo imavomereza zopempha kudzera pa HTTPS, RTMP ndi WSS.

HTTPS - awa ndi zopempha zamitundu yayikulu komanso yam'manja yamasamba: vk.com ndi m.vk.com, ndi makasitomala ena ovomerezeka komanso osavomerezeka a API yathu: makasitomala am'manja, amithenga. Tili ndi phwando RTMP-magalimoto amawayilesi a Live okhala ndi ma seva akutsogolo osiyana ndi WSS- kulumikizana kwa Streaming API.

Kwa HTTPS ndi WSS pa maseva ndikoyenera nginx. Pamawayilesi a RTMP, posachedwapa tasinthira ku yankho lathu ku, koma zapitirira malire a lipotilo. Pofuna kulekerera zolakwika, ma seva awa amalengeza ma adilesi wamba a IP ndikuchita m'magulu kuti ngati pali vuto pa imodzi mwa ma seva, zopempha za ogwiritsa ntchito sizitayika. Kwa HTTPS ndi WSS, ma seva omwewa amabisa magalimoto kuti atenge gawo la CPU pawokha.

Sitidzalankhulanso za WSS ndi RTMP, koma zopempha za HTTPS, zomwe nthawi zambiri zimagwirizanitsidwa ndi polojekiti ya intaneti.

Bwererani

Kumbuyo kutsogolo nthawi zambiri kumakhala ma seva a backend. Amakonza zopempha zomwe seva yakutsogolo imalandira kuchokera kwa makasitomala.

izi ma seva a kPHP, pomwe daemon ya HTTP ikuyenda, chifukwa HTTPS yasinthidwa kale. kPHP ndi seva yomwe imagwira ntchito zitsanzo za prefork: amayambitsa njira yabwino, gulu la machitidwe a ana, amawapatsira zomvera ndipo amakonza zopempha zawo. Pankhaniyi, njira sizikuyambiranso pakati pa pempho lililonse kuchokera kwa wogwiritsa ntchito, koma ingobwezeretsani dziko lawo ku chikhalidwe choyambirira cha zero - pempho pambuyo pa pempho, m'malo moyambiranso.

Kugawa katundu

Ma backends athu onse si gulu lalikulu la makina omwe amatha kukonza pempho lililonse. Ife iwo kugawidwa m'magulu osiyana: general, mobile, api, video, staging... Vuto pa gulu losiyana la makina silidzakhudza ena onse. Pakakhala zovuta ndi kanema, wogwiritsa ntchito yemwe amamvera nyimbo sangadziwe za mavutowo. Zomwe zimabwerera kumbuyo kuti mutumize pempholi zimaganiziridwa ndi nginx kutsogolo malinga ndi config.

Kutolera ma metric ndi kusanjanso

Kuti timvetsetse kuchuluka kwa magalimoto omwe tiyenera kukhala nawo pagulu lililonse, ife osadalira QPS. Ma backends ndi osiyana, ali ndi zopempha zosiyanasiyana, pempho lirilonse liri ndi zovuta zosiyana zowerengera QPS. Ndi chifukwa chake ife timagwira ntchito ndi lingaliro la katundu pa seva yonse - pa CPU ndi perf.

Tili ndi masauzande a maseva otere. Seva iliyonse yakuthupi imayendetsa gulu la kPHP kuti libwezeretsenso ma cores onse (chifukwa kPHP ndi ulusi umodzi).

Seva Yopezeka

CS kapena Content Server ndi yosungirako. CS ndi seva yomwe imasunga mafayilo ndikusinthanso mafayilo omwe adakwezedwa ndi mitundu yonse ya ntchito zofananira zakumbuyo zomwe tsamba lalikulu lakutsogolo limapereka.

Tili ndi ma seva masauzande ambiri omwe amasunga mafayilo. Ogwiritsa ntchito amakonda kukweza mafayilo, ndipo timakonda kuwasunga ndikugawana nawo. Ena mwa masevawa amatsekedwa ndi ma seva apadera a pu/pp.

pa/pp

Mukatsegula tabu ya netiweki mu VK, mudawona pu/pp.

Kodi pu/pp ndi chiyani? Ngati titseka seva imodzi pambuyo pa inzake, ndiye kuti pali njira ziwiri zotsitsa ndikutsitsa fayilo ku seva yomwe idatsekedwa: mwachindunji через http://cs100500.userapi.com/path kapena kudzera pa seva yapakatikati - http://pu.vk.com/c100500/path.

Pu ndi dzina la mbiri yakale loyika zithunzi, ndipo pp ndi chithunzi choyimira. Ndiko kuti, seva imodzi ndi yokweza zithunzi, ndipo ina ndi yokweza. Tsopano osati zithunzi zokha zomwe zakwezedwa, koma dzina lasungidwa.

Ma seva awa kuthetsa magawo a HTTPSkuchotsa katundu wa purosesa kuchokera kusungirako. Komanso, popeza mafayilo ogwiritsira ntchito amasinthidwa pa masevawa, zambiri zomwe zimasungidwa pamakinawa zimakhala bwino. Mwachitsanzo, makiyi a HTTPS encryption.

Popeza makinawa amatsekedwa ndi makina athu ena, sitingathe kuwapatsa ma IP "oyera" akunja, ndi kupereka "grey". Mwanjira iyi tidasunga padziwe la IP ndikuwonetsetsa kuti titeteza makinawo kuti asapezeke kunja - palibe IP yoti mulowemo.

Kukhazikika pama IP omwe amagawana nawo. Pankhani ya kulekerera zolakwika, chiwembucho chimagwira ntchito chimodzimodzi - ma seva angapo akuthupi ali ndi IP wamba, ndipo zida zomwe zili patsogolo pawo zimasankha komwe angatumize pempho. Ndikambirana za njira zina pambuyo pake.

Mfundo yotsutsana ndi yakuti pamenepa kasitomala amasunga zolumikizira zochepa. Ngati pali IP yofanana pamakina angapo - ndi wolandila yemweyo: pu.vk.com kapena pp.vk.com, osatsegula kasitomala ali ndi malire pa kuchuluka kwa zopempha munthawi imodzi kwa wolandila m'modzi. Koma mu nthawi ya HTTP/2 yodziwika bwino, ndikukhulupirira kuti izi sizilinso zofunika.

Kuipa kodziwikiratu kwa chiwembu ndikuti iyenera kutero pompa magalimoto onse, yomwe imapita kosungirako, kupyolera mu seva ina. Popeza timapopa magalimoto kudzera pamakina, sitingathe kutulutsa magalimoto ambiri, mwachitsanzo, kanema, pogwiritsa ntchito njira yomweyo. Timatumiza mwachindunji - kulumikizana kwachindunji kwapadera pazosungirako zamavidiyo. Timatumiza zinthu zopepuka kudzera pa proxy.

Osati kale kwambiri tapeza mtundu wa proxy wowongoleredwa. Tsopano ndikuuzani momwe amasiyanirana ndi anthu wamba komanso chifukwa chake izi ndizofunikira.

Sun

Mu Seputembala 2017, Oracle, yomwe idagula kale Sun, adachotsa antchito ambiri a Sun. Tikhoza kunena kuti panthawiyi kampaniyo inasiya kukhalapo. Posankha dzina la dongosolo latsopano, olamulira athu adaganiza zopereka msonkho ku kukumbukira kwa kampaniyi ndipo adatcha dongosolo latsopano Sun. Pakati pathu timangomutcha "dzuwa".

pp anali ndi zovuta zingapo. IP imodzi pagulu - posungira yosagwira ntchito. Ma seva angapo amagawana ma adilesi wamba a IP, ndipo palibe njira yoyendetsera seva yomwe pempholo likupita. Chifukwa chake, ngati ogwiritsa ntchito osiyanasiyana abwera ku fayilo yomweyi, ndiye ngati pali chosungira pa seva izi, fayiloyo imathera mu cache ya seva iliyonse. Ichi ndi chiwembu chosagwira ntchito, koma palibe chomwe chingachitike.

Chifukwa chake - sitingathe kugawana zomwe zili, chifukwa sitingathe kusankha seva yeniyeni ya gulu ili - ali ndi IP wamba. Komanso pazifukwa zina zamkati zomwe tili nazo sikunali kotheka kukhazikitsa ma seva oterowo m'zigawo. Iwo anaima ku St.

Ndi dzuŵa, tinasintha dongosolo la kusankha. Tsopano tatero mayendedwe aliwonse: mayendedwe osinthika, anycast, dziwoneni nokha daemon. Seva iliyonse ili ndi IP yakeyake, koma subnet wamba. Chilichonse chimakonzedwa mwanjira yakuti ngati seva imodzi ikulephera, magalimoto amafalikira pa ma seva ena a gulu lomwelo basi. Tsopano ndizotheka kusankha seva inayake, palibe caching yofunikira, ndipo kudalirika sikunakhudzidwe.

Thandizo la kulemera. Tsopano titha kuyika makina amagetsi osiyanasiyana ngati pakufunika, komanso, pakagwa mavuto osakhalitsa, sinthani zolemera za "dzuwa" zogwira ntchito kuti muchepetse katundu wawo, kuti "apumule" ndikuyambanso kugwira ntchito.

Kugawana ndi id yopezeka. Chodabwitsa chokhudza sharding: nthawi zambiri timaphwanya zomwe zili mkati kuti ogwiritsa ntchito osiyanasiyana apite ku fayilo imodzi kudzera "dzuwa" lomwelo kuti akhale ndi cache wamba.

Posachedwa tayambitsa pulogalamu ya "Clover". Iyi ndi mafunso a pa intaneti pawailesi yamoyo, pomwe wolandirayo amafunsa mafunso ndipo ogwiritsa ntchito amayankha munthawi yeniyeni, ndikusankha zosankha. Pulogalamuyi ili ndi macheza omwe ogwiritsa ntchito amatha kucheza. Itha kulumikizidwa nthawi imodzi ndi kuwulutsa anthu oposa 100 zikwi. Onse amalemba mauthenga omwe amatumizidwa kwa onse otenga nawo mbali, ndipo avatar imabwera ndi uthengawo. Ngati anthu 100 zikwizikwi abwera ku avatar imodzi mu "dzuwa" limodzi, ndiye kuti nthawi zina imatha kugudubuza kuseri kwa mtambo.

Pofuna kupirira kuphulika kwa zopempha za fayilo yomweyi, ndi zamtundu wina wazinthu zomwe timatsegula chiwembu chopusa chomwe chimafalitsa mafayilo pa "dzuwa" zonse zomwe zilipo m'deralo.

Dzuwa kuchokera mkati

Reverse proxy pa nginx, cache mwina mu RAM kapena pa ma disks othamanga a Optane/NVMe. Chitsanzo: http://sun4-2.userapi.com/c100500/path - cholumikizira ku "dzuwa", lomwe lili m'chigawo chachinayi, gulu lachiwiri la seva. Imatseka fayilo yanjira, yomwe ili pa seva 100500.

chivundikiro

Timawonjezera node ina ku dongosolo lathu la zomangamanga - malo osungira.

Pansipa pali chithunzi chojambula zosungirako dera, pali pafupifupi 20 a iwo. Awa ndi malo omwe ma cache ndi "dzuwa" amapezeka, omwe amatha kusungitsa magalimoto okha.

Uku ndikusungitsa zamitundu yotsatsira; palibe deta yosungidwa pano - nyimbo, kanema, zithunzi.

Kuti tidziwe dera la wogwiritsa ntchito, ife timasonkhanitsa ma prefixes a netiweki a BGP omwe adalengezedwa m'magawo. Pankhani yobwereranso, tiyeneranso kusanthula deta ya geoip ngati sitinapeze IP ndi ma prefixes. Timazindikira dera ndi IP ya wogwiritsa ntchito. Mu code, tikhoza kuyang'ana chigawo chimodzi kapena zingapo za wogwiritsa ntchito - mfundo zomwe ali pafupi kwambiri ndi malo.

Kodi ntchito?

Timawerengera kutchuka kwa mafayilo ndi dera. Pali malo angapo am'deralo komwe wogwiritsa ntchito ali, ndi chizindikiritso cha fayilo - timatenga awiriwa ndikuwonjezera mlingo pakutsitsa kulikonse.

Nthawi yomweyo, ziwanda - mautumiki m'zigawo - nthawi ndi nthawi zimabwera ku API ndikuti: "Ndine chonchi, ndipatseni mndandanda wamafayilo otchuka kwambiri m'chigawo changa omwe sali pa ine. ” API imapereka mulu wa mafayilo osanjidwa ndi mavoti, daemon imawatsitsa, kuwatengera kumadera ndikupereka mafayilo kuchokera pamenepo. Uwu ndiye kusiyana kwakukulu pakati pa pu/pp ndi Sun kuchokera ku cache: amapereka fayilo kudzera mwa iwo okha nthawi yomweyo, ngakhale fayiloyo ilibe posungira, ndipo posungirayo amayamba kutsitsa fayilo yokha, kenako ndikuyamba kuyibweza.

Munkhaniyi tipeza zomwe zili pafupi ndi ogwiritsa ntchito ndi kufalitsa katundu wa netiweki. Mwachitsanzo, kokha kuchokera ku cache ya Moscow timagawira zoposa 1 Tbit / s pa nthawi yachangu.

Koma pali mavuto - ma cache seva si rabara. Pazinthu zodziwika bwino, nthawi zina palibe netiweki yokwanira pa seva yosiyana. Ma seva athu a cache ndi 40-50 Gbit/s, koma pali zomwe zimatseka njira yotere. Tikupita ku kukhazikitsa kusungirako kwa mafayilo odziwika ambiri m'derali. Ndikukhulupirira kuti tidzakwaniritsa izi kumapeto kwa chaka.

Tinayang'ana pa zomangamanga.

• Ma seva apatsogolo omwe amavomereza zopempha.
• Imabwezeretsanso zopempha za ndondomekoyi.
• Zosungira zomwe zimatsekedwa ndi mitundu iwiri ya ma proxies.
• Zosungirako zachigawo.

Chikusowapo ndi chiyani pajambulachi? Inde, nkhokwe zomwe timasungiramo deta.

Databases kapena injini

Timawatcha osati ma database, koma injini - Injini, chifukwa tilibe nkhokwe m'malingaliro ovomerezeka.

Ichi ndi muyezo wofunikira. Izi zidachitika chifukwa mu 2008-2009, pomwe VK idakula kwambiri, ntchitoyi idagwira ntchito kwathunthu pa MySQL ndi Memcache ndipo panali mavuto. MySQL inkakonda kuwononga ndi kuipitsa mafayilo, pambuyo pake sichikanachira, ndipo Memcache pang'onopang'ono inasokoneza ntchito ndipo inayenera kuyambiranso.

Zikuoneka kuti pulojekiti yomwe ikuchulukirachulukirayi inali ndi kusungirako kosalekeza, komwe kumawononga deta, ndi cache, yomwe imachepetsa. M'mikhalidwe yotereyi, zimakhala zovuta kupanga polojekiti yomwe ikukula. Anaganiza zoyesa kulembanso zinthu zofunika kwambiri zomwe polojekitiyi inayang'ana pa njinga zathu.

Yankho lake linali lopambana. Panali mwayi wochita izi, komanso kufunikira kopitilira muyeso, chifukwa njira zina zokulitsira zinalibe panthawiyo. Panalibe gulu lazinthu zambiri, NoSQL kunalibe, panali MySQL, Memcache, PostrgreSQL yokha - ndipo ndi momwemo.

Universal ntchito. Chitukukochi chinatsogozedwa ndi gulu lathu la omanga C ndipo zonse zidachitika mosasinthasintha. Mosasamala za injini, onse anali ndi mawonekedwe ofanana a fayilo olembedwa ku diski, magawo oyambitsira omwewo, ma siginecha okonzedwa mwanjira yomweyo, ndipo amachita chimodzimodzi ngati pakakhala zovuta komanso zovuta. Ndi kukula kwa injini, ndi bwino kuti olamulira agwiritse ntchito dongosolo - palibe zoo zomwe ziyenera kusamalidwa, ndipo amayenera kuphunziranso momwe angagwiritsire ntchito deta yatsopano ya chipani chachitatu, zomwe zinapangitsa kuti ziwonjezeke mwamsanga komanso mosavuta. nambala yawo.

Mitundu ya injini

Gululo linalemba injini zingapo. Nawa ena mwa iwo: abwenzi, malingaliro, chithunzi, ipdb, zilembo, mindandanda, zipika, memcached, meowdb, nkhani, nostradamus, chithunzi, playlist, pmemcached, sandbox, kusaka, kusungirako, zokonda, ntchito, ...

Pa ntchito iliyonse yomwe imafuna dongosolo lachidziwitso lachidziwitso kapena zopempha zachilendo, gulu la C limalemba injini yatsopano. Kulekeranji.

Tili ndi injini yosiyana memcached, yomwe ili yofanana ndi yanthawi zonse, koma yokhala ndi zinthu zambiri zabwino, ndipo sizimachedwa. Osati ClickHouse, koma imagwiranso ntchito. Zikupezeka padera pmemcached Ndi kulimbikira memcached, yomwe imathanso kusunga deta pa diski, komanso, kuposa momwe ikukwanira mu RAM, kuti musataye deta pamene mukuyambiranso. Pali injini zosiyanasiyana za ntchito iliyonse: mizere, mindandanda, seti - chilichonse chomwe polojekiti yathu ikufuna.

Magulu

Kuchokera pamawonekedwe a code, palibe chifukwa choganizira za injini kapena nkhokwe ngati njira, mabungwe, kapena zochitika. Code imagwira ntchito makamaka ndi magulu, ndi magulu a injini - mtundu umodzi pa tsango. Tinene kuti pali gulu la memcached - ndi gulu chabe la makina.

Khodiyo sifunikira kudziwa komwe kuli, kukula, kapena kuchuluka kwa ma seva nkomwe. Amapita kugulu pogwiritsa ntchito chizindikiritso china.

Kuti izi zitheke, muyenera kuwonjezera chinthu chimodzi chomwe chili pakati pa code ndi injini - wothandizira.

Wothandizira RPC

Woyimira kugwirizana basi, pomwe pafupifupi tsamba lonse limayendera. Pa nthawi yomweyo ife palibe kupezeka kwautumiki - m'malo mwake, pali makonzedwe a proxy iyi, yomwe imadziwa malo amagulu onse ndi shards zonse za gululi. Izi ndi zomwe admins amachita.

Okonza mapulogalamu samasamala konse kuti ndi zingati, kuti ndi chiyani - amangopita kumagulu. Izi zimatipatsa mwayi wambiri. Akalandira pempho, woyimira mlandu amawongolera pempholo, akudziwa komwe - amasankha izi.

Pachifukwa ichi, proxy ndi malo otetezera kulephera kwa utumiki. Ngati injini ina ikucheperachepera kapena kugwa, ndiye kuti wothandizira amamvetsetsa izi ndikuyankha molingana ndi kasitomala. Izi zimakulolani kuti muchotse nthawi yopuma - kachidindo sikudikirira kuti injini iyankhe, koma imamvetsetsa kuti sikugwira ntchito ndipo imayenera kuchita mwanjira ina. Khodiyo iyenera kukonzedwa chifukwa nkhokwe sizigwira ntchito nthawi zonse.

Kukhazikitsa kwachindunji

Nthawi zina timafunabe kukhala ndi njira ina yopanda muyezo ngati injini. Panthawi imodzimodziyo, adaganiza kuti asagwiritse ntchito rpc-proxy yathu yokonzekera, yomwe inapangidwira injini zathu, koma kupanga pulojekiti yosiyana ya ntchitoyi.

Kwa MySQL, yomwe tikadali nayo apa ndi apo, timagwiritsa ntchito db-proxy, ndi ClickHouse - Nyumba ya Kitten.

Zimagwira ntchito motere. Pali seva ina, imayendetsa kPHP, Go, Python - kawirikawiri, code iliyonse yomwe ingagwiritse ntchito RPC protocol. Khodiyo imayendera kwanuko pa proxy ya RPC - seva iliyonse pomwe pali code imayendetsa projekiti yake yakwanuko. Akafunsidwa, woyimira mlandu amamvetsetsa koyenera kupita.

Ngati injini imodzi ikufuna kupita ku ina, ngakhale itakhala yoyandikana nayo, imadutsa pa proxy, chifukwa woyandikana naye akhoza kukhala pamalo ena a data. Injini sayenera kudalira kudziwa komwe kuli china chilichonse kupatula iyo yokha - iyi ndiye yankho lathu lokhazikika. Koma zowona, pali zosiyana :)

Chitsanzo cha TL-scheme malinga ndi momwe ma injini onse amagwirira ntchito.

memcache.not_found                                = memcache.Value;
memcache.strvalue	value:string flags:int = memcache.Value;
memcache.addOrIncr key:string flags:int delay:int value:long = memcache.Value;

tasks.task
    fields_mask:#
    flags:int
    tag:%(Vector int)
    data:string
    id:fields_mask.0?long
    retries:fields_mask.1?int
    scheduled_time:fields_mask.2?int
    deadline:fields_mask.3?int
    = tasks.Task;
 
tasks.addTask type_name:string queue_id:%(Vector int) task:%tasks.Task = Long;

Iyi ndi protocol ya binary, analogue yapafupi kwambiri yomwe ili protobuf. Dongosololi limafotokozera magawo omwe mwasankha, mitundu yovuta - zowonjezera za ma scalar omangidwa, ndi mafunso. Chilichonse chimagwira ntchito molingana ndi protocol iyi.

RPC pa TL pa TCP/UDP… UDP?

Tili ndi protocol ya RPC yochita zopempha za injini zomwe zikuyenda pamwamba pa chiwembu cha TL. Zonsezi zimagwira ntchito pa intaneti ya TCP/UDP. TCP ndiyomveka, koma chifukwa chiyani timafunikira UDP nthawi zambiri?

UDP imathandizira pewani vuto la kuchuluka kwa kulumikizana pakati pa ma seva. Ngati seva iliyonse ili ndi proxy ya RPC ndipo, kawirikawiri, imatha kupita ku injini iliyonse, ndiye pali makumi masauzande a maulumikizidwe a TCP pa seva. Katundu alipo, koma ndi wopanda pake. Pankhani ya UDP vutoli kulibe.

Palibe kugwirana chanza kwa TCP kofunikira. Ili ndi vuto lodziwika bwino: injini yatsopano kapena seva yatsopano ikayambitsidwa, maulumikizidwe ambiri a TCP amakhazikitsidwa nthawi imodzi. Pazopempha zazing'ono zopepuka, mwachitsanzo, kubweza kwa UDP, kulumikizana konse pakati pa code ndi injini ndi mapaketi awiri a UDP: imodzi imawulukira mbali ina, yachiwiri inzake. Ulendo umodzi wozungulira - ndipo code idalandira yankho kuchokera ku injini popanda kugwirana chanza.

Inde, zonse zimagwira ntchito ndi gawo laling'ono kwambiri la kutayika kwa paketi. Protocol ili ndi chithandizo cha kubwezeredwa ndi kutha kwa nthawi, koma ngati titaya zambiri, tidzapeza pafupifupi TCP, zomwe sizopindulitsa. Sitiyendetsa UDP kudutsa nyanja.

Tili ndi ma seva oterowo masauzande ambiri, ndipo chiwembuchi ndi chofanana: paketi ya injini imayikidwa pa seva iliyonse yakuthupi. Nthawi zambiri amakhala ndi ulusi umodzi kuti azithamanga mwachangu popanda kutsekereza, ndipo amagawidwa ngati njira zamtundu umodzi. Panthawi imodzimodziyo, tilibe chodalirika kuposa injinizi, ndipo chidwi chachikulu chimaperekedwa kusungirako deta kosalekeza.

Kusunga deta mosalekeza

Ma injini amalemba ma bilogs. Binlog ndi fayilo kumapeto kwake komwe chochitika chosintha boma kapena deta chikuwonjezeredwa. M'mayankho osiyanasiyana amatchedwa mosiyana: chipika cha binary, wal, AOF, koma mfundo yake ndi yofanana.

Pofuna kuteteza injini kuti isawerengenso binlog yonse kwa zaka zambiri pamene ikuyambiranso, injini zimalemba zithunzi - panopa. Ngati ndi kotheka, amawerenga kuchokera pamenepo, ndiyeno amalize kuwerenga kuchokera mu binlog. Ma binlogs onse amalembedwa mumtundu womwewo wa binary - malinga ndi dongosolo la TL, kuti ma admins azitha kuwatsogolera mofanana pogwiritsa ntchito zida zawo. Sipafunikanso zithunzithunzi zotere. Pali mutu wamba womwe ukuwonetsa yemwe chithunzi chake ndi int, matsenga a injini, ndi thupi liti lomwe silili lofunikira kwa aliyense. Ili ndi vuto ndi injini yomwe idalemba chithunzithunzi.

Ndidzalongosola mwamsanga mfundo ya ntchito. Pali seva yomwe injini imayendera. Amatsegula binlog yatsopano yopanda kanthu kuti alembe ndikulemba chochitika kuti asinthe.

Panthawi ina, amasankha kudzijambula yekha, kapena amalandira chizindikiro. Seva imapanga fayilo yatsopano, imalemba momwemo, ndikuwonjezera kukula kwa binlog - kuchotsera - mpaka kumapeto kwa fayilo, ndikupitiriza kulemba. Bilogu yatsopano sinapangidwe.

Panthawi ina, injini ikayambiranso, padzakhala binlog ndi chithunzithunzi pa disk. Injini imawerenga chithunzi chonse ndikukweza malo ake panthawi inayake.

Imawerenga malo omwe anali panthawi yomwe chithunzichi chimapangidwa komanso kukula kwa binlog.

Amawerenga mapeto a binlog kuti adziwe momwe zilili panopa ndikupitiriza kulemba zochitika zina. Ichi ndi chiwembu chosavuta; injini zathu zonse zimagwira ntchito molingana ndi izo.

Kubwereza kwa data

Zotsatira zake, kubwereza kwa data mu athu zotengera mawu - timalemba mu binlog osati tsamba lililonse losintha, koma ndilo kusintha zopempha. Zofanana kwambiri ndi zomwe zimabwera pa intaneti, zosinthidwa pang'ono.

Chiwembu chomwecho chimagwiritsidwa ntchito osati kungobwerezabwereza, komanso kupanga zosunga zobwezeretsera. Tili ndi injini - mbuye wolembera yemwe amalembera binlog. Kumalo ena aliwonse omwe ma admins amayiyika, binlog iyi imakopedwa, ndipo ndizomwezo - tili ndi zosunga zobwezeretsera.

Ngati pakufunika kuwerenga chofaniziraKuti muchepetse kuchuluka kwa CPU kuwerenga, injini yowerengera imangoyambika, yomwe imawerenga kumapeto kwa binlog ndikuchita malamulo awa kwanuko.

Chotsalira apa ndi chaching'ono kwambiri, ndipo ndizotheka kudziwa kuti chofananacho chimatsalira bwanji kumbuyo kwa mbuye.

Kugawana deta mu proxy RPC

Kodi sharding imagwira ntchito bwanji? Kodi projekiti imamvetsetsa bwanji cluster shard yotumiza? Khodiyo sinena kuti: "Tumizani ma shards 15!" - ayi, izi zimachitidwa ndi wothandizira.

Njira yosavuta ndiyo yoyamba - nambala yoyamba mu pempho.

get(photo100_500) => 100 % N.

Ichi ndi chitsanzo cha protocol yosavuta ya memcached, koma, zowona, mafunso amatha kukhala ovuta komanso okhazikika. Chitsanzo chimatenga nambala yoyamba mu funso ndi yotsalayo ikagawidwa ndi kukula kwa masango.

Izi ndizothandiza ngati tikufuna kukhala ndi data yamtundu umodzi. Tinene kuti 100 ndi ID ya ogwiritsa ntchito kapena gulu, ndipo tikufuna kuti zonse za gulu limodzi zikhale pa shard imodzi pamafunso ovuta.

Ngati sitisamala momwe zopempha zimafalidwira pagulu, pali njira ina - kuthirira tsitsi lonse.

hash(photo100_500) => 3539886280 % N

Timapezanso hashi, chotsalira cha magawo ndi nambala ya shard.

Zosankha zonsezi zimagwira ntchito ngati takonzekera kuti pamene tikuwonjezera kukula kwa masango, tidzagawaniza kapena kuwonjezera nthawi zambiri. Mwachitsanzo, tinali ndi shards 16, tilibe zokwanira, tikufuna zambiri - titha kupeza 32 popanda nthawi yopuma. Ngati tikufuna kuwonjezera osati kuchulukitsa, padzakhala nthawi yopuma, chifukwa sitingathe kugawanitsa molondola zonse popanda kutayika. Zosankha izi ndizothandiza, koma osati nthawi zonse.

Ngati tikufuna kuwonjezera kapena kuchotsa ma seva angapo, timagwiritsa ntchito Hashing yokhazikika pa mphete a la Ketama. Koma panthawi imodzimodziyo, timataya kwathunthu malo a deta; tiyenera kuphatikiza pempho ku gululo kuti chidutswa chilichonse chibweretse yankho lake laling'ono, ndikuphatikiza mayankho ku proxy.

Pali zopempha zapadera kwambiri. Zikuwoneka ngati izi: Woyimira wa RPC alandila pempho, amasankha gulu loti apite ndikusankha shard. Ndiye pali olemba masters, kapena, ngati gululo liri ndi chithandizo chofananira, limatumiza ku choyimira pakufunika. Zonsezi zimachitidwa ndi proxy.

Mitengo

Timalemba zipika m'njira zingapo. Chowonekera kwambiri ndi chophweka ndi lembani zipika ku memcache.

ring-buffer: prefix.idx = line

Pali choyambirira chachikulu - dzina la chipika, mzere, ndipo pali kukula kwa chipika ichi - chiwerengero cha mizere. Timatenga nambala yachisawawa kuchokera ku 0 kupita ku nambala ya mizere kuchotsera 1. Chinsinsi cha memcache ndi chilembo cholumikizidwa ndi nambala iyi mwachisawawa. Timasunga mzere wa chipika ndi nthawi yamakono kumtengo wapatali.

Pakafunika kuwerenga zipika, timachita Multi Get makiyi onse, osanjidwa ndi nthawi, motero kupeza chipika kupanga mu nthawi yeniyeni. Chiwembucho chimagwiritsidwa ntchito pamene mukufunikira kusokoneza chinachake pakupanga mu nthawi yeniyeni, osathyola kalikonse, popanda kuyimitsa kapena kulola magalimoto ku makina ena, koma chipikachi sichikhalitsa.

Kusungirako zodalirika kwa zipika tili ndi injini mitengo-injini. Ichi ndichifukwa chake adalengedwa ndipo amagwiritsidwa ntchito kwambiri m'magulu ambiri. Gulu lalikulu kwambiri lomwe ndikudziwa la 600 TB ya zipika zopakidwa.

Injini ndi yakale kwambiri, pali masango omwe ali ndi zaka 6-7. Pali mavuto omwe tikuyesera kuthetsa, mwachitsanzo, tinayamba kugwiritsa ntchito mwakhama ClickHouse kusunga zipika.

Kusonkhanitsa zipika mu ClickHouse

Chithunzichi chikuwonetsa momwe timayendera mu injini zathu.

Pali code yomwe imapita kwanuko kudzera pa RPC kupita ku RPC-proxy, ndipo imamvetsetsa komwe mungapite ku injini. Ngati tikufuna kulemba zipika mu ClickHouse, tiyenera kusintha magawo awiri mu dongosolo ili:

• sinthani injini ina ndi ClickHouse;
• m'malo mwa projekiti ya RPC, yomwe singathe kupeza ClickHouse, ndi yankho lomwe lingathe, komanso kudzera pa RPC.

Injiniyo ndi yosavuta - timayisintha ndi seva kapena gulu la ma seva ndi ClickHouse.

Ndipo kupita ku ClickHouse, tidatero Nyumba ya Kitten. Ngati tipita mwachindunji kuchokera ku KittenHouse kupita ku ClickHouse, sizingathe. Ngakhale popanda zopempha, zimawonjezera kuchokera kumalumikizidwe a HTTP pamakina ambiri. Kuti chiwembucho chigwire ntchito, pa seva yokhala ndi ClickHouse thirakiti yakumbuyo yakwezedwa, zomwe zimalembedwa m'njira yoti zitha kupirira milingo yofunikira yolumikizirana. Ikhozanso kusunga deta mkati mwake modalirika.

Nthawi zina sitifuna kugwiritsa ntchito chiwembu cha RPC munjira zosavomerezeka, mwachitsanzo, mu nginx. Chifukwa chake, KittenHouse imatha kulandira zipika kudzera pa UDP.

Ngati wotumiza ndi wolandira zipikazo akugwira ntchito pamakina omwewo, ndiye kuti mwayi wotaya paketi ya UDP mkati mwa wolandila wakumaloko ndi wotsika kwambiri. Monga kusagwirizana pakati pa kufunikira kokhazikitsa RPC mu njira yachitatu ndi kudalirika, timangogwiritsa ntchito kutumiza kwa UDP. Tibwereranso ku chiwembuchi pambuyo pake.

Kuwunikira

Tili ndi mitundu iwiri ya zipika: zomwe zimasonkhanitsidwa ndi oyang'anira pa maseva awo ndi zomwe zidalembedwa ndi opanga kuchokera pama code. Zimagwirizana ndi mitundu iwiri ya ma metrics: dongosolo ndi mankhwala.

Ma metric a system

Zimagwira ntchito pa ma seva athu onse netdata, yomwe imasonkhanitsa ziwerengero ndikuzitumiza ku Mpweya wa Graphite. Chifukwa chake, ClickHouse imagwiritsidwa ntchito ngati njira yosungira, osati Whisper, mwachitsanzo. Ngati ndi kotheka, mutha kuwerenga mwachindunji kuchokera ku ClickHouse, kapena gwiritsani ntchito grafana kwa ma metric, ma graph ndi malipoti. Monga otukula, tili ndi mwayi wokwanira ku Netdata ndi Grafana.

Ma metrics azinthu

Kuti tikhale omasuka, talemba zinthu zambiri. Mwachitsanzo, pali ntchito wamba zomwe zimakulolani kuti mulembe ma Counts, UniqueCounts mu ziwerengero, zomwe zimatumizidwa kwinakwake.

statlogsCountEvent   ( ‘stat_name’,            $key1, $key2, …)
statlogsUniqueCount ( ‘stat_name’, $uid,    $key1, $key2, …)
statlogsValuetEvent  ( ‘stat_name’, $value, $key1, $key2, …)

$stats = statlogsStatData($params)

Pambuyo pake, titha kugwiritsa ntchito zosefera zosankhidwa ndi magulu ndikuchita chilichonse chomwe tikufuna kuchokera ku ziwerengero - pangani ma graph, sinthani ma Watchdogs.

Timalemba kwambiri metrics ambiri chiwerengero cha zochitika ndi kuchokera 600 biliyoni kufika 1 thililiyoni patsiku. Komabe, tikufuna kuwasunga osachepera zaka zingapokuti mumvetsetse zomwe zikuchitika mu metrics. Kuziyika zonse pamodzi ndi vuto lalikulu lomwe sitinathe kulithetsa. Ndikuuzani momwe zakhalira zaka zingapo zapitazi.

Tili ndi ntchito zomwe zimalemba ma metrics awa ku memcache yakomwekokuchepetsa chiwerengero cha zolembera. Kamodzi mu nthawi yochepa kwanuko anapezerapo stats-daemon amasonkhanitsa zolemba zonse. Kenako, chiwandacho chimaphatikiza ma metric kukhala magawo awiri a maseva mitengo-otolera, zomwe zimaphatikiza ziwerengero kuchokera pagulu la makina athu kuti wosanjikiza kumbuyo kwawo asafe.

Ngati kuli kofunikira, tikhoza kulemba mwachindunji kwa osonkhanitsa zipika.

Koma kulemba kuchokera ku code mwachindunji kupita kwa osonkhanitsa, kudutsa stas-daemom, ndi njira yothetsera vuto chifukwa imawonjezera katundu pa osonkhanitsa. Yankho ndiloyenera pokhapokha ngati pazifukwa zina sitingathe kukweza memcache stats-daemon pamakina, kapena idagwa ndipo tidapita molunjika.

Kenako, osonkhanitsa zipika amaphatikiza ziwerengero mwaDB - iyi ndiye nkhokwe yathu, yomwe imathanso kusunga ma metric.

Kenako titha kupanga zosankha zamabina "pafupi ndi SQL" kuchokera pamakhodi.

Kuyesera

M'chilimwe cha 2018, tinali ndi hackathon yamkati, ndipo lingaliro lidabwera kuti tiyese kusintha gawo lofiira la chithunzicho ndi chinthu chomwe chingasunge ma metrics ku ClickHouse. Tili ndi zipika pa ClickHouse - bwanji osayesa?

Tidali ndi chiwembu chomwe chidalemba zipika kudzera ku KittenHouse.

Tinaganiza onjezani "*Nyumba" ina pachithunzichi, yomwe idzalandira ndendende ma metrics mumpangidwe momwe code yathu imawalembera kudzera pa UDP. Kenako *Nyumbayi imawatembenuza kukhala zolowetsa, monga matabwa, zomwe KittenHouse imamvetsetsa. Amatha kuperekera bwino zipikazi ku ClickHouse, zomwe ziyenera kuziwerenga.

Chiwembu chokhala ndi memcache, stats-daemon ndi logs-collectors database chimasinthidwa ndi ichi.

Chiwembu chokhala ndi memcache, stats-daemon ndi logs-collectors database chimasinthidwa ndi ichi.

• Pali kutumiza kuchokera ku code pano, yomwe yalembedwa kwanuko ku StatsHouse.
• StatsHouse imalemba ma metric a UDP, osinthidwa kale kukhala ma SQL oyika, kupita ku KittenHouse m'magulu.
• KittenHouse imawatumiza ku ClickHouse.
• Ngati tikufuna kuwawerenga, ndiye kuti timawawerenga kudutsa StatsHouse - mwachindunji kuchokera ku ClickHouse pogwiritsa ntchito SQL yokhazikika.

Kodi akadali kuyesa, koma timakonda momwe zimakhalira. Ngati tikonza mavuto ndi chiwembu, ndiye kuti mwina tidzasinthira kwathunthu. Inemwini, ndikuyembekeza choncho.

Chiwembu sichipulumutsa chitsulo. Ma seva ochepa amafunikira, ma stats-daemons am'deralo ndi osonkhanitsa ma logs sakufunika, koma ClickHouse imafuna seva yokulirapo kuposa yomwe ili mu dongosolo lapano. Ma seva ochepa amafunikira, koma ayenera kukhala okwera mtengo komanso amphamvu kwambiri.

Ikani

Choyamba, tiyeni tiwone kutumizidwa kwa PHP. Tikukula mu Pitani: ntchito GitLab и TeamCity za kutumiza. Nthambi zachitukuko zimaphatikizidwa munthambi yayikulu, kuchokera kwa mbuye kuti ayesedwe amaphatikizidwa kukhala masitepe, komanso kuchokera pakupanga mpaka kupanga.

Asanatumizidwe, nthambi yopangira zamakono ndi yapitayi imatengedwa, ndipo mafayilo amasiyana amaganiziridwa mwa iwo - kusintha: kupangidwa, kuchotsedwa, kusinthidwa. Kusintha kumeneku kumalembedwa mu binlog ya injini yapadera ya copyfast, yomwe imatha kubwereza mwamsanga kusintha kwa seva yathu yonse. Zomwe zikugwiritsidwa ntchito pano si kukopera mwachindunji, koma kubwereza miseche, pamene seva imodzi itumiza zosintha kwa oyandikana nawo apafupi, awo kwa anansi awo, ndi zina zotero. Izi zimakupatsani mwayi wosintha ma code mu makumi ndi mayunitsi amasekondi kudutsa zombo zonse. Kusintha kukafika pachifaniziro cha komweko, kumagwiritsa ntchito zigamba zake local file system. Rollback ikuchitikanso molingana ndi chiwembu chomwecho.

Timatumizanso kPHP kwambiri ndipo ilinso ndi chitukuko chake Pitani malinga ndi chithunzi pamwambapa. Kuyambira izi HTTP seva binary, ndiye sitingathe kutulutsa diff - kutulutsidwa kwa binary kumalemera mazana a MB. Choncho, pali njira ina apa - Baibulo lalembedwa binlog copyfast. Ndi kumanga kulikonse kumawonjezeka, ndipo panthawi yobwezeretsa kumawonjezekanso. Baibulo kusinthidwa ku ma seva. Otsatsa am'deralo amawona kuti mtundu watsopano walowa mu binlog, ndipo mwa kubwereza miseche komweko amadzitengera okha mtundu waposachedwa wa binary, osatopetsa seva yathu yayikulu, koma kufalitsa katunduyo mosamala pamanetiweki. Chotsatira kuyambiranso kwachisomo za mtundu watsopano.

Kwa mainjini athu, omwenso ali ma binaries, chiwembucho ndi chofanana kwambiri:

• git master nthambi;
• binary mu .deb;
• Baibulolo linalembedwa ku binlog copyfast;
• kusinthidwa kwa ma seva;
• seva imatulutsa .dep yatsopano;
• dpkg ndi;
• kuyambiranso mwachisomo ku mtundu watsopano.

Kusiyana kwake ndikuti binary yathu imayikidwa m'malo osungira .deb, ndipo popopa iwo dpkg ndi amaikidwa pa dongosolo. Chifukwa chiyani kPHP imayikidwa ngati binary, ndipo injini zimayikidwa ngati dpkg? Izo zinachitika mwanjira imeneyo. Zimagwira ntchito - musakhudze.

Maulalo othandiza:

• Lipoti la Anton Kiryushkin "Vkontakte system administrator. Bwanji?" ndi zambiri za copyfast ndi miseche.
• Lipoti la Yuri Nasretdinov "Momwe VK imayika data mu CLickHouse kuchokera pa ma seva masauzande".
• Lipoti langa "Zomangamanga za polojekiti yomwe ikukula pogwiritsa ntchito chitsanzo cha VKontakte", koma kuchokera ku chitukuko, osati hardware.

Alexey Akulovich ndi mmodzi mwa anthu amene, monga gawo la Komiti Programme, amathandiza PHP Russia pa Meyi 17 idzakhala chochitika chachikulu kwambiri kwa opanga ma PHP posachedwa. Tawonani PC yabwino yomwe tili nayo, chiyani okamba (awiri a iwo akupanga PHP pachimake!) - zikuwoneka ngati chinthu chomwe simungachiphonye ngati mulemba PHP.

Source: www.habr.com