{"id":55009,"date":"2020-01-10T00:00:00","date_gmt":"2020-01-09T21:00:00","guid":{"rendered":"https:\/\/prohoster.info\/blog\/blog_prohoster\/arhitektura-hraneniya-i-otdachi-fotografij-v-badoo"},"modified":"2020-02-18T14:03:04","modified_gmt":"2020-02-18T11:03:04","slug":"arhitektura-hraneniya-i-otdachi-fotografij-v-badoo","status":"publish","type":"post","link":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/arhitektura-hraneniya-i-otdachi-fotografij-v-badoo","title":{"rendered":"Architektura ukl\u00e1d\u00e1n\u00ed a distribuce fotografi\u00ed v Badoo","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p><img decoding=\"async\" alt=\"Architektura ukl\u00e1d\u00e1n\u00ed a distribuce fotografi\u00ed v Badoo\" src=\"\/wp-content\/uploads\/2020\/01\/04d68ff091d9d271bfd352a6639e2f20.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<\/p>\n<h2>Artem Denisov ( <noindex><a rel=\"nofollow\" href=\"https:\/\/habrahabr.ru\/users\/bo0rsh201\/\" class=\"user_link\">bo0rsh201<\/a><\/noindex>, <noindex><a rel=\"nofollow\" href=\"https:\/\/habrahabr.ru\/company\/badoo\/\">Badoo<\/a><\/noindex>)<\/h2>\n<p>\nBadoo je nejv\u011bt\u0161\u00ed seznamovac\u00ed str\u00e1nka na sv\u011bt\u011b. V sou\u010dasnosti m\u00e1me zaregistrov\u00e1no p\u0159ibli\u017en\u011b 330 milion\u016f u\u017eivatel\u016f po cel\u00e9m sv\u011bt\u011b. Ale co je je\u0161t\u011b d\u016fle\u017eit\u011bj\u0161\u00ed v kontextu na\u0161\u00ed dne\u0161n\u00ed debaty, je fakt, \u017ee uchov\u00e1v\u00e1me p\u0159ibli\u017en\u011b 3 petabajty u\u017eivatelsk\u00fdch fotografi\u00ed. Ka\u017ed\u00fd den na\u0161i u\u017eivatel\u00e9 nahr\u00e1vaj\u00ed p\u0159ibli\u017en\u011b 3,5 milionu nov\u00fdch fotografi\u00ed a z\u00e1t\u011b\u017e p\u0159i \u010dten\u00ed \u010din\u00ed p\u0159ibli\u017en\u011b <b>80 tis\u00edc dotaz\u016f za sekundu<\/b>. To je docela hodn\u011b pro n\u00e1\u0161 backend a ob\u010das se s t\u00edm vyskytuj\u00ed pot\u00ed\u017ee.<\/p>\n<p><img decoding=\"async\" alt=\"Architektura ukl\u00e1d\u00e1n\u00ed a distribuce fotografi\u00ed v Badoo\" src=\"\/wp-content\/uploads\/2020\/01\/4e4690f9b0d95bd76fbbc1cd6805c26c.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nPov\u00edm v\u00e1m o designu tohoto syst\u00e9mu, kter\u00fd uchov\u00e1v\u00e1 a poskytuje fotografie v celku, a uk\u00e1\u017eu v\u00e1m pohled na n\u011bj z hlediska v\u00fdvoj\u00e1\u0159e. O tom, jak se vyv\u00edjel, bude kr\u00e1tk\u00e1 retrospektiva, kde ozna\u010d\u00edm hlavn\u00ed miln\u00edky, ale podrobn\u011bji budu mluvit pouze o t\u011bch \u0159e\u0161en\u00edch, kter\u00e1 aktu\u00e1ln\u011b pou\u017e\u00edv\u00e1me.<br \/>\n<noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><br \/>\nA te\u010f za\u010dn\u011bme.<\/p>\n<p><center><div class=\"youtube-placeholder\" data-id=\"SU9ETg39FEg\" onclick=\"loadVideo(this)\">\r\n        <img decoding=\"async\" src=\"https:\/\/img.youtube.com\/vi\/SU9ETg39FEg\/hqdefault.jpg\" alt=\"P\u0159ehr\u00e1t video\" loading=\"lazy\" width=\"480\" height=\"360\" style=\"width:100%;height:auto;\">\r\n        <div class=\"play-button\"><\/div>\r\n    <\/div><\/center><br \/>\nJak u\u017e jsem \u0159\u00edkal, bude to retrospektiva, a abychom s n\u00ed za\u010dali, vezm\u011bme si ten nejsnadn\u011bj\u0161\u00ed p\u0159\u00edklad.<\/p>\n<p><img decoding=\"async\" alt=\"Architektura ukl\u00e1d\u00e1n\u00ed a distribuce fotografi\u00ed v Badoo\" src=\"\/wp-content\/uploads\/2020\/01\/22b9627cda7ccadd9869b64e9d043d89.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nM\u00e1me spole\u010dn\u00fd \u00fakol, mus\u00edme p\u0159ij\u00edmat, uchov\u00e1vat a poskytovat fotografie u\u017eivatel\u016f. V takov\u00e9 podob\u011b je \u00fakol obecn\u00fd, m\u016f\u017eeme pou\u017e\u00edt cokoliv:<\/p>\n<ul>\n<li>modern\u00ed cloudov\u00e9 \u00falo\u017ei\u0161t\u011b,<\/li>\n<li>k dispozici je tak\u00e9 mnoho dal\u0161\u00edch hotov\u00fdch \u0159e\u0161en\u00ed;<\/li>\n<li>m\u016f\u017eeme si nastavit n\u011bkolik stroj\u016f ve sv\u00e9m datov\u00e9m centru a um\u00edstit na n\u011b velk\u00e9 pevn\u00e9 disky a ukl\u00e1dat fotografie tam.<\/li>\n<\/ul>\n<p>\nBadoo historicky \u2014 a v sou\u010dasnosti i tehdy (v dob\u011b, kdy to teprve za\u010d\u00ednalo) \u2014 funguje na vlastn\u00edch serverech, uvnit\u0159 na\u0161ich vlastn\u00edch datov\u00fdch center. Proto pro n\u00e1s byla tato mo\u017enost optim\u00e1ln\u00ed.<\/p>\n<p><img decoding=\"async\" alt=\"Architektura ukl\u00e1d\u00e1n\u00ed a distribuce fotografi\u00ed v Badoo\" src=\"\/wp-content\/uploads\/2020\/01\/9434fbabada15670297c551066ad7e71.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nJednodu\u0161e jsme vzali n\u011bkolik stroj\u016f, pojmenovali je \u201ephotos\u201c a vytvo\u0159ili cluster, kter\u00fd uchov\u00e1v\u00e1 fotografie. Ale zd\u00e1 se, \u017ee n\u011bco chyb\u00ed. Aby to v\u0161e fungovalo, je pot\u0159eba n\u011bjak\u00fdm zp\u016fsobem ur\u010dit, na kter\u00e9m stroji budeme uchov\u00e1vat kter\u00e9 fotografie. A zde tak\u00e9 nen\u00ed t\u0159eba objevovat Ameriku.<\/p>\n<p><img decoding=\"async\" alt=\"Architektura ukl\u00e1d\u00e1n\u00ed a distribuce fotografi\u00ed v Badoo\" src=\"\/wp-content\/uploads\/2020\/01\/36258e7107fd8c60f7e9e563a2cb681c.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nDo na\u0161eho \u00falo\u017ei\u0161t\u011b informac\u00ed o u\u017eivatel\u00edch p\u0159id\u00e1me n\u011bjak\u00e9 pole. To bude kl\u00ed\u010d pro shardov\u00e1n\u00ed. V na\u0161em p\u0159\u00edpad\u011b jsme ho nazvali place_id, a tento id m\u00edst ur\u010duje m\u00edsto, kde jsou uchov\u00e1v\u00e1ny fotografie u\u017eivatel\u016f. Tvo\u0159\u00edme mapy.<\/p>\n<p>V prvn\u00edm kroku to m\u016f\u017eeme dokonce d\u011blat ru\u010dn\u011b \u2014 \u0159\u00edk\u00e1me, \u017ee fotografie tohoto u\u017eivatele s dan\u00fdm m\u00edstem se ulo\u017e\u00ed na tento server. D\u00edky t\u00e9to map\u011b v\u017edy v\u00edme, kdy u\u017eivatel nahr\u00e1v\u00e1 fotografii, kde ji ulo\u017eit, a v\u00edme, odkud ji poskytnout.<\/p>\n<p>Je to naprosto trivi\u00e1ln\u00ed sch\u00e9ma, ale m\u00e1 dost podstatn\u00e9 v\u00fdhody. Prvn\u00ed \u2014 je to jednoduch\u00e9, jak jsem ji\u017e \u0159ekl, a druh\u00e9 \u2014 s takov\u00fdm p\u0159\u00edstupem m\u016f\u017eeme snadno horizont\u00e1ln\u011b \u0161k\u00e1lovat, prost\u011b dod\u00e1vaj\u00edc\u00ed nov\u00e9 stroje a p\u0159id\u00e1vaj\u00edc\u00ed je do mapy. V\u00edce nen\u00ed pot\u0159eba d\u011blat.<\/p>\n<p>Tak to n\u011bjakou dobu bylo i u n\u00e1s.<\/p>\n<p><img decoding=\"async\" alt=\"Architektura ukl\u00e1d\u00e1n\u00ed a distribuce fotografi\u00ed v Badoo\" src=\"\/wp-content\/uploads\/2020\/01\/3effb442702f162f7bad9c18ec54392e.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nBylo to n\u011bkdy v roce 2009. Dod\u00e1vali jsme stroje, dod\u00e1vali...<\/p>\n<p>A v ur\u010dit\u00e9m okam\u017eiku jsme za\u010dali pozorovat, \u017ee toto sch\u00e9ma m\u00e1 ur\u010dit\u00e9 nev\u00fdhody. Jak\u00e9 nev\u00fdhody?<\/p>\n<p>P\u0159edev\u0161\u00edm to je omezen\u00e1 kapacita. Na jeden fyzick\u00fd server nem\u016f\u017eeme um\u00edstit tolik pevn\u00fdch disk\u016f, kolik bychom cht\u011bli. A to se s postupem \u010dasu a r\u016fstem datasetu stalo ur\u010dit\u00fdm probl\u00e9mem.<\/p>\n<p>A druh\u00e9. Je to netypick\u00e1 konfigurace stroj\u016f, proto\u017ee takov\u00e9 stroje se t\u011b\u017eko znovu vyu\u017e\u00edvaj\u00ed v n\u011bjak\u00fdch jin\u00fdch clusterech, jsou dost specifick\u00e9, tedy mus\u00ed b\u00fdt slab\u00e9 v\u00fdkonem, ale z\u00e1rove\u0148 s velk\u00fdm pevn\u00fdm diskem.<\/p>\n<p>To v\u0161e bylo v roce 2009, ale v z\u00e1sad\u011b jsou tyto po\u017eadavky aktu\u00e1ln\u00ed i dnes. M\u00e1me retrospektivu, tak\u017ee v roce 2009 bylo s t\u00edmto v\u0161\u00edm v\u0161echno \u0161patn\u011b.<\/p>\n<p>A posledn\u00ed bod \u2014 cena.<\/p>\n<p><img decoding=\"async\" alt=\"Architektura ukl\u00e1d\u00e1n\u00ed a distribuce fotografi\u00ed v Badoo\" src=\"\/wp-content\/uploads\/2020\/01\/e49010eaeab139199d8fe010caa65bf0.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nCena byla tehdy velmi vysok\u00e1, a my jsme museli hledat n\u011bjak\u00e9 alternativy. To znamen\u00e1, \u017ee jsme museli l\u00e9pe vyu\u017e\u00edvat jak m\u00edsto v datov\u00fdch centrech, tak fyzick\u00e9 servery, na kter\u00fdch je to v\u0161echno um\u00edst\u011bno. A na\u0161i syst\u00e9mov\u00ed in\u017een\u00fd\u0159i zah\u00e1jili rozs\u00e1hl\u00fd v\u00fdzkum, ve kter\u00e9m p\u0159ezkoumali spoustu r\u016fzn\u00fdch mo\u017enost\u00ed. Zkoumali i klastrov\u00e9 souborov\u00e9 syst\u00e9my, jako PolyCeph a Lustre. Tam byly probl\u00e9my s v\u00fdkonem a pom\u011brn\u011b n\u00e1ro\u010dn\u00e1 \u00fadr\u017eba. Ode\u0161li. Zkou\u0161eli jsme mountovat cel\u00fd dataset p\u0159es NFS na ka\u017edou ma\u0161inu, abychom t\u00edmto zp\u016fsobem n\u011bjak\u00fdm zp\u016fsobem \u0161k\u00e1lovali. \u010cten\u00ed tak\u00e9 nedopadlo dob\u0159e, zkou\u0161eli jsme r\u016fzn\u00e9 \u0159e\u0161en\u00ed od r\u016fzn\u00fdch dodavatel\u016f.<\/p>\n<p>A nakonec jsme se rozhodli pou\u017e\u00edvat takzvanou Storage Area Network.<\/p>\n<p><img decoding=\"async\" alt=\"Architektura ukl\u00e1d\u00e1n\u00ed a distribuce fotografi\u00ed v Badoo\" src=\"\/wp-content\/uploads\/2020\/01\/e99402fa641f3bd5fc8d57a9f8995642.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nJedn\u00e1 se o velk\u00e9 SHD, kter\u00e9 jsou ur\u010deny k ukl\u00e1d\u00e1n\u00ed velk\u00e9ho objemu dat. P\u0159edstavuj\u00ed si police s disky, kter\u00e9 jsou namontov\u00e1ny na kone\u010dn\u00e9 servrov\u00e9 jednotky pomoc\u00ed optiky. T\u00edm p\u00e1dem m\u00e1me n\u011bjak\u00fd pool stroj\u016f, kter\u00fd je dost mal\u00fd, a tyto SHD, kter\u00e9 jsou pro na\u0161i logiku vys\u00edl\u00e1n\u00ed transparentn\u00ed, tj. pro n\u00e1\u0161 nginx nebo n\u011bkoho jin\u00e9ho, obsluhuj\u00ed po\u017eadavky za t\u011bmito fotografiemi.<\/p>\n<p>Toto \u0159e\u0161en\u00ed m\u011blo jasn\u00e9 v\u00fdhody. Je to SHD. Je zam\u011b\u0159eno na ukl\u00e1d\u00e1n\u00ed fotografi\u00ed. Je to levn\u011bj\u0161\u00ed, ne\u017e kdy\u017e jednodu\u0161e nastavujeme stroje s pevn\u00fdmi disky.<\/p>\n<p>Druh\u00e1 v\u00fdhoda. <\/p>\n<p><img decoding=\"async\" alt=\"Architektura ukl\u00e1d\u00e1n\u00ed a distribuce fotografi\u00ed v Badoo\" src=\"\/wp-content\/uploads\/2020\/01\/7fd2e7d629cc239096b14c68be5f76bf.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nTo, \u017ee kapacita se stala mnohem v\u011bt\u0161\u00ed, tj. do mnohem men\u0161\u00edho objemu m\u016f\u017eeme um\u00edstit mnohem v\u00edce \u00falo\u017ei\u0161t\u011b.<\/p>\n<p>Ale tak\u00e9 se objevily nev\u00fdhody, kter\u00e9 se uk\u00e1zaly dost rychle. S rostouc\u00edm po\u010dtem u\u017eivatel\u016f a z\u00e1t\u011b\u017e\u00ed na tuto syst\u00e9m za\u010daly vznikat probl\u00e9my s v\u00fdkonem. A probl\u00e9m je dost jasn\u00fd \u2014 jak\u00e9koli SHD ur\u010den\u00e9 k ukl\u00e1d\u00e1n\u00ed velk\u00e9ho mno\u017estv\u00ed fotografi\u00ed v mal\u00e9m objemu, obvykle trp\u00ed intenzivn\u00edm \u010dten\u00edm. To plat\u00ed skute\u010dn\u011b i pro jak\u00e9koli cloudov\u00e9 \u00falo\u017ei\u0161t\u011b, a\u0165 u\u017e se jedn\u00e1 o cokoli. V sou\u010dasnosti neexistuje ide\u00e1ln\u00ed \u00falo\u017ei\u0161t\u011b, kter\u00e9 by bylo nekone\u010dn\u011b \u0161k\u00e1lovateln\u00e9, do kter\u00e9ho byste mohli cokoliv cp\u00e1t a kter\u00e9 by velmi dob\u0159e zvl\u00e1dalo \u010dten\u00ed. Zejm\u00e9na n\u00e1hodn\u00e1 \u010dten\u00ed.<\/p>\n<p><img decoding=\"async\" alt=\"Architektura ukl\u00e1d\u00e1n\u00ed a distribuce fotografi\u00ed v Badoo\" src=\"\/wp-content\/uploads\/2020\/01\/566e836e2fe13a2a3d040ae3fa865d6d.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nJako v p\u0159\u00edpad\u011b na\u0161ich fotografi\u00ed, proto\u017ee fotografie jsou dotazov\u00e1ny nesouvisle, a to velmi negativn\u011b ovliv\u0148uje jejich v\u00fdkon.<\/p>\n<p>I podle dne\u0161n\u00edch \u00fadaj\u016f, pokud se n\u00e1m vyskytne v\u00edce ne\u017e 500 RPS za fotografie na stroj, ke kter\u00e9mu je p\u0159ipojeno \u00falo\u017ei\u0161t\u011b, za\u010d\u00ednaj\u00ed ji\u017e probl\u00e9my. A to bylo pro n\u00e1s velmi \u0161patn\u00e9, proto\u017ee po\u010det u\u017eivatel\u016f roste a v\u0161e by m\u011blo jen zhor\u0161ovat. Je t\u0159eba to n\u011bjak optimalizovat.<\/p>\n<p>Abychom optimalizovali, rozhodli jsme se tehdy, o\u010dividn\u011b, pod\u00edvat se na profil z\u00e1t\u011b\u017ee \u2014 co se vlastn\u011b d\u011bje, co je t\u0159eba optimalizovat.<\/p>\n<p><img decoding=\"async\" alt=\"Architektura ukl\u00e1d\u00e1n\u00ed a distribuce fotografi\u00ed v Badoo\" src=\"\/wp-content\/uploads\/2020\/01\/d67310e6b90c4ee1337065790576c9a0.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nA tady n\u00e1m v\u0161echno hraje do karet.<\/p>\n<p>J\u00e1 jsem ji\u017e v prvn\u00edm slidu \u0159\u00edkal: m\u00e1me 80 tis\u00edc po\u017eadavk\u016f za sekundu na \u010dten\u00ed p\u0159i pouze 3,5 milionu upload\u016f denn\u011b. To znamen\u00e1 rozd\u00edl v t\u0159ech \u0159\u00e1dech. Je o\u010dividn\u00e9, \u017ee je t\u0159eba optimalizovat \u010dten\u00ed a prakticky v\u00edme jak.<\/p>\n<p>Je tu je\u0161t\u011b jeden mal\u00fd moment. Specifika slu\u017eby je takov\u00e1, \u017ee se \u010dlov\u011bk registruje, nahr\u00e1v\u00e1 fotografii, pot\u00e9 za\u010d\u00edn\u00e1 aktivn\u011b prohl\u00ed\u017eet ostatn\u00ed lidi, lajkovat je a je aktivn\u011b ukazov\u00e1n ostatn\u00edm lidem. Pak si najde partnera nebo nenajde, to u\u017e z\u00e1le\u017e\u00ed na okolnostech, a na n\u011bjakou dobu p\u0159est\u00e1v\u00e1 slu\u017ebu pou\u017e\u00edvat. V tu chv\u00edli, kdy ji pou\u017e\u00edv\u00e1, jsou jeho fotky velmi \u017e\u00e1dan\u00e9 \u2014 maj\u00ed velk\u00fd z\u00e1jem, mnoho lid\u00ed je prohl\u00ed\u017e\u00ed. Jakmile s t\u00edm p\u0159estane, dost rychle vypadne z takov\u00fdch intenzivn\u00edch zobrazen\u00ed ostatn\u00edm lidem, jako to bylo d\u0159\u00edve, a jeho fotky se prakticky v\u016fbec nepo\u017eaduj\u00ed.<\/p>\n<p><img decoding=\"async\" alt=\"Architektura ukl\u00e1d\u00e1n\u00ed a distribuce fotografi\u00ed v Badoo\" src=\"\/wp-content\/uploads\/2020\/01\/dcc7e306055d8f2b0895f24679bfc3e2.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nTedy m\u00e1me velmi mal\u00fd 'hot' dataset. Ale p\u0159esto je na n\u011bj opravdu mnoho po\u017eadavk\u016f. A zcela logick\u00fdm \u0159e\u0161en\u00edm je p\u0159idat cache.<\/p>\n<p>Cache s LRU vy\u0159e\u0161\u00ed v\u0161echny na\u0161e probl\u00e9my. Co d\u011bl\u00e1me?<\/p>\n<p><img decoding=\"async\" alt=\"Architektura ukl\u00e1d\u00e1n\u00ed a distribuce fotografi\u00ed v Badoo\" src=\"\/wp-content\/uploads\/2020\/01\/2bd0c88a97d928f386db21549fc02500.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nP\u0159id\u00e1v\u00e1me p\u0159ed n\u00e1\u0161 velk\u00fd cluster se storage je\u0161t\u011b jeden relativn\u011b mal\u00fd, kter\u00fd naz\u00fdv\u00e1me fotoketchem (photoscache). To je vlastn\u011b jen cache proxy.<\/p>\n<p>Jak to funguje zevnit\u0159? Tady je n\u00e1\u0161 u\u017eivatel, tady je storage. V\u0161e jako d\u0159\u00edve. Co mezi nimi p\u0159id\u00e1v\u00e1me?<\/p>\n<p><img decoding=\"async\" alt=\"Architektura ukl\u00e1d\u00e1n\u00ed a distribuce fotografi\u00ed v Badoo\" src=\"\/wp-content\/uploads\/2020\/01\/7b7b5f102f1c012b2aef8f02ec731fac.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nJe to jen stroj s fyzick\u00fdm lok\u00e1ln\u00edm diskem, kter\u00fd je rychl\u00fd. \u0158ekn\u011bme, s SSD. Na tomto disku se ukl\u00e1d\u00e1 n\u011bjak\u00e1 lok\u00e1ln\u00ed cache.<\/p>\n<p>Jak to vypad\u00e1? U\u017eivatel pos\u00edl\u00e1 po\u017eadavek na fotku. NGINX nejprve hled\u00e1 v lok\u00e1ln\u00ed cache. Pokud nen\u00ed, prov\u00e1d\u00ed prost\u011b proxy_pass na n\u00e1\u0161 storage, st\u00e1hne fotografii odtud a poskytne ji u\u017eivateli.<\/p>\n<p>Ale to je velmi ban\u00e1ln\u00ed a nen\u00ed jasn\u00e9, co se uvnit\u0159 d\u011bje. Funguje to zhruba tak.<\/p>\n<p><img decoding=\"async\" alt=\"Architektura ukl\u00e1d\u00e1n\u00ed a distribuce fotografi\u00ed v Badoo\" src=\"\/wp-content\/uploads\/2020\/01\/46be24df171d1008e2a9099b7e4c9be6.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nCache je logicky rozd\u011blena na t\u0159i vrstvy. Kdy\u017e \u0159\u00edk\u00e1m 't\u0159i vrstvy', neznamen\u00e1 to, \u017ee tam existuje n\u011bjak\u00fd slo\u017eit\u00fd syst\u00e9m. Ne, to je prost\u011b podm\u00edn\u011bn\u011b t\u0159i adres\u00e1\u0159e v souborov\u00e9m syst\u00e9mu:<\/p>\n<ol>\n<li>To je buffer, kam se dost\u00e1vaj\u00ed pr\u00e1v\u011b nahran\u00e9 z proxy fotografie.<\/li>\n<li>To je hork\u00e1 cache, ve kter\u00e9 se ukl\u00e1daj\u00ed aktu\u00e1ln\u011b po\u017eadovan\u00e9 fotografie.<\/li>\n<li>A studen\u00e1 cache, kam se postupn\u011b fotografie tla\u010d\u00ed z hork\u00e9ho, kdy\u017e na n\u011b p\u0159ich\u00e1z\u00ed m\u00e9n\u011b po\u017eadavk\u016f.<\/li>\n<\/ol>\n<p>\nAby to fungovalo, mus\u00edme n\u011bjak spravovat tuto cache, mus\u00edme p\u0159estavovat fotografie v n\u00ed atd. To je tak\u00e9 velmi primitivn\u00ed proces.<\/p>\n<p><img decoding=\"async\" alt=\"Architektura ukl\u00e1d\u00e1n\u00ed a distribuce fotografi\u00ed v Badoo\" src=\"\/wp-content\/uploads\/2020\/01\/9b16fe3e99a5f88f06169f6e5eedf2b9.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nNginx na ka\u017edou \u017e\u00e1dost jednodu\u0161e zapisuje do RAMDisk access.log, v n\u011bm\u017e uv\u00e1d\u00ed cestu k fotce, kterou pr\u00e1v\u011b zpracov\u00e1v\u00e1 (relativn\u00ed cesta, samoz\u0159ejm\u011b), a to, jak\u00fdm zp\u016fsobem byla zpracov\u00e1na. Tedy tam m\u016f\u017ee b\u00fdt naps\u00e1no \u201efotka 1\u201c a d\u00e1l bu\u010f buffer, nebo hork\u00fd cache, nebo studen\u00fd cache, nebo proxy.<\/p>\n<p>V z\u00e1vislosti na tom mus\u00edme n\u011bjak rozhodnout, co d\u011blat s fotkou.<\/p>\n<p>Na ka\u017ed\u00e9m stroji b\u011b\u017e\u00ed mal\u00fd d\u00e9mon, kter\u00fd neust\u00e1le \u010dte tento log a ve sv\u00e9 pam\u011bti uchov\u00e1v\u00e1 statistiky o vyu\u017eit\u00ed jednotliv\u00fdch fotografi\u00ed.<\/p>\n<p><img decoding=\"async\" alt=\"Architektura ukl\u00e1d\u00e1n\u00ed a distribuce fotografi\u00ed v Badoo\" src=\"\/wp-content\/uploads\/2020\/01\/0dceda5e5ff432d1609520f0c152f152.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nJednodu\u0161e tam sb\u00edr\u00e1 \u00fadaje, udr\u017euje po\u010d\u00edtadla a periodicky d\u011bl\u00e1 n\u00e1sleduj\u00edc\u00ed. Aktivn\u011b po\u017eadovan\u00e9 fotografie, za nimi\u017e p\u0159ich\u00e1z\u00ed mnoho \u017e\u00e1dost\u00ed, p\u0159esune do hork\u00e9ho cache, a\u0165 u\u017e le\u017e\u00ed kdekoliv.<\/p>\n<p><img decoding=\"async\" alt=\"Architektura ukl\u00e1d\u00e1n\u00ed a distribuce fotografi\u00ed v Badoo\" src=\"\/wp-content\/uploads\/2020\/01\/941c13ecfea88372f3a7eb62d81b0c4e.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nFotografie, kter\u00e9 jsou po\u017eadov\u00e1ny z\u0159\u00eddka a za\u010daly b\u00fdt po\u017eadov\u00e1ny je\u0161t\u011b m\u00e9n\u011b, pomalu vyh\u00e1n\u00ed z hork\u00e9ho cache do studen\u00e9ho.<\/p>\n<p><img decoding=\"async\" alt=\"Architektura ukl\u00e1d\u00e1n\u00ed a distribuce fotografi\u00ed v Badoo\" src=\"\/wp-content\/uploads\/2020\/01\/a07235246995ad2fa4e6aeccae23a629.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nA kdy\u017e n\u00e1m v cache doch\u00e1z\u00ed m\u00edsto, jednodu\u0161e za\u010dneme v\u0161echny bez rozli\u0161ov\u00e1n\u00ed odstra\u0148ovat ze studen\u00e9ho cache. A to, mimochodem, funguje velmi dob\u0159e.<\/p>\n<p>Aby se fotka okam\u017eit\u011b ukl\u00e1dala p\u0159i proxyov\u00e1n\u00ed do bufferu, pou\u017e\u00edv\u00e1me direktivu proxy_store a buffer \u2014 to je tak\u00e9 RAMDisk, tak\u017ee pro u\u017eivatele to funguje velmi rychle. Co se t\u00fd\u010de vnit\u0159nost\u00ed samotn\u00e9ho cache serveru.<\/p>\n<p>Z\u016fstal n\u00e1m probl\u00e9m s t\u00edm, jak rozd\u011blovat \u017e\u00e1dosti mezi tyto servery.<\/p>\n<p>P\u0159edstavme si cluster dvaceti storage stroj\u016f a t\u0159\u00ed cache server\u016f (tak to prost\u011b vy\u0161lo).<\/p>\n<p><img decoding=\"async\" alt=\"Architektura ukl\u00e1d\u00e1n\u00ed a distribuce fotografi\u00ed v Badoo\" src=\"\/wp-content\/uploads\/2020\/01\/6bc970b687701f3b6d57e153e3abed18.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nMus\u00edme n\u011bjak ur\u010dit, kter\u00e9 \u017e\u00e1dosti o jak\u00e9 fotky a kam um\u00edstit.<\/p>\n<p>Nejjednodu\u0161\u0161\u00ed variantou je Round Robin. Nebo to d\u011blat n\u00e1hodn\u011b?<\/p>\n<p>To m\u00e1 samoz\u0159ejm\u011b n\u011bkolik nev\u00fdhod, proto\u017ee v takov\u00e9 situaci budeme velmi neefektivn\u011b vyu\u017e\u00edvat cache. \u017d\u00e1dosti budou p\u0159ist\u00e1vat na n\u011bjak\u00fdch n\u00e1hodn\u00fdch stroj\u00edch: zde se vetkla, na vedlej\u0161\u00ed u\u017e ji nen\u00ed. A fungovat to bude, pokud v\u016fbec, dost \u0161patn\u011b. I p\u0159i mal\u00e9m po\u010dtu stroj\u016f v clusteru.<\/p>\n<p>Mus\u00edme n\u011bjak jednozna\u010dn\u011b ur\u010dit, na kter\u00fd server um\u00edstit kterou \u017e\u00e1dost.<\/p>\n<p>Existuje jednoduch\u00fd zp\u016fsob. Vezmeme hash z URL nebo hash na\u0161eho shardovac\u00edho kl\u00ed\u010de obsa\u017een\u00e9ho v URL a d\u011bl\u00edme ho cel\u00fdm \u010d\u00edslem po\u010dtem server\u016f. Bude to fungovat? Bude.<\/p>\n<p><img decoding=\"async\" alt=\"Architektura ukl\u00e1d\u00e1n\u00ed a distribuce fotografi\u00ed v Badoo\" src=\"\/wp-content\/uploads\/2020\/01\/794a82ac1268b651ab936f0225b802a2.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nTedy m\u00e1me stoprocentn\u00ed po\u017eadavek, nap\u0159\u00edklad na n\u011bjakou \"example_url\", kter\u00fd v\u017edy p\u0159istane na serveru s indexem \"2\", a cache bude neust\u00e1le vyu\u017e\u00edv\u00e1na co nejl\u00e9pe.<\/p>\n<p>Ale v takov\u00e9m sch\u00e9matu nast\u00e1v\u00e1 probl\u00e9m s reshardingem. Resharding \u2014 mysl\u00edm t\u00edm zm\u011bnu po\u010dtu server\u016f.<\/p>\n<p>P\u0159edpokl\u00e1dejme, \u017ee n\u00e1\u0161 caching cluster p\u0159estal zvl\u00e1dat, a rozhodli jsme se p\u0159idat dal\u0161\u00ed stroj.<\/p>\n<p>P\u0159id\u00e1v\u00e1me.<\/p>\n<p><img decoding=\"async\" alt=\"Architektura ukl\u00e1d\u00e1n\u00ed a distribuce fotografi\u00ed v Badoo\" src=\"\/wp-content\/uploads\/2020\/01\/5f2fcba4c4f267c069811c3677bd0213.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nNyn\u00ed m\u00e1me v\u0161echno rozd\u011bleno ne na t\u0159i, ale na \u010dty\u0159i. Tak\u017ee prakticky v\u0161echny kl\u00ed\u010de, kter\u00e9 jsme d\u0159\u00edve m\u011bli, prakticky v\u0161echny URL nyn\u00ed \u017eij\u00ed na jin\u00fdch serverech. Cel\u00e1 cache byla okam\u017eit\u011b zneplatn\u011bna. V\u0161echny po\u017eadavky se hrnuly na n\u00e1\u0161 cluster-storage, bylo mu \u0161patn\u011b, do\u0161lo k selh\u00e1n\u00ed slu\u017eby a nespokojen\u00ed u\u017eivatel\u00e9. To si rozhodn\u011b nep\u0159ejeme.<\/p>\n<p>Tato varianta n\u00e1m tak\u00e9 nevyhovuje.<\/p>\n<p>Tedy co mus\u00edme ud\u011blat? Mus\u00edme n\u011bjak efektivn\u011b vyu\u017e\u00edvat cache, neust\u00e1le pos\u00edlat jeden po\u017eadavek na jeden a ten sam\u00fd server, ale z\u00e1rove\u0148 b\u00fdt odoln\u00ed proti resharding\u016fm. A toto \u0159e\u0161en\u00ed existuje, nen\u00ed to a\u017e tak slo\u017eit\u00e9. \u0158\u00edk\u00e1 se mu consistent hashing.<\/p>\n<p><img decoding=\"async\" alt=\"Architektura ukl\u00e1d\u00e1n\u00ed a distribuce fotografi\u00ed v Badoo\" src=\"\/wp-content\/uploads\/2020\/01\/1380758686274df9755a7ebf7d1c8753.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nJak to vypad\u00e1?<\/p>\n<p><img decoding=\"async\" alt=\"Architektura ukl\u00e1d\u00e1n\u00ed a distribuce fotografi\u00ed v Badoo\" src=\"\/wp-content\/uploads\/2020\/01\/44f97abfac46fa9f361055548819e8a3.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nVezmeme n\u011bjakou funkci od shardovac\u00edho kl\u00ed\u010de a rozprost\u0159eme v\u0161echny jej\u00ed hodnoty na kru\u017enici. Tedy v bod\u011b 0 se setk\u00e1vaj\u00ed jej\u00ed minim\u00e1ln\u00ed a maxim\u00e1ln\u00ed hodnoty. D\u00e1le na t\u00e9to stejn\u00e9 kru\u017enici um\u00edst\u00edme v\u0161echny na\u0161e servery p\u0159ibli\u017en\u011b t\u00edmto zp\u016fsobem:<\/p>\n<p><img decoding=\"async\" alt=\"Architektura ukl\u00e1d\u00e1n\u00ed a distribuce fotografi\u00ed v Badoo\" src=\"\/wp-content\/uploads\/2020\/01\/d470ec615a789913a6baa2726b05acea.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nKa\u017ed\u00fd server je ur\u010den jedn\u00edm bodem, a ten sektor, kter\u00fd jde a\u017e k n\u011bmu po sm\u011bru hodinov\u00fdch ru\u010di\u010dek, je obsluhov\u00e1n t\u00edmto hostitelem. Kdy\u017e k n\u00e1m p\u0159ich\u00e1zej\u00ed po\u017eadavky, okam\u017eit\u011b vid\u00edme, \u017ee nap\u0159\u00edklad po\u017eadavek A \u2014 m\u00e1 takov\u00fd hash \u2014 a je obsluhov\u00e1n serverem 2. Po\u017eadavek B \u2014 serverem 3. A tak d\u00e1le.<\/p>\n<p><img decoding=\"async\" alt=\"Architektura ukl\u00e1d\u00e1n\u00ed a distribuce fotografi\u00ed v Badoo\" src=\"\/wp-content\/uploads\/2020\/01\/fcdbd1caa9f1162cf02e8b0a1c79cb74.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nCo se v t\u00e9to situaci d\u011bje p\u0159i resharding?<\/p>\n<p><img decoding=\"async\" alt=\"Architektura ukl\u00e1d\u00e1n\u00ed a distribuce fotografi\u00ed v Badoo\" src=\"\/wp-content\/uploads\/2020\/01\/7386cda826e9419d5f8a9a0d6d6703c8.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nNezplat\u0148ujeme celou cache, jako d\u0159\u00edve, a neposouv\u00e1me v\u0161echny kl\u00ed\u010de, ale posouv\u00e1me ka\u017ed\u00fd sektor na mal\u00e9 vzd\u00e1lenosti tak, aby se na uvoln\u011bn\u00e9 m\u00edsto, \u0159ekn\u011bme, ve\u0161el n\u00e1\u0161 \u0161est\u00fd server, kter\u00fd chceme p\u0159idat, a p\u0159id\u00e1v\u00e1me ho tam.<\/p>\n<p><img decoding=\"async\" alt=\"Architektura ukl\u00e1d\u00e1n\u00ed a distribuce fotografi\u00ed v Badoo\" src=\"\/wp-content\/uploads\/2020\/01\/8680e0cf023ce911b4d162e9102c0545.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nJist\u011b, v takov\u00e9 situaci kl\u00ed\u010de tak\u00e9 selh\u00e1vaj\u00ed. Ale selh\u00e1vaj\u00ed mnohem m\u00e9n\u011b ne\u017e d\u0159\u00edve. A vid\u00edme, \u017ee na\u0161e dva prvn\u00ed kl\u00ed\u010de z\u016fstaly na sv\u00fdch serverech, a cachiovac\u00ed server se zm\u011bnil pouze pro posledn\u00ed kl\u00ed\u010d. Funguje to dost efektivn\u011b, a pokud p\u0159id\u00e1v\u00e1te nov\u00e9 hosty inkrement\u00e1ln\u011b, nen\u00ed v tom velk\u00fd probl\u00e9m. Po mal\u00fdch kroc\u00edch p\u0159id\u00e1v\u00e1te, \u010dek\u00e1te, a\u017e se cache op\u011bt napln\u00ed, a v\u0161echno dob\u0159e funguje.<\/p>\n<p>Jedin\u00e1 ot\u00e1zka z\u016fst\u00e1v\u00e1 p\u0159i selh\u00e1n\u00edch. P\u0159edpokl\u00e1dejme, \u017ee n\u00e1m n\u011bjak\u00e9 za\u0159\u00edzen\u00ed selhalo.<\/p>\n<p><img decoding=\"async\" alt=\"Architektura ukl\u00e1d\u00e1n\u00ed a distribuce fotografi\u00ed v Badoo\" src=\"\/wp-content\/uploads\/2020\/01\/8e3301af782ce8983716c4bc9add79a5.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nA v tu chv\u00edli bychom necht\u011bli znovu generovat tuto mapu, invalidovat \u010d\u00e1st cache atd., pokud se nap\u0159\u00edklad stroj restartoval a my mus\u00edme n\u011bjak obsluhovat po\u017eadavky. Na ka\u017ed\u00e9 lokalit\u011b m\u00e1me pouze jeden z\u00e1lo\u017en\u00ed fotokache, kter\u00fd pln\u00ed roli n\u00e1hrady pro kter\u00fdkoli stroj, kter\u00fd pr\u00e1v\u011b selhal. A pokud se n\u00e1hodou n\u011bjak\u00fd server stal nedostupn\u00fdm, provoz jde tam. Samoz\u0159ejm\u011b tam v tuto chv\u00edli nem\u00e1me \u017e\u00e1dnou cache, tj. je studen\u00e1, ale minim\u00e1ln\u011b po\u017eadavky u\u017eivatel\u016f jsou zpracov\u00e1v\u00e1ny. Pokud je to kr\u00e1tk\u00fd interval, zvl\u00e1d\u00e1me to zcela b\u011b\u017en\u011b. Jen v\u00edce zat\u00ed\u017een\u00ed na \u00falo\u017ei\u0161ti. Pokud je to del\u0161\u00ed interval, m\u016f\u017eeme se u\u017e rozhodnout \u2014 zda ten server z mapy odstranit nebo ne, nebo t\u0159eba nahradit jin\u00fdm.<\/p>\n<p>To je ohledn\u011b syst\u00e9mu cachingu. Pod\u00edvejme se na v\u00fdsledky.<\/p>\n<p>Na prvn\u00ed pohled se zd\u00e1, \u017ee tady nen\u00ed nic slo\u017eit\u00e9ho. Ale takov\u00fd zp\u016fsob spr\u00e1vy cache n\u00e1m dal hit rate kolem 98 %. Tj. z t\u011bch 80 tis\u00edc po\u017eadavk\u016f za sekundu pouze 1600 doraz\u00ed k \u00falo\u017ei\u0161t\u00edm, a to je zcela norm\u00e1ln\u00ed z\u00e1t\u011b\u017e, kterou to v klidu zvl\u00e1d\u00e1, m\u00e1me v\u017edy rezervu.<\/p>\n<p>Um\u00edstili jsme tyto servery do t\u0159\u00ed na\u0161ich datov\u00fdch center a z\u00edskali t\u0159i m\u00edsta p\u0159\u00edtomnosti \u2014 Praha, Miami a Hongkong.<\/p>\n<p><img decoding=\"async\" alt=\"Architektura ukl\u00e1d\u00e1n\u00ed a distribuce fotografi\u00ed v Badoo\" src=\"\/wp-content\/uploads\/2020\/01\/64eef492b65a014d8ba3c808b6579b41.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nT\u00edm p\u00e1dem jsou v\u00edce m\u00e9n\u011b lok\u00e1ln\u011b um\u00edst\u011bny k ka\u017ed\u00e9mu z na\u0161ich c\u00edlov\u00fdch trh\u016f.<\/p>\n<p>A jako p\u0159\u00edjemn\u00fd bonus jsme z\u00edskali tento cachiovac\u00ed proxy, na kter\u00e9m CPU vlastn\u011b nepracuje, proto\u017ee pro poskytov\u00e1n\u00ed obsahu nen\u00ed tolik pot\u0159eba. A tam jsme pomoc\u00ed NGINX + Lua implementovali opravdu mnoho utilit\u00e1rn\u00ed logiky.<\/p>\n<p><img decoding=\"async\" alt=\"Architektura ukl\u00e1d\u00e1n\u00ed a distribuce fotografi\u00ed v Badoo\" src=\"\/wp-content\/uploads\/2020\/01\/b65c609df5e1985240d04ee65a3e4f55.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nNap\u0159\u00edklad m\u016f\u017eeme experimentovat s webp nebo progresivn\u00edm jpeg (to jsou efektivn\u00ed modern\u00ed form\u00e1ty), sledovat, jak to ovliv\u0148uje provoz, p\u0159ij\u00edmat rozhodnut\u00ed, zap\u00ednat pro ur\u010dit\u00e9 zem\u011b atd.; prov\u00e1d\u011bt dynamick\u00e9 zm\u011bny velikosti nebo o\u0159ez\u00e1v\u00e1n\u00ed fotografi\u00ed za b\u011bhu.<\/p>\n<p>To je dobr\u00fd p\u0159\u00edpad pou\u017eit\u00ed, kdy m\u00e1te nap\u0159\u00edklad mobiln\u00ed aplikaci, kter\u00e1 zobrazuje fotografie, a mobiln\u00ed aplikace nechce zat\u011b\u017eovat CPU klienta t\u00edm, \u017ee by po\u017eadovala velkou fotografii a pot\u00e9 ji zm\u011bnila na n\u011bjakou velikost, aby ji mohla um\u00edstit do zobrazen\u00ed. M\u016f\u017eeme jednodu\u0161e dynamicky ur\u010dit v URL n\u011bjak\u00e9 parametry v hypotetick\u00e9m UPort, a fotokontrola sama zm\u011bn\u00ed velikost fotografie. Obvykle vybere tu velikost, kterou m\u00e1me fyzicky na disku, maxim\u00e1ln\u011b bl\u00edzkou po\u017eadovan\u00e9, a zmen\u0161\u00ed ji v konkr\u00e9tn\u00edch ji\u017e dan\u00fdch rozm\u011brech.<\/p>\n<blockquote><p>Mimochodem, zve\u0159ejnili jsme videonahr\u00e1vky z posledn\u00edch p\u011bti let konferenc\u00ed pro v\u00fdvoj\u00e1\u0159e vysoce zat\u00ed\u017een\u00fdch syst\u00e9m\u016f. <noindex><a rel=\"nofollow\" href=\"http:\/\/www.highload.ru\/\">HighLoad++<\/a><\/noindex>. Sledujte, studujte, sd\u00edlejte a p\u0159ihla\u0161te se k odb\u011bru <noindex><a rel=\"nofollow\" href=\"https:\/\/www.youtube.com\/user\/profyclub\">YouTube kan\u00e1lu<\/a><\/noindex>.<\/p><\/blockquote>\n<p>\nTak\u00e9 m\u016f\u017eeme p\u0159id\u00e1vat spoustu produktov\u00e9 logiky. Nap\u0159\u00edklad m\u016f\u017eeme podle parametr\u016f URL p\u0159id\u00e1vat r\u016fzn\u00e9 vodoznaky, m\u016f\u017eeme rozmaz\u00e1vat fotografie, zaml\u017eovat nebo pixelizovat. To je, kdy\u017e chceme uk\u00e1zat fotografii osoby, ale nechceme uk\u00e1zat jej\u00ed obli\u010dej, to dob\u0159e funguje, v\u0161echno je to realizov\u00e1no zde.<\/p>\n<p>Co jsme z\u00edskali? Z\u00edskali jsme t\u0159i body p\u0159\u00edtomnosti, dobrou m\u00edru hit\u016f a z\u00e1rove\u0148 n\u00e1m neprostoje CPU na t\u011bchto stroj\u00edch. Stal se nyn\u00ed d\u016fle\u017eit\u011bj\u0161\u00edm ne\u017e d\u0159\u00edv. Mus\u00edme nasadit siln\u011bj\u0161\u00ed stroje, ale stoj\u00ed to za to.<\/p>\n<p>Co se t\u00fd\u010de dod\u00e1v\u00e1n\u00ed fotografi\u00ed. Tady je to dost jasn\u00e9 a z\u0159ejm\u00e9. Mysl\u00edm, \u017ee jsem neobjevil Ameriku, takto funguje prakticky ka\u017ed\u00fd CDN.<\/p>\n<p>A pravd\u011bpodobn\u011b by si zku\u0161en\u00fd poslucha\u010d mohl polo\u017eit ot\u00e1zku: pro\u010d prost\u011b nevz\u00edt a nezm\u011bnit v\u0161echno na CDN? Bylo by to p\u0159ibli\u017en\u011b to stejn\u00e9, v\u0161echny modern\u00ed CDN to um\u00ed. A zde je n\u011bkolik d\u016fvod\u016f.<\/p>\n<p>Prvn\u00ed \u2014 to jsou fotografie.<\/p>\n<p><img decoding=\"async\" alt=\"Architektura ukl\u00e1d\u00e1n\u00ed a distribuce fotografi\u00ed v Badoo\" src=\"\/wp-content\/uploads\/2020\/01\/592c10eb74d46191bfcedec120f68556.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nTo je jeden z kl\u00ed\u010dov\u00fdch moment\u016f na\u0161\u00ed infrastruktury, a pot\u0159ebujeme nad nimi co nejv\u011bt\u0161\u00ed kontrolu. Pokud jde o n\u011bjak\u00e9 \u0159e\u0161en\u00ed od t\u0159et\u00ed strany a nem\u00e1te nad n\u00edm \u017e\u00e1dnou moc, bude pro v\u00e1s dost obt\u00ed\u017en\u00e9 s t\u00edm \u017e\u00edt, kdy\u017e m\u00e1te velk\u00fd dataset a kdy\u017e m\u00e1te velmi velk\u00fd proud po\u017eadavk\u016f od u\u017eivatel\u016f.<\/p>\n<p>Uv\u00e1d\u00edm p\u0159\u00edklad. V sou\u010dasnosti m\u016f\u017eeme, nap\u0159\u00edklad v p\u0159\u00edpad\u011b n\u011bjak\u00fdch probl\u00e9m\u016f nebo pulsov\u00e1n\u00ed zem\u011b, vstoupit na server a ladit tam, \u0159ekn\u011bme. M\u016f\u017eeme p\u0159idat sb\u011br n\u011bjak\u00fdch metrik, kter\u00e9 pot\u0159ebujeme, experimentovat, sledovat, jak to ovlivn\u00ed grafy atd. V sou\u010dasnosti sb\u00edr\u00e1me velk\u00e9 mno\u017estv\u00ed statistik o tomto cachovac\u00edm clusteru. A pravideln\u011b se na n\u011b d\u00edv\u00e1me a dlouho zkoum\u00e1me n\u011bjak\u00e9 anom\u00e1lie. Kdyby to bylo na stran\u011b CDN, bylo by mnohem t\u011b\u017e\u0161\u00ed to kontrolovat. Nebo nap\u0159\u00edklad, kdy\u017e dojde k n\u011bjak\u00e9 hav\u00e1rii, v\u00edme, co se stalo, v\u00edme, jak s t\u00edm \u017e\u00edt a jak to p\u0159ekonat. To je prvn\u00ed z\u00e1v\u011br.<\/p>\n<p>Druh\u00fd z\u00e1v\u011br je sp\u00ed\u0161e historick\u00fd, proto\u017ee syst\u00e9m se vyv\u00edj\u00ed ji\u017e dlouho a na r\u016fzn\u00fdch etap\u00e1ch bylo mnoho r\u016fzn\u00fdch po\u017eadavk\u016f na podnik\u00e1n\u00ed, a ne v\u017edy se vejdou do koncepce CDN.<\/p>\n<p>A bod, kter\u00fd vypl\u00fdv\u00e1 z p\u0159edchoz\u00edho \u2013<\/p>\n<p><img decoding=\"async\" alt=\"Architektura ukl\u00e1d\u00e1n\u00ed a distribuce fotografi\u00ed v Badoo\" src=\"\/wp-content\/uploads\/2020\/01\/007cf3aed2f9cc12185fc3865bf9d196.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nJe to, \u017ee na fotok cache m\u00e1me hodn\u011b specifick\u00e9 logiky, kterou nen\u00ed v\u017edy mo\u017en\u00e9 p\u0159idat na po\u017e\u00e1d\u00e1n\u00ed. St\u011b\u017e\u00ed by n\u011bjak\u00e9 CDN na va\u0161i \u017e\u00e1dost p\u0159id\u00e1valo n\u011bjak\u00e9 individu\u00e1ln\u00ed prvky. Nap\u0159\u00edklad \u0161ifrov\u00e1n\u00ed URL, pokud nechcete, aby klient mohl n\u011bco m\u011bnit. Chcete zm\u011bnit URL na serveru a za\u0161ifrovat ji, a potom sem p\u0159edat n\u011bjak\u00e9 dynamick\u00e9 parametry.<\/p>\n<p>Jak\u00fd z\u00e1v\u011br tedy vypl\u00fdv\u00e1? V na\u0161em p\u0159\u00edpad\u011b CDN nen\u00ed p\u0159\u00edli\u0161 dobr\u00e1 alternativa.<\/p>\n<p><img decoding=\"async\" alt=\"Architektura ukl\u00e1d\u00e1n\u00ed a distribuce fotografi\u00ed v Badoo\" src=\"\/wp-content\/uploads\/2020\/01\/cfc2963cca2357cf6b0f63419a4dc436.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nAle ve va\u0161em p\u0159\u00edpad\u011b, pokud m\u00e1te n\u011bjak\u00e9 specifick\u00e9 obchodn\u00ed po\u017eadavky, m\u016f\u017eete si bezstarostn\u011b realizovat to, co jsem v\u00e1m uk\u00e1zal. A p\u0159i podobn\u00e9m zat\u00ed\u017een\u00ed to bude skv\u011ble fungovat.<\/p>\n<p>Ale pokud m\u00e1te n\u011bjak\u00e9 obecn\u00e9 \u0159e\u0161en\u00ed a \u00fakol nen\u00ed p\u0159\u00edli\u0161 specifick\u00fd, m\u016f\u017eete s klidem pou\u017e\u00edt CDN. Nebo pokud je pro v\u00e1s mnohem d\u016fle\u017eit\u011bj\u0161\u00ed \u010das a zdroje ne\u017e kontrola.<\/p>\n<p><img decoding=\"async\" alt=\"Architektura ukl\u00e1d\u00e1n\u00ed a distribuce fotografi\u00ed v Badoo\" src=\"\/wp-content\/uploads\/2020\/01\/b76d25404918a5d0d8bf1d6644c8d873.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nA modern\u00ed CDN maj\u00ed prakticky v\u0161e, co jsem v\u00e1m pr\u00e1v\u011b \u0159ekl, s v\u00fdjimkou +\/- n\u011bjak\u00fdch funkc\u00ed.<\/p>\n<p>To je ohledn\u011b p\u0159ed\u00e1v\u00e1n\u00ed fotografi\u00ed.<\/p>\n<p>Te\u010f se posu\u0148me trochu vp\u0159ed v na\u0161em retrospektiv\u011b a poj\u010fme mluvit o ukl\u00e1d\u00e1n\u00ed.<\/p>\n<p>\u0160el rok 2013.<\/p>\n<p><img decoding=\"async\" alt=\"Architektura ukl\u00e1d\u00e1n\u00ed a distribuce fotografi\u00ed v Badoo\" src=\"\/wp-content\/uploads\/2020\/01\/f218bfc899512041967842430c02aa9b.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nCache servery byly p\u0159id\u00e1ny, probl\u00e9my s v\u00fdkonem zmizely. V\u0161echno je v po\u0159\u00e1dku. Dataset roste. V roce 2013 jsme m\u011bli p\u0159ibli\u017en\u011b 80 server\u016f p\u0159ipojen\u00fdch k \u00falo\u017ei\u0161t\u00edm a p\u0159ibli\u017en\u011b 40 cache server\u016f v ka\u017ed\u00e9m datov\u00e9m centru. To je 560 terabajt\u016f dat na ka\u017ed\u00e9m datov\u00e9m centru, co\u017e je dohromady kolem petabajtu.<\/p>\n<p><img decoding=\"async\" alt=\"Architektura ukl\u00e1d\u00e1n\u00ed a distribuce fotografi\u00ed v Badoo\" src=\"\/wp-content\/uploads\/2020\/01\/ef741ba4a9164e6940f55e93dfb46ef0.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nA se zv\u00fd\u0161en\u00edm datasetu za\u010daly dramaticky r\u016fst provozn\u00ed n\u00e1klady. V \u010dem to spo\u010d\u00edvalo?<\/p>\n<p><img decoding=\"async\" alt=\"Architektura ukl\u00e1d\u00e1n\u00ed a distribuce fotografi\u00ed v Badoo\" src=\"\/wp-content\/uploads\/2020\/01\/6cf5e489a4e97bc6e26fc6d4d16505b5.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nV tomto sch\u00e9matu, kter\u00e9 je zobrazeno \u2014 se SANem, p\u0159ipojen\u00fdmi stroji a cache servery \u2014 existuje mnoho bod\u016f selh\u00e1n\u00ed. Zat\u00edmco s v\u00fdpadky cache server\u016f jsme se ji\u017e d\u0159\u00edve vypo\u0159\u00e1dali, tam je to v\u00edce m\u00e9n\u011b p\u0159edv\u00eddateln\u00e9 a jasn\u00e9, na stran\u011b \u00falo\u017ei\u0161t\u011b to bylo mnohem hor\u0161\u00ed.<\/p>\n<p>Za prv\u00e9, samotn\u00e1 Storage Area Network (SAN) m\u016f\u017ee selhat.<\/p>\n<p>Za druh\u00e9, je p\u0159ipojena optikou k koncov\u00fdm stroj\u016fm. M\u016f\u017ee b\u00fdt probl\u00e9m s optick\u00fdmi kartami a switchemi.<\/p>\n<p><img decoding=\"async\" alt=\"Architektura ukl\u00e1d\u00e1n\u00ed a distribuce fotografi\u00ed v Badoo\" src=\"\/wp-content\/uploads\/2020\/01\/6959b79284c5b1244a70a1d64f191893.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nSamoz\u0159ejm\u011b, t\u011bch nen\u00ed tolik jako s samotn\u00fdm SANem, ale p\u0159esto jsou to tak\u00e9 body selh\u00e1n\u00ed.<\/p>\n<p>D\u00e1le samotn\u00fd stroj, kter\u00fd je p\u0159ipojen k \u00falo\u017ei\u0161ti. Ten tak\u00e9 m\u016f\u017ee selhat.<\/p>\n<p><img decoding=\"async\" alt=\"Architektura ukl\u00e1d\u00e1n\u00ed a distribuce fotografi\u00ed v Badoo\" src=\"\/wp-content\/uploads\/2020\/01\/1801cb502663a4de14329a5e67dfed79.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nCelkem m\u00e1me t\u0159i body selh\u00e1n\u00ed.<\/p>\n<p>Krom\u011b bod\u016f selh\u00e1n\u00ed je to n\u00e1ro\u010dn\u00fd \u00fadr\u017ebov\u00fd proces samotn\u00fdch \u00falo\u017ei\u0161\u0165.<\/p>\n<p>Je to slo\u017eit\u00fd v\u00edcestup\u0148ov\u00fd syst\u00e9m, a syst\u00e9mov\u00fdm in\u017een\u00fdr\u016fm se s n\u00edm n\u011bkdy t\u011b\u017eko pracuje.<\/p>\n<p>A posledn\u00ed, nejd\u016fle\u017eit\u011bj\u0161\u00ed bod. Pokud v kter\u00e9koliv z t\u011bchto t\u0159\u00ed bod\u016f dojde k selh\u00e1n\u00ed, m\u00e1me n\u011bjakou pravd\u011bpodobnost ztr\u00e1ty u\u017eivatelsk\u00fdch dat, proto\u017ee m\u016f\u017ee b\u00fdt po\u0161kozen souborov\u00fd syst\u00e9m.<\/p>\n<p><img decoding=\"async\" alt=\"Architektura ukl\u00e1d\u00e1n\u00ed a distribuce fotografi\u00ed v Badoo\" src=\"\/wp-content\/uploads\/2020\/01\/4bf8bc1c73a2beef74cb4ee0ee45679b.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nP\u0159edstavme si, \u017ee n\u00e1m byl po\u0161kozen souborov\u00fd syst\u00e9m. Jeho obnova trv\u00e1, za prv\u00e9, dlouho \u2014 m\u016f\u017ee to trvat t\u00fdden p\u0159i velk\u00e9m objemu dat. A za druh\u00e9, pravd\u011bpodobn\u011b nakonec z\u00edsk\u00e1me spoustu nejasn\u00fdch soubor\u016f, kter\u00e9 bude t\u0159eba n\u011bjak p\u0159i\u0159adit k fotografi\u00edm u\u017eivatel\u016f. A my rizikujeme ztr\u00e1tu dat. Riziko je dost vysok\u00e9. A \u010d\u00edm \u010dast\u011bji se takov\u00e9 situace vyskytuj\u00ed a \u010d\u00edm v\u00edce probl\u00e9m\u016f se ve v\u0161ech t\u011bchto vazb\u00e1ch objevuje, t\u00edm je toto riziko vy\u0161\u0161\u00ed.<\/p>\n<p>S t\u00edm bylo t\u0159eba n\u011bco ud\u011blat. A rozhodli jsme, \u017ee jednodu\u0161e z\u00e1lohujeme data. To je vlastn\u011b z\u0159ejm\u00e9 a dobr\u00e9 \u0159e\u0161en\u00ed. Co jsme ud\u011blali?<\/p>\n<p><img decoding=\"async\" alt=\"Architektura ukl\u00e1d\u00e1n\u00ed a distribuce fotografi\u00ed v Badoo\" src=\"\/wp-content\/uploads\/2020\/01\/f80ec2b65003c40bd2fabf88966b3e05.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nTakto vypadal n\u00e1\u0161 server, kter\u00fd byl d\u0159\u00edve p\u0159ipojen k \u00falo\u017ei\u0161ti. To je jedna hlavn\u00ed partition, co\u017e je v podstat\u011b blokov\u00e9 za\u0159\u00edzen\u00ed, kter\u00e9 p\u0159edstavuje skute\u010dn\u00fd mount na vzd\u00e1len\u00e9 \u00falo\u017ei\u0161t\u011b optikou.<\/p>\n<p>Jednodu\u0161e jsme p\u0159idali druhou partition.<\/p>\n<p><img decoding=\"async\" alt=\"Architektura ukl\u00e1d\u00e1n\u00ed a distribuce fotografi\u00ed v Badoo\" src=\"\/wp-content\/uploads\/2020\/01\/d3dba75ef477269d92145cd69d587527.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nUm\u00edstili jsme druh\u00e9 storage vedle (na\u0161t\u011bst\u00ed to nen\u00ed tak drah\u00e9), a pojmenovali jsme ho jako z\u00e1lo\u017en\u00ed odd\u00edl. Tak\u00e9 je p\u0159ipojen optikou, nach\u00e1z\u00ed se na stejn\u00e9 ma\u0161in\u011b. Ale mus\u00edme n\u011bjak synchronizovat data mezi nimi.<\/p>\n<p>Zde jednodu\u0161e vytv\u00e1\u0159\u00edme vedle asynchronn\u00ed frontu.<\/p>\n<p><img decoding=\"async\" alt=\"Architektura ukl\u00e1d\u00e1n\u00ed a distribuce fotografi\u00ed v Badoo\" src=\"\/wp-content\/uploads\/2020\/01\/67bced2b45c589009cfb01a1c21398a5.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nNen\u00ed to p\u0159\u00edli\u0161 zat\u00ed\u017een\u00e9. V\u00edme, \u017ee m\u00e1me m\u00e1lo z\u00e1znam\u016f. Fronta je jednodu\u0161e tabulka v MySQL, do kter\u00e9 se zapisuj\u00ed \u0159\u00e1dky typu \"je t\u0159eba z\u00e1lohovat tento obr\u00e1zek\". P\u0159i jak\u00e9koliv zm\u011bn\u011b nebo uploadu kop\u00edrujeme z hlavn\u00edho odd\u00edlu na z\u00e1lo\u017en\u00ed asynchronn\u011b nebo n\u011bjak\u00fdm background workerem.<\/p>\n<p>A t\u00edmto zp\u016fsobem m\u00e1me v\u017edy dva konzistentn\u00ed odd\u00edly. I kdy\u017e jedna \u010d\u00e1st tohoto syst\u00e9mu sel\u017ee, v\u017edy m\u016f\u017eeme vym\u011bnit hlavn\u00ed odd\u00edl se z\u00e1lo\u017en\u00edm a v\u0161echno bude pokra\u010dovat v chodu.<\/p>\n<p>Ale kv\u016fli tomu v\u00fdrazn\u011b nar\u016fst\u00e1 z\u00e1t\u011b\u017e na \u010dten\u00ed, proto\u017ee krom\u011b klient\u016f, kte\u0159\u00ed \u010dtou z hlavn\u00edho odd\u00edlu, proto\u017ee si nejd\u0159\u00edve prohl\u00ed\u017eej\u00ed obr\u00e1zek tam (je tam nov\u011bj\u0161\u00ed), a pak hledaj\u00ed na z\u00e1lo\u017en\u00edm, pokud nena\u0161li (ale to u\u017e d\u011bl\u00e1 pouze NGINX), tak\u00e9 na\u0161e z\u00e1lohovac\u00ed syst\u00e9m nyn\u00ed \u010dte z hlavn\u00edho odd\u00edlu. Nen\u00ed to tak, \u017ee by to bylo \u00fazk\u00e9 m\u00edsto, ale necht\u011bli jsme zbyte\u010dn\u011b zvy\u0161ovat z\u00e1t\u011b\u017e.<\/p>\n<p>A p\u0159idali jsme t\u0159et\u00ed disk, kter\u00fd je mal\u00fd SSD, a nazvali jsme ho bufferem.<\/p>\n<p><img decoding=\"async\" alt=\"Architektura ukl\u00e1d\u00e1n\u00ed a distribuce fotografi\u00ed v Badoo\" src=\"\/wp-content\/uploads\/2020\/01\/fd35481822b4170087e5f670bf338dc5.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nJak to nyn\u00ed funguje.<\/p>\n<p>U\u017eivatel uploaduje fotku do bufferu, pot\u00e9 se ode\u0161le ud\u00e1lost do fronty o tom, \u017ee je t\u0159eba ji zkop\u00edrovat na dva odd\u00edly. Je zkop\u00edrov\u00e1na a fotografie n\u011bjak\u00fd \u010das (\u0159ekn\u011bme jeden den) \u017eije v bufferu, a teprve pot\u00e9 se odstran\u00ed. To v\u00fdrazn\u011b zlep\u0161uje u\u017eivatelskou zku\u0161enost, proto\u017ee u\u017eivatel nahr\u00e1v\u00e1 fotografii, a obvykle se ihned za\u010d\u00ednaj\u00ed objevovat po\u017eadavky, nebo s\u00e1m aktualizuje str\u00e1nku, refreshuje. Ale to v\u0161echno z\u00e1vis\u00ed na aplikaci, kter\u00e1 prov\u00e1d\u00ed upload.<\/p>\n<p>Nebo, nap\u0159\u00edklad, jin\u00e9 lidi, kter\u00fdm za\u010dal ukazovat, ihned za touto fotkou pos\u00edlaj\u00ed po\u017eadavky. V cache je\u0161t\u011b nen\u00ed, prvn\u00ed po\u017eadavek prob\u00edh\u00e1 velmi rychle. V podstat\u011b, stejn\u011b jako s fotokashou. Pomal\u00e9 storage se na tom v\u016fbec nepod\u00edl\u00ed. A kdy\u017e bude po dni odstran\u011bna, ji\u017e je bu\u010f ulo\u017eena v na\u0161\u00ed cache vrstv\u011b, nebo pravd\u011bpodobn\u011b nikomu nen\u00ed pot\u0159eba. Tedy, u\u017eivatelsk\u00e1 zku\u0161enost zde v\u00fdrazn\u011b vzrostla d\u00edky takov\u00fdm jednoduch\u00fdm manipulac\u00edm.<\/p>\n<p>A co je nejd\u016fle\u017eit\u011bj\u0161\u00ed: p\u0159estali jsme ztr\u00e1cet data.<\/p>\n<p><img decoding=\"async\" alt=\"Architektura ukl\u00e1d\u00e1n\u00ed a distribuce fotografi\u00ed v Badoo\" src=\"\/wp-content\/uploads\/2020\/01\/526f549457dd0c438f32b0adb76df76c.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n\u0158ekn\u011bme, \u017ee jsme p\u0159estali <em>potenci\u00e1ln\u011b<\/em> ztr\u00e1cet data, proto\u017ee jsme je vlastn\u011b p\u0159\u00edli\u0161 neztr\u00e1celi. Ale nebezpe\u010d\u00ed tu bylo. Vid\u00edme, \u017ee takov\u00e9 \u0159e\u0161en\u00ed je ur\u010dit\u011b dobr\u00e9, ale vypad\u00e1 to trochu jako zm\u00edrn\u011bn\u00ed symptom\u016f probl\u00e9mu, m\u00edsto abychom ho vy\u0159e\u0161ili do hloubky. N\u011bkter\u00e9 probl\u00e9my zde z\u016fstaly.<\/p>\n<p>Za prv\u00e9, jde o bod selh\u00e1n\u00ed v podob\u011b samotn\u00e9ho fyzick\u00e9ho hostitele, na n\u011bm\u017e cel\u00e1 tato ma\u0161inerie b\u011b\u017e\u00ed, ten nikam nezmizel.<\/p>\n<p><img decoding=\"async\" alt=\"Architektura ukl\u00e1d\u00e1n\u00ed a distribuce fotografi\u00ed v Badoo\" src=\"\/wp-content\/uploads\/2020\/01\/0eb3d292655a7edf912793ed81e8931c.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nZa druh\u00e9, z\u016fstaly probl\u00e9my se SANy, z\u016fstal jejich t\u011b\u017ek\u00fd provoz a tak d\u00e1le. To nebylo kritick\u00fdm faktorem, ale cht\u011bli jsme zkusit \u017e\u00edt bez toho.<\/p>\n<p>A vytvo\u0159ili jsme t\u0159et\u00ed verzi (ve skute\u010dnosti druhou) \u2013 verzi z\u00e1lohov\u00e1n\u00ed. Jak to vypadalo?<\/p>\n<p>To je to, co bylo \u2013<\/p>\n<p><img decoding=\"async\" alt=\"Architektura ukl\u00e1d\u00e1n\u00ed a distribuce fotografi\u00ed v Badoo\" src=\"\/wp-content\/uploads\/2020\/01\/8b1749b6feed200f8a64529ee2316c1a.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nHlavn\u00ed probl\u00e9my m\u00e1me s t\u00edm, \u017ee je to fyzick\u00fd hostitel.<\/p>\n<p>Nejprve odstra\u0148ujeme SANy, proto\u017ee chceme experimentovat, chceme zkusit prost\u011b lok\u00e1ln\u00ed pevn\u00e9 disky.<\/p>\n<p><img decoding=\"async\" alt=\"Architektura ukl\u00e1d\u00e1n\u00ed a distribuce fotografi\u00ed v Badoo\" src=\"\/wp-content\/uploads\/2020\/01\/fd0f5a8627d9179fcbc7097c1c18505f.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nJe to u\u017e rok 2014-2015 a tehdy se situace s disky a jejich kapacitou na jeden host zlep\u0161ila. Rozhodli jsme se, pro\u010d to nevyzkou\u0161et.<\/p>\n<p>A pot\u00e9 jednodu\u0161e vezmeme na\u0161i z\u00e1lohovac\u00ed \u010d\u00e1st a fyzicky ji p\u0159eneseme na samostatn\u00fd stroj.<\/p>\n<p><img decoding=\"async\" alt=\"Architektura ukl\u00e1d\u00e1n\u00ed a distribuce fotografi\u00ed v Badoo\" src=\"\/wp-content\/uploads\/2020\/01\/456e00bc8557d3340b06efcd1dde8a08.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nTak dostaneme takovou sch\u00e9matickou strukturu. M\u00e1me dv\u011b za\u0159\u00edzen\u00ed, kter\u00e1 uchov\u00e1vaj\u00ed identick\u00e9 datasetty. Kompletn\u011b se vz\u00e1jemn\u011b z\u00e1lohuj\u00ed a synchronizuj\u00ed data p\u0159es s\u00ed\u0165 pomoc\u00ed asynchronn\u00ed fronty ve stejn\u00e9m MySQL.<\/p>\n<p><img decoding=\"async\" alt=\"Architektura ukl\u00e1d\u00e1n\u00ed a distribuce fotografi\u00ed v Badoo\" src=\"\/wp-content\/uploads\/2020\/01\/d44cdb7b440ad4c58cc625123306a8c8.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nPro\u010d to dob\u0159e funguje \u2014 proto\u017ee m\u00e1me m\u00e1lo z\u00e1pis\u016f. Tedy, kdyby byl z\u00e1pis srovnateln\u00fd se \u010dten\u00edm, mo\u017en\u00e1 bychom dostali n\u011bjak\u00fd s\u00ed\u0165ov\u00fd nadbytek a probl\u00e9my. Z\u00e1pis je m\u00e1lo, \u010dten\u00ed hodn\u011b \u2014 tento zp\u016fsob funguje dob\u0159e, tedy dost z\u0159\u00eddka kop\u00edrujeme fotografie mezi t\u011bmito dv\u011bma servery.<\/p>\n<p>Jak to p\u0159esn\u011b funguje, pokud se na to pod\u00edv\u00e1me podrobn\u011bji.<\/p>\n<p><img decoding=\"async\" alt=\"Architektura ukl\u00e1d\u00e1n\u00ed a distribuce fotografi\u00ed v Badoo\" src=\"\/wp-content\/uploads\/2020\/01\/071c11abb6ee1d7734c53052139ff04b.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nNahr\u00e1v\u00e1n\u00ed. Zat\u011b\u017eovac\u00ed balancer jednodu\u0161e vyb\u00edr\u00e1 n\u00e1hodn\u00e9 hostitele s p\u00e1rem a prov\u00e1d\u00ed na n\u011bj nahr\u00e1v\u00e1n\u00ed. P\u0159itom samoz\u0159ejm\u011b prov\u00e1d\u00ed zdravotn\u00ed kontroly, aby zkontroloval, \u017ee stroj nevy\u0161el. Tedy nahr\u00e1v\u00e1 fotky pouze na \u017eiv\u00fd server, a pak se to v\u0161e p\u0159es asynchronn\u00ed frontu kop\u00edruje na jeho souseda. S nahr\u00e1v\u00e1n\u00edm je to naprosto jednoduch\u00e9.<\/p>\n<p>S \u00fakolem je to o n\u011bco slo\u017eit\u011bj\u0161\u00ed.<\/p>\n<p><img decoding=\"async\" alt=\"Architektura ukl\u00e1d\u00e1n\u00ed a distribuce fotografi\u00ed v Badoo\" src=\"\/wp-content\/uploads\/2020\/01\/f494904aec823456e18b987a8c98bf56.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nZde n\u00e1m pomohl Lua, proto\u017ee na vanilkov\u00e9m NGINXu je takovou logiku obt\u00ed\u017en\u00e9 realizovat. Nejd\u0159\u00edve d\u011bl\u00e1me dotaz na prvn\u00ed server, zkontrolujeme, zda tam je fotografie, proto\u017ee potenci\u00e1ln\u011b m\u016f\u017ee b\u00fdt nahran\u00e1 t\u0159eba na sousedn\u00ed server a sem se je\u0161t\u011b nedostala. Pokud je fotografie tam, je to skv\u011bl\u00e9. Hned ji p\u0159ed\u00e1me klientovi a mo\u017en\u00e1 ji ulo\u017e\u00edme do cache.<\/p>\n<p><img decoding=\"async\" alt=\"Architektura ukl\u00e1d\u00e1n\u00ed a distribuce fotografi\u00ed v Badoo\" src=\"\/wp-content\/uploads\/2020\/01\/de5f77e997bb049f4cfc1709da0f1788.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nPokud tam nen\u00ed, jednodu\u0161e ud\u011bl\u00e1me dotaz na souseda a odtud ji garantovan\u011b z\u00edsk\u00e1me.<\/p>\n<p><img decoding=\"async\" alt=\"Architektura ukl\u00e1d\u00e1n\u00ed a distribuce fotografi\u00ed v Badoo\" src=\"\/wp-content\/uploads\/2020\/01\/79aa5592acd28dfbc815261d0276c2e7.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nTak m\u016f\u017eeme op\u011bt \u0159\u00edci: mohou b\u00fdt probl\u00e9my s v\u00fdkonem, proto\u017ee neust\u00e1l\u00e9 round tripy \u2013 fotografii nahr\u00e1li, tady ji nen\u00ed, d\u011bl\u00e1me dva dotazy m\u00edsto jednoho, to by m\u011blo b\u011b\u017eet pomalu.<\/p>\n<p>V na\u0161\u00ed situaci to nefunguje pomalu.<\/p>\n<p><img decoding=\"async\" alt=\"Architektura ukl\u00e1d\u00e1n\u00ed a distribuce fotografi\u00ed v Badoo\" src=\"\/wp-content\/uploads\/2020\/01\/52ca3902d80990f512201672cd5e6441.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nShroma\u017e\u010fujeme spoustu metrik podle tohoto syst\u00e9mu a podm\u00ednkov\u00fd hit rate tohoto mechanizmu \u010din\u00ed kolem 95 %. Tedy zpo\u017ed\u011bn\u00ed tohoto z\u00e1lohov\u00e1n\u00ed je mal\u00e9 a d\u00edky tomu prakticky garantujeme, \u017ee pot\u00e9, co byla fotka nahran\u00e1, ji z\u00edsk\u00e1me u\u017e na prvn\u00ed pokus a nikam dvakr\u00e1t nechod\u00edme.<\/p>\n<p>Tak tedy, co jsme je\u0161t\u011b z\u00edskali, a co je opravdu skv\u011bl\u00e9?<\/p>\n<p>D\u0159\u00edve jsme m\u011bli hlavn\u00ed z\u00e1lohovac\u00ed odd\u00edl a \u010detli jsme z n\u011bj postupn\u011b. Tedy v\u017edy jsme nejprve hledali na hlavn\u00edm a pak na z\u00e1lo\u017en\u00edm. To byla jedna cesta.<\/p>\n<p>Te\u010f vyu\u017e\u00edv\u00e1me \u010dten\u00ed ze dvou stroj\u016f najednou. Rozd\u011blujeme dotazy Round Robinem. V mal\u00e9m procentu p\u0159\u00edpad\u016f d\u011bl\u00e1me dva dotazy. Ale celkov\u011b m\u00e1me nyn\u00ed dvakr\u00e1t v\u011bt\u0161\u00ed rezervu pro \u010dten\u00ed, ne\u017e jsme m\u011bli p\u0159edt\u00edm. A z\u00e1t\u011b\u017e se opravdu v\u00fdrazn\u011b sn\u00ed\u017eila jak na v\u00fdstupn\u00edch stroj\u00edch, tak p\u0159\u00edmo na \u00falo\u017ei\u0161t\u00edch, kter\u00e1 jsme v t\u00e9 dob\u011b tak\u00e9 m\u011bli.<\/p>\n<p>Pokud jde o odolnost proti poruch\u00e1m. O to jsme se p\u0159edev\u0161\u00edm sna\u017eili. S odolnost\u00ed zde v\u0161echno dopadlo skv\u011ble.<\/p>\n<p><img decoding=\"async\" alt=\"Architektura ukl\u00e1d\u00e1n\u00ed a distribuce fotografi\u00ed v Badoo\" src=\"\/wp-content\/uploads\/2020\/01\/a030a477e60d6e1f3d0b4e3395013df5.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nJedna ma\u0161ina sel\u017ee.<\/p>\n<p><img decoding=\"async\" alt=\"Architektura ukl\u00e1d\u00e1n\u00ed a distribuce fotografi\u00ed v Badoo\" src=\"\/wp-content\/uploads\/2020\/01\/42900b24e340c03ea642fd49f5b410ff.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n\u017d\u00e1dn\u00e9 probl\u00e9my! Syst\u00e9mov\u00fd in\u017een\u00fdr si dokonce m\u016f\u017ee v noci ani nevst\u00e1vat, po\u010dk\u00e1 do r\u00e1na, nic hrozn\u00e9ho se nestane.<\/p>\n<p>Pokud by p\u0159i selh\u00e1n\u00ed t\u00e9to ma\u0161iny selhala i fronta, tak\u00e9 \u017e\u00e1dn\u00e9 probl\u00e9my, log se nejprve hromad\u00ed na \u017eiv\u00e9 ma\u0161in\u011b a pak se ji\u017e dost\u00e1v\u00e1 do fronty, a pot\u00e9 na tu ma\u0161inu, kter\u00e1 se uvede do provozu po n\u011bjak\u00e9m \u010dase.<\/p>\n<p><img decoding=\"async\" alt=\"Architektura ukl\u00e1d\u00e1n\u00ed a distribuce fotografi\u00ed v Badoo\" src=\"\/wp-content\/uploads\/2020\/01\/ef42eadaaf3e3ecbf70847e9f88c1582.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nTo sam\u00e9 plat\u00ed pro \u00fadr\u017ebu. Jenom vypneme jeden ze stroj\u016f, ru\u010dn\u011b jej vyjmeme ze v\u0161ech pool\u016f, p\u0159estane k n\u011bmu proudit provoz, provedeme n\u011bjakou \u00fadr\u017ebu, n\u011bco uprav\u00edme a pak ho vr\u00e1t\u00edme do provozu. A tento backup se dostihne dost rychle. Tedy, za den v\u00fdpadku jednoho za\u0159\u00edzen\u00ed se dohon\u00ed v \u0159\u00e1du n\u011bkolika minut. To je opravdu velmi m\u00e1lo. Co se t\u00fd\u010de odolnosti, op\u011bt \u0159\u00edk\u00e1m, zde je v\u0161e skv\u011bl\u00e9.<\/p>\n<p>Jak\u00e9 z\u00e1v\u011bry m\u016f\u017eeme u\u010dinit z t\u00e9to sch\u00e9matu zaji\u0161t\u011bn\u00ed redundancy?<\/p>\n<p>Z\u00edskali jsme odolnost.<\/p>\n<p>Jednoduch\u00e9 pou\u017e\u00edv\u00e1n\u00ed. Proto\u017ee na stroj\u00edch jsou lok\u00e1ln\u00ed tvrd\u00e9 disky, je to mnohem pohodln\u011bj\u0161\u00ed z pohledu provozu in\u017een\u00fdr\u016f, kte\u0159\u00ed s t\u00edm pracuj\u00ed.<\/p>\n<p>Z\u00edskali jsme dvojn\u00e1sobnou rezervu pro \u010dten\u00ed.<\/p>\n<p>To je velmi dobr\u00fd bonus k odolnosti.<\/p>\n<p>Ale jsou tu i probl\u00e9my. Te\u010f m\u00e1me mnohem slo\u017eit\u011bj\u0161\u00ed v\u00fdvoj n\u011bjak\u00fdch funkc\u00ed, kter\u00e9 s t\u00edm souvisej\u00ed, proto\u017ee syst\u00e9m se stal na 100 % nakonec konzistentn\u00edm.<\/p>\n<p><img decoding=\"async\" alt=\"Architektura ukl\u00e1d\u00e1n\u00ed a distribuce fotografi\u00ed v Badoo\" src=\"\/wp-content\/uploads\/2020\/01\/b650276a41730cc303180e8304c5dabb.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nMus\u00edme \u0159\u00edct nap\u0159\u00edklad v n\u011bjak\u00e9m background jobu neust\u00e1le p\u0159em\u00fd\u0161let: \"Na jak\u00e9m serveru jsme te\u010f spu\u0161t\u011bni?\", \"Existuje zde opravdu aktu\u00e1ln\u00ed fotka?\" apod. To je samoz\u0159ejm\u011b v\u0161e zabaleno do wrapper\u016f a pro program\u00e1tora, kter\u00fd p\u00ed\u0161e obchodn\u00ed logiku, je to transparentn\u00ed. Nicm\u00e9n\u011b, i tak se objevil v\u011bt\u0161\u00ed slo\u017eit\u00fd vrstvu. Ale jsme ochotni se s t\u00edm sm\u00ed\u0159it v\u00fdm\u011bnou za ty bonusy, kter\u00e9 jsme z toho z\u00edskali.<\/p>\n<p>A op\u011bt se objev\u00ed n\u011bjak\u00fd konflikt.<\/p>\n<p>Na za\u010d\u00e1tku jsem \u0159\u00edkal, \u017ee uchov\u00e1vat v\u0161e na lok\u00e1ln\u00edch tvrd\u00fdch disc\u00edch je \u0161patn\u00e9. A te\u010f \u0159\u00edk\u00e1m, \u017ee se n\u00e1m to l\u00edb\u00ed.<\/p>\n<p>Ano, v pr\u016fb\u011bhu \u010dasu se situace v\u00fdrazn\u011b zm\u011bnila a nyn\u00ed m\u00e1 tento p\u0159\u00edstup mnoho v\u00fdhod. Za prv\u00e9, z\u00edsk\u00e1v\u00e1me mnohem jednodu\u0161\u0161\u00ed pou\u017eit\u00ed.<\/p>\n<p>Za druh\u00e9, je to v\u00fdkonn\u011bj\u0161\u00ed, proto\u017ee nem\u00e1me tyto automatick\u00e9 kontrolery, p\u0159ipojen\u00ed na diskov\u00e9 pole.<\/p>\n<p>Tam je obrovsk\u00e1 ma\u0161inerie, a to jsou prost\u011b n\u011bkolik disk\u016f, kter\u00e9 jsou konkr\u00e9tn\u011b zde v za\u0159\u00edzen\u00ed uspo\u0159\u00e1d\u00e1ny do RAIDu.<\/p>\n<p>Ale jsou tu i nev\u00fdhody.<\/p>\n<p><img decoding=\"async\" alt=\"Architektura ukl\u00e1d\u00e1n\u00ed a distribuce fotografi\u00ed v Badoo\" src=\"\/wp-content\/uploads\/2020\/01\/3c090e0801f1a686dc0d5fefce697c56.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nJe to p\u0159ibli\u017en\u011b 1,5kr\u00e1t dra\u017e\u0161\u00ed ne\u017e pou\u017e\u00edvat SANy, i za dne\u0161n\u00ed ceny. Proto jsme se rozhodli tak sm\u011ble nekonvertovat cel\u00fd n\u00e1\u0161 velk\u00fd cluster na za\u0159\u00edzen\u00ed s lok\u00e1ln\u00edmi tvrd\u00fdmi disky a rozhodli jsme se ponechat hybridn\u00ed \u0159e\u0161en\u00ed.<\/p>\n<p>Polovina stroj\u016f u n\u00e1s pracuje s pevn\u00fdmi disky (no, ne polovina \u2013 asi tak 30 procent). A zb\u00fdvaj\u00edc\u00ed \u010d\u00e1st tvo\u0159\u00ed star\u00e9 stroje, na kter\u00fdch d\u0159\u00edve b\u011b\u017eela prvn\u00ed verze z\u00e1lohov\u00e1n\u00ed. Jen jsme je znovu zamontovali, proto\u017ee nepot\u0159ebujeme \u017e\u00e1dn\u00e1 nov\u00e1 data ani nic jin\u00e9ho, prost\u011b jsme p\u0159ehodili mounty z jednoho fyzick\u00e9ho hostitele na dva.<\/p>\n<p>A m\u00e1me velkou rezervu v \u010dten\u00ed, a zjednodu\u0161ili jsme to. Pokud jsme d\u0159\u00edve montovali jeden storage na jeden stroj, nyn\u00ed montujeme \u010dty\u0159i na jeden p\u00e1r, nap\u0159\u00edklad. A to funguje norm\u00e1ln\u011b.<\/p>\n<p>Shr\u0148me kr\u00e1tce, co jsme dos\u00e1hli, za co jsme bojovali, a zda se n\u00e1m to povedlo.<\/p>\n<h3>Z\u00e1v\u011bry<\/h3>\n<p>\nM\u00e1me u\u017eivatele \u2013 cel\u00fdch 33 milion\u016f.<\/p>\n<p>M\u00e1me t\u0159i m\u00edsta p\u0159\u00edtomnosti \u2013 Praha, Miami, Hongkong.<\/p>\n<p>V nich je um\u00edst\u011bn cache layer, kter\u00fd tvo\u0159\u00ed stroje s rychl\u00fdmi m\u00edstn\u00edmi disky (SSD), na kter\u00fdch b\u011b\u017e\u00ed jednoduch\u00e1 architektura z NGINX, jeho access.loga a d\u00e9mon\u016f na Pythonu, kte\u0159\u00ed to v\u0161echno zpracov\u00e1vaj\u00ed a spravuj\u00ed cache.<\/p>\n<p>Pokud chcete, m\u016f\u017eete ve sv\u00e9m projektu, pokud pro v\u00e1s nejsou fotky tak kritick\u00e9 jako pro n\u00e1s, nebo pokud je pro v\u00e1s trade-off mezi kontrolou a rychlost\u00ed v\u00fdvoje a n\u00e1klady na zdroje v jin\u00e9m pom\u011bru, pak m\u016f\u017eete v klidu nahradit jeho CDN, modern\u00ed CDN to d\u011blaj\u00ed dob\u0159e.<\/p>\n<p>D\u00e1le n\u00e1sleduje \u00falo\u017en\u00fd layer, na kter\u00e9m m\u00e1me clustery ze stroj\u016f, kter\u00e9 se vz\u00e1jemn\u011b z\u00e1lohuj\u00ed, asynchronn\u011b kop\u00edruj\u00ed soubory z jednoho na druh\u00fd p\u0159i jak\u00e9koli zm\u011bn\u011b.<\/p>\n<p>P\u0159itom \u010d\u00e1st t\u011bchto stroj\u016f pracuje s m\u00edstn\u00edmi pevn\u00fdmi disky.<\/p>\n<p>\u010c\u00e1st t\u011bchto stroj\u016f je p\u0159ipojena k SAN.<\/p>\n<p><img decoding=\"async\" alt=\"Architektura ukl\u00e1d\u00e1n\u00ed a distribuce fotografi\u00ed v Badoo\" src=\"\/wp-content\/uploads\/2020\/01\/5ddbc4f73d86aa1f83734885d8c9dea0.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nNa jedn\u00e9 stran\u011b je to pohodln\u011bj\u0161\u00ed p\u0159i provozu a trochu v\u00edce v\u00fdkonn\u00e9, na druh\u00e9 stran\u011b je to pohodln\u00e9 z hlediska hustoty um\u00edst\u011bn\u00ed a ceny za gigabyte.<\/p>\n<p>To je takov\u00fd stru\u010dn\u00fd p\u0159ehled architektury toho, co jsme dos\u00e1hli a jak to v\u0161echno prob\u00edhalo.<\/p>\n<p>Je\u0161t\u011b n\u011bkolik jednoduch\u00fdch rad od l\u00eddra.<\/p>\n<p>Za prv\u00e9, pokud se n\u00e1hle rozhodnete, \u017ee mus\u00edte v\u0161echno zlep\u0161it ve va\u0161\u00ed infrastruktu\u0159e fotografi\u00ed, nejprve zm\u011b\u0159te, proto\u017ee mo\u017en\u00e1 nen\u00ed t\u0159eba nic zlep\u0161ovat.<\/p>\n<p><img decoding=\"async\" alt=\"Architektura ukl\u00e1d\u00e1n\u00ed a distribuce fotografi\u00ed v Badoo\" src=\"\/wp-content\/uploads\/2020\/01\/7770f41d0b190eb78ae0d19f4c8182a6.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nD\u00e1m p\u0159\u00edklad. M\u00e1me cluster stroj\u016f, kter\u00fd poskytuje fotografie z p\u0159\u00edloh v chatech, a tam st\u00e1le funguje sch\u00e9ma z roku 2009, a nikdo z toho netrp\u00ed. V\u0161em je dob\u0159e, v\u0161ichni jsou spokojeni.<\/p>\n<p>Aby se m\u011b\u0159ilo, nejprve nastavte n\u011bkolik m\u011b\u0159\u00edtek, pod\u00edvejte se na n\u011b a pak se rozhodn\u011bte, co v\u00e1m nevyhovuje a co je t\u0159eba zlep\u0161it. Pro m\u011b\u0159en\u00ed m\u00e1me skv\u011bl\u00fd n\u00e1stroj, kter\u00fd se naz\u00fdv\u00e1 Pinba.<\/p>\n<p>Umo\u017e\u0148uje sb\u00edrat velmi podrobn\u00e9 statistiky z NGINX pro ka\u017ed\u00fd po\u017eadavek, k\u00f3dy odpov\u011bd\u00ed a rozd\u011blen\u00ed \u010das\u016f - cokoliv budete pot\u0159ebovat. M\u00e1 vazby na r\u016fzn\u00e9 analytick\u00e9 syst\u00e9my, tak\u017ee to m\u016f\u017eete kr\u00e1sn\u011b sledovat.<\/p>\n<p>Nejprve jsme zm\u011b\u0159ili - pak jsme vylep\u0161ili.<\/p>\n<p>D\u00e1le. \u010cten\u00ed optimalizujeme pomoc\u00ed cache, z\u00e1pis - shardingem, ale to je samoz\u0159ejm\u00fd bod.<\/p>\n<p><img decoding=\"async\" alt=\"Architektura ukl\u00e1d\u00e1n\u00ed a distribuce fotografi\u00ed v Badoo\" src=\"\/wp-content\/uploads\/2020\/01\/f53a4777f1a2c7793eba4a72c94263a4.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nD\u00e1le. Pokud pr\u00e1v\u011b te\u010f za\u010d\u00edn\u00e1te budovat sv\u016fj syst\u00e9m, je mnohem lep\u0161\u00ed uchov\u00e1vat fotografie jako immutable soubory. Tak se vyhnete mnoha probl\u00e9m\u016fm s invalidac\u00ed cache, jak by logika m\u011bla naj\u00edt spr\u00e1vnou verzi fotografie apod.<\/p>\n<p><img decoding=\"async\" alt=\"Architektura ukl\u00e1d\u00e1n\u00ed a distribuce fotografi\u00ed v Badoo\" src=\"\/wp-content\/uploads\/2020\/01\/f345c59681275326cda1320b4f188ce9.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nP\u0159edstavte si, \u017ee jste nahr\u00e1li stovku, pot\u00e9 ji oto\u010dili, ud\u011bl\u00e1te tak, aby to byl fyzicky jin\u00fd soubor. Tedy nen\u00ed t\u0159eba p\u0159em\u00fd\u0161let: te\u010f u\u0161et\u0159\u00edm trochu m\u00edsta, zap\u00ed\u0161u do toho sam\u00e9ho souboru, zm\u011bn\u00edm verzi. To nikdy nefunguje dob\u0159e, s t\u00edm pozd\u011bji bude mnoho starost\u00ed.<\/p>\n<p>Dal\u0161\u00ed bod. O resize na\u017eivo.<\/p>\n<p>D\u0159\u00edve, kdy\u017e u\u017eivatel\u00e9 nahr\u00e1vali fotografii, ihned jsme vytvo\u0159ili celou \u0159adu velikost\u00ed na v\u0161echny mo\u017en\u00e9 p\u0159\u00edpady pro r\u016fzn\u00e9 klienty, a v\u0161echny tyto obr\u00e1zky le\u017eely na disku. Te\u010f jsme od toho upustili.<\/p>\n<p>Nechali jsme pouze t\u0159i z\u00e1kladn\u00ed velikosti: mal\u00e9, st\u0159edn\u00ed a velk\u00e9. V\u0161e ostatn\u00ed jednodu\u0161e zmen\u0161ujeme z velikosti, kter\u00e1 odpov\u00edd\u00e1 tomu, co bylo po\u017eadov\u00e1no v Uportu, prost\u011b provedeme downsizing a p\u0159ed\u00e1me u\u017eivateli.<\/p>\n<p>CPU cache vrstvy je zde mnohem levn\u011bj\u0161\u00ed, ne\u017e kdybychom neust\u00e1le znovu generovali tyto velikosti na ka\u017ed\u00e9m \u00falo\u017ei\u0161ti. Pokud nap\u0159\u00edklad chceme p\u0159idat novou, trv\u00e1 to m\u011bs\u00edc - spustit skript, kter\u00fd by to v\u0161e \u0159\u00e1dn\u011b ud\u011blal, ani\u017e by se zhroutila cluster. Tedy pokud je to nyn\u00ed mo\u017en\u00e9, l\u00e9pe m\u00edt co nejm\u00e9n\u011b fyzick\u00fdch velikost\u00ed, ale aby alespo\u0148 n\u011bjak\u00e9 rozlo\u017een\u00ed bylo, dejme tomu t\u0159i. A v\u0161echno ostatn\u00ed jednodu\u0161e resize na\u017eivo pomoc\u00ed p\u0159ipraven\u00fdch modul\u016f. To je te\u010f velmi snadn\u00e9 a dostupn\u00e9.<\/p>\n<p>A inkrement\u00e1ln\u00ed asynchronn\u00ed z\u00e1loha - to je dobr\u00e9.<\/p>\n<p>Jak uk\u00e1zala na\u0161e praxe, takov\u00fd syst\u00e9m skv\u011ble funguje s odlo\u017een\u00fdm kop\u00edrov\u00e1n\u00edm zm\u011bn\u011bn\u00fdch soubor\u016f.<\/p>\n<p><img decoding=\"async\" alt=\"Architektura ukl\u00e1d\u00e1n\u00ed a distribuce fotografi\u00ed v Badoo\" src=\"\/wp-content\/uploads\/2020\/01\/0e550721e838654e61e1fa4d08f9bb93.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nPosledn\u00ed bod je tak\u00e9 z\u0159ejm\u00fd. Pokud ve va\u0161\u00ed infrastruktu\u0159e moment\u00e1ln\u011b takov\u00e9 probl\u00e9my nejsou, ale existuje n\u011bco, co by mohlo selhat, ur\u010dit\u011b to sel\u017ee, jakmile se to m\u00edrn\u011b zhor\u0161\u00ed. Proto je lep\u0161\u00ed o tom p\u0159em\u00fd\u0161let p\u0159edem a vyhnout se takov\u00fdm probl\u00e9m\u016fm. To je ode m\u011b v\u0161e.<\/p>\n<h3>Kontakty<\/h3>\n<p>\n\u00bb <noindex><a rel=\"nofollow\" href=\"https:\/\/habrahabr.ru\/users\/bo0rsh201\/\" class=\"user_link\">bo0rsh201<\/a><\/noindex> <br \/>\n\u00bb <noindex><a rel=\"nofollow\" href=\"https:\/\/habrahabr.ru\/company\/badoo\/\">Blog spole\u010dnosti Badoo<\/a><\/noindex><\/p>\n<blockquote><p>Tato prezentace je p\u0159episem jednoho z nejlep\u0161\u00edch vystoupen\u00ed na konferenci o v\u00fdvoji syst\u00e9m\u016f s vysokou z\u00e1t\u011b\u017e\u00ed. <noindex><a rel=\"nofollow\" href=\"http:\/\/highload.ru\/?utm_source=habr&amp;utm_medium=media&amp;utm_campaign=past.articles&amp;utm_content=common\">HighLoad++<\/a><\/noindex>. Do konference HighLoad++ 2017 zb\u00fdv\u00e1 m\u00e9n\u011b ne\u017e m\u011bs\u00edc.<\/p>\n<p>M\u00e1me u\u017e p\u0159ipraveno <noindex><a rel=\"nofollow\" href=\"http:\/\/www.highload.ru\/2017\/abstracts\">Program konference<\/a><\/noindex>, aktu\u00e1ln\u011b se aktivn\u011b sestavuje rozvrh.<\/p>\n<p>Tento rok pokra\u010dujeme ve zkoum\u00e1n\u00ed t\u00e9matu architektur a \u0161k\u00e1lov\u00e1n\u00ed:<\/p>\n<ul>\n<li><noindex><a rel=\"nofollow\" href=\"http:\/\/www.highload.ru\/2017\/abstracts\/2946.html?utm_source=habr&amp;utm_medium=media&amp;utm_campaign=past.articles&amp;utm_content=common\">Jak obslou\u017eit miliardu u\u017eivatel\u016f a poskytnout terabitov\u00e9 p\u0159enosy dat<\/a><\/noindex> \/ \u0418\u0433\u043e\u0440\u044c \u0412\u0430\u0441\u0438\u043b\u044c\u0435\u0432<\/li>\n<li><noindex><a rel=\"nofollow\" href=\"http:\/\/www.highload.ru\/2017\/abstracts\/3088.html?utm_source=habr&amp;utm_medium=media&amp;utm_campaign=past.articles&amp;utm_content=common\">Jak p\u0159epsat z nulov\u00e9ho datab\u00e1zi osobn\u00edch zpr\u00e1v VKontakte a migrovat na ni bez prostoj\u016f<\/a><\/noindex> \/ \u0414\u043c\u0438\u0442\u0440\u0438\u0439 \u0415\u0433\u043e\u0440\u043e\u0432<\/li>\n<li><noindex><a rel=\"nofollow\" href=\"http:\/\/www.highload.ru\/2017\/abstracts\/2843.html?utm_source=habr&amp;utm_medium=media&amp;utm_campaign=past.articles&amp;utm_content=common\">Smrt z v\u00fdprodeje: jak se Yandex.Money sna\u017eili rozjet k Black Friday a udr\u017eet se<\/a><\/noindex> \/ \u0410\u043d\u0430\u0442\u043e\u043b\u0438\u0439 \u041f\u043b\u0430\u0441\u043a\u043e\u0432\u0441\u043a\u0438\u0439<\/li>\n<li><noindex><a rel=\"nofollow\" href=\"http:\/\/www.highload.ru\/2017\/abstracts\/3003.html?utm_source=habr&amp;utm_medium=media&amp;utm_campaign=past.articles&amp;utm_content=common\">Odoln\u00e1 architektura front\u00e1ln\u00edho syst\u00e9mu banky.<\/a><\/noindex> \/ \u0420\u043e\u043c\u0430\u043d \u0428\u0435\u0445\u043e\u0432\u0446\u043e\u0432, \u0410\u043b\u0435\u043a\u0441\u0435\u0439 \u0413\u0440\u043e\u043c\u0430\u0442\u0447\u0438\u043a\u043e\u0432<\/li>\n<li><noindex><a rel=\"nofollow\" href=\"http:\/\/www.highload.ru\/2017\/abstracts\/2948.html?utm_source=habr&amp;utm_medium=media&amp;utm_campaign=past.articles&amp;utm_content=common\">Architektura platebn\u00edho syst\u00e9mu: t\u00e9m\u011b\u0159 enterprise.<\/a><\/noindex> \/ \u0424\u0438\u043b\u0438\u043f\u043f \u0414\u0435\u043b\u044c\u0433\u044f\u0434\u043e<\/li>\n<\/ul>\n<p>\nN\u011bkter\u00e9 z t\u011bchto materi\u00e1l\u016f pou\u017e\u00edv\u00e1me tak\u00e9 v na\u0161em online kurzu o v\u00fdvoji syst\u00e9m\u016f s vysokou z\u00e1t\u011b\u017e\u00ed. <noindex><a rel=\"nofollow\" href=\"http:\/\/highload.guide\/?utm_source=habr&amp;utm_medium=media&amp;utm_campaign=past.articles&amp;utm_content=common\">HighLoad.Guide<\/a><\/noindex> \u2014 je to \u0159et\u011bzec speci\u00e1ln\u011b vybran\u00fdch e-mail\u016f, \u010dl\u00e1nk\u016f, materi\u00e1l\u016f a vide\u00ed. Ji\u017e nyn\u00ed v na\u0161ich u\u010debnic\u00edch m\u00e1me v\u00edce ne\u017e 30 jedine\u010dn\u00fdch materi\u00e1l\u016f. P\u0159ipojte se!\n<\/p><\/blockquote>\n<p>Zdroj: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/oleg-bunin\/blog\/340976\/\">habr.com<\/a><\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u0410\u0440\u0442\u0435\u043c \u0414\u0435\u043d\u0438\u0441\u043e\u0432 ( bo0rsh201, Badoo) Badoo \u2014 \u044d\u0442\u043e \u043a\u0440\u0443\u043f\u043d\u0435\u0439\u0448\u0438\u0439 \u0432 \u043c\u0438\u0440\u0435 \u0441\u0430\u0439\u0442 \u0437\u043d\u0430\u043a\u043e\u043c\u0441\u0442\u0432. \u041d\u0430 \u0434\u0430\u043d\u043d\u044b\u0439 \u043c\u043e\u043c\u0435\u043d\u0442 \u0443 \u043d\u0430\u0441 \u0437\u0430\u0440\u0435\u0433\u0438\u0441\u0442\u0440\u0438\u0440\u043e\u0432\u0430\u043d\u043e \u043f\u043e\u0440\u044f\u0434\u043a\u0430 330 \u043c\u0438\u043b\u043b\u0438\u043e\u043d\u043e\u0432 \u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u0442\u0435\u043b\u0435\u0439 \u043f\u043e \u0432\u0441\u0435\u043c\u0443 \u043c\u0438\u0440\u0443. \u041d\u043e, \u0447\u0442\u043e \u0433\u043e\u0440\u0430\u0437\u0434\u043e \u0431\u043e\u043b\u0435\u0435 \u0432\u0430\u0436\u043d\u043e \u0432 \u043a\u043e\u043d\u0442\u0435\u043a\u0441\u0442\u0435 \u043d\u0430\u0448\u0435\u0433\u043e \u0441\u0435\u0433\u043e\u0434\u043d\u044f\u0448\u043d\u0435\u0433\u043e \u0440\u0430\u0437\u0433\u043e\u0432\u043e\u0440\u0430, \u2014 \u044d\u0442\u043e \u0442\u043e, \u0447\u0442\u043e \u043c\u044b \u0445\u0440\u0430\u043d\u0438\u043c \u043e\u043a\u043e\u043b\u043e 3 \u043f\u0435\u0442\u0430\u0431\u0430\u0439\u0442 \u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u0442\u0435\u043b\u044c\u0441\u043a\u0438\u0445 \u0444\u043e\u0442\u043e\u0433\u0440\u0430\u0444\u0438\u0439. \u041a\u0430\u0436\u0434\u044b\u0439 \u0434\u0435\u043d\u044c \u043d\u0430\u0448\u0438 \u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u0442\u0435\u043b\u0438 \u0437\u0430\u043b\u0438\u0432\u0430\u044e\u0442 \u043f\u043e\u0440\u044f\u0434\u043a\u0430 3,5 \u043c\u0438\u043b\u043b\u0438\u043e\u043d\u043e\u0432 [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-55009","post","type-post","status-publish","format-standard","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.3 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u0410\u0440\u0442\u0435\u043c \u0414\u0435\u043d\u0438\u0441\u043e\u0432 (\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/arhitektura-hraneniya-i-otdachi-fotografij-v-badoo\" \/>\n\t\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.3\" \/>\n\t\t<meta property=\"og:locale\" content=\"cs_CZ\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47\u0410\u0440\u0445\u0438\u0442\u0435\u043a\u0442\u0443\u0440\u0430 \u0445\u0440\u0430\u043d\u0435\u043d\u0438\u044f \u0438 \u043e\u0442\u0434\u0430\u0447\u0438 \u0444\u043e\u0442\u043e\u0433\u0440\u0430\u0444\u0438\u0439 \u0432 Badoo | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u0410\u0440\u0442\u0435\u043c \u0414\u0435\u043d\u0438\u0441\u043e\u0432 (\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/arhitektura-hraneniya-i-otdachi-fotografij-v-badoo\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2020-01-09T21:00:00+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-02-18T11:03:04+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47Architektura ukl\u00e1d\u00e1n\u00ed a poskytov\u00e1n\u00ed fotografi\u00ed v Badoo | ProHoster","description":"Artem Denisov (","canonical_url":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/arhitektura-hraneniya-i-otdachi-fotografij-v-badoo","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"cs_CZ","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u0410\u0440\u0445\u0438\u0442\u0435\u043a\u0442\u0443\u0440\u0430 \u0445\u0440\u0430\u043d\u0435\u043d\u0438\u044f \u0438 \u043e\u0442\u0434\u0430\u0447\u0438 \u0444\u043e\u0442\u043e\u0433\u0440\u0430\u0444\u0438\u0439 \u0432 Badoo | ProHoster","og:description":"\u0410\u0440\u0442\u0435\u043c \u0414\u0435\u043d\u0438\u0441\u043e\u0432 (","og:url":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/arhitektura-hraneniya-i-otdachi-fotografij-v-badoo","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-01-09T21:00:00+00:00","article:modified_time":"2020-02-18T11:03:04+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"55009","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":"2026-01-24 13:30:15","breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 19:53:49","updated":"2026-01-24 13:30:15","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts\/55009","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/comments?post=55009"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts\/55009\/revisions"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/media?parent=55009"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/categories?post=55009"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/tags?post=55009"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}