{"id":91090,"date":"2020-08-08T13:42:11","date_gmt":"2020-08-08T11:42:11","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/arhitektura-s3-3-goda-evolyuczii-mail-ru-cloud-storage"},"modified":"2020-08-08T13:42:11","modified_gmt":"2020-08-08T11:42:11","slug":"arhitektura-s3-3-goda-evolyuczii-mail-ru-cloud-storage","status":"publish","type":"post","link":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/arhitektura-s3-3-goda-evolyuczii-mail-ru-cloud-storage","title":{"rendered":"Architektura S3: 3 roky evoluce Mail.ru Cloud Storage","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p><img decoding=\"async\" alt=\"Architektura S3: 3 roky evoluce Mail.ru Cloud Storage\" src=\"\/wp-content\/uploads\/2020\/08\/a53775776b3fc969cc15c9d13a64039d.jpeg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<em><noindex><a rel=\"nofollow\" href=\"https:\/\/www.deviantart.com\/st-pete\/art\/Storage-Corridor-408874509\">\u00dalo\u017en\u00e1 chodba<\/a><\/noindex> od St-Pete<\/em><\/p>\n<p><\/p>\n<p>Ahoj v\u0161ichni! Jsem Mons Anderson, architekt platformy <noindex><a rel=\"nofollow\" href=\"https:\/\/mcs.mail.ru\/\">Mail.ru Cloud Solutions<\/a><\/noindex>, pov\u00edm v\u00e1m, jak jsme vybudovali na\u0161e \u00falo\u017ei\u0161t\u011b S3, jak funguje, jak\u00e1 \u0159e\u0161en\u00ed se osv\u011bd\u010dila a co bychom zm\u011bnili, kdybychom za\u010d\u00ednali takov\u00fd projekt znovu.<\/p>\n<p><\/p>\n<p>\u010cl\u00e1nek byl p\u0159ipraven na z\u00e1klad\u011b prezentace na <noindex><a rel=\"nofollow\" href=\"https:\/\/corp.mail.ru\/ru\/press\/events\/databases-2\/\">@Databases Meetup<\/a><\/noindex> od Mail.ru Cloud Solutions &amp; Tarantool. V \u010dl\u00e1nku se budeme v\u011bnovat:<\/p>\n<p><\/p>\n<ul>\n<li>jak bylo uspo\u0159\u00e1d\u00e1no \u00falo\u017ei\u0161t\u011b Mail.ru, na jeho\u017e z\u00e1klad\u011b jsme vybudovali \u00falo\u017ei\u0161t\u011b S3;<\/li>\n<li>co jsme p\u0159idali, abychom vytvo\u0159ili Mail.ru Cloud Storage;<\/li>\n<li>jak funguje objektov\u00fd model ukl\u00e1d\u00e1n\u00ed a jak\u00e9 byly kroky k nasazen\u00ed do produkce;<\/li>\n<li>o \u00faprav\u00e1ch provozn\u00edho syst\u00e9mu: failover a \u0161k\u00e1lov\u00e1n\u00ed;<\/li>\n<li>jak jsme implementovali shardov\u00e1n\u00ed a resharing;<\/li>\n<li>a tak\u00e9 o pr\u00e1ci s SSL certifik\u00e1ty.<\/li>\n<\/ul>\n<p><\/p>\n<p>Pokud nechcete \u010d\u00edst, m\u016f\u017eete <noindex><a rel=\"nofollow\" href=\"https:\/\/youtu.be\/NEgm1nsv-qg\">se pod\u00edvat<\/a><\/noindex>.<\/p>\n<p><noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><\/p>\n<h2 id=\"kak-bylo-ustroeno-hranilische-mailru-poverh-kotorogo-my-stroili-s3-hranilische\">Jak bylo uspo\u0159\u00e1d\u00e1no \u00falo\u017ei\u0161t\u011b Mail.ru, na jeho\u017e z\u00e1klad\u011b jsme vybudovali \u00falo\u017ei\u0161t\u011b S3<\/h2>\n<p><\/p>\n<p>V\u00fdvoj na\u0161eho S3 za\u010dal na z\u00e1klad\u011b \u00falo\u017ei\u0161t\u011b Cloud Mail.ru, tak\u017ee je nejprve pot\u0159eba \u0159\u00edci, jak je uspo\u0159\u00e1d\u00e1no a co um\u00ed.<\/p>\n<p><\/p>\n<p>\u00dalo\u017ei\u0161t\u011b Cloud Mail.ru se skl\u00e1d\u00e1 ze server\u016f s disky. V pr\u016fm\u011bru modern\u00ed storage server m\u00e1 36 disk\u016f po 12\u201314 terabajtech. D\u0159\u00edve byly disky men\u0161\u00ed, ale za t\u0159i roky se objemy disk\u016f zv\u00fd\u0161ily a dnes jde t\u00e9m\u011b\u0159 o p\u016fl petabajtu syrov\u00fdch dat. <\/p>\n<p><\/p>\n<p>Disky z r\u016fzn\u00fdch server\u016f \u00falo\u017ei\u0161t\u011b jsou spojeny do tzv. \"p\u00e1r\u016f\" (pair). P\u00e1r je jedinou jednotkou ukl\u00e1d\u00e1n\u00ed soubor\u016f. V podstat\u011b jde o disk namontovan\u00fd do ur\u010dit\u00e9ho odd\u00edlu na ur\u010dit\u00e9 cest\u011b, kde mohou le\u017eet soubory identifikovan\u00e9 hash hodnotami. <\/p>\n<p><\/p>\n<p>P\u00e1r je historick\u00fd n\u00e1zev, kter\u00fd se uchoval a\u017e do dne\u0161n\u00edch dn\u016f, i kdy\u017e dnes p\u00e1r nemus\u00ed m\u00edt jen dva disky. M\u016f\u017ee jich b\u00fdt t\u0159i, a tak\u00e9 mohou existovat r\u016fzn\u00e9 hybridn\u00ed \u00falo\u017ei\u0161t\u011b, nap\u0159\u00edklad 3\/2. <\/p>\n<p>\n<img decoding=\"async\" alt=\"Architektura S3: 3 roky evoluce Mail.ru Cloud Storage\" src=\"\/wp-content\/uploads\/2020\/08\/544ab79846539262e2d4dff6ab95ffff.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><em>P\u00e1ry (pair) \u2014 jednotky ukl\u00e1d\u00e1n\u00ed objekt\u016f<\/em><\/p>\n<p><\/p>\n<p>V\u0161echny p\u00e1ry jsou uchov\u00e1v\u00e1ny v PairDB \u2014 to je aplikace na b\u00e1zi Tarantool. V\u0161echny datab\u00e1ze v na\u0161em \u00falo\u017ei\u0161ti, po\u010d\u00ednaje t\u011bmi nejstar\u0161\u00edmi, jsou zalo\u017eeny na Tarantool, jin\u00fdch datab\u00e1z\u00ed nepou\u017e\u00edv\u00e1me.<\/p>\n<p><\/p>\n<p>PairDB uchov\u00e1v\u00e1 v\u0161echny p\u00e1ry, jejich stavy, voln\u00e9 m\u00edsto, mo\u017enosti selh\u00e1n\u00ed, posledn\u00ed chyby. Tak\u00e9 m\u016f\u017ee sama kontrolovat p\u00e1ry, aktualizovat jejich stav, kontrolovat, zda funguj\u00ed nebo ne. To znamen\u00e1, \u017ee PairDB je jak\u00fdmsi obecn\u00fdm sn\u00edmkem stavu v\u0161ech disk\u016f na\u0161\u00ed syst\u00e9mu.<\/p>\n<p>\n<img decoding=\"async\" alt=\"Architektura S3: 3 roky evoluce Mail.ru Cloud Storage\" src=\"\/wp-content\/uploads\/2020\/08\/d50781b8b51ed5fe126bc84e19c934d1.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><em>Pair DB: datab\u00e1ze se stavem p\u00e1r\u016f<\/em><\/p>\n<p><\/p>\n<p>Na p\u00e1rech se ukl\u00e1daj\u00ed soubory, a abychom v\u011bd\u011bli, na kter\u00e9 p\u00e1\u0159e se jak\u00fd soubor nach\u00e1z\u00ed, je pot\u0159eba dal\u0161\u00ed datab\u00e1ze \u2014 FileDB. Ta uchov\u00e1v\u00e1 mapov\u00e1n\u00ed, definici souvislosti: tento soubor se ukl\u00e1d\u00e1 na t\u00e9to p\u00e1\u0159e, stejn\u011b jako mal\u00e9 mno\u017estv\u00ed pot\u0159ebn\u00fdch atribut\u016f.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Architektura S3: 3 roky evoluce Mail.ru Cloud Storage\" src=\"\/wp-content\/uploads\/2020\/08\/b1a64737cf7f916bff446c10c1f01d4c.jpeg\" style=\"display:block;margin: 0 auto;\" \/><em>File DB: m\u00edsto, kde se ukl\u00e1d\u00e1 soubor<\/em><\/p>\n<p>Dal\u0161\u00ed d\u016fle\u017eit\u00fd prvek \u2014 slu\u017eba Nylon, sm\u011brova\u010d pro pr\u00e1ci s datab\u00e1zemi. Je to jedin\u00e9 vstupn\u00ed m\u00edsto, kter\u00e9 umo\u017e\u0148uje pracovat prost\u0159ednictv\u00edm jednotn\u00e9ho rozhran\u00ed jak s PairDB, tak s FileDB. Je to stateless slu\u017eba, prov\u00e1d\u00ed vyva\u017eov\u00e1n\u00ed po\u017eadavk\u016f, rozum\u00ed, na kter\u00fd shard FileDB je pot\u0159eba j\u00edt, v\u00ed, kter\u00e9 p\u00e1ry jsou aktivn\u00ed a kter\u00e9 ne.<\/p>\n<p>\n<img decoding=\"async\" alt=\"Architektura S3: 3 roky evoluce Mail.ru Cloud Storage\" src=\"\/wp-content\/uploads\/2020\/08\/3e2225f229c8cfa462d620d2b5fb581e.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><em>Nylon: sm\u011brova\u010d pro pr\u00e1ci s datab\u00e1zemi<\/em><\/p>\n<p><\/p>\n<p>Tak\u00e9 je pot\u0159eba n\u011bjak\u00fdm zp\u016fsobem ukl\u00e1dat obsah do \u00falo\u017ei\u0161t\u011b. K tomu existuje slu\u017eba \u2014 Streamer. Nab\u00edz\u00ed dv\u011b HTTP metody: metodu PUT pro nahr\u00e1n\u00ed obsahu do \u00falo\u017ei\u0161t\u011b a metodu GET pro jeho sta\u017een\u00ed. HTTP je pom\u011brn\u011b popul\u00e1rn\u00ed a pohodln\u00fd protokol pro p\u0159enos dat. <\/p>\n<p><\/p>\n<p>Kdy\u017e se obrac\u00edme na Streamer\u2019y, ten skrze Nylon komunikuje s PairDB, zji\u0161\u0165uje, na kter\u00fd p\u00e1r lze nahr\u00e1t soubor, a pot\u00e9 p\u0159ed\u00e1v\u00e1 data po WebDAV na tento p\u00e1r. <\/p>\n<p><\/p>\n<p>V podstat\u011b je ka\u017ed\u00fd storage server \u2014 to je nginx plus disky, kter\u00e9 jsou p\u0159ipojeny na stanoven\u00e9 cesty. M\u016f\u017eeme ze Streamer nahr\u00e1t do \u00falo\u017ei\u0161t\u011b soubor, odstranit ho, p\u0159ejmenovat nebo zkontrolovat jeho integritu. To znamen\u00e1, \u017ee to je pohodln\u00e9 rozhran\u00ed pro n\u00edzko\u00farov\u0148ovou interakci s \u00falo\u017ei\u0161t\u011bm. <\/p>\n<p>\n<img decoding=\"async\" alt=\"Architektura S3: 3 roky evoluce Mail.ru Cloud Storage\" src=\"\/wp-content\/uploads\/2020\/08\/3722a89ed44cace36e9cb1f8bee4bd1e.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><em>Streamer: vstupn\u00ed bod do \u00falo\u017ei\u0161t\u011b<\/em><\/p>\n<p><\/p>\n<h2 id=\"chto-my-dobavili-chtoby-sdelat-s3-hranilische\">Co jsme p\u0159idali, abychom vytvo\u0159ili S3 \u00falo\u017ei\u0161t\u011b<\/h2>\n<p><\/p>\n<p>Tak\u017ee jsme se pod\u00edvali na z\u00e1kladn\u00ed strukturu \u00falo\u017ei\u0161t\u011b v dob\u011b, kdy jsme se chystali spustit S3 \u00falo\u017ei\u0161t\u011b. Pomoc\u00ed metody PUT jsme mohli um\u00edstit jak\u00fdkoliv obsah a jako identifik\u00e1tor t\u011bchto dat z\u00edskat hash. S t\u00edmto identifik\u00e1torem bylo pozd\u011bji mo\u017en\u00e9 p\u0159ij\u00edt a vyzvednout p\u016fvodn\u00ed soubor. Ale to nesta\u010d\u00ed pro realizaci S3. V protokolu S3, krom\u011b samotn\u00e9ho ukl\u00e1d\u00e1n\u00ed objekt\u016f, existuje:<\/p>\n<p><\/p>\n<ul>\n<li>\u00falo\u017ei\u0161t\u011b metadat \u2014 dodate\u010dn\u00fdch vlastnost\u00ed objekt\u016f;<\/li>\n<li>organizace p\u0159\u00edstupu k objekt\u016fm prost\u0159ednictv\u00edm HTTP;<\/li>\n<li>skupinov\u00e1n\u00ed objekt\u016f do kolekc\u00ed \u2014 bucket\u016f;<\/li>\n<li>HTTP-S3 Endpoint. S3 organizuje data do ur\u010dit\u00fdch struktur \u2014 bucket\u016f, z nich\u017e ka\u017ed\u00fd poskytuje vstupn\u00ed bod pro ukl\u00e1d\u00e1n\u00ed soubor\u016f.<\/li>\n<\/ul>\n<p><\/p>\n<p>Pro realizaci t\u00e9to logiky pot\u0159ebujeme samostatnou slu\u017ebu. Tak\u00e9 jsme cht\u011bli ihned p\u0159edpokl\u00e1dat architekturu pro budouc\u00ed r\u016fst slu\u017eby s line\u00e1rn\u00ed \u0161k\u00e1lovatelnost\u00ed.<\/p>\n<p><\/p>\n<h2 id=\"pervye-komponenty\">Prvn\u00ed komponenty<\/h2>\n<p><\/p>\n<p>D\u00e9mon implementuj\u00edc\u00ed S3 API. To je standardn\u00ed S3 API Amazonu, kter\u00e9 podporuje pr\u00e1ci p\u0159es XML pro metadata a umo\u017e\u0148uje p\u0159\u00edm\u00fd p\u0159enos obsahu. Nemuseli jsme nic vym\u00fd\u0161let, v\u0161e je pops\u00e1no a zdokumentov\u00e1no.<\/p>\n<p><\/p>\n<p>P\u0159ed slu\u017ebou jsme tak\u00e9 nasadili Nginx. Ten jsme pou\u017eili pro terminaci SSL, vyva\u017eov\u00e1n\u00ed z\u00e1t\u011b\u017ee, a tak\u00e9 pro ur\u010ditou logiku na Lua (metriky, logov\u00e1n\u00ed a trasov\u00e1n\u00ed).<\/p>\n<p><\/p>\n<p>Pro ukl\u00e1d\u00e1n\u00ed metadat S3 jsme tak\u00e9 zvolili Tarantool. V prvn\u00ed verzi za metadaty d\u00e9mon S3 chodil do t\u00e9to datab\u00e1ze, samotn\u00fd obsah byl uchov\u00e1v\u00e1n ve velk\u00e9m \u00falo\u017ei\u0161ti p\u0159es Streamer.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Architektura S3: 3 roky evoluce Mail.ru Cloud Storage\" src=\"\/wp-content\/uploads\/2020\/08\/03dddec282b5bb41eb63748ccfb6c9a4.jpeg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<em>Nginx + S3 API + metadata<\/em><\/p>\n<p><\/p>\n<h2 id=\"obektnaya-model-hraneniya\">Objektov\u00fd model ukl\u00e1d\u00e1n\u00ed<\/h2>\n<p><\/p>\n<p>Pod\u00edvejme se, jak funguje S3. U\u017eivatel m\u016f\u017ee vytvo\u0159it bucket \u2014 kolekci objekt\u016f. Bucket je adresov\u00e1n n\u00e1zvem hostitele a je subdom\u00e9nou slu\u017eby. V r\u00e1mci bucketu m\u016f\u017ee u\u017eivatel vytv\u00e1\u0159et objekty. Identifik\u00e1torem objektu bude URL. Obsah objektu \u2014 blob, pole bin\u00e1rn\u00edch dat, kter\u00e9 budeme uchov\u00e1vat ve skladu. Objekty maj\u00ed tak\u00e9 atributy: jm\u00e9no \u2014 to je ono URL, ACL (seznam \u0159\u00edzen\u00ed p\u0159\u00edstupu), a dal\u0161\u00ed dodate\u010dn\u00e9 nebo libovoln\u00e9 atributy \u2014 to v\u0161e je uchov\u00e1v\u00e1no v metadatech. <\/p>\n<p><\/p>\n<p>Normalizovan\u00e1 sch\u00e9ma t\u011bchto dat m\u016f\u017ee vypadat takto: jsou projekty, kter\u00fdm pat\u0159\u00ed buckety, kter\u00fdm pat\u0159\u00ed objekty, a objekty mohou b\u00fdt slo\u017een\u00e9. Proto\u017ee jedn\u00edm ze zp\u016fsob\u016f, jak lze objekt nahr\u00e1vat, je po \u010d\u00e1stech, pro nahr\u00e1v\u00e1n\u00ed existuj\u00ed dv\u011b pomocn\u00e9 tabulky: uploads a chunks. Projekty tak\u00e9 maj\u00ed p\u0159ihla\u0161ovac\u00ed \u00fadaje pro p\u0159\u00edstup a billing. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Architektura S3: 3 roky evoluce Mail.ru Cloud Storage\" src=\"\/wp-content\/uploads\/2020\/08\/e2bede45ba4044f658b47a9840fd5224.jpeg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<em>Sch\u00e9ma dat<\/em><\/p>\n<p><\/p>\n<p>Proto\u017ee jsme vytv\u00e1\u0159eli B2B slu\u017ebu s placen\u00fdm p\u0159\u00edstupem, byla v t\u00e9to sch\u00e9m\u011b pot\u0159ebn\u00e1 billing.<br \/>\nSlu\u017ebu billing jsme tak\u00e9 realizovali na Tarantool. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Architektura S3: 3 roky evoluce Mail.ru Cloud Storage\" src=\"\/wp-content\/uploads\/2020\/08\/d5b1a63e2273e933c95deecdb054bfe6.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<h2 id=\"dorabotki-s3-hranilischa-shagi-k-prodakshenu\">Dokon\u010den\u00ed S3 \u00falo\u017ei\u0161t\u011b: kroky k produkci<\/h2>\n<p><\/p>\n<p>Ji\u017e jsme vytvo\u0159ili funk\u010dn\u00ed model, kter\u00fd lze pou\u017e\u00edt: objekty a metadata byla uchov\u00e1v\u00e1na, ale pro p\u0159echod do produkce chyb\u011blo n\u011bkolik aspekt\u016f.<\/p>\n<p><\/p>\n<p>Za prv\u00e9, syst\u00e9m limit\u016f p\u0159enosu. Pokud spust\u00edme slu\u017ebu bez n\u011bj, mohli bychom p\u0159i \u0161pi\u010dkov\u00e9 z\u00e1t\u011b\u017ei nep\u0159edv\u00eddateln\u011b p\u0159et\u00ed\u017eit n\u011bjakou \u010d\u00e1st syst\u00e9mu. Limit p\u0159enosu by m\u011bl fungovat takto: jak\u00fdkoli S3 po\u017eadavek p\u0159ich\u00e1z\u00ed na konkr\u00e9tn\u00ed hostitele, tento hostitel je identifik\u00e1torem bucketu a bucket pat\u0159\u00ed n\u011bjak\u00e9mu klientovi. Mus\u00edme ur\u010dit n\u011bjakou funkci pro bucket, kter\u00e1 by umo\u017enila vypo\u010d\u00edtat limit p\u0159enosu. <\/p>\n<p><\/p>\n<p>Krom\u011b toho mus\u00ed b\u00fdt syst\u00e9m limit\u016f p\u0159enosu dostate\u010dn\u011b v\u00fdkonn\u00fd, aby zvl\u00e1dl zat\u00ed\u017een\u00ed, kter\u00e9 p\u0159ich\u00e1z\u00ed na S3.<\/p>\n<p><\/p>\n<p>Zde jsme op\u011bt pou\u017eili Tarantool. Limity p\u0159enosu jsou klastr 21 instanc\u00ed, kter\u00e9 jsou rozd\u011bleny do skupin, rozlo\u017eeny na t\u0159ech fyzick\u00fdch uzlech a spojeny do velk\u00e9ho topologick\u00e9ho klastru. Konfigura\u010dn\u00ed zm\u011bny se automaticky \u0161\u00ed\u0159\u00ed: nastavuj\u00ed se limity p\u0159enosu, defaulty a konfigurace. Ka\u017ed\u00fd bucket je obsluhov\u00e1n p\u0159esn\u011b jednou instanc\u00ed. Kdy\u017e na konkr\u00e9tn\u00ed bucket doraz\u00ed po\u017eadavek, vypo\u010d\u00edt\u00e1 se instance odpov\u00eddaj\u00edc\u00ed tomuto bucketu. V r\u00e1mci toho uzlu se po\u010d\u00edt\u00e1 aktu\u00e1ln\u00ed p\u0159enosov\u00fd rate podle algoritmu podobn\u00e9ho Token Bucket. D\u00e1le syst\u00e9m limit\u016f p\u0159enosu, na z\u00e1klad\u011b aktu\u00e1ln\u00edch ukazatel\u016f z\u00e1t\u011b\u017ee a vlastnost\u00ed nastaven\u00fdch pro konkr\u00e9tn\u00ed bucket, \u0159\u00edk\u00e1, zda lze po\u017eadavek prov\u00e9st nebo ne. Kontrola limit\u016f se prov\u00e1d\u00ed v nejran\u011bj\u0161\u00ed f\u00e1zi vykon\u00e1n\u00ed S3 po\u017eadavku, chr\u00e1n\u00ed v\u0161echny ostatn\u00ed prvky syst\u00e9mu p\u0159ed nadm\u011brn\u00fdm zat\u00ed\u017een\u00edm. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Architektura S3: 3 roky evoluce Mail.ru Cloud Storage\" src=\"\/wp-content\/uploads\/2020\/08\/1b91e5d1fe067c859ba9b36d58827579.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Tak\u00e9 je p\u0159i z\u00e1t\u011b\u017ei docela obt\u00ed\u017en\u00e9 se obej\u00edt bez cache. V S3 se p\u0159edpokl\u00e1d\u00e1 opakovan\u00e9 p\u0159\u00edstup k t\u011bm sam\u00fdm objekt\u016fm, tedy jde o hork\u00e9 \u00falo\u017ei\u0161t\u011b. V b\u011b\u017en\u00e9m p\u0159\u00edpad\u011b je p\u0159\u00edstup k jednotkov\u00e9mu souboru obsluhov\u00e1n cel\u00fdm \u0159et\u011bzcem: Streamer, FileDB, PairDB, Storage. Ale p\u0159i opakovan\u00e9m p\u0159\u00edstupu k souboru optimalizujeme p\u0159\u00edstup k tomuto obsahu pomoc\u00ed lok\u00e1ln\u00ed cache.<\/p>\n<p><\/p>\n<p>Cache je v\u00edcestup\u0148ov\u00e1 a je realizov\u00e1na pomoc\u00ed nginx, lok\u00e1ln\u00edch, SSD a RAM disk\u016f. Zde jsme nepou\u017eili Tarantool, proto\u017ee je pohodln\u011bj\u0161\u00ed p\u0159ed\u00e1vat objekty z file syst\u00e9mu, tak m\u016f\u017eeme ud\u011blat tiering cache. Nav\u00edc m\u00e1me velk\u00e9 objekty s maxim\u00e1ln\u00ed velikost\u00ed 32 gigabajt\u016f, a v Tarantool lze cache pouze mal\u00e9 objekty.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Architektura S3: 3 roky evoluce Mail.ru Cloud Storage\" src=\"\/wp-content\/uploads\/2020\/08\/faa02bd52a1d70642589d775f22c5e75.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Tento prvn\u00ed syst\u00e9m, se kter\u00fdm jsme za\u010dali, m\u011bl ur\u010ditou vypo\u010dtenou kapacitu, kter\u00e1 sta\u010dila pro prozkoum\u00e1n\u00ed a pochopen\u00ed produktu a toho, \u017ee bude fungovat. <\/p>\n<p><\/p>\n<h2 id=\"dorabotki-boevoy-sistemy-feylover-i-masshtabirovanie\">\u00dapravy bojov\u00e9ho syst\u00e9mu: failover a \u0161k\u00e1lovatelnost<\/h2>\n<p><\/p>\n<p>Syst\u00e9m byl ji\u017e v provozu, ale p\u0159i spu\u0161t\u011bn\u00ed jsme n\u011bco p\u0159ehl\u00e9dli \u2014 bylo t\u0159eba p\u0159idat failover a \u0161k\u00e1lovatelnost. <\/p>\n<p><\/p>\n<p>N\u00e1\u0161 S3 d\u00e9mon z\u00edsk\u00e1val metadata p\u0159es protokol Tarantool. Na m\u00edsto p\u016fvodn\u00ed datab\u00e1ze jsme nasadili Tarantool, kter\u00fd fungoval jako proxy-router pro dotazy na metadata. Z pohledu aplikace, kter\u00e1 implementovala API, se nic nezm\u011bnilo \u2014 nad\u00e1le komunikovala s datab\u00e1z\u00ed p\u0159es protokol Tarantool, ale router m\u011bl schopnost aktivn\u00edho failoveru. To znamenalo, \u017ee jsme mohli kontrolovat dostupnost uzlu, vydr\u017eet pauzu b\u011bhem p\u0159ep\u00edn\u00e1n\u00ed a selh\u00e1n\u00ed atd. Aplikaci jsme p\u0159itom nemodifikovali. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Architektura S3: 3 roky evoluce Mail.ru Cloud Storage\" src=\"\/wp-content\/uploads\/2020\/08\/978ff8a458f8d48b98672121cf6eca5d.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<h2 id=\"podrobnee-o-tom-kak-my-realizovali-shardirovanie\">V\u00edce informac\u00ed o tom, jak jsme implementovali \u0161ardov\u00e1n\u00ed<\/h2>\n<p><\/p>\n<p>Dal\u0161\u00ed ot\u00e1zka, kterou jsme museli \u0159e\u0161it, byla \u0161ardov\u00e1n\u00ed. Syst\u00e9m rostl, zvy\u0161ovalo se mno\u017estv\u00ed objekt\u016f a bylo t\u0159eba zajistit mo\u017enosti pro dal\u0161\u00ed r\u016fst.<\/p>\n<p><\/p>\n<p>Vra\u0165me se ke sch\u00e9matu dat: jsou projekty, jsou buckety, kredity a billing. To jsou objekty, kter\u00e9 s vysokou pravd\u011bpodobnost\u00ed v dohledn\u00e9 dob\u011b nep\u0159ekro\u010d\u00ed hranice jednoho instance ani co do objemu, ani co do dotaz\u016f. To znamen\u00e1, \u017ee nem\u00e1 smysl je shardovat, a proto jsme je dali do samostatn\u00e9ho instance, kter\u00fd z\u016fstane ne\u0161ardovan\u00fd. To umo\u017e\u0148uje konzistentn\u011bj\u0161\u00ed spr\u00e1vu projekt\u016f a bucket\u016f, proto\u017ee existuje jedna ne\u0161ardovan\u00e1 jednotka. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Architektura S3: 3 roky evoluce Mail.ru Cloud Storage\" src=\"\/wp-content\/uploads\/2020\/08\/c5a1c1a875e756e3dceb5f885fbda2c8.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Ve sch\u00e9matu jsou tak\u00e9 objekty, kter\u00e9 rostou line\u00e1rn\u011b \u2014 zpo\u010d\u00e1tku jich bylo stovky tis\u00edc, nyn\u00ed se jejich po\u010det m\u011b\u0159\u00ed v n\u011bkolika miliard\u00e1ch. Takov\u00e9 objekty a jejich \u010d\u00e1sti bylo t\u0159eba p\u0159en\u00e9st do shardovan\u00e9ho clustera. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Architektura S3: 3 roky evoluce Mail.ru Cloud Storage\" src=\"\/wp-content\/uploads\/2020\/08\/00bb418cc61ad2e3e8797caf5d1ab202.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Sch\u00e9ma jsme rozd\u011blili, ale objekty mus\u00ed pracovat s buckety: objekt v\u017edy pat\u0159\u00ed konkr\u00e9tn\u00edmu bucketu, nav\u00edc na bucketu funguje ACL. Proto pro ka\u017ed\u00fd shard s objekty uchov\u00e1v\u00e1me st\u00ednovou kopii ka\u017ed\u00e9ho bucketu. Krom\u011b toho, b\u011bhem modifikace objekt\u016f a prov\u00e1d\u011bn\u00ed dotaz\u016f, je t\u0159eba po\u010d\u00edtat objem pro \u00fa\u010dely billing, proto na ka\u017ed\u00e9m shardu existuj\u00ed po\u010d\u00edtadla pro billing.<\/p>\n<p><\/p>\n<p>Tak\u00e9 jsme p\u0159idali n\u011bkolik dal\u0161\u00edch tabulek a komponent:<\/p>\n<p><\/p>\n<ul>\n<li>ko\u0161 pro eliminaci star\u00fdch projekt\u016f, kter\u00e9 jsou maz\u00e1ny nebo zmra\u017eeny;<\/li>\n<li>fronta pro pozad\u00ed \u00faloh, co\u017e znamen\u00e1, \u017ee hlavn\u00ed \u00falo\u017ei\u0161t\u011b m\u016f\u017ee vykon\u00e1vat pozad\u00ed \u00fakoly, kter\u00e9 je t\u0159eba prov\u00e9st v clusteru;<\/li>\n<li>podpora lifecycle \u2014 mechanizmus, kter\u00fd umo\u017e\u0148uje pracovat s objekty a spravovat jejich \u017eivotn\u00ed cyklus.<\/li>\n<\/ul>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Architektura S3: 3 roky evoluce Mail.ru Cloud Storage\" src=\"\/wp-content\/uploads\/2020\/08\/933e98dc559eb8e6bac14003215a6836.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Jeliko\u017e jsme \u010d\u00e1st dat p\u0159esunuli na shardy, bylo pot\u0159eba sharding proxy. Mohli jsme znovu vyu\u017e\u00edt router pro tuto roli, ale samostatn\u00e1 sharding proxy, kter\u00e1 se zab\u00fdv\u00e1 pouze shardov\u00e1n\u00edm dat, umo\u017e\u0148uje routeru z\u00edsk\u00e1vat data v celku, ani\u017e by se staral o sharding.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Architektura S3: 3 roky evoluce Mail.ru Cloud Storage\" src=\"\/wp-content\/uploads\/2020\/08\/4b181ed4629180e15558f25b1e4d5718.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Samostatn\u011b vysv\u011btl\u00edm, pro\u010d jsme nevzali hotov\u00e9 \u0159e\u0161en\u00ed, ale cht\u011bli jsme vytvo\u0159it vlastn\u00ed funkci shardingu. <\/p>\n<p><\/p>\n<p>Pod\u00edvejme se, jak je to uspo\u0159\u00e1d\u00e1no. M\u00e1me 256 dostupn\u00fdch shard\u016f. Pro ka\u017ed\u00fd bucket vymezujeme rozsah pomoc\u00ed ur\u010dit\u00e9 konzistentn\u00ed funkce. Je to jednoduch\u00e9 \u2014 stejn\u011b jako ur\u010dujete pomoc\u00ed konzistentn\u00ed funkce p\u0159\u00edslu\u0161nost k jednomu shardu, ur\u010dujete v\u00fdchoz\u00ed shard a vymezujete rozsah:<\/p>\n<p><\/p>\n<p><code>f(bucket, shards) = subset<\/code><\/p>\n<p><\/p>\n<p>To znamen\u00e1, \u017ee pokud vezmete bucket, m\u016f\u017eete \u0159\u00edci, \u017ee on a jeho data budou v\u017edy le\u017eet v konkr\u00e9tn\u00edm podmno\u017estv\u00ed v\u0161ech shard\u016f. To sni\u017euje vliv n\u011bkter\u00fdch bucket\u016f na jin\u00e9 a zjednodu\u0161uje pr\u00e1ci dotaz\u016f map-reduce, kdy je t\u0159eba nap\u0159\u00edklad prov\u00e9st v\u00fdpis objekt\u016f bucketu. K tomu je t\u0159eba dotazovat v\u0161echny shardy, kde jsou tyto objekty ulo\u017eeny. Kdyby objekty byly na v\u0161ech shardech, jak\u00fdkoli v\u00fdpis by zas\u00e1hl cel\u00fd syst\u00e9m, ale tady zas\u00e1hne pouze konkr\u00e9tn\u00ed podmno\u017estv\u00ed.<\/p>\n<p><\/p>\n<p>D\u00e1le \u2014 ka\u017ed\u00fd objekt pat\u0159\u00ed konkr\u00e9tn\u00edmu bucketu, tak\u017ee kdy\u017e se obrac\u00edme na objekt, obrac\u00edme se na n\u011bj podle jm\u00e9na v konkr\u00e9tn\u00edm bucketu. To znamen\u00e1, \u017ee m\u016f\u017eeme definovat funkci pro objekt, nikoli z cel\u00e9ho dostupn\u00e9ho rozsahu shard\u016f, ale pouze z podmno\u017estv\u00ed jeho bucketu:<\/p>\n<p><\/p>\n<p><code>f(object, subset) = shard<\/code><\/p>\n<p><\/p>\n<p>Vezmeme konkr\u00e9tn\u00ed objekt, jako argumenty funkce p\u0159ed\u00e1v\u00e1me ne v\u0161echny shardy, ale podmno\u017einu jeho bucketu \u2014 a dost\u00e1v\u00e1me konkr\u00e9tn\u00ed shard. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Architektura S3: 3 roky evoluce Mail.ru Cloud Storage\" src=\"\/wp-content\/uploads\/2020\/08\/10480ad2430c228c511ff3cdc5071596.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Tak\u017ee, sharding je realizov\u00e1n, m\u00e1me sharding proxy. D\u00e1le zb\u00fdv\u00e1, aby router a datab\u00e1ze s metadaty chodily do sharding proxy. Nap\u0159\u00edklad pro vytv\u00e1\u0159en\u00ed objekt\u016f st\u00ednov\u00fdch kopi\u00ed \u2014 kdy\u017e vytv\u00e1\u0159\u00edme bucket, hlavn\u00ed \u00falo\u017ei\u0161t\u011b mus\u00ed vytvo\u0159it z\u00e1stupce tohoto bucketu na v\u0161ech shartech, kde by m\u011bl b\u00fdt p\u0159\u00edtomen.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Architektura S3: 3 roky evoluce Mail.ru Cloud Storage\" src=\"\/wp-content\/uploads\/2020\/08\/bb34d1f747903a3c2f08f582e4424787.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<h2 id=\"kak-my-realizovali-resharding\">Jak jsme implementovali resharding<\/h2>\n<p><\/p>\n<p>Nejv\u011bt\u0161\u00edm probl\u00e9mem shardov\u00e1n\u00ed je reshardov\u00e1n\u00ed. Bylo pro n\u00e1s d\u016fle\u017eit\u00e9 prov\u00e9st to bez v\u00fdpadku, proto\u017ee syst\u00e9m byl ji\u017e v provozu. Uk\u00e1\u017eu, jak jsme tento probl\u00e9m vy\u0159e\u0161ili na p\u0159\u00edkladu podobn\u00e9ho \u00fakolu s \u017eivou migrac\u00ed dat z jednoho projektu do druh\u00e9ho.<\/p>\n<p><\/p>\n<p>N\u00ed\u017ee je sch\u00e9ma na\u0161eho klastru, kter\u00e9 vzniklo po zaveden\u00ed shardov\u00e1n\u00ed. M\u00e1me tu nginx, S3 API, router, prim\u00e1rn\u00ed datab\u00e1zi s projekty, sharduj\u00edc\u00ed proxy a samotn\u00e9 shardy.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Architektura S3: 3 roky evoluce Mail.ru Cloud Storage\" src=\"\/wp-content\/uploads\/2020\/08\/0e6bd16c7fc47437b3b7d96b6d9809a4.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>V p\u0159edchoz\u00edm textu jsem opomn\u011bl zm\u00ednit, \u017ee v ur\u010dit\u00e9 f\u00e1zi projektu byla produktov\u00e1 \u00faloha: \"Spustit dal\u0161\u00ed \u00falo\u017ei\u0161t\u011b, Icebox, \u2014 jako Hotbox, jen pro studen\u00e1 data.\" V podstat\u011b to bylo stejn\u00e9 \u00falo\u017ei\u0161t\u011b, ale na jin\u00fdch URL a bez cache.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Architektura S3: 3 roky evoluce Mail.ru Cloud Storage\" src=\"\/wp-content\/uploads\/2020\/08\/9ce9085e1d03e9a99b6b8ccc0717c653.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Icebox byl vyu\u017e\u00edv\u00e1n m\u00e9n\u011b ne\u017e Hotbox, proto se dlouho obe\u0161el bez jak\u00e9hokoli shardov\u00e1n\u00ed. Nakonec jsme se rozhodli ho zru\u0161it a slou\u010dit Hotbox a Icebox do jedn\u00e9 slu\u017eby, jednodu\u0161e rozd\u011blen\u00edm t\u0159\u00edd \u00falo\u017ei\u0161t\u011b. <\/p>\n<p><\/p>\n<p>Bucket v \u00falo\u017ei\u0161t\u00edch se nep\u0159ekr\u00fdvaly, bylo je snadn\u00e9 slou\u010dit a p\u0159esunout, ale klienti pou\u017e\u00edvali ob\u011b \u00falo\u017ei\u0161t\u011b, co\u017e znamenalo, \u017ee bylo t\u0159eba vy\u0159e\u0161it probl\u00e9m s nedostatkem v\u00fdpadku. Nebylo mo\u017en\u00e9 jednodu\u0161e vypnout a zkop\u00edrovat. Migraci jsme realizovali ve v\u00edce kroc\u00edch.<\/p>\n<p><\/p>\n<p>Nejprve jsme synchronizovali prim\u00e1rn\u00ed \u00falo\u017ei\u0161t\u011b. M\u011bli jsme Tarantool a p\u0159i vytv\u00e1\u0159en\u00ed objektu jsme mohli ud\u011blat n\u00e1sleduj\u00edc\u00ed: <\/p>\n<p><\/p>\n<ul>\n<li>do datab\u00e1ze p\u0159ich\u00e1z\u00ed po\u017eadavek na vytvo\u0159en\u00ed bucketu, nap\u0159\u00edklad v Hotbox;<\/li>\n<li>Tarantool kontroluje v jin\u00e9 datab\u00e1zi (v tomto p\u0159\u00edpad\u011b v Icebox), \u017ee takov\u00fd bucket neexistuje;<\/li>\n<li>pokud bucket existuje, datab\u00e1ze \u0159\u00edk\u00e1, \u017ee ho nen\u00ed mo\u017en\u00e9 vytvo\u0159it, a on byl synchronizov\u00e1n jako existuj\u00edc\u00ed.<br \/>\n<img decoding=\"async\" alt=\"Architektura S3: 3 roky evoluce Mail.ru Cloud Storage\" src=\"\/wp-content\/uploads\/2020\/08\/6694c9804f09942416c1717b455b9fd3.jpeg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<em>Synchronizace bucket\u016f<\/em><\/li>\n<\/ul>\n<p><\/p>\n<p>V tom \u00falo\u017ei\u0161ti, kter\u00e9 m\u011blo p\u0159ij\u00edmat v\u0161echna data, jsme pro projekty a buckety zaviedli znak, kter\u00fd \u0159\u00edk\u00e1, kde se tento objekt nach\u00e1z\u00ed. Mohl b\u00fdt ulo\u017een lok\u00e1ln\u011b, tedy v Hotbox, v Icebox \u2014 pak v nov\u00e9m \u00falo\u017ei\u0161ti nen\u00ed \u017e\u00e1dn\u00fd z jeho dat, nebo mohl b\u00fdt ve stavu migrace.<\/p>\n<p><\/p>\n<p>Pokud m\u011bl projekt nebo bucket znak Migrating, pak b\u011bhem migrace se po\u017eadavek nejprve vykon\u00e1val v nov\u00e9m \u00falo\u017ei\u0161ti, kde m\u011bly b\u00fdt data, a pokud tam nebyla, byly po\u017eadavky chybov\u00e1ny na alternativn\u00ed \u00falo\u017ei\u0161t\u011b.<\/p>\n<p><\/p>\n<p>D\u00e1le jsme p\u0159epnuli provoz. Proto\u017ee API m\u016f\u017ee obsluhovat jak po\u017eadavky Icebox, tak po\u017eadavky Hotbox, dok\u00e1zali jsme p\u0159epnout provoz bez v\u00fdpadku jednodu\u0161e p\u0159esunem hostitel\u016f a p\u0159id\u00e1n\u00edm odpov\u00eddaj\u00edc\u00edch z\u00e1znam\u016f do Nginxu. <\/p>\n<p><\/p>\n<p>Po p\u0159esm\u011brov\u00e1n\u00ed provozu jsme mohli odstranit Nginx a API od Icebox.<br \/>\nPak jsme odstranili Icebox Nginx a S3 API \u2014 a v\u0161e za\u010dalo fungovat:<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Architektura S3: 3 roky evoluce Mail.ru Cloud Storage\" src=\"\/wp-content\/uploads\/2020\/08\/62d9f960fc1f11b875a8f8a5aa8e8152.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>D\u00e1le jsme spustili proces migrace na pozad\u00ed, kter\u00fd pracuje uvnit\u0159 datab\u00e1ze \u2014 proch\u00e1z\u00ed jednotliv\u00e9 projekty a jejich bucket a nastavuje pro n\u011b ozna\u010den\u00ed Migrating, p\u0159en\u00e1\u0161\u00ed data a po dokon\u010den\u00ed p\u0159eveden\u00ed nastavuje ozna\u010den\u00ed Local.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Architektura S3: 3 roky evoluce Mail.ru Cloud Storage\" src=\"\/wp-content\/uploads\/2020\/08\/1a3129136f7ccacfdcff260f10cb0605.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Po p\u0159enesen\u00ed dat ji\u017e nepot\u0159ebujeme star\u00e9 \u00falo\u017ei\u0161t\u011b, tak\u017ee odstran\u00edme zb\u00fdvaj\u00edc\u00ed \u010d\u00e1sti star\u00e9ho syst\u00e9mu a tak\u00e9 vyjmeme podporu statusu migrace z k\u00f3du.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Architektura S3: 3 roky evoluce Mail.ru Cloud Storage\" src=\"\/wp-content\/uploads\/2020\/08\/13b6f11add40c21a3b9afc05dd0dfea9.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Podobn\u011b bylo realizov\u00e1no tak\u00e9 resharding ze star\u00e9ho \u00falo\u017ei\u0161t\u011b do shardovan\u00e9ho:<\/p>\n<p><\/p>\n<ul>\n<li>Ozmarkedli jsme v\u0161echny buckety jako <code>Non-sharded<\/code>. V\u0161echny \u017e\u00e1dosti k nim sm\u011b\u0159ovaly do p\u016fvodn\u00edho, ne\u0161ardovan\u00e9ho \u00falo\u017ei\u0161t\u011b.<\/li>\n<li>Nov\u00e9 buckety byly okam\u017eit\u011b vytv\u00e1\u0159eny v statusu <code>Sharded<\/code>.<\/li>\n<li>Po jednom jsme brali buckety, nastavovali status <code>Migrating<\/code> a p\u0159en\u00e1\u0161eli data.<\/li>\n<\/ul>\n<p><\/p>\n<p>\u017d\u00e1dosti byly obsluhov\u00e1ny podle principu:<\/p>\n<p><\/p>\n<ul>\n<li>\u010cteme nejprve v nov\u00e9m, pak ve star\u00e9m.<\/li>\n<li>Vytv\u00e1\u0159\u00edme pouze v nov\u00e9m.<\/li>\n<li>Aktualizujeme dvouf\u00e1zov\u011b: pokud v nov\u00e9m nen\u00ed, p\u0159eneseme ze star\u00e9ho do nov\u00e9ho, pak aktualizujeme.<\/li>\n<\/ul>\n<p><\/p>\n<h2 id=\"rabota-s-ssl-sertifikatami\">Pr\u00e1ce s SSL certifik\u00e1ty<\/h2>\n<p><\/p>\n<blockquote><p>Na front-endu pou\u017e\u00edv\u00e1me Nginx. V na\u0161em p\u0159\u00edpad\u011b to nen\u00ed b\u011b\u017en\u00fd Nginx, ale OpenResty, Nginx s podporou LuaJIT.<\/p><\/blockquote>\n<p>Dal\u0161\u00ed \u010d\u00e1st syst\u00e9mu \u2014 pr\u00e1ce s SSL certifik\u00e1ty. V S3 \u00falo\u017ei\u0161ti m\u016f\u017eete nastavit vlastn\u00ed dom\u00e9nu pro p\u0159\u00edstup k ur\u010dit\u00e9mu bucketu, jednodu\u0161e pomoc\u00ed <code>CNAME<\/code>. Ale bez HTTPS to dnes u\u017e nejde: vlastn\u00ed dom\u00e9na p\u0159edpokl\u00e1d\u00e1 vlastn\u00ed SSL certifik\u00e1t. <\/p>\n<p><\/p>\n<p>Jak jsem u\u017e \u0159ekl, za vyva\u017eov\u00e1n\u00ed a terminaci SSL odpov\u00edd\u00e1 Nginx. V na\u0161em p\u0159\u00edpad\u011b to nen\u00ed b\u011b\u017en\u00fd Nginx, ale OpenResty, Nginx s podporou LuaJIT.<\/p>\n<p><\/p>\n<p>To n\u00e1m umo\u017enilo relativn\u011b snadno nau\u010dit n\u00e1\u0161 Nginx p\u0159ed\u00e1vat libovoln\u00e9 certifik\u00e1ty. Bylo t\u0159eba certifik\u00e1ty vyd\u00e1vat dynamicky (bez nutnosti je zapisovat do konfigura\u010dn\u00edho souboru). Vyu\u017eili jsme roz\u0161\u00ed\u0159en\u00ed <code>ssl_certificate_by_lua<\/code>, kter\u00e9 umo\u017e\u0148uje p\u0159e\u010d\u00edst certifik\u00e1t z libovoln\u00e9ho zdroje p\u0159\u00edmo b\u011bhem TLS handshake. Jako \u00falo\u017ei\u0161t\u011b certifik\u00e1t\u016f jsme rovn\u011b\u017e pou\u017eili Tarantool: to umo\u017e\u0148uje spravovat certifik\u00e1ty zvn\u011bj\u0161ku a zaji\u0161\u0165uje velmi rychl\u00e9 dod\u00e1n\u00ed.<\/p>\n<p><\/p>\n<p>Tak\u00e9 byl implementov\u00e1n samostatn\u00fd d\u00e9mon, jeho\u017e \u00fakolem je pravideln\u011b aktualizovat certifik\u00e1ty, kter\u00e9 byly vyd\u00e1ny pomoc\u00ed Let's Encrypt.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Architektura S3: 3 roky evoluce Mail.ru Cloud Storage\" src=\"\/wp-content\/uploads\/2020\/08\/8d61c1cb3084d9eb4eb8faa745e212db.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<h2 id=\"chto-by-ya-sohranil-a-chto-sdelal-po-drugomu-esli-razrabatyvat-hranilische-zanovo\">Co bych zachoval a co ud\u011blal jinak, kdybych vyv\u00edjel \u00falo\u017ei\u0161t\u011b od za\u010d\u00e1tku<\/h2>\n<p><\/p>\n<h3 id=\"chto-nuzhno-bylo-ispolzovat-s-samogo-nachala\">Co bylo t\u0159eba pou\u017e\u00edt hned na za\u010d\u00e1tku<\/h3>\n<p><\/p>\n<p><strong>\u0160ardov\u00e1n\u00ed hned<\/strong>. P\u0159\u00edli\u0161 mnoho probl\u00e9m\u016f zp\u016fsobilo reshardov\u00e1n\u00ed. Je to snadn\u00e9 ud\u011blat, ale pokud za\u010d\u00edn\u00e1te projekty, kter\u00e9 je t\u0159eba \u0161k\u00e1lovat, je lep\u0161\u00ed hned vz\u00edt shardovan\u00fd cluster, i kdyby na minim\u00e1ln\u00edm po\u010dtu uzl\u016f. Realizace \u0161ardov\u00e1n\u00ed na za\u010d\u00e1tku je t\u00e9m\u011b\u0159 zdarma ve srovn\u00e1n\u00ed s zav\u00e1d\u011bn\u00edm \u0161ardov\u00e1n\u00ed do funk\u010dn\u00edho syst\u00e9mu.<\/p>\n<p><\/p>\n<p><strong>Pr\u00e1ce s Tarantool p\u0159es vyrovn\u00e1va\u010de z\u00e1t\u011b\u017ee<\/strong>. Nyn\u00ed v\u0161echny nov\u00e9 datab\u00e1ze okam\u017eit\u011b p\u0159ipojujeme do pr\u00e1ce p\u0159es vyrovn\u00e1va\u010de z\u00e1t\u011b\u017ee. To umo\u017e\u0148uje roz\u0161i\u0159ovat funkcionalitu a dosahovat vy\u0161\u0161\u00ed odolnosti proti v\u00fdpadk\u016fm. <\/p>\n<p><\/p>\n<p><strong>Automatick\u00e9 p\u0159ep\u00edn\u00e1n\u00ed<\/strong>. Nainstaloval bych v\u0161echny n\u00e1stroje pot\u0159ebn\u00e9 pro automatick\u00e9 p\u0159ep\u00edn\u00e1n\u00ed, proto\u017ee prvn\u00ed ne\u00fasp\u011bchy po spu\u0161t\u011bn\u00ed byly spojen\u00e9 s jeho absenc\u00ed. Po zku\u0161enostech s S3 byly v\u0161echny n\u00e1sleduj\u00edc\u00ed produkty spu\u0161t\u011bny s ohledem na to.<\/p>\n<p><\/p>\n<p><strong>Funkce S3 \"Verzov\u00e1n\u00ed\"<\/strong>. P\u016fvodn\u011b se zd\u00e1lo, \u017ee to nen\u00ed p\u0159\u00edli\u0161 \u017e\u00e1dan\u00e1 funk\u010dnost. Integrovat tuto mo\u017enost do architektury funguj\u00edc\u00edho syst\u00e9mu je velmi obt\u00ed\u017en\u00e9.<\/p>\n<p><\/p>\n<p><strong>Samostatn\u00e9 fakturace<\/strong>. Zp\u016fsob, jak\u00fdm jsme integrovali fakturaci do na\u0161eho syst\u00e9mu, se na za\u010d\u00e1tku osv\u011bd\u010dil, ale pozd\u011bji se stal p\u0159ek\u00e1\u017ekou, bylo by lep\u0161\u00ed m\u00edt ji jako naprosto samostatnou slu\u017ebu.<\/p>\n<p><\/p>\n<h3 id=\"chto-bylo-udachnym-resheniem\">Co bylo \u00fasp\u011b\u0161n\u00fdm rozhodnut\u00edm<\/h3>\n<p><\/p>\n<p><strong>Model dat<\/strong>. Historie uk\u00e1zala, \u017ee v r\u00e1mci rozvoje slu\u017eby se docela p\u0159esn\u011b shodujeme s modelem dat Amazonu, tak\u017ee m\u016f\u017eeme implementovat ty funkce, kter\u00e9 tam jsou.<\/p>\n<p><\/p>\n<p><strong>Sch\u00e9ma \u0161ardov\u00e1n\u00ed<\/strong>. Podporoval bych stejn\u00e9 rozsahov\u00e9 \u0161ardov\u00e1n\u00ed podle bucket\u016f, proto\u017ee to umo\u017e\u0148uje dob\u0159e rozd\u011blovat dotazy z r\u016fzn\u00fdch bucket\u016f po velk\u00e9m clusteru.<\/p>\n<p><\/p>\n<p><strong>Pou\u017eit\u00ed Tarantool<\/strong>. Tarantool velmi pomohl p\u0159i rozvoji slu\u017eby a jej\u00ed modifikaci, snadno jsme pracovali s daty, transformovali a \u0161ardovali \u00falo\u017ei\u0161t\u011b, ani\u017e bychom pot\u0159ebovali p\u0159ech\u00e1zet na vrstvu aplikace. <\/p>\n<p><\/p>\n<blockquote><p>Tento refer\u00e1t byl poprv\u00e9 p\u0159edstaven na <noindex><a rel=\"nofollow\" href=\"https:\/\/corp.mail.ru\/ru\/press\/events\/databases-2\/\">@Databases Meetup<\/a><\/noindex> by Mail.ru Cloud Solutions&amp;Tarantool. Pod\u00edvejte se na <noindex><a rel=\"nofollow\" href=\"https:\/\/www.youtube.com\/watch?v=jM4hL2u6JM0&amp;list=PLQzTaxmOHjntyNRPWhaWHqlHp8UN_wgQ3\">video<\/a><\/noindex> dal\u0161\u00ed vystoupen\u00ed a p\u0159ihlaste se k ozn\u00e1men\u00edm akc\u00ed na Telegramu <noindex><a rel=\"nofollow\" href=\"https:\/\/t.me\/k8s_mail\">Kolem Kubernetes v Mail.ru Group<\/a><\/noindex>.<\/p><\/blockquote>\n<p>Tak\u00e9 m\u016f\u017eete vid\u011bt moji starou p\u0159edn\u00e1\u0161ku o S3 nebo si p\u0159e\u010d\u00edst \u010dl\u00e1nek m\u00e9ho kolegy o blokov\u00e9m \u00falo\u017ei\u0161ti.<\/p>\n<p><\/p>\n<ul>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/www.youtube.com\/watch?v=O0iIADHgBVc\">Reverzn\u00ed in\u017een\u00fdrstv\u00ed architektury Amazon S3 a jak vypadalo S3-\u00falo\u017ei\u0161t\u011b MCS p\u0159ed 3 lety<\/a><\/noindex>.<\/li>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/mailru\/blog\/472694\/\">V\u00edce ne\u017e Ceph: blokov\u00e9 \u00falo\u017ei\u0161t\u011b cloudu MCS<\/a><\/noindex>.<\/li>\n<\/ul>\n<p>Zdroj: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/mailru\/blog\/513356\/\">habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>Storage Corridor by St-Pete \u0412\u0441\u0435\u043c \u043f\u0440\u0438\u0432\u0435\u0442! \u042f Mons Anderson, \u0430\u0440\u0445\u0438\u0442\u0435\u043a\u0442\u043e\u0440 \u043f\u043b\u0430\u0442\u0444\u043e\u0440\u043c\u044b Mail.ru Cloud Solutions, \u0440\u0430\u0441\u0441\u043a\u0430\u0436\u0443, \u043a\u0430\u043a \u043c\u044b \u043f\u043e\u0441\u0442\u0440\u043e\u0438\u043b\u0438 \u043d\u0430\u0448\u0435 S3-\u0445\u0440\u0430\u043d\u0438\u043b\u0438\u0449\u0435, \u043a\u0430\u043a \u043e\u043d\u043e \u0440\u0430\u0431\u043e\u0442\u0430\u0435\u0442, \u043a\u0430\u043a\u0438\u0435 \u0440\u0435\u0448\u0435\u043d\u0438\u044f \u043e\u043a\u0430\u0437\u0430\u043b\u0438\u0441\u044c \u0443\u0434\u0430\u0447\u043d\u044b\u043c\u0438, \u0430 \u043a\u0430\u043a\u0438\u0435 \u0441\u0442\u043e\u0438\u043b\u043e \u0438\u0437\u043c\u0435\u043d\u0438\u0442\u044c, \u0435\u0441\u043b\u0438 \u0431\u044b \u043c\u044b \u043d\u0430\u0447\u0430\u043b\u0438 \u0442\u0430\u043a\u043e\u0439 \u0436\u0435 \u043f\u0440\u043e\u0435\u043a\u0442 \u0441 \u043d\u0443\u043b\u044f \u0441\u0435\u0439\u0447\u0430\u0441. \u0421\u0442\u0430\u0442\u044c\u044f \u043f\u043e\u0434\u0433\u043e\u0442\u043e\u0432\u043b\u0435\u043d\u0430 \u043d\u0430 \u043e\u0441\u043d\u043e\u0432\u0435 \u0434\u043e\u043a\u043b\u0430\u0434\u0430 \u043d\u0430 @Databases Meetup by Mail.ru Cloud Solutions &amp; Tarantool. [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":91091,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-91090","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.3 - aioseo.com -->\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/arhitektura-s3-3-goda-evolyuczii-mail-ru-cloud-storage\" \/>\n\t\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.3\" \/>\n\t\t<meta property=\"og:locale\" content=\"cs_CZ\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47\u0410\u0440\u0445\u0438\u0442\u0435\u043a\u0442\u0443\u0440\u0430 S3: 3 \u0433\u043e\u0434\u0430 \u044d\u0432\u043e\u043b\u044e\u0446\u0438\u0438 Mail.ru Cloud Storage | ProHoster\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/arhitektura-s3-3-goda-evolyuczii-mail-ru-cloud-storage\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2020-08-08T11:42:11+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-08-08T11:42:11+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47Architektura S3: 3 roky evoluce Mail.ru Cloud Storage | ProHoster","description":"","canonical_url":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/arhitektura-s3-3-goda-evolyuczii-mail-ru-cloud-storage","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"cs_CZ","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u0410\u0440\u0445\u0438\u0442\u0435\u043a\u0442\u0443\u0440\u0430 S3: 3 \u0433\u043e\u0434\u0430 \u044d\u0432\u043e\u043b\u044e\u0446\u0438\u0438 Mail.ru Cloud Storage | ProHoster","og:url":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/arhitektura-s3-3-goda-evolyuczii-mail-ru-cloud-storage","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-08-08T11:42:11+00:00","article:modified_time":"2020-08-08T11:42:11+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"91090","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 12:34:25","updated":"2022-10-02 13:27:50","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts\/91090","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/comments?post=91090"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts\/91090\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/media\/91091"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/media?parent=91090"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/categories?post=91090"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/tags?post=91090"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}