{"id":91090,"date":"2020-08-08T13:42:11","date_gmt":"2020-08-08T11:42:11","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/arhitektura-s3-3-goda-evolyuczii-mail-ru-cloud-storage"},"modified":"2020-08-08T13:42:11","modified_gmt":"2020-08-08T11:42:11","slug":"arhitektura-s3-3-goda-evolyuczii-mail-ru-cloud-storage","status":"publish","type":"post","link":"https:\/\/prohoster.info\/ro\/blog\/administrirovanie\/arhitektura-s3-3-goda-evolyuczii-mail-ru-cloud-storage","title":{"rendered":"Arhitectura S3: 3 ani de evolu\u021bie a Mail.ru Cloud Storage","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p><img decoding=\"async\" alt=\"Arhitectura S3: 3 ani de evolu\u021bie a Mail.ru Cloud Storage\" src=\"\/wp-content\/uploads\/2020\/08\/a53775776b3fc969cc15c9d13a64039d.jpeg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<em><noindex><a rel=\"nofollow\" href=\"https:\/\/www.deviantart.com\/st-pete\/art\/Storage-Corridor-408874509\">Coridorul de Stocare<\/a><\/noindex> de la St-Pete<\/em><\/p>\n<p><\/p>\n<p>Salut tuturor! Eu sunt Mons Anderson, arhitectul platformei <noindex><a rel=\"nofollow\" href=\"https:\/\/mcs.mail.ru\/\">Solu\u021bii Cloud Mail.ru<\/a><\/noindex>, voi povesti cum am construit stocarea noastr\u0103 S3, cum func\u021bioneaz\u0103, ce solu\u021bii s-au dovedit a fi reu\u0219ite \u0219i ce ar fi trebuit s\u0103 schimb\u0103m dac\u0103 am \u00eencepe un astfel de proiect de la zero acum.<\/p>\n<p><\/p>\n<p>Articolul este preg\u0103tit pe baza unei prezent\u0103ri la <noindex><a rel=\"nofollow\" href=\"https:\/\/corp.mail.ru\/ru\/press\/events\/databases-2\/\">@Databases Meetup<\/a><\/noindex> de la Mail.ru Cloud Solutions &amp; Tarantool. \u00cen acest articol vom discuta:<\/p>\n<p><\/p>\n<ul>\n<li>cum a fost organizat stocarea Mail.ru, pe care am construit stocarea S3;<\/li>\n<li>ce am ad\u0103ugat pentru a crea Mail.ru Cloud Storage;<\/li>\n<li>cum func\u021bioneaz\u0103 modelul de stocare a obiectelor \u0219i ce pa\u0219i au fost f\u0103cu\u021bi pentru a ajunge \u00een produc\u021bie;<\/li>\n<li>despre \u00eembun\u0103t\u0103\u021birile sistemului de produc\u021bie: failover \u0219i scalare;<\/li>\n<li>cum am implementat sharding \u0219i resharding;<\/li>\n<li>\u0219i, de asemenea, despre lucrul cu certificatele SSL.<\/li>\n<\/ul>\n<p><\/p>\n<p>Dac\u0103 nu dori\u021bi s\u0103 citi\u021bi, pute\u021bi <noindex><a rel=\"nofollow\" href=\"https:\/\/youtu.be\/NEgm1nsv-qg\">s\u0103 vede\u021bi<\/a><\/noindex>.<\/p>\n<p><noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><\/p>\n<h2 id=\"kak-bylo-ustroeno-hranilische-mailru-poverh-kotorogo-my-stroili-s3-hranilische\">Cum a fost organizat stocarea Mail.ru, pe care am construit stocarea S3<\/h2>\n<p><\/p>\n<p>Dezvoltarea S3-ului nostru a \u00eenceput pe stocarea Cloud Mail.ru, deci este important s\u0103 discut\u0103m mai \u00eent\u00e2i cum este organizat\u0103 \u0219i ce poate face.<\/p>\n<p><\/p>\n<p>Stocarea cloud Mail.ru este format\u0103 din servere cu discuri. \u00cen medie, un server de stocare modern are 36 de discuri de 12\u201314 terabytes. Anterior, discurile erau mai mici, dar \u00een trei ani, capacit\u0103\u021bile discurilor au crescut \u0219i ast\u0103zi avem aproape o jum\u0103tate de petabyte de date brute. <\/p>\n<p><\/p>\n<p>Discurile din diferite servere de stocare sunt unite \u00een a\u0219a-numitele \u201epere\u201d (pair). O pere este un unitate unic\u0103 de stocare a fi\u0219ierelor. Practic, este un disc montat \u00eentr-o anumit\u0103 parti\u021bie pe un anumit drum, unde pot fi stocate fi\u0219iere identificate prin hash-uri. <\/p>\n<p><\/p>\n<p>Pere este un nume istoric, a r\u0103mas p\u00e2n\u0103 \u00een zilele noastre, de\u0219i acum nu sunt obligate s\u0103 fie doar dou\u0103 discuri. Pot exista trei discuri, iar de asemenea pot exista stoc\u0103ri hibride, cum ar fi 3\/2. <\/p>\n<p>\n<img decoding=\"async\" alt=\"Arhitectura S3: 3 ani de evolu\u021bie a Mail.ru Cloud Storage\" src=\"\/wp-content\/uploads\/2020\/08\/544ab79846539262e2d4dff6ab95ffff.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><em>Pere (pair) \u2013 unit\u0103\u021bi de stocare a obiectelor<\/em><\/p>\n<p><\/p>\n<p>Toate perele sunt stocate \u00een PairDB \u2013 o aplica\u021bie bazat\u0103 pe Tarantool. Toate bazele din stocarea noastr\u0103, \u00eencep\u00e2nd cu cele mai vechi, sunt Tarantool, nu folosim alte baze.<\/p>\n<p><\/p>\n<p>PairDB stocheaz\u0103 toate perele, st\u0103rile lor, spa\u021biul liber, posibilit\u0103\u021bile de rezerv\u0103, ultimele erori. De asemenea, poate s\u0103 se conecteze la pere, s\u0103 actualizeze starea lor, s\u0103 verifice dac\u0103 func\u021bioneaz\u0103 sau nu. Deci, PairDB este o imagine general\u0103 a st\u0103rii tuturor discurilor sistemului nostru.<\/p>\n<p>\n<img decoding=\"async\" alt=\"Arhitectura S3: 3 ani de evolu\u021bie a Mail.ru Cloud Storage\" src=\"\/wp-content\/uploads\/2020\/08\/d50781b8b51ed5fe126bc84e19c934d1.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><em>Pair DB: baza de date cu starea perechilor<\/em><\/p>\n<p><\/p>\n<p>Pe pere\u021bi sunt stocate fi\u0219iere, iar pentru a \u0219ti pe care pere\u021bi se afl\u0103 fiecare fi\u0219ier, este nevoie de o alt\u0103 baz\u0103 de date \u2014 FileDB. Aceasta p\u0103streaz\u0103 mapping-ul, defini\u021bia coresponden\u021bei: fi\u0219ierul respectiv se afl\u0103 pe pere\u021bii respectivi, precum \u0219i un num\u0103r mic de atribute necesare.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Arhitectura S3: 3 ani de evolu\u021bie a Mail.ru Cloud Storage\" src=\"\/wp-content\/uploads\/2020\/08\/b1a64737cf7f916bff446c10c1f01d4c.jpeg\" style=\"display:block;margin: 0 auto;\" \/><em>File DB: locul unde este stocat fi\u0219ierul<\/em><\/p>\n<p>Un alt element important \u2014 serviciul Nylon, un router pentru lucrul cu bazele de date. Acesta este un punct de acces unic, care permite lucrul printr-o interfa\u021b\u0103 unificat\u0103 at\u00e2t cu PairDB, c\u00e2t \u0219i cu FileDB. Este un serviciu stateless, efectueaz\u0103 balansarea cererilor, \u00een\u021belege pe ce shard trebuie s\u0103 mearg\u0103 FileDB, \u0219tie care pere\u021bi sunt activi \u0219i care nu.<\/p>\n<p>\n<img decoding=\"async\" alt=\"Arhitectura S3: 3 ani de evolu\u021bie a Mail.ru Cloud Storage\" src=\"\/wp-content\/uploads\/2020\/08\/3e2225f229c8cfa462d620d2b5fb581e.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><em>Nylon: router pentru lucrul cu bazele de date<\/em><\/p>\n<p><\/p>\n<p>De asemenea, trebuie s\u0103 plas\u0103m con\u021binut \u00een depozit \u00eentr-un anumit mod. Pentru aceasta, exist\u0103 serviciul \u2014 Streamer. Acesta ofer\u0103 dou\u0103 metode HTTP: metoda PUT, pentru a \u00eenc\u0103rca con\u021binut \u00een depozit, \u0219i metoda GET, pentru a-l recupera. HTTP este un protocol destul de popular \u0219i convenabil pentru transferul de date. <\/p>\n<p><\/p>\n<p>C\u00e2nd ne adres\u0103m la Streamer, acesta se conecteaz\u0103 la PairDB prin Nylon, afl\u00e2nd ce pereche poate primi fi\u0219ierul, dup\u0103 care transmite datele prin WebDAV c\u0103tre aceast\u0103 pereche. <\/p>\n<p><\/p>\n<p>\u00cen esen\u021b\u0103, orice server de stocare este un nginx plus discuri montate la c\u0103ile specificate. Putem \u00eenc\u0103rca fi\u0219iere \u00een depozit din Streamer, le putem \u0219terge, redenumi sau verifica integritatea. Aceasta este o interfa\u021b\u0103 convenabil\u0103 pentru interac\u021biunea la nivel de baz\u0103 cu depozitul. <\/p>\n<p>\n<img decoding=\"async\" alt=\"Arhitectura S3: 3 ani de evolu\u021bie a Mail.ru Cloud Storage\" src=\"\/wp-content\/uploads\/2020\/08\/3722a89ed44cace36e9cb1f8bee4bd1e.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><em>Streamer: punct de acces \u00een depozit<\/em><\/p>\n<p><\/p>\n<h2 id=\"chto-my-dobavili-chtoby-sdelat-s3-hranilische\">Ce am ad\u0103ugat pentru a crea un depozit S3<\/h2>\n<p><\/p>\n<p>A\u0219adar, am examinat structura de baz\u0103 a depozitului \u00een momentul \u00een care ne-am propus s\u0103 lans\u0103m depozitul S3. Cu ajutorul metodei PUT, am putea plasa acolo con\u021binut arbitrar \u0219i ob\u021bine ca identificator al acestor date un hash. Cu acest identificator, ulterior, puteam veni \u0219i recupera fi\u0219ierul ini\u021bial. Dar acest lucru nu este suficient pentru implementarea S3. \u00cen protocolul S3, pe l\u00e2ng\u0103 stocarea efectiv\u0103 a obiectelor, exist\u0103 \u0219i:<\/p>\n<p><\/p>\n<ul>\n<li>stocarea metadatelor \u2014 propriet\u0103\u021bi suplimentare ale obiectelor;<\/li>\n<li>organizarea accesului la obiecte prin intermediul HTTP;<\/li>\n<li>gruparea obiectelor \u00een colec\u021bii \u2014 buc\u0103\u021bi;<\/li>\n<li>Endpoint HTTP-S3. S3 organizeaz\u0103 datele \u00een structuri specifice \u2014 buc\u0103\u021bi, fiecare dintre acestea oferind un punct de acces pentru stocarea fi\u0219ierelor.<\/li>\n<\/ul>\n<p><\/p>\n<p>Pentru implementarea acestei logici a fost necesar un serviciu separatat. De asemenea, am dorit s\u0103 preconiz\u0103m de la \u00eenceput arhitectura pentru o cre\u0219tere ulterioar\u0103 a serviciului cu scalabilitate liniar\u0103.<\/p>\n<p><\/p>\n<h2 id=\"pervye-komponenty\">Primele componente<\/h2>\n<p><\/p>\n<p>Demonul care implementeaz\u0103 S3 API. Acesta este standardul S3 API Amazon, care suport\u0103 lucrul cu XML pentru metadate \u0219i permite transferul direct de con\u021binut. Nu a fost nevoie s\u0103 invent\u0103m nimic, totul este descris \u0219i documentat.<\/p>\n<p><\/p>\n<p>De asemenea, \u00eenaintea serviciului am plasat Nginx. L-am folosit pentru terminarea SSL, balansarea \u00eenc\u0103rc\u0103rii, precum \u0219i pentru anumite logici \u00een Lua (metrice, logare \u0219i tracing).<\/p>\n<p><\/p>\n<p>Pentru stocarea metadatelor S3 am ales de asemenea Tarantool. \u00cen prima versiune, demonul S3 accesa aceast\u0103 baz\u0103 pentru metadate, \u00een timp ce con\u021binutul era stocat \u00eentr-un mare depozit prin intermediul Streamer.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Arhitectura S3: 3 ani de evolu\u021bie a Mail.ru Cloud Storage\" src=\"\/wp-content\/uploads\/2020\/08\/03dddec282b5bb41eb63748ccfb6c9a4.jpeg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<em>Nginx + S3 API + metadate<\/em><\/p>\n<p><\/p>\n<h2 id=\"obektnaya-model-hraneniya\">Modelul obiectelor de stocare<\/h2>\n<p><\/p>\n<p>S\u0103 vedem cum func\u021bioneaz\u0103 S3. Utilizatorul poate crea un bucket - o colec\u021bie de obiecte. Bucket-ul este adresat prin numele gazdei \u0219i reprezint\u0103 un subdomeniu al serviciului. \u00cen cadrul bucket-ului, utilizatorul poate crea obiecte. Identificatorul obiectului va fi URL-ul. Con\u021binutul obiectului este un blob, un array de date binare pe care le vom p\u0103stra \u00een depozit. De asemenea, obiectul are atribute: numele - acel URL, ACL (lista de control al accesului), alte atribute suplimentare sau arbitrare - toate acestea sunt p\u0103strate \u00een metadate. <\/p>\n<p><\/p>\n<p>Schema normalizat\u0103 a acestor date poate ar\u0103ta astfel: exist\u0103 proiecte care de\u021bin bucket-uri, care de\u021bin obiecte, iar obiectele pot fi compuse. Deoarece unul dintre moduri pentru a \u00eenc\u0103rca un obiect este pe p\u0103r\u021bi, exist\u0103 dou\u0103 tabele auxiliare pentru \u00eenc\u0103rcare: uploads \u0219i chunks. De asemenea, proiectele au acreditive pentru acces \u0219i facturare. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Arhitectura S3: 3 ani de evolu\u021bie a Mail.ru Cloud Storage\" src=\"\/wp-content\/uploads\/2020\/08\/e2bede45ba4044f658b47a9840fd5224.jpeg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<em>Schema de date<\/em><\/p>\n<p><\/p>\n<p>Deoarece am realizat un serviciu B2B cu acces pe baz\u0103 de plat\u0103, era necesar\u0103 o schem\u0103 de facturare \u00een acest sistem.<br \/>\nServiciul de facturare l-am implementat de asemenea pe Tarantool. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Arhitectura S3: 3 ani de evolu\u021bie a Mail.ru Cloud Storage\" src=\"\/wp-content\/uploads\/2020\/08\/d5b1a63e2273e933c95deecdb054bfe6.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<h2 id=\"dorabotki-s3-hranilischa-shagi-k-prodakshenu\">\u00cembun\u0103t\u0103\u021biri ale stoc\u0103rii S3: pa\u0219i c\u0103tre produc\u021bie<\/h2>\n<p><\/p>\n<p>Am realizat deja un model func\u021bional care poate fi utilizat: obiectele \u0219i metadatele erau stocate, dar pentru a ie\u0219i \u00een produc\u021bie lipseau c\u00e2teva aspecte.<\/p>\n<p><\/p>\n<p>\u00cen primul r\u00e2nd, avem un sistem de limitare a ratei. Dac\u0103 pornim serviciul f\u0103r\u0103 acesta, la o \u00eenc\u0103rcare maxim\u0103, am putea suprasolicita imprevizibil o parte a sistemului. Limitarea ratei ar trebui s\u0103 func\u021bioneze astfel: orice solicitare S3 ajunge pe un anumit gazduitor, acest gazduitor este identificatorul bucket-ului, iar bucket-ul apar\u021bine unui anumit client. Trebuie s\u0103 definim o func\u021bie legat\u0103 de bucket care s\u0103 permit\u0103 calcularea limit\u0103rii ratei. <\/p>\n<p><\/p>\n<p>\u00cen plus, sistemul de limitare a ratei trebuie s\u0103 fie destul de performant pentru a face fa\u021b\u0103 \u00eenc\u0103rc\u0103rii care vine pe S3.<\/p>\n<p><\/p>\n<p>Aici am folosit din nou Tarantool. Limit\u0103rile ratei constituie un cluster format din 21 de instan\u021be, instan\u021bele sunt grupate, distribuite pe trei noduri fizice \u0219i unite \u00eentr-un mare cluster topologic. Schimb\u0103rile de configurare sunt propagate automat prin acesta: se stabilesc limit\u0103rile ratei, valorile implicite \u0219i configura\u021bia. Fiecare bucket este servit de strict o singur\u0103 instan\u021b\u0103. Atunci c\u00e2nd o solicitare ajunge la un anumit bucket, se calculeaz\u0103 instan\u021ba responsabil\u0103 pentru acel bucket. \u00cen cadrul acestui nod se efectueaz\u0103 calcularea ratei curente de solicit\u0103ri folosind un algoritm similar cu Token Bucket. Ulterior, sistemul de limitare a ratei, pe baza indicatorilor curen\u021bi de \u00eenc\u0103rcare \u0219i a propriet\u0103\u021bilor stabilite pentru bucketul specific, decide dac\u0103 solicitarea poate fi executat\u0103 sau nu. Verificarea limitelor se efectueaz\u0103 \u00een cea mai timpurie etap\u0103 a execut\u0103rii solicit\u0103rii S3, protej\u00e2nd celelalte componente ale sistemului de o suprasolicitare excesiv\u0103. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Arhitectura S3: 3 ani de evolu\u021bie a Mail.ru Cloud Storage\" src=\"\/wp-content\/uploads\/2020\/08\/1b91e5d1fe067c859ba9b36d58827579.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>De asemenea, sub \u00eenc\u0103rcare este destul de dificil s\u0103 ne descurc\u0103m f\u0103r\u0103 cache. \u00cen S3 se presupune o accesare repetat\u0103 a acelora\u0219i obiecte, adic\u0103 acesta este un depozit fierbinte. \u00cen condi\u021bii normale, accesul la un singur fi\u0219ier este gestionat de \u00eentreaga este: Streamer, FileDB, PairDB, Storage. Dar la accesarea repetat\u0103 a unui fi\u0219ier, optimiz\u0103m accesul la acest con\u021binut printr-un cache local.<\/p>\n<p><\/p>\n<p>Cache-ul este multi-strat \u0219i este implementat prin nginx, unit\u0103\u021bi de SSD \u0219i RAM. Aici nu am folosit Tarantool, deoarece este mai convenabil s\u0103 livr\u0103m obiectele din sistemul de fi\u0219iere, astfel put\u00e2nd realiza o stratificare a cache-ului. \u00cen plus, avem obiecte mari cu o dimensiune maxim\u0103 de 32 de gigaocte\u021bi, iar \u00een Tarantool se pot cache-ui doar obiecte mici.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Arhitectura S3: 3 ani de evolu\u021bie a Mail.ru Cloud Storage\" src=\"\/wp-content\/uploads\/2020\/08\/faa02bd52a1d70642589d775f22c5e75.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Acesta este primul sistem cu care am \u00eenceput, av\u00e2nd o capacitate calculat\u0103, suficient\u0103 pentru a explora \u0219i a \u00een\u021belege produsul, asigur\u00e2ndu-ne c\u0103 func\u021bioneaz\u0103. <\/p>\n<p><\/p>\n<h2 id=\"dorabotki-boevoy-sistemy-feylover-i-masshtabirovanie\">\u00cembun\u0103t\u0103\u021biri pentru sistemul de produc\u021bie: failover \u0219i scalabilitate<\/h2>\n<p><\/p>\n<p>Sistemul era deja \u00een produc\u021bie, \u00eens\u0103 la \u00eenceput am omis unele aspecte - era necesar\u0103 ad\u0103ugarea failover-ului \u0219i scalabilit\u0103\u021bii. <\/p>\n<p><\/p>\n<p>Demonul nostru S3 prelua metadate prin protocolul Tarantool. Am \u00eenlocuit baza original\u0103 cu Tarantool, care func\u021biona ca un router proxy pentru cererile de metadate. Din perspectiva aplica\u021biei care implementa API-ul, nimic nu s-a schimbat - aceasta a continuat s\u0103 acceseze baza de date prin protocolul Tarantool, dar router-ul a putut asigura failover activ. Asta \u00eenseamn\u0103 c\u0103 am putut verifica disponibilitatea nodului, s\u0103 avem pauze \u00een timpul comut\u0103rilor \u0219i defec\u021biunilor, \u0219i a\u0219a mai departe. \u00cen plus, aplica\u021bia \u00eens\u0103\u0219i nu a fost modificat\u0103. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Arhitectura S3: 3 ani de evolu\u021bie a Mail.ru Cloud Storage\" src=\"\/wp-content\/uploads\/2020\/08\/978ff8a458f8d48b98672121cf6eca5d.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<h2 id=\"podrobnee-o-tom-kak-my-realizovali-shardirovanie\">Mai multe informa\u021bii despre cum am implementat sharding-ul<\/h2>\n<p><\/p>\n<p>Urm\u0103toarea problem\u0103 care a trebuit abordat\u0103 a fost sharding-ul. Sistemul cre\u0219tea, num\u0103rul de obiecte cre\u0219tea \u0219i a trebuit s\u0103 asigur\u0103m capacit\u0103\u021bi pentru o expansiune ulterioar\u0103.<\/p>\n<p><\/p>\n<p>S\u0103 ne \u00eentoarcem la schema de date: exist\u0103 proiecte, bucket-uri, creden\u021biale \u0219i facturare. Acestea sunt obiecte care, cu o mare probabilitate, \u00een viitorul apropiat nu vor dep\u0103\u0219i limitele unui singur instan\u021b \u00een ceea ce prive\u0219te dimensiunea sau cererile. Asta \u00eenseamn\u0103 c\u0103 nu are sens s\u0103 le shard-uim, a\u0219a c\u0103 le-am mutat \u00eentr-o instan\u021b\u0103 separat\u0103, care va r\u0103m\u00e2ne nesharduit\u0103. Aceasta permite o gestionare mai consistent\u0103 a proiectelor \u0219i bucket-urilor, deoarece exist\u0103 un singur punct nesharduit. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Arhitectura S3: 3 ani de evolu\u021bie a Mail.ru Cloud Storage\" src=\"\/wp-content\/uploads\/2020\/08\/c5a1c1a875e756e3dceb5f885fbda2c8.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>De asemenea, \u00een schem\u0103 exist\u0103 obiecte care cresc linear - la \u00eenceput erau sute de mii, acum num\u0103rul lor se m\u0103soar\u0103 \u00een miliarde. Aceste obiecte, \u00eempreun\u0103 cu p\u0103r\u021bile lor, au trebuit mutate \u00eentr-un cluster sharded. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Arhitectura S3: 3 ani de evolu\u021bie a Mail.ru Cloud Storage\" src=\"\/wp-content\/uploads\/2020\/08\/00bb418cc61ad2e3e8797caf5d1ab202.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Am \u00eemp\u0103r\u021bit schema, dar obiectele trebuie s\u0103 interac\u021bioneze cu bucket-urile: fiecare obiect apar\u021bine \u00eentotdeauna unui bucket specific, plus c\u0103 pe bucket exist\u0103 ACL. De aceea, pentru fiecare shard cu obiecte p\u0103str\u0103m o copie de rezerv\u0103 a fiec\u0103rui bucket. \u00cen plus, \u00een timpul modific\u0103rii obiectelor \u0219i execut\u0103rii cererilor, trebuie s\u0103 calcul\u0103m volumele pentru facturare, a\u0219a c\u0103 fiecare shard are contori pentru facturare.<\/p>\n<p><\/p>\n<p>De asemenea, am ad\u0103ugat c\u00e2teva tabele \u0219i componente suplimentare:<\/p>\n<p><\/p>\n<ul>\n<li>co\u0219ul de reciclare, pentru distrugerea vechilor proiecte care sunt \u0219terse sau suspendate;<\/li>\n<li>o coad\u0103 pentru sarcini de fundal, adic\u0103 stocarea principal\u0103 poate executa sarcini de fundal care trebuie realizate \u00een cluster;<\/li>\n<li>suport pentru lifecycle \u2014 un mecanism care permite gestionarea obiectelor \u0219i administrarea ciclului lor de via\u021b\u0103.<\/li>\n<\/ul>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Arhitectura S3: 3 ani de evolu\u021bie a Mail.ru Cloud Storage\" src=\"\/wp-content\/uploads\/2020\/08\/933e98dc559eb8e6bac14003215a6836.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Fiindc\u0103 o parte din date le-am mutat pe shard-uri, a fost necesar\u0103 o proxy de shard-ing. Ar fi fost posibil s\u0103 reutiliz\u0103m router-ul pentru acest rol, \u00eens\u0103 o proxy de shard-ing separat\u0103, care se ocup\u0103 doar de shard-ing-ul datelor, permite router-ului s\u0103 acceseze datele \u00een \u00eentregime, f\u0103r\u0103 a se g\u00e2ndi la shard-ing.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Arhitectura S3: 3 ani de evolu\u021bie a Mail.ru Cloud Storage\" src=\"\/wp-content\/uploads\/2020\/08\/4b181ed4629180e15558f25b1e4d5718.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Voi explica separat de ce nu am ales o solu\u021bie gata f\u0103cut\u0103, ci am dorit s\u0103 facem o func\u021bie de shard-ing personalizat\u0103. <\/p>\n<p><\/p>\n<p>S\u0103 vedem cum este construit\u0103. Avem 256 de shard-uri disponibile. Pentru fiecare bucket, aloc\u0103m un interval folosind o func\u021bie de consisten\u021b\u0103. Este simplu \u2014 la fel cum folosi\u021bi o func\u021bie de consisten\u021b\u0103 pentru a determina apartenen\u021ba la un shard, determina\u021bi shard-ul de \u00eenceput \u0219i aloca\u021bi intervalul:<\/p>\n<p><\/p>\n<p><code>f(bucket, shards) = subset<\/code><\/p>\n<p><\/p>\n<p>A\u0219adar, dac\u0103 lu\u0103m un bucket, putem spune c\u0103 acesta \u0219i datele sale vor fi \u00eentotdeauna stocate \u00eentr-un subset specific din toate shard-urile. Acest lucru reduce influen\u021ba unora dintre bucket-uri asupra altora \u0219i simplific\u0103 procesarea interog\u0103rilor map-reduce, c\u00e2nd este necesar, de exemplu, s\u0103 face\u021bi o list\u0103 a obiectelor din bucket. Pentru aceasta, trebuie s\u0103 interoga\u021bi toate shard-urile unde aceste obiecte sunt stocate. Dac\u0103 obiectele ar fi stocate pe toate shard-urile, orice listare ar afecta \u00eentreaga sistem\u0103, iar aici afecteaz\u0103 doar un subset specific.<\/p>\n<p><\/p>\n<p>\u00cen continuare \u2014 fiecare obiect apar\u021bine unui anumit bucket, a\u0219a c\u0103 atunci c\u00e2nd ne adres\u0103m pentru un obiect, ne adres\u0103m pentru un obiect dup\u0103 nume \u00een bucket-ul specific. Asta \u00eenseamn\u0103 c\u0103 putem defini o func\u021bie pentru un obiect nu din \u00eentregul interval disponibil de shard-uri, ci doar din subsetul bucket-ului s\u0103u:<\/p>\n<p><\/p>\n<p><code>f(object, subset) = shard<\/code><\/p>\n<p><\/p>\n<p>Lu\u0103m un obiect specific, \u0219i ca argumente ale func\u021biei transmitem nu toate shard-urile, ci subsetul bucket-ului s\u0103u \u2014 \u0219i ob\u021binem un shard specific. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Arhitectura S3: 3 ani de evolu\u021bie a Mail.ru Cloud Storage\" src=\"\/wp-content\/uploads\/2020\/08\/10480ad2430c228c511ff3cdc5071596.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>A\u0219adar, shard-ingul este implementat, exist\u0103 o proxy de shard-ing. R\u0103m\u00e2ne doar s\u0103 ob\u021binem din router \u0219i din baza de metadate \u00een proxy-ul de shard-ing. De exemplu, pentru a crea obiecte de copii shadow \u2014 atunci c\u00e2nd cre\u0103m un bucket, stocarea principal\u0103 trebuie s\u0103 creeze un reprezentant al acestui bucket pe toate shard-urile unde trebuie s\u0103 fie prezent.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Arhitectura S3: 3 ani de evolu\u021bie a Mail.ru Cloud Storage\" src=\"\/wp-content\/uploads\/2020\/08\/bb34d1f747903a3c2f08f582e4424787.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<h2 id=\"kak-my-realizovali-resharding\">Cum am implementat resharding<\/h2>\n<p><\/p>\n<p>Cea mai mare problem\u0103 a shardingului este resharingul. A fost important pentru noi s\u0103-l facem f\u0103r\u0103 downtime, deoarece sistemul era deja \u00een produc\u021bie. Voi ar\u0103ta cum am rezolvat problema printr-un exemplu similar de migrarea \u00een direct a datelor dintr-un proiect \u00een altul.<\/p>\n<p><\/p>\n<p>Mai jos este schema clusterului nostru, care a rezultat dup\u0103 implementarea shardingului. Avem nginx, API S3, un router, baza principal\u0103 cu proiectele, un proxy de sharding \u0219i efectiv shardurile.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Arhitectura S3: 3 ani de evolu\u021bie a Mail.ru Cloud Storage\" src=\"\/wp-content\/uploads\/2020\/08\/0e6bd16c7fc47437b3b7d96b6d9809a4.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Mai sus am omis s\u0103 men\u021bionez c\u0103 \u00eentr-un anumit moment al proiectului a existat o cerin\u021b\u0103 de produs: \u201eLansa\u021bi un alt stoc, Icebox, - ca Hotbox, doar pentru datele reci\u201d. Practic, acela\u0219i tip de stocare, dar pe alte URL-uri \u0219i f\u0103r\u0103 cache.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Arhitectura S3: 3 ani de evolu\u021bie a Mail.ru Cloud Storage\" src=\"\/wp-content\/uploads\/2020\/08\/9ce9085e1d03e9a99b6b8ccc0717c653.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Icebox a fost utilizat mai pu\u021bin dec\u00e2t Hotbox, a\u0219adar, a func\u021bionat destul de mult f\u0103r\u0103 vreo sharding. \u00cen final, am decis s\u0103 renun\u021b\u0103m la el \u0219i s\u0103 combin\u0103m Hotbox \u0219i Icebox \u00eentr-un singur serviciu, \u00eemp\u0103r\u021bind doar clasele de stocare. <\/p>\n<p><\/p>\n<p>Buclele din stocare nu se suprapuneau, era u\u0219or s\u0103 le unim \u0219i s\u0103 le mut\u0103m, dar clien\u021bii foloseau ambele stocuri, ceea ce \u00eensemna c\u0103 trebuia s\u0103 rezolv\u0103m problema absen\u021bei downtime-ului. Nu puteam pur \u0219i simplu s\u0103 oprim \u0219i s\u0103 copiem. Am realizat migrarea \u00een mai multe etape.<\/p>\n<p><\/p>\n<p>Pentru \u00eenceput, am sincronizat stoc\u0103rile primare. Aveam Tarantool, iar c\u00e2nd se crea un obiect, puteam face a\u0219a: <\/p>\n<p><\/p>\n<ul>\n<li>o cerere de creare a unei buc\u0103\u021bi ajunge la baza de date, de exemplu, \u00een Hotbox;<\/li>\n<li>Tarantool verific\u0103 \u00een cealalt\u0103 baz\u0103 (\u00een acest caz - \u00een Icebox), c\u0103 nu exist\u0103 o astfel de bucat\u0103;<\/li>\n<li>dac\u0103 bucata exist\u0103, baza spune c\u0103 nu poate fi creat\u0103, iar aceasta s-a sincronizat ca existent\u0103.<br \/>\n<img decoding=\"async\" alt=\"Arhitectura S3: 3 ani de evolu\u021bie a Mail.ru Cloud Storage\" src=\"\/wp-content\/uploads\/2020\/08\/6694c9804f09942416c1717b455b9fd3.jpeg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<em>Sincronizarea bucatelor<\/em><\/li>\n<\/ul>\n<p><\/p>\n<p>\u00cen stocarea care urma s\u0103 primeasc\u0103 toate datele, am introdus pentru proiecte \u0219i buc\u0103\u021bi un atribut care s\u0103 indice unde este stocat acest obiect. Acesta putea fi stocat local, adic\u0103 \u00een Hotbox, \u00een Icebox - atunci nu exist\u0103 date de la el \u00een noul stoc, sau putea fi \u00een starea de migrare.<\/p>\n<p><\/p>\n<p>Dac\u0103 un proiect sau o bucat\u0103 avea atributul Migrating, atunci \u00een timpul migra\u021biei cererea a fost realizat\u0103 mai \u00eent\u00e2i \u00een noul stoc, \u00een care datele ar trebui s\u0103 fie, iar dac\u0103 acestea nu erau acolo, cererile erau redirec\u021bionate c\u0103tre stocarea alternativ\u0103.<\/p>\n<p><\/p>\n<p>Apoi am transferat traficul. Deoarece API-ul putea deservi at\u00e2t cererile Icebox, c\u00e2t \u0219i cererile Hotbox, am reu\u0219it s\u0103 transfer\u0103m traficul f\u0103r\u0103 downtime, mut\u00e2nd pur \u0219i simplu gazdele \u0219i ad\u0103ug\u00e2nd \u00eenregistr\u0103rile corespunz\u0103toare \u00een Nginx. <\/p>\n<p><\/p>\n<p>Dup\u0103 ce traficul a fost redirec\u021bionat, Nginx \u0219i API-ul de la Icebox au putut fi eliminate.<br \/>\nApoi am eliminat Nginx-ul Icebox \u0219i API-ul S3 \u2014 \u0219i totul a \u00eenceput s\u0103 func\u021bioneze:<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Arhitectura S3: 3 ani de evolu\u021bie a Mail.ru Cloud Storage\" src=\"\/wp-content\/uploads\/2020\/08\/62d9f960fc1f11b875a8f8a5aa8e8152.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Apoi am lansat un proces de migrare \u00een fundal, care func\u021bioneaz\u0103 \u00een cadrul bazei de date \u2014 parcurge toate proiectele \u0219i bucket-urile acestora, le marcheaz\u0103 cu statusul Migrating, transfer\u0103 datele \u0219i, la finalizarea transferului, le marcheaz\u0103 ca Local.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Arhitectura S3: 3 ani de evolu\u021bie a Mail.ru Cloud Storage\" src=\"\/wp-content\/uploads\/2020\/08\/1a3129136f7ccacfdcff260f10cb0605.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Dup\u0103 transferul datelor, nu mai avem nevoie de vechiul storage, a\u0219a c\u0103 elimin\u0103m p\u0103r\u021bile r\u0103mase ale vechii sisteme \u0219i elimin\u0103m suportul pentru statusul migra\u021biei din cod.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Arhitectura S3: 3 ani de evolu\u021bie a Mail.ru Cloud Storage\" src=\"\/wp-content\/uploads\/2020\/08\/13b6f11add40c21a3b9afc05dd0dfea9.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Acelea\u0219i principii au fost aplicate \u0219i pentru resharding-ul din vechiul storage \u00een cel shardat:<\/p>\n<p><\/p>\n<ul>\n<li>Am marcat toate bucket-urile ca <code>Non-sharded<\/code>. Toate cererile c\u0103tre ele erau direc\u021bionate c\u0103tre vechea storage, ne-shardat\u0103.<\/li>\n<li>Noile bucket-uri erau create direct cu statusul <code>Sharded<\/code>.<\/li>\n<li>Am luat bucket-urile pe r\u00e2nd, le-am setat statusul <code>Migrating<\/code> \u0219i am transferat datele.<\/li>\n<\/ul>\n<p><\/p>\n<p>\u00cen acest timp, cererile erau gestionate dup\u0103 principiul:<\/p>\n<p><\/p>\n<ul>\n<li>Citire din noul, apoi din vechiul.<\/li>\n<li>Cre\u0103m doar \u00een noul.<\/li>\n<li>Actualiz\u0103m \u00een dou\u0103 faze: dac\u0103 nu este \u00een nou, transfer\u0103m din vechi \u00een nou, apoi actualiz\u0103m.<\/li>\n<\/ul>\n<p><\/p>\n<h2 id=\"rabota-s-ssl-sertifikatami\">Lucrul cu certificatele SSL<\/h2>\n<p><\/p>\n<blockquote><p>Pe frontend folosim Nginx. \u00cen cazul nostru, nu este un Nginx obi\u0219nuit, ci OpenResty, Nginx cu suport pentru LuaJIT.<\/p><\/blockquote>\n<p>O alt\u0103 component\u0103 a sistemului este gestionarea certificatelor SSL. \u00cen storage-ul S3, pute\u021bi seta un domeniu propriu pentru a accesa un anumit bucket, pur \u0219i simplu folosind <code>CNAME<\/code>. Dar f\u0103r\u0103 HTTPS \u00een ziua de azi nu se poate: un domeniu propriu implic\u0103 un certificat SSL propriu. <\/p>\n<p><\/p>\n<p>Dup\u0103 cum am spus, echilibrarea \u0219i terminarea SSL sunt gestionate de Nginx. \u00cen cazul nostru, nu este un Nginx obi\u0219nuit, ci OpenResty, Nginx cu suport pentru LuaJIT.<\/p>\n<p><\/p>\n<p>Acest lucru ne-a permis s\u0103 \u00eenv\u0103\u021b\u0103m destul de u\u0219or Nginx-ul nostru s\u0103 returneze certificate arbitrare. De asemenea, era necesar s\u0103 return\u0103m certificatele dinamic (f\u0103r\u0103 a fi nevoie s\u0103 le configur\u0103m \u00een fi\u0219ierul de configurare). Am folosit extensia <code>ssl_certificate_by_lua<\/code>, care permite citirea certificatului dintr-o surs\u0103 arbitrar\u0103 \u00een timpul procesului de handshaking TLS. Ca stocare a certificatelor, am folosit \u0219i Tarantool: acest lucru permite gestionarea certificatelor din exterior \u0219i asigur\u0103 o returnare extrem de rapid\u0103.<\/p>\n<p><\/p>\n<p>De asemenea, a fost implementat un daemon separat, a c\u0103rui sarcin\u0103 este s\u0103 actualizeze regulat certificatele emise prin Let\u2019s Encrypt.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Arhitectura S3: 3 ani de evolu\u021bie a Mail.ru Cloud Storage\" src=\"\/wp-content\/uploads\/2020\/08\/8d61c1cb3084d9eb4eb8faa745e212db.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<h2 id=\"chto-by-ya-sohranil-a-chto-sdelal-po-drugomu-esli-razrabatyvat-hranilische-zanovo\">Ce a\u0219 salva \u0219i ce a\u0219 face diferit dac\u0103 a\u0219 dezvolta stocarea din nou<\/h2>\n<p><\/p>\n<h3 id=\"chto-nuzhno-bylo-ispolzovat-s-samogo-nachala\">Ce ar fi trebuit s\u0103 folosesc de la \u00eenceput<\/h3>\n<p><\/p>\n<p><strong>Sharding de la \u00eenceput<\/strong>. A cauzat destul de multe probleme re-shardingul. Se face u\u0219or, \u00eens\u0103, dac\u0103 \u00eencepi proiecte care trebuie s\u0103 se scaleze, este mai bine s\u0103 iei direct un cluster sharding, chiar \u0219i cu un num\u0103r minim de noduri. Implementarea shardingului de la \u00eenceput este aproape gratuit\u0103 comparativ cu integrarea acestuia \u00eentr-un sistem \u00een produc\u021bie.<\/p>\n<p><\/p>\n<p><strong>Lucrul cu Tarantool prin load balancers<\/strong>. Acum toate noile baze sunt imediat conectate la lucru prin load balancers. Acest lucru permite extinderea func\u021bionalit\u0103\u021bii, ob\u021bin\u00e2nd o disponibilitate mai mare. <\/p>\n<p><\/p>\n<p><strong>Auto-failover<\/strong>. A\u0219 instala toate instrumentele necesare pentru auto-failover, deoarece primele e\u0219ecuri dup\u0103 lansare au fost legate de lipsa acestuia. Dup\u0103 experien\u021ba cu S3, toate produsele ulterioare au fost lansate av\u00e2nd \u00een vedere acest lucru.<\/p>\n<p><\/p>\n<p><strong>Func\u021bia S3 \"Versionare\"<\/strong>. Ini\u021bial p\u0103rea c\u0103 aceasta nu este o func\u021bionalitate foarte cerut\u0103. Integrarea acestei op\u021biuni \u00een arhitectura unui sistem func\u021bional este extrem de complicat\u0103.<\/p>\n<p><\/p>\n<p><strong>Facturare separat\u0103<\/strong>. Modul \u00een care am integrat facturarea \u00een sistemul nostru \u0219i-a dovedit eficien\u021ba la \u00eenceput, dar ulterior a \u00eenceput s\u0103 deranjeze, ar fi fost mai bine s\u0103 o implement\u0103m ca un serviciu complet separat\u0103.<\/p>\n<p><\/p>\n<h3 id=\"chto-bylo-udachnym-resheniem\">Ce a fost o solu\u021bie bun\u0103<\/h3>\n<p><\/p>\n<p><strong>Modelul de date<\/strong>. Istoria a ar\u0103tat c\u0103 pe m\u0103sur\u0103 ce serviciul s-a dezvoltat, ne-am potrivit destul de bine cu modelul de date Amazon, a\u0219a c\u0103 putem implementa acele func\u021bii care sunt acolo.<\/p>\n<p><\/p>\n<p><strong>Schema de sharding<\/strong>. A\u0219 sus\u021bine acelea\u0219i shardinguri pe intervale pe bachete, deoarece asta permite o bun\u0103 distribu\u021bie a cererilor din diferite bachete pe un cluster mare.<\/p>\n<p><\/p>\n<p><strong>Utilizarea Tarantool<\/strong>. Tarantool a ajutat foarte mult la dezvoltarea serviciului \u0219i la modific\u0103rile acestuia, am lucrat u\u0219or cu datele, transform\u00e2nd \u0219i sharding stocarea, f\u0103r\u0103 a fi nevoie s\u0103 urc\u0103m pe stratul aplica\u021biei. <\/p>\n<p><\/p>\n<blockquote><p>Aceast\u0103 prezentare a avut loc pentru prima dat\u0103 la <noindex><a rel=\"nofollow\" href=\"https:\/\/corp.mail.ru\/ru\/press\/events\/databases-2\/\">@Databases Meetup<\/a><\/noindex> by Mail.ru Cloud Solutions&amp;Tarantool. Vezi <noindex><a rel=\"nofollow\" href=\"https:\/\/www.youtube.com\/watch?v=jM4hL2u6JM0&amp;list=PLQzTaxmOHjntyNRPWhaWHqlHp8UN_wgQ3\">video<\/a><\/noindex> alte prezent\u0103ri \u0219i aboneaz\u0103-te la anun\u021burile evenimentelor \u00een Telegram <noindex><a rel=\"nofollow\" href=\"https:\/\/t.me\/k8s_mail\">\u00cen jurul Kubernetes \u00een Mail.ru Group<\/a><\/noindex>.<\/p><\/blockquote>\n<p>De asemenea, po\u021bi viziona vechiul meu raport despre S3 sau s\u0103 cite\u0219ti articolul colegului meu despre stocarea pe bloc.<\/p>\n<p><\/p>\n<ul>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/www.youtube.com\/watch?v=O0iIADHgBVc\">Reversarea arhitecturii Amazon S3 \u0219i cum ar\u0103ta stocarea S3 MCS acum 3 ani<\/a><\/noindex>.<\/li>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/mailru\/blog\/472694\/\">Mai mult dec\u00e2t Ceph: stocarea de blocuri a cloud-ului MCS<\/a><\/noindex>.<\/li>\n<\/ul>\n<p>Sursa: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/mailru\/blog\/513356\/\">habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>Storage Corridor by St-Pete \u0412\u0441\u0435\u043c \u043f\u0440\u0438\u0432\u0435\u0442! \u042f Mons Anderson, \u0430\u0440\u0445\u0438\u0442\u0435\u043a\u0442\u043e\u0440 \u043f\u043b\u0430\u0442\u0444\u043e\u0440\u043c\u044b Mail.ru Cloud Solutions, \u0440\u0430\u0441\u0441\u043a\u0430\u0436\u0443, \u043a\u0430\u043a \u043c\u044b \u043f\u043e\u0441\u0442\u0440\u043e\u0438\u043b\u0438 \u043d\u0430\u0448\u0435 S3-\u0445\u0440\u0430\u043d\u0438\u043b\u0438\u0449\u0435, \u043a\u0430\u043a \u043e\u043d\u043e \u0440\u0430\u0431\u043e\u0442\u0430\u0435\u0442, \u043a\u0430\u043a\u0438\u0435 \u0440\u0435\u0448\u0435\u043d\u0438\u044f \u043e\u043a\u0430\u0437\u0430\u043b\u0438\u0441\u044c \u0443\u0434\u0430\u0447\u043d\u044b\u043c\u0438, \u0430 \u043a\u0430\u043a\u0438\u0435 \u0441\u0442\u043e\u0438\u043b\u043e \u0438\u0437\u043c\u0435\u043d\u0438\u0442\u044c, \u0435\u0441\u043b\u0438 \u0431\u044b \u043c\u044b \u043d\u0430\u0447\u0430\u043b\u0438 \u0442\u0430\u043a\u043e\u0439 \u0436\u0435 \u043f\u0440\u043e\u0435\u043a\u0442 \u0441 \u043d\u0443\u043b\u044f \u0441\u0435\u0439\u0447\u0430\u0441. \u0421\u0442\u0430\u0442\u044c\u044f \u043f\u043e\u0434\u0433\u043e\u0442\u043e\u0432\u043b\u0435\u043d\u0430 \u043d\u0430 \u043e\u0441\u043d\u043e\u0432\u0435 \u0434\u043e\u043a\u043b\u0430\u0434\u0430 \u043d\u0430 @Databases Meetup by Mail.ru Cloud Solutions &amp; Tarantool. [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":91091,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-91090","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.2.1 - aioseo.com -->\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/ro\/blog\/administrirovanie\/arhitektura-s3-3-goda-evolyuczii-mail-ru-cloud-storage\" \/>\n\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.2.1\" \/>\n\t\t<meta property=\"og:locale\" content=\"ro_RO\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47\u0410\u0440\u0445\u0438\u0442\u0435\u043a\u0442\u0443\u0440\u0430 S3: 3 \u0433\u043e\u0434\u0430 \u044d\u0432\u043e\u043b\u044e\u0446\u0438\u0438 Mail.ru Cloud Storage | ProHoster\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/ro\/blog\/administrirovanie\/arhitektura-s3-3-goda-evolyuczii-mail-ru-cloud-storage\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2020-08-08T11:42:11+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-08-08T11:42:11+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47Arhitectura S3: 3 ani de evolu\u021bie a Mail.ru Cloud Storage | ProHoster","description":"","canonical_url":"https:\/\/prohoster.info\/ro\/blog\/administrirovanie\/arhitektura-s3-3-goda-evolyuczii-mail-ru-cloud-storage","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"ro_RO","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u0410\u0440\u0445\u0438\u0442\u0435\u043a\u0442\u0443\u0440\u0430 S3: 3 \u0433\u043e\u0434\u0430 \u044d\u0432\u043e\u043b\u044e\u0446\u0438\u0438 Mail.ru Cloud Storage | ProHoster","og:url":"https:\/\/prohoster.info\/ro\/blog\/administrirovanie\/arhitektura-s3-3-goda-evolyuczii-mail-ru-cloud-storage","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-08-08T11:42:11+00:00","article:modified_time":"2020-08-08T11:42:11+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"91090","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 12:34:25","updated":"2022-10-02 13:27:50","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/posts\/91090","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/comments?post=91090"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/posts\/91090\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/media\/91091"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/media?parent=91090"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/categories?post=91090"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/tags?post=91090"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}