{"id":74327,"date":"2020-03-16T08:42:33","date_gmt":"2020-03-16T05:42:33","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/tonkoe-rezervirovanie-fajlovyh-sistem-linux-kak-sozdavat-rabochie-kopii-trehterabajtnoj-subd-mysql-za-20-sekund"},"modified":"2020-03-16T08:42:33","modified_gmt":"2020-03-16T05:42:33","slug":"tonkoe-rezervirovanie-fajlovyh-sistem-linux-kak-sozdavat-rabochie-kopii-trehterabajtnoj-subd-mysql-za-20-sekund","status":"publish","type":"post","link":"https:\/\/prohoster.info\/ro\/blog\/administrirovanie\/tonkoe-rezervirovanie-fajlovyh-sistem-linux-kak-sozdavat-rabochie-kopii-trehterabajtnoj-subd-mysql-za-20-sekund","title":{"rendered":"Rezervarea fin\u0103 a sistemelor de fi\u0219iere Linux. Cum s\u0103 creezi copii de rezerv\u0103 ale unei baze de date MySQL de trei terabai\u021bi \u00een 20 de secunde","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p><img decoding=\"async\" alt=\"Rezervarea fin\u0103 a sistemelor de fi\u0219iere Linux. Cum s\u0103 creezi copii de rezerv\u0103 ale unei baze de date MySQL de trei terabai\u021bi \u00een 20 de secunde\" src=\"\/wp-content\/uploads\/2020\/03\/7502785a78bd6f069913e97829ff2f63.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>M\u0103 numesc Yuri, sunt liderul grupului de administrare a sistemelor la Citimobil. Ast\u0103zi voi \u00eemp\u0103rt\u0103\u0219i experien\u021ba mea \u00een lucrul cu tehnologia rezerv\u0103rii reduse (thin provisioning) a sistemelor de fi\u0219iere Linux \u0219i voi explica cum poate fi utilizat\u0103 \u00een procesele tehnologice CI\/CD ale companiei. Vom analiza situa\u021bia \u00een care pentru testarea automat\u0103 a codului la livrarea acestuia \u00een produc\u021bie avem nevoie c\u00e2t mai rapid de copii ale bazei de date MySQL, c\u00e2t mai apropiate de versiunea de \u201eprod\u201d disponibil\u0103 pentru citire \u0219i scriere.<\/p>\n<p><noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><\/p>\n<h3 id=\"vvedenie-zachem-davat-vrednye-sovety\">Introducere: de ce s\u0103 oferi sfaturi d\u0103un\u0103toare?<\/h3>\n<p><\/p>\n<p>O \u00eentrebare logic\u0103, av\u00e2nd \u00een vedere c\u0103 exist\u0103 mecanisme bine stabilite pentru migrarea schemelor bazei de date \u00een medii de testare. De ce s\u0103 ducem baza de date principal\u0103 nespart\u0103 la astfel de volume? \u00cen plus, pentru testare nu sunt necesare toate datele. Voi \u00eencerca s\u0103 explic.<\/p>\n<p><\/p>\n<p>Aproximativ cu un an \u00een urm\u0103, pe fondul cre\u0219terii active a agregatorului nostru de taxiuri (\u00een 2018 am crescut de aproximativ 15 ori \u00een deplas\u0103ri finalizate), au crescut volumul de date, \u00eenc\u0103rc\u0103tura pe servere \u0219i frecven\u021ba livr\u0103rilor. Ne-am aflat \u00een urm\u0103toarea situa\u021bie:<\/p>\n<p><\/p>\n<ul>\n<li>Baza de date MySQL principal\u0103 a crescut la aproximativ 1000 de tabele cu un volum total de 2,5 TB \u0219i continua s\u0103 creasc\u0103.<\/li>\n<li>Nu a existat posibilitatea de a ne \u00eemp\u0103r\u021bi rapid \u0219i de a dispersa baza. Acest lucru a fost imposibil din cauza vechiului principiu \u201escriu \u00een baza de date ce vreau \u0219i cum vreau\u201d, multe JOIN-uri \u0219i dependen\u021be interne ale tabelelor.<\/li>\n<li>Nu a existat un mecanism pentru migrarea schemei bazei de date \u00een medii de testare.<\/li>\n<li>Nu a existat testare automat\u0103 a codului la livrarea \u00een produc\u021bie.<\/li>\n<\/ul>\n<p><\/p>\n<p>Ultima problem\u0103 a fost dorit\u0103 s\u0103 fie rezolvat\u0103 c\u00e2t mai repede. Au fost deja scrise teste Postman pentru verificarea monolitului PHP principal, dar lipsea o baz\u0103 de date actualizat\u0103. \u00cen acela\u0219i timp, nu puteam crea noaptea o replic\u0103, s\u0103 o facem master \u0219i s\u0103 o oferim \u00een cursul zilei: un num\u0103r foarte mare de livr\u0103ri \u0219i modific\u0103ri, inclusiv \u00een date \u0219i schem\u0103, ar fi f\u0103cut ca standul s\u0103 devin\u0103 nefunc\u021bional deja la mijlocul zilei. \u0218i limitarea livr\u0103rilor doar la ziua lucr\u0103toare ar fi fost ineficient\u0103. <\/p>\n<p><\/p>\n<p>Cu toate acestea, sarcina a fost \u00eendeplinit\u0103: primul stand opera\u021bional l-am ob\u021binut deja \u00een dou\u0103 s\u0103pt\u0103m\u00e2ni. \u00cen trecutul an, acesta a suferit multe modific\u0103ri \u0219i continu\u0103 s\u0103 fie folosit.<\/p>\n<p><\/p>\n<p>\u00cen continuare, voi descrie \u00een detaliu toate pa\u0219ii \u0219i etapele dezvolt\u0103rii solu\u021biei noastre. Ve\u021bi constata c\u0103 aceast\u0103 metod\u0103 merit\u0103 s\u0103 existe.<\/p>\n<p><\/p>\n<p><strong>Ce este \u201erezervarea redus\u0103\u201d?<\/strong><br \/>\nAceasta este o tehnologie hardware sau software (cunoscut\u0103 \u0219i sub numele de volume sparse), care permite alocarea unei cantit\u0103\u021bi mai mari de resurse necesare dec\u00e2t cele disponibile. Astfel, volumul alocat trebuie s\u0103 respecte criteriile just-enough (c\u00e2t este nevoie) \u0219i just-in-time (\u00een timpul necesar). \u00cen principal, rezervarea sub\u021bire este utilizat\u0103 \u00een diverse SCD-uri pentru a oferi spa\u021biu pe disc \u00een volumele necesare, dep\u0103\u0219ind efectiv resursele disponibile. Tehnologia este sus\u021binut\u0103 de diverse sisteme de fi\u0219iere, cum ar fi LVM2, ZFS, BTRFS. Este folosit\u0103 pe scar\u0103 larg\u0103 \u00een hipervizoare de virtualizare. Rezervarea sub\u021bire ne-a permis s\u0103 cre\u0103m rapid din instantanee ale part\u021biei principale cu date at\u00e2t de multe copii ale acestei parti\u021bii c\u00e2t am avut nevoie (directorul de date al SGBD MySQL).<\/p>\n<p><\/p>\n<h3 id=\"pervyy-stend-tehnologiya-thin-lvm\">Prima sta\u021bie, tehnologia Thin LVM<\/h3>\n<p><\/p>\n<p>Aceast\u0103 capitol poate fi numit \u0219i \u201eCum s\u0103 facem instantanee extrem de rapide ale volumelor mari de date folosind <noindex><a rel=\"nofollow\" href=\"http:\/\/man7.org\/linux\/man-pages\/man7\/lvmthin.7.html\">Thin LVM<\/a><\/noindex>, reduc\u00e2nd stabilitatea sistemului de fi\u0219iere \u0219i a SGBD MySQL la valori inacceptabile\u201d. <\/p>\n<p><\/p>\n<p>Deoarece am folosit deja LVM pentru construirea partitiilor principale ale sistemului de operare, am decis s\u0103 \u00eencepem cu aceasta. La \u00eenceput, a fost necesar\u0103 o ma\u0219in\u0103 fizic\u0103 separat\u0103 \u2014 replica bazei noastre principale MySQL, pe care am putut s\u0103 cre\u0103m la cerere o instantanee a replicii \u0219i s\u0103 o ridic\u0103m \u00eentr-un exemplu separat de MySQL. Pe timpul test\u0103rii, am permis opera\u021bii modificatoare pe acest exemplu \u0219i, la finalizarea testelor, l-am \u0219ters cu succes. Configura\u021bia serverului a fost urm\u0103toarea:<\/p>\n<p><\/p>\n<ul>\n<li>2 x Intel Silver 4114 (10\u00d72,2 GHz HT)<\/li>\n<li>8 x 32 GB DDR4 <\/li>\n<li>8 x 1920 GB Intel SSD \u00een controller RAID Adaptec \u00een RAID-10<\/li>\n<\/ul>\n<p><\/p>\n<p>Despre alegerea \u00eentre controller RAID \u0219i RAID software MD s-ar putea scrie un articol separat. Voi spune doar c\u0103 alegerea noastr\u0103 a fost influen\u021bat\u0103 de doi factori:<\/p>\n<p><\/p>\n<ul>\n<li>\u00cen vremurile stabilirii sarcinii, toate SGBD-urile le instalasem pe controlere RAID, a\u0219a c\u0103 se poate spune c\u0103 a\u0219a s-a \u00eent\u00e2mplat istoric. <\/li>\n<li>Diferen\u021ba de performan\u021b\u0103 \u00een teste sintetice ale sistemului de fi\u0219iere \u0219i teste cu diverse opera\u021bii \u00een MySQL a fost minim\u0103. <\/li>\n<\/ul>\n<p><\/p>\n<p>Am \u00eemp\u0103r\u021bit RAID-10 rezultat: am creat un singur grup de volum (VG) pentru \u00eentreaga capacitate (cu cheltuieli suplimentare de aproximativ 6,7 Gb) \u0219i am creat o parti\u021bie logical\u0103 (Logical Volume, LV) pentru sistem de 50 Gb. \u00cen mod normal, restul spa\u021biului este alocat pentru parti\u021bia cu MySQL. Dar aveam nevoie de rezervare flexibil\u0103, a\u0219a c\u0103 mai \u00eent\u00e2i am creat a\u0219a-numitul pool, \u00een interiorul c\u0103ruia am creat o parti\u021bie pentru \/var\/lib\/mysql de 3,5 Tb (pe baza volumelor estimate ale bazei de date):<\/p>\n<p><\/p>\n<pre><code class=\"bash\">lvcreate -l 100%FREE -T vga\/thin\nlvcreate -V 3.5T -T vga\/thin -n mysql<\/code><\/pre>\n<p><\/p>\n<p>Am formatat parti\u021bia \u00een ext4, am montat-o, am scris o replic\u0103 \u0219i am ob\u021binut configura\u021bia ini\u021bial\u0103. Apoi, am realizat o interfa\u021b\u0103 de tip API, care ar trebui s\u0103 creeze instantanee, s\u0103 porneasc\u0103 un exemplar MySQL pe un port specificat \u0219i s\u0103 \u0219terg\u0103 exemplar creat. Deoarece folosim exclusiv apeluri de sistem, am ales ca limbaj de scripting bash obi\u0219nuit, iar ca interfa\u021b\u0103 API HTTP \u2192 bash am implementat o solu\u021bie open source. <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/phonkee\/goexpose\">goexpose<\/a><\/noindex>, scris \u00een Go. <\/p>\n<p><\/p>\n<p>C\u00e2ndva, vom publica scripturile noastre bash \u00een open source, dar deocamdat\u0103 voi descrie doar algoritmul principal:<\/p>\n<p><\/p>\n<p>Crearea instantanei principale snapmain:<\/p>\n<p><\/p>\n<ol>\n<li>Oprim replica principal\u0103.<\/li>\n<li>Punem o blocare pe opera\u021biunile cu instantanea snapmain.<\/li>\n<li>Cre\u0103m o nou\u0103 instantanee snapmain.<\/li>\n<li>Pornim MySQL \u0219i elimin\u0103m blocarea.<\/li>\n<\/ol>\n<p><\/p>\n<p>Crearea unei baze de date pe un port arbitrar din snapmain:<\/p>\n<p><\/p>\n<ol>\n<li>Punem o blocare pe un anumit exemplar de baz\u0103 de date (port).<\/li>\n<li>Verific\u0103m existen\u021ba bloc\u0103rii pentru crearea instantanei principale. Dac\u0103 aceasta exist\u0103, atunci a\u0219tept\u0103m \u0219i verific\u0103m din nou la fiecare 5 secunde.<\/li>\n<li>Verific\u0103m dac\u0103 exist\u0103 o veche parti\u021bie LV a exemplarului.<br \/>\n3.1 Dac\u0103 da, oprim exemplar MySQL folosind kill -9 \u0219i \u0219tergem parti\u021bia LV.<\/li>\n<li>Cre\u0103m un nou exemplar din snapmain.<\/li>\n<li>Preg\u0103tim \u0219i mont\u0103m directoarele pentru acest exemplar.<\/li>\n<li>Elimin\u0103m semnele de slava (fi\u0219iere) \u0219i pornim exemplar MySQL.<\/li>\n<li>\u00cel facem master.<\/li>\n<li>Elimin\u0103m blocarea.<\/li>\n<\/ol>\n<p><\/p>\n<p>\u0218tergerea unei baze de date pe un port arbitrar:<\/p>\n<p><\/p>\n<ol>\n<li>Punem o blocare pe un anumit exemplar de baz\u0103 de date (port).<\/li>\n<li>\u00cempu\u0219c\u0103m exemplar MySQL folosind kill -9.<\/li>\n<li>Demont\u0103m directoarele.<\/li>\n<li>\u0218tergem parti\u021bia LV \u0219i elimin\u0103m blocarea.<\/li>\n<\/ol>\n<p><\/p>\n<p>Exemple de comenzi pentru clonarea parti\u021biilor noii baze de date:<\/p>\n<p><\/p>\n<pre><code class=\"bash\">lvcreate -n stage_3307 -s vga\/snapmain\nlvchange -ay -K vga\/stage_3307\nmount -o noatime,nodiratime,data=writeback \/dev\/mapper\/vga-stage_3307 \/mnt\/stage_3307<\/code><\/pre>\n<p><\/p>\n<p>Acum voi discuta despre problema principal\u0103 cu care ne-am confruntat \u00een timpul utiliz\u0103rii rezerv\u0103rii sub\u021biri. Ne-am lovit de performan\u021ba dispunerilor SSD. Asta s-a \u00eent\u00e2mplat din cauza specificit\u0103\u021bilor Thin LVM: aceasta opereaz\u0103, \u00een esen\u021ba sa, la nivel de dispozitiv cu blocuri de dimensiune de 4 MB, \u00een mod implicit. Cum a ar\u0103tat asta:<\/p>\n<p><\/p>\n<ol>\n<li>Cre\u0103m un snapshot din part\u021bia principal\u0103 \/var\/lib\/mysql.<\/li>\n<li>Pornim replicarea pentru a ajunge din urm\u0103 masterul.<\/li>\n<li>Orice modificare \u00een tabelele replicii for\u021beaz\u0103 p\u0103strarea vechilor blocuri de date nemodificate \u00een sec\u021biunea snapshot-ului.<\/li>\n<li>Orice modificare \u00een exemplarul de test ridicat for\u021beaz\u0103 p\u0103strarea vechilor blocuri de date nemodificate \u00een sec\u021biunea snapshot-ului clonat pentru acest exemplar.<\/li>\n<li>Ob\u021binem o utilizare a opera\u021biunilor de intrare-ie\u0219ire de 100% pe dispozitiv, oprind orice opera\u021biune \u0219i \u00eencetinind treptat replica.<\/li>\n<li>La sf\u00e2r\u0219itul zilei de lucru, ob\u021binem un stand \u00eent\u00e2rziat cu c\u00e2teva ore.<\/li>\n<\/ol>\n<p><\/p>\n<p>Cum ne-am luptat cu asta, pentru a ob\u021bine un rezultat mai rezonabil (punctele principale):<\/p>\n<p><\/p>\n<p>Controller RAID:<\/p>\n<p><\/p>\n<ul>\n<li>Am dezactivat toate tipurile de caching \u00een mod implicit. <\/li>\n<li>Am setat writeback (atunci c\u00e2nd datele ajung \u00een buffer, scrierea se finalizeaz\u0103 \u00eenainte de salvarea efectiv\u0103 pe disc).<\/li>\n<\/ul>\n<p><\/p>\n<p>Sistemul de fi\u0219iere:<\/p>\n<p><\/p>\n<ul>\n<li>\u00cen punctul de montare \/var\/lib\/mysql am specificat <em>noatime,nodiratime,data=writeback<\/em><\/li>\n<li>Am dezactivat jurnalizarea ext4 prin tune2fs.<\/li>\n<\/ul>\n<p><\/p>\n<p>MySQL:<\/p>\n<p><\/p>\n<ul>\n<li>Am specificat <em>innodb_flush_method = O_DSYNC<\/em> (am crescut viteza de scriere, sc\u0103z\u00e2nd astfel fiabilitatea).<\/li>\n<li>Am dezactivat jurnalizarea, logurile nu ne sunt necesare.<\/li>\n<li>Am specificat <em>innodb_buffer_pool_size = 4G<\/em> (cu c\u00e2t dimensiunea pool-ului InnoDB este mai mic\u0103, cu at\u00e2t MySQL se va opri mai repede, \u0219i cu at\u00e2t mai repede vom crea snapshotul).<\/li>\n<\/ul>\n<p><\/p>\n<p>Aceasta nu este o list\u0103 complet\u0103, mai ales privind MySQL. Totu\u0219i, celelalte modific\u0103ri sunt minore \u0219i adesea nu sunt aplicabile \u00eentotdeauna \u0219i nu exact. De exemplu, \u00een \u00eencercarea de a desc\u0103rca discurile, am mutat <em>innodb_parallel_doublewrite_path<\/em> \u00een \/dev\/shm, ceea ce, \u00een unele cazuri, la pornirea unui exemplu necorespunz\u0103tor \u00eencheiat, ne economisea p\u00e2n\u0103 la 5 secunde.<\/p>\n<p><\/p>\n<p>De ce oprim MySQL \u00eenainte de a face snapshot? Deoarece putem lua unul de pe replica func\u021bional\u0103. A\u0219a este, dar noul exemplu de baz\u0103 de date de pe acest snapshot va fi considerat, \u00een mod implicit, corupt \u0219i va necesita o scanare complet\u0103 la pornire. Oprirea replicii este cu siguran\u021b\u0103 mai rapid\u0103, de\u0219i aceasta este, \u00een final, cea mai lung\u0103 opera\u021biune din \u00eentregul proces.<\/p>\n<p><\/p>\n<p>\u00cen urma analizei, am ob\u021binut timpi mai acceptabili \u0219i un stand func\u021bional. Totu\u0219i, dup\u0103 cum se vede din graficul elocvent al \u00eent\u00e2rzierii replic\u0103rii principale, situa\u021bia este \u00eenc\u0103 departe de ideal:<br \/>\n<img decoding=\"async\" alt=\"Rezervarea fin\u0103 a sistemelor de fi\u0219iere Linux. Cum s\u0103 creezi copii de rezerv\u0103 ale unei baze de date MySQL de trei terabai\u021bi \u00een 20 de secunde\" src=\"\/wp-content\/uploads\/2020\/03\/9bb7bb7d9abfea124a75616deec57f05.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Printre alte dezavantaje, merit\u0103 men\u021bionat\u0103 imposibilitatea practic\u0103 de a monitoriza pool-ul Thin LVM: dincolo de func\u021biile sistemului standard iostat, nu este posibil s\u0103 \u00een\u021belegem, de exemplu, care element al pool-ului genereaz\u0103 acum cea mai mare \u00eenc\u0103rcare pe sistemul de fi\u0219iere.<\/p>\n<p><\/p>\n<p>Un alt dezavantaj semnificativ, legat de optimizarea men\u021bionat\u0103 anterior, este c\u0103 am ob\u021binut un stand YOLO. Aproape o dat\u0103 la una-dou\u0103 luni, ext4 nu rezista abuzurilor \u0219i se strica ireversibil, necesit\u00e2nd reformatare \u0219i re\u00eenc\u0103rcare a replicii. C\u00e2\u0219tig\u00e2nd \u00een vitez\u0103, am compromis complet stabilitatea.<\/p>\n<p><\/p>\n<p>Ce metrici ar trebui s\u0103 monitoriz\u0103m \u00een timpul utiliz\u0103rii Thin LVM:<\/p>\n<p><\/p>\n<ul>\n<li>Procentaj de date din pool-ul Thin<\/li>\n<li>Procentaj de metadate din pool-ul Thin<\/li>\n<\/ul>\n<p><\/p>\n<p>Dac\u0103 standul nostru va supravie\u021bui lipsei de spa\u021biu pentru date (este suficient s\u0103 cur\u0103\u021b\u0103m discurile), lipsa de spa\u021biu pentru metadate va duce la o pr\u0103bu\u0219ire complet\u0103 a pool-ului \u0219i va necesita recrearea acestuia de la zero.<\/p>\n<p><\/p>\n<p>Sistemul de fi\u0219iere din interiorul pool-ului se fragmenteaz\u0103 considerabil \u00een timp. Recomand s\u0103 rula\u021bi zilnic comanda prin cron, <em>fstrim -v \/var\/lib\/mysql<\/em>.<\/p>\n<p><\/p>\n<p>Rezultate intermediare:<\/p>\n<p><\/p>\n<ul>\n<li>Tehnologia este u\u0219or aplicabil\u0103, la fel ca \u0219i LVM, \u0219i nu necesit\u0103 o calificare special\u0103 a inginerului.<\/li>\n<li>Este bine s\u0103 fie utilizat\u0103 pentru baze de date de dimensiuni mici \u0219i care nu sunt prea solicitate. Cu c\u00e2t baza de date este mai mic\u0103, cu at\u00e2t mai pu\u021bini chunk-uri se mut\u0103 prin sistemul de fi\u0219iere din interiorul pool-ului \u0219i cu at\u00e2t mai mic\u0103 este \u00eenc\u0103rcarea pe discuri.<\/li>\n<li>Pentru sarcina noastr\u0103, am \u00eenceput s\u0103 c\u0103ut\u0103m alte solu\u021bii, despre care vom discuta \u00een urm\u0103toarea sec\u021biune.<\/li>\n<\/ul>\n<p><\/p>\n<h3 id=\"vtoroy-stend-tehnologiya-zfs\">Al doilea stand, tehnologia ZFS<\/h3>\n<p><\/p>\n<p>Cu mult timp \u00een urm\u0103 am avut de-a face cu sistemul de fi\u0219iere ZFS, dar atunci ZFS func\u021biona cu adev\u0103rat bine pe familia sa nativ\u0103 de sisteme de operare Solaris. Existau versiuni portate pe FreeBSD cu un nivel de implementare destul de bun. De asemenea, a existat un port neterminat pe Linux, care era rar utilizat. Din cauza structurii de stocare a datelor B-tree (de altfel, aceea\u0219i structur\u0103 de stocare este utilizat\u0103 de InnoDB MySQL), ZFS s-a dovedit a fi slab \u00een instala\u021biile cu un num\u0103r foarte mare de fi\u0219iere. Toate acestea, combinate cu necesitatea de a \u00eenv\u0103\u021ba detalii tehnice \u00eenainte de utilizare, au dus la eliminarea acestei file de sistem din practica mea pentru o lung\u0103 perioad\u0103 de timp. Au ap\u0103rut ext4 \u0219i xfs, care au devenit standarde. Dar av\u00e2nd \u00een vedere c\u0103 ZFS se potrive\u0219te mai mult dec\u00e2t bine pentru sarcina noastr\u0103, iar versiunea de Linux, judec\u00e2nd dup\u0103 recenzii, a evoluat \u00eentr-un produs decent (de\u0219i nu beneficiaz\u0103 de suport complet, ceea ce face ca instalarea complet\u0103 a sistemului pe ZFS s\u0103 fie posibil\u0103 doar prin diverse trucuri), am decis s\u0103 o \u00eencerc\u0103m.<\/p>\n<p><\/p>\n<p>Din motive evidente, am ales un stand cu o configura\u021bie similar\u0103 (cu excep\u021bia controller-ului RAID). Am instalat opt SSD-uri de 1920 Gb. Nu am dorit s\u0103 scriem imaginea de re\u021bea pentru \u00eenc\u0103rcarea serverului pe ZFS gol, a\u0219a c\u0103 am t\u0103iat c\u00e2te 50 Gb de pe toate discurile \u0219i am creat un RAID-10 MD pentru sistem. Celelalte 1950 Gb de pe fiecare disc le-am combinat \u00eentr-un analog ZFS al RAID-10:<\/p>\n<p><\/p>\n<pre><code class=\"bash\">zpool create zpool mirror \/dev\/sda2 \/dev\/sdb2 mirror \/dev\/sdc2 \/dev\/sdd2 mirror \/dev\/sde2 \/dev\/sdf2 mirror \/dev\/sdg2 \/dev\/sdh2<\/code><\/pre>\n<p><\/p>\n<p>Am creat parti\u021bii pentru MySQL:<\/p>\n<p><\/p>\n<pre><code class=\"bash\">zfs create zpool\/mysql\nzfs set compression=gzip zpool\/mysql\nzfs set recordsize=128k zpool\/mysql\nzfs set atime=off zpool\/mysql\nzfs create zpool\/mysql\/data\nzfs set recordsize=16k zpool\/mysql\/data\nzfs set primarycache=metadata zpool\/mysql\/data\nzfs set mountpoint=\/var\/lib\/mysql zpool\/mysql\/data<\/code><\/pre>\n<p><\/p>\n<p>V\u0103 rug\u0103m s\u0103 re\u021bine\u021bi c\u0103 am activat compresia de date gzip standard. Avem multe resurse de procesor pe server, iar acestea nu sunt utilizate complet. Ca rezultat, 3 TB din baza noastr\u0103 de date s-au transformat \u00een 1,6 TB, iar deoarece punctul slab, ca \u0219i \u00een cazul anterior, este performan\u021ba maxim\u0103 a discurilor, cu c\u00e2t sunt mai pu\u021bine date \u2014 cu at\u00e2t mai bine, ob\u021binem de la \u00eenceput un bonus excelent de la ZFS! \u00cen orele de v\u00e2rf, sub o sarcin\u0103 complet\u0103, men\u021binerea func\u021bion\u0103rii gzip necesit\u0103 p\u00e2n\u0103 la 4 nuclee, dar nu ne deranjeaz\u0103.<\/p>\n<p><\/p>\n<p>\u00centre timp, implementarea a decurs mai repede. Am copiat set\u0103rile replica MySQL de pe standul LVM. A trebuit s\u0103 ne petrecem ceva timp rescriind scripturile \u00een comenzi ZFS, dar \u00een general algoritmii au r\u0103mas aceia\u0219i. Un exemplu de creare a unui snapshot:<\/p>\n<p><\/p>\n<pre><code class=\"plaintext\">zfs set snapdir=visible zpool\/mysql\/data\nzfs create zpool\/stage_3307\nzfs clone zpool\/mysql\/data@snapmain zpool\/stage_3307\/data\nzfs set mountpoint=\/mnt\/stage_3307 zpool\/stage_3307\/data<\/code><\/pre>\n<p><\/p>\n<p>De la tuning suplimentar: am mutat \u00een memorie sec\u021biunile ZFS cu metadate \u0219i log-uri l2arc \u0219i zil. Pentru sarcina noastr\u0103, a\u0219a cum s-a dovedit ulterior, aceasta era excesiv\u0103, dar deocamdat\u0103 am p\u0103strat aceast\u0103 optimizare, schimbarea este u\u0219oar\u0103 \u00een caz de nevoie. Din efectele negative \u2014 trebuie s\u0103 recre\u0103m zonele corespunz\u0103toare ale memoriei dup\u0103 repornirea serverului. Datele nu se pierd. Extrac\u021bie zpool status:<\/p>\n<p><\/p>\n<pre><code class=\"bash\">logs\n      \/dev\/shm\/zil_slog.img  ONLINE       0     0     0\ncache\n      \/dev\/shm\/l2arc.img     ONLINE       0     0     0<\/code><\/pre>\n<p><\/p>\n<p>\u00cen aceast\u0103 configura\u021bie am \u00eenceput s\u0103 test\u0103m standul \u0219i am ob\u021binut rezultate excelente: cu dou\u0103 instan\u021be de baze de date func\u021bion\u00e2nd simultan (\u0219i replicat principal activ) pe snapshot-uri, am ob\u021binut o \u00eenc\u0103rcare a discurilor de 50-60%.<\/p>\n<p><\/p>\n<p>Am sc\u0103pat de problema noastr\u0103 principal\u0103, ceea ce se vede \u00een graficul \u00eent\u00e2rzierii replic\u0103rii (comp\u0103ra\u021bi cu graficul anterior din sec\u021biunea Thin LVM):<br \/>\n<img decoding=\"async\" alt=\"Rezervarea fin\u0103 a sistemelor de fi\u0219iere Linux. Cum s\u0103 creezi copii de rezerv\u0103 ale unei baze de date MySQL de trei terabai\u021bi \u00een 20 de secunde\" src=\"\/wp-content\/uploads\/2020\/03\/15df9283838a4662ff0f68accfde4089.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>\u00cen plus \u0219i datorit\u0103 acestui lucru, am accelerat semnificativ toate opera\u021biile: crearea complet\u0103 a snapshot-ului cu oprirea \u0219i repornirea replicatului dureaz\u0103 p\u00e2n\u0103 la 40 de secunde, desf\u0103\u0219urarea unui nou exemplar MySQL din snapshot dureaz\u0103 p\u00e2n\u0103 la 20 de secunde. Ceea ce ne mul\u021bume\u0219te pe noi, dar \u0219i testele noastre de cod.<\/p>\n<p><\/p>\n<p>Rezultate intermediare:<\/p>\n<p><\/p>\n<ul>\n<li>Rezultatele au satisf\u0103cut pe deplin nevoia noastr\u0103 de a ob\u021bine o copie a bazei de date \u00een produc\u021bie pentru testarea codului.<\/li>\n<li>Tehnologia necesit\u0103 o \u00een\u021belegere: trebuie s\u0103 \u0219tii ce este ZFS \u0219i cum s\u0103 lucrezi cu ea.<\/li>\n<li>Nu am verificat starea curent\u0103 a func\u021bion\u0103rii ZFS cu un num\u0103r mare (de la 1 milion) de fi\u0219iere mici. Dar presupunem c\u0103 problema persist\u0103, prin urmare nu a\u0219 recomanda acest sistem de fi\u0219iere pentru niciun fel de stoc\u0103ri de fi\u0219iere.<\/li>\n<\/ul>\n<p><\/p>\n<h3 id=\"chto-dalshe\">Ce urmeaz\u0103?<\/h3>\n<p><\/p>\n<p>\u00cen cadrul standului nu facem mai nimic, rezultatul ne mul\u021bume\u0219te. Este posibil ca \u00een viitor s\u0103 ad\u0103ug\u0103m \u00een configurarea replic\u0103rii standului excluderea tabelelor inutile pentru testare, ceea ce va reduce \u0219i mai mult volumul bazei de date. Nu am testat sistemul BTRFS \u0219i implementarea sa a tehnologiei de rezervare sub\u021bire. Totu\u0219i, o astfel de sarcin\u0103 nu mai este relevant\u0103, deoarece obiectivul principal a fost atins. \u00cen general, desigur, ne dorim s\u0103 ne \u00eendep\u0103rt\u0103m de abordarea descris\u0103 mai sus \u2014 s\u0103 implement\u0103m migra\u021bii func\u021bionale ale bazei de date \u00een medii de testare, s\u0103 cre\u0103m un circuit separat de testare pentru baza de date, s\u0103 ne ocup\u0103m de sharding-ul bazei de date principale. Multe dintre acestea le punem deja \u00een practic\u0103, despre ceea ce vom vorbi cu siguran\u021b\u0103 \u00een articolele viitoare. <\/p>\n<p><\/p>\n<h3 id=\"itogi\">Concluzii<\/h3>\n<p><\/p>\n<p>Sarcina ini\u021bial\u0103 a fost rezolvat\u0103, de\u0219i \u00eentr-un mod neobi\u0219nuit. \u00cen concluziile intermediare au fost descrise avantajele \u0219i dezavantajele fiec\u0103rei tehnologii utilizate, a\u0219a c\u0103 s\u0103 decidem ce tehnologie \u0219i c\u00e2nd poate fi folosit\u0103:<\/p>\n<p><\/p>\n<ul>\n<li>Thin LVM \u2014 pentru baze de date mici \u0219i c\u00e2nd nu ai dorin\u021ba sau timpul necesar pentru a studia ZFS.<\/li>\n<li>ZFS \u2014 dac\u0103 ai experien\u021b\u0103 \u00een utilizarea sa sau oportunitatea de a dedica timp pentru a \u00eenv\u0103\u021ba \u00een orice situa\u021bie. <\/li>\n<\/ul>\n<p><\/p>\n<p>La un nivel mai \u00eenalt, acest articol nu este doar o compara\u021bie a tehnologiilor celor dou\u0103 sisteme de fi\u0219iere. Ideea principal\u0103 pe care a\u0219 dori s\u0103 o transmit \u0219i s\u0103 o consolidez este c\u0103 nu ar trebui s\u0103 ne temem s\u0103 g\u00e2ndim neconven\u021bional \u00een situa\u021bii critice pentru afaceri \u0219i s\u0103 folosim doar re\u021bete gata f\u0103cute. C\u00e2ndva, \u00eentreaga noastr\u0103 echip\u0103 tehnic\u0103 putea s\u0103 clatin\u0103 din cap \u0219i s\u0103 spun\u0103 c\u0103 sarcina de a crea copii de baz\u0103 de trei terabytes \u00een mai pu\u021bin de un minut este imposibil\u0103, \u0219i c\u0103 nu avem nevoie de tehnologii riscante, a\u0219a c\u0103 haide\u021bi s\u0103 facem cum trebuie. Ar fi fost posibil, dar am fi pierdut aproximativ \u0219ase luni p\u00e2n\u0103 la un an \u0219i multe c\u0103l\u0103torii ale clien\u021bilor (c\u0103l\u0103toriile sunt indicatorul nostru de afaceri principal) f\u0103r\u0103 teste \u0219i \u00een timpul implement\u0103rii. Proced\u00e2nd neconven\u021bional, am pierdut mult mai pu\u021bin timp cu implementarea, am c\u00e2\u0219tigat experien\u021b\u0103 \u00een tehnologii noi \u0219i vechi uitate, \u0219i am oferit testarea exact \u00een acel moment \u00een care aveam cu adev\u0103rat nevoie. F\u0103r\u0103 \u00eendoial\u0103, acest lucru a avut un impact pozitiv asupra tuturor indicatorilor no\u0219tri. Alegerea este \u00eentotdeauna a voastr\u0103, iar noi, din partea noastr\u0103, vom continua s\u0103 povestim \u00een blogul nostru despre realiz\u0103rile interesante actuale \u0219i viitoare.<\/p>\n<p>Sursa: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/citymobil\/blog\/492172\/\">habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u041c\u0435\u043d\u044f \u0437\u043e\u0432\u0443\u0442 \u042e\u0440\u0438\u0439, \u044f \u0440\u0443\u043a\u043e\u0432\u043e\u0434\u0438\u0442\u0435\u043b\u044c \u0433\u0440\u0443\u043f\u043f\u044b \u0441\u0438\u0441\u0442\u0435\u043c\u043d\u043e\u0433\u043e \u0430\u0434\u043c\u0438\u043d\u0438\u0441\u0442\u0440\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u044f \u0432 \u0421\u0438\u0442\u0438\u043c\u043e\u0431\u0438\u043b. \u0421\u0435\u0433\u043e\u0434\u043d\u044f \u043f\u043e\u0434\u0435\u043b\u044e\u0441\u044c \u043e\u043f\u044b\u0442\u043e\u043c \u0440\u0430\u0431\u043e\u0442\u044b \u0441 \u0442\u0435\u0445\u043d\u043e\u043b\u043e\u0433\u0438\u0435\u0439 \u0442\u043e\u043d\u043a\u043e\u0433\u043e \u0440\u0435\u0437\u0435\u0440\u0432\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u044f (thin provisioning) \u0444\u0430\u0439\u043b\u043e\u0432\u044b\u0445 \u0441\u0438\u0441\u0442\u0435\u043c Linux \u0438 \u0440\u0430\u0441\u0441\u043a\u0430\u0436\u0443, \u043a\u0430\u043a \u0435\u0435 \u043c\u043e\u0436\u043d\u043e \u043f\u0440\u0438\u043c\u0435\u043d\u044f\u0442\u044c \u0432 \u0442\u0435\u0445\u043d\u043e\u043b\u043e\u0433\u0438\u0447\u0435\u0441\u043a\u0438\u0445 CI\/CD-\u043f\u0440\u043e\u0446\u0435\u0441\u0441\u0430\u0445 \u043a\u043e\u043c\u043f\u0430\u043d\u0438\u0438. \u041c\u044b \u0440\u0430\u0437\u0431\u0435\u0440\u0435\u043c \u0441\u0438\u0442\u0443\u0430\u0446\u0438\u044e, \u043a\u043e\u0433\u0434\u0430 \u0434\u043b\u044f \u0430\u0432\u0442\u043e\u043c\u0430\u0442\u0438\u0447\u0435\u0441\u043a\u043e\u0433\u043e \u0442\u0435\u0441\u0442\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u044f \u043a\u043e\u0434\u0430 \u043f\u0440\u0438 \u0434\u043e\u0441\u0442\u0430\u0432\u043a\u0435 \u0435\u0433\u043e \u0432 production \u043d\u0430\u043c \u043a\u0430\u043a \u043c\u043e\u0436\u043d\u043e \u0431\u044b\u0441\u0442\u0440\u0435\u0435 \u043d\u0435\u043e\u0431\u0445\u043e\u0434\u0438\u043c\u044b \u043a\u043e\u043f\u0438\u0438 \u0411\u0414 MySQL, \u043c\u0430\u043a\u0441\u0438\u043c\u0430\u043b\u044c\u043d\u043e [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":74328,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-74327","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.2.1 - aioseo.com -->\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/ro\/blog\/administrirovanie\/tonkoe-rezervirovanie-fajlovyh-sistem-linux-kak-sozdavat-rabochie-kopii-trehterabajtnoj-subd-mysql-za-20-sekund\" \/>\n\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.2.1\" \/>\n\t\t<meta property=\"og:locale\" content=\"ro_RO\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47\u0422\u043e\u043d\u043a\u043e\u0435 \u0440\u0435\u0437\u0435\u0440\u0432\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u0435 \u0444\u0430\u0439\u043b\u043e\u0432\u044b\u0445 \u0441\u0438\u0441\u0442\u0435\u043c Linux. \u041a\u0430\u043a \u0441\u043e\u0437\u0434\u0430\u0432\u0430\u0442\u044c \u0440\u0430\u0431\u043e\u0447\u0438\u0435 \u043a\u043e\u043f\u0438\u0438 \u0442\u0440\u0435\u0445\u0442\u0435\u0440\u0430\u0431\u0430\u0439\u0442\u043d\u043e\u0439 \u0421\u0423\u0411\u0414 MySQL \u0437\u0430 20 \u0441\u0435\u043a\u0443\u043d\u0434 | ProHoster\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/ro\/blog\/administrirovanie\/tonkoe-rezervirovanie-fajlovyh-sistem-linux-kak-sozdavat-rabochie-kopii-trehterabajtnoj-subd-mysql-za-20-sekund\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2020-03-16T05:42:33+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-03-16T05:42:33+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47Rezervarea fin\u0103 a sistemelor de fi\u0219iere Linux. Cum s\u0103 crea\u021bi copii de siguran\u021b\u0103 pentru o baz\u0103 de date MySQL de trei terabytes \u00een 20 de secunde | ProHoster","description":"","canonical_url":"https:\/\/prohoster.info\/ro\/blog\/administrirovanie\/tonkoe-rezervirovanie-fajlovyh-sistem-linux-kak-sozdavat-rabochie-kopii-trehterabajtnoj-subd-mysql-za-20-sekund","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"ro_RO","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u0422\u043e\u043d\u043a\u043e\u0435 \u0440\u0435\u0437\u0435\u0440\u0432\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u0435 \u0444\u0430\u0439\u043b\u043e\u0432\u044b\u0445 \u0441\u0438\u0441\u0442\u0435\u043c Linux. \u041a\u0430\u043a \u0441\u043e\u0437\u0434\u0430\u0432\u0430\u0442\u044c \u0440\u0430\u0431\u043e\u0447\u0438\u0435 \u043a\u043e\u043f\u0438\u0438 \u0442\u0440\u0435\u0445\u0442\u0435\u0440\u0430\u0431\u0430\u0439\u0442\u043d\u043e\u0439 \u0421\u0423\u0411\u0414 MySQL \u0437\u0430 20 \u0441\u0435\u043a\u0443\u043d\u0434 | ProHoster","og:url":"https:\/\/prohoster.info\/ro\/blog\/administrirovanie\/tonkoe-rezervirovanie-fajlovyh-sistem-linux-kak-sozdavat-rabochie-kopii-trehterabajtnoj-subd-mysql-za-20-sekund","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-03-16T05:42:33+00:00","article:modified_time":"2020-03-16T05:42:33+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"74327","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 18:16:23","updated":"2022-10-01 23:42:57","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/posts\/74327","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/comments?post=74327"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/posts\/74327\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/media\/74328"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/media?parent=74327"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/categories?post=74327"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/tags?post=74327"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}