{"id":52113,"date":"2019-11-01T00:00:00","date_gmt":"2019-10-31T21:00:00","guid":{"rendered":"https:\/\/prohoster.info\/blog\/blog_prohoster\/bioyino-raspredelyonnyj-masshtabiruemyj-agregator-metrik"},"modified":"2020-02-18T13:59:47","modified_gmt":"2020-02-18T10:59:47","slug":"bioyino-raspredelyonnyj-masshtabiruemyj-agregator-metrik","status":"publish","type":"post","link":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/bioyino-raspredelyonnyj-masshtabiruemyj-agregator-metrik","title":{"rendered":"Bioyino \u2014 distribuovan\u00fd, \u0161k\u00e1lovateln\u00fd agreg\u00e1tor metrik","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p>Tak\u017ee shroma\u017e\u010fujete metriky. Stejn\u011b jako my. My tak\u00e9 shroma\u017e\u010fujeme metriky. Samoz\u0159ejm\u011b, \u017ee ty pot\u0159ebn\u00e9 pro podnik\u00e1n\u00ed. Dnes v\u00e1m pov\u00edme o prvn\u00edm \u010dl\u00e1nku v syst\u00e9mu na\u0161eho monitorov\u00e1n\u00ed \u2014 serveru pro agregaci kompatibiln\u00edm se statsd. <noindex><a rel=\"nofollow\" href=\"http:\/\/bit.ly\/2Noxg1M\">bioyino<\/a><\/noindex>, pro\u010d jsme jej napsali a pro\u010d jsme se vzdali brubecku. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Bioyino \u2014 distribuovan\u00fd, \u0161k\u00e1lovateln\u00fd agreg\u00e1tor metrik\" src=\"\/wp-content\/uploads\/2019\/11\/312c9a706828acc65f638a6a8abfc5b5.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><\/p>\n<p>Z na\u0161ich p\u0159edchoz\u00edch \u010dl\u00e1nk\u016f (<noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/company\/avito\/blog\/335410\/\">1<\/a><\/noindex>, <noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/company\/avito\/blog\/343928\/\">2<\/a><\/noindex>) se m\u016f\u017eete dozv\u011bd\u011bt, \u017ee do ur\u010dit\u00e9 doby jsme zna\u010dky shroma\u017e\u010fovali pomoc\u00ed <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/github\/brubeck\">brubecku<\/a><\/noindex>. Je naps\u00e1n v C. Z pohledu k\u00f3du \u2014 jednoduch\u00fd jako z\u00e1tka (to je d\u016fle\u017eit\u00e9, kdy\u017e chcete p\u0159isp\u011bt) a, co\u017e je nejd\u016fle\u017eit\u011bj\u0161\u00ed, zvl\u00e1d\u00e1 na\u0161e objemy 2 miliony metrik za sekundu (MPS) v maxim\u00e1ln\u00ed z\u00e1t\u011b\u017ei bez v\u011bt\u0161\u00edch probl\u00e9m\u016f. Dokumentace uv\u00e1d\u00ed podporu 4 miliony MPS s hv\u011bzdi\u010dkou. To znamen\u00e1, \u017ee uveden\u00fd \u00fadaj dos\u00e1hnete, pokud spr\u00e1vn\u011b nastav\u00edte s\u00ed\u0165 na Linuxu. (Kolik MPS lze dos\u00e1hnout, pokud s\u00ed\u0165 nech\u00e1te tak jak je, nev\u00edme). Navzdory t\u011bmto v\u00fdhod\u00e1m jsme m\u011bli k brubecku n\u011bkolik v\u00e1\u017en\u00fdch v\u00fdhrad.<\/p>\n<p><\/p>\n<p><em>V\u00fdhrada 1.<\/em> Github \u2014 v\u00fdvoj\u00e1\u0159 projektu \u2014 p\u0159estal jej podporovat: publikovat opravy a z\u00e1platy, p\u0159ij\u00edmat na\u0161e a (nejen na\u0161e) PR. V posledn\u00edch n\u011bkolika m\u011bs\u00edc\u00edch (n\u011bkdy od \u00fanora-b\u0159ezna 2018) se aktivita obnovila, ale p\u0159edt\u00edm byla t\u00e9m\u011b\u0159 2 roky \u00fapln\u00e9ho ticha. Krom\u011b toho je projekt vyv\u00edjen <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/github\/brubeck\/pull\/31#issuecomment-325907734\">pro intern\u00ed pot\u0159eby Githubu<\/a><\/noindex>, co\u017e m\u016f\u017ee p\u0159edstavovat v\u00e1\u017enou p\u0159ek\u00e1\u017eku pro zav\u00e1d\u011bn\u00ed nov\u00fdch funkc\u00ed.<\/p>\n<p><\/p>\n<p><em>V\u00fdhrada 2.<\/em> P\u0159esnost v\u00fdpo\u010dt\u016f. Brubeck shroma\u017e\u010fuje pro agregaci pouze 65536 hodnot. V na\u0161em p\u0159\u00edpad\u011b m\u016f\u017ee b\u011bhem agrega\u010dn\u00edho obdob\u00ed (30 s) p\u0159ich\u00e1zet mnohem v\u00edce hodnot (1 527 392 v maxim\u00e1ln\u00edm zat\u00ed\u017een\u00ed). V d\u016fsledku tohoto vzorkov\u00e1n\u00ed hodnoty maxim a minim\u016f vypadaj\u00ed naprosto bezcenn\u011b. Nap\u0159\u00edklad takto: <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Bioyino \u2014 distribuovan\u00fd, \u0161k\u00e1lovateln\u00fd agreg\u00e1tor metrik\" src=\"\/wp-content\/uploads\/2019\/11\/4222a1a7d1e127137220595241f5276c.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<em>Jak bylo<\/em><\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Bioyino \u2014 distribuovan\u00fd, \u0161k\u00e1lovateln\u00fd agreg\u00e1tor metrik\" src=\"\/wp-content\/uploads\/2019\/11\/c8f0025c61f708584328ea30aacf7bc0.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<em>Jak m\u011blo b\u00fdt<\/em><\/p>\n<p><\/p>\n<p>Z toho d\u016fvodu jsou sou\u010dty pova\u017eov\u00e1ny za zcela nep\u0159esn\u00e9. P\u0159idejte sem chybu s p\u0159ete\u010den\u00edm 32-bitov\u00e9ho floatu, kter\u00e1 server zcela pos\u00edl\u00e1 do segfaultu p\u0159i p\u0159ij\u00edm\u00e1n\u00ed na pohled nevinn\u00e9 metriky, a je to dokonal\u00e9. Chyba, mimochodem, je\u0161t\u011b nebyla opravena.<\/p>\n<p><\/p>\n<p>A nakonec, <em>V\u00fdhrada X<\/em>. V dob\u011b psan\u00ed tohoto \u010dl\u00e1nku jsme p\u0159ipraveni uk\u00e1zat ji v\u0161em 14 v\u00edcem\u00e9n\u011b funguj\u00edc\u00edm implementac\u00edm statsd, kter\u00e9 se n\u00e1m poda\u0159ilo naj\u00edt. P\u0159edstavme si, \u017ee n\u011bjak\u00e1 konkr\u00e9tn\u00ed infrastruktura vyrostla natolik, \u017ee p\u0159ij\u00edmat 4 miliony MPS u\u017e nesta\u010d\u00ed. Nebo mo\u017en\u00e1 je\u0161t\u011b nevyrostla, ale metriky jsou pro v\u00e1s u\u017e natolik d\u016fle\u017eit\u00e9, \u017ee i kr\u00e1tk\u00e9, 2-3 minutov\u00e9 v\u00fdpadky na grafech mohou b\u00fdt kritick\u00e9 a vyvolat neodolateln\u00e9 deprese u mana\u017eer\u016f. Proto\u017ee l\u00e9\u010den\u00ed deprese je vd\u011b\u010dn\u00e9, pot\u0159ebujeme technick\u00e1 \u0159e\u0161en\u00ed.<\/p>\n<p><\/p>\n<p>Za prv\u00e9, odolnost v\u016f\u010di chyb\u00e1m, aby n\u00e1hl\u00fd probl\u00e9m na serveru nezp\u016fsobil v kancel\u00e1\u0159i psychiatrick\u00fd zombie apokalypsu. Za druh\u00e9, \u0161k\u00e1lovatelnost, abychom m\u011bli mo\u017enost p\u0159ij\u00edmat v\u00edce ne\u017e 4 miliony MPS, ani\u017e bychom museli kopat do j\u00e1dra s\u00ed\u0165ov\u00e9ho stacku Linux a klidn\u011b r\u016fst \u201edo \u0161\u00ed\u0159ky\u201c na po\u017eadovan\u00e9 rozm\u011bry.<\/p>\n<p><\/p>\n<p>Jeliko\u017e jsme m\u011bli rezervy pro \u0161k\u00e1lov\u00e1n\u00ed, rozhodli jsme se za\u010d\u00edt s odolnost\u00ed v\u016f\u010di chyb\u00e1m. \u201eOh! Odolnost v\u016f\u010di chyb\u00e1m! To je snadn\u00e9, to um\u00edme,\u201c pomysleli jsme si a spustili 2 servery, p\u0159i\u010dem\u017e na ka\u017ed\u00e9m jsme zprovoznili kopii brubeck. K tomu jsme museli zkop\u00edrovat provoz s metrikami na oba servery a dokonce jsme napsali pro to <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/Albibek\/udpdup\">mal\u00fd n\u00e1stroj<\/a><\/noindex>. Tento probl\u00e9m odolnosti v\u016f\u010di chyb\u00e1m jsme t\u00edm vy\u0159e\u0161ili, ale\u2026 ne p\u0159\u00edli\u0161 dob\u0159e. Zpo\u010d\u00e1tku to vypadalo docela dob\u0159e: ka\u017ed\u00fd brubeck sb\u00edr\u00e1 sv\u016fj variant agregace, zapisuje data do Graphite ka\u017ed\u00fdch 30 sekund, p\u0159i\u010dem\u017e p\u0159episuje star\u00fd interval (to se d\u011bje na stran\u011b Graphite). Pokud jeden server sel\u017ee, v\u017edy m\u00e1me druh\u00fd s vlastn\u00ed kopi\u00ed agregovan\u00fdch dat. Ale je tu probl\u00e9m: pokud server sel\u017ee, na grafech vznik\u00e1 \u201epila\u201c. To souvis\u00ed s t\u00edm, \u017ee 30sekundov\u00e9 intervaly u brubeck nejsou synchronizov\u00e1ny, a v okam\u017eiku selh\u00e1n\u00ed se jeden z nich nep\u0159episuje. V okam\u017eiku spu\u0161t\u011bn\u00ed druh\u00e9ho serveru se d\u011bje to sam\u00e9. Je to vcelku snesiteln\u00e9, ale cht\u011bli bysme n\u011bco lep\u0161\u00edho! Probl\u00e9m \u0161k\u00e1lovatelnosti tak\u00e9 nezmizel. V\u0161echny metriky st\u00e1le \u201elet\u00ed\u201c na jedin\u00fd server, a proto jsme omezeni t\u011bmi sam\u00fdmi 2-4 miliony MPS v z\u00e1vislosti na s\u00ed\u0165ov\u00e9m v\u00fdkonu.<\/p>\n<p><\/p>\n<p>Pokud se na probl\u00e9m zamysl\u00edme a z\u00e1rove\u0148 budeme hrabat lopatou ve sn\u011bhu, m\u016f\u017ee n\u00e1s napadnout jasn\u00e1 my\u0161lenka: pot\u0159ebujeme statsd, kter\u00fd um\u00ed fungovat v distribuovan\u00e9m re\u017eimu. To znamen\u00e1, \u017ee by m\u011bla b\u00fdt implementov\u00e1na synchronizace mezi uzly podle \u010dasu a metrik. \u201eSamoz\u0159ejm\u011b, takov\u00e9 \u0159e\u0161en\u00ed u\u017e jist\u011b existuje,\u201c \u0159ekli jsme a \u0161li to googlit... A nic jsme nena\u0161li. Po prozkoum\u00e1n\u00ed dokumentace k r\u016fzn\u00fdm statsd (<noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/etsy\/statsd\/wiki#server-implementations\">https:\/\/github.com\/etsy\/statsd\/wiki#server-implementations<\/a><\/noindex> k datu 11.12.2017), jsme nenalezli naprosto nic. Z\u0159ejm\u011b se s takov\u00fdm mno\u017estv\u00edm metrik dosud nesetkali ani v\u00fdvoj\u00e1\u0159i, ani u\u017eivatel\u00e9 t\u011bchto \u0159e\u0161en\u00ed, jinak by ur\u010dit\u011b n\u011bco vymysleli.<\/p>\n<p><\/p>\n<p>A tady jsme si vzpomn\u011bli na \u201ehra\u010dkov\u00fd\u201c statsd \u2014 bioyino, kter\u00fd jsme vyvinuli na hackathonu jen tak pro z\u00e1bavu (n\u00e1zev projektu vygeneroval skript p\u0159ed za\u010d\u00e1tkem hackathonu) a uv\u011bdomili si, \u017ee nal\u00e9hav\u011b pot\u0159ebujeme vlastn\u00ed statsd. Pro\u010d?<\/p>\n<p><\/p>\n<ul>\n<li>proto\u017ee na sv\u011bt\u011b je p\u0159\u00edli\u0161 m\u00e1lo klon\u016f statsd,<\/li>\n<li>proto\u017ee m\u016f\u017eeme zajistit po\u017eadovanou nebo bl\u00edzkou po\u017eadovan\u00e9 odolnost proti selh\u00e1n\u00ed a \u0161k\u00e1lovatelnost (v\u010detn\u011b synchronizace agregovan\u00fdch metrik mezi servery a vy\u0159e\u0161en\u00ed probl\u00e9m\u016f s konflikty p\u0159i odes\u00edl\u00e1n\u00ed),<\/li>\n<li>proto\u017ee m\u016f\u017eeme m\u011b\u0159it metriky p\u0159esn\u011bji, ne\u017e to d\u011bl\u00e1 brubeck,<\/li>\n<li>proto\u017ee si m\u016f\u017eeme samostatn\u011b sb\u00edrat podrobn\u011bj\u0161\u00ed statistiky, kter\u00e9 n\u00e1m brubeck prakticky neposkytoval,<\/li>\n<li>proto\u017ee se n\u00e1m naskytla p\u0159\u00edle\u017eitost naprogramovat na\u0161i vlastn\u00ed hyperv\u00fdkonnostn\u00ed distribuovanou aplikaci, kter\u00e1 nebude zcela kop\u00edrovat architekturu jin\u00e9 takov\u00e9 hyperv\u00fdkonnostn\u00ed\u2026 n\u016fvyponeli. <\/li>\n<\/ul>\n<p><\/p>\n<p>Na \u010dem ps\u00e1t? Pochopiteln\u011b na Rust. Pro\u010d?<\/p>\n<p><\/p>\n<ul>\n<li>proto\u017ee u\u017e existoval prototyp \u0159e\u0161en\u00ed,<\/li>\n<li>proto\u017ee autor \u010dl\u00e1nku v t\u00e9 dob\u011b u\u017e znal Rust a cht\u011bl na n\u011bm napsat n\u011bco pro produk\u010dn\u00ed prost\u0159ed\u00ed s mo\u017enost\u00ed to zve\u0159ejnit jako open-source,<\/li>\n<li>proto\u017ee jazyky s GC n\u00e1m nevyhovuj\u00ed d\u00edky povaze p\u0159ij\u00edman\u00e9ho provozu (prakticky realtime) a pauzy GC jsou prakticky nep\u0159ijateln\u00e9, <\/li>\n<li>proto\u017ee pot\u0159ebujeme maxim\u00e1ln\u00ed v\u00fdkon srovnateln\u00fd s C,<\/li>\n<li>proto\u017ee Rust n\u00e1m poskytuje bezstarostnou soub\u011b\u017enost, a za\u010dneme-li to ps\u00e1t v C\/C++, nabrali bychom je\u0161t\u011b v\u00edc zranitelnost\u00ed, p\u0159ete\u010den\u00ed buffer\u016f, race conditions a dal\u0161\u00edch hrozn\u00fdch slov.<\/li>\n<\/ul>\n<p><\/p>\n<p>Existoval tak\u00e9 argument proti Rustu. Spole\u010dnost nem\u011bla \u017e\u00e1dn\u00e9 zku\u0161enosti s tvorbou projekt\u016f v Rustu a nyn\u00ed tak\u00e9 nepl\u00e1nujeme jeho pou\u017eit\u00ed v hlavn\u00edm projektu. Proto byly v\u00e1\u017en\u00e9 obavy, \u017ee se n\u00e1m nic nepoda\u0159\u00ed, ale rozhodli jsme se risknout a zkusili jsme to.<\/p>\n<p><\/p>\n<p>\u010cas plynul...<\/p>\n<p><\/p>\n<p>Kone\u010dn\u011b, po n\u011bkolika ne\u00fasp\u011b\u0161n\u00fdch pokusech byla p\u0159ipravena prvn\u00ed funk\u010dn\u00ed verze. Co se stalo? Vypadalo to takto.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Bioyino \u2014 distribuovan\u00fd, \u0161k\u00e1lovateln\u00fd agreg\u00e1tor metrik\" src=\"\/wp-content\/uploads\/2019\/11\/322c4136ed033b80b31a89d7b5f63af8.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Ka\u017ed\u00fd uzel p\u0159ij\u00edm\u00e1 sv\u016fj vlastn\u00ed soubor metrik a ukl\u00e1d\u00e1 je u sebe, p\u0159i\u010dem\u017e neagreguje metriky pro ty typy, kde je pro kone\u010dnou agregaci pot\u0159ebn\u00fd jejich pln\u00fd soubor. Uzel jsou mezi sebou spojeny ur\u010dit\u00fdm protokolem distribuovan\u00e9 z\u00e1mky (distributed lock), kter\u00fd umo\u017e\u0148uje vybrat z nich ten jeden (tady jsme plakali), kter\u00fd je hoden odes\u00edlat metriky Velk\u00e9mu. V tuto chv\u00edli je tento probl\u00e9m vy\u0159e\u0161en prost\u0159edky <noindex><a rel=\"nofollow\" href=\"https:\/\/www.consul.io\/docs\/guides\/leader-election.html\">Consul<\/a><\/noindex>, ale v budoucnu se autorovy ambice rozprost\u00edraj\u00ed k <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/Albibek\/raft-consensus\">vlastn\u00edmu<\/a><\/noindex> <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/Albibek\/raft-tokio\">prov\u00e1d\u011bn\u00ed<\/a><\/noindex> Raft, kde ten jedin\u00fd hoden bude, samoz\u0159ejm\u011b, uzel-l\u00eddr konsensu. Krom\u011b konsensu uzly dost \u010dasto (ve v\u00fdchoz\u00edm nastaven\u00ed jednou za sekundu) odes\u00edlaj\u00ed sv\u00fdm soused\u016fm ty \u010d\u00e1sti p\u0159edagregovan\u00fdch metrik, kter\u00e9 se jim poda\u0159ilo za tuto sekundu shrom\u00e1\u017edit. V\u00fdsledkem je, \u017ee \u0161k\u00e1lovatelnost a odolnost proti poruch\u00e1m z\u016fst\u00e1vaj\u00ed zachov\u00e1ny \u2014 ka\u017ed\u00fd z uzl\u016f si st\u00e1le uchov\u00e1v\u00e1 \u00fapln\u00fd soubor metrik, ale metriky jsou nyn\u00ed odes\u00edl\u00e1ny ji\u017e agregovan\u011b, p\u0159es TCP a s k\u00f3dov\u00e1n\u00edm v bin\u00e1rn\u00edm protokolu, co\u017e znamen\u00e1, \u017ee n\u00e1klady na duplicitu v porovn\u00e1n\u00ed s UDP se v\u00fdrazn\u011b sni\u017euj\u00ed. I p\u0159es pom\u011brn\u011b velk\u00e9 mno\u017estv\u00ed p\u0159\u00edchoz\u00edch metrik si akumulace \u017e\u00e1d\u00e1 jen velmi m\u00e1lo pam\u011bti a je\u0161t\u011b m\u00e9n\u011b CPU. Pro na\u0161e dob\u0159e komprimovateln\u00e9 metriky to jsou pouh\u00e9 des\u00edtky megabajt\u016f dat. Dal\u0161\u00edm bonusem je absence zbyte\u010dn\u00fdch p\u0159epis\u016f dat v Graphite, jak tomu bylo v p\u0159\u00edpad\u011b burbeck.<\/p>\n<p><\/p>\n<p>UDP pakety s metrikami jsou vyv\u00e1\u017eeny mezi uzly na s\u00ed\u0165ov\u00e9m za\u0159\u00edzen\u00ed pomoc\u00ed jednoduch\u00e9ho Round Robin. Samoz\u0159ejm\u011b, s\u00ed\u0165ov\u00e9 za\u0159\u00edzen\u00ed nerozli\u0161uje obsah paket\u016f, a proto m\u016f\u017ee zpracov\u00e1vat mnohem v\u00edce ne\u017e 4M paket\u016f za sekundu, nemluv\u011b o metrik\u00e1ch, o kter\u00fdch v\u016fbec nic nev\u00ed. Vzhledem k tomu, \u017ee metriky p\u0159ich\u00e1zej\u00ed ne po jedn\u00e9 v ka\u017ed\u00e9m paketu, neo\u010dek\u00e1v\u00e1me \u017e\u00e1dn\u00e9 probl\u00e9my s v\u00fdkonem na tomto m\u00edst\u011b. V p\u0159\u00edpad\u011b p\u00e1du serveru s\u00ed\u0165ov\u00e9 za\u0159\u00edzen\u00ed rychle (b\u011bhem 1-2 sekund) zjist\u00ed tento fakt a odstran\u00ed spu\u0161t\u011bn\u00fd server z rotace. V d\u016fsledku toho mohou b\u00fdt pasivn\u00ed (tzn. nejsou-li vedouc\u00ed) uzly zap\u00edn\u00e1ny a vyp\u00edn\u00e1ny, ani\u017e by byly na grafech znateln\u00e9 poklesy. Maxim\u00e1ln\u011b ztr\u00e1c\u00edme \u010d\u00e1st metrik, kter\u00e9 p\u0159i\u0161ly za posledn\u00ed sekundu. N\u00e1hl\u00e1 ztr\u00e1ta\/vypnut\u00ed\/p\u0159epnut\u00ed l\u00eddra st\u00e1le vykresl\u00ed malou anom\u00e1lii (30sekundov\u00fd interval je st\u00e1le nesynchronizov\u00e1n), ale p\u0159i zachov\u00e1n\u00ed spojen\u00ed mezi uzly lze tyto probl\u00e9my minimalizovat nap\u0159\u00edklad rozes\u00edl\u00e1n\u00edm synchroniza\u010dn\u00edch paket\u016f.<\/p>\n<p><\/p>\n<p>N\u011bco o vnit\u0159n\u00ed struktu\u0159e. Aplikace je samoz\u0159ejm\u011b v\u00edcerozm\u011brn\u00e1, ale architektura vl\u00e1ken se li\u0161\u00ed od t\u00e9, kter\u00e1 byla pou\u017eita v brubeck. Vl\u00e1ken v brubeck je stejn\u00fdch \u2014 ka\u017ed\u00e9 z nich odpov\u00edd\u00e1 z\u00e1rove\u0148 za sb\u011br informac\u00ed, tak i za agregaci. Ve bioyino jsou pracovn\u00ed vl\u00e1kna (workers) rozd\u011blena do dvou skupin: odpov\u011bdn\u00e9 za s\u00ed\u0165 a odpov\u011bdn\u00e9 za agregaci. Toto rozd\u011blen\u00ed umo\u017e\u0148uje flexibiln\u011bj\u0161\u00ed \u0159\u00edzen\u00ed aplikace v z\u00e1vislosti na typu metrik: tam, kde je pot\u0159ebn\u00e1 intenzivn\u00ed agregace, lze p\u0159idat agreg\u00e1tory, tam, kde je vysok\u00fd s\u00ed\u0165ov\u00fd provoz \u2014 zv\u00fd\u0161it po\u010det s\u00ed\u0165ov\u00fdch vl\u00e1ken. V sou\u010dasn\u00e9 dob\u011b na na\u0161ich serverech pracujeme se 8 s\u00ed\u0165ov\u00fdmi a 4 agreguj\u00edc\u00edmi vl\u00e1kny.<\/p>\n<p><\/p>\n<p>Agrega\u010dn\u00ed \u010d\u00e1st je dost nudn\u00e1. Napln\u011bn\u00e9 s\u00ed\u0165ov\u00fdmi vl\u00e1kny buffery se rozd\u011bluj\u00ed mezi agreguj\u00edc\u00ed vl\u00e1kna, kde jsou d\u00e1le parsov\u00e1ny a agregov\u00e1ny. Na po\u017e\u00e1d\u00e1n\u00ed jsou metriky odes\u00edl\u00e1ny na jin\u00e9 uzly. To v\u0161e, v\u010detn\u011b p\u0159enosu dat mezi uzly a pr\u00e1ce s Consul, se prov\u00e1d\u00ed asynchronn\u011b na r\u00e1mci <noindex><a rel=\"nofollow\" href=\"https:\/\/tokio.rs\">tokio<\/a><\/noindex>.<\/p>\n<p><\/p>\n<p>Mnohem v\u00edce probl\u00e9m\u016f p\u0159i v\u00fdvoji zp\u016fsobila s\u00ed\u0165ov\u00e1 \u010d\u00e1st zodpov\u011bdn\u00e1 za p\u0159\u00edjem metrik. Hlavn\u00edm c\u00edlem odd\u011blen\u00ed s\u00ed\u0165ov\u00fdch tok\u016f do samostatn\u00fdch entit bylo sn\u00ed\u017eit \u010das, kter\u00fd tok str\u00e1v\u00ed <em>ne<\/em> p\u0159i \u010dten\u00ed dat ze socketu. Varianta s pou\u017eit\u00edm asynchronn\u00edho UDP a b\u011b\u017en\u00e9ho recvmsg brzy odpadla: prvn\u00ed spot\u0159ebov\u00e1v\u00e1 p\u0159\u00edli\u0161 mnoho CPU v u\u017eivatelsk\u00e9m prostoru na zpracov\u00e1n\u00ed ud\u00e1lost\u00ed, druh\u00fd \u2014 p\u0159\u00edli\u0161 mnoho p\u0159ep\u00edn\u00e1n\u00ed kontextu. Proto se nyn\u00ed pou\u017e\u00edv\u00e1 <noindex><a rel=\"nofollow\" href=\"https:\/\/linux.die.net\/man\/2\/recvmmsg\">recvmmsg<\/a><\/noindex> s velk\u00fdmi buffery (a buffery, p\u00e1nov\u00e9 d\u016fstojn\u00edci, to nen\u00ed jen tak!). Podpora b\u011b\u017en\u00e9ho UDP byla ponech\u00e1na pro nen\u00e1ro\u010dn\u00e9 p\u0159\u00edpady, kdy nen\u00ed pot\u0159eba recvmmsg. V re\u017eimu multimessage se da\u0159\u00ed dos\u00e1hnout hlavn\u00edho c\u00edle: drtiv\u00e1 v\u011bt\u0161ina \u010dasu s\u00ed\u0165ov\u00fd tok zpracov\u00e1v\u00e1 frontu OS \u2014 \u010dte data ze socketu a p\u0159esouv\u00e1 je do bufferu u\u017eivatelsk\u00e9ho prostoru, jen ob\u010das p\u0159ep\u00edn\u00e1, aby odevzdal napln\u011bn\u00fd buffer agreg\u00e1tor\u016fm. Fronta v socketu se prakticky nenakup\u00ed, po\u010det odm\u00edtnut\u00fdch paket\u016f t\u00e9m\u011b\u0159 neroste. <\/p>\n<p>\n<b class=\"spoiler_title\">Pozn\u00e1mka<\/b><\/p>\n<p>V nastaven\u00ed je v\u00fdchoz\u00ed velikost bufferu nastavena na pom\u011brn\u011b velkou. Pokud se n\u00e1hodou rozhodnete vyzkou\u0161et server samostatn\u011b, mo\u017en\u00e1 se setk\u00e1te s t\u00edm, \u017ee po odesl\u00e1n\u00ed mal\u00e9ho mno\u017estv\u00ed metrik nedoraz\u00ed do Graphite, z\u016fst\u00e1vaj\u00ed v bufferu s\u00ed\u0165ov\u00e9ho toku. Pro pr\u00e1ci s mal\u00fdm mno\u017estv\u00edm metrik je pot\u0159eba v konfiguraci nastavit bufsize a task-queue-size na men\u0161\u00ed hodnoty.<\/p>\n<p><\/p>\n<p>Nakonec \u2014 n\u011bkolik graf\u016f pro milovn\u00edky graf\u016f.<\/p>\n<p><\/p>\n<p>Statistika po\u010dtu p\u0159\u00edchoz\u00edch metrik na ka\u017ed\u00fd server: v\u00edce ne\u017e 2 miliony MPS.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Bioyino \u2014 distribuovan\u00fd, \u0161k\u00e1lovateln\u00fd agreg\u00e1tor metrik\" src=\"\/wp-content\/uploads\/2019\/11\/c6cb1a36c267657d55ec22df518f8d03.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Vypnut\u00ed jednoho z uzl\u016f a p\u0159erozd\u011blen\u00ed p\u0159\u00edchoz\u00edch metrik.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Bioyino \u2014 distribuovan\u00fd, \u0161k\u00e1lovateln\u00fd agreg\u00e1tor metrik\" src=\"\/wp-content\/uploads\/2019\/11\/106af265a6bc2a8b31fe34df69e9c200.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Statistika o odchoz\u00edch metrik\u00e1ch: odes\u00edl\u00e1 v\u017edy pouze jeden uzel \u2014 raidboss.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Bioyino \u2014 distribuovan\u00fd, \u0161k\u00e1lovateln\u00fd agreg\u00e1tor metrik\" src=\"\/wp-content\/uploads\/2019\/11\/582730b501a42bb3a371a84bf2e2394b.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Statistika fungov\u00e1n\u00ed ka\u017ed\u00e9ho uzlu s ohledem na chyby v r\u016fzn\u00fdch modulech syst\u00e9mu.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Bioyino \u2014 distribuovan\u00fd, \u0161k\u00e1lovateln\u00fd agreg\u00e1tor metrik\" src=\"\/wp-content\/uploads\/2019\/11\/101377281c3add8c598c6439ca25d271.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Detailizace p\u0159\u00edchoz\u00edch metrik (n\u00e1zvy metrik jsou skryty).<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Bioyino \u2014 distribuovan\u00fd, \u0161k\u00e1lovateln\u00fd agreg\u00e1tor metrik\" src=\"\/wp-content\/uploads\/2019\/11\/6cf0b61b454b0414799460c482e03242.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Co s t\u00edm v\u0161\u00edm pl\u00e1nujeme d\u011blat d\u00e1l? Samoz\u0159ejm\u011b ps\u00e1t k\u00f3d, bl\u2026! Projekt byl od za\u010d\u00e1tku pl\u00e1nov\u00e1n jako open-source a z\u016fstane tak po celou svou existenci. V nejbli\u017e\u0161\u00edch pl\u00e1nech je p\u0159echod na vlastn\u00ed verzi Raft, zm\u011bna peer protokolu na p\u0159enositeln\u011bj\u0161\u00ed, p\u0159id\u00e1n\u00ed dal\u0161\u00ed vnit\u0159n\u00ed statistiky, nov\u00fdch typ\u016f metrik, oprava chyb a dal\u0161\u00ed vylep\u0161en\u00ed. <\/p>\n<p><\/p>\n<p>Samoz\u0159ejm\u011b v\u00edt\u00e1me v\u0161echny, kte\u0159\u00ed cht\u011bj\u00ed pomoci s rozvojem projektu: vytv\u00e1\u0159ejte PR, Issues, budeme se sna\u017eit odpov\u00eddat a vylep\u0161ovat, apod.<\/p>\n<p><\/p>\n<p>Na tom, jak se \u0159\u00edk\u00e1, to je v\u0161echno, p\u0159\u00e1tel\u00e9, kupujte na\u0161e slony!<\/p>\n<p>\n<center><div class=\"youtube-placeholder\" data-id=\"siCiIyg4ZZY\" onclick=\"loadVideo(this)\">\r\n        <img decoding=\"async\" src=\"https:\/\/img.youtube.com\/vi\/siCiIyg4ZZY\/hqdefault.jpg\" alt=\"P\u0159ehr\u00e1t video\" loading=\"lazy\" width=\"480\" height=\"360\" style=\"width:100%;height:auto;\">\r\n        <div class=\"play-button\"><\/div>\r\n    <\/div><\/center><br \/>\n<br \/>Zdroj: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/avito\/blog\/354714\/\">habr.com<\/a><\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u0418\u0442\u0430\u043a, \u0432\u044b \u0441\u043e\u0431\u0438\u0440\u0430\u0435\u0442\u0435 \u043c\u0435\u0442\u0440\u0438\u043a\u0438. \u041a\u0430\u043a \u0438 \u043c\u044b. \u041c\u044b \u0442\u043e\u0436\u0435 \u0441\u043e\u0431\u0438\u0440\u0430\u0435\u043c \u043c\u0435\u0442\u0440\u0438\u043a\u0438. \u041a\u043e\u043d\u0435\u0447\u043d\u043e \u0436\u0435, \u043d\u0443\u0436\u043d\u044b\u0435 \u0434\u043b\u044f \u0431\u0438\u0437\u043d\u0435\u0441\u0430. \u0421\u0435\u0433\u043e\u0434\u043d\u044f \u043c\u044b \u0440\u0430\u0441\u0441\u043a\u0430\u0436\u0435\u043c \u043e \u0441\u0430\u043c\u043e\u043c \u043f\u0435\u0440\u0432\u043e\u043c \u0437\u0432\u0435\u043d\u0435 \u0441\u0438\u0441\u0442\u0435\u043c\u044b \u043d\u0430\u0448\u0435\u0433\u043e \u043c\u043e\u043d\u0438\u0442\u043e\u0440\u0438\u043d\u0433\u0430 \u2014 statsd-\u0441\u043e\u0432\u043c\u0435\u0441\u0442\u0438\u043c\u043e\u043c \u0441\u0435\u0440\u0432\u0435\u0440\u0435 \u0430\u0433\u0440\u0435\u0433\u0430\u0446\u0438\u0438 bioyino, \u0437\u0430\u0447\u0435\u043c \u043c\u044b \u0435\u0433\u043e \u043d\u0430\u043f\u0438\u0441\u0430\u043b\u0438 \u0438 \u043f\u043e\u0447\u0435\u043c\u0443 \u043e\u0442\u043a\u0430\u0437\u0430\u043b\u0438\u0441\u044c \u043e\u0442 brubeck. \u0418\u0437 \u043f\u0440\u0435\u0434\u044b\u0434\u0443\u0449\u0438\u0445 \u043d\u0430\u0448\u0438\u0445 \u0441\u0442\u0430\u0442\u0435\u0439 (1, 2) \u043c\u043e\u0436\u043d\u043e \u0443\u0437\u043d\u0430\u0442\u044c, \u0447\u0442\u043e \u0434\u043e \u043d\u0435\u043a\u043e\u0442\u043e\u0440\u043e\u0433\u043e \u0432\u0440\u0435\u043c\u0435\u043d\u0438 \u043c\u0435\u0442\u043a\u0438 \u043c\u044b \u0441\u043e\u0431\u0438\u0440\u0430\u043b\u0438 [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-52113","post","type-post","status-publish","format-standard","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.3 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u0418\u0442\u0430\u043a, \u0432\u044b \u0441\u043e\u0431\u0438\u0440\u0430\u0435\u0442\u0435 \u043c\u0435\u0442\u0440\u0438\u043a\u0438. \u041a\u0430\u043a \u0438 \u043c\u044b. \u041c\u044b \u0442\u043e\u0436\u0435 \u0441\u043e\u0431\u0438\u0440\u0430\u0435\u043c \u043c\u0435\u0442\u0440\u0438\u043a\u0438. \u041a\u043e\u043d\u0435\u0447\u043d\u043e \u0436\u0435, \u043d\u0443\u0436\u043d\u044b\u0435 \u0434\u043b\u044f \u0431\u0438\u0437\u043d\u0435\u0441\u0430.\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/bioyino-raspredelyonnyj-masshtabiruemyj-agregator-metrik\" \/>\n\t\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.3\" \/>\n\t\t<meta property=\"og:locale\" content=\"cs_CZ\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47Bioyino \u2014 \u0440\u0430\u0441\u043f\u0440\u0435\u0434\u0435\u043b\u0451\u043d\u043d\u044b\u0439, \u043c\u0430\u0441\u0448\u0442\u0430\u0431\u0438\u0440\u0443\u0435\u043c\u044b\u0439 \u0430\u0433\u0440\u0435\u0433\u0430\u0442\u043e\u0440 \u043c\u0435\u0442\u0440\u0438\u043a | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u0418\u0442\u0430\u043a, \u0432\u044b \u0441\u043e\u0431\u0438\u0440\u0430\u0435\u0442\u0435 \u043c\u0435\u0442\u0440\u0438\u043a\u0438. \u041a\u0430\u043a \u0438 \u043c\u044b. \u041c\u044b \u0442\u043e\u0436\u0435 \u0441\u043e\u0431\u0438\u0440\u0430\u0435\u043c \u043c\u0435\u0442\u0440\u0438\u043a\u0438. \u041a\u043e\u043d\u0435\u0447\u043d\u043e \u0436\u0435, \u043d\u0443\u0436\u043d\u044b\u0435 \u0434\u043b\u044f \u0431\u0438\u0437\u043d\u0435\u0441\u0430.\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/bioyino-raspredelyonnyj-masshtabiruemyj-agregator-metrik\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2019-10-31T21:00:00+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-02-18T10:59:47+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47Bioyino \u2014 distribuovan\u00fd, \u0161k\u00e1lovateln\u00fd agreg\u00e1tor metrik | ProHoster","description":"Tak\u017ee sb\u00edr\u00e1te metriky. Stejn\u011b jako my. My tak\u00e9 sb\u00edr\u00e1me metriky. Samoz\u0159ejm\u011b, ty, kter\u00e9 jsou pot\u0159ebn\u00e9 pro podnik\u00e1n\u00ed.","canonical_url":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/bioyino-raspredelyonnyj-masshtabiruemyj-agregator-metrik","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"cs_CZ","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47Bioyino \u2014 \u0440\u0430\u0441\u043f\u0440\u0435\u0434\u0435\u043b\u0451\u043d\u043d\u044b\u0439, \u043c\u0430\u0441\u0448\u0442\u0430\u0431\u0438\u0440\u0443\u0435\u043c\u044b\u0439 \u0430\u0433\u0440\u0435\u0433\u0430\u0442\u043e\u0440 \u043c\u0435\u0442\u0440\u0438\u043a | ProHoster","og:description":"\u0418\u0442\u0430\u043a, \u0432\u044b \u0441\u043e\u0431\u0438\u0440\u0430\u0435\u0442\u0435 \u043c\u0435\u0442\u0440\u0438\u043a\u0438. \u041a\u0430\u043a \u0438 \u043c\u044b. \u041c\u044b \u0442\u043e\u0436\u0435 \u0441\u043e\u0431\u0438\u0440\u0430\u0435\u043c \u043c\u0435\u0442\u0440\u0438\u043a\u0438. \u041a\u043e\u043d\u0435\u0447\u043d\u043e \u0436\u0435, \u043d\u0443\u0436\u043d\u044b\u0435 \u0434\u043b\u044f \u0431\u0438\u0437\u043d\u0435\u0441\u0430.","og:url":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/bioyino-raspredelyonnyj-masshtabiruemyj-agregator-metrik","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2019-10-31T21:00:00+00:00","article:modified_time":"2020-02-18T10:59:47+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"52113","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":"2026-01-24 02:30:22","breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 20:49:49","updated":"2026-01-24 02:30:22","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts\/52113","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/comments?post=52113"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts\/52113\/revisions"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/media?parent=52113"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/categories?post=52113"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/tags?post=52113"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}