{"id":81888,"date":"2020-05-17T13:42:23","date_gmt":"2020-05-17T11:42:23","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/thanos-masshtabiruemyj-prometheus"},"modified":"2020-05-17T13:42:23","modified_gmt":"2020-05-17T11:42:23","slug":"thanos-masshtabiruemyj-prometheus","status":"publish","type":"post","link":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/thanos-masshtabiruemyj-prometheus","title":{"rendered":"Thanos \u2013 \u0161k\u00e1lovateln\u00fd Prometheus","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

P\u0159eklad \u010dl\u00e1nku byl p\u0159ipraven speci\u00e1ln\u011b pro studenty kurzu \u201eDevOps praktiky a n\u00e1stroje\u201c<\/a><\/noindex>.<\/i><\/b><\/p>\n

\nFabian Reinartz<\/a><\/noindex> \u2013 softwarov\u00fd v\u00fdvoj\u00e1\u0159, fanou\u0161ek Go a milovn\u00edk \u0159e\u0161en\u00ed slo\u017eit\u00fdch probl\u00e9m\u016f. Je tak\u00e9 spr\u00e1vcem Prometheus a spoluzakladatelem Kubernetes SIG instrumentation. D\u0159\u00edve byl produk\u010dn\u00edm in\u017een\u00fdrem ve spole\u010dnosti SoundCloud a vedl monitoring t\u00fdm v CoreOS. V sou\u010dasn\u00e9 dob\u011b pracuje v Googlu.<\/p>\n

Bartek Plotka<\/a><\/noindex> \u2013 infrastruktur\u00e1ln\u00ed in\u017een\u00fdr ve spole\u010dnosti Improbable. Zaj\u00edm\u00e1 se o nov\u00e9 technologie a probl\u00e9my distribuovan\u00fdch syst\u00e9m\u016f. M\u00e1 zku\u0161enosti s n\u00edzko\u00farov\u0148ov\u00fdm programov\u00e1n\u00edm ve spole\u010dnosti Intel, zku\u0161enosti jako p\u0159isp\u011bvatel v Mesos a glob\u00e1ln\u00ed produk\u010dn\u00ed zku\u0161enosti SRE ve spole\u010dnosti Improbable. Pracuje na zlep\u0161en\u00ed sv\u011bta mikroslu\u017eeb. T\u0159i jeho v\u00e1\u0161n\u011b: Golang, open source a volejbal.<\/i><\/p>\n

S ohledem na n\u00e1\u0161 vlajkov\u00fd produkt SpatialOS si m\u016f\u017eete domyslet, \u017ee pro spole\u010dnost Improbable je pot\u0159eba vysoce v\u00fdkonn\u00e1 cloudov\u00e1 infrastruktura glob\u00e1ln\u00edho m\u011b\u0159\u00edtka s des\u00edtkami Kubernetes cluster\u016f. Byli jsme jedni z prvn\u00edch, kdo za\u010dal pou\u017e\u00edvat monitorovac\u00ed syst\u00e9m Prometheus<\/a><\/noindex>. Prometheus dok\u00e1\u017ee sledovat miliony metrik v re\u00e1ln\u00e9m \u010dase a dod\u00e1v\u00e1 se s v\u00fdkonn\u00fdm dotazovac\u00edm jazykem, kter\u00fd umo\u017e\u0148uje extrakci pot\u0159ebn\u00fdch informac\u00ed.<\/p>\n

Jednoduchost a spolehlivost Prometheus je jednou z jeho hlavn\u00edch v\u00fdhod. Nicm\u00e9n\u011b p\u0159i ur\u010dit\u00e9m m\u011b\u0159\u00edtku jsme se setkali s n\u011bkolika nedostatky. Pro \u0159e\u0161en\u00ed t\u011bchto probl\u00e9m\u016f jsme vyvinuli Thanos<\/a><\/noindex> \u2013 projekt s otev\u0159en\u00fdm zdrojov\u00fdm k\u00f3dem, vytvo\u0159en\u00fd spole\u010dnost\u00ed Improbable, pro bezprobl\u00e9movou transformaci existuj\u00edc\u00edch cluster\u016f Prometheus na jednotn\u00fd monitorovac\u00ed syst\u00e9m s neomezen\u00fdm \u00falo\u017ei\u0161t\u011bm historick\u00fdch dat. Thanos je dostupn\u00fd na Githubu. zde<\/a><\/noindex>.<\/a><\/noindex><\/p>\n

Bu\u010fte informov\u00e1ni o nejnov\u011bj\u0161\u00edch zpr\u00e1v\u00e1ch od Improbable.<\/a><\/noindex><\/p>\n

Na\u0161e c\u00edle s Thanosem<\/h2>\n

\nP\u0159i ur\u010dit\u00e9m m\u011b\u0159\u00edtku vznikaj\u00ed probl\u00e9my, kter\u00e9 p\u0159ekra\u010duj\u00ed mo\u017enosti vanilkov\u00e9ho Prometheusu. Jak spolehliv\u011b a ekonomicky ukl\u00e1dat petabajty historick\u00fdch dat? Je mo\u017en\u00e9 to ud\u011blat bez zhor\u0161en\u00ed doby odezvy na dotazy? Lze p\u0159istupovat ke v\u0161em metrik\u00e1m rozlo\u017een\u00fdm na r\u016fzn\u00fdch serverech Prometheus pomoc\u00ed jednoho API dotazu? Lze n\u011bjak\u00fdm zp\u016fsobem slou\u010dit replikovan\u00e1 data shrom\u00e1\u017ed\u011bn\u00e1 pomoc\u00ed Prometheus HA?<\/p>\n

Pro \u0159e\u0161en\u00ed t\u011bchto ot\u00e1zek jsme vytvo\u0159ili Thanos. V n\u00e1sleduj\u00edc\u00edch odd\u00edlech popisujeme, jak jsme p\u0159istoupili k \u0159e\u0161en\u00ed t\u011bchto probl\u00e9m\u016f a vysv\u011btlujeme c\u00edle, kter\u00e9 jsme sledovali.<\/p>\n

Dotazov\u00e1n\u00ed dat z v\u00edce instanc\u00ed Prometheus (glob\u00e1ln\u00ed dotaz)<\/h4>\n

\nPrometheus nab\u00edz\u00ed funk\u010dn\u00ed p\u0159\u00edstup k shardingu. I jeden server Prometheus poskytuje dostate\u010dnou \u0161k\u00e1lovatelnost, aby uvolnil u\u017eivatele od slo\u017eitost\u00ed horizont\u00e1ln\u00edho shardingu t\u00e9m\u011b\u0159 ve v\u0161ech sc\u00e9n\u00e1\u0159\u00edch pou\u017eit\u00ed.<\/p>\n

I kdy\u017e je to skv\u011bl\u00fd model nasazen\u00ed, \u010dasto je t\u0159eba z\u00edskat p\u0159\u00edstup k dat\u016fm na r\u016fzn\u00fdch serverech Prometheus prost\u0159ednictv\u00edm jednotn\u00e9ho API nebo UI \u2014 global view. Samoz\u0159ejm\u011b je mo\u017en\u00e9 zobrazit n\u011bkolik dotaz\u016f na jednom panelu Grafana, ale ka\u017ed\u00fd dotaz m\u016f\u017ee b\u00fdt proveden pouze na jednom serveru Prometheus. Na druhou stranu, pomoc\u00ed Thanos m\u016f\u017eete dotazovat a agregovat data z n\u011bkolika server\u016f Prometheus, proto\u017ee v\u0161echny jsou k dispozici z jednoho koncov\u00e9ho bodu.<\/p>\n

D\u0159\u00edve jsme pro z\u00edsk\u00e1n\u00ed global view v Improbable zorganizovali na\u0161e instance Prometheus do v\u00edcestup\u0148ov\u00e9 Hierarchical Federation<\/a><\/noindex>. To znamenalo vytvo\u0159en\u00ed jednoho meta-serveru Prometheus, kter\u00fd shroma\u017e\u010fuje \u010d\u00e1st metrik z ka\u017ed\u00e9ho \"listov\u00e9ho\" serveru.<\/p>\n

\"Thanos
\n
\nTento p\u0159\u00edstup se uk\u00e1zal jako problematick\u00fd. Vedl k slo\u017eit\u011bj\u0161\u00ed konfiguraci, p\u0159id\u00e1n\u00ed dal\u0161\u00ed potenci\u00e1ln\u00edho bodu selh\u00e1n\u00ed a pou\u017eit\u00ed slo\u017eit\u00fdch pravidel pro poskytov\u00e1n\u00ed federovan\u00e9 koncov\u00e9mu bodu pouze pot\u0159ebn\u00e1 data. Krom\u011b toho takov\u00e1 federace neumo\u017e\u0148uje z\u00edskat skute\u010dn\u00e9 global view, proto\u017ee v\u0161echna data nejsou dostupn\u00e1 z jednoho API dotazu.<\/p>\n

S t\u00edm \u00fazce souvis\u00ed jednotn\u00e9 zobrazen\u00ed dat shrom\u00e1\u017ed\u011bn\u00fdch na vysoce dostupn\u00fdch (high-availability, HA) serverech Prometheus. HA model Prometheus sb\u00edr\u00e1 data nez\u00e1visle dvakr\u00e1t, co\u017e je tak jednoduch\u00e9, \u017ee snaz\u0161\u00ed to b\u00fdt nem\u016f\u017ee. P\u0159esto by bylo mnohem pohodln\u011bj\u0161\u00ed pou\u017e\u00edvat slou\u010den\u00e9 a deduplifikovan\u00e9 zobrazen\u00ed obou tok\u016f.<\/p>\n

Samoz\u0159ejm\u011b je v serverech Prometheus s vysokou dostupnost\u00ed pot\u0159eba. V Improbable se opravdu v\u00e1\u017en\u011b v\u011bnujeme minutov\u00e9mu monitorov\u00e1n\u00ed dat, ale m\u00edt jeden instance Prometheus v clusteru je jedin\u00fd bod selh\u00e1n\u00ed. Jak\u00e1koliv chybov\u00e1 konfigurace nebo selh\u00e1n\u00ed hardwaru mohou potenci\u00e1ln\u011b v\u00e9st ke ztr\u00e1t\u011b d\u016fle\u017eit\u00fdch dat. I jednoduch\u00e9 nasazen\u00ed m\u016f\u017ee v\u00e9st k drobn\u00fdm v\u00fdpadk\u016fm p\u0159i shroma\u017e\u010fov\u00e1n\u00ed metrik, proto\u017ee restart m\u016f\u017ee b\u00fdt v\u00fdrazn\u011b del\u0161\u00ed ne\u017e interval sb\u011bru.<\/p>\n

Spolehliv\u00e9 uchov\u00e1v\u00e1n\u00ed historick\u00fdch dat<\/h4>\n

\nLevn\u00e9, rychl\u00e9 a dlouhodob\u00e9 \u00falo\u017ei\u0161t\u011b metrik je na\u0161\u00edm snem (sd\u00edlen\u00fdm v\u011bt\u0161inou u\u017eivatel\u016f Prometheus). V Improbable jsme byli nuceni nastavit dobu uchov\u00e1v\u00e1n\u00ed metrik na dev\u011bt dn\u00ed (pro Prometheus 1.8). To p\u0159in\u00e1\u0161\u00ed zjevn\u00e1 omezen\u00ed v tom, jak daleko m\u016f\u017eeme pohl\u00e9dnout zp\u011bt.<\/p>\n

Prometheus 2.0 se v tomto ohledu zlep\u0161il, proto\u017ee po\u010det \u010dasov\u00fdch \u0159ad ji\u017e nem\u00e1 vliv na celkov\u00fd v\u00fdkon serveru (viz KubeCon keynote o Prometheus 2<\/a><\/noindex>). P\u0159esto Prometheus uchov\u00e1v\u00e1 data na m\u00edstn\u00edm disku. A\u010dkoli vysoce \u00fa\u010dinn\u00e9 zpracov\u00e1n\u00ed dat m\u016f\u017ee v\u00fdznamn\u011b sn\u00ed\u017eit vyu\u017eit\u00ed m\u00edstn\u00edho SSD, st\u00e1le existuje omezen\u00ed na objem uchov\u00e1van\u00fdch historick\u00fdch dat.<\/p>\n

Krom\u011b toho n\u00e1m v Improbable z\u00e1le\u017e\u00ed na spolehlivosti, jednoduchosti a n\u00e1kladech. Velk\u00e9 m\u00edstn\u00ed disky jsou slo\u017eit\u011bj\u0161\u00ed na \u00fadr\u017ebu a z\u00e1lohov\u00e1n\u00ed. Jsou dra\u017e\u0161\u00ed a vy\u017eaduj\u00ed v\u00edce n\u00e1stroj\u016f pro z\u00e1lohov\u00e1n\u00ed, co\u017e vede k nadm\u011brn\u00e9 slo\u017eitosti.<\/p>\n

Sn\u00ed\u017een\u00ed vzorkov\u00e1n\u00ed<\/h4>\n

\nJakmile jsme za\u010dali pracovat s historick\u00fdmi daty, uv\u011bdomili jsme si, \u017ee existuj\u00ed z\u00e1kladn\u00ed komplikace s O-velk\u00fdm, kter\u00e9 d\u011blaj\u00ed dotazy st\u00e1le pomalej\u0161\u00edmi, pokud pracujeme s daty za t\u00fddny, m\u011bs\u00edce a roky.<\/p>\n

Standardn\u00edm \u0159e\u0161en\u00edm tohoto probl\u00e9mu bude sn\u00ed\u017een\u00ed vzorkov\u00e1n\u00ed<\/a><\/noindex> (downsampling) \u2014 sn\u00ed\u017een\u00ed frekvence vzorkov\u00e1n\u00ed sign\u00e1lu. Pomoc\u00ed sn\u00ed\u017een\u00ed vzorkov\u00e1n\u00ed m\u016f\u017eeme \"p\u0159e\u0161k\u00e1lovat\" na del\u0161\u00ed \u010dasov\u00e9 obdob\u00ed a udr\u017eet p\u016fvodn\u00ed po\u010det vzork\u016f, co\u017e umo\u017en\u00ed zachovat rychlost dotaz\u016f.<\/p>\n

Sn\u00ed\u017een\u00ed vzorkov\u00e1n\u00ed star\u00fdch dat je nevyhnutelnou sou\u010d\u00e1st\u00ed jak\u00e9hokoli \u0159e\u0161en\u00ed pro dlouhodob\u00e9 uchov\u00e1v\u00e1n\u00ed a p\u0159esahuje mo\u017enosti vanilkov\u00e9ho Prometheus.<\/p>\n

Dal\u0161\u00ed c\u00edle<\/h4>\n

\nJedn\u00edm z prvotn\u00edch c\u00edl\u016f projektu Thanos byla bezprobl\u00e9mov\u00e1 integrace s jakoukoli existuj\u00edc\u00ed instalac\u00ed Prometheus. Druh\u00fdm c\u00edlem byla snadn\u00e1 \u00fadr\u017eba s minim\u00e1ln\u00edmi vstupn\u00edmi p\u0159ek\u00e1\u017ekami. Jak\u00e9koli z\u00e1vislosti by m\u011bly b\u00fdt snadno splniteln\u00e9 jak pro mal\u00e9, tak pro velk\u00e9 u\u017eivatele, co\u017e tak\u00e9 znamen\u00e1 n\u00edzk\u00e9 z\u00e1kladn\u00ed n\u00e1klady.<\/p>\n

Architektura Thanos<\/h2>\n

\nPot\u00e9, co jsme v p\u0159edchoz\u00edm odd\u00edle uvedli na\u0161e c\u00edle, poj\u010fme se s nimi vypo\u0159\u00e1dat a pod\u00edvat se, jak Thanos \u0159e\u0161\u00ed tyto probl\u00e9my.<\/p>\n

Glob\u00e1ln\u00ed pohled<\/h4>\n

\nAbychom z\u00edskali global view nad existuj\u00edc\u00edmi instancemi Prometheu, mus\u00edme spojit jednotn\u00fd vstupn\u00ed bod dotaz\u016f se v\u0161emi servery. To je \u00fakol komponenty Thanos. Sidecar<\/a><\/noindex>. Nasazuje se vedle ka\u017ed\u00e9ho serveru Promethea a funguje jako proxy, obsluhuj\u00edc\u00ed m\u00edstn\u00ed data Promethea prost\u0159ednictv\u00edm gRPC rozhran\u00ed Store API, kter\u00e9 umo\u017e\u0148uje vyb\u00edrat \u010dasov\u00e9 \u0159ady podle zna\u010dek a \u010dasov\u00e9ho rozmez\u00ed.<\/p>\n

Na druh\u00e9 stran\u011b se nach\u00e1z\u00ed horizont\u00e1ln\u011b \u0161k\u00e1lovateln\u00e1 komponenta Querier bez udr\u017eov\u00e1n\u00ed stavu, kter\u00e1 d\u011bl\u00e1 trochu v\u00edce ne\u017e jen odpov\u00edd\u00e1 na dotazy PromQL prost\u0159ednictv\u00edm standardn\u00edho HTTP API Promethea. Komponenten Querier, Sidecar a dal\u0161\u00ed Thanos komunikuj\u00ed prost\u0159ednictv\u00edm gossip protokolu.<\/a><\/noindex>.<\/p>\n

\"Thanos
\n<\/p>\n

    \n
  1. Querier p\u0159i p\u0159\u00edjmu dotazu p\u0159ipojuje k p\u0159\u00edslu\u0161n\u00e9mu serveru Store API, tedy k na\u0161im Sidecar\u016fm, a z\u00edsk\u00e1v\u00e1 \u010dasov\u00e9 \u0159ady z p\u0159\u00edslu\u0161n\u00fdch server\u016f Promethea.<\/li>\n
  2. Pot\u00e9 spojuje odpov\u011bdi a prov\u00e1d\u00ed na nich dotaz PromQL. Querier m\u016f\u017ee kombinovat jak nep\u0159ekr\u00fdvaj\u00edc\u00ed se data, tak duplikovan\u00e1 data z HA server\u016f Promethea.<\/li>\n<\/ol>\n

    \nTo \u0159e\u0161\u00ed v\u011bt\u0161inu na\u0161\u00ed h\u00e1danky \u2014 spojen\u00ed dat z izolovan\u00fdch server\u016f Promethea do jednoho pohledu. Vlastn\u011b lze Thanos pou\u017e\u00edt pouze kv\u016fli t\u00e9to funk\u010dnosti. Do existuj\u00edc\u00edch server\u016f Promethea nen\u00ed t\u0159eba prov\u00e1d\u011bt \u017e\u00e1dn\u00e9 zm\u011bny!<\/p>\n

    Neomezen\u00e1 doba uchov\u00e1v\u00e1n\u00ed!<\/h4>\n

    \nAv\u0161ak d\u0159\u00edve \u010di pozd\u011bji budeme cht\u00edt uchovat data, kter\u00e1 p\u0159ekra\u010duj\u00ed b\u011b\u017enou dobu uchov\u00e1v\u00e1n\u00ed Promethea. Pro uchov\u00e1v\u00e1n\u00ed historick\u00fdch dat jsme zvolili objektov\u00e9 \u00falo\u017ei\u0161t\u011b. Je \u0161iroce dostupn\u00e9 v jak\u00e9mkoli cloudu, stejn\u011b jako v m\u00edstn\u00edch datov\u00fdch centrech, a je velmi ekonomick\u00e9. Nav\u00edc je prakticky jak\u00e9koli objektov\u00e9 \u00falo\u017ei\u0161t\u011b dostupn\u00e9 p\u0159es dob\u0159e zn\u00e1m\u00e9 S3 API.<\/p>\n

    Prometheus zapisuje data z opera\u010dn\u00ed pam\u011bti na disk p\u0159ibli\u017en\u011b ka\u017ed\u00e9 dv\u011b hodiny. Blok ukl\u00e1dan\u00fdch dat obsahuje v\u0161echna data pro pevn\u00e9 \u010dasov\u00e9 obdob\u00ed a je nem\u011bnn\u00fd. To je velmi pohodln\u00e9, proto\u017ee Thanos Sidecar m\u016f\u017ee jednodu\u0161e sledovat adres\u00e1\u0159 dat Promethea a jakmile se objev\u00ed nov\u00e9 bloky, nahr\u00e1t je do kbel\u00edk\u016f objektov\u00e9ho \u00falo\u017ei\u0161t\u011b.<\/p>\n

    \"Thanos
    \n
    \nNahr\u00e1v\u00e1n\u00ed do objektov\u00e9ho \u00falo\u017ei\u0161t\u011b hned po z\u00e1pisu na disk tak\u00e9 umo\u017e\u0148uje udr\u017eet jednoduchost \u201escraperu\u201c (Prometheus a Thanos Sidecar). To usnad\u0148uje \u00fadr\u017ebu, n\u00e1klady a design syst\u00e9mu.<\/p>\n

    Jak vid\u00edte, z\u00e1lohov\u00e1n\u00ed dat se prov\u00e1d\u00ed velmi jednodu\u0161e. Ale co dotazy na data v objektov\u00e9m \u00falo\u017ei\u0161ti?<\/p>\n

    Komponenta Thanos Store p\u016fsob\u00ed jako proxy pro na\u010d\u00edt\u00e1n\u00ed dat z objektov\u00e9ho \u00falo\u017ei\u0161t\u011b. Stejn\u011b jako Thanos Sidecar se \u00fa\u010dastn\u00ed gossip clusteru a implementuje Store API. Existuj\u00edc\u00ed Querier je tak m\u016f\u017ee vn\u00edmat jako Sidecar, jako dal\u0161\u00ed zdroj \u010dasov\u00fdch \u0159ad dat \u2014 \u017e\u00e1dn\u00e9 speci\u00e1ln\u00ed nastaven\u00ed nen\u00ed pot\u0159eba.<\/p>\n

    \"Thanos
    \n
    \nBloky \u010dasov\u00fdch \u0159ad dat se skl\u00e1daj\u00ed z n\u011bkolika velk\u00fdch soubor\u016f. Na\u010d\u00edt\u00e1n\u00ed na vy\u017e\u00e1d\u00e1n\u00ed by bylo pom\u011brn\u011b neefektivn\u00ed a m\u00edstn\u00ed cache by vy\u017eadovala obrovsk\u00e9 mno\u017estv\u00ed pam\u011bti a \u00falo\u017en\u00e9ho prostoru.<\/p>\n

    Store Gateway m\u00edsto toho v\u00ed, jak pracovat s form\u00e1tem \u00falo\u017ei\u0161t\u011b Prometheus. D\u00edky inteligentn\u00edmu pl\u00e1nova\u010di dotaz\u016f a ukl\u00e1d\u00e1n\u00ed pouze pot\u0159ebn\u00fdch indexov\u00fdch \u010d\u00e1st\u00ed blok\u016f se poda\u0159ilo zjednodu\u0161it slo\u017eit\u00e9 dotazy na minim\u00e1ln\u00ed po\u010det HTTP po\u017eadavk\u016f na soubory objektov\u00e9ho \u00falo\u017ei\u0161t\u011b. T\u00edm se lze dostat k po\u010dtu dotaz\u016f sn\u00ed\u017een\u00e9mu o \u010dty\u0159i a\u017e \u0161est \u0159\u00e1d\u016f a dos\u00e1hnout doby odezvy, kter\u00e1 se celkov\u011b t\u011b\u017eko odli\u0161uje od dotaz\u016f na data na m\u00edstn\u00edm SSD.<\/p>\n

    \"Thanos
    \n
    \nJak ukazuje v\u00fd\u0161e uveden\u00fd diagram, Thanos Querier v\u00fdznamn\u011b sni\u017euje n\u00e1klady na jeden dotaz na data v objektov\u00e9m \u00falo\u017ei\u0161ti, pou\u017e\u00edvaj\u00edc\u00ed form\u00e1t \u00falo\u017ei\u0161t\u011b Prometheus a umis\u0165uj\u00edc\u00ed souvisej\u00edc\u00ed data bl\u00edzko sebe. S t\u00edmto p\u0159\u00edstupem m\u016f\u017eeme slou\u010dit mno\u017estv\u00ed jednotliv\u00fdch dotaz\u016f do minim\u00e1ln\u00edho po\u010dtu hromadn\u00fdch operac\u00ed.<\/p>\n

    Kompakce a downsampling<\/h4>\n

    \nJakmile je nov\u00fd blok \u010dasov\u00fdch \u0159ad dat \u00fasp\u011b\u0161n\u011b nahr\u00e1n do objektov\u00e9ho \u00falo\u017ei\u0161t\u011b, pova\u017eujeme ho za \u201ehistorick\u00e1\u201c data, kter\u00e1 jsou ihned k dispozici prost\u0159ednictv\u00edm Store Gateway.<\/p>\n

    Po n\u011bjak\u00e9 dob\u011b se bloky z jednoho zdroje (Prometheus se Sidecar) hromad\u00ed a ji\u017e nevyu\u017e\u00edvaj\u00ed cel\u00fd potenci\u00e1l indexace. Pro vy\u0159e\u0161en\u00ed tohoto probl\u00e9mu jsme zavedli dal\u0161\u00ed komponentu nazvanou Compactor. Ta jednodu\u0161e aplikuje m\u00edstn\u00ed mechanismus kompakce Prometheus na historick\u00e1 data v objektov\u00e9m \u00falo\u017ei\u0161ti a m\u016f\u017ee b\u00fdt spu\u0161t\u011bna jako jednoduch\u00fd periodick\u00fd d\u00e1vkov\u00fd \u00fakol.<\/p>\n

    \"Thanos
    \n
    \nD\u00edky efektivn\u00edmu kompresi nen\u00ed pro dlouhodob\u00fd po\u017eadavek na \u00falo\u017ei\u0161t\u011b probl\u00e9mem velikost dat. Potenci\u00e1ln\u00ed n\u00e1klady na dekomprese miliardy hodnot a jejich zpracov\u00e1n\u00ed prost\u0159ednictv\u00edm dotazov\u00e9ho zpracovatele v\u0161ak nevyhnuteln\u011b povedou k dramatick\u00e9mu zv\u00fd\u0161en\u00ed doby prov\u00e1d\u011bn\u00ed dotaz\u016f. Na druhou stranu, proto\u017ee na ka\u017ed\u00fd pixel obrazovky p\u0159ipad\u00e1 stovky datov\u00fdch bod\u016f, st\u00e1v\u00e1 se vizualizace dat v pln\u00e9m rozli\u0161en\u00ed nemo\u017enou. Downsampling je tedy nejen mo\u017en\u00fd, ale tak\u00e9 nezp\u016fsob\u00ed v\u00fdznamnou ztr\u00e1tu p\u0159esnosti. <\/p>\n

    \"Thanos
    \n
    \nPro downsampling dat Compactor nep\u0159etr\u017eit\u011b agreguje data s rozli\u0161en\u00edm p\u011bt minut a jedna hodina. Pro ka\u017ed\u00fd nezpracovan\u00fd fragment, kter\u00fd je k\u00f3dov\u00e1n pomoc\u00ed TSDB XOR komprese, jsou ukl\u00e1d\u00e1ny r\u016fzn\u00e9 typy agregovan\u00fdch dat, jako nap\u0159. min, max nebo sum pro jeden blok. To umo\u017e\u0148uje Querieru automaticky vybrat agreg\u00e1t, kter\u00fd je vhodn\u00fd pro dan\u00fd PromQL dotaz. <\/p>\n

    K pou\u017e\u00edv\u00e1n\u00ed dat s ni\u017e\u0161\u00ed p\u0159esnost\u00ed u\u017eivatel nemus\u00ed m\u00edt \u017e\u00e1dnou speci\u00e1ln\u00ed konfiguraci. Querier se automaticky p\u0159ep\u00edn\u00e1 mezi r\u016fzn\u00fdmi rozli\u0161en\u00edmi a nezpracovan\u00fdmi daty, kdy\u017e u\u017eivatel zv\u011bt\u0161uje a zmen\u0161uje m\u011b\u0159\u00edtko. Pokud si to u\u017eivatel p\u0159eje, m\u016f\u017ee to \u0159\u00eddit p\u0159\u00edmo pomoc\u00ed parametru \u201cstep\u201d ve dotazu. <\/p>\n

    Proto\u017ee n\u00e1klady na ulo\u017een\u00e9 1 GB jsou mal\u00e9, Thanos ve v\u00fdchoz\u00edm nastaven\u00ed uchov\u00e1v\u00e1 p\u016fvodn\u00ed data, data s rozli\u0161en\u00edm p\u011bt minut a jednu hodinu. Nen\u00ed pot\u0159eba mazat p\u016fvodn\u00ed data.<\/p>\n

    Recording rules<\/h2>\n

    \nI s Thanos jsou recording rules d\u016fle\u017eitou sou\u010d\u00e1st\u00ed monitorovac\u00edho stacku. Sni\u017euj\u00ed slo\u017eitost, latenci a n\u00e1klady na dotazy. Jsou tak\u00e9 u\u017eite\u010dn\u00e9 pro u\u017eivatele, kte\u0159\u00ed pot\u0159ebuj\u00ed agregovan\u00e1 data o metrik\u00e1ch. Thanos je zalo\u017een na vanilkov\u00fdch instanc\u00edch Promethea, a proto je zcela p\u0159\u00edpustn\u00e9 uchov\u00e1vat recording rules a alerting rules na st\u00e1vaj\u00edc\u00edm serveru Promethea. Nicm\u00e9n\u011b v n\u011bkter\u00fdch p\u0159\u00edpadech to m\u016f\u017ee b\u00fdt nedostate\u010dn\u00e9:<\/p>\n