{"id":79467,"date":"2020-04-27T07:42:23","date_gmt":"2020-04-27T05:42:23","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/operativnaya-analitika-v-mikroservisnoj-arhitekture-p%cc%b6o%cc%b6n%cc%b6ya%cc%b6t%cc%b6%cc%b6-%cc%b6i%cc%b6-%cc%b6p%cc%b6r%cc%b6o%cc%b6s%cc%b6t%cc%b6i%cc%b6t%cc%b6%cc%b6-pomoch-i-podskazat-postgres"},"modified":"2020-04-27T07:42:23","modified_gmt":"2020-04-27T05:42:23","slug":"operativnaya-analitika-v-mikroservisnoj-arhitekture-p%cc%b6o%cc%b6n%cc%b6ya%cc%b6t%cc%b6%cc%b6-%cc%b6i%cc%b6-%cc%b6p%cc%b6r%cc%b6o%cc%b6s%cc%b6t%cc%b6i%cc%b6t%cc%b6%cc%b6-pomoch-i-podskazat-postgres","status":"publish","type":"post","link":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/operativnaya-analitika-v-mikroservisnoj-arhitekture-p%cc%b6o%cc%b6n%cc%b6ya%cc%b6t%cc%b6%cc%b6-%cc%b6i%cc%b6-%cc%b6p%cc%b6r%cc%b6o%cc%b6s%cc%b6t%cc%b6i%cc%b6t%cc%b6%cc%b6-pomoch-i-podskazat-postgres","title":{"rendered":"Opera\u010dn\u00ed analitika v mikroslu\u017ebov\u00e9 architektu\u0159e: p\u0336o\u0336n\u0336y\u0336m\u0336e\u0336j\u0336 \u0336i\u0336 \u0336p\u0336o\u0336m\u0336o\u0336\u017e\u0336 te\u010f porad\u00edme Postgres FDW","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

Mikroslu\u017ebov\u00e1 architektura, stejn\u011b jako cokoliv jin\u00e9ho na tomto sv\u011bt\u011b, m\u00e1 sv\u00e9 v\u00fdhody a nev\u00fdhody. N\u011bkter\u00e9 procesy se s n\u00ed zjednodu\u0161uj\u00ed, jin\u00e9 se naopak komplikuje. A ve prosp\u011bch rychlosti zm\u011bn a lep\u0161\u00ed \u0161k\u00e1lovatelnosti je nutn\u00e9 p\u0159in\u00e1\u0161et ob\u011bti. Jednou z nich je slo\u017eitost analytiky. Pokud lze v monolitu ve\u0161kerou operativn\u00ed analytiku redukovat na SQL dotazy na analytickou replikaci, pak v multi-slu\u017ebov\u00e9 architektu\u0159e m\u00e1 ka\u017ed\u00e1 slu\u017eba svou datab\u00e1zi a zd\u00e1 se, \u017ee jedn\u00edm dotazem se to nezvl\u00e1dne (ale mo\u017en\u00e1 to zvl\u00e1dne?). Pro ty, kte\u0159\u00ed se zaj\u00edmaj\u00ed, jak jsme vy\u0159e\u0161ili probl\u00e9m operativn\u00ed analytiky ve na\u0161\u00ed spole\u010dnosti a jak jsme se nau\u010dili \u017e\u00edt s t\u00edmto \u0159e\u0161en\u00edm \u2014 v\u00edtejte.<\/p>\n

\"Opera\u010dn\u00ed
\nJmenuji se Pavel Siva\u0161, v DomClicku pracuji v t\u00fdmu, kter\u00fd se star\u00e1 o \u00fadr\u017ebu analytick\u00e9ho datov\u00e9ho skladu. Na\u0161e \u010dinnost lze podm\u00edn\u011bn\u011b za\u0159adit do oblasti datov\u00e9ho in\u017een\u00fdrstv\u00ed, ale ve skute\u010dnosti je spektrum \u00fakol\u016f mnohem \u0161ir\u0161\u00ed. Existuj\u00ed standardn\u00ed \u00fakoly pro datov\u00e9 in\u017een\u00fdrstv\u00ed ETL\/ELT, podpora a p\u0159izp\u016fsoben\u00ed n\u00e1stroj\u016f pro anal\u00fdzu dat a v\u00fdvoj vlastn\u00edch n\u00e1stroj\u016f. Konkr\u00e9tn\u011b pro operativn\u00ed reportov\u00e1n\u00ed jsme se rozhodli \u201ep\u0159edst\u00edrat\u201c, \u017ee m\u00e1me monolit a poskytnout analytik\u016fm jednu datab\u00e1zi, v n\u00ed\u017e budou v\u0161echna pot\u0159ebn\u00e1 data. <\/a><\/noindex><\/p>\n

Ve skute\u010dnosti jsme zva\u017eovali r\u016fzn\u00e9 mo\u017enosti. Mohli jsme vybudovat plnohodnotn\u00e9 \u00falo\u017ei\u0161t\u011b \u2014 dokonce jsme to zkusili, ale up\u0159\u00edmn\u011b \u0159e\u010deno, nedok\u00e1zali jsme sladit dost \u010dast\u00e9 zm\u011bny v logice s dost pomal\u00fdm procesem budov\u00e1n\u00ed \u00falo\u017ei\u0161t\u011b a prov\u00e1d\u011bn\u00edm zm\u011bn (pokud se to n\u011bkomu poda\u0159ilo, napi\u0161te do koment\u00e1\u0159\u016f, jak). Mohli jsme \u0159\u00edct analytik\u016fm: \u201eChlapci, u\u010dte se Python a p\u0159ejd\u011bte do analytick\u00fdch replik\u201c, ale to by znamenalo dal\u0161\u00ed po\u017eadavek na v\u00fdb\u011br person\u00e1lu, a zd\u00e1lo se, \u017ee se tomu chceme vyhnout, pokud to jde. Rozhodli jsme se vyzkou\u0161et technologii FDW (Foreign Data Wrapper): to je v podstat\u011b standardn\u00ed dblink, kter\u00fd existuje ve standardu SQL, ale s mnohem p\u0159\u00edv\u011btiv\u011bj\u0161\u00edm rozhran\u00edm. Na jeho z\u00e1klad\u011b jsme vytvo\u0159ili \u0159e\u0161en\u00ed, kter\u00e9 se nakonec uchytilo, na n\u011bm\u017e jsme se zastavili. Jeho podrobnosti jsou t\u00e9matem samostatn\u00e9ho \u010dl\u00e1nku, mo\u017en\u00e1 i n\u011bkolika, proto\u017ee je toho spousta, co bychom cht\u011bli \u0159\u00edct: od synchronizace sch\u00e9mat datab\u00e1z\u00ed po spr\u00e1vu p\u0159\u00edstup\u016f a anonymizaci osobn\u00edch \u00fadaj\u016f. Tak\u00e9 je t\u0159eba uv\u00e9st, \u017ee toto \u0159e\u0161en\u00ed nenahrazuje skute\u010dn\u00e9 analytick\u00e9 datab\u00e1ze a \u00falo\u017ei\u0161t\u011b, ale \u0159e\u0161\u00ed pouze konkr\u00e9tn\u00ed \u00fakol.<\/p>\n

Na \u00farovni p\u0159\u00edb\u011bhu to vypad\u00e1 takto:<\/p>\n

\"Opera\u010dn\u00ed
\nExistuje datab\u00e1ze PostgreSQL, kde si u\u017eivatel\u00e9 mohou ukl\u00e1dat sv\u00e1 pracovn\u00ed data, a co je nejd\u016fle\u017eit\u011bj\u0161\u00ed \u2014 k t\u00e9to datab\u00e1zi je prost\u0159ednictv\u00edm FDW p\u0159ipojeno analytick\u00e9 replikace v\u0161ech slu\u017eeb. To umo\u017e\u0148uje napsat dotaz na n\u011bkolik datab\u00e1z\u00ed, bez ohledu na to, zda jde o PostgreSQL, MySQL, MongoDB nebo n\u011bco jin\u00e9ho (soubor, API, pokud n\u00e1hodou neexistuje vhodn\u00fd wrapper, m\u016f\u017eete napsat sv\u016fj). No, vypad\u00e1 to, \u017ee je v\u0161echno super! Tak se rozejdeme?<\/p>\n

Kdyby v\u0161e kon\u010dilo tak rychle a jednodu\u0161e, pravd\u011bpodobn\u011b by tento \u010dl\u00e1nek ani neexistoval.<\/p>\n

Je d\u016fle\u017eit\u00e9 jasn\u011b ch\u00e1pat, jak Postgres zpracov\u00e1v\u00e1 dotazy na vzd\u00e1len\u00e9 servery. To se m\u016f\u017ee zd\u00e1t logick\u00e9, p\u0159esto si toho lid\u00e9 \u010dasto nev\u0161imnou: Postgres d\u011bl\u00ed dotaz na \u010d\u00e1sti, kter\u00e9 jsou vykon\u00e1v\u00e1ny na vzd\u00e1len\u00fdch serverech nez\u00e1visle, shroma\u017e\u010fuje tato data a kone\u010dn\u00e9 v\u00fdpo\u010dty prov\u00e1d\u00ed s\u00e1m, tak\u017ee rychlost prov\u00e1d\u011bn\u00ed dotazu bude z\u00e1sadn\u011b z\u00e1viset na tom, jak je naps\u00e1n. Je tak\u00e9 t\u0159eba zd\u016fraznit: kdy\u017e data p\u0159ich\u00e1zej\u00ed z vzd\u00e1len\u00e9ho serveru, ji\u017e nemaj\u00ed indexy, nic, co by pomohlo pl\u00e1nova\u010di, proto mu m\u016f\u017eeme pomoci a poradit pouze my sami. A pr\u00e1v\u011b o tom bych cht\u011bl mluvit podrobn\u011bji.<\/p>\n

Jednoduch\u00fd dotaz a pl\u00e1n s n\u00edm<\/h1>\n

\nAbychom uk\u00e1zali, jak PostgreSQL prov\u00e1d\u00ed dotaz na tabulku o 6 milionech \u0159\u00e1dk\u016f na vzd\u00e1len\u00e9m serveru<\/a>, pod\u00edvejme se na jednoduch\u00fd pl\u00e1n.<\/p>\n

vysv\u011btlit analyzovat podrobn\u011b  \nSELECT count(1)\nFROM fdw_schema.table;\n\nAgregace  (n\u00e1klady=418383.23..418383.24 \u0159\u00e1dk\u016f=1 \u0161\u00ed\u0159ka=8) (skute\u010dn\u00fd \u010das=3857.198..3857.198 \u0159\u00e1dk\u016f=1 smy\u010dek=1)\n  V\u00fdstup: count(1)\n  ->  Vzd\u00e1len\u00e9 skenov\u00e1n\u00ed na fdw_schema."table"  (n\u00e1klady=100.00..402376.14 \u0159\u00e1dk\u016f=6402838 \u0161\u00ed\u0159ka=0) (skute\u010dn\u00fd \u010das=4.874..3256.511 \u0159\u00e1dk\u016f=6406868 smy\u010dek=1)\n        V\u00fdstup: "table".id, "table".is_active, "table".meta, "table".created_dt\n        Vzd\u00e1len\u00e9 SQL: SELECT NULL FROM fdw_schema.table\nPl\u00e1novac\u00ed \u010das: 0.986 ms\n\u010cas proveden\u00ed: 3857.436 ms<\/code><\/pre>\n
\nPou\u017eit\u00ed p\u0159\u00edkazu VERBOSE umo\u017e\u0148uje vid\u011bt dotaz, kter\u00fd bude odesl\u00e1n na vzd\u00e1len\u00fd server, a v\u00fdsledky, kter\u00e9 obdr\u017e\u00edme pro dal\u0161\u00ed zpracov\u00e1n\u00ed (\u0159\u00e1dek RemoteSQL).<\/p>\n
Poj\u010fme o krok d\u00e1l a p\u0159id\u00e1me do na\u0161eho dotazu n\u011bkolik filtr\u016f: jeden podle boolean<\/b> pole, jeden podle v\u00fdskytu timestamp<\/b> v intervalu a jeden podle jsonb<\/b>.<\/p>\n
vysv\u011btlit analyzovat podrobn\u011b\nSELECT count(1)\nFROM fdw_schema.table \nWHERE is_active is True\nAND created_dt BETWEEN CURRENT_DATE - INTERVAL '7 month' \nAND CURRENT_DATE - INTERVAL '6 month'\nAND meta->>'source' = 'test';\n\nAgregace  (n\u00e1klady=577487.69..577487.70 \u0159\u00e1dk\u016f=1 \u0161\u00ed\u0159ka=8) (skute\u010dn\u00fd \u010das=27473.818..25473.819 \u0159\u00e1dk\u016f=1 smy\u010dek=1)\n  V\u00fdstup: count(1)\n  ->  Vzd\u00e1len\u00e9 skenov\u00e1n\u00ed na fdw_schema."table"  (n\u00e1klady=100.00..577469.21 \u0159\u00e1dk\u016f=7390 \u0161\u00ed\u0159ka=0) (skute\u010dn\u00fd \u010das=31.369..25372.466 \u0159\u00e1dk\u016f=1360025 smy\u010dek=1)\n        V\u00fdstup: "table".id, "table".is_active, "table".meta, "table".created_dt\n        Filtr: (("table".is_active IS TRUE) AND (("table".meta ->> 'source'::text) = 'test'::text) AND ("table".created_dt >= (('now'::cstring)::date - '7 mons'::interval)) AND ("table".created_dt <= ((('now'::cstring)::date)::timestamp with time zone - '6 mons'::interval)))\n        Odstran\u011bn\u00e9 \u0159\u00e1dky filtrem: 5046843\n        Vzd\u00e1len\u00e9 SQL: SELECT created_dt, is_active, meta FROM fdw_schema.table\nPl\u00e1novac\u00ed \u010das: 0.665 ms\n\u010cas proveden\u00ed: 27474.118 ms<\/code><\/pre>\n
\nPr\u00e1v\u011b zde se skr\u00fdv\u00e1 moment, na kter\u00fd je t\u0159eba db\u00e1t p\u0159i psan\u00ed dotaz\u016f. Filtry nebyly p\u0159ed\u00e1ny na vzd\u00e1len\u00fd server, co\u017e znamen\u00e1, \u017ee PostgreSQL vytahuje v\u0161ech 6 milion\u016f \u0159\u00e1dk\u016f, aby je pak lok\u00e1ln\u011b filtroval (\u0159\u00e1dek Filter) a provedl agregaci. Kl\u00ed\u010dem k \u00fasp\u011bchu je napsat dotaz tak, aby filtry byly p\u0159ed\u00e1ny na vzd\u00e1len\u00fd stroj a my dostali a agregovali pouze pot\u0159ebn\u00e9 \u0159\u00e1dky. <\/p>\n
To je n\u011bjak\u00fd nesmysl.<\/h1>\n
\nS boolean poli je v\u0161e jednoduch\u00e9. V p\u016fvodn\u00edm dotazu vznikl probl\u00e9m kv\u016fli oper\u00e1toru is<\/b>. Pokud jej nahrad\u00edme =<\/b>, dostaneme n\u00e1sleduj\u00edc\u00ed v\u00fdsledek:<\/p>\n
vysv\u011btlit analyzovat podrobn\u011b\nSELECT count(1)\nFROM fdw_schema.table\nWHERE is_active = True\nAND created_dt BETWEEN CURRENT_DATE - INTERVAL '7 month' \nAND CURRENT_DATE - INTERVAL '6 month'\nAND meta->>'source' = 'test';\n\nAgregace  (n\u00e1klady=508010.14..508010.15 \u0159\u00e1dk\u016f=1 \u0161\u00ed\u0159ka=8) (skute\u010dn\u00fd \u010das=19064.314..19064.314 \u0159\u00e1dk\u016f=1 smy\u010dek=1)\n  V\u00fdstup: count(1)\n  ->  Vzd\u00e1len\u00e9 skenov\u00e1n\u00ed na fdw_schema."table"  (n\u00e1klady=100.00..507988.44 \u0159\u00e1dk\u016f=8679 \u0161\u00ed\u0159ka=0) (skute\u010dn\u00fd \u010das=33.035..18951.278 \u0159\u00e1dk\u016f=1360025 smy\u010dek=1)\n        V\u00fdstup: "table".id, "table".is_active, "table".meta, "table".created_dt\n        Filtr: ((("table".meta ->> 'source'::text) = 'test'::text) AND ("table".created_dt >= (('now'::cstring)::date - '7 mons'::interval)) AND ("table".created_dt <= ((('now'::cstring)::date)::timestamp with time zone - '6 mons'::interval)))\n        Odstran\u011bn\u00e9 \u0159\u00e1dky filtrem: 3567989\n        Vzd\u00e1len\u00e9 SQL: SELECT created_dt, meta FROM fdw_schema.table WHERE (is_active)\nPl\u00e1novac\u00ed \u010das: 0.834 ms\n\u010cas proveden\u00ed: 19064.534 ms<\/code><\/pre>\n
\nJak vid\u00edte, filtr byl odesl\u00e1n na vzd\u00e1len\u00fd server a \u010das prov\u00e1d\u011bn\u00ed se sn\u00ed\u017eil z 27 na 19 sekund. <\/p>\n
Je t\u0159eba poznamenat, \u017ee oper\u00e1tor is<\/b> se li\u0161\u00ed od oper\u00e1toru =<\/b> t\u00edm, \u017ee um\u00ed pracovat s hodnotou Null. To znamen\u00e1, \u017ee is not True<\/b> ve filtru ponech\u00e1 hodnoty False a Null, zat\u00edmco != True<\/b> ponech\u00e1 pouze hodnoty False. Proto p\u0159i nahrazov\u00e1n\u00ed oper\u00e1toru is not<\/b> je t\u0159eba ve filtr p\u0159idat dv\u011b podm\u00ednky s oper\u00e1torem OR, nap\u0159\u00edklad, WHERE (col != True) OR (col is null)<\/b>.<\/p>\n
S booleanem jsme se vyrovnali, pokra\u010dujeme d\u00e1l. Zat\u00edm vr\u00e1t\u00edme filtr pro logickou hodnotu do p\u016fvodn\u00edho stavu, abychom nez\u00e1visle prozkoumali efekt ostatn\u00edch zm\u011bn.<\/p>\n
timestamptz? hz<\/h1>\n
\n\u010casto mus\u00edme experimentovat s t\u00edm, jak spr\u00e1vn\u011b napsat dotaz, kter\u00fd zahrnuje vzd\u00e1len\u00e9 servery, a pot\u00e9 hledat vysv\u011btlen\u00ed, pro\u010d to tak bylo. Velmi m\u00e1lo informac\u00ed k tomu lze nal\u00e9zt na internetu. V na\u0161ich experimentech jsme zjistili, \u017ee filtrov\u00e1n\u00ed podle pevn\u00e9ho data prob\u00edh\u00e1 hladce na vzd\u00e1len\u00e9m serveru, ale kdy\u017e se pokus\u00edme nastavit datum dynamicky, nap\u0159\u00edklad pomoc\u00ed now() nebo CURRENT_DATE, to neprob\u011bhne. V na\u0161em p\u0159\u00edkladu jsme p\u0159idali takov\u00fd filtr, aby sloupec created_at obsahoval data p\u0159esn\u011b za 1 m\u011bs\u00edc zp\u011bt (BETWEEN CURRENT_DATE \u2014 INTERVAL '7 month' AND CURRENT_DATE \u2014 INTERVAL '6 month'). Co jsme tedy v tomto p\u0159\u00edpad\u011b ud\u011blali?<\/p>\n
vysv\u011btli analyzovat podrobn\u011b\nSELECT count(1)\nFROM fdw_schema.table \nWHERE is_active is True\nAND created_dt >= (SELECT CURRENT_DATE::timestamptz - INTERVAL '7 m\u011bs\u00edc\u016f') \nAND created_dt >'source' = 'test';\n\nAgregace  (n\u00e1klady=306875.17..306875.18 \u0159\u00e1dk\u016f=1 \u0161\u00ed\u0159ka=8) (skute\u010dn\u00fd \u010das=4789.114..4789.115 \u0159\u00e1dk\u016f=1 smy\u010dek=1)\n  V\u00fdstup: count(1)\n  InitPlan 1 (vrac\u00ed $0)\n    ->  V\u00fdsledek  (n\u00e1klady=0.00..0.02 \u0159\u00e1dk\u016f=1 \u0161\u00ed\u0159ka=8) (skute\u010dn\u00fd \u010das=0.007..0.008 \u0159\u00e1dk\u016f=1 smy\u010dek=1)\n          V\u00fdstup: ((('nyn\u00ed'::cstring)::datum)::\u010dasov\u00e9 raz\u00edtko s \u010dasovou z\u00f3nou - '7 m\u011bs\u00edc\u016f'::interval)\n  InitPlan 2 (vrac\u00ed $1)\n    ->  V\u00fdsledek  (n\u00e1klady=0.00..0.02 \u0159\u00e1dk\u016f=1 \u0161\u00ed\u0159ka=8) (skute\u010dn\u00fd \u010das=0.002..0.002 \u0159\u00e1dk\u016f=1 smy\u010dek=1)\n          V\u00fdstup: ((('nyn\u00ed'::cstring)::datum)::\u010dasov\u00e9 raz\u00edtko s \u010dasovou z\u00f3nou - '6 m\u011bs\u00edc\u016f'::interval)\n  ->  Vzd\u00e1len\u00e9 skenov\u00e1n\u00ed na fdw_schema.\"table\"  (n\u00e1klady=100.02..306874.86 \u0159\u00e1dk\u016f=105 \u0161\u00ed\u0159ka=0) (skute\u010dn\u00fd \u010das=23.475..4681.419 \u0159\u00e1dk\u016f=1360025 smy\u010dek=1)\n        V\u00fdstup: \"table\".id, \"table\".is_active, \"table\".meta, \"table\".created_dt\n        Filtr: ((\"table\".is_active IS TRUE) AND ((\"table\".meta ->> 'source'::text) = 'test'::text))\n        \u0158\u00e1dky odebran\u00e9 filtrem: 76934\n        Vzd\u00e1len\u00e9 SQL: SELECT is_active, meta FROM fdw_schema.table WHERE ((created_dt >= $1::\u010dasov\u00e9 raz\u00edtko s \u010dasovou z\u00f3nou)) AND ((created_dt < $2::\u010dasov\u00e9 raz\u00edtko s \u010dasovou z\u00f3nou))\n\u010cas pl\u00e1nov\u00e1n\u00ed: 0.703 ms\n\u010cas prov\u00e1d\u011bn\u00ed: 4789.379 ms<\/code><\/pre>\n
\nDali jsme pl\u00e1nova\u010di pokyn, aby p\u0159edem vypo\u010d\u00edtal datum v poddotazu a p\u0159edal ji\u017e p\u0159ipravenou prom\u011bnnou do filtru. Tato rada n\u00e1m p\u0159inesla skv\u011bl\u00fd v\u00fdsledek, dotaz se zrychlil t\u00e9m\u011b\u0159 \u0161estkr\u00e1t!<\/p>\n
Op\u011bt je d\u016fle\u017eit\u00e9 b\u00fdt opatrn\u00fd: datov\u00fd typ v poddotazu mus\u00ed b\u00fdt stejn\u00fd jako u pole, podle kter\u00e9ho filtrujeme, jinak pl\u00e1nova\u010d usoud\u00ed, \u017ee typy jsou r\u016fzn\u00e9 a je pot\u0159eba nejprve vyt\u00e1hnout v\u0161echna data a pot\u00e9 je lokaln\u011b filtrovat.<\/p>\n
Vra\u0165me filtr podle data na v\u00fdchoz\u00ed hodnotu.<\/p>\n
Freddy vs. Jsonb<\/h1>\n
\nV podstat\u011b u\u017e booleovsk\u00e9 sloupce a data dostate\u010dn\u011b zrychlily n\u00e1\u0161 dotaz, nicm\u00e9n\u011b z\u016fst\u00e1val je\u0161t\u011b jeden datov\u00fd typ. Bitva s filtrac\u00ed podle n\u011bj, up\u0159\u00edmn\u011b \u0159e\u010deno, dosud neskon\u010dila, i kdy\u017e i zde dos\u00e1hli \u00fasp\u011bch\u016f. Tak\u017ee, takto se n\u00e1m poda\u0159ilo p\u0159edat filtr podle jsonb<\/b> pole na vzd\u00e1len\u00fd server.<\/p>\n
vysv\u011btli analyzovat podrobn\u011b\nSELECT count(1)\nFROM fdw_schema.table \nWHERE is_active is True\nAND created_dt BETWEEN CURRENT_DATE - INTERVAL '7 m\u011bs\u00edc\u016f' \nAND CURRENT_DATE - INTERVAL '6 m\u011bs\u00edc\u016f'\nAND meta @> '{\"source\":\"test\"}'::jsonb;\n\nAgregace  (n\u00e1klady=245463.60..245463.61 \u0159\u00e1dk\u016f=1 \u0161\u00ed\u0159ka=8) (skute\u010dn\u00fd \u010das=6727.589..6727.590 \u0159\u00e1dk\u016f=1 smy\u010dek=1)\n  V\u00fdstup: count(1)\n  ->  Vzd\u00e1len\u00e9 skenov\u00e1n\u00ed na fdw_schema.\"table\"  (n\u00e1klady=1100.00..245459.90 \u0159\u00e1dk\u016f=1478 \u0161\u00ed\u0159ka=0) (skute\u010dn\u00fd \u010das=16.213..6634.794 \u0159\u00e1dk\u016f=1360025 smy\u010dek=1)\n        V\u00fdstup: \"table\".id, \"table\".is_active, \"table\".meta, \"table\".created_dt\n        Filtr: ((\"table\".is_active IS TRUE) AND ("table".created_dt >= (('nyn\u00ed'::cstring)::datum - '7 m\u011bs\u00edc\u016f'::interval)) AND (\"table\".created_dt  '{\"source\": \"test\"}'::jsonb))\n\u010cas pl\u00e1nov\u00e1n\u00ed: 0.747 ms\n\u010cas prov\u00e1d\u011bn\u00ed: 6727.815 ms<\/code><\/pre>\n
\nM\u00edsto oper\u00e1tor\u016f filtrace je nutn\u00e9 pou\u017e\u00edt oper\u00e1tor p\u0159\u00edtomnosti jednoho jsonb<\/b> v jin\u00e9m. 7 sekund m\u00edsto p\u016fvodn\u00edch 29. Zat\u00edm je to jedin\u00e1 \u00fasp\u011b\u0161n\u00e1 varianta p\u0159enosu filtr\u016f podle jsonb<\/b> na vzd\u00e1len\u00fd server, ale zde je d\u016fle\u017eit\u00e9 vz\u00edt v \u00favahu jedno omezen\u00ed: pou\u017e\u00edv\u00e1me verzi datab\u00e1ze 9.6, nicm\u00e9n\u011b do konce dubna pl\u00e1nujeme dokon\u010dit posledn\u00ed testy a p\u0159ej\u00edt na verzi 12. Jakmile se aktualizujeme, nap\u00ed\u0161eme, jak to ovlivnilo, proto\u017ee zm\u011bn, na kter\u00e9 se hodn\u011b spol\u00e9h\u00e1me, je pom\u011brn\u011b hodn\u011b: json_path, nov\u00e9 chov\u00e1n\u00ed CTE, push down (existuj\u00edc\u00ed od verze 10). Opravdu se t\u011b\u0161\u00edme, a\u017e to vyzkou\u0161\u00edme.<\/p>\n
Dohodnojte to<\/h1>\n
\nZkontrolovali jsme, jak ka\u017ed\u00e1 zm\u011bna ovliv\u0148uje rychlost dotazu jednotliv\u011b. Nyn\u00ed se pod\u00edvejme, co se stane, kdy\u017e budou v\u0161echny t\u0159i filtry naps\u00e1ny spr\u00e1vn\u011b.<\/p>\n
vysv\u011btli analyzovat podrobn\u011b\nSELECT count(1)\nFROM fdw_schema.table \nWHERE is_active = True\nAND created_dt >= (SELECT CURRENT_DATE::timestamptz - INTERVAL '7 month') \nAND created_dt  '{\"source\":\"test\"}'::jsonb;\n\nAgregace  (n\u00e1klady=322041.51..322041.52 \u0159\u00e1dky=1 \u0161\u00ed\u0159ka=8) (skute\u010dn\u00fd \u010das=2278.867..2278.867 \u0159\u00e1dky=1 smy\u010dky=1)\n  V\u00fdstup: count(1)\n  InitPlan 1 (vrac\u00ed $0)\n    ->  V\u00fdsledek  (n\u00e1klady=0.00..0.02 \u0159\u00e1dky=1 \u0161\u00ed\u0159ka=8) (skute\u010dn\u00fd \u010das=0.010..0.010 \u0159\u00e1dky=1 smy\u010dky=1)\n          V\u00fdstup: ((('now'::cstring)::date)::timestamp with time zone - '7 mons'::interval)\n  InitPlan 2 (vrac\u00ed $1)\n    ->  V\u00fdsledek  (n\u00e1klady=0.00..0.02 \u0159\u00e1dky=1 \u0161\u00ed\u0159ka=8) (skute\u010dn\u00fd \u010das=0.003..0.003 \u0159\u00e1dky=1 smy\u010dky=1)\n          V\u00fdstup: ((('now'::cstring)::date)::timestamp with time zone - '6 mons'::interval)\n  ->  Vzd\u00e1len\u00e9 skenov\u00e1n\u00ed na fdw_schema.\"table\"  (n\u00e1klady=100.02..322041.41 \u0159\u00e1dky=25 \u0161\u00ed\u0159ka=0) (skute\u010dn\u00fd \u010das=8.597..2153.809 \u0159\u00e1dky=1360025 smy\u010dky=1)\n        V\u00fdstup: \"table\".id, \"table\".is_active, \"table\".meta, \"table\".created_dt\n        Vzd\u00e1len\u00fd SQL: SELECT NULL FROM fdw_schema.table WHERE (is_active) AND ((created_dt >= $1::timestamp with time zone)) AND ((created_dt  '{\"source\": \"test\"}'::jsonb))\n\u010cas pl\u00e1nov\u00e1n\u00ed: 0.820 ms\n\u010cas prov\u00e1d\u011bn\u00ed: 2279.087 ms<\/code><\/pre>\n
\nAno, dotaz vypad\u00e1 slo\u017eit\u011bji, je to nevyhnuteln\u00e1 da\u0148, ale rychlost proveden\u00ed je 2 sekundy, co\u017e je v\u00edce ne\u017e 10kr\u00e1t rychleji! A to mluv\u00edme o jednoduch\u00e9m dotazu na relativn\u011b mal\u00fd soubor dat. U re\u00e1ln\u00fdch dotaz\u016f jsme z\u00edskali n\u00e1r\u016fst a\u017e n\u011bkolik setkr\u00e1t.<\/p>\n
Shr\u0148me to: pokud pou\u017e\u00edv\u00e1te PostgreSQL s FDW, v\u017edy zkontrolujte, zda jsou v\u0161echny filtry odes\u00edl\u00e1ny na vzd\u00e1len\u00fd server, a budete \u0161\u0165astn\u00ed\u2026 Minim\u00e1ln\u011b dokud se nedostanete k join\u016fm mezi tabulkami z r\u016fzn\u00fdch servery<\/a>. Ale to u\u017e je p\u0159\u00edb\u011bh na dal\u0161\u00ed \u010dl\u00e1nek.<\/p>\n
D\u011bkuji za pozornost! R\u00e1d usly\u0161\u00edm ot\u00e1zky, koment\u00e1\u0159e a tak\u00e9 p\u0159\u00edb\u011bhy o va\u0161ich zku\u0161enostech v koment\u00e1\u0159\u00edch.
\n
Zdroj: habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"
\u041c\u0438\u043a\u0440\u043e\u0441\u0435\u0440\u0432\u0438\u0441\u043d\u0430\u044f \u0430\u0440\u0445\u0438\u0442\u0435\u043a\u0442\u0443\u0440\u0430, \u043a\u0430\u043a \u0438 \u0432\u0441\u0435 \u0432 \u044d\u0442\u043e\u043c \u043c\u0438\u0440\u0435, \u0438\u043c\u0435\u0435\u0442 \u0441\u0432\u043e\u0438 \u043f\u043b\u044e\u0441\u044b \u0438 \u0441\u0432\u043e\u0438 \u043c\u0438\u043d\u0443\u0441\u044b. \u041e\u0434\u043d\u0438 \u043f\u0440\u043e\u0446\u0435\u0441\u0441\u044b \u0441 \u043d\u0435\u0439 \u0441\u0442\u0430\u043d\u043e\u0432\u044f\u0442\u0441\u044f \u043f\u0440\u043e\u0449\u0435, \u0434\u0440\u0443\u0433\u0438\u0435 \u2014 \u0441\u043b\u043e\u0436\u043d\u0435\u0435. \u0418 \u0432 \u0443\u0433\u043e\u0434\u0443 \u0441\u043a\u043e\u0440\u043e\u0441\u0442\u0438 \u0438\u0437\u043c\u0435\u043d\u0435\u043d\u0438\u0439 \u0438 \u043b\u0443\u0447\u0448\u0435\u0439 \u043c\u0430\u0441\u0448\u0442\u0430\u0431\u0438\u0440\u0443\u0435\u043c\u043e\u0441\u0442\u0438 \u043d\u0443\u0436\u043d\u043e \u043f\u0440\u0438\u043d\u043e\u0441\u0438\u0442\u044c \u0441\u0432\u043e\u0438 \u0436\u0435\u0440\u0442\u0432\u044b. \u041e\u0434\u043d\u0430 \u0438\u0437 \u043d\u0438\u0445 \u2014 \u0443\u0441\u043b\u043e\u0436\u043d\u0435\u043d\u0438\u0435 \u0430\u043d\u0430\u043b\u0438\u0442\u0438\u043a\u0438. \u0415\u0441\u043b\u0438 \u0432 \u043c\u043e\u043d\u043e\u043b\u0438\u0442\u0435 \u0432\u0441\u044e \u043e\u043f\u0435\u0440\u0430\u0442\u0438\u0432\u043d\u0443\u044e \u0430\u043d\u0430\u043b\u0438\u0442\u0438\u043a\u0443 \u043c\u043e\u0436\u043d\u043e \u0441\u0432\u0435\u0441\u0442\u0438 \u043a SQL \u0437\u0430\u043f\u0440\u043e\u0441\u0430\u043c \u043a \u0430\u043d\u0430\u043b\u0438\u0442\u0438\u0447\u0435\u0441\u043a\u043e\u0439 \u0440\u0435\u043f\u043b\u0438\u043a\u0435, […]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":79468,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-79467","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\n","aioseo_head_json":{"title":"\ud83e\udd47Opera\u010dn\u00ed anal\u00fdza v mikroslu\u017ebov\u00e9 architektu\u0159e: pomoci a poradit Postgres FDW | ProHoster","description":"Mikroslu\u017ebov\u00e1 architektura, stejn\u011b jako v\u0161echno na tomto sv\u011bt\u011b, m\u00e1 sv\u00e9 v\u00fdhody a nev\u00fdhody. N\u011bkter\u00e9 procesy se s n\u00ed st\u00e1vaj\u00ed jednodu\u0161\u0161\u00edmi, jin\u00e9 \u2013 slo\u017eit\u011bj\u0161\u00ed.","canonical_url":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/operativnaya-analitika-v-mikroservisnoj-arhitekture-p%cc%b6o%cc%b6n%cc%b6ya%cc%b6t%cc%b6%cc%b6-%cc%b6i%cc%b6-%cc%b6p%cc%b6r%cc%b6o%cc%b6s%cc%b6t%cc%b6i%cc%b6t%cc%b6%cc%b6-pomoch-i-podskazat-postgres","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"cs_CZ","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u041e\u043f\u0435\u0440\u0430\u0442\u0438\u0432\u043d\u0430\u044f \u0430\u043d\u0430\u043b\u0438\u0442\u0438\u043a\u0430 \u0432 \u043c\u0438\u043a\u0440\u043e\u0441\u0435\u0440\u0432\u0438\u0441\u043d\u043e\u0439 \u0430\u0440\u0445\u0438\u0442\u0435\u043a\u0442\u0443\u0440\u0435: \u043f\u0336\u043e\u0336\u043d\u0336\u044f\u0336\u0442\u0336\u044c\u0336 \u0336\u0438\u0336 \u0336\u043f\u0336\u0440\u0336\u043e\u0336\u0441\u0336\u0442\u0336\u0438\u0336\u0442\u0336\u044c\u0336 \u043f\u043e\u043c\u043e\u0447\u044c \u0438 \u043f\u043e\u0434\u0441\u043a\u0430\u0437\u0430\u0442\u044c Postgres FDW | ProHoster","og:description":"\u041c\u0438\u043a\u0440\u043e\u0441\u0435\u0440\u0432\u0438\u0441\u043d\u0430\u044f \u0430\u0440\u0445\u0438\u0442\u0435\u043a\u0442\u0443\u0440\u0430, \u043a\u0430\u043a \u0438 \u0432\u0441\u0435 \u0432 \u044d\u0442\u043e\u043c \u043c\u0438\u0440\u0435, \u0438\u043c\u0435\u0435\u0442 \u0441\u0432\u043e\u0438 \u043f\u043b\u044e\u0441\u044b \u0438 \u0441\u0432\u043e\u0438 \u043c\u0438\u043d\u0443\u0441\u044b. \u041e\u0434\u043d\u0438 \u043f\u0440\u043e\u0446\u0435\u0441\u0441\u044b \u0441 \u043d\u0435\u0439 \u0441\u0442\u0430\u043d\u043e\u0432\u044f\u0442\u0441\u044f \u043f\u0440\u043e\u0449\u0435, \u0434\u0440\u0443\u0433\u0438\u0435 \u2014 \u0441\u043b\u043e\u0436\u043d\u0435\u0435.","og:url":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/operativnaya-analitika-v-mikroservisnoj-arhitekture-p%cc%b6o%cc%b6n%cc%b6ya%cc%b6t%cc%b6%cc%b6-%cc%b6i%cc%b6-%cc%b6p%cc%b6r%cc%b6o%cc%b6s%cc%b6t%cc%b6i%cc%b6t%cc%b6%cc%b6-pomoch-i-podskazat-postgres","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-04-27T05:42:23+00:00","article:modified_time":"2020-04-27T05:42:23+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"79467","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 16:37:24","updated":"2026-02-09 21:38:01","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts\/79467","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/comments?post=79467"}],"version-history":[{"count":2,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts\/79467\/revisions"}],"predecessor-version":[{"id":159871,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts\/79467\/revisions\/159871"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/media\/79468"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/media?parent=79467"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/categories?post=79467"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/tags?post=79467"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}