{"id":90103,"date":"2020-07-29T13:42:32","date_gmt":"2020-07-29T11:42:32","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/patroni-failure-stories-or-how-to-crash-your-postgresql-cluster-aleksej-lesovskij"},"modified":"2020-07-29T13:42:32","modified_gmt":"2020-07-29T11:42:32","slug":"patroni-failure-stories-or-how-to-crash-your-postgresql-cluster-aleksej-lesovskij","status":"publish","type":"post","link":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/patroni-failure-stories-or-how-to-crash-your-postgresql-cluster-aleksej-lesovskij","title":{"rendered":"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/6404cba13214f499d1f347cca0aacbe7.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Hlavn\u00edm c\u00edlem Patroni je zajistit vysokou dostupnost pro PostgreSQL. Ale Patroni je jen \u0161ablona, nikoli hotov\u00fd n\u00e1stroj (co\u017e je ostatn\u011b uvedeno v dokumentaci). Na prvn\u00ed pohled, kdy\u017e nastav\u00edte Patroni v testovac\u00edm prost\u0159ed\u00ed, m\u016f\u017eete vid\u011bt, jak\u00fd je to skv\u011bl\u00fd n\u00e1stroj a jak snadno zvl\u00e1d\u00e1 na\u0161e pokusy o rozpad klastru. Nicm\u00e9n\u011b v praxi v produk\u010dn\u00edm prost\u0159ed\u00ed se to v\u017edy ned\u011bje tak kr\u00e1sn\u011b a elegantn\u011b jako v testovac\u00ed laborato\u0159i.<\/p>\n<p><noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/ff445e6d76a2ad46a232b705e104446f.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Trochu o sob\u011b. Za\u010d\u00ednal jsem jako syst\u00e9mov\u00fd administr\u00e1tor. Pracoval jsem ve webov\u00e9m v\u00fdvoji. Od roku 2014 pracuji ve spole\u010dnosti Data Egret. Firma se zab\u00fdv\u00e1 poradenstv\u00edm v oblasti Postgres. A my se specializujeme p\u0159\u00edmo na Postgres a ka\u017ed\u00fd den s n\u00edm pracujeme, tak\u017ee m\u00e1me r\u016fznou odbornost spojenou s jeho provozem. <\/p>\n<p><\/p>\n<p>Na konci roku 2018 jsme za\u010dali pomalu pou\u017e\u00edvat Patroni. Nasb\u00edrali jsme ur\u010dit\u00e9 zku\u0161enosti. N\u011bjak jsme ho diagnostikovali, vyladili, dosp\u011bli jsme k vlastn\u00edm osv\u011bd\u010den\u00fdm postup\u016fm. A v t\u00e9to prezentaci o nich budu vypr\u00e1v\u011bt.<\/p>\n<p><\/p>\n<p>Krom\u011b Postgresu m\u00e1m r\u00e1d Linux. R\u00e1d v n\u011bm kut\u00edm a prozkoum\u00e1v\u00e1m, r\u00e1d sestavuji j\u00e1dra. Zaj\u00edm\u00e1m se o virtualizaci, kontejnery, Docker, Kubernetes. Zauj\u00edm\u00e1 m\u011b to, proto\u017ee se projevuj\u00ed star\u00e9 administr\u00e1torsk\u00e9 zvyky. R\u00e1d se zab\u00fdv\u00e1m monitorov\u00e1n\u00edm. A m\u00e1m r\u00e1d postgresov\u00e9 v\u011bci spojen\u00e9 s administrac\u00ed, jako je replikace, z\u00e1lohov\u00e1n\u00ed. A ve voln\u00e9m \u010dase p\u00ed\u0161u v Go. Nejsem softwarov\u00fd in\u017een\u00fdr, jen si pro sebe p\u00ed\u0161u v Go. A d\u011bl\u00e1 mi to radost. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/02cfd9293af5a8410c91e39afe2c75e6.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<ul>\n<li>Mysl\u00edm, \u017ee mnoz\u00ed z v\u00e1s v\u00ed, \u017ee v Postgresu nen\u00ed vysok\u00e1 dostupnost (HA) out-of-the-box. Abychom dos\u00e1hli HA, mus\u00edme n\u011bco nainstalovat, nastavit, vynalo\u017eit \u00fasil\u00ed a z\u00edskat to. <\/li>\n<li>Existuje n\u011bkolik n\u00e1stroj\u016f a Patroni je jedn\u00edm z nich, kter\u00fd velmi dob\u0159e \u0159e\u0161\u00ed HA. Ale kdy\u017e to v\u0161echno nastav\u00edme v testovac\u00ed laborato\u0159i a spust\u00edme, m\u016f\u017eeme vid\u011bt, \u017ee to funguje, m\u016f\u017eeme reprodukovat n\u011bjak\u00e9 probl\u00e9my a pod\u00edvat se, jak s nimi Patroni zach\u00e1z\u00ed. A uvid\u00edme, \u017ee v\u0161echno skv\u011ble funguje. <\/li>\n<li>Ale v praxi jsme se setkali s r\u016fzn\u00fdmi probl\u00e9my. A o t\u011bchto probl\u00e9mech budu vypr\u00e1v\u011bt.<\/li>\n<li>Pov\u00edm, jak jsme to diagnostikovali, co jsme nastavovali \u2013 pomohlo n\u00e1m to nebo ne. <\/li>\n<\/ul>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/7126b45b40dd08521e80a3c150382bec.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<ul>\n<li>Nebudu vypr\u00e1v\u011bt, jak nainstalovat Patroni, proto\u017ee to lze snadno naj\u00edt na internetu, m\u016f\u017eete se pod\u00edvat na konfigura\u010dn\u00ed soubory, abyste pochopili, jak to v\u0161echno b\u011b\u017e\u00ed a jak se nastavuje. M\u016f\u017eete se sezn\u00e1mit se sch\u00e9maty a architekturami, kdy\u017e najdete informace o tom na internetu. <\/li>\n<li>Nebudu vypr\u00e1v\u011bt o ciz\u00edch zku\u0161enostech. Budu mluvit pouze o probl\u00e9mech, se kter\u00fdmi jsme se setkali my sami. <\/li>\n<li>A nebudu hovo\u0159it o probl\u00e9mech, kter\u00e9 jsou mimo Patroni a PostgreSQL. Nap\u0159\u00edklad o probl\u00e9mech souvisej\u00edc\u00edch s vyva\u017eov\u00e1n\u00edm, kdy\u017e se n\u00e1\u0161 klastr rozpadl, o tom hovo\u0159it nebudu. <\/li>\n<\/ul>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/3f8d179e7c78ce83e8842ef5c72fe30d.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>A mal\u00fd disclaimer p\u0159edt\u00edm, ne\u017e za\u010dneme n\u00e1\u0161 p\u0159\u00edsp\u011bvek. <\/p>\n<p><\/p>\n<p>V\u0161echny tyto probl\u00e9my, se kter\u00fdmi jsme se pot\u00fdkali, jsme za\u017eili b\u011bhem prvn\u00edch 6-7-8 m\u011bs\u00edc\u016f provozu. \u010casem jsme p\u0159i\u0161li na sv\u00e9 intern\u00ed osv\u011bd\u010den\u00e9 postupy. A probl\u00e9my zmizely. Proto byl p\u0159\u00edsp\u011bvek ohl\u00e1\u0161en p\u0159ed asi p\u016fl rokem, kdy\u017e to v\u0161echno bylo \u010derstv\u00e9 v pam\u011bti a j\u00e1 si to dob\u0159e pamatoval. <\/p>\n<p><\/p>\n<p>P\u0159i p\u0159\u00edprav\u011b p\u0159\u00edsp\u011bvku jsem ji\u017e proch\u00e1zel star\u00e9 postmortemy, zkoumal logy. A n\u011bkter\u00e9 detaily se mohly zapomenout nebo \u010d\u00e1st n\u011bkter\u00fdch detail\u016f mohla b\u00fdt b\u011bhem anal\u00fdzy probl\u00e9m\u016f nedostate\u010dn\u011b prozkoum\u00e1na, tak\u017ee se m\u016f\u017ee zd\u00e1t, \u017ee probl\u00e9my nebyly pln\u011b probr\u00e1ny, nebo \u017ee chyb\u00ed n\u011bjak\u00e9 informace. A proto se omlouv\u00e1m za tuto situaci. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/30d414284a41fcbc3bf417c9da28ca52.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Co je Patroni?<\/p>\n<p><\/p>\n<ul>\n<li>Je to \u0161ablona pro vytv\u00e1\u0159en\u00ed HA. Tak je to naps\u00e1no v dokumentaci. A z m\u00e9ho pohledu je to velmi spr\u00e1vn\u00e9 up\u0159esn\u011bn\u00ed. Patroni nen\u00ed st\u0159\u00edbrn\u00e1 kulka, kter\u00e1 vy\u0159e\u0161\u00ed v\u0161echny va\u0161e probl\u00e9my, tzn. je pot\u0159eba vynalo\u017eit \u00fasil\u00ed, aby to za\u010dalo fungovat a p\u0159in\u00e1\u0161elo prosp\u011bch. <\/li>\n<li>Je to agentn\u00ed slu\u017eba, kter\u00e1 je nainstalov\u00e1na na ka\u017ed\u00e9 slu\u017eb\u011b s datab\u00e1z\u00ed a kter\u00e1 je jak\u00fdmsi init syst\u00e9mem pro v\u00e1\u0161 Postgres. Spou\u0161t\u00ed Postgres, zastavuje ho, restartuje, m\u011bn\u00ed konfiguraci a m\u011bn\u00ed topologii va\u0161eho klastru. <\/li>\n<li>Aby bylo mo\u017en\u00e9 uchov\u00e1vat stav klastru, jeho aktu\u00e1ln\u00ed reprezentaci, jak vypad\u00e1, je pot\u0159eba n\u011bjak\u00e9 \u00falo\u017ei\u0161t\u011b. A z tohoto pohledu Patroni zvolil cestu uchov\u00e1v\u00e1n\u00ed stavu v extern\u00edm syst\u00e9mu. Toto je syst\u00e9m distribuovan\u00e9ho \u00falo\u017ei\u0161t\u011b konfigurace. Mohou to b\u00fdt Etcd, Consul, ZooKeeper nebo kubernetesk\u00fd Etcd, tedy n\u011bkter\u00e1 z t\u011bchto variant. <\/li>\n<li>A jednou z vlastnost\u00ed Patroni je to, \u017ee autofailover m\u00e1te z krabice, sta\u010d\u00ed ho nastavit. Pokud to srovn\u00e1me s Repmgr, tam je failover ji\u017e sou\u010d\u00e1st\u00ed balen\u00ed. S Repmgr dost\u00e1v\u00e1me switchover, ale pokud chceme autofailover, mus\u00edme ho nastavit dodate\u010dn\u011b. Patroni ji\u017e autofailover z krabice m\u00e1.<\/li>\n<li>A existuje mnoho dal\u0161\u00edch v\u011bc\u00ed. Nap\u0159\u00edklad spr\u00e1va konfigurac\u00ed, zav\u00e1d\u011bn\u00ed nov\u00fdch replik, z\u00e1lohov\u00e1n\u00ed atd. Ale to je mimo t\u00e9ma prezentace, o tom mluvit nebudu. <\/li>\n<\/ul>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/fed0f1ce931df99ca76ae1116f8098cc.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>A mal\u00fd z\u00e1v\u011br \u2013 hlavn\u00ed \u00fakol Patroni je dob\u0159e a spolehliv\u011b prov\u00e1d\u011bt autofailover, aby byl n\u00e1\u0161 cluster funk\u010dn\u00ed a aplikace nezaznamenala zm\u011bny v topologii clusteru. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/fac52620957efa47ced330e7cc8084e0.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Ale kdy\u017e za\u010dneme pou\u017e\u00edvat Patroni, n\u00e1\u0161 syst\u00e9m se st\u00e1v\u00e1 trochu slo\u017eit\u011bj\u0161\u00edm. Pokud jsme m\u011bli d\u0159\u00edve Postgres, p\u0159i pou\u017eit\u00ed Patroni dost\u00e1v\u00e1me samotn\u00fd Patroni, dost\u00e1v\u00e1me DCS, kde je ulo\u017een stav. A to v\u0161e mus\u00ed n\u011bjak fungovat. Tak\u017ee, co se m\u016f\u017ee pokazit?<\/p>\n<p><\/p>\n<p>M\u016f\u017ee se pokazit:<\/p>\n<p><\/p>\n<ul>\n<li>M\u016f\u017ee se pokazit Postgres. M\u016f\u017ee to b\u00fdt master nebo replikace, n\u011bco z toho m\u016f\u017ee selhat. <\/li>\n<li>M\u016f\u017ee se pokazit samotn\u00fd Patroni. <\/li>\n<li>M\u016f\u017ee se pokazit DCS, kde je ulo\u017een stav.<\/li>\n<li>A m\u016f\u017ee se pokazit s\u00ed\u0165. <\/li>\n<\/ul>\n<p><\/p>\n<p>Tyto body budu zkoumat v prezentaci. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/1aeaf46d37bf1935b514b0581e0dbd0b.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Budeme zkoumat p\u0159\u00edpady podle jejich slo\u017eitosti, ne z pohledu toho, \u017ee p\u0159\u00edpad zahrnuje mnoho komponent. Ale z pohledu subjektivn\u00edch pocit\u016f, \u017ee tento p\u0159\u00edpad byl pro m\u011b slo\u017eit\u00fd, t\u011b\u017eko se rozeb\u00edral... a naopak, n\u011bjak\u00fd p\u0159\u00edpad byl snadn\u00fd a bylo ho snadn\u00e9 rozeb\u00edrat. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/65b0c98bf2284610e955ced5eb998f39.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>A prvn\u00ed p\u0159\u00edpad je nejjednodu\u0161\u0161\u00ed. Je to p\u0159\u00edpad, kdy jsme vzali cluster datab\u00e1z\u00ed a na tomto stejn\u00e9m clusteru jsme nasadili na\u0161e \u00falo\u017ei\u0161t\u011b DCS. To je nejb\u011b\u017en\u011bj\u0161\u00ed chyba. To je chyba architektur\u00e1ln\u00edho n\u00e1vrhu, tj. spojen\u00ed r\u016fzn\u00fdch komponent na jednom m\u00edst\u011b. <\/p>\n<p><\/p>\n<p>Tak\u017ee do\u0161lo k failoveru, poj\u010fme se pod\u00edvat na to, co se stalo. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/244b69d623fe2205c2d9ed2aa4620067.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>A zde n\u00e1s zaj\u00edm\u00e1, kdy do\u0161lo k failoveru. Tj. zaj\u00edm\u00e1 n\u00e1s tento okam\u017eik, kdy do\u0161lo ke zm\u011bn\u011b stavu clusteru. <\/p>\n<p><\/p>\n<p>Ale failover nen\u00ed v\u017edy okam\u017eit\u00fd, tj. nezabere n\u011bjakou jednotku \u010dasu, m\u016f\u017ee se prot\u00e1hnout. M\u016f\u017ee trvat del\u0161\u00ed dobu.<\/p>\n<p><\/p>\n<p>Proto m\u00e1 \u010das za\u010d\u00e1tku a \u010das ukon\u010den\u00ed, tedy se jedn\u00e1 o trvalou ud\u00e1lost. V\u0161echny ud\u00e1losti d\u011bl\u00edme na t\u0159i intervaly: m\u00e1me \u010das p\u0159ed failoverem, b\u011bhem failoveru a po failoveru. Tedy zva\u017eujeme v\u0161echny ud\u00e1losti na t\u00e9to \u010dasov\u00e9 ose. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/a0ed61afe55d1a157147e0d80b17e5d1.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>A prvn\u00edm krokem, kdy\u017e do\u0161lo k failoveru, je hled\u00e1n\u00ed p\u0159\u00ed\u010diny, co se stalo, co vedlo k failoveru. <\/p>\n<p><\/p>\n<p>Pokud se pod\u00edv\u00e1me na logy, budou to klasick\u00e9 logy Patroni. V nich n\u00e1m sd\u011bluje, \u017ee server se stal mistrem a role mistra p\u0159e\u0161la na tento uzel. Zde je to zv\u00fdrazn\u011bno. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/9998c69f7d0d4358b277f4e0e7b4350f.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>D\u00e1le mus\u00edme pochopit, pro\u010d do\u0161lo k failoveru, tedy jak\u00e9 ud\u00e1losti vedly k tomu, \u017ee se role mistra p\u0159est\u011bhovala z jednoho uzlu na druh\u00fd. V tomto p\u0159\u00edpad\u011b je to jasn\u00e9. M\u00e1me chybu p\u0159i interakci se syst\u00e9mem \u00falo\u017ei\u0161t\u011b. Mistr si uv\u011bdomil, \u017ee nem\u016f\u017ee pracovat s DCS, tedy do\u0161lo k n\u011bjak\u00e9mu probl\u00e9mu s interakc\u00ed. A \u0159\u00edk\u00e1, \u017ee u\u017e nem\u016f\u017ee b\u00fdt mistrem a vzd\u00e1v\u00e1 se sv\u00fdch pravomoc\u00ed. Tato \u0159\u00e1dka \u201edemoted self\u201c p\u0159esn\u011b o tom mluv\u00ed. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/4a8eca27689546b4b0fc3c666a3c37c4.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Pokud se pod\u00edv\u00e1me na ud\u00e1losti, kter\u00e9 p\u0159edch\u00e1zely failoveru, m\u016f\u017eeme tam vid\u011bt pr\u00e1v\u011b ty p\u0159\u00ed\u010diny, kter\u00e9 zp\u016fsobily probl\u00e9m pro pokra\u010dov\u00e1n\u00ed pr\u00e1ce mistra. <\/p>\n<p><\/p>\n<p>Pokud se pod\u00edv\u00e1me na logy Patroni, uvid\u00edme, \u017ee m\u00e1me spoustu r\u016fzn\u00fdch chyb, \u010dasov\u00fdch limit\u016f, tedy agent Patroni nem\u016f\u017ee pracovat s DCS. V tomto p\u0159\u00edpad\u011b je to Consul agent, se kter\u00fdm se komunikuje p\u0159es port 8500. <\/p>\n<p><\/p>\n<p>A probl\u00e9m zde spo\u010d\u00edv\u00e1 v tom, \u017ee Patroni a datab\u00e1ze b\u011b\u017e\u00ed na jednom hostiteli. A na tomto uzlu byly spu\u0161t\u011bny tak\u00e9 servery Consul. Zat\u00ed\u017een\u00edm serveru jsme vytvo\u0159ili probl\u00e9my i pro <a class=\"wpil_keyword_link\" href=\"https:\/\/prohoster.info\/cs\/server\/\"   title=\"servery\" data-wpil-keyword-link=\"linked\"  data-wpil-monitor-id=\"1515\">servery<\/a> Consul. Nedok\u00e1zali norm\u00e1ln\u011b komunikovat. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/0d8dba6641809560e6b32825d481e30b.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Po n\u011bjak\u00e9 dob\u011b, kdy\u017e z\u00e1t\u011b\u017e opadla, mohl n\u00e1\u0161 Patroni op\u011bt komunikovat s agenty. Norm\u00e1ln\u00ed provoz byl obnoven. A ten sam\u00fd server Pgdb-2 se znovu stal mistrem. Tedy do\u0161lo k mal\u00e9 zm\u011bn\u011b, kv\u016fli kter\u00e9 uzel vzdal sv\u00e9 pravomoci mistra, a pot\u00e9 je op\u011bt p\u0159ijal, tedy v\u0161e se vr\u00e1tilo do p\u016fvodn\u00edho stavu. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/131b93e0bd83099e1b99b53742419e13.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>A to lze pova\u017eovat za fale\u0161nou aktivaci, nebo to m\u016f\u017eeme vn\u00edmat tak, \u017ee Patroni ud\u011blal v\u0161e spr\u00e1vn\u011b. Tedy pochopil, \u017ee nem\u016f\u017ee udr\u017eovat stav clustera a vzdal se sv\u00fdch pravomoc\u00ed.<\/p>\n<p><\/p>\n<p>A zde vznik\u00e1 probl\u00e9m, proto\u017ee servery Consul jsou na stejn\u00e9m hardware jako datab\u00e1ze. Ka\u017ed\u00e1 z\u00e1t\u011b\u017e, a\u0165 u\u017e z\u00e1t\u011b\u017e na disky nebo procesory, tak\u00e9 ovliv\u0148uje interakci s klastrem Consul.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/142c7cf839c8da078451f7ba9d5499e5.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>A rozhodli jsme se, \u017ee to nem\u016f\u017ee \u017e\u00edt spolu, vy\u010dlenili jsme samostatn\u00fd cluster pro Consul. A Patroni u\u017e pracoval s odd\u011blen\u00fdm Consulem, tj. byl zvl\u00e1\u0161\u0165 cluster Postgres, zvl\u00e1\u0161\u0165 cluster Consul. Toto je z\u00e1kladn\u00ed pokyn, jak je t\u0159eba v\u0161echny tyto v\u011bci rozd\u011blit a udr\u017eovat, aby ne\u017eily spolu. <\/p>\n<p><\/p>\n<p>Jako alternativu m\u016f\u017eeme upravit parametry ttl, loop_wait, retry_timeout, tedy se pokusit zv\u00fd\u0161en\u00edm t\u011bchto parametr\u016f p\u0159e\u010dkat kr\u00e1tkodob\u00e9 \u0161pi\u010dky z\u00e1t\u011b\u017ee. Ale to nen\u00ed \u00fapln\u011b vhodn\u00e1 mo\u017enost, proto\u017ee z\u00e1t\u011b\u017e m\u016f\u017ee b\u00fdt dlouhodob\u00e1. Mohli bychom se prost\u011b dostat za limity t\u011bchto parametr\u016f, co\u017e nemus\u00ed nutn\u011b pomoci. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/c51f35dcd88b1c792acec231c0c6c66c.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Prvn\u00ed probl\u00e9m, jak jste pochopili, je jednoduch\u00fd. Vzali jsme DCS a um\u00edstili ho spolu s datab\u00e1z\u00ed, a t\u00edm jsme vytvo\u0159ili probl\u00e9m. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/cc2a98d9790441577fb4080406cbc53f.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Druh\u00fd probl\u00e9m je podobn\u00fd prvn\u00edmu. Je podobn\u00fd t\u00edm, \u017ee m\u00e1me znovu probl\u00e9my s interakc\u00ed se syst\u00e9mem DCS.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/e5c01b105adca48cc620085db8dd2d2b.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Pokud se pod\u00edv\u00e1me na logy, uvid\u00edme, \u017ee znovu doch\u00e1z\u00ed k chyb\u011b komunikace. Patroni \u0159\u00edk\u00e1, \u017ee nem\u016f\u017ee komunikovat s DCS, proto aktu\u00e1ln\u00ed master p\u0159ech\u00e1z\u00ed do re\u017eimu repliky.<\/p>\n<p><\/p>\n<p>Star\u00fd master se st\u00e1v\u00e1 replikou, Patroni zde funguje, jak m\u00e1. Spou\u0161t\u00ed pg_rewind, aby se vr\u00e1til k transak\u010dn\u00edmu logu a pak se p\u0159ipojil k nov\u00e9mu masterovi a vyrovnal se s n\u00edm. Patroni zde funguje, jak by m\u011bl. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/29255f4790abb15b64dbf9ee0b41c7fc.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Mus\u00edme naj\u00edt m\u00edsto, kter\u00e9 p\u0159edch\u00e1zelo chyb\u011b souboru, tedy chyby, kter\u00e9 zp\u016fsobily vznik t\u00e9to situace. Z tohoto pohledu se s logy Patroni pracuje docela snadno. Pravideln\u011b zapisuje stejn\u00e9 zpr\u00e1vy. A pokud rychle proj\u00ed\u017ed\u00edme logy, uvid\u00edme, \u017ee se zpr\u00e1vy zm\u011bnily, co\u017e nazna\u010duje, \u017ee n\u011bjak\u00e9 probl\u00e9my za\u010daly. Rychle se vr\u00e1t\u00edme na toto m\u00edsto a pod\u00edv\u00e1me se, co se d\u011bje. <\/p>\n<p><\/p>\n<p>V norm\u00e1ln\u00ed situaci logy vypadaj\u00ed p\u0159ibli\u017en\u011b takto. Kontroluje se vlastn\u00edk z\u00e1mku. A pokud se vlastn\u00edk nap\u0159\u00edklad zm\u011bnil, mohou nastat n\u011bjak\u00e9 ud\u00e1losti, na kter\u00e9 by m\u011bl Patroni reagovat. Ale v tomto p\u0159\u00edpad\u011b m\u00e1me v\u0161e v po\u0159\u00e1dku. Hled\u00e1me m\u00edsto, kdy za\u010daly chyby. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/f48ebd22a405f5b2900621451c68f543.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>A kdy\u017e se posuneme na m\u00edsto, kde za\u010daly chyby, vid\u00edme, \u017ee do\u0161lo k automatick\u00e9mu selh\u00e1n\u00ed. A jeliko\u017e byly na\u0161e chyby spojeny se spoluprac\u00ed s DCS a v na\u0161em p\u0159\u00edpad\u011b jsme pou\u017e\u00edvali Consul, pod\u00edv\u00e1me se tak\u00e9 do log\u016f Consulu, co se tam d\u011blo. <\/p>\n<p><\/p>\n<p>P\u0159ibli\u017en\u011b srovn\u00e1n\u00edm \u010dasu selh\u00e1n\u00ed a \u010dasu v logech Consulu vid\u00edme, \u017ee na\u0161i soused\u00e9 z Consul clusteru za\u010dali pochybovat o existenci ostatn\u00edch \u00fa\u010dastn\u00edk\u016f Consul clusteru.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/63b02ac5b4f5a34a2822f1eee2b1f2f8.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>A pokud se pod\u00edv\u00e1me na logy dal\u0161\u00edch agent\u016f Consulu, pak tam tak\u00e9 vid\u00edme, \u017ee doch\u00e1z\u00ed k n\u011bjak\u00e9mu s\u00ed\u0165ov\u00e9mu kolapsu. A v\u0161echny \u00fa\u010dastn\u00edky Consul clusteru maj\u00ed pochybnosti o existenci jeden druh\u00e9ho. A to bylo podn\u011btem k selh\u00e1n\u00ed. <\/p>\n<p><\/p>\n<p>Pokud se pod\u00edv\u00e1me, co se d\u011blo p\u0159ed t\u011bmito chybami, m\u016f\u017eeme vid\u011bt, \u017ee se objevuj\u00ed r\u016fzn\u00e9 chyby, nap\u0159\u00edklad deadline, RPC selhal, tj. z\u0159ejm\u011b n\u011bjak\u00fd probl\u00e9m p\u0159i vz\u00e1jemn\u00e9m p\u016fsoben\u00ed \u00fa\u010dastn\u00edk\u016f Consul clusteru. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/b15c912f2afb49d9d85e7bb9361ce449.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Nejjednodu\u0161\u0161\u00ed odpov\u011b\u010f je opravit s\u00ed\u0165. Ale pro m\u011b, kdy\u017e stoj\u00edm na tribun\u011b, je snadn\u00e9 to \u0159\u00edct. Ale okolnosti jsou takov\u00e9, \u017ee ne v\u017edy si klient m\u016f\u017ee dovolit opravit s\u00ed\u0165. M\u016f\u017ee \u017e\u00edt v datov\u00e9m centru a nemus\u00ed m\u00edt mo\u017enost opravit s\u00ed\u0165, ovlivnit hardware. A proto jsou pot\u0159eba n\u011bjak\u00e9 jin\u00e9 mo\u017enosti. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/1dbdb66233acc3bc321a30a1c5fe14f9.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Mo\u017enosti existuj\u00ed:<\/p>\n<p><\/p>\n<ul>\n<li>Nejjednodu\u0161\u0161\u00ed mo\u017enost, kter\u00e1 je napsan\u00e1, mysl\u00edm, \u017ee i v dokumentaci, je vypnout kontroln\u00ed mechanismy Consulu, tj. jednodu\u0161e p\u0159edat pr\u00e1zdn\u00e9 pole. \u0158\u00edk\u00e1me agentu Consulu, aby nepou\u017e\u00edval \u017e\u00e1dn\u00e9 kontroly. D\u00edky t\u011bmto kontrol\u00e1m m\u016f\u017eeme ignorovat tyto s\u00ed\u0165ov\u00e9 bou\u0159e a nevyvol\u00e1vat selh\u00e1n\u00ed. <\/li>\n<li>Dal\u0161\u00ed mo\u017enost\u00ed je zkontrolovat raft_multiplier. To je parametr samotn\u00e9ho serveru Consul. Ve v\u00fdchoz\u00edm nastaven\u00ed je nastaven na hodnotu 5. Tato hodnota je doporu\u010den\u00e1 podle dokumentace pro staging prost\u0159ed\u00ed. V podstat\u011b to ovliv\u0148uje frekvenci v\u00fdm\u011bny zpr\u00e1v mezi \u00fa\u010dastn\u00edky s\u00edt\u011b Consul. V podstat\u011b tento parametr ovliv\u0148uje rychlost slu\u017eebn\u00ed komunikace mezi \u00fa\u010dastn\u00edky Consul clusteru. A pro produkci se doporu\u010duje sni\u017eovat, aby se uzly \u010dast\u011bji vym\u011b\u0148ovaly zpr\u00e1vy. <\/li>\n<li>Jinou mo\u017enost\u00ed, kterou jsme za\u010dali pou\u017e\u00edvat, je zv\u00fd\u0161en\u00ed priority proces\u016f Consul mezi ostatn\u00edmi procesy pro pl\u00e1nova\u010d opera\u010dn\u00edho syst\u00e9mu. Existuje takov\u00fd parametr \"nice\", kter\u00fd p\u0159esn\u011b ur\u010duje prioritu proces\u016f, je\u017e pl\u00e1nova\u010d OS zohled\u0148uje p\u0159i pl\u00e1nov\u00e1n\u00ed. Sn\u00ed\u017eili jsme hodnotu nice pro agenti Consul, tedy jsme zv\u00fd\u0161ili prioritu, aby opera\u010dn\u00ed syst\u00e9m d\u00e1val proces\u016fm Consul v\u00edce \u010dasu na pr\u00e1ci a na vykon\u00e1v\u00e1n\u00ed sv\u00e9ho k\u00f3du. V na\u0161em p\u0159\u00edpad\u011b to vy\u0159e\u0161ilo n\u00e1\u0161 probl\u00e9m. <\/li>\n<li>Dal\u0161\u00ed mo\u017enost\u00ed je nepou\u017e\u00edvat Consul. M\u00e1m p\u0159\u00edtele, kter\u00fd je velk\u00fdm zast\u00e1ncem Etcd. A pravideln\u011b se s n\u00edm h\u00e1d\u00e1me, co je lep\u0161\u00ed, Etcd nebo Consul. Ale co se t\u00fd\u010de toho, co je lep\u0161\u00ed, obvykle se shodujeme v n\u00e1zoru, \u017ee Consul m\u00e1 agenta, kter\u00fd mus\u00ed b\u00fdt spu\u0161t\u011bn na ka\u017ed\u00e9m uzlu s datab\u00e1z\u00ed. Tedy interakce Patroni s klastrem Consul prob\u00edh\u00e1 p\u0159es tohoto agenta. A tento agent se st\u00e1v\u00e1 \u00fazk\u00fdm m\u00edstem. Pokud se s agentem n\u011bco stane, Patroni u\u017e nem\u016f\u017ee pracovat s klastrem Consul. A to je probl\u00e9m. Co se t\u00fd\u010de Etcd, \u017e\u00e1dn\u00e9ho agenta nen\u00ed. Patroni m\u016f\u017ee p\u0159\u00edmo pracovat se seznamem Etcd server\u016f a komunikovat s nimi. V tomto ohledu, pokud ve va\u0161\u00ed firm\u011b pou\u017e\u00edv\u00e1te Etcd, m\u016f\u017ee b\u00fdt Etcd pravd\u011bpodobn\u011b lep\u0161\u00ed volbou ne\u017e Consul. Ale my jsme u na\u0161ich z\u00e1kazn\u00edk\u016f v\u017edy omezeni t\u00edm, co si vybral a pou\u017e\u00edv\u00e1 klient. A v\u011bt\u0161inou m\u00e1me Consul u v\u0161ech klient\u016f. <\/li>\n<li>A posledn\u00ed bod \u2013 je p\u0159ehodnotit hodnoty parametr\u016f. M\u016f\u017eeme tyto parametry zv\u00fd\u0161it v nad\u011bji, \u017ee na\u0161e kr\u00e1tkodob\u00e9 s\u00ed\u0165ov\u00e9 probl\u00e9my budou kr\u00e1tk\u00e9 a nebudou spadat do intervalu t\u011bchto parametr\u016f. T\u00edmto zp\u016fsobem m\u016f\u017eeme sn\u00ed\u017eit agresivitu Patroni p\u0159i prov\u00e1d\u011bn\u00ed automatick\u00e9ho p\u0159evodu, pokud se vyskytnou n\u011bjak\u00e9 s\u00ed\u0165ov\u00e9 probl\u00e9my.<\/li>\n<\/ul>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/f0043050da20f6368dabf66ec9a4eade.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Mysl\u00edm, \u017ee mnoz\u00ed, kdo pou\u017e\u00edvaj\u00ed Patroni, znaj\u00ed tento p\u0159\u00edkaz. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/9f78f59dab166e47ac78436802156d04.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Tento p\u0159\u00edkaz ukazuje aktu\u00e1ln\u00ed stav klastru. A na prvn\u00ed pohled se m\u016f\u017ee tato situace zd\u00e1t norm\u00e1ln\u00ed. M\u00e1me master, m\u00e1me repliku, \u017e\u00e1dn\u00e1 replikace nen\u00ed zpo\u017ed\u011bn\u00e1. Ale tato situace je norm\u00e1ln\u00ed jen do t\u00e9 doby, dokud nev\u00edme, \u017ee v tomto klastru by m\u011bly b\u00fdt t\u0159i uzly, nikoli dva. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/a39c891f8620ba210b6bce0727e0fa8b.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>V d\u016fsledku toho do\u0161lo k automatick\u00e9mu p\u0159epnut\u00ed souboru. Po tomto p\u0159epnut\u00ed souboru n\u00e1m ode\u0161la replikace. Mus\u00edme zjistit, pro\u010d ode\u0161la, a vr\u00e1tit ji zp\u011bt, obnovit. A op\u011bt se d\u00edv\u00e1me do log\u016f a zkoum\u00e1me, pro\u010d k automatick\u00e9mu p\u0159epnut\u00ed souboru do\u0161lo.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/e6e67d46cb20c8d7e58c7505157f68e1.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>V tomto p\u0159\u00edpad\u011b se druh\u00e1 replikace stala mistrem. V\u0161echno je v po\u0159\u00e1dku. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/b86ccea68b0c6013a23bbd2804e48320.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>A mus\u00edme se pod\u00edvat na replikaci, kter\u00e1 se odpojila a nen\u00ed v clusteru. Otev\u00edr\u00e1me logy Patroni a sledujeme, \u017ee p\u0159i pokusu o p\u0159ipojen\u00ed k clusteru do\u0161lo k probl\u00e9mu ve f\u00e1zi pg_rewind. Abychom se p\u0159ipojili k clusteru, mus\u00edme se vr\u00e1tit v protokolu transakc\u00ed, po\u017e\u00e1dat o pot\u0159ebn\u00fd protokol transakc\u00ed z mistra a podle n\u011bj dohnat mistra. <\/p>\n<p><\/p>\n<p>V tomto p\u0159\u00edpad\u011b nem\u00e1me protokol transakc\u00ed a replikace se nem\u016f\u017ee spustit. D\u016fsledkem toho zastavujeme Postgres s chybou. A proto nen\u00ed v clusteru. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/dadbd44b766674b719d85781ef7f0caa.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Mus\u00edme zjistit, pro\u010d nen\u00ed v clusteru a pro\u010d chyb\u00ed logy. Jdeme na nov\u00e9ho mistra a zkoum\u00e1me, co m\u00e1 v log\u00e1ch. Ukazuje se, \u017ee p\u0159i prov\u00e1d\u011bn\u00ed pg_rewind do\u0161lo k checkpointu. A \u010d\u00e1st star\u00fdch protokol\u016f transakc\u00ed byla jednodu\u0161e p\u0159ejmenov\u00e1na. Kdy\u017e se star\u00fd mistr pokusil p\u0159ipojit k nov\u00e9mu mistrovi a po\u017e\u00e1dat o tyto logy, u\u017e byly p\u0159ejmenov\u00e1ny, jednodu\u0161e neexistovaly.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/f2d5c37324f9436b2bdad1290612d86f.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Porovn\u00e1val jsem \u010dasov\u00e9 zna\u010dky, kdy k t\u011bmto ud\u00e1lostem do\u0161lo. A rozd\u00edl je doslova 150 milisekund, t. j. checkpoint byl dokon\u010den za 369 milisekund, WAL segmenty byly p\u0159ejmenov\u00e1ny. A doslova po 517 milisekund\u00e1ch, po 150 milisekund\u00e1ch, se spustil rewind na star\u00e9 replikaci. T. j. sta\u010dilo n\u00e1m 150 milisekund, aby se replikace nemohla p\u0159ipojit a fungovat. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/99dd80deb8466bfc3aff568456e07102.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Jak\u00e9 jsou mo\u017enosti?<\/p>\n<p><\/p>\n<p>Zpo\u010d\u00e1tku jsme pou\u017e\u00edvali replikace sloty. Zd\u00e1lo se n\u00e1m, \u017ee je to dobr\u00e9. I kdy\u017e jsme v prvn\u00ed f\u00e1zi provozu sloty vypnuli. M\u011bli jsme pocit, \u017ee pokud budou sloty hromadit mnoho WAL segment\u016f, m\u016f\u017eeme po\u0161kodit mistra. Spadne. Chv\u00edli jsme se bez slot\u016f tr\u00e1pili. A pochopili jsme, \u017ee sloty pot\u0159ebujeme, vr\u00e1tili jsme sloty zp\u011bt. <\/p>\n<p><\/p>\n<p>Ale tady je probl\u00e9m, \u017ee kdy\u017e se master p\u0159ejde na replikaci, odstran\u00ed sloty a spolu se sloty odstran\u00ed WAL segmenty. Abychom p\u0159ede\u0161li t\u00e9to situaci, rozhodli jsme se zv\u00fd\u0161it parametr wal_keep_segments. Ten je ve v\u00fdchoz\u00edm stavu nastaven na 8 segment\u016f. My jsme ho zv\u00fd\u0161ili na 1 000 a pod\u00edvali se, kolik m\u00e1me voln\u00e9ho m\u00edsta. A na wal_keep_segments jsme darovali 16 gigabajt\u016f. Tedy p\u0159i p\u0159ep\u00edn\u00e1n\u00ed m\u00e1me v\u017edy na v\u0161ech uzlech rezervu 16 gigabajt\u016f transak\u010dn\u00edch log\u016f.<\/p>\n<p><\/p>\n<p>A nav\u00edc \u2013 to je tak\u00e9 aktu\u00e1ln\u00ed pro dlouhotrvaj\u00edc\u00ed \u00fadr\u017ebov\u00e9 \u00fakoly. \u0158ekn\u011bme, \u017ee pot\u0159ebujeme aktualizovat jednu z replik. A chceme ji vypnout. Mus\u00edme aktualizovat software, mo\u017en\u00e1 opera\u010dn\u00ed syst\u00e9m, n\u011bco dal\u0161\u00edho. A kdy\u017e vypneme replikaci, pro tuto replikaci se tak\u00e9 odstran\u00ed slot. A pokud pou\u017eijeme mal\u00fd wal_keep_segments, p\u0159i dlouh\u00e9 absenci repliky se transak\u010dn\u00ed logy p\u0159ehraj\u00ed. Spust\u00edme replikaci, ta si vy\u017e\u00e1d\u00e1 ty transak\u010dn\u00ed logy, kde skon\u010dila, ale na masteru m\u016f\u017ee \u017e\u00e1dn\u00e9 b\u00fdt. A replikace se tak\u00e9 nebude moct p\u0159ipojit. Proto udr\u017eujeme velkou rezervu log\u016f.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/dad55c828128ce92f649a99cc53ce54b.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/63e70f0b098567f3a6d4fd66a0c293bb.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>M\u00e1me produk\u010dn\u00ed datab\u00e1zi. Tam u\u017e b\u011b\u017e\u00ed projekty. <\/p>\n<p><\/p>\n<p>Do\u0161lo k failoveru. Pod\u00edvali jsme se a v\u0161echno je v po\u0159\u00e1dku, repliky jsou na sv\u00e9m m\u00edst\u011b, \u017e\u00e1dn\u00fd lag replikace. Chyb tak\u00e9 nen\u00ed, v\u0161e je v po\u0159\u00e1dku. <\/p>\n<p><\/p>\n<p>Produktov\u00fd t\u00fdm \u0159\u00edk\u00e1, \u017ee zd\u00e1nliv\u011b by m\u011bly b\u00fdt n\u011bjak\u00e9 \u00fadaje, ale vid\u00edme je v jednom zdroji, zat\u00edmco v datab\u00e1zi je nevid\u00edme. A mus\u00edme pochopit, co se s nimi stalo. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/568ca9aa31680d5e0c9d4ed9693b14e6.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Je jasn\u00e9, \u017ee pg_rewind je smazal. To jsme pochopili hned, ale \u0161li jsme se pod\u00edvat, co se d\u011blo. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/a8fee389201d96e81761cd276c5265c3.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>V log\u00e1ch v\u017edy m\u016f\u017eeme naj\u00edt, kdy k failoveru do\u0161lo, kdo se stal mistrem a m\u016f\u017eeme ur\u010dit, kdo byl star\u00fd master a kdy se cht\u011bl st\u00e1t replikou, tedy pot\u0159ebujeme tyto logy, abychom zjistili objem transak\u010dn\u00edch log\u016f, kter\u00e9 byly ztraceny.<\/p>\n<p><\/p>\n<p>N\u00e1\u0161 star\u00fd master se restartoval. A v automatick\u00e9m spu\u0161t\u011bn\u00ed byl zaps\u00e1n Patroni. Spustil se Patroni. Ten n\u00e1sledn\u011b spustil Postgres. P\u0159esn\u011bji, p\u0159ed spu\u0161t\u011bn\u00edm Postgresu a p\u0159ed t\u00edm, ne\u017e se stal jeho replikou, Patroni spustil proces pg_rewind. T\u00edm p\u00e1dem smazal \u010d\u00e1st transak\u010dn\u00edch \u017eurn\u00e1l\u016f, st\u00e1hl nov\u00e9 a p\u0159ipojil se. Zde Patroni fungoval skv\u011ble, tj. jak m\u011bl. N\u00e1\u0161 cluster se obnovil. M\u011bli jsme 3 nody, po failoveru 3 nody \u2013 v\u0161echno je super. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/b747a24f53e20ed098c2dc0afc79bead.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Ztratili jsme \u010d\u00e1st dat. A pot\u0159ebujeme zjistit, kolik jsme ztratili. Hled\u00e1me pr\u00e1v\u011b ten okam\u017eik, kdy do\u0161lo k rewindu. To m\u016f\u017eeme naj\u00edt podle takov\u00fdchto z\u00e1znam\u016f v logu. Rewind se spustil, n\u011bco ud\u011blal a dokon\u010dil se.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/5b7bacffb52e3ccd529a3a734c441295.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Mus\u00edme naj\u00edt pozici v transak\u010dn\u00edm logu, na kter\u00e9 se zastavil star\u00fd master. V tomto p\u0159\u00edpad\u011b \u2013 je to tato zna\u010dka. A pot\u0159ebujeme druhou zna\u010dku, tedy vzd\u00e1lenost, o kterou se star\u00fd master li\u0161\u00ed od nov\u00e9ho. <\/p>\n<p><\/p>\n<p>Bereme b\u011b\u017en\u00fd pg_wal_lsn_diff a porovn\u00e1v\u00e1me tyto dv\u011b zna\u010dky. A v tomto p\u0159\u00edpad\u011b dostaneme 17 megabajt\u016f. To, zda je to mnoho nebo m\u00e1lo, si ka\u017ed\u00fd rozhoduje s\u00e1m. Proto\u017ee pro n\u011bkoho 17 megabajt\u016f \u2013 to je m\u00e1lo, pro n\u011bkoho je to mnoho a nep\u0159ijateln\u00e9. Tady u\u017e si ka\u017ed\u00fd individu\u00e1ln\u011b ur\u010duje podle pot\u0159eb sv\u00e9ho podnik\u00e1n\u00ed. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/55335fce85961e159459c38300a0ad12.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Ale co jsme zjistili pro sebe? <\/p>\n<p><\/p>\n<p>Za prv\u00e9, mus\u00edme si rozhodnout \u2013 pot\u0159ebujeme v\u017edy automatick\u00e9 spu\u0161t\u011bn\u00ed Patroni po restartu syst\u00e9mu? \u010cast\u011bji se st\u00e1v\u00e1, \u017ee mus\u00edme j\u00edt na star\u00fd master, pod\u00edvat se, jak daleko se vzd\u00e1lil. Mo\u017en\u00e1 prozkoumat segmenty transak\u010dn\u00edho logu, pod\u00edvat se, co tam je. A pochopit \u2013 m\u016f\u017eeme tato data ztratit, nebo pot\u0159ebujeme spustit star\u00fd master v standalone re\u017eimu, abychom tato data z\u00edskali. <\/p>\n<p><\/p>\n<p>A teprve pot\u00e9 bychom m\u011bli p\u0159ijmout rozhodnut\u00ed, \u017ee tato data m\u016f\u017eeme zahodit, nebo je m\u016f\u017eeme obnovit, p\u0159ipojit tento uzel jako repliku do na\u0161eho clusteru.<\/p>\n<p><\/p>\n<p>Krom\u011b toho existuje parametr \u201emaximum_lag_on_failover\u201c. Pokud se nem\u00fdl\u00edm, tento parametr m\u00e1 ve v\u00fdchoz\u00edm nastaven\u00ed hodnotu 1 megabajt. <\/p>\n<p><\/p>\n<p>Jak to funguje? Pokud na\u0161e replika zpo\u017e\u010fuje o 1 megabajt dat kv\u016fli zpo\u017ed\u011bn\u00ed replikace, tato replika se ne\u00fa\u010dastn\u00ed voleb. A pokud dojde k failoveru, Patroni kontroluje, kter\u00e9 repliky se zpo\u017e\u010fuj\u00ed. Pokud se zpo\u017e\u010fuj\u00ed o velk\u00e9 mno\u017estv\u00ed transak\u010dn\u00edch log\u016f, nemohou se st\u00e1t masterem. To je velmi dobr\u00e1 ochrann\u00e1 funkce, kter\u00e1 zabra\u0148uje ztr\u00e1t\u011b velk\u00e9ho mno\u017estv\u00ed dat. <\/p>\n<p><\/p>\n<p>Ale zde je probl\u00e9m v tom, \u017ee zpo\u017ed\u011bn\u00ed replikace v clusteru Patroni a DCS se aktualizuje s ur\u010ditou periodou. Podle m\u011b je to ve v\u00fdchoz\u00edm nastaven\u00ed 30 sekund pro ttl.<\/p>\n<p><\/p>\n<p>M\u016f\u017ee nastat situace, kdy zpo\u017ed\u011bn\u00ed replikace pro repliky v DCS je jedin\u00e9, ale ve skute\u010dnosti m\u016f\u017ee b\u00fdt \u00fapln\u011b jin\u00e9 zpo\u017ed\u011bn\u00ed nebo dokonce nemus\u00ed b\u00fdt \u017e\u00e1dn\u00e9, tj. tento proces neprob\u00edh\u00e1 v re\u00e1ln\u00e9m \u010dase. A nikdy neodr\u00e1\u017e\u00ed skute\u010dn\u00fd obraz. Nen\u00ed vhodn\u00e9 na n\u011bm stav\u011bt slo\u017eitou logiku. <\/p>\n<p><\/p>\n<p>A riziko ztr\u00e1ty dat v\u017edy z\u016fst\u00e1v\u00e1. V nejhor\u0161\u00edm p\u0159\u00edpad\u011b jedna formule, v pr\u016fm\u011brn\u00e9m p\u0159\u00edpad\u011b jin\u00e1 formule. Kdy\u017e pl\u00e1nujeme implementaci Patroni a odhadujeme, kolik dat m\u016f\u017eeme ztratit, mus\u00edme po\u010d\u00edtat s t\u011bmito formulami a p\u0159ibli\u017en\u011b m\u00edt p\u0159edstavu o mno\u017estv\u00ed ztracen\u00fdch dat. <\/p>\n<p><\/p>\n<p>A m\u00e1m dobrou zpr\u00e1vu. Kdy\u017e star\u00fd master p\u0159e\u0161el vp\u0159ed, m\u016f\u017ee tak u\u010dinit d\u00edky n\u011bkter\u00fdm pozadov\u00fdm proces\u016fm. Nap\u0159\u00edklad pokud byl spu\u0161t\u011bn automatick\u00fd vakuum, zapsal data a ulo\u017eil je do transak\u010dn\u00edho logu. Tato data m\u016f\u017eeme bez probl\u00e9m\u016f ignorovat a ztratit. Na tom nen\u00ed nic \u0161patn\u00e9ho. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/a86e8971ab4ef41b89af9cf0bdf519ba.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>A takto vypadaj\u00ed logy, pokud je nastavena maximum_lag_on_failover a do\u0161lo k failoveru, kdy je t\u0159eba vybrat nov\u00e9ho mastera. Replika se sama hodnot\u00ed jako neschopn\u00e1 \u00fa\u010dastnit se voleb. A odm\u00edt\u00e1 \u00fa\u010dast v z\u00e1vodu o l\u00eddra. \u010cek\u00e1, a\u017e bude vybr\u00e1n nov\u00fd master, ke kter\u00e9mu se pak p\u0159ipoj\u00ed. To je dodate\u010dn\u00e9 opat\u0159en\u00ed proti ztr\u00e1t\u011b dat.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/f760788e5951facdd08b2069037830fa.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/c32f26e7526e1873f3bfd7d3693fcc72.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Zde m\u00e1me od produktov\u00e9ho t\u00fdmu informaci, \u017ee jejich produkt m\u00e1 probl\u00e9my p\u0159i pr\u00e1ci s Postgres. P\u0159itom nen\u00ed mo\u017en\u00e9 se p\u0159ipojit k samotn\u00e9mu masterovi, proto\u017ee nen\u00ed dostupn\u00fd p\u0159es SSH. A automatick\u00fd failover tak\u00e9 neprob\u00edh\u00e1. <\/p>\n<p><\/p>\n<p>Tento host byl nucen\u011b restartov\u00e1n. Kv\u016fli restartu do\u0161lo k automatick\u00e9mu failoveru, p\u0159i\u010dem\u017e bylo mo\u017en\u00e9 prov\u00e9st i manu\u00e1ln\u00ed automatick\u00fd failover, jak nyn\u00ed ch\u00e1pu. A po restartu jdeme ji\u017e zkontrolovat, co se d\u011blo s aktu\u00e1ln\u00edm masterem. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/95efb82e0647a396a21683cd53a69547.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>P\u0159itom jsme ji\u017e p\u0159edem v\u011bd\u011bli, \u017ee m\u00e1me probl\u00e9my s disky, tj. podle monitorov\u00e1n\u00ed jsme ji\u017e v\u011bd\u011bli, kde p\u00e1trat a co hledat. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/367aaab607d833d96a573ddbbbced502.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Pod\u00edvali jsme se do postgresov\u00e9ho logu a za\u010dali jsme zji\u0161\u0165ovat, co se tam d\u011bje. Vid\u011bli jsme commity, kter\u00e9 trvaly jednu, dv\u011b, t\u0159i sekundy, co\u017e v\u016fbec nen\u00ed norm\u00e1ln\u00ed. Vid\u011bli jsme, \u017ee n\u00e1\u0161 automatick\u00fd vakuum se spou\u0161t\u00ed velmi dlouho a zvl\u00e1\u0161tn\u011b. A vid\u011bli jsme do\u010dasn\u00e9 soubory na disku. Tj. to v\u0161e jsou ukazatele probl\u00e9m\u016f s disky. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/7e0cfce7a81901ef9ef736c3fefc8dce.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Pod\u00edvali jsme se do syst\u00e9mov\u00e9ho dmesg (do logu j\u00e1drov\u00fdch zpr\u00e1v). A zjistili jsme, \u017ee m\u00e1me probl\u00e9m s jedn\u00edm z disk\u016f. Diskov\u00fd subsyst\u00e9m p\u0159edstavoval softwarov\u00fd Raid. Pod\u00edvali jsme se na \/proc\/mdstat a zjistili, \u017ee n\u00e1m chyb\u00ed jeden disk. Tj. je to Raid z 8 disk\u016f, m\u00e1me jeden chyb\u011bj\u00edc\u00ed. Pokud se pozorn\u011b pod\u00edv\u00e1te na sn\u00edmek, m\u016f\u017eete v v\u00fdstupu vid\u011bt, \u017ee n\u00e1m tam chyb\u00ed sde. Podm\u00edn\u011bn\u011b \u0159e\u010deno, disk vypadl. To spustilo diskov\u00e9 probl\u00e9my a aplikace tak\u00e9 za\u017e\u00edvaly probl\u00e9my p\u0159i pr\u00e1ci s clusterem Postgres.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/4b09b3aa761cb504fa79173ef280faac.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>A v tomto p\u0159\u00edpad\u011b by n\u00e1m Patroni nijak nepomohl, proto\u017ee Patroni nem\u00e1 \u00fakol sledovat stav serveru, stav disku. Tyto situace mus\u00edme sledovat extern\u00edm monitoringem. Rychle jsme do extern\u00edho monitoringu p\u0159idali monitoring disk\u016f. <\/p>\n<p><\/p>\n<p>A byla takov\u00e1 my\u0161lenka \u2013 mohl by n\u00e1m pomoci fencing, nebo softwarov\u00fd watchdog? Mysleli jsme, \u017ee by n\u00e1m v tomto p\u0159\u00edpad\u011b asi moc nepomohl, proto\u017ee b\u011bhem probl\u00e9m\u016f Patroni nad\u00e1le interagoval s clusterem DCS a nevid\u011bl \u017e\u00e1dn\u00fd probl\u00e9m. Tj. z pohledu DCS a Patroni bylo s clusterem v\u0161echno v po\u0159\u00e1dku, i kdy\u017e ve skute\u010dnosti byly probl\u00e9my s diskem, byly probl\u00e9my s dostupnost\u00ed datab\u00e1ze. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/15fdac78535355d4eb889efcc27efc46.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Podle m\u00e9ho n\u00e1zoru je to jeden z nejpodivn\u011bj\u0161\u00edch probl\u00e9m\u016f, kter\u00e9 jsem dlouho zkoumal, p\u0159e\u010detl jsem spoustu log\u016f a nazval jsem ho cluster-simulant. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/b5259774931c290eda6673deb14b6c48.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Probl\u00e9m spo\u010d\u00edval v tom, \u017ee star\u00fd master se nemohl st\u00e1t norm\u00e1ln\u00ed replikou, tj. Patroni ho spou\u0161t\u011bl, Patroni ukazoval, \u017ee tento uzel je p\u0159\u00edtomen jako replikovan\u00e1, ale z\u00e1rove\u0148 to nebyla norm\u00e1ln\u00ed replikace. Te\u010f uvid\u00edte, pro\u010d. To mi z\u016fstalo z anal\u00fdzy toho probl\u00e9mu. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/c35a5d102a46764bbf96e5a75ee88171.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>A jak to v\u0161echno za\u010dalo? Za\u010dalo to, jak v p\u0159edchoz\u00edm probl\u00e9mu, s diskov\u00fdmi zpo\u017ed\u011bn\u00edmi. M\u011bli jsme commity za sekundu, za dv\u011b. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/435c1b350a89f288e0e10ca9d5a2ec6e.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Byly p\u0159eru\u0161en\u00ed spojen\u00ed, tj. klienti se odpojovali. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/920cb1f3f3b40578e15e6ff711e85c50.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Byly blokace r\u016fzn\u00e9 z\u00e1va\u017enosti. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/285d6292d89b928a85e95602ee57d38a.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>A odpov\u00eddaj\u00edc\u00edm zp\u016fsobem diskov\u00fd subsyst\u00e9m nebyl p\u0159\u00edli\u0161 rychl\u00fd. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/ebe71d49360e7f604ecd6a3c8276cb73.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>A to nejz\u00e1hadn\u011bj\u0161\u00ed pro m\u011b \u2013 to byl p\u0159ijat\u00fd immediate shutdown request. Postgres m\u00e1 t\u0159i re\u017eimy vypnut\u00ed:<\/p>\n<p><\/p>\n<ul>\n<li>To je graceful, kdy \u010dek\u00e1me, ne\u017e se v\u0161ichni klienti odpoj\u00ed sami. <\/li>\n<li>Existuje fast, kdy donut\u00edme klienty, aby se odpojili, proto\u017ee se chyst\u00e1me na vypnut\u00ed. <\/li>\n<li>A immediate. In this case, immediate doesn't even notify clients that they need to disconnect; it simply shuts down without warning. Meanwhile, all clients receive a message from the operating system, RST (a TCP message indicating that the connection has been interrupted and there is nothing more for the client to catch). <\/li>\n<\/ul>\n<p><\/p>\n<p>Who sent this signal? Background processes of Postgres don't send such signals to each other, meaning this is kill-9. They don't send such signals to each other; they only react to them, indicating an emergency restart of Postgres. I do not know who sent it. <\/p>\n<p><\/p>\n<p>I checked using the 'last' command and saw one person who also logged into this server with us, but I hesitated to ask. It could have been kill -9. I would have seen kill -9 in the logs because Postgres logs that it received kill -9, but I didn't see this in the logs. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/fe8e844649cea3384ebfde86e36eba44.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Upon further investigation, I noticed that Patroni hadn't written to the logs for quite a while\u201454 seconds. When comparing the two timestamps, there were no messages for approximately 54 seconds. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/ffb1c4db3d7de591d77d56ed2fcc4f2a.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>During this time, an auto-failover occurred. Patroni worked excellently again. Our old master was unavailable; something was happening with it. Elections for a new master began. Everything worked well here. Our pgsql01 became the new leader. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/6b2457653e9f77af44cd64b4aff021a0.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>We have a replica that became the master, and there is a second replica. We had issues with the second replica. It was trying to reconfigure itself. As I understand it, it was trying to change recovery.conf, restart Postgres, and connect to the new master. Every 10 seconds, it sent messages that it was trying, but it was unsuccessful. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/381818f189e19c3e1154074b4652f837.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>During these attempts, an immediate-shutdown signal reached the old master. The master restarted. Recovery also stopped because the old master went into reboot mode. This means the replica couldn't connect to it because it was shutting down. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/d8a8cde925fa8fe601d44ab908ae69dc.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>At some point, it managed to start, but replication did not initiate. <\/p>\n<p><\/p>\n<p>I have a single hypothesis that the recovery.conf contained the address of the old master. And when the new master appeared, the second replica continued to try to connect to the old master. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/425ea3af1f1e186f4760e1fb1f5419fa.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>When Patroni started on the second replica, the node started but could not connect for replication. This created a replication lag that looked something like this. That is, all three nodes were in place, but the second node was lagging. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/f564eacd9944666c9a3f5bea634ede6b.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>P\u0159i pohledu do z\u00e1znam\u016f, kter\u00e9 byly vytvo\u0159eny, bylo mo\u017eno vid\u011bt, \u017ee replikace nemohla za\u010d\u00edt, proto\u017ee transak\u010dn\u00ed logy se li\u0161ily. A tyto transak\u010dn\u00ed logy, kter\u00e9 nab\u00edz\u00ed master, uveden\u00e9 v recovery.conf, jednodu\u0161e nevyhovovaly na\u0161emu aktu\u00e1ln\u00edmu uzlu. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/6b09947d157c9ef0a190da2df2cb5892.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>A tady jsem se dopustil chyby. M\u011bl jsem se pod\u00edvat, co je v recovery.conf, abych ov\u011b\u0159il svou hypot\u00e9zu, \u017ee se p\u0159ipojujeme k nespr\u00e1vn\u00e9mu masteru. Ale tehdy jsem se s t\u00edm teprve seznamoval a nenapadlo m\u011b to, nebo jsem vid\u011bl, \u017ee replika zaost\u00e1v\u00e1 a \u017ee ji budu muset p\u0159enastavit, tzn. odpracoval jsem to tak n\u011bjak na rychlo. To byla moje chyba. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/159f2256fd81428fd0d82a255528887d.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Po t\u0159iceti minut\u00e1ch u\u017e p\u0159i\u0161el administr\u00e1tor, tzn. j\u00e1 jsem restartoval Patroni na replice. U\u017e jsem ji odbyl, myslel jsem, \u017ee ji budu muset p\u0159enastavit. A pomyslel jsem si \u2013 restartuji Patroni, mo\u017en\u00e1 se n\u011bco povede. Obnovil se recovery. A datab\u00e1ze se dokonce otev\u0159ela, byla p\u0159ipravena p\u0159ij\u00edmat spojen\u00ed. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/aeac98750386c8389d39e006a229cc85.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Replikace se rozb\u011bhla. Ale po minut\u011b se to zhroutilo s chybou, \u017ee j\u00ed nevyhovuj\u00ed transak\u010dn\u00ed logy. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/17eaba000a79c80c29d1bc3902de05e6.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Pomyslel jsem si, \u017ee to zkus\u00edm restartovat je\u0161t\u011b jednou. Znovu jsem restartoval Patroni, a to jsem nerestartoval Postgres, ale pr\u00e1v\u011b Patroni, v nad\u011bji, \u017ee kouzeln\u00fdm zp\u016fsobem spust\u00ed datab\u00e1zi. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/a85b38e033c45ad73da317f41e0ef24a.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Replikace se znovu rozb\u011bhla, ale polo\u017eky v transak\u010dn\u00edm logu se li\u0161ily, nebyly to ty, kter\u00e9 byly p\u0159i p\u0159edchoz\u00edm pokusu o spu\u0161t\u011bn\u00ed. Replikace se znovu zastavila. A zpr\u00e1va byla u\u017e trochu jin\u00e1. A nebyla pro m\u011b p\u0159\u00edli\u0161 informativn\u00ed. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/fd90d708230bd54af685786c7e361ee1.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>A tu mi blesklo hlavou \u2013 co kdy\u017e restartuji Postgres, p\u0159i\u010dem\u017e na aktu\u00e1ln\u00edm masteru ud\u011bl\u00e1m checkpoint, abych posunul m\u00edsto v transak\u010dn\u00edm logu trochu dop\u0159edu, abych spustil recovery z jin\u00e9ho bodu? Nav\u00edc tam byly je\u0161t\u011b rezervy WAL. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/19e67f95d1a74141a013947c9aa39e38.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Restartoval jsem Patroni, ud\u011blal p\u00e1r checkpoint\u016f na masteru, p\u00e1r restartovac\u00edch bod\u016f na replice, kdy\u017e se otev\u0159ela. A to pomohlo. Dlouho jsem p\u0159em\u00fd\u0161lel, pro\u010d to pomohlo a jak to fungovalo. A replika se rozb\u011bhla. A replikace u\u017e se neodpojila. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/418ca59a681be84f51ed374a73a759c9.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Takov\u00fd probl\u00e9m mi p\u0159ijde jako jeden z v\u00edce z\u00e1hadn\u00fdch, nad kter\u00fdm st\u00e1le p\u0159em\u00fd\u0161l\u00edm, co se tam vlastn\u011b d\u011blo. <\/p>\n<p><\/p>\n<p>Jak\u00e9 z\u00e1v\u011bry lze u\u010dinit? Patroni m\u016f\u017ee fungovat, jak bylo zam\u00fd\u0161leno, a to bez jak\u00fdchkoli chyb. Ale to neznamen\u00e1, \u017ee to zaru\u010duje, \u017ee v\u0161echno funguje spr\u00e1vn\u011b. Replika m\u016f\u017ee b\u00fdt aktivov\u00e1na, ale m\u016f\u017ee b\u00fdt v polovi\u010dn\u00edm provozn\u00edm stavu, a aplikace nem\u016f\u017ee pracovat s takovou replikou, proto\u017ee tam budou star\u00e1 data. <\/p>\n<p><\/p>\n<p>A po failoveru je v\u017edy t\u0159eba zkontrolovat, zda je n\u00e1\u0161 cluster v po\u0159\u00e1dku, tj. \u017ee m\u00e1me pot\u0159ebn\u00fd po\u010det replik a \u017e\u00e1dn\u00fd zpo\u017ed\u011bn\u00ed replikace.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/aca7047ff33d3efe14071b798b554091.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>A b\u011bhem zkoum\u00e1n\u00ed t\u011bchto probl\u00e9m\u016f formuluj\u00ed doporu\u010den\u00ed. Sna\u017eil jsem se je shrnout do dvou slid\u016f. Pravd\u011bpodobn\u011b by bylo mo\u017en\u00e9 v\u0161echny p\u0159\u00edb\u011bhy slou\u010dit do dvou slid\u016f a pouze je p\u0159edn\u00e9st.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/d4ad31b26d83c8846fe4097a11d70849.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Kdy\u017e pou\u017e\u00edv\u00e1te Patroni, mus\u00edte m\u00edt neust\u00e1l\u00fd monitoring. Mus\u00edte v\u017edy v\u011bd\u011bt, kdy do\u0161lo k automatick\u00e9mu failoveru, proto\u017ee pokud nev\u00edte, \u017ee do\u0161lo k automatick\u00e9mu failoveru, nekontrolujete cluster. A to je \u0161patn\u011b.<\/p>\n<p><\/p>\n<p>Po ka\u017ed\u00e9m failoveru bychom m\u011bli ru\u010dn\u011b zkontrolovat cluster. Mus\u00edme se ujistit, \u017ee m\u00e1me v\u017edy aktu\u00e1ln\u00ed po\u010det replik, \u017ee neexistuje zpo\u017ed\u011bn\u00ed replikace, v log\u00e1ch nejsou \u017e\u00e1dn\u00e9 chyby souvisej\u00edc\u00ed s tokovou replikac\u00ed, s Patroni, se syst\u00e9mem DCS.<\/p>\n<p><\/p>\n<p>Automatizace m\u016f\u017ee \u00fasp\u011b\u0161n\u011b fungovat, Patroni je velmi dobr\u00fd n\u00e1stroj. M\u016f\u017ee fungovat, ale to neprovede cluster do po\u017eadovan\u00e9ho stavu. A pokud se o tom nedozv\u00edme, budeme m\u00edt probl\u00e9my.<\/p>\n<p><\/p>\n<p>A Patroni nen\u00ed st\u0159\u00edbrn\u00e1 kulka. P\u0159esto mus\u00edme m\u00edt p\u0159edstavu o tom, jak funguje Postgres, jak funguje replikace a jak Patroni spolupracuje s Postgres a jak je zaji\u0161t\u011bna interakce mezi uzly. To je nezbytn\u00e9 pro to, abychom mohli sami opravovat vznikl\u00e9 probl\u00e9my.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/ea4162df3561d2ea7001f2ef6788eaaa.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Jak p\u0159istupuji k diagnostice? Tak se stalo, \u017ee pracujeme s r\u016fzn\u00fdmi klienty a nikdo nem\u00e1 stack ELK, a tak mus\u00edme t\u0159\u00eddit logy, otev\u0159\u00edt 6 konzol\u00ed a 2 z\u00e1lo\u017eky. V jedn\u00e9 z\u00e1lo\u017ece jsou logy Patroni pro ka\u017ed\u00fd uzel, ve druh\u00e9 z\u00e1lo\u017ece jsou logy Consul nebo Postgres podle pot\u0159eby. Diagnostikovat to je velmi t\u011b\u017ek\u00e9. <\/p>\n<p><\/p>\n<p>Jak\u00e9 p\u0159\u00edstupy jsem vyvinul? Za prv\u00e9, v\u017edy sleduji, kdy do\u0161lo k failoveru. A pro m\u011b je to jist\u00fd vodod\u011bl. Sleduji, co se stalo p\u0159ed failoverem, b\u011bhem failoveru a po failoveru. Failover m\u00e1 dv\u011b \u010dasov\u00e9 zna\u010dky: \u010das za\u010d\u00e1tku a konce. <\/p>\n<p><\/p>\n<p>D\u00e1le se d\u00edv\u00e1m v logech na ud\u00e1losti p\u0159ed failoverem, co mu p\u0159edch\u00e1zelo, tj. hled\u00e1m d\u016fvody, pro\u010d k failoveru do\u0161lo. <\/p>\n<p><\/p>\n<p>A to d\u00e1v\u00e1 obr\u00e1zek porozum\u011bn\u00ed, co se d\u011blo a co lze ud\u011blat v budoucnu, aby se takov\u00e9 okolnosti neopakovaly (a t\u00edm p\u00e1dem se nestal failover).<\/p>\n<p><\/p>\n<p>A kam se obvykle d\u00edv\u00e1me? J\u00e1 se d\u00edv\u00e1m:<\/p>\n<p><\/p>\n<ul>\n<li>Nejprve do log\u016f Patroni. <\/li>\n<li>D\u00e1le se d\u00edv\u00e1m do log\u016f Postgres nebo do log\u016f DCS, v z\u00e1vislosti na tom, co jsem na\u0161el v logech Patroni. <\/li>\n<li>A logy syst\u00e9mu tak\u00e9 ob\u010das poskytuj\u00ed porozum\u011bn\u00ed tomu, co bylo p\u0159\u00ed\u010dinou failoveru. <\/li>\n<\/ul>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/a53618e366020e1a550d852fc308c188.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Jak se d\u00edv\u00e1m na Patroni? Na Patroni se d\u00edv\u00e1m velmi pozitivn\u011b. Podle m\u011b je to to nejlep\u0161\u00ed, co dnes existuje. Zn\u00e1m mnoho dal\u0161\u00edch produkt\u016f. To jsou Stolon, Repmgr, Pg_auto_failover, PAF. \u010cty\u0159i n\u00e1stroje. Vyzkou\u0161el jsem v\u0161echny. Patroni se mi l\u00edbilo nejv\u00edce. <\/p>\n<p><\/p>\n<p>Pokud se m\u011b zept\u00e1te: \u201eDoporu\u010duji Patroni?\u201c. Odpov\u00edm, \u017ee ano, proto\u017ee Patroni se mi l\u00edb\u00ed. A mysl\u00edm, \u017ee jsem se ho nau\u010dil dob\u0159e pou\u017e\u00edvat. <\/p>\n<p><\/p>\n<p>Pokud v\u00e1s zaj\u00edm\u00e1, jak\u00e9 dal\u0161\u00ed probl\u00e9my mohou b\u00fdt s Patroni krom\u011b t\u011bch, kter\u00e9 jsem zm\u00ednil, m\u016f\u017eete se v\u017edy pod\u00edvat na str\u00e1nku <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/zalando\/patroni\/issues\/\">issues<\/a><\/noindex> na GitHubu. Tam je mnoho r\u016fzn\u00fdch p\u0159\u00edb\u011bh\u016f a diskutuje se tam o mnoha zaj\u00edmav\u00fdch probl\u00e9mech. A nakonec byly odhaleny a vy\u0159e\u0161eny n\u011bjak\u00e9 chyby, tj. je to zaj\u00edmav\u00e9 \u010dten\u00ed. <\/p>\n<p><\/p>\n<p>Jsou tam zaj\u00edmav\u00e9 p\u0159\u00edb\u011bhy o tom, jak si lid\u00e9 st\u0159\u00edlej\u00ed do vlastn\u00edch nohou. Velmi pou\u010dn\u00e9. \u010cte\u0161 a ch\u00e1pe\u0161, \u017ee takto to d\u011blat nesm\u00ed\u0161. Ud\u011blal jsem si za to \u010d\u00e1rku. <\/p>\n<p><\/p>\n<p>A cht\u011bl bych moc pod\u011bkovat spole\u010dnosti Zalando za to, \u017ee tento projekt rozv\u00edjej\u00ed, konkr\u00e9tn\u011b Alexandru Kuku\u0161kovi a Alexejovi Kljukinu. Alexej Kljukin je jedn\u00edm ze spoluautor\u016f, ji\u017e v Zalando nepracuje, ale to jsou dva lid\u00e9, kte\u0159\u00ed za\u010dali pracovat s t\u00edmto produktem. <\/p>\n<p><\/p>\n<p>A pova\u017euji Patroni za velmi skv\u011blou v\u011bc. Jsem spokojen\u00fd, \u017ee existuje, je s n\u00edm zaj\u00edmav\u00e9 pracovat. A velk\u00e9 d\u00edky v\u0161em p\u0159isp\u011bvatel\u016fm, kte\u0159\u00ed p\u00ed\u0161\u00ed opravy pro Patroni. Douf\u00e1m, \u017ee Patroni s v\u011bkem bude st\u00e1le vyzr\u00e1lej\u0161\u00ed, skv\u011blej\u0161\u00ed a funk\u010dn\u011bj\u0161\u00ed. Ji\u017e te\u010f je funk\u010dn\u00ed, ale douf\u00e1m, \u017ee se to bude je\u0161t\u011b zlep\u0161ovat. Proto, pokud pl\u00e1nujete pou\u017e\u00edt Patroni, nebojte se. Je to dobr\u00e9 \u0159e\u0161en\u00ed, kter\u00e9 lze implementovat a pou\u017e\u00edvat. <\/p>\n<p><\/p>\n<p>Na tom je v\u0161e. Pokud m\u00e1te n\u011bjak\u00e9 ot\u00e1zky, ptejte se.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories nebo Jak rozb\u00edt sv\u016fj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/85c2fef2455999b48413c9fc3b235ed6.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Ot\u00e1zky<\/p>\n<p><\/p>\n<p><em>D\u011bkuji za prezentaci! Pokud i po failoveru mus\u00edme znovu d\u00e1vat pozor, pro\u010d tedy pot\u0159ebujeme automatick\u00fd failover?<\/em> <\/p>\n<p><\/p>\n<p>Proto\u017ee je to nov\u00e1 v\u011bc. Pracujeme s n\u00ed teprve rok. Je lep\u0161\u00ed se pojistit. Chceme se pod\u00edvat a ujistit se, \u017ee v\u0161echno skute\u010dn\u011b fungovalo tak, jak m\u011blo. Je to \u00farove\u0148 dosp\u011bl\u00e9ho ned\u016fv\u011bry \u2013 rad\u011bji znovu zkontrolovat a pod\u00edvat se. <\/p>\n<p><\/p>\n<p><em>Nap\u0159\u00edklad, r\u00e1no jsme se pod\u00edvali, \u017ee?<\/em><\/p>\n<p><\/p>\n<p>Ne r\u00e1no, obvykle se dozv\u00edd\u00e1me o auto failoveru prakticky okam\u017eit\u011b. Dost\u00e1v\u00e1me upozorn\u011bn\u00ed, vid\u00edme, \u017ee do\u0161lo k auto failoveru. V\u011bt\u0161inou se hned pod\u00edv\u00e1me. Ale v\u0161echny tyto kontroly by m\u011bly b\u00fdt p\u0159esunuty na \u00farove\u0148 monitorov\u00e1n\u00ed. Pokud se obr\u00e1t\u00edme na Patroni p\u0159es REST API, existuje historie. Podle historie m\u016f\u017eeme sledovat \u010dasov\u00e9 zna\u010dky, kdy do\u0161lo k failoveru. Na z\u00e1klad\u011b toho lze ud\u011blat monitorov\u00e1n\u00ed. M\u016f\u017eeme sledovat historii, kolik tam bylo ud\u00e1lost\u00ed. Pokud n\u00e1m p\u0159ibylo ud\u00e1lost\u00ed, do\u0161lo k auto failoveru. M\u016f\u017eeme se pod\u00edvat. Anebo na\u0161e automatizace v monitorov\u00e1n\u00ed zkontrolovala, \u017ee v\u0161echny repliky jsou na m\u00edst\u011b, nen\u00ed \u017e\u00e1dn\u00e9 zpo\u017ed\u011bn\u00ed a v\u0161e je v po\u0159\u00e1dku. <\/p>\n<p><\/p>\n<p><em>D\u011bkuji!<\/em><\/p>\n<p><\/p>\n<p><em>Mockr\u00e1t d\u011bkuji za skv\u011bl\u00fd p\u0159\u00edb\u011bh! Pokud jsme p\u0159esunuli DCS cluster n\u011bkam daleko od Postgres clusteru, je nutn\u00e9 tento cluster tak\u00e9 pravideln\u011b udr\u017eovat? Jak\u00e9 jsou osv\u011bd\u010den\u00e9 postupy, co se t\u00fdk\u00e1 vyp\u00edn\u00e1n\u00ed n\u011bkter\u00fdch \u010d\u00e1st\u00ed DCS clusteru, co s nimi d\u011blat atd.? Jak v tomto p\u0159\u00edpad\u011b funguje cel\u00e1 tato konstrukce? A jak tyto v\u011bci d\u011blat?<\/em><\/p>\n<p><\/p>\n<p>Pro jednu spole\u010dnost bylo nutn\u00e9 vytvo\u0159it matici probl\u00e9m\u016f, co se stane, pokud n\u011bkter\u00e1 nebo n\u011bkolik komponent vypadne. Na z\u00e1klad\u011b t\u00e9to matice postupn\u011b proch\u00e1zej\u00edme v\u0161echny komponenty a stav\u00edme sc\u00e9n\u00e1\u0159e pro p\u0159\u00edpad poruchy t\u011bchto komponent. Na ka\u017ed\u00fd sc\u00e9n\u00e1\u0159 poruchy lze m\u00edt pl\u00e1n pro obnovu. A v p\u0159\u00edpad\u011b DCS to jde jako sou\u010d\u00e1st standardn\u00ed infrastruktury. A admin to spravuje, a my u\u017e se spol\u00e9h\u00e1me na adminy, kte\u0159\u00ed to spravuj\u00ed, a na jejich schopnosti to opravit v p\u0159\u00edpad\u011b hav\u00e1rie. Pokud DCS v\u016fbec nen\u00ed, nasazujeme ho my, ale p\u0159itom se o n\u011bj zvl\u00e1\u0161\u0165 nestar\u00e1me, proto\u017ee za infrastrukturu neodpov\u00edd\u00e1me, ale d\u00e1v\u00e1me doporu\u010den\u00ed, jak a co monitorovat. <\/p>\n<p><\/p>\n<p><em>To znamen\u00e1, rozum\u00edm tomu spr\u00e1vn\u011b, \u017ee je t\u0159eba vypnout Patroni, vypnout failover, vypnout v\u0161e, ne\u017e n\u011bco ud\u011bl\u00e1me s hostiteli?<\/em><\/p>\n<p><\/p>\n<p>To z\u00e1vis\u00ed na tom, kolik m\u00e1me uzl\u016f v DCS clusteru. Pokud je uzl\u016f hodn\u011b a odstaven\u00ed nastane jen u jednoho uzlu (repliky), z\u016fst\u00e1v\u00e1 v clusteru kv\u00f3rum. A Patroni z\u016fst\u00e1v\u00e1 funk\u010dn\u00ed. A nic se neaktivuje. Pokud m\u00e1me n\u011bjak\u00e9 slo\u017eit\u00e9 operace, kter\u00e9 zasahuj\u00ed v\u00edce uzl\u016f a jejich\u017e absence m\u016f\u017ee rozpadnout kv\u00f3rum, pak ano, mo\u017en\u00e1 m\u00e1 smysl pozastavit Patroni. M\u00e1 k tomu odpov\u00eddaj\u00edc\u00ed p\u0159\u00edkaz \u2013 patronictl pause, patronictl resume. Jenom pozastav\u00edme a automatick\u00e9 p\u0159epnut\u00ed (failover) se v tuto dobu nespust\u00ed. Provedeme \u00fadr\u017ebu na DCS clusteru, pak pauzu zru\u0161\u00edme a pokra\u010dujeme d\u00e1l.<\/p>\n<p><\/p>\n<p><em>Moc d\u011bkuji!<\/em><\/p>\n<p><\/p>\n<p><em>Velmi d\u011bkuji za prezentaci! Jak se produktov\u00fd t\u00fdm stav\u00ed k tomu, \u017ee data mohou b\u00fdt ztracena?<\/em> <\/p>\n<p><\/p>\n<p>Produktov\u00fdm t\u00fdm\u016fm je to jedno, ale v\u016fdcov\u00e9 t\u00fdm\u016f maj\u00ed obavy. <\/p>\n<p><\/p>\n<p><em>Jak\u00e9 jsou tam z\u00e1ruky?<\/em><\/p>\n<p><\/p>\n<p>Se z\u00e1rukami je to velice obt\u00ed\u017en\u00e9. Je zde prezentace od Alexandra Kuku\u0161kina \u201eJak vypo\u010d\u00edtat RPO a RTO\u201c, tj. \u010das obnoven\u00ed a kolik dat m\u016f\u017eeme ztratit. Mysl\u00edm, \u017ee je pot\u0159eba tyto diapozitivy naj\u00edt a prostudovat. Podle m\u00fdch vzpom\u00ednek tam jsou konkr\u00e9tn\u00ed kroky, jak tyto v\u011bci vypo\u010d\u00edtat. Kolik transakc\u00ed m\u016f\u017eeme ztratit, kolik dat m\u016f\u017eeme ztratit. Jako variantu m\u016f\u017eeme pou\u017e\u00edt synchronn\u00ed replikaci na \u00farovni Patroni, ale to je dvouse\u010dn\u00fd me\u010d: bu\u010f m\u00e1me spolehlivost dat, nebo ztr\u00e1c\u00edme na rychlosti. Existuje synchronn\u00ed replikace, ale ani ta nezaru\u010duje 100% ochranu p\u0159ed ztr\u00e1tou dat.<\/p>\n<p><\/p>\n<p><em>Alexei, d\u011bkuji za skv\u011blou prezentaci! M\u00e1te zku\u0161enosti s pou\u017e\u00edv\u00e1n\u00edm Patroni pro ochranu na nulov\u00e9 \u00farovni? Tj. v kombinaci se synchronn\u00edm standby? To je prvn\u00ed ot\u00e1zka. A druh\u00e1 ot\u00e1zka. Pou\u017e\u00edvali jste r\u016fzn\u00e1 \u0159e\u0161en\u00ed. My jsme pou\u017e\u00edvali Repmgr, ale bez automatizovan\u00e9ho p\u0159epnut\u00ed a te\u010f pl\u00e1nujeme implementaci automatizovan\u00e9ho p\u0159epnut\u00ed. Zva\u017eujeme Patroni jako alternativn\u00ed \u0159e\u0161en\u00ed. Co m\u016f\u017eete \u0159\u00edct jako v\u00fdhody pr\u00e1v\u011b ve srovn\u00e1n\u00ed s Repmgr?<\/em><\/p>\n<p><\/p>\n<p>Prvn\u00ed ot\u00e1zka se t\u00fdkala synchronn\u00edch replik. U n\u00e1s nikdo synchronn\u00ed replikaci nepou\u017e\u00edv\u00e1, proto\u017ee se v\u0161ichni boj\u00ed (Ji\u017e ji pou\u017e\u00edv\u00e1 n\u011bkolik klient\u016f, probl\u00e9my s v\u00fdkonnost\u00ed obecn\u011b nezaznamenali \u2014 <em>Pozn\u00e1mka p\u0159edn\u00e1\u0161ej\u00edc\u00edho<\/em>). Pro sebe jsme si stanovili pravidlo, \u017ee v clusteru synchronn\u00ed replikace mus\u00ed b\u00fdt minim\u00e1ln\u011b t\u0159i uzly, proto\u017ee pokud m\u00e1me dva uzly a pokud dojde k selh\u00e1n\u00ed masteru nebo repliky, Patroni p\u0159evede tento uzel do re\u017eimu Standalone, aby aplikace mohla pokra\u010dovat v chodu. V tomto p\u0159\u00edpad\u011b existuj\u00ed rizika ztr\u00e1ty dat.<\/p>\n<p><\/p>\n<p>Co se t\u00fd\u010de druh\u00e9 ot\u00e1zky, pou\u017eili jsme Repmgr a st\u00e1le ho pou\u017e\u00edv\u00e1me u n\u011bkter\u00fdch klient\u016f z historick\u00fdch d\u016fvod\u016f. Co lze \u0159\u00edci? V Patroni je auto failover zabudovan\u00fd v z\u00e1kladu, v Repmgr je auto failover dodate\u010dn\u00e1 funkce, kterou je t\u0159eba zapnout. Je nutn\u00e9 spustit Repmgr daemon na ka\u017ed\u00e9m uzlu a pak m\u016f\u017eeme nastavit auto failover. <\/p>\n<p><\/p>\n<p>Repmgr kontroluje, zda jsou uzly Postgres \u017eiv\u00e9. Procesy Repmgr kontroluj\u00ed p\u0159\u00edtomnost jeden druh\u00e9ho, co\u017e nen\u00ed p\u0159\u00edli\u0161 efektivn\u00ed p\u0159\u00edstup, proto\u017ee mohou nastat slo\u017eit\u00e9 p\u0159\u00edpady s\u00ed\u0165ov\u00e9 izolace, kdy se velk\u00fd Repmgr cluster m\u016f\u017ee rozpadnout na n\u011bkolik men\u0161\u00edch a p\u0159esto pokra\u010dovat v pr\u00e1ci. Dlouho jsem nesledoval Repmgr, mo\u017en\u00e1 to opravili\u2026 nebo tak\u00e9 ne. P\u0159en\u00e1\u0161en\u00ed informac\u00ed o stavu clusteru do DCS, jak to d\u011bl\u00e1 Stolon, Patroni, je nej\u017eivotaschopn\u011bj\u0161\u00ed varianta. <\/p>\n<p><\/p>\n<p><em>Alexei, m\u00e1m ot\u00e1zku, mo\u017en\u00e1 naivn\u00ed. V jednom z prvn\u00edch p\u0159\u00edklad\u016f DCS jste p\u0159enesli z lok\u00e1ln\u00edho stroje na vzd\u00e1len\u00fd uzel. Ch\u00e1peme, \u017ee s\u00ed\u0165 \u2013 to je v\u011bc, kter\u00e1 m\u00e1 sv\u00e9 zvl\u00e1\u0161tnosti, \u017eije sama o sob\u011b. A co se stane, pokud se z n\u011bjak\u00e9ho d\u016fvodu cluster DCS stane nedostupn\u00fdm? D\u016fvody zde nebudu rozeb\u00edrat, m\u016f\u017ee jich b\u00fdt spousta: od ne\u0161ikovn\u00fdch s\u00ed\u0165a\u0159\u016f po skute\u010dn\u00e9 probl\u00e9my.<\/em> <\/p>\n<p><\/p>\n<p>Ne\u0159ekl jsem to nahlas, ale cluster DCS by m\u011bl b\u00fdt tak\u00e9 odoln\u00fd proti chyb\u00e1m, tj. mus\u00ed m\u00edt lich\u00fd po\u010det uzl\u016f, aby se mohla shrom\u00e1\u017edit kvorum. Co se stane, pokud se cluster DCS stane nedostupn\u00fdm, nebo se nem\u016f\u017ee shrom\u00e1\u017edit kvorum, tj. n\u011bjak\u00fd s\u00ed\u0165ov\u00fd split nebo selh\u00e1n\u00ed uzl\u016f? V tomto p\u0159\u00edpad\u011b p\u0159ech\u00e1z\u00ed cluster Patroni do re\u017eimu read only. Cluster Patroni nem\u016f\u017ee ur\u010dit stav clusteru a co by m\u011bl d\u011blat. Nem\u016f\u017ee se spojit s DCS a ulo\u017eit tam nov\u00fd stav clusteru, a proto cel\u00fd cluster p\u0159ech\u00e1z\u00ed do re\u017eimu read only. A \u010dek\u00e1 na ru\u010dn\u00ed z\u00e1sah oper\u00e1tora nebo a\u017e se DCS obnov\u00ed.<\/p>\n<p><\/p>\n<p><em>Jednodu\u0161e \u0159e\u010deno, DCS se pro n\u00e1s st\u00e1v\u00e1 slu\u017ebou stejn\u011b d\u016fle\u017eitou jako samotn\u00e1 datab\u00e1ze?<\/em><\/p>\n<p><\/p>\n<p>Ano, ano. V mnoha modern\u00edch spole\u010dnostech je Service Discovery ned\u00edlnou sou\u010d\u00e1st\u00ed infrastruktury. Implementuje se dokonce d\u0159\u00edve, ne\u017e se v infrastruktu\u0159e objev\u00ed datab\u00e1ze. \u0158ekn\u011bme, \u017ee jsme spustili infrastrukturu, rozvinuli se v datov\u00e9m centru, a my u\u017e m\u00e1me Service Discovery. Pokud je to Consul, m\u016f\u017ee na n\u011bm b\u00fdt postaveno i DNS. Pokud je to Etcd, m\u016f\u017ee to b\u00fdt \u010d\u00e1st z clusteru Kubernetes, ve kter\u00e9m bude rozvinuto v\u0161e ostatn\u00ed. M\u00e1m pocit, \u017ee Service Discovery je u\u017e ned\u00edlnou sou\u010d\u00e1st\u00ed modern\u00edch infrastruktur. A o n\u011bm se p\u0159em\u00fd\u0161l\u00ed mnohem d\u0159\u00edve, ne\u017e o datab\u00e1z\u00edch. <\/p>\n<p><\/p>\n<p><em>D\u011bkuji!<\/em><\/p>\n<p>Zdroj: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/post\/512768\/\">habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u041e\u0441\u043d\u043e\u0432\u043d\u0430\u044f \u0446\u0435\u043b\u044c Patroni \u2014 \u044d\u0442\u043e \u043e\u0431\u0435\u0441\u043f\u0435\u0447\u0435\u043d\u0438\u0435 High Availability \u0434\u043b\u044f PostgreSQL. \u041d\u043e Patroni \u2014 \u044d\u0442\u043e \u043b\u0438\u0448\u044c template, \u0430 \u043d\u0435 \u0433\u043e\u0442\u043e\u0432\u044b\u0439 \u0438\u043d\u0441\u0442\u0440\u0443\u043c\u0435\u043d\u0442 (\u0447\u0442\u043e, \u0432 \u043e\u0431\u0449\u0435\u043c, \u0438 \u0441\u043a\u0430\u0437\u0430\u043d\u043e \u0432 \u0434\u043e\u043a\u0443\u043c\u0435\u043d\u0442\u0430\u0446\u0438\u0438). \u041d\u0430 \u043f\u0435\u0440\u0432\u044b\u0439 \u0432\u0437\u0433\u043b\u044f\u0434, \u043d\u0430\u0441\u0442\u0440\u043e\u0438\u0432 Patroni \u0432 \u0442\u0435\u0441\u0442\u043e\u0432\u043e\u0439 \u043b\u0430\u0431\u0435, \u043c\u043e\u0436\u043d\u043e \u0443\u0432\u0438\u0434\u0435\u0442\u044c, \u043a\u0430\u043a\u043e\u0439 \u044d\u0442\u043e \u043f\u0440\u0435\u043a\u0440\u0430\u0441\u043d\u044b\u0439 \u0438\u043d\u0441\u0442\u0440\u0443\u043c\u0435\u043d\u0442 \u0438 \u043a\u0430\u043a \u043e\u043d \u043b\u0435\u0433\u043a\u043e \u043e\u0431\u0440\u0430\u0431\u0430\u0442\u044b\u0432\u0430\u0435\u0442 \u043d\u0430\u0448\u0438 \u043f\u043e\u043f\u044b\u0442\u043a\u0438 \u0440\u0430\u0437\u0432\u0430\u043b\u0438\u0442\u044c \u043a\u043b\u0430\u0441\u0442\u0435\u0440. \u041e\u0434\u043d\u0430\u043a\u043e \u043d\u0430 \u043f\u0440\u0430\u043a\u0442\u0438\u043a\u0435 \u0432 \u043f\u0440\u043e\u0438\u0437\u0432\u043e\u0434\u0441\u0442\u0432\u0435\u043d\u043d\u043e\u0439 [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":90104,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-90103","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.3 - aioseo.com -->\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/patroni-failure-stories-or-how-to-crash-your-postgresql-cluster-aleksej-lesovskij\" \/>\n\t\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.3\" \/>\n\t\t<meta property=\"og:locale\" content=\"cs_CZ\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47Patroni Failure Stories or How to crash your PostgreSQL cluster. \u0410\u043b\u0435\u043a\u0441\u0435\u0439 \u041b\u0435\u0441\u043e\u0432\u0441\u043a\u0438\u0439 | ProHoster\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/patroni-failure-stories-or-how-to-crash-your-postgresql-cluster-aleksej-lesovskij\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2020-07-29T11:42:32+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-07-29T11:42:32+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47P\u0159\u00edb\u011bhy selh\u00e1n\u00ed Patroni nebo jak zkolabovat sv\u016fj cluster PostgreSQL. Alexej Lesovsk\u00fd | ProHoster","description":"","canonical_url":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/patroni-failure-stories-or-how-to-crash-your-postgresql-cluster-aleksej-lesovskij","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"cs_CZ","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47Patroni Failure Stories or How to crash your PostgreSQL cluster. \u0410\u043b\u0435\u043a\u0441\u0435\u0439 \u041b\u0435\u0441\u043e\u0432\u0441\u043a\u0438\u0439 | ProHoster","og:url":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/patroni-failure-stories-or-how-to-crash-your-postgresql-cluster-aleksej-lesovskij","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-07-29T11:42:32+00:00","article:modified_time":"2020-07-29T11:42:32+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"90103","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 12:58:48","updated":"2026-02-09 16:50:35","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts\/90103","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/comments?post=90103"}],"version-history":[{"count":1,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts\/90103\/revisions"}],"predecessor-version":[{"id":158759,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts\/90103\/revisions\/158759"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/media\/90104"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/media?parent=90103"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/categories?post=90103"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/tags?post=90103"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}