{"id":90103,"date":"2020-07-29T13:42:32","date_gmt":"2020-07-29T11:42:32","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/patroni-failure-stories-or-how-to-crash-your-postgresql-cluster-aleksej-lesovskij"},"modified":"2020-07-29T13:42:32","modified_gmt":"2020-07-29T11:42:32","slug":"patroni-failure-stories-or-how-to-crash-your-postgresql-cluster-aleksej-lesovskij","status":"publish","type":"post","link":"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/patroni-failure-stories-or-how-to-crash-your-postgresql-cluster-aleksej-lesovskij","title":{"rendered":"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/6404cba13214f499d1f347cca0aacbe7.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Hlavn\u00fdm cie\u013eom Patroni je zabezpe\u010di\u0165 vysok\u00fa dostupnos\u0165 pre PostgreSQL. Ale Patroni je len \u0161abl\u00f3na, nie hotov\u00fd n\u00e1stroj (\u010do je v podstate uveden\u00e9 aj v dokument\u00e1cii). Na prv\u00fd poh\u013ead, po nastaven\u00ed Patroni v testovacej laborat\u00f3rii, m\u00f4\u017eeme vidie\u0165, ak\u00fd je to skvel\u00fd n\u00e1stroj a ako \u013eahko spracov\u00e1va na\u0161e pokusy o rozpad klasteru. Av\u0161ak v praxi, v produk\u010dnom prostred\u00ed, sa nie v\u017edy v\u0161etko odohr\u00e1va tak kr\u00e1sne a elegantne ako v testovacej laborat\u00f3rii.<\/p>\n<p><noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/ff445e6d76a2ad46a232b705e104446f.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Povedzme si nie\u010do o mne. Za\u010d\u00ednal som ako syst\u00e9mov\u00fd administr\u00e1tor. Pracoval som vo webovom v\u00fdvoji. Od roku 2014 pracujem v spolo\u010dnosti Data Egret. Firma sa zaober\u00e1 konzult\u00e1ciami v oblasti Postgres. A my sa star\u00e1me pr\u00e1ve o Postgres a ka\u017ed\u00fd de\u0148 pracujeme s Postgres, tak\u017ee m\u00e1me r\u00f4znu odbornos\u0165 t\u00fdkaj\u00facu sa prev\u00e1dzky. <\/p>\n<p><\/p>\n<p>A na konci roku 2018 sme za\u010dali pomaly pou\u017e\u00edva\u0165 Patroni. A z\u00edskali sme ur\u010dit\u00e9 sk\u00fasenosti. Diagnostikovali sme ho, ladili sme ho a dostali sme sa k svojim osved\u010den\u00fdm praxiam. A v tejto predn\u00e1\u0161ke v\u00e1m o nich poviem.<\/p>\n<p><\/p>\n<p>Okrem Postgres m\u00e1m r\u00e1d Linux. R\u00e1d sa v \u0148om hrabem a sk\u00famam, r\u00e1d zostavujem jadro. Zauj\u00edmaj\u00fa ma virtualiz\u00e1cia, kontajnery, Docker, Kubernetes. Zauj\u00edma ma to v\u0161etko, preto\u017ee sa prejavuj\u00fa star\u00e9 administr\u00e1torsk\u00e9 n\u00e1vyky. R\u00e1d sa zaober\u00e1m monitorovan\u00edm. A m\u00e1m r\u00e1d postgresov\u00e9 z\u00e1le\u017eitosti, s\u00favisiace s administr\u00e1ciou, ako je replik\u00e1cia a z\u00e1lohovanie. Vo svojom vo\u013enom \u010dase p\u00ed\u0161em v Go. Nie som softv\u00e9rov\u00fd in\u017einier, p\u00ed\u0161em len pre seba v Go. A bav\u00ed ma to. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/02cfd9293af5a8410c91e39afe2c75e6.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<ul>\n<li>Mysl\u00edm, \u017ee mnoh\u00ed z v\u00e1s vedia, \u017ee v Postgres nie je vysok\u00e1 dostupnos\u0165 (HA) priamo z krabice. Aby sme dosiahli HA, mus\u00edme nie\u010do nain\u0161talova\u0165, nastavi\u0165, vynalo\u017ei\u0165 \u00fasilie a z\u00edska\u0165 to. <\/li>\n<li>Existuje nieko\u013eko n\u00e1strojov a Patroni je jedn\u00fdm z nich, ktor\u00fd ve\u013emi dobre rie\u0161i HA. Ale po nain\u0161talovan\u00ed v\u0161etk\u00e9ho v testovacej laborat\u00f3rii a spusten\u00ed m\u00f4\u017eeme vidie\u0165, \u017ee v\u0161etko funguje, m\u00f4\u017eeme reprodukova\u0165 nejak\u00e9 probl\u00e9my a pozrie\u0165 sa, ako ich Patroni spracov\u00e1va. A zist\u00edme, \u017ee v\u0161etko to funguje skvele. <\/li>\n<li>Ale v praxi sme sa stretli s r\u00f4znymi probl\u00e9mami. A o t\u00fdchto probl\u00e9moch v\u00e1m poviem.<\/li>\n<li>Povedzme si, ako sme to diagnostikovali, \u010do sme nastavovali \u2013 pomohlo n\u00e1m to alebo nie. <\/li>\n<\/ul>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/7126b45b40dd08521e80a3c150382bec.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<ul>\n<li>Nebudem rozpr\u00e1va\u0165 o tom, ako nain\u0161talova\u0165 Patroni, preto\u017ee to sa d\u00e1 vyh\u013eada\u0165 na internete, m\u00f4\u017eete si pozrie\u0165 konfigura\u010dn\u00e9 s\u00fabory, aby ste pochopili, ako to v\u0161etko funguje a ako sa nastavuje. M\u00f4\u017eete sa obozn\u00e1mi\u0165 so sch\u00e9mami a architekt\u00farami a n\u00e1js\u0165 inform\u00e1cie o tom na internete. <\/li>\n<li>Nebudem rozpr\u00e1va\u0165 o cudzom sk\u00fasenosti. Budem hovori\u0165 len o t\u00fdch probl\u00e9moch, ktor\u00fdm sme \u010delili my sami. <\/li>\n<li>A nebudem rozpr\u00e1va\u0165 o probl\u00e9moch, ktor\u00e9 presahuj\u00fa Patroni a PostgreSQL. Ak napr\u00edklad ide o probl\u00e9my spojen\u00e9 s vyva\u017eovan\u00edm, ke\u010f sa n\u00e1m rozpadol klaster, o tom hovori\u0165 nebudem. <\/li>\n<\/ul>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/3f8d179e7c78ce83e8842ef5c72fe30d.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>A mal\u00fd disclaimer predt\u00fdm, ne\u017e za\u010dneme na\u0161u prezent\u00e1ciu. <\/p>\n<p><\/p>\n<p>V\u0161etky tieto probl\u00e9my, s ktor\u00fdmi sme sa stretli, sa vyskytli v prv\u00fdch 6-7-8 mesiacoch pou\u017e\u00edvania. S \u010dasom sme si vyvinuli svoje vlastn\u00e9 intern\u00e9 best practices. A probl\u00e9my n\u00e1m zmizli. Preto sa prezent\u00e1cia pl\u00e1novala pribli\u017ene pred pol rokom, ke\u010f to v\u0161etko bolo fresh v hlave a dobre som to v\u0161etko pam\u00e4tal. <\/p>\n<p><\/p>\n<p>Po\u010das pr\u00edpravy prezent\u00e1cie som op\u00e4\u0165 pre\u0161tudoval star\u00e9 postmortemy, pozeral logy. A niektor\u00e9 detaily mohli zabudn\u00fa\u0165, alebo niektor\u00e9 detaily mohli by\u0165 nedostato\u010dne presk\u00faman\u00e9 pri anal\u00fdze probl\u00e9mov, preto v niektor\u00fdch momentoch m\u00f4\u017ee p\u00f4sobi\u0165, \u017ee probl\u00e9my boli pos\u00faden\u00e9 ne\u00faplne, alebo je tam nedostatok inform\u00e1ci\u00ed. A preto sa za tento moment ospravedl\u0148ujem. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/30d414284a41fcbc3bf417c9da28ca52.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>\u010co je to Patroni?<\/p>\n<p><\/p>\n<ul>\n<li>Je to vzor na vytvorenie HA. Tak je nap\u00edsan\u00e9 v dokument\u00e1cii. A z m\u00f4jho poh\u013eadu je to ve\u013emi spr\u00e1vne upresnenie. Patroni nie je strieborn\u00e1 gu\u013eka, ktor\u00e1 vyrie\u0161i v\u0161etky va\u0161e probl\u00e9my, t.j. je potrebn\u00e9 vyvin\u00fa\u0165 \u00fasilie, aby to za\u010dalo fungova\u0165 a prin\u00e1\u0161alo prospech. <\/li>\n<li>Je to agenta slu\u017eba, ktor\u00e1 sa in\u0161taluje na ka\u017ed\u00fa slu\u017ebu s datab\u00e1zou a ktor\u00e1 je ak\u00fdmsi init syst\u00e9mom pre v\u00e1\u0161 Postgres. Sp\u00fa\u0161\u0165a Postgres, zastavuje, re\u0161tartuje, men\u00ed konfigur\u00e1ciu a men\u00ed topol\u00f3giu v\u00e1\u0161ho klastra. <\/li>\n<li>Aby sa teda uchov\u00e1val stav klastra a jeho aktu\u00e1lne zobrazenie, ako vyzer\u00e1, je potrebn\u00e9 nejak\u00e9 \u00falo\u017eisko. A z tohto h\u013eadiska sa Patroni vybral cestou uchov\u00e1vania stavu vo vonkaj\u0161ej s\u00fastave. Tento syst\u00e9m je distribuovan\u00e9 \u00falo\u017eisko konfigur\u00e1cie. M\u00f4\u017eu to by\u0165 Etcd, Consul, ZooKeeper, alebo kubernetes\u2019k\u00fd Etcd, teda nejak\u00e1 z t\u00fdchto mo\u017enost\u00ed. <\/li>\n<li>Jednou z vlastnost\u00ed Patroni je, \u017ee automatick\u00fd failover dostanete z krabice, sta\u010d\u00ed ho nastavit. Kdy\u017e porovn\u00e1me s Repmgr, failover je zde sou\u010d\u00e1st\u00ed. S Repmgr z\u00edsk\u00e1me switchover, ale pokud chceme automatick\u00fd failover, mus\u00edme ho nastavit zvl\u00e1\u0161\u0165. V Patroni je automatick\u00fd failover ji\u017e v z\u00e1kladu.<\/li>\n<li>A je tu mnoho dal\u0161\u00edch v\u011bc\u00ed. Nap\u0159\u00edklad spr\u00e1va konfigurac\u00ed, zav\u00e1d\u011bn\u00ed nov\u00fdch replik, z\u00e1lohov\u00e1n\u00ed atd. Ale to je mimo rozsah t\u00e9to prezentace, o tom mluvit nebudu. <\/li>\n<\/ul>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/fed0f1ce931df99ca76ae1116f8098cc.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>A mal\u00fd z\u00e1v\u011br \u2013 hlavn\u00ed \u00fakol Patroni je dob\u0159e a spolehliv\u011b prov\u00e1d\u011bt automatick\u00fd failover, aby z\u016fstal n\u00e1\u0161 cluster funk\u010dn\u00ed a aplikace si nev\u0161imla zm\u011bn v topologii clusteru. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/fac52620957efa47ced330e7cc8084e0.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Ale kdy\u017e za\u010dneme pou\u017e\u00edvat Patroni, n\u00e1\u0161 syst\u00e9m se trochu zkomplikuje. Pokud jsme p\u0159edt\u00edm m\u011bli Postgres, s pou\u017eit\u00edm Patroni z\u00edsk\u00e1v\u00e1me samotnou Patroni, DCS, kde se uchov\u00e1v\u00e1 stav. A v\u0161echno to mus\u00ed n\u011bjak fungovat. Tak\u017ee, co se m\u016f\u017ee pokazit?<\/p>\n<p><\/p>\n<p>M\u016f\u017ee se pokazit:<\/p>\n<p><\/p>\n<ul>\n<li>M\u016f\u017ee se pokazit Postgres. M\u016f\u017ee to b\u00fdt master nebo replikace, jedna z nich m\u016f\u017ee p\u0159estat fungovat. <\/li>\n<li>M\u016f\u017ee se pokazit samotn\u00e1 Patroni. <\/li>\n<li>M\u016f\u017ee se pokazit DCS, kde se uchov\u00e1v\u00e1 stav.<\/li>\n<li>A m\u016f\u017ee se pokazit s\u00ed\u0165. <\/li>\n<\/ul>\n<p><\/p>\n<p>V\u0161echny tyto aspekty budu zva\u017eovat v prezentaci. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/1aeaf46d37bf1935b514b0581e0dbd0b.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Budu zva\u017eovat p\u0159\u00edpady podle jejich slo\u017eitosti, ne z hlediska toho, \u017ee p\u0159\u00edpad zasahuje do mnoha komponent\u016f. Ale z hlediska subjektivn\u00edch pocit\u016f, \u017ee tento p\u0159\u00edpad byl pro m\u011b slo\u017eit\u00fd, bylo t\u011b\u017ek\u00e9 ho rozebrat... a naopak, n\u011bkter\u00e9 p\u0159\u00edpady byly lehk\u00e9 a bylo snadn\u00e9 je rozebrat. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/65b0c98bf2284610e955ced5eb998f39.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>A prvn\u00ed p\u0159\u00edpad je nejjednodu\u0161\u0161\u00ed. To je ten p\u0159\u00edpad, kdy jsme vzali cluster datab\u00e1z\u00ed a na tom sam\u00e9m clusteru jsme nasadili na\u0161e DCS \u00falo\u017ei\u0161t\u011b. To je nej\u010dast\u011bj\u0161\u00ed chyba. Je to chyba v architektu\u0159e, tedy kombinov\u00e1n\u00ed r\u016fzn\u00fdch komponent\u016f na jednom m\u00edst\u011b. <\/p>\n<p><\/p>\n<p>Tak\u017ee, do\u0161lo k failoveru, jdeme zjistit, co se stalo. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/244b69d623fe2205c2d9ed2aa4620067.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>A n\u00e1s zaj\u00edm\u00e1, kdy do\u0161lo k failoveru. Tedy n\u00e1s zaj\u00edm\u00e1 ten okam\u017eik, kdy do\u0161lo k zm\u011bn\u011b stavu clusteru. <\/p>\n<p><\/p>\n<p>Ale failover nen\u00ed v\u017edy okam\u017eit\u00fd, tedy nezab\u00edr\u00e1 ur\u010ditou \u010dasovou jednotku, m\u016f\u017ee se prot\u00e1hnout. M\u016f\u017ee trvat del\u0161\u00ed dobu.<\/p>\n<p><\/p>\n<p>Preto m\u00e1 \u010das za\u010datia a \u010das ukon\u010denia, t. j. ide o dlhodob\u00fa udalos\u0165. A v\u0161etky udalosti del\u00edme na tri intervaly: m\u00e1me \u010das pred failoverom, po\u010das failoveru a po failoveri. T. j. zoh\u013ead\u0148ujeme v\u0161etky udalosti v tejto \u010dasovej osi. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/a0ed61afe55d1a157147e0d80b17e5d1.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>A prv\u00e9, \u010do sprav\u00edme, ke\u010f nastane failover, je h\u013eadanie pr\u00ed\u010diny, \u010do sa stalo, \u010do sp\u00f4sobilo, \u017ee do\u0161lo k failoveru. <\/p>\n<p><\/p>\n<p>Ak sa pozrieme na logy, tak to bud\u00fa klasick\u00e9 logy Patroni. V nich n\u00e1m oznamuje, \u017ee server sa stal hlavn\u00fdm, a \u00faloha hlavn\u00e9ho pre\u0161la na tento uzol. Tu je to zv\u00fdraznen\u00e9. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/9998c69f7d0d4358b277f4e0e7b4350f.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>\u010ealej mus\u00edme pochopi\u0165, pre\u010do do\u0161lo k failoveru, t. j. ak\u00e9 sa udiali udalosti, ktor\u00e9 sp\u00f4sobili presun \u00falohy hlavn\u00e9ho z jedn\u00e9ho uzla na druh\u00fd. A v tomto pr\u00edpade je to jednoduch\u00e9. M\u00e1me chybu pri komunik\u00e1cii so syst\u00e9mom \u00falo\u017eiska. Hlavn\u00fd server zistil, \u017ee nem\u00f4\u017ee pracova\u0165 s DCS, t. j. vznikol nejak\u00fd probl\u00e9m s interakciou. A hovor\u00ed, \u017ee u\u017e nem\u00f4\u017ee by\u0165 hlavn\u00fdm a vzd\u00e1va sa svojich pr\u00e1vomoc\u00ed. T\u00e1to veta \u201edemoted self\u201c to presne vyjadruje. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/4a8eca27689546b4b0fc3c666a3c37c4.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Ak sa pozrieme na udalosti, ktor\u00e9 predch\u00e1dzali failoveru, m\u00f4\u017eeme tam vidie\u0165 tie ist\u00e9 pr\u00ed\u010diny, ktor\u00e9 boli probl\u00e9mom pre pokra\u010dovanie \u010dinnosti hlavn\u00e9ho servera. <\/p>\n<p><\/p>\n<p>Ak sa pozrieme na logy Patroni, uvid\u00edme, \u017ee m\u00e1me mno\u017estvo r\u00f4znych ch\u00fdb, \u010dasov\u00fdch limitov, t. j. agent Patroni nem\u00f4\u017ee pracova\u0165 s DCS. V tomto pr\u00edpade to je Consul agent, s ktor\u00fdm sa komunikuje cez port 8500. <\/p>\n<p><\/p>\n<p>A probl\u00e9m predstavuje to, \u017ee Patroni a datab\u00e1za s\u00fa spusten\u00e9 na tom istom hostite\u013eovi. A na tomto uzle boli spusten\u00e9 servery Consul. Vytvoren\u00edm z\u00e1\u0165a\u017ee na serveri sme sp\u00f4sobili probl\u00e9my aj pre <a class=\"wpil_keyword_link\" href=\"https:\/\/prohoster.info\/sk\/server\/\"   title=\"serverov\" data-wpil-keyword-link=\"linked\"  data-wpil-monitor-id=\"1515\">serverov<\/a> Consul. Neboli schopn\u00ed norm\u00e1lne komunikova\u0165. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/0d8dba6641809560e6b32825d481e30b.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Po ur\u010ditom \u010dase, ke\u010f z\u00e1\u0165a\u017e klesla, n\u00e1\u0161 Patroni mohol znovu komunikova\u0165 s agentmi. Norm\u00e1lna \u010dinnos\u0165 sa obnovila. A ten ist\u00fd server Pgdb-2 sa op\u00e4\u0165 stal hlavn\u00fdm. T. j. do\u0161lo k mal\u00e9mu prevratu, kv\u00f4li ktor\u00e9mu uzol vzdal svoje pr\u00e1va hlavn\u00e9ho, a potom si ich znova prevzal, t. j. v\u0161etko sa vr\u00e1tilo do p\u00f4vodn\u00e9ho stavu. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/131b93e0bd83099e1b99b53742419e13.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>A to m\u00f4\u017ee by\u0165 pova\u017eovan\u00e9 za falo\u0161n\u00e9 vyvolanie, alebo sa m\u00f4\u017ee pova\u017eova\u0165 za to, \u017ee Patroni v\u0161etko spravil spr\u00e1vne. T. j. pochopil, \u017ee nedok\u00e1\u017ee udr\u017ea\u0165 stav klastre a vzdal sa svojich pr\u00e1vomoc\u00ed.<\/p>\n<p><\/p>\n<p>A probl\u00e9m\u00e1t az okozta, hogy a Consul-szerverek ugyanazon a hardveren vannak, mint az adatb\u00e1zisok. Ennek megfelel\u0151en b\u00e1rmilyen terhel\u00e9s, legyen az lemez- vagy processzorterhel\u00e9s, szint\u00e9n hat\u00e1ssal van a Consullal val\u00f3 interakci\u00f3ra.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/142c7cf839c8da078451f7ba9d5499e5.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>A rozhodli jsme se, \u017ee to nem\u00e1 b\u00fdt spole\u010dn\u011b, vy\u010dlenili jsme samostatn\u00fd klastr pro Consul. A Patroni u\u017e pracoval s odd\u011blen\u00fdm Consul, t. j. byl samostatn\u00fd klastr Postgres a samostatn\u00fd klastr Consul. Toto je z\u00e1kladn\u00ed instrukce, jak tyto v\u011bci rozd\u011blit a udr\u017eovat, aby ne\u017eily spole\u010dn\u011b. <\/p>\n<p><\/p>\n<p>Alternat\u00edv megold\u00e1sk\u00e9nt lehet j\u00e1tszani a ttl, loop_wait, retry_timeout param\u00e9terekkel, azaz pr\u00f3b\u00e1lkozni ezen param\u00e9terek n\u00f6vel\u00e9s\u00e9vel, hogy t\u00fal\u00e9lj\u00fck ezeket a r\u00f6vid ideig tart\u00f3 terhel\u00e9si cs\u00facsokat. De ez nem a legjobb megold\u00e1s, mert ez a terhel\u00e9s id\u0151ben elh\u00faz\u00f3dhat. \u00c9s egyszer\u0171en t\u00fall\u00e9pj\u00fck a param\u00e9terek hat\u00e1rait. Ez nem felt\u00e9tlen\u00fcl seg\u00edt. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/c51f35dcd88b1c792acec231c0c6c66c.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Az els\u0151 probl\u00e9ma, ahogy \u00e9rtett\u00fck, egyszer\u0171. A DCS-t az adatb\u00e1zissal egy\u00fctt helyezt\u00fck el, \u00e9s probl\u00e9m\u00e1t kaptunk. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/cc2a98d9790441577fb4080406cbc53f.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>A m\u00e1sodik probl\u00e9ma hasonl\u00edt az els\u0151re. Az a hasonl\u00f3s\u00e1ga, hogy ism\u00e9t probl\u00e9m\u00e1ink vannak a DCS rendszerrel val\u00f3 interakci\u00f3val.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/e5c01b105adca48cc620085db8dd2d2b.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Ha megn\u00e9zz\u00fck a napl\u00f3kat, l\u00e1tni fogjuk, hogy ism\u00e9t kommunik\u00e1ci\u00f3s hiba t\u00f6rt\u00e9nt. A Patroni azt mondja, hogy nem tudok kommunik\u00e1lni a DCS-sel, ez\u00e9rt az aktu\u00e1lis mester replika m\u00f3dba l\u00e9p.<\/p>\n<p><\/p>\n<p>A r\u00e9gi mester replika lesz, itt a Patroni \u00fagy m\u0171k\u00f6dik, ahogy kell. Elind\u00edtja a pg_rewind-t, hogy visszatekerje a tranzakci\u00f3s napl\u00f3t, majd csatlakozik az \u00faj mesterhez, \u00e9s m\u00e1r ut\u00f3l\u00e9ri az \u00faj mestert. Itt a Patroni igenis azt teszi, amit kell. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/29255f4790abb15b64dbf9ee0b41c7fc.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Itt meg kell tal\u00e1lnunk azt a helyet, amely megel\u0151zte a file veret, azaz azokat a hib\u00e1kat, amelyek oka volt annak, hogy a file verett t\u00f6rt\u00e9nt. Ilyen szempontb\u00f3l a napl\u00f3kkal viszonylag k\u00f6nny\u0171 dolgozni a Patroni eset\u00e9ben. Meghat\u00e1rozott id\u0151k\u00f6z\u00f6nk\u00e9nt ugyanazokat az \u00fczeneteket \u00edrja. Ha gyorsan kezdj\u00fck el visszatekerni ezeket a napl\u00f3kat, akkor a napl\u00f3k v\u00e1ltozottak, teh\u00e1t valamilyen probl\u00e9ma kezd\u0151d\u00f6tt. Gyorsan visszat\u00e9r\u00fcnk erre a pontra, \u00e9s megn\u00e9zz\u00fck, mi t\u00f6rt\u00e9nik. <\/p>\n<p><\/p>\n<p>Norm\u00e1lis helyzetben a napl\u00f3k nagyj\u00e1b\u00f3l \u00edgy n\u00e9znek ki. Ellen\u0151rizz\u00fck a z\u00e1rol\u00e1s tulajdonos\u00e1t. Ha a tulajdonos p\u00e9ld\u00e1ul megv\u00e1ltozott, akkor bek\u00f6vetkezhetnek olyan esem\u00e9nyek, amelyekre a Patroni reag\u00e1lnia kell. De ebben az esetben minden rendben van. Keress\u00fck azt a helyet, amikor a hib\u00e1k megkezd\u0151dtek. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/f48ebd22a405f5b2900621451c68f543.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Ak posunieme a\u017e na miesto, kde sa za\u010dali objavova\u0165 chyby, vid\u00edme, \u017ee sa n\u00e1m aktivoval auto-failover. A ke\u010f\u017ee na\u0161e chyby mali s\u00favislos\u0165 s interakciou s DCS a v na\u0161om pr\u00edpade sme pou\u017e\u00edvali Consul, pozrieme sa aj do logov Consulu, \u010do sa tam dialo. <\/p>\n<p><\/p>\n<p>Pribli\u017ene porovn\u00e1vaj\u00fac \u010das failoveru a \u010das v logoch Consulu, vid\u00edme, \u017ee na\u0161i susedia v klastri Consul za\u010dali pochybova\u0165 o existencii ostatn\u00fdch \u00fa\u010dastn\u00edkov klastra Consul.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/63b02ac5b4f5a34a2822f1eee2b1f2f8.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>A ak sa pozrieme aj na logy ostatn\u00fdch agentov Consul, m\u00f4\u017eete vidie\u0165, \u017ee tam sa deje nejak\u00fd sie\u0165ov\u00fd kolaps. A v\u0161etci \u00fa\u010dastn\u00edci klastra Consul pochybuj\u00fa o existencii jeden druh\u00e9ho. A to viedlo k failoveru. <\/p>\n<p><\/p>\n<p>Ak sa pozrieme na to, \u010do sa dialo pred t\u00fdmito chybami, m\u00f4\u017eeme vidie\u0165, \u017ee existuj\u00fa r\u00f4zne chyby, napr\u00edklad deadline, RPC failed, t. j. jasne nejak\u00fd probl\u00e9m s interakciou medzi \u00fa\u010dastn\u00edkmi klastra Consul. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/b15c912f2afb49d9d85e7bb9361ce449.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Najjednoduch\u0161ia odpove\u010f je opravi\u0165 sie\u0165. No ja, stojac na trib\u00fane, m\u00f4\u017eem o tom \u013eahko vyhl\u00e1si\u0165. Ale okolnosti s\u00fa tak\u00e9, \u017ee nie v\u017edy si z\u00e1kazn\u00edk m\u00f4\u017ee dovoli\u0165 opravi\u0165 sie\u0165. M\u00f4\u017ee \u017ei\u0165 v DC a nemus\u00ed ma\u0165 mo\u017enosti opravova\u0165 sie\u0165, ovplyv\u0148ova\u0165 zariadenia. A preto s\u00fa potrebn\u00e9 nejak\u00e9 in\u00e9 mo\u017enosti. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/1dbdb66233acc3bc321a30a1c5fe14f9.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Mo\u017enosti s\u00fa:<\/p>\n<p><\/p>\n<ul>\n<li>Najjednoduch\u0161ia mo\u017enos\u0165, ktor\u00e1 je nap\u00edsan\u00e1, mysl\u00edm, \u017ee dokonca aj v dokument\u00e1cii, je vypn\u00fa\u0165 kontroly Consul, t. j. jednoducho odovzda\u0165 pr\u00e1zdne pole. A hovor\u00edme agentovi Consulu, aby nepou\u017e\u00edval \u017eiadne kontroly. V\u010faka t\u00fdmto kontrol\u00e1m m\u00f4\u017eeme ignorova\u0165 tieto sie\u0165ov\u00e9 b\u00farky a nepo\u017eiada\u0165 o failover. <\/li>\n<li>In\u00e1 mo\u017enos\u0165 je prekontrolova\u0165 raft_multiplier. To je parameter samotn\u00e9ho servera Consul. Predvolene je nastaven\u00fd na hodnotu 5. T\u00e1to hodnota je odpor\u00fa\u010dan\u00e1 v dokument\u00e1cii pre staging prostredia. V podstate to ovplyv\u0148uje frekvenciu v\u00fdmeny spr\u00e1v medzi \u00fa\u010dastn\u00edkmi siete Consul. Tento parameter ovplyv\u0148uje r\u00fdchlos\u0165 slu\u017eobnej komunik\u00e1cie medzi \u00fa\u010dastn\u00edkmi klastra Consul. A pre produkciu sa u\u017e odpor\u00fa\u010da zn\u00ed\u017ei\u0165 ho, aby uzly \u010dastej\u0161ie vymenili spr\u00e1vy. <\/li>\n<li>In\u00fd pr\u00edstup, ktor\u00fd sme za\u010dali pou\u017e\u00edva\u0165, je zv\u00fd\u0161enie priority procesov Consul v porovnan\u00ed s ostatn\u00fdmi procesmi pre pl\u00e1nova\u010d opera\u010dn\u00e9ho syst\u00e9mu. Existuje parameter \"nice\", ktor\u00fd presne ur\u010duje prioritu procesov, ktor\u00fa pl\u00e1nova\u010d OS zoh\u013ead\u0148uje pri pl\u00e1novan\u00ed. Zn\u00ed\u017eili sme hodnotu nice pre agentov Consul, teda sme zv\u00fd\u0161ili prioritu, aby opera\u010dn\u00fd syst\u00e9m poskytoval procesom Consul viac \u010dasu na pr\u00e1cu a vykonanie svojho k\u00f3du. V na\u0161om pr\u00edpade to vyrie\u0161ilo n\u00e1\u0161 probl\u00e9m. <\/li>\n<li>\u010eal\u0161ia mo\u017enos\u0165 je, \u017ee nepou\u017e\u00edvame Consul. M\u00e1m zn\u00e1meho, ktor\u00fd je ve\u013ek\u00fd zast\u00e1nca Etcd. A pravidelne o tom diskutujeme, \u010do je lep\u0161ie, Etcd alebo Consul. \u010co sa t\u00fdka toho, \u010do je lep\u0161ie, obvykle sa zhodneme na tom, \u017ee Consul m\u00e1 agenta, ktor\u00fd mus\u00ed by\u0165 spusten\u00fd na ka\u017edom uzle s datab\u00e1zou. T.j. interakcia Patroni s klastrom Consul prebieha cez tohto agenta. A tento agent sa st\u00e1va \u00fazkym miestom. Ak sa nie\u010do stane s agentom, Patroni u\u017e nem\u00f4\u017ee pracova\u0165 s klastrom Consul. A to je probl\u00e9m. Na strane Etcd nie je \u017eiadny agent. Patroni m\u00f4\u017ee priamo pracova\u0165 so zoznamom serverov Etcd a komunikova\u0165 s nimi. V tomto oh\u013eade, ak vo va\u0161ej spolo\u010dnosti pou\u017e\u00edvate Etcd, bude asi Etcd lep\u0161ou vo\u013ebou ako Consul. Ale sme u na\u0161ich z\u00e1kazn\u00edkov v\u017edy obmedzen\u00ed t\u00fdm, \u010do si zvolil a pou\u017e\u00edva svoj klient. A v\u00e4\u010d\u0161inou m\u00e1me Consul u v\u0161etk\u00fdch klientov. <\/li>\n<li>A posledn\u00fd bod \u2013 je prehodnoti\u0165 hodnoty parametrov. M\u00f4\u017eeme zv\u00fd\u0161i\u0165 tieto parametre, d\u00fafaj\u00fac, \u017ee na\u0161e kr\u00e1tkodob\u00e9 sie\u0165ov\u00e9 probl\u00e9my bud\u00fa kr\u00e1tke a nepresiahnu interval t\u00fdchto parametrov. Takto m\u00f4\u017eeme zn\u00ed\u017ei\u0165 agresivitu Patroni pri vykon\u00e1van\u00ed auto-failover, ak sa vyskytuj\u00fa nejak\u00e9 sie\u0165ov\u00e9 probl\u00e9my.<\/li>\n<\/ul>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/f0043050da20f6368dabf66ec9a4eade.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Mysl\u00edm, \u017ee mnoh\u00ed, ktor\u00ed pou\u017e\u00edvaj\u00fa Patroni, s\u00fa obozn\u00e1men\u00ed s touto pr\u00edkazom. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/9f78f59dab166e47ac78436802156d04.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Tento pr\u00edkaz ukazuje aktu\u00e1lny stav klastru. A na prv\u00fd poh\u013ead sa t\u00e1to situ\u00e1cia m\u00f4\u017ee javi\u0165 ako norm\u00e1lna. M\u00e1me majstra, m\u00e1me replik\u00e1ciu, \u017eiadna latencia replik\u00e1cie nie je. Ale t\u00e1to situ\u00e1cia je norm\u00e1lna len dovtedy, k\u00fdm nevieme, \u017ee v tomto klastre by mali by\u0165 tri uzly, nie dva. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/a39c891f8620ba210b6bce0727e0fa8b.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Vznikol autofailover. Po tomto autofailoveri n\u00e1m zmizla replika. Mus\u00edme zisti\u0165 pre\u010do zmizla a obnovi\u0165 ju sp\u00e4\u0165. Op\u00e4\u0165 sa ideme pozrie\u0165 do logov a zisti\u0165, pre\u010do nastal autofailover.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/e6e67d46cb20c8d7e58c7505157f68e1.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>V tomto pr\u00edpade sa druh\u00e1 replika stala majstrom. Tu je v\u0161etko v poriadku. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/b86ccea68b0c6013a23bbd2804e48320.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Mus\u00edme sa pozrie\u0165 na replik\u00e1ciu, ktor\u00e1 vypadla a nie je v klusteri. Otvor\u00edme logy Patroni a skontrolujeme, \u017ee pri pokuse o pripojenie ku klasteru nastal probl\u00e9m po\u010das f\u00e1zy pg_rewind. Na pripojenie ku klasteru je potrebn\u00e9 resetova\u0165 transak\u010dn\u00fd log, po\u017eiada\u0165 spr\u00e1vne transak\u010dn\u00e9 logy od majstra a pod\u013ea nich dobehn\u00fa\u0165 majstra. <\/p>\n<p><\/p>\n<p>V tomto pr\u00edpade nem\u00e1me transak\u010dn\u00fd log a replika sa nem\u00f4\u017ee spusti\u0165. Preto zastav\u00edme PostgreSQL s chybou. A preto nie je v klusteri. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/dadbd44b766674b719d85781ef7f0caa.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Mus\u00edme zisti\u0165, pre\u010do nie je v klusteri a pre\u010do neexistuj\u00fa logy. Ideme na nov\u00e9ho majstra a pozrieme sa, \u010do m\u00e1 v logoch. Ukazuje sa, \u017ee po\u010das pg_rewind do\u0161lo k checkpointu. A \u010das\u0165 star\u00fdch transak\u010dn\u00fdch logov bola jednoducho premenovan\u00e1. Ke\u010f sa star\u00fd majster pok\u00fasil pripoji\u0165 k nov\u00e9mu majstrovi a po\u017eiadal o tieto logy, u\u017e boli premenovan\u00e9, jednoducho ich nebolo.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/f2d5c37324f9436b2bdad1290612d86f.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Porovn\u00e1val som \u010dasov\u00e9 pe\u010diatky, kedy sa tieto udalosti stali. A tam je rozdiel doslova 150 milisek\u00fand, teda v 369 milisekund\u00e1ch sa dokon\u010dil checkpoint a WAL segmenty boli premenovan\u00e9. A len v 517 milisekund\u00e1ch po 150 milisekund\u00e1ch sa za\u010dal rewind na starej replice. Teda doslova 150 milisek\u00fand n\u00e1m sta\u010dilo, aby sa replika nemohla pripoji\u0165 a rozbehn\u00fa\u0165. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/99dd80deb8466bfc3aff568456e07102.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Ak\u00e9 s\u00fa mo\u017enosti?<\/p>\n<p><\/p>\n<p>P\u00f4vodne sme pou\u017e\u00edvali replika\u010dn\u00e9 sloty. Zdalo sa n\u00e1m, \u017ee je to dobr\u00e9. Hoci v prvom \u0161t\u00e1diu prev\u00e1dzky sme sloty vyp\u00ednali. Mysleli sme si, \u017ee ak sloty nahromadia ve\u013ea WAL segmentov, m\u00f4\u017eeme zr\u00fati\u0165 majstra. Ten spadne. Niekedy sme sa mu\u010dili bez slotov. A pochopili sme, \u017ee sloty potrebujeme, a vr\u00e1tili sme ich sp\u00e4\u0165. <\/p>\n<p><\/p>\n<p>Ale je tu probl\u00e9m, \u017ee ke\u010f sa hlavn\u00fd server prepnul na replik\u00e1ciu, zmazal sloty a spolu so slotmi aj WAL segmenty. Aby sme predi\u0161li tomuto probl\u00e9mu, rozhodli sme sa zv\u00fd\u0161i\u0165 parametre wal_keep_segments. Predvolene je to 8 segmentov. Zv\u00fd\u0161ili sme to na 1 000 a skontrolovali, ko\u013eko m\u00e1me vo\u013en\u00e9ho miesta. A darovali sme 16 gigabajtov na wal_keep_segments. T.j. pri prep\u00ednan\u00ed m\u00e1me v\u017edy na v\u0161etk\u00fdch uzloch rezervu 16 gigabajtov transak\u010dn\u00fdch z\u00e1znamov.<\/p>\n<p><\/p>\n<p>A navy\u0161e \u2013 to je e\u0161te relevantn\u00e9 pre dlhodob\u00e9 \u00fadr\u017ebov\u00e9 \u00falohy. Predpokladajme, \u017ee potrebujeme aktualizova\u0165 jednu z replik. A chceme ju vypn\u00fa\u0165. Mus\u00edme aktualizova\u0165 softv\u00e9r, mo\u017eno opera\u010dn\u00fd syst\u00e9m, e\u0161te nie\u010do. A ke\u010f vypneme replik\u00e1ciu, aj pre t\u00fato replik\u00e1ciu sa zma\u017ee slot. A ak pou\u017e\u00edvame mal\u00fd wal_keep_segments, tak pri dlhodobom vypnut\u00ed repliky sa transak\u010dn\u00e9 z\u00e1znamy prehr\u00e1vaj\u00fa. Zapneme replik\u00e1ciu, po\u017eiada o tie transak\u010dn\u00e9 z\u00e1znamy, kde zastavila, ale na hlavnom serveri ich nemus\u00ed by\u0165. A replik\u00e1cia sa tie\u017e nebude m\u00f4c\u0165 pripoji\u0165. Preto dr\u017e\u00edme ve\u013ek\u00fa rezervu z\u00e1znamov.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/dad55c828128ce92f649a99cc53ce54b.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/63e70f0b098567f3a6d4fd66a0c293bb.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>M\u00e1me produk\u010dn\u00fa datab\u00e1zu. Tam u\u017e funguj\u00fa projekty. <\/p>\n<p><\/p>\n<p>Do\u0161lo k s\u00faborov\u00e9mu zlyhaniu. Pri\u0161li sme a pozreli sa \u2013 v\u0161etko je v poriadku, repliky s\u00fa na svojich miestach, nie je \u017eiadne oneskorenie replik\u00e1cie. Chyby v z\u00e1znamoch tie\u017e nie s\u00fa, v\u0161etko je v poriadku. <\/p>\n<p><\/p>\n<p>Produktov\u00fd t\u00edm hovor\u00ed, \u017ee by mali by\u0165 nejak\u00e9 \u00fadaje, ale vid\u00edme ich v jednom zdroji, av\u0161ak v datab\u00e1ze ich nevid\u00edme. A mus\u00edme zisti\u0165, \u010do sa s nimi stalo. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/568ca9aa31680d5e0c9d4ed9693b14e6.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Je jasn\u00e9, \u017ee pg_rewind ich prep\u00edsal. Okam\u017eite sme to pochopili, ale i\u0161li sme sa pozrie\u0165, \u010do sa stalo. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/a8fee389201d96e81761cd276c5265c3.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>V logoch v\u017edy m\u00f4\u017eeme n\u00e1js\u0165, kedy do\u0161lo k s\u00faborov\u00e9mu zlyhaniu, kto sa stal hlavn\u00fdm serverom a m\u00f4\u017eeme ur\u010di\u0165, kto bol star\u00fdm hlavn\u00fdm serverom a kedy chcel by\u0165 replikou, t.j. potrebujeme tieto logy, aby sme zistili objem transak\u010dn\u00fdch z\u00e1znamov, ktor\u00e9 boli straten\u00e9.<\/p>\n<p><\/p>\n<p>N\u00e1\u0161 star\u00fd master se restartoval. A v automatick\u00e9m spu\u0161t\u011bn\u00ed byl zaregistrov\u00e1n Patroni. Patroni se spustil. N\u00e1sledn\u011b spustil Postgres. P\u0159esn\u011bji, p\u0159ed spu\u0161t\u011bn\u00edm Postgres a p\u0159ed t\u00edm, ne\u017e ho ud\u011blal replikou, Patroni spustil proces pg_rewind. T\u00edm p\u00e1dem smazal \u010d\u00e1st transak\u010dn\u00edch z\u00e1znam\u016f, st\u00e1hl nov\u00e9 a p\u0159ipojil se. Zde Patroni fungoval skv\u011ble, t. j. jak m\u00e1. N\u00e1\u0161 klastr se obnovil. M\u011bli jsme 3 uzly, po failoveru 3 uzly \u2013 v\u0161echno je skv\u011bl\u00e9. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/b747a24f53e20ed098c2dc0afc79bead.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Stratili sme \u010das\u0165 d\u00e1t. Mus\u00edme zisti\u0165, ko\u013eko sme stratili. H\u013ead\u00e1me presne ten okamih, ke\u010f sa uskuto\u010dnil rewind. M\u00f4\u017eeme to n\u00e1js\u0165 pod\u013ea tak\u00fdchto z\u00e1znamov v denn\u00edku. Spustil sa rewind, nie\u010do tam spravil a skon\u010dil.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/5b7bacffb52e3ccd529a3a734c441295.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Mus\u00edme n\u00e1js\u0165 t\u00fa poz\u00edciu v denn\u00edku transakci\u00ed, na ktorej sa zastavil star\u00fd master. V tomto pr\u00edpade \u2013 to je t\u00e1to zna\u010dka. A potrebujeme druh\u00fa zna\u010dku, t.j. vzdialenos\u0165, o ktor\u00fa sa star\u00fd master l\u00ed\u0161i od nov\u00e9ho. <\/p>\n<p><\/p>\n<p>Berieme be\u017en\u00fd pg_wal_lsn_diff a porovn\u00e1vame tieto dve zna\u010dky. A v tomto pr\u00edpade dost\u00e1vame 17 megabajtov. \u010ci je to ve\u013ea alebo m\u00e1lo, si rozhodne ka\u017ed\u00fd s\u00e1m. Preto\u017ee pre niekoho je 17 megabajtov m\u00e1lo, pre niekoho je to ve\u013ea a neprijate\u013en\u00e9. Tu u\u017e ka\u017ed\u00fd ur\u010duje individu\u00e1lne pod\u013ea potrieb biznisu. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/55335fce85961e159459c38300a0ad12.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Ale \u010do sme zistili pre seba? <\/p>\n<p><\/p>\n<p>Po prv\u00e9, mus\u00edme si rozhodn\u00fa\u0165 \u2013 potrebujeme v\u017edy automatick\u00e9 sp\u00fa\u0161\u0165anie Patroni po re\u0161tarte syst\u00e9mu? \u010castej\u0161ie je to tak, \u017ee mus\u00edme \u00eds\u0165 na star\u00fd master, pozrie\u0165 sa, ako \u010faleko sa dostal. Mo\u017eno prein\u0161pektova\u0165 segmenty denn\u00edka transakci\u00ed, pozrie\u0165 sa, \u010do tam je. A zisti\u0165 \u2013 m\u00f4\u017eeme tieto d\u00e1ta strati\u0165 alebo potrebujeme spusti\u0165 star\u00fd master v standalone re\u017eime, aby sme z\u00edskali tieto d\u00e1ta. <\/p>\n<p><\/p>\n<p>A a\u017e potom by sme mali prija\u0165 rozhodnutia, \u017ee m\u00f4\u017eeme tieto d\u00e1ta zahodi\u0165 alebo ich m\u00f4\u017eeme obnovi\u0165, pripoji\u0165 tento uzol ako repliku do n\u00e1\u0161ho klastrov\u00e9ho syst\u00e9mu.<\/p>\n<p><\/p>\n<p>Okrem toho existuje parameter \"maximum_lag_on_failover\". Ako si pam\u00e4t\u00e1m, tento parameter m\u00e1 predvolen\u00e9 hodnoty 1 megabajt. <\/p>\n<p><\/p>\n<p>Ako to funguje? Ak na\u0161a replika zaost\u00e1va o 1 megabajt d\u00e1t pod\u013ea replik\u00e1cie, t\u00e1to replika sa nez\u00fa\u010dast\u0148uje volieb. A ak n\u00e1hodou d\u00f4jde k failoveru, Patroni sleduje, ktor\u00e9 repliky zaost\u00e1vaj\u00fa. Ak zaost\u00e1vaj\u00fa o ve\u013ek\u00e9 mno\u017estvo denn\u00edkov transakci\u00ed, nem\u00f4\u017eu sa sta\u0165 majstrom. Je to ve\u013emi dobr\u00e1 ochrann\u00e1 funkcia, ktor\u00e1 zabra\u0148uje strate ve\u013ek\u00e9ho mno\u017estva d\u00e1t. <\/p>\n<p><\/p>\n<p>Ale probl\u00e9m je v tom, \u017ee oneskorenie replik\u00e1cie v klastre Patroni a DCS sa aktualizuje s ur\u010dit\u00fdm intervalom. Pod\u013ea m\u00f4jho n\u00e1zoru je predvolen\u00e9 nastavenie ttl 30 sek\u00fand.<\/p>\n<p><\/p>\n<p>M\u00f4\u017ee sa sta\u0165, \u017ee latencia replik\u00e1cie pre repliky v DCS je jedna, ale v skuto\u010dnosti m\u00f4\u017ee by\u0165 \u00faplne in\u00e1 latencia alebo m\u00f4\u017ee v\u00f4bec neexistova\u0165, t. j. t\u00e1to vec nie je realtime. A nie v\u017edy odr\u00e1\u017ea skuto\u010dn\u00fd obraz. Tak\u017ee na nej nie je rozumn\u00e9 budova\u0165 zlo\u017eit\u00fa logiku. <\/p>\n<p><\/p>\n<p>A riziko str\u00e1t v\u017edy zost\u00e1va. A v najhor\u0161om pr\u00edpade jedna rov nice, a v priemernom pr\u00edpade in\u00e1 rov nice. T. j. ke\u010f pl\u00e1nujeme zavedenie Patroni a posudzujeme, ko\u013eko d\u00e1t m\u00f4\u017eeme strati\u0165, mali by sme sa spolieha\u0165 na tieto vzorce a ma\u0165 pribli\u017en\u00fa predstavu o tom, ko\u013eko d\u00e1t m\u00f4\u017eeme strati\u0165. <\/p>\n<p><\/p>\n<p>A dobr\u00e1 spr\u00e1va je. Ke\u010f star\u00fd master pre\u0161iel pred, m\u00f4\u017ee \u00eds\u0165 vpred v\u010faka nejak\u00fdm pozadiu procesom. T. j. nejak\u00fd bol autovakuum, zap\u00edsal d\u00e1ta, ulo\u017eil ich do \u017eurn\u00e1lu transakci\u00ed. A tieto d\u00e1ta m\u00f4\u017eeme pokojne ignorova\u0165 a strati\u0165. V tom nie je \u017eiadny probl\u00e9m. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/a86e8971ab4ef41b89af9cf0bdf519ba.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Takto vyzeraj\u00fa logy, ak je nastaven\u00fd maximum_lag_on_failover a do\u0161lo k failoveru, a treba vybra\u0165 nov\u00e9ho majstra. Replika sa hodnot\u00ed ako neschopn\u00e1 z\u00fa\u010dastni\u0165 sa volieb. A odmieta \u00fa\u010dastni\u0165 sa pretekov o l\u00eddra. A \u010dak\u00e1, k\u00fdm bude vybran\u00fd nov\u00fd majster, aby sa potom k nemu pripojila. To je dodato\u010dn\u00e9 opatrenie proti strate \u00fadajov.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/f760788e5951facdd08b2069037830fa.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/c32f26e7526e1873f3bfd7d3693fcc72.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Na\u0161a produktov\u00e1 t\u00edm nap\u00edsal, \u017ee ich produkt m\u00e1 probl\u00e9my pri pr\u00e1ci s Postgres. Z\u00e1rove\u0148 nie je mo\u017en\u00e9 sa pripoji\u0165 na samotn\u00e9ho majstra, preto\u017ee nie je dostupn\u00fd cez SSH. A autofailover sa tie\u017e nedeje. <\/p>\n<p><\/p>\n<p>Tento host bol n\u00fatene re\u0161tartovan\u00fd. Kv\u00f4li re\u0161tartu do\u0161lo k autofailoveru, aj ke\u010f bolo mo\u017en\u00e9 vykona\u0165 aj ru\u010dn\u00fd autofailover, ako teraz ch\u00e1pem. A po re\u0161tarte u\u017e ideme skontrolova\u0165, \u010do sme mali s aktu\u00e1lnym majstrom. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/95efb82e0647a396a21683cd53a69547.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Z\u00e1rove\u0148 sme vopred vedeli, \u017ee m\u00e1me probl\u00e9my s diskami, t. j. sme u\u017e pod\u013ea monitorovania vedeli, kde h\u013eada\u0165 a \u010do sk\u00fama\u0165. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/367aaab607d833d96a573ddbbbced502.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Vo\u0161li sme do postgresov\u00e9ho logu, za\u010dali sme sledova\u0165, \u010do sa tam deje. Videli sme komity, ktor\u00e9 trvaj\u00fa jednu-dve-tri sekundy, \u010do je \u00faplne nenorm\u00e1lne. Videli sme, \u017ee sa autovakuum sp\u00fa\u0161\u0165a ve\u013emi dlho a zvl\u00e1\u0161tne. A videli sme do\u010dasn\u00e9 s\u00fabory na disku. T. j. to v\u0161etko s\u00fa znaky probl\u00e9mov s diskami. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/7e0cfce7a81901ef9ef736c3fefc8dce.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Prejavili sme do syst\u00e9mov\u00e9ho dmesg (do logu jadrov\u00fdch spr\u00e1v). A videli sme, \u017ee m\u00e1me probl\u00e9m s jedn\u00fdm z diskov. Diskov\u00e1 subsyst\u00e9mov\u00e1 predstavovala software Raid. Pozreli sme sa na \/proc\/mdstat a zistili sme, \u017ee n\u00e1m ch\u00fdba jeden disk. To znamen\u00e1, \u017ee tu je Raid z 8 diskov, jedn\u00e9ho n\u00e1m ch\u00fdba. Ak sa pozrieme pozorne na sn\u00edmku, vo v\u00fdstupe m\u00f4\u017eeme vidie\u0165, \u017ee n\u00e1m tam ch\u00fdba sde. Pod\u013ea v\u0161etk\u00e9ho n\u00e1m vypadol disk. To spustilo diskov\u00e9 probl\u00e9my a aplik\u00e1cie tie\u017e za\u017eili probl\u00e9my pri pr\u00e1ci s klastrom Postgres.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/4b09b3aa761cb504fa79173ef280faac.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>A v tomto pr\u00edpade by n\u00e1m Patroni nijako nepomohol, preto\u017ee Patroni nem\u00e1 \u00falohu monitorova\u0165 stav servera, stav disku. Mus\u00edme tak\u00e9to situ\u00e1cie sledova\u0165 extern\u00fdm monitorovan\u00edm. Do extern\u00e9ho monitorovania sme prakticky pridali monitorovanie diskov. <\/p>\n<p><\/p>\n<p>A mala som tak\u00fa my\u0161lienku \u2013 mohol by n\u00e1m pom\u00f4c\u0165 fencing, alebo soft watchdog? Mysleli sme si, \u017ee to n\u00e1m asi nepom\u00f4\u017ee v tomto pr\u00edpade, preto\u017ee po\u010das probl\u00e9mov Patroni na\u010falej interagoval s klastrom DCS a nevidel \u017eiadny probl\u00e9m. To znamen\u00e1, \u017ee z poh\u013eadu DCS a Patroni s klastrom bolo v\u0161etko v poriadku, hoci v skuto\u010dnosti boli probl\u00e9my s diskom, boli probl\u00e9my s dostupnos\u0165ou datab\u00e1zy. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/15fdac78535355d4eb889efcc27efc46.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Pod\u013ea m\u00f4jho n\u00e1zoru ide o jeden z najstrrangej\u0161\u00edch probl\u00e9mov, ktor\u00e9 som sk\u00famala ve\u013emi dlho, pre\u010d\u00edtala som si ve\u013emi ve\u013ea logov, prekopala som a nazvala som ho klaster-simulantom. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/b5259774931c290eda6673deb14b6c48.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Probl\u00e9m spo\u010d\u00edval v tom, \u017ee star\u00fd master nemohol by\u0165 norm\u00e1lnou replikou, to znamen\u00e1, \u017ee Patroni ho sp\u00fa\u0161\u0165al, Patroni ukazoval, \u017ee tento uzol je pr\u00edtomn\u00fd ako replika, ale z\u00e1rove\u0148 nebol norm\u00e1lnou replikou. Teraz uvid\u00edte, pre\u010do. Toto som si uchovala z rozboru toho probl\u00e9mu. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/c35a5d102a46764bbf96e5a75ee88171.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>A od \u010doho to v\u0161etko za\u010dalo? Za\u010dalo to, ako v predch\u00e1dzaj\u00facom probl\u00e9me, s diskov\u00fdmi spoma\u013eovaniami. Mali sme komitovanie po sekunde, dve. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/435c1b350a89f288e0e10ca9d5a2ec6e.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Boli preru\u0161enia pripojen\u00ed, to znamen\u00e1, \u017ee klienti sa odpojovali. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/920cb1f3f3b40578e15e6ff711e85c50.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Boli z\u00e1mky r\u00f4znej z\u00e1va\u017enosti. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/285d6292d89b928a85e95602ee57d38a.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>A pod\u013ea toho diskov\u00e1 subsyst\u00e9m nebola ve\u013emi pohotov\u00e1. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/ebe71d49360e7f604ecd6a3c8276cb73.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>A najz\u00e1hadnej\u0161ie pre m\u0148a \u2013 to bola ihne\u010f doru\u010den\u00e1 po\u017eiadavka na vypnutie. V Postgres existuj\u00fa tri re\u017eimy vyp\u00ednania:<\/p>\n<p><\/p>\n<ul>\n<li>To je graceful, ke\u010f \u010dak\u00e1me, k\u00fdm sa v\u0161etci klienti odpoja sami. <\/li>\n<li>Existuje fast, ke\u010f nut\u00edme klientov, aby sa odpojili, preto\u017ee ideme na vypnutie. <\/li>\n<li>A immedi\u00e1tna. V tomto pr\u00edpade immedi\u00e1tna ani neinformuje klientov, \u017ee sa maj\u00fa odpoji\u0165, jednoducho sa vypne bez varovania. A v\u0161etk\u00fdm klientom u\u017e opera\u010dn\u00fd syst\u00e9m odosiela spr\u00e1vu RST (TCP spr\u00e1va, \u017ee pripojenie bolo preru\u0161en\u00e9 a klient u\u017e nem\u00e1 \u010do robi\u0165). <\/li>\n<\/ul>\n<p><\/p>\n<p>Kto poslal tento sign\u00e1l? Pozadia procesy Postgres si navz\u00e1jom tak\u00e9to sign\u00e1ly neposielaj\u00fa, t. j. je to kill-9. Navz\u00e1jom si tak\u00e9to neposielaj\u00fa, len na tak\u00e9 reaguj\u00fa, t. j. ide o n\u00fadzov\u00fd re\u0161tart Postgres. Kto to poslal, neviem. <\/p>\n<p><\/p>\n<p>Pozrel som sa na pr\u00edkaz \u201elast\u201c a videl som jednu osobu, ktor\u00e1 sa tie\u017e prihl\u00e1sila na tento server spolu s nami, ale hanbil som sa polo\u017ei\u0165 ot\u00e1zku. Mo\u017eno, to bolo kill -9. V logoch by som videl kill -9, preto\u017ee Postgres zapisuje, \u017ee prijal kill -9, ale toto som v logoch nevidel. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/fe8e844649cea3384ebfde86e36eba44.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Pri hlb\u0161om sk\u00faman\u00ed som si v\u0161imol, \u017ee Patroni dlho nep\u00edsal do logu \u2013 54 sek\u00fand. A ak porovn\u00e1m dva \u010dasov\u00e9 odtla\u010dky, tu pribli\u017ene 54 sek\u00fand neboli \u017eiadne spr\u00e1vy. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/ffb1c4db3d7de591d77d56ed2fcc4f2a.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>A za tento \u010das do\u0161lo k autofailoveru. Patroni op\u00e4\u0165 fungoval skvele. Na\u0161a star\u00e1 u\u010dite\u013eka bola nepr\u00edstupn\u00e1, nie\u010do sa s \u0148ou dialo. A za\u010dali sa vo\u013eby nov\u00e9ho u\u010dite\u013ea. Tu to v\u0161etko fungovalo dobre. N\u00e1\u0161 pgsql01 sa stal nov\u00fdm l\u00eddrom. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/6b2457653e9f77af44cd64b4aff021a0.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>M\u00e1me replik\u00e1ciu, ktor\u00e1 sa stala u\u010dite\u013eom. A je tu druh\u00e1 replik\u00e1cia. A s druhou replik\u00e1ciou boli pr\u00e1ve probl\u00e9my. Sna\u017eila sa prekonfigurova\u0165. Ako ch\u00e1pem, sna\u017eila sa zmeni\u0165 recovery.conf, re\u0161tartova\u0165 Postgres a pripoji\u0165 sa k nov\u00e9mu u\u010dite\u013eovi. Ka\u017ed\u00fdch 10 sek\u00fand p\u00ed\u0161e spr\u00e1vy, \u017ee sa sna\u017e\u00ed, ale nedar\u00ed sa jej. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/381818f189e19c3e1154074b4652f837.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>A po\u010das t\u00fdchto pokusov dost\u00e1va star\u00fd u\u010dite\u013e sign\u00e1l immediate-shutdown. U\u010dite\u013e sa re\u0161tartuje. A tie\u017e sa obnova zastavuje, preto\u017ee star\u00fd u\u010dite\u013e ide do re\u0161tartovania. T. j. replik\u00e1cia sa k nemu nem\u00f4\u017ee pripoji\u0165, preto\u017ee je v re\u017eime vyp\u00ednania. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/d8a8cde925fa8fe601d44ab908ae69dc.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>V nejakom momente sa to samo rozbehlo, ale replik\u00e1cia pri tom sa nespustila. <\/p>\n<p><\/p>\n<p>M\u00e1m jedin\u00fa hypot\u00e9zu, \u017ee v recovery.conf bola adresa star\u00e9ho u\u010dite\u013ea. A ke\u010f sa objavil nov\u00fd u\u010dite\u013e, druh\u00e1 replik\u00e1cia sa st\u00e1le sna\u017eila pripoji\u0165 k star\u00e9mu u\u010dite\u013eovi. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/425ea3af1f1e186f4760e1fb1f5419fa.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Ke\u010f sa Patroni spustil na druhej replik\u00e1cii, uzol sa spustil, ale nemohol sa pripoji\u0165 k replik\u00e1cii. A vytvoril sa oneskorenie replik\u00e1cie, ktor\u00e9 vyzeralo asi takto. T. j. v\u0161etky tri uzly boli na mieste, ale druh\u00fd uzol zaostal. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/f564eacd9944666c9a3f5bea634ede6b.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Ak sa pozrieme na logy, ktor\u00e9 sa p\u00edsali, bolo vidie\u0165, \u017ee replik\u00e1cia sa nem\u00f4\u017ee spusti\u0165, preto\u017ee transak\u010dn\u00e9 z\u00e1znamy sa l\u00ed\u0161ia. A tie transak\u010dn\u00e9 z\u00e1znamy, ktor\u00e9 pon\u00faka master a s\u00fa uveden\u00e9 v recovery.conf, jednoducho nevyhovuj\u00fa n\u00e1\u0161mu aktu\u00e1lnemu uzlu. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/6b09947d157c9ef0a190da2df2cb5892.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>A tu som urobil chybu. Mal som pr\u00eds\u0165 a pozrie\u0165 sa, \u010do je v recovery.conf, aby som overil svoju hypot\u00e9zu, \u017ee sa prip\u00e1jame k nespr\u00e1vnemu masterovi. Ale vtedy som sa len za\u010d\u00ednal orientova\u0165 a nenapadlo ma to, alebo som si v\u0161imol, \u017ee replik\u00e1cia zaost\u00e1va a bude ju potrebn\u00e9 znova previes\u0165, t. j. pracoval som na to len tak naoko. To bola moja chyba. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/159f2256fd81428fd0d82a255528887d.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>O 30 min\u00fat pri\u0161iel administr\u00e1tor, t. j. ja som re\u0161tartoval Patroni na replike. U\u017e som na nej zanechal n\u00e1dej, myslel som, \u017ee ju treba prein\u0161talova\u0165. A povedal som si - re\u0161tartujem Patroni, mo\u017eno sa objav\u00ed nie\u010do dobr\u00e9. Spustil sa recovery. A datab\u00e1za sa dokonca otvorila, bola pripraven\u00e1 prij\u00edma\u0165 pripojenia. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/aeac98750386c8389d39e006a229cc85.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Replik\u00e1cia sa spustila. Ale o min\u00fatu sa op\u00e4\u0165 odpojila s chybou, \u017ee jej nevyhovuj\u00fa transak\u010dn\u00e9 z\u00e1znamy. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/17eaba000a79c80c29d1bc3902de05e6.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Pomyslel som si, \u017ee ju sk\u00fasim re\u0161tartova\u0165 e\u0161te raz. Znovu som re\u0161tartoval Patroni, pri\u010dom som nera\u0161tartoval Postgres, ale iba Patroni v n\u00e1deji, \u017ee magicky spust\u00ed datab\u00e1zu. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/a85b38e033c45ad73da317f41e0ef24a.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Replik\u00e1cia sa op\u00e4\u0165 spustila, ale z\u00e1znamy v transak\u010dnom logu sa l\u00ed\u0161ili, boli in\u00e9, ne\u017e pri predch\u00e1dzaj\u00facej pokuse o spustenie. Replik\u00e1cia sa op\u00e4\u0165 zastavila. A spr\u00e1va bola op\u00e4\u0165 tro\u0161ku in\u00e1. A nebola pre m\u0148a ve\u013emi informat\u00edvna. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/fd90d708230bd54af685786c7e361ee1.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>A tu mi napadlo \u2013 \u010do ak re\u0161tartujem Postgres, zatia\u013e na aktu\u00e1lnom masterovi urob\u00edm checkpoint, aby som posunul bod v transak\u010dnom logu o k\u00fasok \u010falej, aby sa recovery za\u010dalo z in\u00e9ho momentu? Plus, mali sme tam aj z\u00e1soby WAL. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/19e67f95d1a74141a013947c9aa39e38.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Re\u0161tartoval som Patroni, urobil som nieko\u013eko checkpoints na majstrovskom serveri, nieko\u013eko restart-poinov na replike, ke\u010f sa otvorila. A to pomohlo. Dlho som rozm\u00fd\u0161\u013eal, pre\u010do to pomohlo a ako to fungovalo. A replik\u00e1cia sa spustila. A replik\u00e1cia u\u017e viac nepreru\u0161ovala. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/418ca59a681be84f51ed374a73a759c9.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Tak\u00fdto probl\u00e9m je pre m\u0148a jedn\u00fdm z najz\u00e1hadnej\u0161\u00edch, nad ktor\u00fdm sa st\u00e1le zam\u00fd\u0161\u013eam, \u010do sa tam naozaj dialo. <\/p>\n<p><\/p>\n<p>Ak\u00e9 s\u00fa tu z\u00e1very? Patroni m\u00f4\u017ee fungova\u0165, ako sa zam\u00fd\u0161\u013ealo, a bez ak\u00fdchko\u013evek ch\u00fdb. Ale nie je to 100 % z\u00e1ruka, \u017ee je v\u0161etko v poriadku. Replika sa m\u00f4\u017ee spusti\u0165, ale m\u00f4\u017ee by\u0165 v polofunk\u010dnom stave, a aplik\u00e1cia nem\u00f4\u017ee pracova\u0165 s takouto replikou, preto\u017ee tam bud\u00fa star\u00e9 \u00fadaje. <\/p>\n<p><\/p>\n<p>A po failoveri je v\u017edy potrebn\u00e9 skontrolova\u0165, \u010di je s klastrom v\u0161etko v poriadku, t.j. je potrebn\u00fd po\u010det replik, \u017eiadny lag replik\u00e1cie.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/aca7047ff33d3efe14071b798b554091.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>A po\u010das presk\u00fam\u00e1vania t\u00fdchto probl\u00e9mov formulujem odpor\u00fa\u010dania. Sna\u017eil som sa ich zhrn\u00fa\u0165 do dvoch sn\u00edmok. Pravdepodobne by sa v\u0161etky pr\u00edbehy dali zhrn\u00fa\u0165 do dvoch sn\u00edmok a len ich rozpr\u00e1va\u0165.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/d4ad31b26d83c8846fe4097a11d70849.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Ke\u010f pou\u017e\u00edvate Patroni, mus\u00edte ma\u0165 monitoring. V\u017edy mus\u00edte vedie\u0165, kedy do\u0161lo k autofailoveru, preto\u017ee ak neviete, \u017ee ste mali autofailover, nekontrolujete klaster. A to je zle.<\/p>\n<p><\/p>\n<p>Po ka\u017edom failoveri v\u017edy mus\u00edme ru\u010dne skontrolova\u0165 klaster. Mus\u00edme sa uisti\u0165, \u017ee m\u00e1me v\u017edy aktu\u00e1lny po\u010det replik, \u017eiadny lag replik\u00e1cie, v logoch nie s\u00fa \u017eiadne chyby s\u00favisiace s pr\u00fadovou replik\u00e1ciou, s Patroni, so syst\u00e9mom DCS.<\/p>\n<p><\/p>\n<p>Automatika m\u00f4\u017ee \u00faspe\u0161ne fungova\u0165, Patroni je ve\u013emi dobr\u00fd n\u00e1stroj. M\u00f4\u017ee fungova\u0165, ale to neprivedie klaster do po\u017eadovan\u00e9ho stavu. A ak to nezist\u00edme, budeme ma\u0165 probl\u00e9my.<\/p>\n<p><\/p>\n<p>A Patroni nie je strieborn\u00e1 gu\u013eka. Mus\u00edme ma\u0165 aj predstavu, ako funguje Postgres, ako funguje replik\u00e1cia a o tom, ako Patroni funguje s Postgres, a ako je zabezpe\u010den\u00e1 interakcia medzi uzlami. To je potrebn\u00e9 na to, aby sme vedeli vlastn\u00fdmi rukami opravi\u0165 vznikaj\u00face probl\u00e9my.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/ea4162df3561d2ea7001f2ef6788eaaa.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Ako sa pristupujem k ot\u00e1zke diagnostiky? Tak sa stalo, \u017ee pracujeme s r\u00f4znymi klientmi a nikto nem\u00e1 ELK stack, a mus\u00edme sa prekop\u00e1va\u0165 v logoch, otvoren\u00edm 6 konzol a 2 kariet. Na jednej karte s\u00fa logy Patroni pre ka\u017ed\u00fd uzol, na druhej karte s\u00fa logy Consul alebo Postgres pod\u013ea potreby. Diagnostikova\u0165 to je ve\u013emi \u0165a\u017ek\u00e9. <\/p>\n<p><\/p>\n<p>Ak\u00e9 pr\u00edstupy som si vypracoval? Po prv\u00e9, v\u017edy sa pozer\u00e1m, kedy do\u0161lo k failoveru. A pre m\u0148a je to ur\u010dit\u00fd zlomy. Pozer\u00e1m sa, \u010do sa stalo pred failoverom, po\u010das failoveru a po failoveru. Failover m\u00e1 dva \u010dasov\u00e9 \u00fadaje: \u010das za\u010diatku a konca. <\/p>\n<p><\/p>\n<p>\u010ealej v logoch sledujem udalosti pred failoverom, \u010do predch\u00e1dzalo failoveru, t. j. h\u013ead\u00e1m d\u00f4vody, pre\u010do do\u0161lo k failoveru. <\/p>\n<p><\/p>\n<p>A to d\u00e1va obraz o tom, \u010do sa dialo a \u010do m\u00f4\u017eeme urobi\u0165 v bud\u00facnosti, aby sa tak\u00e9to okolnosti neopakovali (a aby k failoveru nedo\u0161lo).<\/p>\n<p><\/p>\n<p>A kam sa zvy\u010dajne pozer\u00e1me? Pozriem sa:<\/p>\n<p><\/p>\n<ul>\n<li>Najprv na logy Patroni. <\/li>\n<li>Potom sa pozriem na logy Postgres, alebo na logy DCS v z\u00e1vislosti od toho, \u010do som na\u0161iel v logoch Patroni. <\/li>\n<li>A logy syst\u00e9mu niekedy tie\u017e d\u00e1vaj\u00fa pochopenie toho, \u010do bolo pr\u00ed\u010dinou failoveru. <\/li>\n<\/ul>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/a53618e366020e1a550d852fc308c188.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Ako som na tom s Patroni? K Patroni sa staviam ve\u013emi pozit\u00edvne. Pod\u013ea m\u0148a je to najlep\u0161ie, \u010do je dnes k dispoz\u00edcii. Pozn\u00e1m ve\u013ea in\u00fdch produktov. To s\u00fa Stolon, Repmgr, Pg_auto_failover, PAF. 4 n\u00e1stroje. Vysk\u00fa\u0161al som ich v\u0161etky. Patroni sa mi p\u00e1\u010di najviac. <\/p>\n<p><\/p>\n<p>Ak sa ma niekto op\u00fdta: \u201eOdpor\u00fa\u010dam Patroni?\u201c. Povedal by som, \u017ee \u00e1no, preto\u017ee sa mi Patroni p\u00e1\u010di. A mysl\u00edm, \u017ee som sa nau\u010dil, ako ho spravova\u0165. <\/p>\n<p><\/p>\n<p>Ak v\u00e1s zauj\u00edma, ak\u00e9 \u010fal\u0161ie probl\u00e9my m\u00f4\u017eu by\u0165 s Patroni, okrem t\u00fdch, ktor\u00e9 som spomenul, m\u00f4\u017eete sa v\u017edy pozrie\u0165 na str\u00e1nku <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/zalando\/patroni\/issues\/\">issues<\/a><\/noindex> na GitHub. Tam je mno\u017estvo r\u00f4znych pr\u00edbehov a diskutuje sa tam o mnoh\u00fdch zauj\u00edmav\u00fdch probl\u00e9moch. A nakoniec sa nejak\u00e9 chyby nahl\u00e1sili a vyrie\u0161ili, t. j. je to zauj\u00edmav\u00e9 \u010d\u00edtanie. <\/p>\n<p><\/p>\n<p>Tam s\u00fa zauj\u00edmav\u00e9 pr\u00edbehy o tom, ako si \u013eudia strie\u013eaj\u00fa do vlastnej nohy. Je to ve\u013emi pou\u010dn\u00e9. \u010c\u00edta\u0161 a ch\u00e1pe\u0161, \u017ee takto sa to robi\u0165 ned\u00e1. Mal som si to zap\u00edsa\u0165. <\/p>\n<p><\/p>\n<p>A chcel by som sa ve\u013emi po\u010fakova\u0165 spolo\u010dnosti Zalando za to, \u017ee rozv\u00edjaj\u00fa tento projekt, a to konkr\u00e9tne Aleksandrovi Kuku\u0161kovi a Alexejovi Kljukinovi. Alexej Kljukin je jedn\u00fdm z spoluautorov, u\u017e v Zalando nepracuje, ale s\u00fa to dvaja \u013eudia, ktor\u00ed za\u010dali pracova\u0165 s t\u00fdmto produktom. <\/p>\n<p><\/p>\n<p>A m\u00e1m pocit, \u017ee Patroni je ve\u013emi cool vec. Som r\u00e1d, \u017ee existuje, je s \u0148ou zauj\u00edmav\u00e9 pracova\u0165. A ve\u013ek\u00e9 \u010fakujem v\u0161etk\u00fdm prispievate\u013eom, ktor\u00ed p\u00ed\u0161u z\u00e1platy pre Patroni. D\u00fafam, \u017ee Patroni s \u010dasom bude st\u00e1le zrel\u0161\u00ed, lep\u0161\u00ed a funk\u010dnej\u0161\u00ed. U\u017e teraz je funk\u010dn\u00fd, ale d\u00fafam, \u017ee sa e\u0161te zlep\u0161\u00ed. Tak\u017ee ak pl\u00e1nujete implementova\u0165 Patroni, nebojte sa. Je to dobr\u00e9 rie\u0161enie, d\u00e1 sa nasadi\u0165 a pou\u017e\u00edva\u0165. <\/p>\n<p><\/p>\n<p>Na tomto je v\u0161etko. Ak m\u00e1te nejak\u00e9 ot\u00e1zky, p\u00fdtajte sa.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Patroni Failure Stories alebo Ako zr\u00fati\u0165 svoj PostgreSQL cluster. Alexej Lesovskij\" src=\"\/wp-content\/uploads\/2020\/07\/85c2fef2455999b48413c9fc3b235ed6.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Ot\u00e1zky<\/p>\n<p><\/p>\n<p><em>\u010eakujem za prezent\u00e1ciu! Ak po failovere je potrebn\u00e9 sa tam pozrie\u0165 ve\u013emi d\u00f4kladne, tak na \u010do potrebujeme automatick\u00fd failover?<\/em> <\/p>\n<p><\/p>\n<p>Preto\u017ee je to nov\u00e1 vec. Pracujeme s \u0148ou len rok. Rad\u0161ej sa poist\u00edme. Chceme sa prihl\u00e1si\u0165 a skontrolova\u0165, \u010di v\u0161etko fungovalo spr\u00e1vne. Toto je \u00farove\u0148 dospel\u00e9ho ned\u00f4very \u2013 rad\u0161ej to skontrolova\u0165 a pozrie\u0165 sa. <\/p>\n<p><\/p>\n<p><em>Napr\u00edklad, r\u00e1no sme sa prihl\u00e1sili a pozreli sa, \u017ee?<\/em><\/p>\n<p><\/p>\n<p>Nie r\u00e1no, zvy\u010dajne sa dozvieme o autofailovere takmer okam\u017eite. Prich\u00e1dzaj\u00fa n\u00e1m upozornenia, vid\u00edme, \u017ee do\u0161lo k autofailoveru. Takmer okam\u017eite sa prihl\u00e1sime a pozrieme sa. Ale v\u0161etky tieto kontroly by mali by\u0165 prenesen\u00e9 na \u00farove\u0148 monitorovania. Ak sa obraciame na Patroni cez REST API, existuje hist\u00f3ria. Pod\u013ea hist\u00f3rie m\u00f4\u017eeme sledova\u0165 \u010dasov\u00e9 pe\u010diatky, kedy do\u0161lo k failoveru. Na z\u00e1klade toho m\u00f4\u017eeme robi\u0165 monitorovanie. M\u00f4\u017eeme sledova\u0165 hist\u00f3riu, ko\u013eko tam bolo udalost\u00ed. Ak n\u00e1m pribudlo udalost\u00ed, znamen\u00e1 to, \u017ee do\u0161lo k autofailoveru. M\u00f4\u017eeme sa prihl\u00e1si\u0165 a pozrie\u0165 sa. Alebo na\u0161a automatiz\u00e1cia v monitorovan\u00ed skontrolovala, \u017ee m\u00e1me v\u0161etky repliky na mieste, nie je \u017eiadne oneskorenie a v\u0161etko je v poriadku. <\/p>\n<p><\/p>\n<p><em>\u010eakujeme!<\/em><\/p>\n<p><\/p>\n<p><em>Ve\u013ek\u00e9 \u010fakujem za skvel\u00fd pr\u00edbeh! Ak sme presunuli DCS klaster \u010faleko od Postgres klastru, mus\u00edme tento klaster pravidelne spravova\u0165? Ak\u00e9 s\u00fa najlep\u0161ie praktiky na to, \u017ee niektor\u00e9 \u010dasti DCS klastru treba vypn\u00fa\u0165, urobi\u0165 s nimi nie\u010do a podobne? Ako pri tom \u017eije cel\u00e1 t\u00e1to kon\u0161trukcia? A ak\u00fdm sp\u00f4sobom tieto veci robi\u0165?<\/em><\/p>\n<p><\/p>\n<p>Pre jednu spolo\u010dnos\u0165 bolo potrebn\u00e9 vytvori\u0165 maticu probl\u00e9mov, \u010do sa stane, ak zlyh\u00e1 nejak\u00e1 s\u00fa\u010das\u0165 alebo viacero s\u00fa\u010dast\u00ed. Pod\u013ea tejto matice postupne prech\u00e1dzame v\u0161etk\u00fdmi s\u00fa\u010das\u0165ami a vytv\u00e1rame scen\u00e1re v pr\u00edpade zlyhania t\u00fdchto komponentov. Zodpovedaj\u00face k ka\u017ed\u00e9mu scen\u00e1ru zlyhania m\u00f4\u017ee existova\u0165 pl\u00e1n akci\u00ed na obnovenie. A v pr\u00edpade DCS je to s\u00fa\u010das\u0165 \u0161tandardnej infra\u0161trukt\u00fary. A administr\u00e1tor to spravuje a u\u017e sa spoliehame na administr\u00e1torov, ktor\u00ed to spravuj\u00fa a na jeho schopnosti to opravi\u0165 v pr\u00edpade hav\u00e1rie. Ak DCS nie je v\u00f4bec pr\u00edtomn\u00fd, tak ho nasadzujeme my, ale pritom ho pr\u00edli\u0161 nesledujeme, preto\u017ee nezodpoved\u00e1me za infra\u0161trukt\u00faru, ale d\u00e1vame odpor\u00fa\u010dania ako a \u010do monitorova\u0165. <\/p>\n<p><\/p>\n<p><em>T. j. spr\u00e1vne som pochopil, \u017ee treba vypn\u00fa\u0165 Patroni, vypn\u00fa\u0165 failover, vypn\u00fa\u0165 v\u0161etko predt\u00fdm, ako rob\u00edme nie\u010do s hostite\u013emi?<\/em><\/p>\n<p><\/p>\n<p>To z\u00e1vis\u00ed na tom, kolik uzl\u016f m\u00e1me v DCS klastru. Pokud je uzl\u016f hodn\u011b a pokud poru\u0161\u00edme pouze jeden z uzl\u016f (replikaci), klastra si udr\u017euje kvorum. A Patroni z\u016fst\u00e1v\u00e1 funk\u010dn\u00ed. A nic se nespou\u0161t\u00ed. Pokud m\u00e1me slo\u017eit\u00e9 operace, kter\u00e9 zasahuj\u00ed v\u00edce uzl\u016f, jejich\u017e absence m\u016f\u017ee rozlo\u017eit kvorum, pak ano, mo\u017en\u00e1 m\u00e1 smysl pozastavit Patroni. M\u00e1 p\u0159\u00edslu\u0161n\u00fd p\u0159\u00edkaz \u2013 patronictl pause, patronictl resume. Jednodu\u0161e pozastav\u00edme, a automatick\u00e9 p\u0159epnut\u00ed se v tuto dobu nespust\u00ed. Prov\u00e1d\u00edme \u00fadr\u017ebu na DCS klastru, pot\u00e9 pozastaven\u00ed zru\u0161\u00edme a pokra\u010dujeme v pr\u00e1ci.<\/p>\n<p><\/p>\n<p><em>\u010eakujem pekne!<\/em><\/p>\n<p><\/p>\n<p><em>Ve\u013emi pekne \u010fakujem za prezent\u00e1ciu! Ako produktov\u00fd t\u00edm reaguje na to, \u017ee m\u00f4\u017eu d\u00f4js\u0165 k strate d\u00e1t?<\/em> <\/p>\n<p><\/p>\n<p>Produktov\u00fdm t\u00edmom to je jedno, ale t\u00edmov\u00ed l\u00eddri s\u00fa nerv\u00f3zni. <\/p>\n<p><\/p>\n<p><em>Ak\u00e9 s\u00fa tam z\u00e1ruky?<\/em><\/p>\n<p><\/p>\n<p>S z\u00e1rukami je to ve\u013emi \u0165a\u017ek\u00e9. Existuje prezent\u00e1cia od Alexandra Kuku\u0161kina 'Ako vypo\u010d\u00edta\u0165 RPO a RTO', t. j. \u010das obnovy a ko\u013eko d\u00e1t m\u00f4\u017eeme strati\u0165. Mysl\u00edm, \u017ee je potrebn\u00e9 n\u00e1js\u0165 tieto sn\u00edmky a \u0161tudova\u0165 ich. Pokia\u013e si pam\u00e4t\u00e1m, s\u00fa tam konkr\u00e9tne kroky, ako tieto veci po\u010d\u00edta\u0165. Ko\u013eko transakci\u00ed m\u00f4\u017eeme strati\u0165, ko\u013eko d\u00e1t m\u00f4\u017eeme strati\u0165. Ako alternat\u00edvu m\u00f4\u017eeme pou\u017ei\u0165 synchronn\u00fa replik\u00e1ciu na \u00farovni Patroni, ale to je dvojse\u010dn\u00e1 zbra\u0148: bu\u010f m\u00e1me spo\u013eahlivos\u0165 d\u00e1t, alebo str\u00e1came na r\u00fdchlosti. Existuje synchronn\u00e1 replik\u00e1cia, ale ona tie\u017e nezaru\u010duje 100%-n\u00fa ochranu pred stratou d\u00e1t.<\/p>\n<p><\/p>\n<p><em>Alexej, \u010fakujem za skvel\u00fa prezent\u00e1ciu! M\u00e1te sk\u00fasenosti s pou\u017eit\u00edm Patroni pre ochranu na nule? T. j. spolu so synchronn\u00fdm uzlom? To je prv\u00e1 ot\u00e1zka. A druh\u00e1 ot\u00e1zka. Pou\u017eili ste r\u00f4zne rie\u0161enia. Pou\u017eili sme Repmgr, ale bez automatick\u00e9ho prep\u00ednania a teraz pl\u00e1nujeme prida\u0165 automatick\u00e9 prep\u00ednanie. Zva\u017eujeme Patroni ako alternat\u00edvne rie\u0161enie. \u010co m\u00f4\u017eete poveda\u0165 o v\u00fdhod\u00e1ch v porovnan\u00ed s Repmgr?<\/em><\/p>\n<p><\/p>\n<p>Prv\u00e1 ot\u00e1zka bola o synchronn\u00fdch replik\u00e1ch. Nikto u n\u00e1s synchronn\u00fa replik\u00e1ciu nepou\u017e\u00edva, preto\u017ee sa v\u0161etci boja (U\u017e pou\u017e\u00edvaj\u00fa niektor\u00ed klienti, probl\u00e9my s v\u00fdkonom v podstate nezaznamenali \u2014 <em>Pozn\u00e1mka predn\u00e1\u0161aj\u00faceho<\/em>). Ale pre seba sme si vypracovali pravidlo, \u017ee v klastri synchronnej replik\u00e1cie musia by\u0165 minim\u00e1lne tri uzly, preto\u017ee ak m\u00e1me dva uzly a ak master alebo replika vypadn\u00fa, Patroni prevedie tento uzol do samostatn\u00e9ho re\u017eimu, aby aplik\u00e1cia mohla pokra\u010dova\u0165 v prev\u00e1dzke. V tomto pr\u00edpade existuje riziko straty d\u00e1t.<\/p>\n<p><\/p>\n<p>Pokia\u013e ide o druh\u00fa ot\u00e1zku, pou\u017e\u00edvali sme Repmgr a st\u00e1le ho pou\u017e\u00edvame u niektor\u00fdch klientov z historick\u00fdch d\u00f4vodov. \u010co k tomu poveda\u0165? V Patroni je automatick\u00e9 prech\u00e1dzanie na failover out-of-the-box, zatia\u013e \u010do v Repmgr ide automatick\u00e9 prech\u00e1dzanie na failover ako dodato\u010dn\u00e1 funkcia, ktor\u00fa treba aktivova\u0165. Mus\u00edme spusti\u0165 Repmgr daemon na ka\u017edom uzle a potom m\u00f4\u017eeme nastavi\u0165 automatick\u00e9 prech\u00e1dzanie na failover. <\/p>\n<p><\/p>\n<p>Repmgr kontroluje, \u010di s\u00fa uzly Postgres \u017eiv\u00e9. Procesy Repmgr kontroluj\u00fa existenciu navz\u00e1jom, to nie je ve\u013emi efekt\u00edvny pr\u00edstup, preto\u017ee m\u00f4\u017eu nasta\u0165 zlo\u017eit\u00fd pr\u00edpady sie\u0165ovej izol\u00e1cie, pri ktor\u00fdch sa ve\u013ek\u00fd Repmgr klaster m\u00f4\u017ee rozpadn\u00fa\u0165 na nieko\u013eko mal\u00fdch a pokra\u010dova\u0165 v prev\u00e1dzke. U\u017e dlh\u00fa dobu nesledujem Repmgr, mo\u017eno to u\u017e opravili... alebo nie. No vy\u010dlenenie inform\u00e1ci\u00ed o stave klastru do DCS, ako to rob\u00ed Stolon, Patroni, je naj\u017eivotaschopnej\u0161ia mo\u017enos\u0165. <\/p>\n<p><\/p>\n<p><em>Alexej, m\u00e1m ot\u00e1zku, mo\u017eno nevyberan\u00fa. Vo jednom z prv\u00fdch pr\u00edkladov DCS ste presunuli z lok\u00e1lneho stroja na vzdialen\u00fd uzol. Ch\u00e1peme, \u017ee sie\u0165 je vec, ktor\u00e1 m\u00e1 svoje \u0161pecifik\u00e1, \u017eije sama o sebe. A \u010do sa stane, ak sa z nejak\u00e9ho d\u00f4vodu DCS-klaster stane nedostupn\u00fdm? D\u00f4vody spom\u00edna\u0165 nebudem, m\u00f4\u017ee ich by\u0165 mno\u017estvo: od kriv\u00fdch r\u00fak sie\u0165arov po re\u00e1lne probl\u00e9my.<\/em> <\/p>\n<p><\/p>\n<p>Nepreriekol som to nahlas, ale klaster DCS by mal by\u0165 taktie\u017e odoln\u00fd vo\u010di poruch\u00e1m, t.j. mal by ma\u0165 nep\u00e1rny po\u010det uzlov, aby sa mohol z\u00edska\u0165 kvorum. \u010co sa stane, ak sa klaster DCS stane nedostupn\u00fdm, alebo sa nem\u00f4\u017ee z\u00edska\u0165 kvorum, t.j. prich\u00e1dzaj\u00fa k nejak\u00e9mu sie\u0165ov\u00e9mu rozdeleniu alebo zlyhaniu uzlov? V tomto pr\u00edpade prech\u00e1dza klaster Patroni do re\u017eimu len na \u010d\u00edtanie. Klaster Patroni nem\u00f4\u017ee ur\u010di\u0165 stav klastru a \u010do m\u00e1 robi\u0165. Nem\u00f4\u017ee sa spoji\u0165 s DCS a ulo\u017ei\u0165 tam nov\u00fd stav klastru, preto cel\u00fd klaster prech\u00e1dza do re\u017eimu len na \u010d\u00edtanie. A \u010dak\u00e1 bu\u010f na manu\u00e1lny z\u00e1sah od oper\u00e1tora, alebo k\u00fdm sa DCS neobnov\u00ed.<\/p>\n<p><\/p>\n<p><em>V podstate sa DCS pre n\u00e1s st\u00e1va slu\u017ebou rovnako d\u00f4le\u017eitou, ako je samotn\u00e1 datab\u00e1za?<\/em><\/p>\n<p><\/p>\n<p>\u00c1no, v mnoh\u00fdch modern\u00fdch spolo\u010dnostiach je Service Discovery neoddelite\u013enou s\u00fa\u010das\u0165ou infra\u0161trukt\u00fary. Zav\u00e1dza sa dokonca sk\u00f4r, ako sa v infra\u0161trukt\u00fare objav\u00ed datab\u00e1za. Povedzme, \u017ee infra\u0161trukt\u00fara bola spusten\u00e1, vybavila sa v DCentr\u00e1le a hne\u010f m\u00e1me Service Discovery. Ak je to Consul, m\u00f4\u017ee by\u0165 postaven\u00fd aj DNS. Ak je to Etcd, m\u00f4\u017ee to by\u0165 \u010das\u0165 z klasteru Kubernetes, v ktorom sa u\u017e rozbehne v\u0161etko ostatn\u00e9. Mysl\u00edm si, \u017ee Service Discovery je u\u017e neoddelite\u013enou s\u00fa\u010das\u0165ou modern\u00fdch infra\u0161trukt\u00far. A prem\u00fd\u0161\u013ea sa o \u0148om ove\u013ea sk\u00f4r, ako o datab\u00e1zach. <\/p>\n<p><\/p>\n<p><em>\u010eakujeme!<\/em><\/p>\n<p>Zdroj: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/post\/512768\/\">habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u041e\u0441\u043d\u043e\u0432\u043d\u0430\u044f \u0446\u0435\u043b\u044c Patroni \u2014 \u044d\u0442\u043e \u043e\u0431\u0435\u0441\u043f\u0435\u0447\u0435\u043d\u0438\u0435 High Availability \u0434\u043b\u044f PostgreSQL. \u041d\u043e Patroni \u2014 \u044d\u0442\u043e \u043b\u0438\u0448\u044c template, \u0430 \u043d\u0435 \u0433\u043e\u0442\u043e\u0432\u044b\u0439 \u0438\u043d\u0441\u0442\u0440\u0443\u043c\u0435\u043d\u0442 (\u0447\u0442\u043e, \u0432 \u043e\u0431\u0449\u0435\u043c, \u0438 \u0441\u043a\u0430\u0437\u0430\u043d\u043e \u0432 \u0434\u043e\u043a\u0443\u043c\u0435\u043d\u0442\u0430\u0446\u0438\u0438). \u041d\u0430 \u043f\u0435\u0440\u0432\u044b\u0439 \u0432\u0437\u0433\u043b\u044f\u0434, \u043d\u0430\u0441\u0442\u0440\u043e\u0438\u0432 Patroni \u0432 \u0442\u0435\u0441\u0442\u043e\u0432\u043e\u0439 \u043b\u0430\u0431\u0435, \u043c\u043e\u0436\u043d\u043e \u0443\u0432\u0438\u0434\u0435\u0442\u044c, \u043a\u0430\u043a\u043e\u0439 \u044d\u0442\u043e \u043f\u0440\u0435\u043a\u0440\u0430\u0441\u043d\u044b\u0439 \u0438\u043d\u0441\u0442\u0440\u0443\u043c\u0435\u043d\u0442 \u0438 \u043a\u0430\u043a \u043e\u043d \u043b\u0435\u0433\u043a\u043e \u043e\u0431\u0440\u0430\u0431\u0430\u0442\u044b\u0432\u0430\u0435\u0442 \u043d\u0430\u0448\u0438 \u043f\u043e\u043f\u044b\u0442\u043a\u0438 \u0440\u0430\u0437\u0432\u0430\u043b\u0438\u0442\u044c \u043a\u043b\u0430\u0441\u0442\u0435\u0440. \u041e\u0434\u043d\u0430\u043a\u043e \u043d\u0430 \u043f\u0440\u0430\u043a\u0442\u0438\u043a\u0435 \u0432 \u043f\u0440\u043e\u0438\u0437\u0432\u043e\u0434\u0441\u0442\u0432\u0435\u043d\u043d\u043e\u0439 [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":90104,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-90103","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.4 - aioseo.com -->\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/patroni-failure-stories-or-how-to-crash-your-postgresql-cluster-aleksej-lesovskij\" \/>\n\t\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.4\" \/>\n\t\t<meta property=\"og:locale\" content=\"sk_SK\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47Patroni Failure Stories or How to crash your PostgreSQL cluster. \u0410\u043b\u0435\u043a\u0441\u0435\u0439 \u041b\u0435\u0441\u043e\u0432\u0441\u043a\u0438\u0439 | ProHoster\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/patroni-failure-stories-or-how-to-crash-your-postgresql-cluster-aleksej-lesovskij\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2020-07-29T11:42:32+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-07-29T11:42:32+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47Patroni Failure Stories alebo Ako zhodi\u0165 svoj PostgreSQL klaster. Alexej Lesovsk\u00fd | ProHoster","description":"","canonical_url":"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/patroni-failure-stories-or-how-to-crash-your-postgresql-cluster-aleksej-lesovskij","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"sk_SK","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47Patroni Failure Stories or How to crash your PostgreSQL cluster. \u0410\u043b\u0435\u043a\u0441\u0435\u0439 \u041b\u0435\u0441\u043e\u0432\u0441\u043a\u0438\u0439 | ProHoster","og:url":"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/patroni-failure-stories-or-how-to-crash-your-postgresql-cluster-aleksej-lesovskij","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-07-29T11:42:32+00:00","article:modified_time":"2020-07-29T11:42:32+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"90103","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 12:58:48","updated":"2026-02-09 16:50:35","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/posts\/90103","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/comments?post=90103"}],"version-history":[{"count":1,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/posts\/90103\/revisions"}],"predecessor-version":[{"id":158759,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/posts\/90103\/revisions\/158759"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/media\/90104"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/media?parent=90103"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/categories?post=90103"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/tags?post=90103"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}