{"id":35706,"date":"2019-10-31T22:05:50","date_gmt":"2019-10-31T19:05:50","guid":{"rendered":"https:\/\/prohoster.info\/blog\/otkazoustojchivyj-klaster-postgresql-patroni-opyt-vnedreniya\/"},"modified":"2026-05-18T20:58:46","modified_gmt":"2026-05-18T18:58:46","slug":"otkazoustojchivyj-klaster-postgresql-patroni-opyt-vnedreniya","status":"publish","type":"post","link":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/otkazoustojchivyj-klaster-postgresql-patroni-opyt-vnedreniya","title":{"rendered":"Odoln\u00fd cluster PostgreSQL + Patroni. Zku\u0161enosti s implementac\u00ed","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p>V \u010dl\u00e1nku pop\u00ed\u0161u, jak jsme p\u0159istoupili k ot\u00e1zce odolnosti PostgreSQL, pro\u010d se to stalo d\u016fle\u017eit\u00e9 a co jsme nakonec dos\u00e1hli.<\/p>\n<p>M\u00e1me vysoce zat\u011b\u017eovan\u00fd servis: 2,5 milionu u\u017eivatel\u016f po cel\u00e9m sv\u011bt\u011b, p\u0159es 50 000 aktivn\u00edch u\u017eivatel\u016f ka\u017ed\u00fd den. Servery se nach\u00e1z\u00ed na Amazone v jednom regionu Irska: v provozu je nep\u0159etr\u017eit\u011b 100 a v\u00edce r\u016fzn\u00fdch server\u016f, z nich\u017e t\u00e9m\u011b\u0159 50 je s datab\u00e1zemi.<\/p>\n<p>Cel\u00fd backend je velk\u00e1 monolitick\u00e1 stateful aplikace napsan\u00e1 v Jav\u011b, kter\u00e1 udr\u017euje trval\u00e9 websocketov\u00e9 spojen\u00ed s klientem. P\u0159i sou\u010dasn\u00e9 pr\u00e1ci n\u011bkolika u\u017eivatel\u016f na jednom panelu v\u0161ichni vid\u00ed zm\u011bny v re\u00e1ln\u00e9m \u010dase, proto\u017ee ka\u017edou zm\u011bnu zapisujeme do datab\u00e1ze. M\u00e1me asi 10 000 dotaz\u016f za sekundu na na\u0161e datab\u00e1ze. V dob\u011b nejvy\u0161\u0161\u00ed z\u00e1t\u011b\u017ee v Redis zapisujeme 80-100 tis\u00edc dotaz\u016f za sekundu.<br \/>\n<img decoding=\"async\" style=\"display: block; margin: 0 auto;\" src=\"\/wp-content\/uploads\/2019\/06\/443f85815b0560fade7db5b639942935.png\" alt=\"Odoln\u00fd cluster PostgreSQL + Patroni. Zku\u0161enosti s implementac\u00ed\" \/><br \/>\n<a rel=\"nofollow\" name=\"habracut\"><\/a><\/p>\n<h2>Pro\u010d jsme p\u0159e\u0161li z Redis na PostgreSQL<\/h2>\n<p>Zpo\u010d\u00e1tku n\u00e1\u0161 servis pracoval s Redis, key-value \u00falo\u017ei\u0161t\u011bm, kter\u00e9 uchov\u00e1v\u00e1 v\u0161echna data v opera\u010dn\u00ed pam\u011bti. <a href=\"https:\/\/prohoster.info\/cs\/server\/\">serveru<\/a>.<\/p>\n<p>V\u00fdhody Redis:<\/p>\n<ol>\n<li>Vysok\u00e1 rychlost odezvy, proto\u017ee v\u0161echno je ulo\u017eeno v pam\u011bti;<\/li>\n<li>Pohodln\u00e9 z\u00e1lohov\u00e1n\u00ed a replikace.<\/li>\n<\/ol>\n<p>Nev\u00fdhody Redis pro n\u00e1s:<\/p>\n<ol>\n<li>Chyb\u00ed skute\u010dn\u00e9 transakce. Pokou\u0161eli jsme se je simulovat na \u00farovni na\u0161\u00ed aplikace. Bohu\u017eel to ne v\u017edy fungovalo dob\u0159e a vy\u017eadovalo to psan\u00ed velmi slo\u017eit\u00e9ho k\u00f3du.<\/li>\n<li>Objem dat je omezen mno\u017estv\u00edm pam\u011bti. P\u0159i zvy\u0161ov\u00e1n\u00ed objemu dat pam\u011b\u0165 roste a nakonec se dostaneme na charakteristiky vybran\u00e9ho instance, co\u017e v AWS vy\u017eaduje zastaven\u00ed na\u0161eho servisu pro zm\u011bnu typu instance.<\/li>\n<li>Je nutn\u00e9 neust\u00e1le udr\u017eovat n\u00edzkou latenci, proto\u017ee m\u00e1me velmi vysok\u00fd po\u010det dotaz\u016f. Optim\u00e1ln\u00ed \u00farove\u0148 zpo\u017ed\u011bn\u00ed pro n\u00e1s je 17-20 ms. P\u0159i \u00farovni 30-40 ms dost\u00e1v\u00e1me dlouh\u00e9 odpov\u011bdi na dotazy na\u0161\u00ed aplikace a degradaci servisu. Bohu\u017eel se to stalo v z\u00e1\u0159\u00ed 2018, kdy jeden z instance s Redis n\u011bjak\u00fdm d\u016fvodem z\u00edskal latenci dvakr\u00e1t v\u011bt\u0161\u00ed ne\u017e obvykle. Pro \u0159e\u0161en\u00ed probl\u00e9mu jsme zastavili servis uprost\u0159ed pracovn\u00edho dne kv\u016fli nepl\u00e1novan\u00e9 \u00fadr\u017eb\u011b a vym\u011bnili probl\u00e9mov\u00fd instance Redis.<\/li>\n<li>Snadno se m\u016f\u017ee vyskytnout nekonzistentnost dat i p\u0159i drobn\u00fdch chyb\u00e1ch v k\u00f3du a pot\u00e9 str\u00e1v\u00edme hodn\u011b \u010dasu psan\u00edm k\u00f3du pro opravu t\u011bchto dat.<\/li>\n<\/ol>\n<p>Zohlednili jsme nedostatky a uv\u011bdomili si, \u017ee pot\u0159ebujeme p\u0159ej\u00edt na n\u011bco pohodln\u011bj\u0161\u00edho, s norm\u00e1ln\u00edmi transakcemi a men\u0161\u00ed z\u00e1vislost\u00ed na latenci. Provedli jsme v\u00fdzkum, analyzovali jsme mnoho mo\u017enost\u00ed a vybrali PostgreSQL.<\/p>\n<p>Na novou datab\u00e1zi p\u0159ech\u00e1z\u00edme ji\u017e 1,5 roku a p\u0159enesli jsme pouze malou \u010d\u00e1st dat, proto nyn\u00ed pracujeme sou\u010dasn\u011b s Redis a PostgreSQL. Podrobnosti o f\u00e1z\u00edch p\u0159echodu a p\u0159ep\u00edn\u00e1n\u00ed dat mezi datab\u00e1zemi jsou uvedeny v <a href=\"https:\/\/habr.com\/ru\/company\/miro\/blog\/437826\/\" rel=\"nofollow\">\u010dl\u00e1nku m\u00e9ho kolegy.<\/a>.<\/p>\n<p>Kdy\u017e jsme za\u010d\u00ednali s p\u0159echodem, na\u0161e aplikace komunikovala p\u0159\u00edmo s datab\u00e1z\u00ed a spojovala se s masterem Redis a PostgreSQL. Klaster PostgreSQL se skl\u00e1dal z masteru a repliky s asynchronn\u00ed replikac\u00ed. Takto vypadala sch\u00e9ma pr\u00e1ce s datab\u00e1zemi:<br \/>\n<img decoding=\"async\" style=\"display: block; margin: 0 auto;\" src=\"\/wp-content\/uploads\/2019\/06\/674dd77de4c8b48a946c9f64c0b2fdd1.png\" alt=\"Odoln\u00fd cluster PostgreSQL + Patroni. Zku\u0161enosti s implementac\u00ed\" \/><\/p>\n<h2>Nasazen\u00ed PgBouncer.<\/h2>\n<p>B\u011bhem na\u0161eho p\u0159echodu se tak\u00e9 produkt vyv\u00edjel: zvy\u0161oval se po\u010det u\u017eivatel\u016f a po\u010det server\u016f, kter\u00e9 pracovaly s PostgreSQL, a n\u00e1m za\u010dalo chyb\u011bt spojen\u00ed. PostgreSQL na ka\u017ed\u00e9 p\u0159ipojen\u00ed vytv\u00e1\u0159\u00ed samostatn\u00fd proces a spot\u0159ebov\u00e1v\u00e1 zdroje. Zvy\u0161ovat po\u010det spojen\u00ed lze do ur\u010dit\u00e9 m\u00edry, jinak hroz\u00ed, \u017ee datab\u00e1ze bude fungovat suboptim\u00e1ln\u011b. Ide\u00e1ln\u00edm \u0159e\u0161en\u00edm v takov\u00e9 situaci by bylo zvolit mana\u017eera p\u0159ipojen\u00ed, kter\u00fd se postav\u00ed p\u0159ed datab\u00e1zi.<\/p>\n<p>M\u011bli jsme dv\u011b mo\u017enosti pro mana\u017eera p\u0159ipojen\u00ed: Pgpool a PgBouncer. Ale prvn\u00ed nepodporuje transak\u010dn\u00ed re\u017eim pr\u00e1ce s datab\u00e1z\u00ed, proto jsme zvolili PgBouncer.<\/p>\n<p>Nastavili jsme n\u00e1sleduj\u00edc\u00ed sch\u00e9ma pr\u00e1ce: na\u0161e aplikace se p\u0159ipojuje k jednomu PgBounceru, za kter\u00fdm se nach\u00e1zej\u00ed masters PostgreSQL, a za ka\u017ed\u00fdm masterem je jedna replikace s asynchronn\u00ed replikac\u00ed.<br \/>\n<img decoding=\"async\" style=\"display: block; margin: 0 auto;\" src=\"\/wp-content\/uploads\/2019\/06\/dabcb7f6006520c4fec85fb925e8975e.png\" alt=\"Odoln\u00fd cluster PostgreSQL + Patroni. Zku\u0161enosti s implementac\u00ed\" \/><\/p>\n<p>P\u0159itom jsme nemohli uchov\u00e1vat cel\u00fd objem dat v PostgreSQL a pro n\u00e1s byla d\u016fle\u017eit\u00e1 rychlost pr\u00e1ce s datab\u00e1z\u00ed, proto jsme za\u010dali shardovat PostgreSQL na aplika\u010dn\u00ed \u00farovni. V\u00fd\u0161e popsan\u00e9 sch\u00e9ma je pro to relativn\u011b pohodln\u00e9: p\u0159i p\u0159id\u00e1n\u00ed nov\u00e9ho shardu sta\u010d\u00ed aktualizovat konfiguraci PgBounceru a aplikace m\u016f\u017ee okam\u017eit\u011b pracovat s nov\u00fdm shardem.<\/p>\n<h3>Odolnost PgBounceru.<\/h3>\n<p>Tento syst\u00e9m fungoval, dokud jedin\u00fd instance PgBouncer nezkolaboval. Nach\u00e1z\u00edme se v AWS, kde v\u0161echny instance b\u011b\u017e\u00ed na hardwaru, kter\u00fd ob\u010das vypov\u00edd\u00e1 slu\u017ebu. V takov\u00fdch p\u0159\u00edpadech se instance jednodu\u0161e p\u0159esune na nov\u00fd hardware a znovu za\u010dne fungovat. Stejn\u011b to bylo i s PgBouncer, av\u0161ak stal se nedostupn\u00fdm. V\u00fdsledkem tohoto p\u00e1du byla nedostupnost na\u0161\u00ed slu\u017eby po dobu 25 minut. AWS pro takov\u00e9 situace doporu\u010duje pou\u017e\u00edvat redundanci na stran\u011b u\u017eivatele, co\u017e jsme v tu dobu nem\u011bli implementov\u00e1no.<\/p>\n<p>Po tom jsme se v\u00e1\u017en\u011b zamysleli nad odolnost\u00ed PgBouncer a klastr\u016f PostgreSQL, proto\u017ee podobn\u00e1 situace se mohla opakovat s jakoukoli instanc\u00ed v na\u0161em AWS \u00fa\u010dtu.<\/p>\n<p>Sch\u00e9ma odolnosti PgBouncer jsme navrhli n\u00e1sledovn\u011b: v\u0161echny aplika\u010dn\u00ed servery se obracej\u00ed na Network Load Balancer, za kter\u00fdm stoj\u00ed dva PgBouncer. Ka\u017ed\u00fd PgBouncer se d\u00edv\u00e1 na stejn\u00e9 master PostgreSQL ka\u017ed\u00e9ho shardu. V p\u0159\u00edpad\u011b opakov\u00e1n\u00ed situace s p\u00e1dem instance AWS je ve\u0161ker\u00fd provoz p\u0159esm\u011brov\u00e1n p\u0159es jin\u00fd PgBouncer. Odolnost Network Load Balancer zabezpe\u010duje AWS.<\/p>\n<p>Takov\u00e9 sch\u00e9ma umo\u017e\u0148uje bez probl\u00e9m\u016f p\u0159id\u00e1vat nov\u00e9 servery PgBouncer.<br \/>\n<img decoding=\"async\" style=\"display: block; margin: 0 auto;\" src=\"\/wp-content\/uploads\/2019\/06\/2c1f1e7d9f7be4fdeb43858520d55e36.png\" alt=\"Odoln\u00fd cluster PostgreSQL + Patroni. Zku\u0161enosti s implementac\u00ed\" \/><\/p>\n<h2>Vytvo\u0159en\u00ed odoln\u00e9ho klastru PostgreSQL<\/h2>\n<p>P\u0159i \u0159e\u0161en\u00ed tohoto \u00fakolu jsme zva\u017eovali r\u016fzn\u00e9 varianty: vlastn\u00ed failover, repmgr, AWS RDS, Patroni.<\/p>\n<h3>Vlastn\u00ed skripty<\/h3>\n<p>Mohou monitorovat \u010dinnost mastera a v p\u0159\u00edpad\u011b jeho selh\u00e1n\u00ed pov\u00fd\u0161it repliku na mastera a aktualizovat konfiguraci PgBouncer.<\/p>\n<p>V\u00fdhody tohoto p\u0159\u00edstupu spo\u010d\u00edvaj\u00ed v maxim\u00e1ln\u00ed jednoduchosti, proto\u017ee skripty p\u00ed\u0161ete sami a p\u0159esn\u011b v\u00edte, jak funguj\u00ed.<\/p>\n<p>Nev\u00fdhody:<\/p>\n<ul>\n<li>Master mohl nezkolabovat, m\u00edsto toho mohlo doj\u00edt k v\u00fdpadku s\u00edt\u011b. Failover, nev\u011bda o tom, povy\u0161uje repliku na mastera, zat\u00edmco star\u00fd master st\u00e1le pracuje. V d\u016fsledku toho m\u00e1me dva servery v roli master a nebudeme v\u011bd\u011bt, na kter\u00e9m z nich jsou posledn\u00ed aktu\u00e1ln\u00ed data. Takov\u00e1 situace se naz\u00fdv\u00e1 tak\u00e9 split-brain.<\/li>\n<li>Z\u016fstali jsme bez repliky. V na\u0161\u00ed konfiguraci je master a jedna replika, po p\u0159epnut\u00ed je replika povy\u0161ov\u00e1na na mastera a nem\u00e1me v\u00edce replik, co\u017e znamen\u00e1, \u017ee mus\u00edme ru\u010dn\u011b p\u0159idat novou repliku.<\/li>\n<li>Pot\u0159ebujeme dal\u0161\u00ed monitorov\u00e1n\u00ed pr\u00e1ce failoveru, p\u0159i\u010dem\u017e m\u00e1me 12 shard\u016f PostgreSQL, co\u017e znamen\u00e1, \u017ee mus\u00edme monitorovat 12 klastr\u016f. P\u0159i zvy\u0161ov\u00e1n\u00ed po\u010dtu shard\u016f nesm\u00edme zapomenout aktualizovat failover.<\/li>\n<\/ul>\n<p>Napsan\u00fd failover vypad\u00e1 velmi slo\u017eit\u011b a vy\u017eaduje netrivi\u00e1ln\u00ed podporu. S jedn\u00edm PostgreSQL \u043a\u043b\u0430\u0441\u0442\u0435\u0440em to bude nejjednodu\u0161\u0161\u00ed varianta, ale nen\u00ed \u0161k\u00e1lovateln\u00fd, tak\u017ee pro n\u00e1s se nehod\u00ed.<\/p>\n<h3>Repmgr<\/h3>\n<p>Replication Manager pro PostgreSQL \u043a\u043b\u0430ustry, kter\u00fd um\u00ed spravovat pr\u00e1ci PostgreSQL \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0430. P\u0159itom v n\u011bm nen\u00ed automatick\u00fd failover \"z krabice\", tak\u017ee pro spr\u00e1vnou funkci je pot\u0159eba napsat vlastn\u00ed \"obal\" nad hotov\u00fdm \u0159e\u0161en\u00edm. Tak\u017ee v\u0161e m\u016f\u017ee b\u00fdt dokonce slo\u017eit\u011bj\u0161\u00ed ne\u017e s napsan\u00fdmi skripty, proto jsme Repmgr ani nezkou\u0161eli.<\/p>\n<h3>AWS RDS<\/h3>\n<p>Podporuje v\u0161e pot\u0159ebn\u00e9 pro n\u00e1s, um\u00ed d\u011blat z\u00e1lohy a podporuje pool p\u0159ipojen\u00ed. M\u00e1 automatick\u00e9 p\u0159ep\u00edn\u00e1n\u00ed: p\u0159i smrti masteru se replikace stane nov\u00fdm masterem a AWS zm\u011bn\u00ed DNS z\u00e1znam na nov\u00e9ho mastera, p\u0159i\u010dem\u017e repliky mohou b\u00fdt v r\u016fzn\u00fdch AZ.<\/p>\n<p>Mezi nev\u00fdhody lze po\u010d\u00edtat absenci jemn\u00e9ho nastaven\u00ed. Jako p\u0159\u00edklad jemn\u00e9ho nastaven\u00ed: na na\u0161ich instanc\u00edch jsou nastaveny omezen\u00ed pro TCP p\u0159ipojen\u00ed, co\u017e bohu\u017eel nelze ud\u011blat v RDS:<\/p>\n<pre><code class=\"python\">net.ipv4.tcp_keepalive_time=10\nnet.ipv4.tcp_keepalive_intvl=1\nnet.ipv4.tcp_keepalive_probes=5\nnet.ipv4.tcp_retries2=3\n<\/code><\/pre>\n<p>Krom\u011b toho je cena AWS RDS t\u00e9m\u011b\u0159 dvakr\u00e1t vy\u0161\u0161\u00ed ne\u017e b\u011b\u017en\u00e1 cena instance, co\u017e se stalo hlavn\u00edm d\u016fvodem odm\u00edtnut\u00ed tohoto \u0159e\u0161en\u00ed.<\/p>\n<h3>Patroni<\/h3>\n<p>To je \u0161ablona na python pro \u0159\u00edzen\u00ed PostgreSQL s dobrou dokumentac\u00ed, automatick\u00fdm failoverem a zdrojov\u00fdm k\u00f3dem na githubu.<\/p>\n<p>V\u00fdhody Patroni:<\/p>\n<ul>\n<li>Ka\u017ed\u00fd konfigura\u010dn\u00ed parametr je podrobn\u011b pops\u00e1n, jasn\u011b je uvedeno, jak co funguje;<\/li>\n<li>Automatick\u00fd failover funguje z krabice;<\/li>\n<li>Naps\u00e1n v pythonu, a proto\u017ee my sami hodn\u011b p\u00ed\u0161eme v pythonu, bude pro n\u00e1s snaz\u0161\u00ed \u0159e\u0161it probl\u00e9my a mo\u017en\u00e1 dokonce pomoci s rozvojem projektu;<\/li>\n<li>\u00dapln\u011b spravuje PostgreSQL, umo\u017e\u0148uje m\u011bnit konfiguraci okam\u017eit\u011b na v\u0161ech uzlech \u043a\u043b\u0430\uc2a4\ud130u, a pokud pro pou\u017eit\u00ed nov\u00e9 konfigurace je pot\u0159eba restartovat \u043a\u043b\u0430\u0441\u0442\u0435\u0440, lze to op\u011bt ud\u011blat pomoc\u00ed Patroni.<\/li>\n<\/ul>\n<p>Nev\u00fdhody:<\/p>\n<ul>\n<li>Z dokumentace nen\u00ed jasn\u00e9, jak spr\u00e1vn\u011b pracovat s PgBouncer. A\u010dkoli to nen\u00ed snadn\u00e9 nazvat nev\u00fdhodou, proto\u017ee \u00fakolem Patroni je spravovat PostgreSQL, a jak budou prob\u00edhat p\u0159ipojen\u00ed k Patroni \u2014 to u\u017e je n\u00e1\u0161 probl\u00e9m;<\/li>\n<li>Malo p\u0159\u00edklad\u016f zaveden\u00ed Patroni na velk\u00fdch objemech, zat\u00edmco existuje mnoho p\u0159\u00edklad\u016f zaveden\u00ed od nuly.<\/li>\n<\/ul>\n<p>Nakonec jsme pro vytvo\u0159en\u00ed odoln\u00e9ho \u043a\u043b\u0430\u0441\u0442\u0440\u0443 zvolili pr\u00e1v\u011b Patroni.<\/p>\n<h2>Proces zaveden\u00ed Patroni<\/h2>\n<p>P\u0159ed zaveden\u00edm Patroni jsme m\u011bli 12 shard\u016f PostgreSQL v konfiguraci jeden master a jedna replikace s asynchronn\u00ed replikac\u00ed. Aplika\u010dn\u00ed servery komunikovaly s datab\u00e1zemi p\u0159es Network Load Balancer, za kter\u00fdm byly dva instance s PgBouncer, a za nimi se nach\u00e1zely v\u0161echny servery PostgreSQL.<br \/>\n<img decoding=\"async\" style=\"display: block; margin: 0 auto;\" src=\"\/wp-content\/uploads\/2019\/06\/8e5350d2466311593e11add30d5b1bdc.png\" alt=\"Odoln\u00fd cluster PostgreSQL + Patroni. Zku\u0161enosti s implementac\u00ed\" \/><\/p>\n<p>Pro zaveden\u00ed Patroni jsme museli vybrat distribuovan\u00e9 \u00falo\u017ei\u0161t\u011b konfigurace clustera. Patroni pracuje s distribuovan\u00fdmi syst\u00e9my \u00falo\u017ei\u0161t\u011b konfigurac\u00ed, jako jsou etcd, Zookeeper, Consul. Na na\u0161em produk\u010dn\u00edm prost\u0159ed\u00ed m\u00e1me pln\u011b funk\u010dn\u00ed cluster Consul, kter\u00fd pracuje ve spojen\u00ed s Vault a jinak ho nepou\u017e\u00edv\u00e1me. Skv\u011bl\u00e1 p\u0159\u00edle\u017eitost za\u010d\u00edt pou\u017e\u00edvat Consul podle jeho \u00fa\u010delu.<\/p>\n<h3>Jak funguje Patroni s Consul<\/h3>\n<p>M\u00e1me cluster Consul, kter\u00fd se skl\u00e1d\u00e1 ze t\u0159\u00ed uzl\u016f a cluster Patroni, kter\u00fd se skl\u00e1d\u00e1 z l\u00eddra a repliky (v Patroni je master naz\u00fdv\u00e1n l\u00eddrem clustera a slaves \u2014 replikami). Ka\u017ed\u00fd instance clustera Patroni neust\u00e1le pos\u00edl\u00e1 do Consul informace o stavu clustera. Proto ze Consul v\u017edy m\u016f\u017eete zjistit aktu\u00e1ln\u00ed konfiguraci clustera Patroni a kdo je pr\u00e1v\u011b l\u00eddrem.<\/p>\n<p><img decoding=\"async\" style=\"display: block; margin: 0 auto;\" src=\"\/wp-content\/uploads\/2019\/06\/3234594b363904424fdce64c9d77fca5.png\" alt=\"Odoln\u00fd cluster PostgreSQL + Patroni. Zku\u0161enosti s implementac\u00ed\" \/><\/p>\n<p>Pro p\u0159ipojen\u00ed Patroni k Consul sta\u010d\u00ed prostudovat ofici\u00e1ln\u00ed dokumentaci, kde je uvedeno, \u017ee je t\u0159eba uv\u00e9st hostitele ve tvaru http nebo https v z\u00e1vislosti na tom, jak pracujeme s Consul, a sch\u00e9ma p\u0159ipojen\u00ed, voliteln\u011b:<\/p>\n<pre><code class=\"plaintext\">host: host:port pro endpoint Consul, ve form\u00e1tu: http(s):\/\/host:port\nscheme: (voliteln\u00e9) http nebo https, v\u00fdchoz\u00ed je http<\/code><\/pre>\n<p>Vypad\u00e1 to jednodu\u0161e, ale zde za\u010d\u00ednaj\u00ed \u00faskal\u00ed. S Consul pracujeme p\u0159es zabezpe\u010den\u00e9 spojen\u00ed p\u0159es https a na\u0161e konfigura\u010dn\u00ed nastaven\u00ed bude vypadat n\u00e1sledovn\u011b:<\/p>\n<pre><code class=\"python\">consul:\n  host: https:\/\/server.production.consul:8080 \n  verify: true\n  cacert: {{ consul_cacert }}\n  cert: {{ consul_cert }}\n  key: {{ consul_key }}<\/code><\/pre>\n<p>Ale takto to nefunguje. P\u0159i spu\u0161t\u011bn\u00ed Patroni se nem\u016f\u017ee p\u0159ipojit k Consul, proto\u017ee se pokou\u0161\u00ed i tak j\u00edt p\u0159es http.<\/p>\n<p>Pomohl vy\u0159e\u0161it probl\u00e9m zdrojov\u00fd k\u00f3d Patroni. Na\u0161t\u011bst\u00ed je naps\u00e1n v pythonu. Ukazuje se, \u017ee parametr host se nijak nezpracov\u00e1v\u00e1, a protokol je t\u0159eba uv\u00e9st ve sch\u00e9matu. Takhle vypad\u00e1 funk\u010dn\u00ed blok konfigurace pro pr\u00e1ci s Consul u n\u00e1s:<\/p>\n<pre><code class=\"python\">consul:\n  host: server.production.consul:8080\n  scheme: https\n  verify: true\n  cacert: {{ consul_cacert }}\n  cert: {{ consul_cert }}\n  key: {{ consul_key }}<\/code><\/pre>\n<h3>Consul-template<\/h3>\n<p>Tak\u017ee jsme vybrali \u00falo\u017ei\u0161t\u011b pro konfiguraci. Nyn\u00ed mus\u00edme pochopit, jak PgBouncer p\u0159epne svou konfiguraci p\u0159i zm\u011bn\u011b l\u00eddra v clusteru Patroni. V dokumentaci na tuto ot\u00e1zku odpov\u011b\u010f nen\u00ed, proto\u017ee tam nen\u00ed obecn\u011b pops\u00e1na pr\u00e1ce s PgBouncer.<\/p>\n<p>Hled\u00e1n\u00edm \u0159e\u0161en\u00ed jsme na\u0161li \u010dl\u00e1nek (n\u00e1zev bohu\u017eel nev\u00edm), kde bylo uvedeno, \u017ee Consul-template velmi pomohl ve spojen\u00ed PgBouncer a Patroni. To n\u00e1s povzbudilo k prozkoum\u00e1n\u00ed pr\u00e1ce Consul-template.<\/p>\n<p>Uk\u00e1zalo se, \u017ee Consul-template neust\u00e1le monitoruje konfiguraci clusteru PostgreSQL v Consulu. P\u0159i zm\u011bn\u011b l\u00eddra aktualizuje konfiguraci PgBouncer a pos\u00edl\u00e1 p\u0159\u00edkaz k jej\u00edmu restartov\u00e1n\u00ed.<\/p>\n<p><img decoding=\"async\" style=\"display: block; margin: 0 auto;\" src=\"\/wp-content\/uploads\/2019\/06\/6cf92996a127bb6637ab81dc45fdd60a.png\" alt=\"Odoln\u00fd cluster PostgreSQL + Patroni. Zku\u0161enosti s implementac\u00ed\" \/><\/p>\n<p>Velkou v\u00fdhodou template je, \u017ee je uchov\u00e1v\u00e1n ve form\u011b k\u00f3du, tak\u017ee p\u0159i p\u0159id\u00e1v\u00e1n\u00ed nov\u00e9ho shardu sta\u010d\u00ed ud\u011blat nov\u00fd commit a automaticky aktualizovat template, \u010d\u00edm\u017e se zachov\u00e1v\u00e1 princip Infrastructure as code.<\/p>\n<h3>Nov\u00e1 architektura s Patroni<\/h3>\n<p>V\u00fdsledkem jsme z\u00edskali takovou sch\u00e9ma pr\u00e1ce:<br \/>\n<img decoding=\"async\" style=\"display: block; margin: 0 auto;\" src=\"\/wp-content\/uploads\/2019\/06\/bf9a9eff675a26039e3f76b16c6cd713.png\" alt=\"Odoln\u00fd cluster PostgreSQL + Patroni. Zku\u0161enosti s implementac\u00ed\" \/><\/p>\n<p>V\u0161echny servery aplikace se obracej\u00ed na vyrovn\u00e1va\u010d z\u00e1t\u011b\u017ee \u2192 za n\u00edm stoj\u00ed dva instance PgBouncer \u2192 na ka\u017ed\u00e9m instance b\u011b\u017e\u00ed Consul-template, kter\u00fd monitoruje stav ka\u017ed\u00e9ho clusteru Patroni a sleduje aktu\u00e1lnost konfigurace PgBouncer, kter\u00fd sm\u011b\u0159uje dotazy na aktu\u00e1ln\u00edho l\u00eddra ka\u017ed\u00e9ho clusteru.<\/p>\n<h3>Manu\u00e1ln\u00ed testov\u00e1n\u00ed<\/h3>\n<p>Tuto sch\u00e9ma jsme p\u0159ed uveden\u00edm do produkce spustili na mal\u00e9 testovac\u00ed prost\u0159ed\u00ed a ov\u011b\u0159ili fungov\u00e1n\u00ed automatick\u00e9ho p\u0159ep\u00edn\u00e1n\u00ed. Otev\u0159eli jsme n\u00e1st\u011bnku, posunuli n\u00e1lepku a v tu chv\u00edli jsme \"zabili\" l\u00eddra clusteru. V AWS je k tomu dostate\u010dn\u00e9 vypnout instance p\u0159es konzoli.<\/p>\n<p><img decoding=\"async\" style=\"display: block; margin: 0 auto;\" src=\"\/wp-content\/uploads\/2019\/06\/d670fe373b774f1b52bfd8a8c3b53a21.png\" alt=\"Odoln\u00fd cluster PostgreSQL + Patroni. Zku\u0161enosti s implementac\u00ed\" \/><\/p>\n<p>N\u00e1lepka se b\u011bhem 10-20 sekund vr\u00e1tila zp\u011bt, a pak op\u011bt za\u010dala norm\u00e1ln\u011b fungovat. Znamen\u00e1 to, \u017ee cluster Patroni fungoval spr\u00e1vn\u011b: zm\u011bnil l\u00eddra, poslal informace do Consulu, a Consul-template okam\u017eit\u011b p\u0159evzal tyto informace, nahradil konfiguraci PgBouncer a poslal p\u0159\u00edkaz k reload.<\/p>\n<h2>Jak p\u0159e\u017e\u00edt pod vysok\u00fdm zat\u00ed\u017een\u00edm a zachovat minim\u00e1ln\u00ed prostoje?<\/h2>\n<p>V\u0161echno funguje skv\u011ble! Ale objevuj\u00ed se nov\u00e9 ot\u00e1zky: Jak to bude fungovat pod vysok\u00fdm zat\u00ed\u017een\u00edm? Jak rychle a bezpe\u010dn\u011b nasadit v\u0161e na produkci?<\/p>\n<p>Na prvou ot\u00e1zku n\u00e1m pom\u00e1h\u00e1 testovac\u00ed prost\u0159ed\u00ed, na kter\u00e9m prov\u00e1d\u00edme z\u00e1t\u011b\u017eov\u00e9 testov\u00e1n\u00ed. Je pln\u011b identick\u00e9 s produk\u010dn\u00edm z hlediska architektury a m\u00e1 generovan\u00e1 testovac\u00ed data, kter\u00e1 objemov\u011b p\u0159ibli\u017en\u011b odpov\u00eddaj\u00ed produkci. Jednodu\u0161e se rozhodneme \"zab\u00edt\" jednoho z master\u016f PostgreSQL b\u011bhem testu a pod\u00edvat se, co se stane. Ale p\u0159ed t\u00edm je d\u016fle\u017eit\u00e9 zkontrolovat automatickou instalaci, proto\u017ee v tomto prost\u0159ed\u00ed m\u00e1me n\u011bkolik shard\u016f PostgreSQL, tak\u017ee z\u00edsk\u00e1me skv\u011bl\u00e9 testov\u00e1n\u00ed konfigura\u010dn\u00edch skript\u016f p\u0159ed produkc\u00ed.<\/p>\n<p>Oba \u00fakoly vypadaj\u00ed ambici\u00f3zn\u011b, ale m\u00e1me PostgreSQL 9.6. Mo\u017en\u00e1 se rovnou aktualizujeme na 11.2?<\/p>\n<p>Rozhodli jsme se to ud\u011blat ve dvou kroc\u00edch: nejprve aktualizovat verzi na 11.2, pak spustit Patroni.<\/p>\n<h3>Aktualizace PostgreSQL<\/h3>\n<p>Pro rychlou aktualizaci verze PostgreSQL je nutn\u00e9 pou\u017e\u00edt mo\u017enost <b>-k<\/b>, kter\u00e1 vytv\u00e1\u0159\u00ed hard linky na disku a nen\u00ed nutn\u00e9 kop\u00edrovat va\u0161e data. Na datab\u00e1z\u00edch o velikosti 300-400 GB aktualizace trv\u00e1 1 sekundu.<\/p>\n<p>M\u00e1me mnoho shard\u016f, tak\u017ee aktualizace mus\u00ed prob\u011bhnout automaticky. K tomu jsme napsali Ansible playbook, kter\u00fd prov\u00e1d\u00ed cel\u00fd proces aktualizace za n\u00e1s:<\/p>\n<pre><code class=\"plaintext\">\/usr\/lib\/postgresql\/11\/bin\/pg_upgrade \n&lt;b&gt;--odkaz &lt;\/b&gt;\n--star&aacute;-datadir=&#039;&#039; --nov&aacute;-datadir=&#039;&#039; \n --star&aacute;-bindir=&#039;&#039;  --nov&aacute;-bindir=&#039;&#039; \n --star&aacute;-options=&#039; -c config_file=&#039; \n --nov&eacute;-options=&#039; -c config_file=&#039;<\/code><\/pre>\n<p>Je d\u016fle\u017eit\u00e9 poznamenat, \u017ee p\u0159ed spu\u0161t\u011bn\u00edm upgradu je nutn\u00e9 jej prov\u00e9st s parametrem <b>\u2014zkontrolovat<\/b>, abychom si byli jist\u00ed mo\u017enost\u00ed upgradu. Tak\u00e9 n\u00e1\u0161 skript d\u011bl\u00e1 v\u00fdm\u011bnu konfigurac\u00ed na dobu upgradu. Skript se n\u00e1m provedl za 30 sekund, co\u017e je vynikaj\u00edc\u00ed v\u00fdsledek.<\/p>\n<h3>Spu\u0161t\u011bn\u00ed Patroni<\/h3>\n<p>Pro vy\u0159e\u0161en\u00ed druh\u00e9ho probl\u00e9mu sta\u010d\u00ed pohl\u00e9dnout na konfiguraci Patroni. V ofici\u00e1ln\u00edm repozit\u00e1\u0159i je p\u0159\u00edklad konfigurace s initdb, kter\u00e1 odpov\u00edd\u00e1 za inicializaci nov\u00e9 datab\u00e1ze p\u0159i prvn\u00edm spu\u0161t\u011bn\u00ed Patroni. Ale proto\u017ee m\u00e1me ji\u017e hotovou datab\u00e1zi, jednodu\u0161e jsme tuto \u010d\u00e1st z konfigurace odstranili.<\/p>\n<p>Kdy\u017e jsme za\u010dali instalovat Patroni na ji\u017e hotov\u00fd cluster PostgreSQL a spou\u0161t\u011bt ho, narazili jsme na nov\u00fd probl\u00e9m: oba servery se spou\u0161t\u011bly jako leader. Patroni nic nev\u00ed o p\u0159edchoz\u00edm stavu clusteru a pokou\u0161\u00ed se spustit oba servery jako dva samostatn\u00e9 clustery se stejn\u00fdm n\u00e1zvem. Pro vy\u0159e\u0161en\u00ed tohoto probl\u00e9mu je nutn\u00e9 smazat adres\u00e1\u0159 s daty na slave:<\/p>\n<pre><code class=\"plaintext\">rm -rf \/var\/lib\/postgresql\/<\/code><\/pre>\n<p><b>To je nutn\u00e9 ud\u011blat pouze na slave!<\/b><\/p>\n<p>P\u0159i p\u0159ipojen\u00ed \u010dist\u00e9 repliky Patroni d\u011bl\u00e1 basebackup leadera a obnovuje ho na repliku, a pak doh\u00e1n\u00ed aktu\u00e1ln\u00ed stav podle wal-log\u016f.<\/p>\n<p>Dal\u0161\u00ed probl\u00e9m, se kter\u00fdm jsme se pot\u00fdkali, je, \u017ee v\u0161echny PostgreSQL clustery jsou ve v\u00fdchoz\u00edm nastaven\u00ed pojmenov\u00e1ny main. Kdy\u017e ka\u017ed\u00fd cluster nev\u00ed nic o ostatn\u00edch, je to v po\u0159\u00e1dku. Ale kdy\u017e chcete pou\u017e\u00edvat Patroni, mus\u00ed m\u00edt v\u0161echny clustery unik\u00e1tn\u00ed jm\u00e9no. \u0158e\u0161en\u00edm je zm\u011bnit n\u00e1zev clusteru v konfiguraci PostgreSQL.<\/p>\n<h3>Z\u00e1t\u011b\u017eov\u00fd test<\/h3>\n<p>Spustili jsme test, kter\u00fd simuluje pr\u00e1ci u\u017eivatel\u016f na f\u00f3rech. Kdy\u017e zat\u00ed\u017een\u00ed dos\u00e1hlo na\u0161eho pr\u016fm\u011brn\u00e9ho denn\u00edho v\u00fdkonu, zopakovali jsme p\u0159esn\u011b stejn\u00fd test, vypnuli jsme jeden instance s leaderem PostgreSQL. Automatick\u00fd failover fungoval tak, jak jsme o\u010dek\u00e1vali: Patroni zm\u011bnil l\u00eddra, Consul-template aktualizoval konfiguraci PgBouncer a poslal p\u0159\u00edkaz k reloadu. Na na\u0161ich grafech v Grafan\u011b bylo vid\u011bt, \u017ee do\u0161lo k prodlev\u00e1m 20-30 sekund a k mal\u00e9mu objemu chyb ze server\u016f spojen\u00fdch s p\u0159ipojen\u00edm k datab\u00e1zi. To je norm\u00e1ln\u00ed situace, takov\u00e9 hodnoty jsou p\u0159\u00edpustn\u00e9 pro n\u00e1\u0161 failover a ur\u010dit\u011b lep\u0161\u00ed ne\u017e downtime slu\u017eby.<\/p>\n<h2>Nasazen\u00ed Patroni na produkci<\/h2>\n<p>Nakonec jsme dostali n\u00e1sleduj\u00edc\u00ed pl\u00e1n:<\/p>\n<ul>\n<li>Nasazen\u00ed Consul-template na servery PgBouncer a spu\u0161t\u011bn\u00ed;<\/li>\n<li>Aktualizace PostgreSQL na verzi 11.2;<\/li>\n<li>Zm\u011bna n\u00e1zvu clusteru;<\/li>\n<li>Spu\u0161t\u011bn\u00ed clusteru Patroni.<\/li>\n<\/ul>\n<p>Na\u0161e sch\u00e9ma n\u00e1m v\u0161ak umo\u017e\u0148uje prov\u00e9st prvn\u00ed bod prakticky kdykoli, m\u016f\u017eeme postupn\u011b odstranit ka\u017ed\u00fd PgBouncer z provozu a spustit na n\u011bm nasazen\u00ed a spu\u0161t\u011bn\u00ed consul-template. Tak jsme tak\u00e9 ud\u011blali.<\/p>\n<p>Pro rychl\u00e9 nasazen\u00ed jsme pou\u017eili Ansible, proto\u017ee jsme v\u0161echny playbooky ji\u017e otestovali v testovac\u00edm prost\u0159ed\u00ed a doba proveden\u00ed cel\u00e9ho sc\u00e9n\u00e1\u0159e byla od 1,5 do 2 minut pro ka\u017edou shardu. M\u011bli jsme mo\u017enost v\u0161e nasadit postupn\u011b na ka\u017ed\u00fd shard bez zastaven\u00ed na\u0161\u00ed slu\u017eby, ale museli bychom na n\u011bkolik minut vypnout ka\u017ed\u00fd PostgreSQL. V tomto p\u0159\u00edpad\u011b by u\u017eivatel\u00e9, jejich\u017e data se nach\u00e1zej\u00ed na tomto shardu, nemohli v tu dobu pln\u011b pracovat, co\u017e je pro n\u00e1s nep\u0159ijateln\u00e9.<\/p>\n<p>\u0158e\u0161en\u00edm t\u00e9to situace byla pl\u00e1novan\u00e1 \u00fadr\u017eba, kter\u00e1 prob\u00edh\u00e1 ka\u017ed\u00e9 3 m\u011bs\u00edce. Toto je okno pro pl\u00e1novan\u00e9 pr\u00e1ce, kdy zcela vypneme na\u0161i slu\u017ebu a aktualizujeme instance datab\u00e1z\u00ed. Do dal\u0161\u00edho okna zb\u00fdval jeden t\u00fdden, a tak jsme se rozhodli jednodu\u0161e po\u010dkat a dal\u0161\u00ed se p\u0159ipravit. B\u011bhem doby \u010dek\u00e1n\u00ed jsme ud\u011blali n\u011bkolik pojistn\u00fdch krok\u016f: pro ka\u017ed\u00e9 sharding PostgreSQL jsme zvedli z\u00e1lo\u017en\u00ed repliky pro p\u0159\u00edpad ne\u00fasp\u011bchu, abychom uchovali nejnov\u011bj\u0161\u00ed data, a p\u0159idali novou instanci pro ka\u017ed\u00e9 sharding, kter\u00e1 by m\u011bla st\u00e1t novou replikou v clusteru Patroni, abychom nem\u011bli prov\u00e1d\u011bt p\u0159\u00edkaz pro odstran\u011bn\u00ed dat. To v\u0161e pomohlo maxim\u00e1ln\u011b sn\u00ed\u017eit riziko chyby.<br \/>\n<img decoding=\"async\" style=\"display: block; margin: 0 auto;\" src=\"\/wp-content\/uploads\/2019\/06\/29e5c5f50dbfebfe5665fa0aeb009728.png\" alt=\"Odoln\u00fd cluster PostgreSQL + Patroni. Zku\u0161enosti s implementac\u00ed\" \/><\/p>\n<p>Restartovali jsme na\u0161i slu\u017ebu, v\u0161echno fungovalo jak m\u011blo, u\u017eivatel\u00e9 pokra\u010dovali v pr\u00e1ci, ale na grafech jsme zaznamenali abnorm\u00e1ln\u011b vysokou z\u00e1t\u011b\u017e na serverech Consul.<br \/>\n<img decoding=\"async\" style=\"display: block; margin: 0 auto;\" src=\"\/wp-content\/uploads\/2019\/06\/af0eef9ac235706c0aac62d2e1ee179d.png\" alt=\"Odoln\u00fd cluster PostgreSQL + Patroni. Zku\u0161enosti s implementac\u00ed\" \/><\/p>\n<p>Pro\u010d jsme to nevid\u011bli v testovac\u00edm prost\u0159ed\u00ed? Tento probl\u00e9m velmi dob\u0159e ilustruje, \u017ee je t\u0159eba se \u0159\u00eddit principem Infrastructure as code a vylep\u0161it celou infrastrukturu, po\u010d\u00ednaje testovac\u00edmi prost\u0159ed\u00edmi a kon\u010de produkc\u00ed. Jinak je velmi snadn\u00e9 dostat se k takov\u00e9mu probl\u00e9mu, jak\u00fd jsme m\u011bli my. Co se stalo? Consul nejprve vznikl na produkci a pot\u00e9 na testovac\u00edch prost\u0159ed\u00edch, tak\u017ee na testovac\u00edch prost\u0159ed\u00edch byla verze Consul vy\u0161\u0161\u00ed ne\u017e na produkci. P\u0159esn\u011b v jednom z release\u016f byla vy\u0159e\u0161ena \u00fanik CPU p\u0159i pr\u00e1ci s consul-template. Tak\u017ee jsme jednodu\u0161e aktualizovali Consul a t\u00edmto zp\u016fsobem vy\u0159e\u0161ili probl\u00e9m.<\/p>\n<h3>Restartovat cluster Patroni<\/h3>\n<p>Nicm\u00e9n\u011b jsme dostali nov\u00fd probl\u00e9m, o kter\u00e9m jsme ani nep\u0159edpokl\u00e1dali. P\u0159i aktualizaci Consulu jednodu\u0161e odstran\u00edme uzel Consul z clustru pomoc\u00ed p\u0159\u00edkazu consul leave \u2192 Patroni se p\u0159ipoj\u00ed k jin\u00e9mu serveru Consul \u2192 v\u0161e funguje. Ale kdy\u017e jsme se dostali k posledn\u00ed instanci clustru Consul a poslali mu p\u0159\u00edkaz consul leave, v\u0161echny clustry Patroni se prost\u011b restartovaly, a v logu jsme vid\u011bli n\u00e1sleduj\u00edc\u00ed chybu:<\/p>\n<pre><code class=\"plaintext\">CHYBA: get_cluster\nSledov&aacute;n&iacute; (posledn&iacute; vol&aacute;n&iacute; naposledy):\n...\nRetryFailedError: &#039;P\u0159ekro\u010dena doba pokusu&#039;\nCHYBA: Chyba p\u0159i komunikaci s DCS\n&lt;b&gt;LOG: syst&eacute;m datab&aacute;z&iacute; byl vypnut&lt;\/b&gt;<\/code><\/pre>\n<p>Cluster Patroni nebyl schopen z\u00edskat informace o sv\u00e9m clusteru a restartoval se.<\/p>\n<p>Pro hled\u00e1n\u00ed \u0159e\u0161en\u00ed jsme se obr\u00e1tili na autory Patroni prost\u0159ednictv\u00edm issue na githubu. Navrhli vylep\u0161en\u00ed na\u0161ich konfigura\u010dn\u00edch soubor\u016f:<\/p>\n<pre><code class=\"python\">consul:\n consul.checks: []\nbootstrap:\n dcs:\n   retry_timeout: 8<\/code><\/pre>\n<p>Poda\u0159ilo se n\u00e1m probl\u00e9m opakovat v testovac\u00edm prost\u0159ed\u00ed a otestovali jsme tam tyto parametry, ale bohu\u017eel nefungovaly.<\/p>\n<p>Probl\u00e9m st\u00e1le z\u016fst\u00e1v\u00e1 nevy\u0159e\u0161en\u00fd. Pl\u00e1nujeme vyzkou\u0161et n\u00e1sleduj\u00edc\u00ed mo\u017enosti \u0159e\u0161en\u00ed:<\/p>\n<ul>\n<li>Pou\u017e\u00edt agent Consul na ka\u017ed\u00e9m instance clusteru Patroni;<\/li>\n<li>Opravit probl\u00e9m v k\u00f3du.<\/li>\n<\/ul>\n<p>Ch\u00e1peme m\u00edsto vzniku chyby: pravd\u011bpodobn\u011b je probl\u00e9m v pou\u017eit\u00ed v\u00fdchoz\u00edho \u010dasov\u00e9ho limitu, kter\u00fd nen\u00ed p\u0159edefinov\u00e1n v konfigura\u010dn\u00edm souboru. P\u0159i odstran\u011bn\u00ed posledn\u00edho serveru Consul z clusteru dojde k zamrznut\u00ed cel\u00e9ho clusteru Consul, kter\u00e9 trv\u00e1 d\u00e9le ne\u017e sekundu, kv\u016fli \u010demu\u017e Patroni nem\u016f\u017ee z\u00edskat stav clusteru a kompletn\u011b restartuje cel\u00fd cluster.<\/p>\n<p>Na\u0161t\u011bst\u00ed jsme se nesetkali s \u017e\u00e1dn\u00fdmi dal\u0161\u00edmi chybami.<\/p>\n<h2>Shrnut\u00ed pou\u017eit\u00ed Patroni<\/h2>\n<p>Po \u00fasp\u011b\u0161n\u00e9m spu\u0161t\u011bn\u00ed Patroni jsme p\u0159idali po jedn\u00e9 dal\u0161\u00ed replikaci do ka\u017ed\u00e9ho clusteru. Nyn\u00ed m\u00e1 ka\u017ed\u00fd cluster podobu kv\u00f3ra: jeden v\u016fdce a dv\u011b repliky, - pro zaji\u0161t\u011bn\u00ed v p\u0159\u00edpad\u011b split-brain p\u0159i p\u0159ep\u00edn\u00e1n\u00ed.<br \/>\n<img decoding=\"async\" style=\"display: block; margin: 0 auto;\" src=\"\/wp-content\/uploads\/2019\/06\/3bb4c0495fea274b04edd2e99b59eacb.png\" alt=\"Odoln\u00fd cluster PostgreSQL + Patroni. Zku\u0161enosti s implementac\u00ed\" \/><\/p>\n<p>Na produkci Patroni funguje v\u00edce ne\u017e t\u0159i m\u011bs\u00edce. B\u011bhem t\u00e9to doby n\u00e1s u\u017e n\u011bkolikr\u00e1t zachr\u00e1nil. Ned\u00e1vno v AWS selhal v\u016fdce jednoho z cluster\u016f, automatick\u00fd failover se spustil a u\u017eivatel\u00e9 pokra\u010dovali v pr\u00e1ci. Patroni splnil sv\u016fj hlavn\u00ed \u00fakol.<\/p>\n<p><b>Kr\u00e1tk\u00e9 shrnut\u00ed pou\u017eit\u00ed Patroni:<\/b><\/p>\n<ul>\n<li>Pohodl\u00ed p\u0159i zm\u011bn\u011b konfigurace. Sta\u010d\u00ed zm\u011bnit konfiguraci na jednom instance a ta se automaticky pou\u017eije na cel\u00fd cluster. Pokud je t\u0159eba restartovat, aby se nov\u00e1 konfigurace projevila, Patroni o tom informuje. Patroni m\u016f\u017ee restartovat cel\u00fd cluster jedin\u00fdm p\u0159\u00edkazem, co\u017e je tak\u00e9 velmi pohodln\u00e9.<\/li>\n<li>Automatick\u00fd failover funguje a u\u017e n\u00e1s n\u011bkolikr\u00e1t zachr\u00e1nil.<\/li>\n<li>Aktualizace PostgreSQL bez v\u00fdpadku aplikace. Nejprve je nutn\u00e9 aktualizovat repliky na novou verzi, pot\u00e9 zm\u011bnit v\u016fdce v clusteru Patroni a aktualizovat star\u00e9ho v\u016fdce. P\u0159i tom se prov\u00e1d\u00ed nezbytn\u00e9 testov\u00e1n\u00ed automatick\u00e9ho failoveru.<\/li>\n<\/ul>\n<p>Zdroj: <a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/miro\/blog\/457326\/\">habr.com<\/a><\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u0412 \u0441\u0442\u0430\u0442\u044c\u0435 \u044f \u0440\u0430\u0441\u0441\u043a\u0430\u0436\u0443, \u043a\u0430\u043a \u043c\u044b \u043f\u043e\u0434\u043e\u0448\u043b\u0438 \u043a \u0432\u043e\u043f\u0440\u043e\u0441\u0443 \u043e\u0442\u043a\u0430\u0437\u043e\u0443\u0441\u0442\u043e\u0439\u0447\u0438\u0432\u043e\u0441\u0442\u0438 PostgreSQL, \u043f\u043e\u0447\u0435\u043c\u0443 \u044d\u0442\u043e \u0441\u0442\u0430\u043b\u043e \u0434\u043b\u044f \u043d\u0430\u0441 \u0432\u0430\u0436\u043d\u043e \u0438 \u0447\u0442\u043e \u0432 \u0438\u0442\u043e\u0433\u0435 \u043f\u043e\u043b\u0443\u0447\u0438\u043b\u043e\u0441\u044c. \u0423 \u043d\u0430\u0441 \u0432\u044b\u0441\u043e\u043a\u043e\u043d\u0430\u0433\u0440\u0443\u0436\u0435\u043d\u043d\u044b\u0439 \u0441\u0435\u0440\u0432\u0438\u0441: 2,5 \u043c\u043b\u043d \u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u0442\u0435\u043b\u0435\u0439 \u043f\u043e \u0432\u0441\u0435\u043c\u0443 \u043c\u0438\u0440\u0443, 50\u041a+ \u0430\u043a\u0442\u0438\u0432\u043d\u044b\u0445 \u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u0442\u0435\u043b\u0435\u0439 \u043a\u0430\u0436\u0434\u044b\u0439 \u0434\u0435\u043d\u044c. \u0421\u0435\u0440\u0432\u0435\u0440\u0430 \u043d\u0430\u0445\u043e\u0434\u044f\u0442\u0441\u044f \u0432 Amazone \u0432 \u043e\u0434\u043d\u043e\u043c \u0440\u0435\u0433\u0438\u043e\u043d\u0435 \u0418\u0440\u043b\u0430\u043d\u0434\u0438\u0438: \u0432 \u0440\u0430\u0431\u043e\u0442\u0435 \u043f\u043e\u0441\u0442\u043e\u044f\u043d\u043d\u043e 100+ \u0440\u0430\u0437\u043b\u0438\u0447\u043d\u044b\u0445 \u0441\u0435\u0440\u0432\u0435\u0440\u043e\u0432, \u0438\u0437 \u043d\u0438\u0445 \u043f\u043e\u0447\u0442\u0438 50 [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":26749,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-35706","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.3 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u0412 \u0441\u0442\u0430\u0442\u044c\u0435 \u044f \u0440\u0430\u0441\u0441\u043a\u0430\u0436\u0443, \u043a\u0430\u043a \u043c\u044b \u043f\u043e\u0434\u043e\u0448\u043b\u0438 \u043a \u0432\u043e\u043f\u0440\u043e\u0441\u0443 \u043e\u0442\u043a\u0430\u0437\u043e\u0443\u0441\u0442\u043e\u0439\u0447\u0438\u0432\u043e\u0441\u0442\u0438 PostgreSQL, \u043f\u043e\u0447\u0435\u043c\u0443 \u044d\u0442\u043e \u0441\u0442\u0430\u043b\u043e \u0434\u043b\u044f \u043d\u0430\u0441 \u0432\u0430\u0436\u043d\u043e \u0438 \u0447\u0442\u043e \u0432 \u0438\u0442\u043e\u0433\u0435 \u043f\u043e\u043b\u0443\u0447\u0438\u043b\u043e\u0441\u044c.\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/otkazoustojchivyj-klaster-postgresql-patroni-opyt-vnedreniya\" \/>\n\t\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.3\" \/>\n\t\t<meta property=\"og:locale\" content=\"cs_CZ\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47\u041e\u0442\u043a\u0430\u0437\u043e\u0443\u0441\u0442\u043e\u0439\u0447\u0438\u0432\u044b\u0439 \u043a\u043b\u0430\u0441\u0442\u0435\u0440 PostgreSQL + Patroni. \u041e\u043f\u044b\u0442 \u0432\u043d\u0435\u0434\u0440\u0435\u043d\u0438\u044f | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u0412 \u0441\u0442\u0430\u0442\u044c\u0435 \u044f \u0440\u0430\u0441\u0441\u043a\u0430\u0436\u0443, \u043a\u0430\u043a \u043c\u044b \u043f\u043e\u0434\u043e\u0448\u043b\u0438 \u043a \u0432\u043e\u043f\u0440\u043e\u0441\u0443 \u043e\u0442\u043a\u0430\u0437\u043e\u0443\u0441\u0442\u043e\u0439\u0447\u0438\u0432\u043e\u0441\u0442\u0438 PostgreSQL, \u043f\u043e\u0447\u0435\u043c\u0443 \u044d\u0442\u043e \u0441\u0442\u0430\u043b\u043e \u0434\u043b\u044f \u043d\u0430\u0441 \u0432\u0430\u0436\u043d\u043e \u0438 \u0447\u0442\u043e \u0432 \u0438\u0442\u043e\u0433\u0435 \u043f\u043e\u043b\u0443\u0447\u0438\u043b\u043e\u0441\u044c.\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/otkazoustojchivyj-klaster-postgresql-patroni-opyt-vnedreniya\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2019-10-31T19:05:50+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2026-05-18T18:58:46+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47Vysoce dostupn\u00fd cluster PostgreSQL + Patroni. Zku\u0161enosti z implementace | ProHoster","description":"V \u010dl\u00e1nku pop\u00ed\u0161u, jak jsme p\u0159istoupili k ot\u00e1zce odolnosti PostgreSQL, pro\u010d se to stalo d\u016fle\u017eit\u00e9 a co jsme nakonec dos\u00e1hli.","canonical_url":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/otkazoustojchivyj-klaster-postgresql-patroni-opyt-vnedreniya","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"cs_CZ","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u041e\u0442\u043a\u0430\u0437\u043e\u0443\u0441\u0442\u043e\u0439\u0447\u0438\u0432\u044b\u0439 \u043a\u043b\u0430\u0441\u0442\u0435\u0440 PostgreSQL + Patroni. \u041e\u043f\u044b\u0442 \u0432\u043d\u0435\u0434\u0440\u0435\u043d\u0438\u044f | ProHoster","og:description":"\u0412 \u0441\u0442\u0430\u0442\u044c\u0435 \u044f \u0440\u0430\u0441\u0441\u043a\u0430\u0436\u0443, \u043a\u0430\u043a \u043c\u044b \u043f\u043e\u0434\u043e\u0448\u043b\u0438 \u043a \u0432\u043e\u043f\u0440\u043e\u0441\u0443 \u043e\u0442\u043a\u0430\u0437\u043e\u0443\u0441\u0442\u043e\u0439\u0447\u0438\u0432\u043e\u0441\u0442\u0438 PostgreSQL, \u043f\u043e\u0447\u0435\u043c\u0443 \u044d\u0442\u043e \u0441\u0442\u0430\u043b\u043e \u0434\u043b\u044f \u043d\u0430\u0441 \u0432\u0430\u0436\u043d\u043e \u0438 \u0447\u0442\u043e \u0432 \u0438\u0442\u043e\u0433\u0435 \u043f\u043e\u043b\u0443\u0447\u0438\u043b\u043e\u0441\u044c.","og:url":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/otkazoustojchivyj-klaster-postgresql-patroni-opyt-vnedreniya","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2019-10-31T19:05:50+00:00","article:modified_time":"2026-05-18T18:58:46+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"35706","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":"2026-01-22 00:26:19","breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 13:51:06","updated":"2026-01-22 00:26:19","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts\/35706","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/comments?post=35706"}],"version-history":[{"count":1,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts\/35706\/revisions"}],"predecessor-version":[{"id":172654,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts\/35706\/revisions\/172654"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/media\/26749"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/media?parent=35706"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/categories?post=35706"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/tags?post=35706"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}