{"id":41051,"date":"2020-02-05T21:28:58","date_gmt":"2020-02-05T18:28:58","guid":{"rendered":"https:\/\/prohoster.info\/blog\/blog_prohoster\/consul-iptables-3"},"modified":"2020-02-05T21:28:58","modified_gmt":"2020-02-05T18:28:58","slug":"consul-iptables-3","status":"publish","type":"post","link":"https:\/\/prohoster.info\/cs\/blog\/consul-iptables-3","title":{"rendered":"Consul + iptables = :3","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p>V roce 2010 m\u011bla firma <noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/wargaming\/\">Wargaming<\/a><\/noindex> 50 server\u016f a jednoduch\u00fd s\u00ed\u0165ov\u00fd model: backend, frontend a firewall. Po\u010det server\u016f rostl, model se komplikoval: stagingy, izolovan\u00e9 VLAN s ACL, pot\u00e9 VPN s VRF, VLAN s ACL na L2, VRF s ACL na L3. To je trochu matouc\u00ed? D\u00e1l to bude z\u00e1bavn\u011bj\u0161\u00ed.<\/p>\n<p>Kdy\u017e vzrostl po\u010det server\u016f na 16 000, pr\u00e1ce s takov\u00fdm mno\u017estv\u00edm r\u016fznorod\u00fdch segment\u016f se stala nemo\u017enou. Proto jsme p\u0159i\u0161li s jin\u00fdm \u0159e\u0161en\u00edm. Vzali jsme stack Netfilter, p\u0159idali k n\u011bmu Consul jako zdroj dat, a vznikl rychl\u00fd distribuovan\u00fd firewall. Ten nahradil ACL na routerech a slou\u017eil jako extern\u00ed i intern\u00ed firewall. Pro dynamick\u00e9 \u0159\u00edzen\u00ed n\u00e1stroje jsme vyvinuli syst\u00e9m BEFW, kter\u00fd jsme pou\u017eili v\u0161ude: od \u0159\u00edzen\u00ed p\u0159\u00edstupu u\u017eivatel\u016f do produktov\u00e9 s\u00edt\u011b po izolaci s\u00ed\u0165ov\u00fdch segment\u016f od sebe.<\/p>\n<p><img decoding=\"async\" alt=\"Consul + iptables = :3\" src=\"\/wp-content\/uploads\/2020\/02\/246e74278f413749a676599b5ce986a5.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nJak to v\u0161echno funguje a pro\u010d byste m\u011bli v\u011bnovat pozornost tomuto syst\u00e9mu, v\u00e1m pov\u00ed <strong>Ivan Agarov<\/strong> (<noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/users\/annmuor\/\" class=\"user_link\">annmuor<\/a><\/noindex>) \u2014 vedouc\u00ed skupiny zabezpe\u010den\u00ed infrastruktury odd\u011blen\u00ed Maintenance v Minsk\u00e9 v\u00fdvojov\u00e9 centr\u00e1le firmy. Ivan je fanou\u0161ek SELinuxu, miluje Perl a p\u00ed\u0161e k\u00f3d. Jako vedouc\u00ed skupiny I\u017d pravideln\u011b pracuje s logy, z\u00e1lohami a v\u00fdzkumem a v\u00fdvojem, aby chr\u00e1nil Wargaming p\u0159ed hackery a zajistil provoz v\u0161ech hern\u00edch server\u016f ve firm\u011b.<br \/>\n<noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><br \/>\n<center><div class=\"youtube-placeholder\" data-id=\"4zP67uYnsR4\" onclick=\"loadVideo(this)\">\r\n        <img decoding=\"async\" src=\"https:\/\/img.youtube.com\/vi\/4zP67uYnsR4\/hqdefault.jpg\" alt=\"P\u0159ehr\u00e1t video\" loading=\"lazy\" width=\"480\" height=\"360\" style=\"width:100%;height:auto;\">\r\n        <div class=\"play-button\"><\/div>\r\n    <\/div><\/center><\/p>\n<h2>Historie<\/h2>\n<p>\nNe\u017e pov\u00edm, jak jsme to d\u011blali, pov\u00edm, jak jsme se k tomu dostali a pro\u010d to bylo pot\u0159eba. K tomu se p\u0159eneseme o 9 let zp\u011bt: rok 2010, pr\u00e1v\u011b se objevily World of Tanks. Firma Wargaming m\u011bla asi 50 server\u016f.<\/p>\n<p><img decoding=\"async\" alt=\"Consul + iptables = :3\" src=\"\/wp-content\/uploads\/2020\/02\/64e1c2839363dbdf59ba5f11bb13530e.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<em>Graf r\u016fstu server\u016f firmy.<\/em><\/p>\n<p>M\u011bli jsme s\u00ed\u0165ov\u00fd model. Pro tu dobu byl optim\u00e1ln\u00ed.<\/p>\n<p><img decoding=\"async\" alt=\"Consul + iptables = :3\" src=\"\/wp-content\/uploads\/2020\/02\/c0509324e2a83882a9792ba75612ceea.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<em>S\u00ed\u0165ov\u00fd model v roce 2010.<\/em><\/p>\n<p>Na frontendu \u017eij\u00ed \u0161patn\u00ed kluci, kte\u0159\u00ed n\u00e1s cht\u011bj\u00ed zni\u010dit, ale tam je firewall. Na backendu firewall nen\u00ed, ale m\u00e1me tam 50 server\u016f, v\u0161echny je zn\u00e1me. V\u0161echno funguje dob\u0159e.<\/p>\n<p>Za 4 roky vzrostl park server\u016f sto n\u00e1sobn\u011b, na 5000. Objevily se prvn\u00ed izolovan\u00e9 s\u00edt\u011b \u2014 stagingy: nemohou j\u00edt do produkce, a tam \u010dasto b\u011b\u017eelo n\u011bco, co mohlo b\u00fdt nebezpe\u010dn\u00e9.<\/p>\n<p><img decoding=\"async\" alt=\"Consul + iptables = :3\" src=\"\/wp-content\/uploads\/2020\/02\/08e1d7670ce1a1346e49d87c67c4be1b.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<em>S\u00ed\u0165ov\u00fd model v roce 2014.<\/em><\/p>\n<p>Z setrva\u010dnosti jsme pou\u017e\u00edvali st\u00e1le stejn\u00e1 za\u0159\u00edzen\u00ed a cel\u00e1 pr\u00e1ce prob\u00edhala na izolovan\u00fdch VLAN: na VLAN se zapisuj\u00ed ACL, kter\u00e9 povoluj\u00ed nebo zakazuj\u00ed ur\u010dit\u00e9 spojen\u00ed.<\/p>\n<p>V roce 2016 po\u010det server\u016f dos\u00e1hl 8000. Wargaming p\u0159evzal dal\u0161\u00ed studia a vznikly dal\u0161\u00ed partnersk\u00e9 s\u00edt\u011b. Vypadaj\u00ed, \u017ee jsou na\u0161e, ale nejsou \u00fapln\u011b: pro partnery VLAN \u010dasto nefunguje, je pot\u0159eba pou\u017e\u00edt VPN s VRF, izolace se komplikuj\u00ed. Sm\u011bs izolac\u00ed ACL rostla.<\/p>\n<p><img decoding=\"async\" alt=\"Consul + iptables = :3\" src=\"\/wp-content\/uploads\/2020\/02\/a6580c9551730447e16d7881eb102275.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<em>S\u00ed\u0165ov\u00fd model v roce 2016.<\/em><\/p>\n<p>Na za\u010d\u00e1tku roku 2018 se po\u010det stroj\u016f zv\u00fd\u0161il na 16 000. Segment\u016f bylo 6, ostatn\u00ed jsme nepo\u010d\u00edtali, v\u010detn\u011b uzav\u0159en\u00fdch, kde byly uchov\u00e1v\u00e1ny finan\u010dn\u00ed \u00fadaje. Objevily se kontejnerov\u00e9 s\u00edt\u011b (Kubernetes), DevOps, cloudov\u00e9 s\u00edt\u011b p\u0159ipojen\u00e9 p\u0159es VPN, nap\u0159\u00edklad z IVS. Pravidel bylo p\u0159esp\u0159\u00edli\u0161 \u2014 bylo to bolestiv\u00e9.<\/p>\n<p><img decoding=\"async\" alt=\"Consul + iptables = :3\" src=\"\/wp-content\/uploads\/2020\/02\/ec44e6a60da98daa4bbf2bca2e3b2e1a.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<em>S\u00ed\u0165ov\u00fd model a metody izolace v roce 2018.<\/em><\/p>\n<p>Pro izolaci jsme pou\u017e\u00edvali: VLAN s ACL na L2, VRF s ACL na L3, VPN a mnoho dal\u0161\u00edho. P\u0159\u00edli\u0161 mnoho.<\/p>\n<h2>Probl\u00e9my<\/h2>\n<p>\nV\u0161ichni \u017eij\u00ed s ACL a VLAN. Co je vlastn\u011b \u0161patn\u011b? Na tuto ot\u00e1zku odpov\u00ed Harold, kter\u00fd skr\u00fdv\u00e1 bolest.<\/p>\n<p><img decoding=\"async\" alt=\"Consul + iptables = :3\" src=\"\/wp-content\/uploads\/2020\/02\/4df5105af3bce515d80e8e5f3e14c027.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nProbl\u00e9m\u016f bylo mnoho, ale masivn\u00edch \u2014 p\u011bt.<\/p>\n<ul>\n<li><strong>Geometrick\u00fd n\u00e1r\u016fst ceny pro nov\u00e1 pravidla.<\/strong>. Ka\u017ed\u00e9 nov\u00e9 pravidlo se zav\u00e1d\u011blo d\u00e9le ne\u017e to p\u0159edchoz\u00ed, proto\u017ee bylo t\u0159eba nejprve zkontrolovat, zda u\u017e takov\u00e9 pravidlo neexistuje.<\/li>\n<li><strong>Nen\u00ed firewall uvnit\u0159 segment\u016f.<\/strong>. Segmenty se n\u011bjak odd\u011blily, uvnit\u0159 u\u017e zdroj\u016f nen\u00ed dostatek.<\/li>\n<li><strong>Pravidla se aplikovala dlouho. <\/strong>Jedno m\u00edstn\u00ed pravidlo mohli oper\u00e1to\u0159i napsat za hodinu. Glob\u00e1ln\u00ed trvalo n\u011bkolik dn\u00ed.<\/li>\n<li><strong>Obt\u00ed\u017enosti s auditem pravidel.<\/strong>. P\u0159esn\u011bji, audit nebyl mo\u017en\u00fd. Prvn\u00ed pravidla byla ps\u00e1na u\u017e v roce 2010 a v\u011bt\u0161ina jejich autor\u016f ji\u017e v spole\u010dnosti nepracovala.<\/li>\n<li><strong>N\u00edzk\u00e1 \u00farove\u0148 kontroly nad infrastrukturou.<\/strong>. To je hlavn\u00ed probl\u00e9m \u2014 \u0161patn\u011b jsme v\u011bd\u011bli, co se u n\u00e1s vlastn\u011b d\u011bje.<\/li>\n<\/ul>\n<p>\nTak vypadal s\u00ed\u0165ov\u00fd in\u017een\u00fdr v roce 2018, kdy\u017e sly\u0161el: \u201ePot\u0159ebujeme je\u0161t\u011b trochu ACL.\u201c<\/p>\n<p><img decoding=\"async\" alt=\"Consul + iptables = :3\" src=\"\/wp-content\/uploads\/2020\/02\/8366d3092558ebb15a4ba6d8e5702032.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<h2>\u0158e\u0161en\u00ed.<\/h2>\n<p>\nNa za\u010d\u00e1tku roku 2018 bylo rozhodnuto s t\u00edm n\u011bco ud\u011blat.<\/p>\n<p><strong>Cena integrac\u00ed neust\u00e1le roste. <\/strong>V\u00fdchoz\u00edm bodem bylo to, \u017ee velk\u00e9 datov\u00e9 centra p\u0159estala podporovat izolovan\u00e9 VLAN a ACL, proto\u017ee do\u0161ly pam\u011bti na za\u0159\u00edzen\u00edch.<\/p>\n<p>\u0158e\u0161en\u00ed: odstranili jsme lidsk\u00fd faktor a co nejv\u00edce automatizovali poskytov\u00e1n\u00ed p\u0159\u00edstupu.<\/p>\n<p><strong>Nov\u00e1 pravidla se aplikuj\u00ed dlouho. <\/strong>\u0158e\u0161en\u00ed: urychlit aplikaci pravidel, ud\u011blat ji distribuovanou a paraleln\u00ed. K tomu je pot\u0159eba distribuovan\u00fd syst\u00e9m, aby se pravidla doru\u010dovala samy, bez rsync nebo SFTP na tis\u00edc syst\u00e9m\u016f.<\/p>\n<p><strong>Nedostatek firewallu uvnit\u0159 segment\u016f. <\/strong>Firewall uvnit\u0159 segment\u016f za\u010dal b\u00fdt probl\u00e9m, kdy\u017e se v r\u00e1mci jedn\u00e9 s\u00edt\u011b objevily r\u016fzn\u00e9 slu\u017eby. \u0158e\u0161en\u00ed: pou\u017e\u00edvat firewall na \u00farovni hostitele \u2013 host-based firewalls. Prakticky v\u0161ude m\u00e1me Linux, a v\u0161ude je k dispozici iptables, to nen\u00ed probl\u00e9m.<\/p>\n<p><strong>Obt\u00ed\u017ee s auditem pravidel.<\/strong> \u0158e\u0161en\u00ed: uchov\u00e1vat v\u0161echna pravidla na jednom m\u00edst\u011b pro p\u0159ehled a spr\u00e1vu, takto budeme moci prov\u00e9st audit.<\/p>\n<p><strong>N\u00edzk\u00e1 \u00farove\u0148 kontroly nad infrastrukturou. <\/strong>\u0158e\u0161en\u00ed: prov\u00e9st inventarizaci v\u0161ech slu\u017eeb a p\u0159\u00edstup\u016f mezi nimi.<\/p>\n<p>Je to sp\u00ed\u0161e administrativn\u00ed proces ne\u017e technick\u00fd. N\u011bkdy m\u00e1me 200-300 nov\u00fdch release\u016f za t\u00fdden, zejm\u00e9na b\u011bhem akc\u00ed a o sv\u00e1tc\u00edch. A to je pouze pro jeden t\u00fdm na\u0161ich DevOps. S takov\u00fdm mno\u017estv\u00edm release\u016f je nemo\u017en\u00e9 pochopit, jak\u00e9 porty, IP adresy a integrace jsou pot\u0159eba. Proto jsme pot\u0159ebovali speci\u00e1ln\u011b vy\u0161kolen\u00e9 servisn\u00ed mana\u017eery, kte\u0159\u00ed se dotazovali t\u00fdm\u016f: \u201eCo v\u016fbec existuje a pro\u010d jste to zprovoznili?\u201c<\/p>\n<p>Po v\u0161em, co jsme spustili, v roce 2019 se s\u00ed\u0165ov\u00fd in\u017een\u00fdr ji\u017e za\u010dal jevit takto.<\/p>\n<p><img decoding=\"async\" alt=\"Consul + iptables = :3\" src=\"\/wp-content\/uploads\/2020\/02\/ccca25d7f7bb7f70638725addc6d99ed.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<\/p>\n<h2>Consul<\/h2>\n<p>\nRozhodli jsme se, \u017ee v\u0161e, co jsme na\u0161li pomoc\u00ed servisn\u00edch mana\u017eer\u016f, vlo\u017e\u00edme do Consulu a odtud budeme ps\u00e1t pravidla iptables.<\/p>\n<p>Jak jsme se rozhodli to ud\u011blat?<\/p>\n<ul>\n<li>Shrom\u00e1\u017ed\u00edme v\u0161echny slu\u017eby, s\u00edt\u011b a u\u017eivatele.<\/li>\n<li>Na jejich z\u00e1klad\u011b vytvo\u0159\u00edme pravidla iptables.<\/li>\n<li>Automatizujeme kontrolu.<\/li>\n<li>\u2026.<\/li>\n<li>ZISK.<\/li>\n<\/ul>\n<p>\nConsul nen\u00ed vzd\u00e1len\u00e9 API, m\u016f\u017ee fungovat na ka\u017ed\u00e9m uzlu a ps\u00e1t do iptables. Zb\u00fdv\u00e1 vymyslet automatick\u00e9 kontroln\u00ed prost\u0159edky, kter\u00e9 budou odstra\u0148ovat zbyte\u010dn\u00e9, a v\u011bt\u0161ina probl\u00e9m\u016f bude vy\u0159e\u0161ena! Zbytek se dolad\u00ed v pr\u016fb\u011bhu.<\/p>\n<h3>Pro\u010d Consul?<\/h3>\n<p>\n<strong>Osv\u011bd\u010dil se. <\/strong>V letech 2014-15 jsme ho pou\u017e\u00edvali jako backend pro Vault, kde uchov\u00e1v\u00e1me hesla.<\/p>\n<p><strong>Netrat\u00ed data.<\/strong>. B\u011bhem pou\u017e\u00edv\u00e1n\u00ed Consulu jsme neztratili data ani p\u0159i jedn\u00e9 hav\u00e1rii. To je obrovsk\u00e1 v\u00fdhoda pro syst\u00e9m spr\u00e1vy firewallu.<\/p>\n<p><strong>P2P spojen\u00ed zrychluj\u00ed distribuci zm\u011bn.<\/strong>. S P2P p\u0159ich\u00e1zej\u00ed v\u0161echny zm\u011bny rychle, nen\u00ed pot\u0159eba \u010dekat hodiny.<\/p>\n<p><strong>Pohodln\u00e9 REST API.<\/strong> Zva\u017eovali jsme tak\u00e9 Apache ZooKeeper, ale nem\u00e1 REST API, budeme muset ud\u011blat n\u011bjak\u00e9 workaroundy.<\/p>\n<p><strong>Funguje jak jako \u00falo\u017ei\u0161t\u011b kl\u00ed\u010d\u016f (KV), tak jako katalog (Service Discovery).<\/strong>. Lze ukl\u00e1dat slu\u017eby, katalogy, datov\u00e1 centra. To je pohodln\u00e9 nejen pro n\u00e1s, ale i pro sousedn\u00ed t\u00fdmy, proto\u017ee p\u0159i budov\u00e1n\u00ed glob\u00e1ln\u00ed slu\u017eby mysl\u00edme ve velk\u00e9m.<\/p>\n<p><strong>Naps\u00e1n v Go, kter\u00fd je sou\u010d\u00e1st\u00ed technologick\u00e9ho stacku Wargaming. <\/strong>M\u00e1me r\u00e1di tento jazyk, m\u00e1me mnoho v\u00fdvoj\u00e1\u0159\u016f Go.<\/p>\n<p><strong>Siln\u00fd syst\u00e9m ACL. <\/strong>S pomoc\u00ed ACL lze v Consul spravovat, kdo a co m\u016f\u017ee zapisovat. Zaru\u010dujeme, \u017ee pravidla firewallu se nebudou k\u0159\u00ed\u017eit s ni\u010d\u00edm jin\u00fdm a nebudeme s t\u00edm m\u00edt probl\u00e9my.<\/p>\n<p>Ale Consul m\u00e1 i nev\u00fdhody.<\/p>\n<ul>\n<li><strong>Neskalovateln\u00e9 v r\u00e1mci datov\u00e9ho centra, pokud nem\u00e1te firemn\u00ed verzi. <\/strong>\u0160kaluje se pouze federac\u00ed.<\/li>\n<li><strong>Velmi z\u00e1visl\u00e9 na kvalit\u011b s\u00edt\u011b a zat\u00ed\u017een\u00ed server\u016f. <\/strong>Consul nebude spr\u00e1vn\u011b fungovat jako server na zat\u00ed\u017een\u00e9m serveru, pokud v s\u00edti existuj\u00ed n\u011bjak\u00e9 zpo\u017ed\u011bn\u00ed, nap\u0159\u00edklad kol\u00edsav\u00e1 rychlost. To souvis\u00ed s P2P p\u0159ipojen\u00edmi a modely \u0161\u00ed\u0159en\u00ed aktualizac\u00ed.<\/li>\n<li><strong>Obt\u00ed\u017ee s monitorov\u00e1n\u00edm dostupnosti<\/strong>. Ve stavu Consul m\u016f\u017ee b\u00fdt uvedeno, \u017ee v\u0161e je v po\u0159\u00e1dku, ale on u\u017e d\u00e1vno nefunguje.<\/li>\n<\/ul>\n<p>\nV\u011bt\u0161inu t\u011bchto probl\u00e9m\u016f jsme vy\u0159e\u0161ili b\u011bhem pou\u017e\u00edv\u00e1n\u00ed Consul, proto jsme si ho vybrali. Spole\u010dnost m\u00e1 pl\u00e1ny na alternativn\u00ed backend, ale nau\u010dili jsme se bojovat s probl\u00e9my a zat\u00edm \u017eijeme s Consul.<\/p>\n<h3>Jak funguje Consul<\/h3>\n<p>\nV ur\u010dit\u00fdch datov\u00fdch centrech nainstalujeme servery - od t\u0159\u00ed do p\u011bti. Jeden nebo dva servery nejsou vhodn\u00e9: nejsou schopny organizovat kv\u00f3rum a rozhodnout, kdo m\u00e1 pravdu, kdy\u017e se data neshoduj\u00ed. V\u00edce ne\u017e p\u011bt nem\u00e1 smysl, v\u00fdkon klesne.<\/p>\n<p><img decoding=\"async\" alt=\"Consul + iptables = :3\" src=\"\/wp-content\/uploads\/2020\/02\/96153c4e9075b067b34a63ceba5b80f0.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nKlienti se p\u0159ipojuj\u00ed k server\u016fm v libovoln\u00e9m po\u0159ad\u00ed: ti sam\u00ed agenti, pouze s flagem <code>server = false<\/code>.<\/p>\n<p><img decoding=\"async\" alt=\"Consul + iptables = :3\" src=\"\/wp-content\/uploads\/2020\/02\/0854d1286c18bb5e7c1130c6734154a9.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nPot\u00e9 klienti obdr\u017e\u00ed seznam P2P p\u0159ipojen\u00ed a buduj\u00ed mezi sebou vazby.<\/p>\n<p><img decoding=\"async\" alt=\"Consul + iptables = :3\" src=\"\/wp-content\/uploads\/2020\/02\/2457cb7707c27cc23f6e13415119b91d.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nNa glob\u00e1ln\u00ed \u00farovni propojujeme n\u011bkolik datov\u00fdch center. Ta se tak\u00e9 propojuj\u00ed P2P a komunikuj\u00ed.<\/p>\n<p><img decoding=\"async\" alt=\"Consul + iptables = :3\" src=\"\/wp-content\/uploads\/2020\/02\/3b4ae922eb8d2a6a9a79425e769505d7.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nKdy\u017e chceme z\u00edskat data z jin\u00e9ho datov\u00e9ho centra, \u017e\u00e1dost jde od serveru k serveru. Tento zp\u016fsob se naz\u00fdv\u00e1 <strong>protokol Serf<\/strong>. Protokol Serf, stejn\u011b jako Consul, je v\u00fdvoj spole\u010dnosti HashiCorp.<\/p>\n<h3>N\u011bkolik d\u016fle\u017eit\u00fdch fakt\u016f o Consul<\/h3>\n<p>\nConsul m\u00e1 dokumentaci popisuj\u00edc\u00ed jeho fungov\u00e1n\u00ed. Uvedu pouze vybran\u00e9 fakta, kter\u00e1 stoj\u00ed za to v\u011bd\u011bt.<\/p>\n<p><strong>Consul servery vyb\u00edraj\u00ed master z hlasuj\u00edc\u00edch<\/strong>. Consul vyb\u00edr\u00e1 master ze seznamu server\u016f pro ka\u017ed\u00e9 datov\u00e9 centrum a v\u0161echny \u017e\u00e1dosti jdou pouze k n\u011bmu, bez ohledu na po\u010det server\u016f. Zaseknut\u00ed mastera nevede k nov\u00fdm volb\u00e1m. Pokud master nen\u00ed zvolen, \u017e\u00e1dosti nen\u00ed nikdo obsluhuj\u00ed.<\/p>\n<blockquote><p>Cht\u011bli jste horizont\u00e1ln\u00ed \u0161k\u00e1lov\u00e1n\u00ed? Omlouv\u00e1me se, nen\u00ed.<\/p><\/blockquote>\n<p> \u017d\u00e1dost do jin\u00e9ho datov\u00e9ho centra jde od mistra k mistrovi, bez ohledu na to, na kter\u00fd server p\u0159i\u0161la. Vybran\u00fd mistr p\u0159eb\u00edr\u00e1 100 % z\u00e1t\u011b\u017ee, krom\u011b z\u00e1t\u011b\u017ee na p\u0159esm\u011brov\u00e1n\u00ed \u017e\u00e1dost\u00ed. Aktu\u00e1ln\u00ed kopie dat jsou k dispozici na v\u0161ech serverech datov\u00e9ho centra, ale odpov\u00edd\u00e1 pouze jeden.<\/p>\n<blockquote><p>Jedin\u00fd zp\u016fsob, jak \u0161k\u00e1lovat, je aktivovat re\u017eim stale na klientovi.<\/p><\/blockquote>\n<p>V re\u017eimu stale lze odpov\u00eddat bez kv\u00f3ra. Je to re\u017eim, ve kter\u00e9m se vzd\u00e1v\u00e1me konzistence dat, ale \u010dten\u00ed je o n\u011bco rychlej\u0161\u00ed ne\u017e obvykle a m\u016f\u017ee odpov\u011bd\u011bt jak\u00fdkoli server. Samoz\u0159ejm\u011b, z\u00e1pis prob\u00edh\u00e1 pouze p\u0159es mistra.<\/p>\n<p><strong>Consul nekop\u00edruje data mezi datov\u00fdmi centry.<\/strong>P\u0159i shroma\u017e\u010fov\u00e1n\u00ed federace m\u00e1 ka\u017ed\u00fd server pouze sv\u00e1 data. Ostatn\u00ed informace z\u00edsk\u00e1v\u00e1 v\u017edy od n\u011bkoho jin\u00e9ho.<\/p>\n<p><strong>Atonomnost operac\u00ed nen\u00ed zaru\u010dena mimo transakci.<\/strong>Pamatujte, \u017ee n\u011bco m\u011bnit m\u016f\u017eete nejen vy. Pokud chcete jinak, prov\u00e1d\u011bjte transakci s blokov\u00e1n\u00edm.<\/p>\n<p><strong>Blokuj\u00edc\u00ed operace nezaru\u010duj\u00ed blokaci.<\/strong>\u017d\u00e1dost jde od mistra k mistrovi, a ne p\u0159\u00edmo, tak\u017ee nen\u00ed zaru\u010deno, \u017ee blokace usp\u011bje, pokud prov\u00e1d\u00edte blokaci nap\u0159\u00edklad v jin\u00e9m datov\u00e9m centru.<\/p>\n<p><strong>ACL tak\u00e9 nezaru\u010duje p\u0159\u00edstup (v mnoha p\u0159\u00edpadech).<\/strong>ACL nemus\u00ed fungovat, proto\u017ee se uchov\u00e1v\u00e1 v jednom datov\u00e9m centru federace \u2014 v ACL datov\u00e9m centru (Prim\u00e1rn\u00ed DC). Pokud v\u00e1m DC neodpov\u00ed, ACL nebude fungovat.<\/p>\n<p><strong>Jedna vis\u00edc\u00ed mistr zp\u016fsob\u00ed visen\u00ed cel\u00e9 federace.<\/strong>Nap\u0159\u00edklad v federaci 10 datov\u00fdch center, p\u0159i\u010dem\u017e jedno m\u00e1 \u0161patnou s\u00ed\u0165, a jeden mistr sel\u017ee. V\u0161ichni, kdo s n\u00edm komunikuj\u00ed, se zacykl\u00ed: doch\u00e1z\u00ed k \u017e\u00e1dosti, na kterou nen\u00ed odpov\u011b\u010f, vl\u00e1kno se vis\u00ed. Nen\u00ed mo\u017en\u00e9 zjistit, kdy k tomu dojde, prost\u011b za hodinu nebo dv\u011b spadne cel\u00e1 federace. S t\u00edm nic nenad\u011bl\u00e1te.<\/p>\n<p>Status, kv\u00f3rum a volby jsou zpracov\u00e1v\u00e1ny odd\u011blen\u00fdm vl\u00e1knem. Nedo\u0161lo k op\u011btovn\u00e9mu v\u00fdb\u011bru, status nic neuk\u00e1\u017ee. Mysl\u00edte si, \u017ee m\u00e1te \u017eiv\u00fd Consul, \u017e\u00e1d\u00e1te a nic se ned\u011bje \u2014 nen\u00ed \u017e\u00e1dn\u00e1 odpov\u011b\u010f. P\u0159itom status ukazuje, \u017ee je v\u0161e v po\u0159\u00e1dku.<\/p>\n<p>Setkali jsme se s t\u00edmto probl\u00e9mem, museli jsme p\u0159estavovat konkr\u00e9tn\u00ed \u010d\u00e1sti datov\u00fdch center, abychom se mu vyhnuli.<\/p>\n<p><strong>V podnikov\u00e9 verzi Consul Enterprise chyb\u00ed n\u011bkter\u00e9 v\u00fd\u0161e uveden\u00e9 \u043d\u0435\u0434\u043e\u0441\u0442\u0430\u0442\u043a\u0438.<\/strong>M\u00e1 mnoho u\u017eite\u010dn\u00fdch funkc\u00ed: volby hlasuj\u00edc\u00edch, rozd\u011blen\u00ed, \u0161k\u00e1lov\u00e1n\u00ed. Je tu ale jedno \"ale\" \u2014 licen\u010dn\u00ed syst\u00e9m pro distribuovan\u00fd syst\u00e9m je velmi drah\u00fd.<\/p>\n<p>Tip: <code>rm -rf \/var\/lib\/consul<\/code> \u2014 l\u00e9k na v\u0161echny bolesti agenta. Pokud v\u00e1m n\u011bco nefunguje, sta\u010d\u00ed smazat sv\u00e1 data a nahr\u00e1t data z kopie. Pravd\u011bpodobn\u011b bude Consul fungovat.<\/p>\n<h2>BEFW<\/h2>\n<p>\nNyn\u00ed si pov\u00edme, co jsme p\u0159idali k Consul.<\/p>\n<p><noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/wgnet\/befw\">BEFW<\/a><\/noindex> \u2014 je to akronym od <strong>B<\/strong>ack<strong>E<\/strong>nd<strong>F<\/strong>ire<strong>W<\/strong>all. Musel jsem n\u011bjak pojmenovat produkt, kdy\u017e jsem vytv\u00e1\u0159el repozit\u00e1\u0159, abych do n\u011bj um\u00edstil prvn\u00ed testovac\u00ed commity. Toto jm\u00e9no z\u016fstalo.<\/p>\n<h3>\u0160ablony pravidel<\/h3>\n<p>\nPravidla jsou naps\u00e1na v syntaxi iptables.<\/p>\n<ul>\n<li>-N BEFW<\/li>\n<li>-P INPUT DROP<\/li>\n<li>-A INPUT -m state\u2014state RELATED,ESTABLISHED -j ACCEPT<\/li>\n<li>-A INPUT -i lo -j ACCEPT<\/li>\n<li>-A INPUT -j BEFW<\/li>\n<\/ul>\n<p>\nV\u0161echno proch\u00e1z\u00ed do \u0159et\u011bzce BEFW, krom\u011b <code>ESTABLISHED<\/code>, <code>RELATED<\/code> a localhost. \u0160ablona m\u016f\u017ee b\u00fdt jak\u00e1koli, je to jen p\u0159\u00edklad.<\/p>\n<p>Jak je BEFW u\u017eite\u010dn\u00e9?<\/p>\n<h3>Slu\u017eby<\/h3>\n<p>\nM\u00e1me slu\u017ebu, v\u017edy m\u00e1 port, na kter\u00e9m b\u011b\u017e\u00ed, uzel. Z na\u0161\u00ed uzlu se m\u016f\u017eeme m\u00edstn\u011b zeptat agenta a zjistit, \u017ee m\u00e1me n\u011bjakou slu\u017ebu. Tak\u00e9 m\u016f\u017eeme p\u0159id\u00e1vat tagy.<\/p>\n<p><img decoding=\"async\" alt=\"Consul + iptables = :3\" src=\"\/wp-content\/uploads\/2020\/02\/6d8c74db37e2e1eed971cfa6947cc5c8.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nKa\u017ed\u00e1 slu\u017eba, kter\u00e1 b\u011b\u017e\u00ed a je registrov\u00e1na v Consul, se m\u011bn\u00ed na pravidlo iptables. M\u00e1me SSH \u2014 otev\u00edr\u00e1me port 22. Bash skript je jednoduch\u00fd: curl a iptables, nic v\u00edc nepot\u0159ebujeme.<\/p>\n<h3>Klienti<\/h3>\n<p>\nJak otev\u0159\u00edt p\u0159\u00edstup ne v\u0161em, ale selektivn\u011b? Po n\u00e1zvu slu\u017eby ukl\u00e1dat do KV \u00falo\u017ei\u0161t\u011b seznamy IP.<\/p>\n<p><img decoding=\"async\" alt=\"Consul + iptables = :3\" src=\"\/wp-content\/uploads\/2020\/02\/11a5af5aad9e848565458f168a1eee64.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nNap\u0159\u00edklad chceme, aby v\u0161ichni z des\u00e1t\u00e9 s\u00edt\u011b mohli p\u0159istupovat k slu\u017eb\u011b SSH_TCP_22. P\u0159id\u00e1me jedno mal\u00e9 pole TTL? a nyn\u00ed m\u00e1me do\u010dasn\u00e1 povolen\u00ed, nap\u0159\u00edklad na jeden den.<\/p>\n<h3>P\u0159\u00edstupy<\/h3>\n<p>\nSpojujeme slu\u017eby a klienty: m\u00e1me slu\u017ebu, na ka\u017edou je p\u0159ipraveno KV \u00falo\u017ei\u0161t\u011b. Nyn\u00ed d\u00e1v\u00e1me p\u0159\u00edstup ne v\u0161em, ale selektivn\u011b.<\/p>\n<p><img decoding=\"async\" alt=\"Consul + iptables = :3\" src=\"\/wp-content\/uploads\/2020\/02\/d1119fbb226595184cac8d9ce7a924e3.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<\/p>\n<h3>Skupiny<\/h3>\n<p>\nPokud budeme poka\u017ed\u00e9 ps\u00e1t tis\u00edce IP pro p\u0159\u00edstupy, unav\u00edme se. Vymysl\u00edme skupiny \u2014 samostatn\u00fd subset v KV. Nazveme ho Alias (nebo skupiny) a budeme tam ukl\u00e1dat skupiny podle stejn\u00e9ho principu.<\/p>\n<p><img decoding=\"async\" alt=\"Consul + iptables = :3\" src=\"\/wp-content\/uploads\/2020\/02\/67c6ab76cc25a3973739bdd44b69cef9.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nSpojujeme: nyn\u00ed m\u016f\u017eeme otev\u0159\u00edt SSH ne konkr\u00e9tn\u011b na P2P, ale na celou skupinu nebo n\u011bkolik skupin. Stejn\u011b tak existuje TTL \u2014 m\u016f\u017eeme do skupiny p\u0159id\u00e1vat a do\u010dasn\u011b odstra\u0148ovat.<\/p>\n<p><img decoding=\"async\" alt=\"Consul + iptables = :3\" src=\"\/wp-content\/uploads\/2020\/02\/a00b7db5087a96730c2725090f33e2b0.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<\/p>\n<h3>Integrace<\/h3>\n<p>\nN\u00e1\u0161 probl\u00e9m \u2014 lidsk\u00fd faktor a automatizace. Zat\u00edm jsme ho vy\u0159e\u0161ili takto.<\/p>\n<p><img decoding=\"async\" alt=\"Consul + iptables = :3\" src=\"\/wp-content\/uploads\/2020\/02\/ce7a0c16543d0624453e6bdada3b8f03.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nPracujeme s Puppet a p\u0159en\u00e1\u0161\u00edme v\u0161echny informace t\u00fdkaj\u00edc\u00ed se syst\u00e9mu (k\u00f3d aplikac\u00ed). V puppetdb (b\u011b\u017en\u00fd PostgreSQL) se uchov\u00e1v\u00e1 seznam slu\u017eeb, kter\u00e9 tam b\u011b\u017e\u00ed, a lze je vyhledat podle typu zdroje. Tam tak\u00e9 najdete, kdo kam p\u0159istupuje. D\u00e1le m\u00e1me syst\u00e9m pull request a merge request pro toto.<\/p>\n<p>Napsali jsme befw-sync \u2014 nejjednodu\u0161\u0161\u00ed \u0159e\u0161en\u00ed, kter\u00e9 pom\u00e1h\u00e1 p\u0159en\u00e1\u0161et data. Nejprve se sync cookies obracej\u00ed na puppetdb. Tam je nastaveno HTTP API: dotazujeme se, jak\u00e9 m\u00e1me slu\u017eby a co je t\u0159eba ud\u011blat. Pot\u00e9 se provede dotaz na Consul.<\/p>\n<p>Je integrace? Ano: napsali jsme pravidla a povolili p\u0159ij\u00edmat Pull Requesty. Pot\u0159ebujete n\u011bjak\u00fd port nebo p\u0159idat hosta do n\u011bjak\u00e9 skupiny? Pull Request, revize \u2014 \u017e\u00e1dn\u00e9 dal\u0161\u00ed \"Najdi 200 dal\u0161\u00edch ACL a zkus n\u011bco ud\u011blat s t\u00edm\".<\/p>\n<h3>Optimalizace<\/h3>\n<p>\nPing localhost s pr\u00e1zdn\u00fdm \u0159et\u011bzcem pravidel trv\u00e1 0,075 ms.<\/p>\n<p><img decoding=\"async\" alt=\"Consul + iptables = :3\" src=\"\/wp-content\/uploads\/2020\/02\/39263844f79c4795af05dc3ecb734394.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nP\u0159id\u00e1me do tohoto \u0159et\u011bzce 10 000 adres iptables. V d\u016fsledku toho se ping zv\u00fd\u0161\u00ed p\u011btkr\u00e1t: iptables je zcela line\u00e1rn\u00ed, zpracov\u00e1n\u00ed ka\u017ed\u00e9 adresy trv\u00e1 n\u011bjak\u00fd \u010das.<\/p>\n<p><img decoding=\"async\" alt=\"Consul + iptables = :3\" src=\"\/wp-content\/uploads\/2020\/02\/e6c314155c3a34fdf61e40479ecb26c3.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nPro firewall, do kter\u00e9ho migrujeme tis\u00edce ACL, m\u00e1me hodn\u011b pravidel a to zp\u016fsobuje zpo\u017ed\u011bn\u00ed. Pro hern\u00ed protokoly je to \u0161patn\u00e9.<\/p>\n<p>Ale pokud um\u00edst\u00edme <strong>10 000 adres do ipsetu<\/strong> ping se dokonce zkr\u00e1t\u00ed.<\/p>\n<p><img decoding=\"async\" alt=\"Consul + iptables = :3\" src=\"\/wp-content\/uploads\/2020\/02\/2dd844d9324d057ba9ab098e501a1e51.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nSmyslem je, \u017ee \"O\" (slo\u017eitost algoritmu) pro ipset je v\u017edy rovno 1, bez ohledu na to, kolik tam pravidel je. Pravda, je tam omezen\u00ed \u2014 pravidel nem\u016f\u017ee b\u00fdt v\u00edce ne\u017e 65535. Zat\u00edm s t\u00edm \u017eijeme: lze je kombinovat, roz\u0161i\u0159ovat, ud\u011blat dva ipsety v jednom.<\/p>\n<h3>\u00dalo\u017ei\u0161t\u011b<\/h3>\n<p>\nLogick\u00e9 pokra\u010dov\u00e1n\u00ed procesu iterac\u00ed je uchov\u00e1v\u00e1n\u00ed informac\u00ed o klientech pro slu\u017ebu v ipsetu.<\/p>\n<p><img decoding=\"async\" alt=\"Consul + iptables = :3\" src=\"\/wp-content\/uploads\/2020\/02\/126069799d95594be8611237e935189d.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nNyn\u00ed m\u00e1me stejn\u00fd SSH a ihned nep\u00ed\u0161eme 100 IP, ale zad\u00e1v\u00e1me n\u00e1zev ipsetu, se kter\u00fdm je t\u0159eba komunikovat, a n\u00e1sleduj\u00edc\u00ed pravidlo <code>DROP<\/code>. M\u016f\u017eeme to p\u0159ed\u011blat na jedno pravidlo \"Kdokoli tu nen\u00ed, ten DROP\", ale tak je to p\u0159ehledn\u011bj\u0161\u00ed.<\/p>\n<p>Nyn\u00ed m\u00e1me pravidla a sady. Hlavn\u00edm \u00fakolem je vytvo\u0159it sadu p\u0159ed naps\u00e1n\u00edm pravidla, proto\u017ee jinak iptables pravidlo nezap\u00ed\u0161e.<\/p>\n<h3>Obecn\u00e1 sch\u00e9ma<\/h3>\n<p>\nVe sch\u00e9matu to, co jsem popsal, vypad\u00e1 takto.<\/p>\n<p><img decoding=\"async\" alt=\"Consul + iptables = :3\" src=\"\/wp-content\/uploads\/2020\/02\/4046e921cb5f466f8af5a6af1fcd151c.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nZavazujeme do Puppet, v\u0161echno se ode\u0161le na hostitele, slu\u017eby jsou zde, ipset tam, a kdo tam nen\u00ed uveden, toho nepou\u0161t\u011bj\u00ed.<\/p>\n<h3>Allow &amp; deny<\/h3>\n<p>\nAbychom rychle zachra\u0148ovali sv\u011bt nebo rychle n\u011bkoho odpojili, na za\u010d\u00e1tku v\u0161ech \u0159et\u011bzc\u016f jsme vytvo\u0159ili dv\u011b ipsety: <code>rules_allow<\/code> a <code>rules_deny<\/code>. Jak to funguje?<\/p>\n<p>Nap\u0159\u00edklad, n\u011bkdo vytv\u00e1\u0159\u00ed z\u00e1t\u011b\u017e na na\u0161em webu pomoc\u00ed bot\u016f. D\u0159\u00edve bylo nutn\u00e9 naj\u00edt podle log\u016f jeho IP, p\u0159edat ji s\u00ed\u0165ov\u00fdm in\u017een\u00fdr\u016fm, aby na\u0161li zdroj provozu a zablokovali ho. Te\u010f to vypad\u00e1 jinak.<\/p>\n<p><img decoding=\"async\" alt=\"Consul + iptables = :3\" src=\"\/wp-content\/uploads\/2020\/02\/6e7f906919536cab2c283ba6ceb0472d.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nOdes\u00edl\u00e1me do Consul, \u010dek\u00e1me 2,5 s a hotovo. Vzhledem k tomu, \u017ee Consul rychle rozd\u011bluje d\u00edky P2P, funguje to v\u0161ude na sv\u011bt\u011b.<\/p>\n<p>Jednou jsem \u00fapln\u011b zastavil WOT, kdy\u017e jsem se spletl s firewallem. <code>rules_allow<\/code> \u2014 to je na\u0161e pojistka pro takov\u00e9 p\u0159\u00edpady. Pokud jsme n\u011bkde ud\u011blali chybu s firewallem, n\u011bco se zablokuje, v\u017edy m\u016f\u017eeme poslat podm\u00edn\u011bn\u00fd <code>0.0\/0<\/code>, abychom rychle v\u0161echno zvedli. Pozd\u011bji v\u0161echno oprav\u00edme ru\u010dn\u011b.<\/p>\n<h3>Dal\u0161\u00ed sady<\/h3>\n<p>\nM\u016f\u017eeme p\u0159idat jak\u00e9koli jin\u00e9 sady do prostoru <code>$IPSETS$<\/code>.<\/p>\n<p><img decoding=\"async\" alt=\"Consul + iptables = :3\" src=\"\/wp-content\/uploads\/2020\/02\/4e1e337b35da81cd7324b2adc106b085.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nPro\u010d? N\u011bkdy n\u011bkdo pot\u0159ebuje ipset, nap\u0159\u00edklad aby emuloval odpojen\u00ed n\u011bjak\u00e9 \u010d\u00e1sti clusteru. Ka\u017ed\u00fd m\u016f\u017ee p\u0159in\u00e9st jak\u00e9koli sady, pojmenovat je a ty budou br\u00e1ny z Consul. Tyto sady mohou jak \u00fa\u010dastnit v pravidlech iptables, tak b\u00fdt jako by p\u0159\u00edkazem <code>NOOP<\/code>: konzistence bude udr\u017eov\u00e1na d\u00e9monem.<\/p>\n<h3>U\u017eivatel\u00e9<\/h3>\n<p>\nD\u0159\u00edve to bylo tak: u\u017eivatel se p\u0159ipojil k s\u00edti a p\u0159es dom\u00e9nu z\u00edskal parametry. P\u0159ed zaveden\u00edm nov\u00fdch generac\u00ed firewall\u016f Cisco nedok\u00e1zala rozli\u0161it, kde je u\u017eivatel a kde IP. Proto byl p\u0159\u00edstup ud\u011blov\u00e1n pouze p\u0159es hostname stroje.<\/p>\n<p>Co jsme ud\u011blali? Vkl\u00ednili jsme se do okam\u017eiku z\u00edsk\u00e1v\u00e1n\u00ed adresy. Obvykle je to dot1x, Wi-Fi nebo VPN \u2014 v\u0161e jde p\u0159es RADIUS. Pro ka\u017ed\u00e9ho u\u017eivatele vytv\u00e1\u0159\u00edme skupinu podle jeho jm\u00e9na a umis\u0165ujeme do n\u00ed IP s TTL, kter\u00fd odpov\u00edd\u00e1 jeho dhcp.lease \u2014 jakmile vypr\u0161\u00ed, pravidlo zmiz\u00ed.<\/p>\n<p><img decoding=\"async\" alt=\"Consul + iptables = :3\" src=\"\/wp-content\/uploads\/2020\/02\/04550475c6af49d094c18b1d5b89123a.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nTe\u010f m\u016f\u017eeme otev\u00edrat p\u0159\u00edstup k slu\u017eb\u00e1m, stejn\u011b jako v jin\u00fdch skupin\u00e1ch, podle u\u017eivatelsk\u00e9ho jm\u00e9na. Zbavili jsme se bolesti s hostname, kdy\u017e se m\u011bn\u00ed, a sn\u00ed\u017eili z\u00e1t\u011b\u017e na s\u00ed\u0165ov\u00e9 in\u017een\u00fdry, proto\u017ee u\u017e to nepot\u0159ebuj\u00ed Cisco. Te\u010f si in\u017een\u00fd\u0159i sami zapisuj\u00ed p\u0159\u00edstupy na sv\u00fdch serverech.<\/p>\n<h3>Izolace<\/h3>\n<p>\nSou\u010dasn\u011b jsme za\u010dali rozeb\u00edrat izolaci. Spr\u00e1vci slu\u017eeb provedli inventarizaci, a my jsme analyzovali v\u0161echny na\u0161e s\u00edt\u011b. Rozlo\u017e\u00edme je na stejn\u00e9 skupiny, a na pot\u0159ebn\u00fdch serverech jsme skupiny p\u0159idali, nap\u0159\u00edklad do deny. Te\u010f ta sam\u00e1 izolace stagingu jde do rules_deny na produkci, ale ne do samotn\u00e9 produkce.<\/p>\n<p><img decoding=\"async\" alt=\"Consul + iptables = :3\" src=\"\/wp-content\/uploads\/2020\/02\/60fe7407013aa64b853791f8def627cc.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nSchema funguje rychle a jednodu\u0161e: odstra\u0148ujeme v\u0161echny ACL ze server\u016f, uvol\u0148ujeme hardware, sni\u017eujeme po\u010det izolovan\u00fdch VLAN.<\/p>\n<h3>Kontrola integrity<\/h3>\n<p>\nD\u0159\u00edve jsme m\u011bli speci\u00e1ln\u00ed spou\u0161t\u011b\u010d, kter\u00fd n\u00e1s informoval, kdy\u017e n\u011bkdo ru\u010dn\u011b m\u011bnil pravidlo firewallu. Psal jsem obrovsk\u00fd linter pro kontrolu pravidel firewallu, bylo to slo\u017eit\u00e9. Nyn\u00ed integritu kontroluje BEFW. Pe\u010dliv\u011b db\u00e1 na to, aby se pravidla, kter\u00e1 vytv\u00e1\u0159\u00ed, nem\u011bnila. Pokud n\u011bkdo zm\u011bn\u00ed pravidla firewallu, vr\u00e1t\u00ed v\u0161echno zp\u011bt. \u201eRychle jsem zprovoznil proxy, abych mohl pracovat z domova\u201c \u2013 takov\u00e9 mo\u017enosti u\u017e neexistuj\u00ed.<\/p>\n<p>BEFW kontroluje ipset ze slu\u017eeb a seznamu v befw.conf, pravidla slu\u017eeb v \u0159et\u011bzci BEFW. Ale nesleduje jin\u00e9 \u0159et\u011bzce a pravidla a dal\u0161\u00ed ipset.<\/p>\n<h3>Ochrana proti hav\u00e1ri\u00edm<\/h3>\n<p>\nBEFW v\u017edy ukl\u00e1d\u00e1 posledn\u00ed \u00fasp\u011b\u0161n\u00fd stav p\u0159\u00edmo do bin\u00e1rn\u00ed struktury state.bin. Pokud se n\u011bco pokaz\u00ed, v\u017edy se vr\u00e1t\u00ed zp\u011bt k tomuto state.bin.<\/p>\n<p><img decoding=\"async\" alt=\"Consul + iptables = :3\" src=\"\/wp-content\/uploads\/2020\/02\/724f3537c06a7135fa6397f9ddd5630f.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nJe to pojistka proti nestabiln\u00edmu chov\u00e1n\u00ed Consulu, kdy\u017e neodeslal data nebo n\u011bkdo ud\u011blal chybu a pou\u017eil pravidla, kter\u00e1 nelze aplikovat. Abychom nez\u016fstali bez firewallu, BEFW se vr\u00e1t\u00ed k posledn\u00edmu stavu, pokud v n\u011bjak\u00e9m okam\u017eiku dojde k chyb\u011b.<\/p>\n<p>V kritick\u00fdch situac\u00edch je to z\u00e1ruka, \u017ee z\u016fstaneme s funk\u010dn\u00edm firewallem. Otev\u00edr\u00e1me v\u0161echny \u0161ed\u00e9 s\u00edt\u011b v nad\u011bji, \u017ee spr\u00e1vce p\u0159ijde a oprav\u00ed to. N\u011bkdy to p\u0159enesu do konfigurac\u00ed, ale te\u010f m\u00e1me prost\u011b t\u0159i \u0161ed\u00e9 s\u00edt\u011b: 10\/8, 172\/12 a 192.168\/16. V r\u00e1mci na\u0161eho Consulu je to d\u016fle\u017eit\u00e1 vlastnost, kter\u00e1 n\u00e1m pom\u00e1h\u00e1 d\u00e1le se rozv\u00edjet.<\/p>\n<p><em>Demo: b\u011bhem prezentace Ivan demonstruje demo re\u017eim BEFW. Je pohodln\u011bj\u0161\u00ed sledovat demonstrov\u00e1n\u00ed na <noindex><a rel=\"nofollow\" href=\"https:\/\/youtu.be\/4zP67uYnsR4?t=1663\">video<\/a><\/noindex>. Zdrojov\u00fd k\u00f3d dema je k dispozici <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/wgnet\/befw\/tree\/master\/demo\">na GitHubu<\/a><\/noindex>.<\/em><\/p>\n<h2>\u00daskal\u00ed<\/h2>\n<p>\nPov\u00edm o chyb\u00e1ch, se kter\u00fdmi jsme se setkali.<\/p>\n<p><strong>ipset add set 0.0.0.0\/0. <\/strong>Co se stane, kdy\u017e do ipset p\u0159id\u00e1te 0.0.0.0\/0? P\u0159idaj\u00ed se v\u0161echny IP adresy? Otev\u0159e se p\u0159\u00edstup na internet? <\/p>\n<p>Ne, dostaneme chybu, kter\u00e1 n\u00e1s st\u00e1la dv\u011b hodiny v\u00fdpadku. A to chyba nefunguje od roku 2016, le\u017e\u00ed v RedHat Bugzilla pod \u010d\u00edslem #1297092, a na\u0161li jsme ji n\u00e1hodou \u2013 z reportu v\u00fdvoj\u00e1\u0159e.<\/p>\n<p>Nyn\u00ed v BEFW plat\u00ed p\u0159\u00edsn\u00e9 pravidlo, \u017ee <code>0.0.0.0\/0<\/code> se m\u011bn\u00ed na dv\u011b adresy: <code>0.0.0.0\/1<\/code> a <code>128.0.0.0\/1<\/code>.<\/p>\n<p><strong>ipset restore set &lt; file. <\/strong>Co ipset d\u011bl\u00e1, kdy\u017e mu \u0159eknete, \u017ee <code>restore<\/code>? \u0412\u044b \u0434\u0443\u043c\u0430\u0435\u0442\u0435, \u043e\u043d \u0440\u0430\u0431\u043e\u0442\u0430\u0435\u0442 \u0442\u0430\u043a\u0436\u0435, \u043a\u0430\u043a iptables? \u0412\u043e\u0441\u0441\u0442\u0430\u043d\u043e\u0432\u0438\u0442 \u0434\u0430\u043d\u043d\u044b\u0435?<\/p>\n<p>Nen\u00ed to tak \u2013 d\u011bl\u00e1 merge a star\u00e9 adresy nikam nezmiz\u00ed, p\u0159\u00edstup nezav\u00edr\u00e1te.<\/p>\n<p>Na\u0161li jsme chybu, kdy\u017e jsme testovali izolaci. Nyn\u00ed tam je docela slo\u017eit\u00fd syst\u00e9m \u2013 m\u00edsto <code>restore<\/code> prob\u00edh\u00e1 <code>create temp<\/code>, pak <code>restore flush temp<\/code> a <code>restore temp<\/code>. Na konci swap: pro atomaritu, proto\u017ee pokud nejprve prov\u00e1d\u00edte <code>flush<\/code> A v tu chv\u00edli p\u0159ijde n\u011bjak\u00fd bal\u00edk, kter\u00fd bude odm\u00edtnut, a n\u011bco se pokaz\u00ed. Proto je zde trochu \u010dern\u00e9 magie.<\/p>\n<p><strong>consul kv get -datacenter=other. <\/strong>Jak jsem ji\u017e \u0159\u00edkal, mysl\u00edme si, \u017ee \u017e\u00e1d\u00e1me o n\u011bjak\u00e1 data, ale dostaneme bu\u010f data, nebo chybu. M\u016f\u017eeme to d\u011blat p\u0159es Consul lok\u00e1ln\u011b, ale i v tomto p\u0159\u00edpad\u011b se to oboje zasekne.<\/p>\n<p>Lok\u00e1ln\u00ed klient Consul je ob\u00e1lka nad HTTP API. Ale prost\u011b zamrzne a nereaguje ani na Ctrl+C, ani na Ctrl+Z, prost\u011b na nic, pouze na <code>kill -9<\/code> v vedlej\u0161\u00ed konzole. S t\u00edm jsme se setkali, kdy\u017e jsme budovali velk\u00fd klastr. Ale zat\u00edm nem\u00e1me \u0159e\u0161en\u00ed, chyst\u00e1me se opravit tuto chybu v Consul.<\/p>\n<p><strong>Consul leader nereaguje. <\/strong>N\u00e1\u0161 master v datov\u00e9m centru nereaguje, mysl\u00edme si: \"Snad nyn\u00ed spust\u00ed algoritmus p\u0159evolby?\" <\/p>\n<p>Ne, nespust\u00ed, a monitoring nic neuk\u00e1\u017ee: Consul \u0159\u00edk\u00e1, \u017ee commit index existuje, leader nalezen, v\u0161e v po\u0159\u00e1dku.<\/p>\n<p>Jak se s t\u00edm vyrovn\u00e1v\u00e1me? <code>service consul restart<\/code> v cron ka\u017edou hodinu. Pokud m\u00e1te 50 server\u016f, nen\u00ed to stra\u0161n\u00e9. A\u017e jich bude 16 000, pochop\u00edte, jak to funguje.<\/p>\n<h2>Z\u00e1v\u011br<\/h2>\n<p>\nNakonec jsme z\u00edskali n\u00e1sleduj\u00edc\u00ed v\u00fdhody:<\/p>\n<ul>\n<li>100% pokryt\u00ed v\u0161ech Linuxov\u00fdch stroj\u016f.<\/li>\n<li>Rychlost.<\/li>\n<li>Automatizaci.<\/li>\n<li>Osvobodili jsme hardware a s\u00ed\u0165ov\u00e9 in\u017een\u00fdry z otroctv\u00ed.<\/li>\n<li>Objevily se mo\u017enosti integrace, kter\u00e9 jsou prakticky nekone\u010dn\u00e9: a\u0165 u\u017e s Kubernetes, Ansible nebo Python.<\/li>\n<\/ul>\n<p>\n<strong>Nev\u00fdhody<\/strong>: Consul, se kter\u00fdm te\u010f mus\u00edme \u017e\u00edt, a velmi vysok\u00e1 cena chyby. Nap\u0159\u00edklad jednou jsem v 6 odpoledne (v hlavn\u00edm \u010dase v Rusku) n\u011bco upravoval ve v\u00fd\u010dtech s\u00edt\u00ed. Pr\u00e1v\u011b jsme budovali izolaci na BEFW. N\u011bkde jsem se zm\u00fdlil, zd\u00e1 se, \u017ee jsem uvedl nespr\u00e1vnou masku, ale v\u0161e spadlo b\u011bhem dvou sekund. Monitoring se rozsv\u00edtil, p\u0159ib\u011bhl technik podpory: \"V\u0161echno n\u00e1m pad\u00e1!\" Vedouc\u00ed odd\u011blen\u00ed zest\u00e1rl, kdy\u017e vysv\u011btloval byznysu, pro\u010d k tomu do\u0161lo.<\/p>\n<p>Cena chyby je tak vysok\u00e1, \u017ee jsme vymysleli vlastn\u00ed slo\u017eitou proceduru prevence. Pokud to budete implementovat na velk\u00e9m produk\u010dn\u00edm prost\u0159ed\u00ed, ned\u00e1vejte master token nad Consul v\u0161em bez v\u00fdb\u011bru. To skon\u010d\u00ed \u0161patn\u011b.<\/p>\n<p><strong>N\u00e1klady. <\/strong>Napsal jsem k\u00f3d 400 hodin s\u00e1m. Na podporu moje t\u00fdm o 4 lidech tr\u00e1v\u00ed 10 hodin m\u011bs\u00ed\u010dn\u011b na v\u0161echny. Ve srovn\u00e1n\u00ed s cenou jak\u00e9hokoli firewallu nov\u00e9 generace je to zadarmo.<\/p>\n<p><strong>Pl\u00e1ny. <\/strong>Dlouhodob\u00fd pl\u00e1n je hled\u00e1n\u00ed alternativn\u00ed dopravy jako n\u00e1hrady nebo dopln\u011bn\u00ed k Consul. Mo\u017en\u00e1 to bude Kafka nebo n\u011bco podobn\u00e9ho. Ale v p\u0159\u00ed\u0161t\u00edch letech budeme \u017e\u00edt na Consul.<\/p>\n<p>Nejbli\u017e\u0161\u00ed pl\u00e1ny: integrace s Fail2ban, s monitorov\u00e1n\u00edm, s nftables, mo\u017en\u00e1 s jin\u00fdmi distribucemi, metriky, roz\u0161\u00ed\u0159en\u00e9 monitorov\u00e1n\u00ed, optimalizace. Podpora Kubernetes je tak\u00e9 n\u011bkde v pl\u00e1nech, proto\u017ee nyn\u00ed m\u00e1me n\u011bkolik klastr\u016f a z\u00e1jem.<\/p>\n<p>Dal\u0161\u00ed pl\u00e1ny:<\/p>\n<ul>\n<li>vyhled\u00e1v\u00e1n\u00ed anom\u00e1li\u00ed v provozu;<\/li>\n<li>spr\u00e1va mapy s\u00edt\u011b;<\/li>\n<li>podpora Kubernetes;<\/li>\n<li>kompilace bal\u00ed\u010dk\u016f pro v\u0161echny syst\u00e9my;<\/li>\n<li>Web-UI.<\/li>\n<\/ul>\n<p>\nNeust\u00e1le pracujeme na roz\u0161\u00ed\u0159en\u00ed konfigurace, zvy\u0161ov\u00e1n\u00ed metrik a optimalizaci.<\/p>\n<p><i>P\u0159ipojte se k projektu. Projekt se vyda\u0159il, ale bohu\u017eel je to zat\u00edm projekt jedn\u00e9 osoby. P\u0159ij\u010fte na <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/wgnet\/befw\">GitHub <\/a><\/noindex>a zkuste n\u011bco ud\u011blat: p\u0159isp\u011bt, otestovat, n\u011bco navrhnout, d\u00e1t svou zp\u011btnou vazbu.<\/i><\/p>\n<blockquote><p>Mezit\u00edm se p\u0159ipravujeme na <noindex><a rel=\"nofollow\" href=\"https:\/\/www.highload.ru\/spb\/2020\">Saint HighLoad++<\/a><\/noindex>, kter\u00fd se kon\u00e1 6. a 7. dubna v Petrohrad\u011b, a zveme v\u00fdvoj\u00e1\u0159e vysoce zat\u011b\u017eovan\u00fdch syst\u00e9m\u016f, <noindex><a rel=\"nofollow\" href=\"https:\/\/conf.ontico.ru\/lectures\/propose?conference=hl2020-spb\">aby podali p\u0159ihl\u00e1\u0161ku na p\u0159edn\u00e1\u0161ku<\/a><\/noindex>. Zku\u0161en\u00ed \u0159e\u010dn\u00edci u\u017e v\u011bd\u00ed, co d\u011blat, a nov\u00e1\u010dk\u016fm doporu\u010dujeme alespo\u0148 <noindex><a rel=\"nofollow\" href=\"https:\/\/www.highload.ru\/spb\/2020\/speakers\">vyzkou\u0161et<\/a><\/noindex>. \u00da\u010dast na konferenci jako \u0159e\u010dn\u00edk m\u00e1 \u0159adu v\u00fdhod. Jak\u00e9, si m\u016f\u017eete p\u0159e\u010d\u00edst nap\u0159\u00edklad na konci <noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/oleg-bunin\/blog\/486142\/\">tohoto \u010dl\u00e1nku<\/a><\/noindex>.<\/p><\/blockquote>\n<p>Zdroj: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/oleg-bunin\/blog\/486842\/\">habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u0412 2010 \u0433\u043e\u0434\u0443 \u0443 \u043a\u043e\u043c\u043f\u0430\u043d\u0438\u0438 Wargaming \u0431\u044b\u043b\u043e 50 \u0441\u0435\u0440\u0432\u0435\u0440\u043e\u0432 \u0438 \u043f\u0440\u043e\u0441\u0442\u0430\u044f \u0441\u0435\u0442\u0435\u0432\u0430\u044f \u043c\u043e\u0434\u0435\u043b\u044c: \u0431\u044d\u043a\u0435\u043d\u0434, \u0444\u0440\u043e\u043d\u0442\u0435\u043d\u0434 \u0438 \u0444\u0430\u0439\u0440\u0432\u043e\u043b. \u041a\u043e\u043b\u0438\u0447\u0435\u0441\u0442\u0432\u043e \u0441\u0435\u0440\u0432\u0435\u0440\u043e\u0432 \u0440\u043e\u0441\u043b\u043e, \u043c\u043e\u0434\u0435\u043b\u044c \u0443\u0441\u043b\u043e\u0436\u043d\u044f\u043b\u0430\u0441\u044c: \u0441\u0442\u0435\u0439\u0434\u0436\u0438\u043d\u0433\u0438, \u0438\u0437\u043e\u043b\u0438\u0440\u043e\u0432\u0430\u043d\u043d\u044b\u0435 VLAN \u0441 ACL, \u043f\u043e\u0442\u043e\u043c VPN \u0441 VRF, VLAN c ACL \u043d\u0430 L2, VRF \u0441 ACL \u043d\u0430 L3. \u0417\u0430\u043a\u0440\u0443\u0436\u0438\u043b\u0430\u0441\u044c \u0433\u043e\u043b\u043e\u0432\u0430? \u0414\u0430\u043b\u044c\u0448\u0435 \u0431\u0443\u0434\u0435\u0442 \u0432\u0435\u0441\u0435\u043b\u0435\u0435. \u041a\u043e\u0433\u0434\u0430 \u0441\u0435\u0440\u0432\u0435\u0440\u043e\u0432 \u0441\u0442\u0430\u043b\u043e 16 000 \u0440\u0430\u0431\u043e\u0442\u0430\u0442\u044c \u0431\u0435\u0437 \u0441\u043b\u0435\u0437 \u0441 [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":41052,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[],"tags":[],"class_list":["post-41051","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.3 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u0412 2010 \u0433\u043e\u0434\u0443 \u0443 \u043a\u043e\u043c\u043f\u0430\u043d\u0438\u0438 Wargaming \u0431\u044b\u043b\u043e 50 \u0441\u0435\u0440\u0432\u0435\u0440\u043e\u0432 \u0438 \u043f\u0440\u043e\u0441\u0442\u0430\u044f \u0441\u0435\u0442\u0435\u0432\u0430\u044f \u043c\u043e\u0434\u0435\u043b\u044c: \u0431\u044d\u043a\u0435\u043d\u0434, \u0444\u0440\u043e\u043d\u0442\u0435\u043d\u0434 \u0438 \u0444\u0430\u0439\u0440\u0432\u043e\u043b.\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/cs\/blog\/consul-iptables-3\" \/>\n\t\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.3\" \/>\n\t\t<meta property=\"og:locale\" content=\"cs_CZ\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47Consul + iptables = :3 | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u0412 2010 \u0433\u043e\u0434\u0443 \u0443 \u043a\u043e\u043c\u043f\u0430\u043d\u0438\u0438 Wargaming \u0431\u044b\u043b\u043e 50 \u0441\u0435\u0440\u0432\u0435\u0440\u043e\u0432 \u0438 \u043f\u0440\u043e\u0441\u0442\u0430\u044f \u0441\u0435\u0442\u0435\u0432\u0430\u044f \u043c\u043e\u0434\u0435\u043b\u044c: \u0431\u044d\u043a\u0435\u043d\u0434, \u0444\u0440\u043e\u043d\u0442\u0435\u043d\u0434 \u0438 \u0444\u0430\u0439\u0440\u0432\u043e\u043b.\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/cs\/blog\/consul-iptables-3\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2020-02-05T18:28:58+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-02-05T18:28:58+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47Consul + iptables = :3 | ProHoster","description":"V roce 2010 m\u011bla spole\u010dnost Wargaming 50 server\u016f a jednoduch\u00fd s\u00ed\u0165ov\u00fd model: backend, frontend a firewall.","canonical_url":"https:\/\/prohoster.info\/cs\/blog\/consul-iptables-3","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"cs_CZ","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47Consul + iptables = :3 | ProHoster","og:description":"\u0412 2010 \u0433\u043e\u0434\u0443 \u0443 \u043a\u043e\u043c\u043f\u0430\u043d\u0438\u0438 Wargaming \u0431\u044b\u043b\u043e 50 \u0441\u0435\u0440\u0432\u0435\u0440\u043e\u0432 \u0438 \u043f\u0440\u043e\u0441\u0442\u0430\u044f \u0441\u0435\u0442\u0435\u0432\u0430\u044f \u043c\u043e\u0434\u0435\u043b\u044c: \u0431\u044d\u043a\u0435\u043d\u0434, \u0444\u0440\u043e\u043d\u0442\u0435\u043d\u0434 \u0438 \u0444\u0430\u0439\u0440\u0432\u043e\u043b.","og:url":"https:\/\/prohoster.info\/cs\/blog\/consul-iptables-3","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-02-05T18:28:58+00:00","article:modified_time":"2020-02-05T18:28:58+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"41051","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-03-01 00:24:42","updated":"2022-10-01 10:00:06","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts\/41051","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/comments?post=41051"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts\/41051\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/media\/41052"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/media?parent=41051"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/categories?post=41051"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/tags?post=41051"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}