{"id":37849,"date":"2019-10-31T22:20:09","date_gmt":"2019-10-31T19:20:09","guid":{"rendered":"https:\/\/prohoster.info\/blog\/kafka-i-mikroservisy-obzor\/"},"modified":"2019-10-31T22:20:09","modified_gmt":"2019-10-31T19:20:09","slug":"kafka-i-mikroservisy-obzor","status":"publish","type":"post","link":"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/kafka-i-mikroservisy-obzor","title":{"rendered":"Kafka a mikroservisy: preh\u013ead","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p><img decoding=\"async\" alt=\"Kafka a mikroservisy: preh\u013ead\" src=\"\/wp-content\/uploads\/2019\/09\/ab0534ad5d28b3c3aa03e4e0b2d3101b.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Ahojte. V tomto \u010dl\u00e1nku v\u00e1m prezrad\u00edm, pre\u010do sme v Avite pred deviatimi mesiacmi zvolili Kafka, a \u010do vlastne predstavuje. Podel\u00edm sa o jeden z pr\u00edpadov pou\u017eitia \u2013 broker spr\u00e1v. Na z\u00e1ver sa porozpr\u00e1vame o tom, ak\u00e9 v\u00fdhody sme z\u00edskali z pr\u00edstupu Kafka as a Service.<\/p>\n<p><noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><\/p>\n<h1 id=\"problema\">Probl\u00e9m<\/h1>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Kafka a mikroservisy: preh\u013ead\" src=\"\/wp-content\/uploads\/2019\/09\/dfa5e231847cee67259da17867a11812.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Na za\u010diatok trochu kontextu. Pred nejak\u00fdm \u010dasom sme za\u010dali odch\u00e1dza\u0165 od monolitnej architekt\u00fary, a teraz m\u00e1 Avito u\u017e nieko\u013eko stoviek r\u00f4znych slu\u017eieb. Tieto slu\u017eby maj\u00fa svoje \u00falo\u017eisk\u00e1, svoj technologick\u00fd stack a zodpovedaj\u00fa za svoju \u010das\u0165 biznisovej logiky. <\/p>\n<p><\/p>\n<p>Jedn\u00fdm z probl\u00e9mov s ve\u013ek\u00fdm po\u010dtom slu\u017eieb s\u00fa komunik\u00e1cie. Slu\u017eba A \u010dasto chce z\u00edska\u0165 inform\u00e1cie, ktor\u00e9 m\u00e1 slu\u017eba B. V tomto pr\u00edpade sa slu\u017eba A obracia na slu\u017ebu B prostredn\u00edctvom synchronn\u00e9ho API. Slu\u017eba V chce vedie\u0165, \u010do sa deje so slu\u017ebami G a D, a tie, na druhej strane, sa zauj\u00edmaj\u00fa o slu\u017eby A a B. Ke\u010f po\u010det tak\u00fdchto \u201ezvedav\u00fdch\u201c slu\u017eieb narast\u00e1, v\u00e4zby medzi nimi sa menia na zlo\u017eit\u00fa sie\u0165.<\/p>\n<p><\/p>\n<p>Pri tom sa m\u00f4\u017ee slu\u017eba A kedyko\u013evek sta\u0165 nedostupnou. A \u010do robi\u0165 v takom pr\u00edpade slu\u017ebe B a v\u0161etk\u00fdm ostatn\u00fdm z\u00e1visl\u00fdm slu\u017eb\u00e1m? A ak na uskuto\u010dnenie biznisovej oper\u00e1cie je potrebn\u00e9 vykona\u0165 re\u0165azec postupn\u00fdch synchronn\u00fdch volan\u00ed, pravdepodobnos\u0165 zlyhania celej oper\u00e1cie sa e\u0161te zvy\u0161uje (a t\u00fdm vy\u0161\u0161ia je, \u010d\u00edm dlh\u0161\u00ed je tento re\u0165azec). <\/p>\n<p><\/p>\n<h1 id=\"vybor-tehnologii\">V\u00fdber technol\u00f3gie<\/h1>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Kafka a mikroservisy: preh\u013ead\" src=\"\/wp-content\/uploads\/2019\/09\/3acd7bcf3dded8093bdef6d627eb4071.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Dobre, probl\u00e9my s\u00fa jasn\u00e9. M\u00f4\u017eeme ich vyrie\u0161i\u0165 vytvoren\u00edm centralizovan\u00e9ho syst\u00e9mu v\u00fdmeny spr\u00e1v medzi slu\u017ebami. Teraz ka\u017ed\u00e1 z slu\u017eieb potrebuje vedie\u0165 iba o tomto syst\u00e9me v\u00fdmeny spr\u00e1v. Okrem toho by mal by\u0165 s\u00e1m syst\u00e9m odoln\u00fd vo\u010di zlyhaniu a horizont\u00e1lne \u0161k\u00e1lovate\u013en\u00fd, ako aj v pr\u00edpade nehody zhroma\u017e\u010fova\u0165 vo svojom bufferi \u017eiadosti pre n\u00e1sledn\u00e9 spracovanie. <\/p>\n<p><\/p>\n<p>Teraz si vyberme technol\u00f3giu, na ktorej sa bude realizova\u0165 dod\u00e1vka spr\u00e1v. Najprv si ujasn\u00edme, \u010do od nej o\u010dak\u00e1vame:<\/p>\n<p><\/p>\n<ul>\n<li>spr\u00e1vy medzi slu\u017ebami by nemali by\u0165 straten\u00e9;<\/li>\n<li>spr\u00e1vy sa m\u00f4\u017eu duplicova\u0165;<\/li>\n<li>spr\u00e1vy m\u00f4\u017eeme uchov\u00e1va\u0165 a \u010d\u00edta\u0165 na dobu nieko\u013ek\u00fdch dn\u00ed (perzistentn\u00fd buffer);<\/li>\n<li>slu\u017eby sa m\u00f4\u017eu prihl\u00e1si\u0165 k zauj\u00edmav\u00fdm \u00fadajom;<\/li>\n<li>nieko\u013eko slu\u017eieb m\u00f4\u017ee \u010d\u00edta\u0165 rovnak\u00e9 \u00fadaje;<\/li>\n<li>spr\u00e1vy m\u00f4\u017eu obsahova\u0165 podrobn\u00fd, objemn\u00fd payload (event-carried state transfer);<\/li>\n<li>niekedy je potrebn\u00e1 z\u00e1ruka poradia spr\u00e1v.<\/li>\n<\/ul>\n<p><\/p>\n<p>Rovnako pre n\u00e1s bolo kriticky d\u00f4le\u017eit\u00e9 vybra\u0165 maxim\u00e1lne \u0161k\u00e1lovate\u013en\u00fd a spo\u013eahliv\u00fd syst\u00e9m s vysokou kapacitou (minim\u00e1lne 100k spr\u00e1v pri nieko\u013ek\u00fdch kilobajtoch za sekundu).<\/p>\n<p><\/p>\n<p>V tejto f\u00e1ze sme sa rozl\u00fa\u010dili s RabbitMQ (\u0165a\u017ek\u00e9 udr\u017eiava\u0165 stabilitu pri vysok\u00fdch rps), PGQ od SkyTools (nedostato\u010dne r\u00fdchly a zle \u0161k\u00e1lovate\u013en\u00fd) a NSQ (nepersisten\u010dn\u00fd). V\u0161etky tieto technol\u00f3gie sa u n\u00e1s v spolo\u010dnosti pou\u017e\u00edvaj\u00fa, ale na dan\u00fa \u00falohu sa nehodili. <\/p>\n<p><\/p>\n<p>Potom sme za\u010dali sk\u00fama\u0165 nov\u00e9 technol\u00f3gie pre n\u00e1s \u2014 Apache Kafka, Apache Pulsar a NATS Streaming.<\/p>\n<p><\/p>\n<p>Prv\u00fdm sme vyl\u00fa\u010dili Pulsar. Rozhodli sme sa, \u017ee Kafka a Pulsar s\u00fa dos\u0165 podobn\u00e9 rie\u0161enia. A hoci je Pulsar preverovan\u00fd ve\u013ek\u00fdmi spolo\u010dnos\u0165ami, nov\u0161\u00ed a pon\u00faka teoreticky ni\u017e\u0161iu latenciu, rozhodli sme sa z t\u00fdchto dvoch ponecha\u0165 Kafka ako de facto \u0161tandard pre tak\u00e9to \u00falohy. Pravdepodobne sa v bud\u00facnosti k Apache Pulsar vr\u00e1time. <\/p>\n<p><\/p>\n<p>A tak zostali dvaja kandid\u00e1ti: NATS Streaming a Apache Kafka. Oba sme podrobne presk\u00famali a obidve vyhovovali na\u0161ej \u00falohe. Nakoniec sme sa v\u0161ak ob\u00e1vali relat\u00edvnej mladosti NATS Streaming (a toho, \u017ee jeden z hlavn\u00fdch v\u00fdvoj\u00e1rov, Tyler Treat, sa rozhodol od\u00eds\u0165 z projektu a za\u010da\u0165 svoj vlastn\u00fd \u2014 Liftbridge). Okrem toho Clustering re\u017eim NATS Streaming neumo\u017e\u0148oval v\u00fdrazn\u00e9 horizont\u00e1lne \u0161k\u00e1lovanie (pravdepodobne u\u017e nie je probl\u00e9m po pridan\u00ed partitioning re\u017eimu v roku 2017).<\/p>\n<p><\/p>\n<p>Napriek tomu je NATS Streaming skvel\u00e1 technol\u00f3gia, nap\u00edsan\u00e1 v Go a m\u00e1 podporu Cloud Native Computing Foundation. Na rozdiel od Apache Kafka, nepotrebuje Zookeeper na fungovanie (mo\u017eno, <noindex><a rel=\"nofollow\" href=\"https:\/\/cwiki.apache.org\/confluence\/display\/KAFKA\/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum\">v bud\u00facnosti bude mo\u017en\u00e9 to ist\u00e9 poveda\u0165 aj o Kafka<\/a><\/noindex>), preto\u017ee interne implementuje RAFT. NATS Streaming je z\u00e1rove\u0148 jednoduch\u0161ia na administr\u00e1ciu. Nevyra\u010fujeme, \u017ee sa v bud\u00facnosti e\u0161te vr\u00e1time k tejto technol\u00f3gii. <\/p>\n<p><\/p>\n<p>Napriek tomu sa n\u00e1\u0161 dne\u0161n\u00fd v\u00ed\u0165az stal Apache Kafka. V na\u0161ich testoch sa uk\u00e1zala by\u0165 dostato\u010dne r\u00fdchla (viac ako mili\u00f3n spr\u00e1v za sekundu pri \u010d\u00edtan\u00ed a p\u00edsan\u00ed s objemom spr\u00e1v 1 kilobajt), dostato\u010dne spo\u013eahliv\u00e1, dobre \u0161k\u00e1lovate\u013en\u00e1 a overen\u00e1 v produkcii ve\u013ek\u00fdmi spolo\u010dnos\u0165ami. Okrem toho Kafka podporuje aspo\u0148 nieko\u013eko ve\u013ek\u00fdch komer\u010dn\u00fdch spolo\u010dnost\u00ed (napr\u00edklad pou\u017e\u00edvame Confluent verziu), a tie\u017e m\u00e1 vybudovan\u00fd rozvinut\u00fd ekosyst\u00e9m.<br \/>\n<br clear=\"left\">\n <\/p>\n<p><\/p>\n<h1 id=\"obzor-kafka\">Preh\u013ead Kafka<\/h1>\n<p><\/p>\n<p>Predt\u00fdm, ako za\u010dneme, okam\u017eite odpor\u00fa\u010dam vynikaj\u00facu knihu \u2014 <em>\u201eKafka: The Definitive Guide\u201c<\/em> (existuje aj v slovenskom preklade, ale terminol\u00f3gia trochu zamie\u0148a hlavu). V nej n\u00e1jdete inform\u00e1cie potrebn\u00e9 na z\u00e1kladn\u00e9 pochopenie Kafka a mo\u017eno aj trochu viac. Samotn\u00e1 dokument\u00e1cia od Apache a blog od Confluent s\u00fa tie\u017e v\u00fdborne nap\u00edsan\u00e9 a \u013eahko \u010ditate\u013en\u00e9. <\/p>\n<p><\/p>\n<p>Tak\u017ee sa pozrime na to, ako je Kafka usporiadan\u00e1 z vt\u00e1\u010dej perspekt\u00edvy. Z\u00e1kladn\u00e1 topol\u00f3gia Kafka pozost\u00e1va z producer, consumer, broker a zookeeper.<\/p>\n<p><\/p>\n<h3 id=\"broker\">Brokera<\/h3>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Kafka a mikroservisy: preh\u013ead\" src=\"\/wp-content\/uploads\/2019\/09\/82f34b5a1aadcdc07eef58eb5414b553.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Za ukladanie va\u0161ich \u00fadajov zodpoved\u00e1 broker (broker). V\u0161etky \u00fadaje s\u00fa ulo\u017een\u00e9 v bin\u00e1rnej forme a broker m\u00e1 len m\u00e1lo inform\u00e1ci\u00ed o tom, \u010do predstavuj\u00fa a ak\u00e1 je ich \u0161trukt\u00fara. <\/p>\n<p><\/p>\n<p>Ka\u017ed\u00fd logick\u00fd typ udalost\u00ed sa zvy\u010dajne nach\u00e1dza v samostatnom topicu (topic). Napr\u00edklad, udalos\u0165 zalo\u017eenia inzer\u00e1tu m\u00f4\u017ee pr\u00eds\u0165 do topicu item.created, zatia\u013e \u010do udalos\u0165 jeho zmeny do item.changed. T\u00e9mata je mo\u017en\u00e9 pova\u017eova\u0165 za klasifik\u00e1tory udalost\u00ed. Na \u00farovni topicu je mo\u017en\u00e9 nastavi\u0165 tak\u00e9 konfigura\u010dn\u00e9 parametre, ako s\u00fa: <\/p>\n<p><\/p>\n<ul>\n<li>objem uchov\u00e1van\u00fdch \u00fadajov a\/alebo ich vek (retention.bytes, retention.ms); <\/li>\n<li>faktor redundancie \u00fadajov (replication factor);<\/li>\n<li>maxim\u00e1lna ve\u013ekos\u0165 jednej spr\u00e1vy (max.message.bytes);<\/li>\n<li>minim\u00e1lny po\u010det synchronizovan\u00fdch replik, pri ktor\u00fdch je mo\u017en\u00e9 zapisova\u0165 \u00fadaje do topicu (min.insync.replicas);<\/li>\n<li>mo\u017enos\u0165 vykona\u0165 failover na nesynchronizovan\u00fa zaost\u00e1vaj\u00facu replikou s potenci\u00e1lnou stratou \u00fadajov (unclean.leader.election.enable);<\/li>\n<li>a mnoho \u010fal\u0161\u00edch (<noindex><a rel=\"nofollow\" href=\"https:\/\/kafka.apache.org\/documentation\/#topicconfigs\">https:\/\/kafka.apache.org\/documentation\/#topicconfigs<\/a><\/noindex>).<\/li>\n<\/ul>\n<p><\/p>\n<p>Na druhej strane, ka\u017ed\u00fd topic sa del\u00ed na jednu alebo viac part\u00edci\u00ed (partition). Pr\u00e1ve do part\u00edcie nakoniec putuj\u00fa udalosti. Ak je v klustre viac ako jeden broker, part\u00edcie bud\u00fa rovnomerne rozdelen\u00e9 medzi v\u0161etk\u00fdch brokerov (ako to len bude mo\u017en\u00e9), \u010do umo\u017en\u00ed \u0161k\u00e1lovanie za\u0165a\u017eenia pri z\u00e1pise a \u010d\u00edtan\u00ed do jedn\u00e9ho topicu na viacero brokerov naraz.<\/p>\n<p><\/p>\n<p>Na disku s\u00fa \u00fadaje pre ka\u017ed\u00fa part\u00edciu ulo\u017een\u00e9 vo forme segmentov\u00fdch s\u00faborov, ktor\u00e9 s\u00fa predvolene ve\u013ek\u00e9 jeden gigabajt (riadi sa cez log.segment.bytes). D\u00f4le\u017eit\u00e1 vlastnos\u0165 \u2014 odstra\u0148ovanie \u00fadajov z part\u00edci\u00ed (pri uplynut\u00ed retention) sa uskuto\u010d\u0148uje pr\u00e1ve po segmentoch (nie je mo\u017en\u00e9 odstr\u00e1ni\u0165 jednu udalos\u0165 z part\u00edcie, mo\u017en\u00e9 je len odstr\u00e1ni\u0165 cel\u00fd segment, a to iba neakt\u00edvny).<\/p>\n<p><\/p>\n<h3 id=\"zookeeper\">Zookeeper<\/h3>\n<p><\/p>\n<p>Zookeeper funguje ako \u00falo\u017eisko metad\u00e1t a koordin\u00e1tor. Je to on, kto dok\u00e1\u017ee poveda\u0165, \u010di s\u00fa brokery \u017eiv\u00e9 (na to sa m\u00f4\u017eete pozrie\u0165 o\u010dami zookeepra cez zookeeper-shell pr\u00edkazom <code>ls \/brokers\/ids<\/code>), ktor\u00fd z brokerov je kontrol\u00f3r (<code>get \/controller<\/code>), s\u00fa part\u00edcie v synchronizovanom stave so svojimi replikami (<code>get \/brokers\/topics\/topic_name\/partitions\/partition_number\/state<\/code>). Taktie\u017e pr\u00e1ve do zookeepra najprv p\u00f4jdu producer a consumer, aby zistili, na ktorom brokeri s\u00fa ulo\u017een\u00e9 ak\u00e9 t\u00e9my a part\u00edcie. V pr\u00edpadoch, ke\u010f je pre t\u00e9mu nastaven\u00fd replication factor v\u00e4\u010d\u0161\u00ed ako 1, zookeeper ur\u010d\u00ed, ktor\u00e9 part\u00edcie s\u00fa l\u00eddrami (do nich sa bude zapisova\u0165 a z nich sa bude \u010d\u00edta\u0165). V pr\u00edpade p\u00e1du brokera sa pr\u00e1ve v zookeepra zaznamen\u00e1 inform\u00e1cia o nov\u00fdch l\u00eddersk\u00fdch part\u00edci\u00e1ch (od verzie 1.1.0 asynchr\u00f3nne, <noindex><a rel=\"nofollow\" href=\"https:\/\/www.confluent.io\/blog\/apache-kafka-supports-200k-partitions-per-cluster\">a to je d\u00f4le\u017eit\u00e9<\/a><\/noindex>).<\/p>\n<p><\/p>\n<p>V star\u0161\u00edch verzi\u00e1ch Kafka zookeeper zodpovedal aj za ukladanie offsetov, ale teraz s\u00fa ulo\u017een\u00e9 v \u0161peci\u00e1lnej t\u00e9me <code>__consumer_offsets<\/code> na brokeri (aj ke\u010f m\u00f4\u017eete na tieto \u00fa\u010dely st\u00e1le pou\u017e\u00edva\u0165 zookeeper). <\/p>\n<p><\/p>\n<p>Najjednoduch\u0161\u00ed sp\u00f4sob, ako premeni\u0165 va\u0161e d\u00e1ta na tekvicu, je pr\u00e1ve strata inform\u00e1ci\u00ed zo zookeepra. V takom scen\u00e1ri bude ve\u013emi \u0165a\u017ek\u00e9 pochopi\u0165, \u010do a odkial treba \u010d\u00edta\u0165. <\/p>\n<p><\/p>\n<h3 id=\"producer\">Producer<\/h3>\n<p><\/p>\n<p>Producer je naj\u010dastej\u0161ie slu\u017eba, ktor\u00e1 priamo zapisuje d\u00e1ta do Apache Kafka. Producer vyber\u00e1 t\u00e9mu, v ktorej bud\u00fa ulo\u017een\u00e9 jeho tematick\u00e9 spr\u00e1vy, a za\u010d\u00edna do nej zapisova\u0165 inform\u00e1cie. Napr\u00edklad, producerom m\u00f4\u017ee by\u0165 slu\u017eba na inzerovanie. V takom pr\u00edpade bude odosiela\u0165 do tematick\u00fdch t\u00e9m tak\u00e9 udalosti, ako s\u00fa \"inzer\u00e1t vytvoren\u00fd\", \"inzer\u00e1t aktualizovan\u00fd\", \"inzer\u00e1t odstr\u00e1nen\u00fd\" a pod. Ka\u017ed\u00e1 udalos\u0165 pritom predstavuje p\u00e1r k\u013e\u00fa\u010d-hodnota.<\/p>\n<p><\/p>\n<p>Predvolene sa v\u0161etky udalosti rozde\u013euj\u00fa po part\u00edci\u00e1ch t\u00e9my round-robinom, ak nie je zadan\u00fd k\u013e\u00fa\u010d (str\u00e1ca sa poradie), a prostredn\u00edctvom MurmurHash (k\u013e\u00fa\u010d), ak je k\u013e\u00fa\u010d pr\u00edtomn\u00fd (poradie v r\u00e1mci jednej part\u00edcie).<\/p>\n<p><\/p>\n<p>Tu sa hne\u010f treba poznamena\u0165, \u017ee Kafka zaru\u010duje poradie udalost\u00ed iba v r\u00e1mci jednej part\u00edcie. Ale v skuto\u010dnosti to \u010dasto nie je probl\u00e9m. Napr\u00edklad, v\u0161etky zmeny toho ist\u00e9ho inzer\u00e1tu je mo\u017en\u00e9 zabezpe\u010di\u0165 pridan\u00edm do jednej part\u00edcie (\u010d\u00edm sa zachov\u00e1 poradie t\u00fdchto zmien v r\u00e1mci inzer\u00e1tu). Taktie\u017e je mo\u017en\u00e9 pren\u00e1\u0161a\u0165 poradie \u010d\u00edsla v jednom z pol\u00ed udalosti.<\/p>\n<p><\/p>\n<h3 id=\"consumer\">Consumer<\/h3>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Kafka a mikroservisy: preh\u013ead\" src=\"\/wp-content\/uploads\/2019\/09\/1f69a53dc19bb6c587883b8754cde29f.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Consumer je zodpovedn\u00fd za prij\u00edmanie \u00fadajov z Apache Kafka. Ak sa vr\u00e1time k uveden\u00e9mu pr\u00edkladu, consumerom m\u00f4\u017ee by\u0165 slu\u017eba moderovania. T\u00e1to slu\u017eba bude prihl\u00e1sen\u00e1 na t\u00e9mu slu\u017eby inzer\u00e1tov a pri objaven\u00ed nov\u00e9ho inzer\u00e1tu ho dostane a analyzuje na s\u00falad s niektor\u00fdmi stanoven\u00fdmi politikami.<\/p>\n<p><\/p>\n<p>Apache Kafka si pam\u00e4t\u00e1, ak\u00e9 posledn\u00e9 udalosti prijal consumer (na tento \u00fa\u010del sa pou\u017e\u00edva pomocn\u00e1 t\u00e9ma <code>__consumer__offsets<\/code>), \u010d\u00edm zaru\u010duje, \u017ee ak sa \u00faspe\u0161ne pre\u010d\u00edta, consumer nedostane t\u00fa ist\u00fa spr\u00e1vu dvakr\u00e1t. Av\u0161ak, ak sa pou\u017e\u00edva mo\u017enos\u0165 enable.auto.commit = true a cel\u00fa pr\u00e1cu na sledovanie polohy consumera v t\u00e9me prenech\u00e1me Kafke, m\u00f4\u017eeme <noindex><a rel=\"nofollow\" href=\"https:\/\/blog.newrelic.com\/engineering\/kafka-consumer-config-auto-commit-data-loss\/\">strati\u0165 \u00fadaje<\/a><\/noindex>. V produk\u010dnom k\u00f3de sa zvy\u010dajne poloha consumeru kontroluje manu\u00e1lne (v\u00fdvoj\u00e1r riadi okamih, kedy mus\u00ed nevyhnutne d\u00f4js\u0165 k commitnuti pre\u010d\u00edtanej udalosti).<\/p>\n<p><\/p>\n<p>V pr\u00edpadoch, ke\u010f je jeden consumer nedostato\u010dn\u00fd (napr\u00edklad pr\u00fad nov\u00fdch udalost\u00ed je ve\u013emi ve\u013ek\u00fd), je mo\u017en\u00e9 prida\u0165 nieko\u013eko \u010fal\u0161\u00edch consumerov a spoji\u0165 ich do consumer group. Consumer group logicky predstavuje presne tak\u00e9ho ist\u00e9ho consumer, ale s rozdelen\u00edm \u00fadajov medzi \u00fa\u010dastn\u00edkov skupiny. To umo\u017e\u0148uje ka\u017ed\u00e9mu z \u00fa\u010dastn\u00edkov vzia\u0165 si svoju \u010das\u0165 spr\u00e1v, \u010d\u00edm sa zvy\u0161uje r\u00fdchlos\u0165 \u010d\u00edtania. <\/p>\n<p><\/p>\n<h1 id=\"rezultaty-testirovaniya\">V\u00fdsledky testovania<\/h1>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Kafka a mikroservisy: preh\u013ead\" src=\"\/wp-content\/uploads\/2019\/09\/f7abe4bcf34fdb0d762c8a928bd873ee.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Tu nebudem p\u00edsa\u0165 ve\u013ea vysvet\u013euj\u00faceho textu, len sa podel\u00edm o dosiahnut\u00e9 v\u00fdsledky. Testovanie prebiehalo na 3 fyzick\u00fdch strojoch (12 CPU, 384 GB RAM, 15k SAS DISK, 10 GBit\/s Net), brokeri a zookeeper boli nasaden\u00ed v lxc. <\/p>\n<p><\/p>\n<p><strong>Testovanie v\u00fdkonu<\/strong><\/p>\n<p><\/p>\n<p>Po\u010das testovania boli z\u00edskan\u00e9 nasleduj\u00face v\u00fdsledky.<\/p>\n<p><\/p>\n<ul>\n<li>R\u00fdchlos\u0165 z\u00e1pisu spr\u00e1v ve\u013ekosti 1 KB s\u00fa\u010dasne 9 producermi \u2013 1 300 000 udalost\u00ed za sekundu.<\/li>\n<li>R\u00fdchlos\u0165 \u010d\u00edtania spr\u00e1v ve\u013ekosti 1 KB s\u00fa\u010dasne 9 consumerami \u2013 1 500 000 udalost\u00ed za sekundu.<\/li>\n<\/ul>\n<p><\/p>\n<p><strong>Testovanie odolnosti vo\u010di poruch\u00e1m<\/strong><\/p>\n<p><\/p>\n<p>Po\u010das testovania boli z\u00edskan\u00e9 nasleduj\u00face v\u00fdsledky (3 brokery, 3 zookeeperi).<\/p>\n<p><\/p>\n<ul>\n<li>Ne\u0161tandardn\u00e9 ukon\u010denie jedn\u00e9ho z brokerov nevedie k zastaveniu alebo nedostupnosti klastra. Pr\u00e1ca pokra\u010duje v norm\u00e1lnom re\u017eime, ale na zost\u00e1vaj\u00face brokery sa kladie v\u00e4\u010d\u0161ia z\u00e1\u0165a\u017e.<\/li>\n<li>Ne\u010dakan\u00e9 ukon\u010denie dvoch brokerov v pr\u00edpade klastra z troch brokerov a min.isr = 2 vedie k nedostupnosti klastra pre z\u00e1pis, ale dostupnosti na \u010d\u00edtanie. V pr\u00edpade, ak je min.isr = 1, klaster zost\u00e1va dostupn\u00fd ako pre \u010d\u00edtanie, tak aj pre z\u00e1pis. Napriek tomu tento re\u017eim odporuje po\u017eiadavke na vysok\u00fa zachovanos\u0165 d\u00e1t.<\/li>\n<li>Ne\u010dakan\u00e9 ukon\u010denie jedn\u00e9ho zo serverov Zookeeper nevedie k zastaveniu alebo nedostupnosti klastra. Pr\u00e1ca pokra\u010duje v norm\u00e1lnom re\u017eime.<\/li>\n<li>Ne\u010dakan\u00e9 ukon\u010denie dvoch serverov Zookeeper vedie k nedostupnosti klastra do okamihu obnovenia \u010dinnosti aspo\u0148 jedn\u00e9ho zo serverov Zookeeper. Toto tvrdenie plat\u00ed pre klaster Zookeeper zo 3 serverov. V d\u00f4sledku v\u00fdskumov sa rozhodlo zv\u00fd\u0161i\u0165 klaster Zookeeper na 5 serverov na zv\u00fd\u0161enie odolnosti vo\u010di poruch\u00e1m. <br clear=\"left\">\n <\/li>\n<\/ul>\n<p><\/p>\n<h1 id=\"kafka-as-a-service\">Kafka ako slu\u017eba<\/h1>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Kafka a mikroservisy: preh\u013ead\" src=\"\/wp-content\/uploads\/2019\/09\/19b2356131203cefbadfd90a25ae9674.png\" style=\"display:block;margin: 0 auto;\" \/> <\/p>\n<p><\/p>\n<p>Ujasnili sme si, \u017ee Kafka je vynikaj\u00faca technol\u00f3gia, ktor\u00e1 n\u00e1m umo\u017e\u0148uje vyrie\u0161i\u0165 predlo\u017een\u00fa \u00falohu (implement\u00e1ciu brokeru spr\u00e1v). Napriek tomu sme sa rozhodli zak\u00e1za\u0165 slu\u017eb\u00e1m priamy pr\u00edstup k Kafke a uzavreli sme ju nad slu\u017ebou data-bus. Pre\u010do sme to urobili? V skuto\u010dnosti existuje nieko\u013eko d\u00f4vodov.<\/p>\n<p><\/p>\n<ul>\n<li>\n<p>Data-bus prevzal na seba v\u0161etky \u00falohy s\u00favisiace s integr\u00e1ciou s Kafkou (implement\u00e1cia a nastavenie konsumentov a producentov, monitoring, alerting, logovanie, \u0161k\u00e1lovanie at\u010f.). Takto je integr\u00e1cia s brokerom spr\u00e1v maxim\u00e1lne jednoduch\u00e1. <\/p>\n<p>\n<\/li>\n<li>\n<p>Data-bus umo\u017enil abstrahova\u0165 od konkr\u00e9tneho jazyka alebo kni\u017enice na pr\u00e1cu s Kafkou. <\/p>\n<p>\n<\/li>\n<li>\n<p>Data-bus umo\u017enil in\u00fdm slu\u017eb\u00e1m abstrahova\u0165 od \u00falo\u017en\u00e9ho priestoru. Mo\u017eno, v nejakom okamihu vymen\u00edme Kafku za Pulsar, a nikto si toho ni\u010d nev\u0161imne (v\u0161etky slu\u017eby poznaj\u00fa len API data-bus).<\/p>\n<p>\n<\/li>\n<li>\n<p>Data-bus prevzal na seba valid\u00e1ciu sch\u00e9m udalost\u00ed.<\/p>\n<p>\n<\/li>\n<li>\n<p>S pomocou data-bus bola implementovan\u00e1 autentifik\u00e1cia.<\/p>\n<p>\n<\/li>\n<li>\n<p>Pod z\u00e1\u0161titou data-bus m\u00f4\u017eeme bez prestoja, nepozorovane aktualizova\u0165 verzie Kafky, centralizovane spravova\u0165 konfigur\u00e1cie producentov, konsumentov, brokerov at\u010f.<\/p>\n<p>\n<\/li>\n<li>\n<p>Data-bus n\u00e1m umo\u017enil prid\u00e1va\u0165 potrebn\u00e9 funkcie, ktor\u00e9 v Kafke nie s\u00fa (ako audit t\u00e9m, kontrola anom\u00e1li\u00ed v klastri, vytv\u00e1ranie DLQ at\u010f.).<\/p>\n<p>\n<\/li>\n<li>\n<p>Data-bus umo\u017e\u0148uje centralizovane realizova\u0165 failover pre v\u0161etky slu\u017eby.<\/p>\n<p>\n<\/li>\n<\/ul>\n<p><\/p>\n<p>Na za\u010diatku sta\u010d\u00ed pripoji\u0165 mal\u00fa kni\u017enicu do k\u00f3du svojho servisu pre za\u010datie odosielania udalost\u00ed do spr\u00e1vy. To je v\u0161etko. Z\u00edskate mo\u017enos\u0165 p\u00edsa\u0165, \u010d\u00edta\u0165 a \u0161k\u00e1lova\u0165 jedin\u00fdm riadkom k\u00f3du. Cel\u00e1 realiz\u00e1cia je skryt\u00e1, von sa dost\u00e1va len nieko\u013eko ovl\u00e1dac\u00edch prvkov typu ve\u013ekosti d\u00e1vky. Pod kapotou slu\u017eba data-bus sp\u00fa\u0161\u0165a v Kubernetes potrebn\u00fd po\u010det in\u0161tanci\u00ed producerov a consumerov a poskytuje im potrebn\u00fa konfigur\u00e1ciu, ale v\u0161etko to je pre v\u00e1\u0161 servis transparentn\u00e9. <\/p>\n<p><\/p>\n<p>Samozrejme, niet striebornej gu\u013eky a tento pr\u00edstup m\u00e1 svoje obmedzenia.<\/p>\n<p><\/p>\n<ul>\n<li>Data-bus je potrebn\u00e9 udr\u017eiava\u0165 svojpomocne, na rozdiel od extern\u00fdch kni\u017en\u00edc.<\/li>\n<li>Data-bus zvy\u0161uje po\u010det interakci\u00ed medzi slu\u017ebami a brokerom spr\u00e1v, \u010do vedie k zn\u00ed\u017eeniu efektivity v porovnan\u00ed s holou Kafka.<\/li>\n<li>Nie v\u0161etko sa d\u00e1 tak jednoducho skry\u0165 pred slu\u017ebami, kop\u00edrova\u0165 funk\u010dnos\u0165 KSQL alebo Kafka Streams do data-bus sa n\u00e1m nechce, preto niekedy mus\u00edme povoli\u0165 slu\u017eb\u00e1m \u00eds\u0165 priamo. <\/li>\n<\/ul>\n<p><\/p>\n<p>V na\u0161om pr\u00edpade preva\u017eovali plusy nad minusmi a rozhodnutie skry\u0165 broker spr\u00e1v pod samostatn\u00fa slu\u017ebu sa uk\u00e1zalo ako ospravedlnite\u013en\u00e9. Po roku prev\u00e1dzky sme nemali \u017eiadne v\u00e1\u017ene hav\u00e1rie ani probl\u00e9my.<\/p>\n<p><\/p>\n<p>P.S. \u010eakujem svojej priate\u013eke, Ekaterine Obaljajevovej, za skvel\u00e9 obr\u00e1zky k tomuto \u010dl\u00e1nku. Ak sa v\u00e1m p\u00e1\u010dili, <noindex><a rel=\"nofollow\" href=\"https:\/\/www.instagram.com\/o.k_o.k_\/\">tu<\/a><\/noindex> n\u00e1jdete e\u0161te viac ilustr\u00e1ci\u00ed.<\/p>\n<p>Zdroj: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/avito\/blog\/465315\/\">habr.com<\/a><\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u0412\u0441\u0435\u043c \u043f\u0440\u0438\u0432\u0435\u0442. \u0412 \u044d\u0442\u043e\u0439 \u0441\u0442\u0430\u0442\u044c\u0435 \u044f \u0440\u0430\u0441\u0441\u043a\u0430\u0436\u0443, \u043f\u043e\u0447\u0435\u043c\u0443 \u043c\u044b \u0432 \u0410\u0432\u0438\u0442\u043e \u0434\u0435\u0432\u044f\u0442\u044c \u043c\u0435\u0441\u044f\u0446\u0435\u0432 \u043d\u0430\u0437\u0430\u0434 \u0432\u044b\u0431\u0440\u0430\u043b\u0438 Kafka, \u0438 \u0447\u0442\u043e \u043e\u043d\u0430 \u0438\u0437 \u0441\u0435\u0431\u044f \u043f\u0440\u0435\u0434\u0441\u0442\u0430\u0432\u043b\u044f\u0435\u0442. \u041f\u043e\u0434\u0435\u043b\u044e\u0441\u044c \u043e\u0434\u043d\u0438\u043c \u0438\u0437 \u043a\u0435\u0439\u0441\u043e\u0432 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u043d\u0438\u044f \u2014 \u0431\u0440\u043e\u043a\u0435\u0440 \u0441\u043e\u043e\u0431\u0449\u0435\u043d\u0438\u0439. \u0418 \u043d\u0430\u043f\u043e\u0441\u043b\u0435\u0434\u043e\u043a \u043f\u043e\u0433\u043e\u0432\u043e\u0440\u0438\u043c \u043e \u0442\u043e\u043c, \u043a\u0430\u043a\u0438\u0435 \u043f\u043b\u044e\u0441\u044b \u043c\u044b \u043f\u043e\u043b\u0443\u0447\u0438\u043b\u0438 \u043e\u0442 \u043f\u0440\u0438\u043c\u0435\u043d\u0435\u043d\u0438\u044f \u043f\u043e\u0434\u0445\u043e\u0434\u0430 Kafka as a Service. \u041f\u0440\u043e\u0431\u043b\u0435\u043c\u0430 \u0414\u043b\u044f \u043d\u0430\u0447\u0430\u043b\u0430 \u043d\u0435\u043c\u043d\u043e\u0433\u043e \u043a\u043e\u043d\u0442\u0435\u043a\u0441\u0442\u0430. \u041d\u0435\u043a\u043e\u0442\u043e\u0440\u043e\u0435 \u0432\u0440\u0435\u043c\u044f \u043d\u0430\u0437\u0430\u0434 \u043c\u044b [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":28406,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-37849","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.3 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u0412\u0441\u0435\u043c \u043f\u0440\u0438\u0432\u0435\u0442.\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/kafka-i-mikroservisy-obzor\" \/>\n\t\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.3\" \/>\n\t\t<meta property=\"og:locale\" content=\"sk_SK\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47Kafka \u0438 \u043c\u0438\u043a\u0440\u043e\u0441\u0435\u0440\u0432\u0438\u0441\u044b: \u043e\u0431\u0437\u043e\u0440 | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u0412\u0441\u0435\u043c \u043f\u0440\u0438\u0432\u0435\u0442.\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/kafka-i-mikroservisy-obzor\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2019-10-31T19:20:09+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2019-10-31T19:20:09+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47Kafka a mikroslu\u017eby: preh\u013ead | ProHoster","description":"Ahojte v\u0161etci.","canonical_url":"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/kafka-i-mikroservisy-obzor","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"sk_SK","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47Kafka \u0438 \u043c\u0438\u043a\u0440\u043e\u0441\u0435\u0440\u0432\u0438\u0441\u044b: \u043e\u0431\u0437\u043e\u0440 | ProHoster","og:description":"\u0412\u0441\u0435\u043c \u043f\u0440\u0438\u0432\u0435\u0442.","og:url":"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/kafka-i-mikroservisy-obzor","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2019-10-31T19:20:09+00:00","article:modified_time":"2019-10-31T19:20:09+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"37849","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":"2026-01-23 19:31:03","breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-03-01 01:19:23","updated":"2026-01-23 19:31:03","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/posts\/37849","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/comments?post=37849"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/posts\/37849\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/media\/28406"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/media?parent=37849"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/categories?post=37849"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/tags?post=37849"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}