{"id":37849,"date":"2019-10-31T22:20:09","date_gmt":"2019-10-31T19:20:09","guid":{"rendered":"https:\/\/prohoster.info\/blog\/kafka-i-mikroservisy-obzor\/"},"modified":"2019-10-31T22:20:09","modified_gmt":"2019-10-31T19:20:09","slug":"kafka-i-mikroservisy-obzor","status":"publish","type":"post","link":"https:\/\/prohoster.info\/ro\/blog\/administrirovanie\/kafka-i-mikroservisy-obzor","title":{"rendered":"Kafka \u0219i microservicii: o prezentare general\u0103","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p><img decoding=\"async\" alt=\"Kafka \u0219i microservicii: o prezentare general\u0103\" src=\"\/wp-content\/uploads\/2019\/09\/ab0534ad5d28b3c3aa03e4e0b2d3101b.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Bun\u0103 ziua tuturor. \u00cen acest articol, voi explica de ce noi, la Avito, am ales Kafka acum nou\u0103 luni \u0219i ce reprezint\u0103 acesta. Voi \u00eemp\u0103rt\u0103\u0219i unul dintre cazurile de utilizare \u2014 broker de mesaje. La final, vom discuta despre beneficiile pe care le-am ob\u021binut din aplicarea abord\u0103rii Kafka as a Service.<\/p>\n<p><noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><\/p>\n<h1 id=\"problema\">Problema<\/h1>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Kafka \u0219i microservicii: o prezentare general\u0103\" src=\"\/wp-content\/uploads\/2019\/09\/dfa5e231847cee67259da17867a11812.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Pentru \u00eenceput, pu\u021bin context. Cu ceva timp \u00een urm\u0103, am \u00eenceput s\u0103 ne \u00eendep\u0103rt\u0103m de arhitectura monolitic\u0103, iar acum, la Avito, exist\u0103 deja c\u00e2teva sute de servicii diferite. Acestea au propriile stoc\u0103ri, propriul stiv\u0103 de tehnologii \u0219i sunt responsabile pentru partea lor de logic\u0103 de afaceri. <\/p>\n<p><\/p>\n<p>Una dintre problemele cu un num\u0103r mare de servicii este comunicarea. Serviciul A dore\u0219te adesea s\u0103 afle informa\u021bii pe care le are serviciul B. \u00cen acest caz, serviciul A se adreseaz\u0103 serviciului B printr-o API sincron\u0103. Serviciul C vrea s\u0103 \u0219tie ce se \u00eent\u00e2mpl\u0103 \u00een cazul serviciilor G \u0219i D, iar acestea, la r\u00e2ndul lor, sunt interesate de serviciile A \u0219i B. Pe m\u0103sur\u0103 ce num\u0103rul acestor servicii \"curioase\" cre\u0219te, leg\u0103turile dintre ele devin un ghem complicat.<\/p>\n<p><\/p>\n<p>\u00cen acela\u0219i timp, \u00een orice moment, serviciul A poate deveni indisponibil. \u0218i ce ar trebui s\u0103 fac\u0103 serviciul B \u0219i toate celelalte servicii legate de acesta? Iar dac\u0103 pentru a efectua o opera\u021biune de afaceri trebuie s\u0103 se fac\u0103 o serie de apeluri sincrone consecutive, probabilitatea ca \u00eentreaga opera\u021biune s\u0103 e\u0219ueze cre\u0219te \u0219i mai mult (iar aceasta devine mai mare cu c\u00e2t aceast\u0103 serie este mai lung\u0103). <\/p>\n<p><\/p>\n<h1 id=\"vybor-tehnologii\">Alegerea tehnologiei<\/h1>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Kafka \u0219i microservicii: o prezentare general\u0103\" src=\"\/wp-content\/uploads\/2019\/09\/3acd7bcf3dded8093bdef6d627eb4071.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Bine, problemele sunt clare. Acestea pot fi rezolvate prin crearea unei sisteme centralizate de schimb de mesaje \u00eentre servicii. Acum, fiecare serviciu trebuie s\u0103 \u0219tie doar despre aceast\u0103 sistem\u0103 de schimb de mesaje. \u00cen plus, sistemul \u00eensu\u0219i trebuie s\u0103 fie tolerant la erori \u0219i scalabil orizontal, precum \u0219i, \u00een caz de avarii, s\u0103 acumuleze un buffer de apel pentru procesarea ulterioar\u0103. <\/p>\n<p><\/p>\n<p>Acum s\u0103 alegem tehnologia pe care va fi implementat\u0103 livrarea mesajelor. Pentru aceasta, mai \u00eent\u00e2i s\u0103 \u00een\u021belegem ce a\u0219tept\u0103m de la ea:<\/p>\n<p><\/p>\n<ul>\n<li>mesajele \u00eentre servicii nu trebuie s\u0103 se piard\u0103;<\/li>\n<li>mesajele pot fi duplicate;<\/li>\n<li>mesajele pot fi stocate \u0219i citite pentru o durat\u0103 de c\u00e2teva zile (buffer persistent);<\/li>\n<li>serviciile se pot abona la datele care le intereseaz\u0103;<\/li>\n<li>mai multe servicii pot citi acelea\u0219i date;<\/li>\n<li>mesajele pot con\u021bine un payload detaliat \u0219i voluminos (transfer de stare purtat de evenimente);<\/li>\n<li>uneori este necesar\u0103 o garan\u021bie a ordinii mesajelor.<\/li>\n<\/ul>\n<p><\/p>\n<p>De asemenea, a fost esen\u021bial pentru noi s\u0103 alegem un sistem extrem de scalabil \u0219i de \u00eencredere, cu o capacitate mare de procesare (de cel pu\u021bin 100k mesaje de c\u00e2\u021biva kilobi\u021bi pe secund\u0103).<\/p>\n<p><\/p>\n<p>\u00cen aceast\u0103 etap\u0103, ne-am desp\u0103r\u021bit de RabbitMQ (dificil de men\u021binut stabil la rate mari de procesare), PGQ de la SkyTools (prea lent \u0219i slab scalabil) \u0219i NSQ (nepermanent). Toate aceste tehnologii sunt utilizate \u00een compania noastr\u0103, dar nu se potriveau cu sarcina pe care trebuia s\u0103 o rezolv\u0103m. <\/p>\n<p><\/p>\n<p>Apoi am \u00eenceput s\u0103 explor\u0103m tehnologii noi pentru noi - Apache Kafka, Apache Pulsar \u0219i NATS Streaming.<\/p>\n<p><\/p>\n<p>Primul pe care l-am exclus a fost Pulsar. Am decis c\u0103 Kafka \u0219i Pulsar sunt solu\u021bii destul de asem\u0103n\u0103toare. \u0218i, de\u0219i Pulsar a fost testat de companii mari, este mai nou \u0219i ofer\u0103 laten\u021be mai mici (\u00een teorie), am decis s\u0103 p\u0103str\u0103m Kafka din cele dou\u0103, ca standard de facto pentru astfel de sarcini. Probabil ne vom \u00eentoarce la Apache Pulsar \u00een viitor. <\/p>\n<p><\/p>\n<p>\u0218i iat\u0103 c\u0103 au r\u0103mas doi candida\u021bi: NATS Streaming \u0219i Apache Kafka. Am studiat destul de detaliat ambele solu\u021bii, iar ambele s-au potrivit pentru sarcin\u0103. Dar, \u00een cele din urm\u0103, ne-am temut de t\u00e2n\u0103rul NATS Streaming (\u0219i de faptul c\u0103 unul dintre principalii dezvoltatori, Tyler Treat, a decis s\u0103 p\u0103r\u0103seasc\u0103 proiectul \u0219i s\u0103 \u00eenceap\u0103 unul propriu - Liftbridge). De asemenea, modul de Clustering al NATS Streaming nu oferea posibilitatea de scalare orizontal\u0103 puternic\u0103 (probabil c\u0103 aceasta nu mai este o problem\u0103 dup\u0103 ad\u0103ugarea modului de partitioning \u00een 2017).<\/p>\n<p><\/p>\n<p>Cu toate acestea, NATS Streaming este o tehnologie grozav\u0103, scris\u0103 \u00een Go \u0219i av\u00e2nd suport din partea Cloud Native Computing Foundation. Spre deosebire de Apache Kafka, nu are nevoie de Zookeeper pentru a func\u021biona (posibil, <noindex><a rel=\"nofollow\" href=\"https:\/\/cwiki.apache.org\/confluence\/display\/KAFKA\/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum\">s\u0103 se poat\u0103 spune acela\u0219i lucru \u0219i despre Kafka<\/a><\/noindex>), deoarece interiorul s\u0103u implementa RAFT. \u00cen plus, NATS Streaming este mai simplu de administrat. Nu excluzem c\u0103 ne vom \u00eentoarce la aceast\u0103 tehnologie \u00een viitor. <\/p>\n<p><\/p>\n<p>Cu toate acestea, ast\u0103zi c\u00e2\u0219tig\u0103torul nostru este Apache Kafka. \u00cen testele noastre, a demonstrat a fi suficient de rapid (peste un milion de mesaje pe secund\u0103 la citire \u0219i la scriere pentru un volum de mesaj de 1 kilobit), suficient de fiabil, bine scalabil \u0219i dovedit prin experien\u021b\u0103 \u00een produc\u021bie de companii mari. \u00cen plus, Kafka este sus\u021binut de cel pu\u021bin c\u00e2teva companii comerciale mari (noi, de exemplu, folosim versiunea Confluent), iar Kafka are un ecosistem dezvoltat.<br \/>\n<br clear=\"left\">\n <\/p>\n<p><\/p>\n<h1 id=\"obzor-kafka\">Prezentare general\u0103 Kafka<\/h1>\n<p><\/p>\n<p>\u00cenainte de a \u00eencepe, v\u0103 recomand din start o carte excelent\u0103 - <em>\u201eKafka: The Definitive Guide\u201d<\/em> (exist\u0103 \u0219i \u00een traducerea \u00een rus\u0103, dar termenii sunt pu\u021bin derutanti). \u00cen ea pute\u021bi g\u0103si informa\u021bii necesare pentru o \u00een\u021belegere de baz\u0103 a Kafka \u0219i chiar pu\u021bin mai mult. Documenta\u021bia de la Apache \u0219i blogul de la Confluent sunt, de asemenea, foarte bine scrise \u0219i u\u0219or de citit. <\/p>\n<p><\/p>\n<p>A\u0219adar, s\u0103 arunc\u0103m o privire asupra modului \u00een care este organizat\u0103 Kafka dintr-o perspectiv\u0103 de ansamblu. Topologia de baz\u0103 a Kafka const\u0103 din producer, consumer, broker \u0219i zookeeper.<\/p>\n<p><\/p>\n<h3 id=\"broker\">Broker<\/h3>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Kafka \u0219i microservicii: o prezentare general\u0103\" src=\"\/wp-content\/uploads\/2019\/09\/82f34b5a1aadcdc07eef58eb5414b553.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Brokerul (broker) este responsabil pentru stocarea datelor dumneavoastr\u0103. Toate datele sunt stocate \u00een form\u0103 binar\u0103, iar brokerul \u0219tie pu\u021bin despre ce reprezint\u0103 acestea \u0219i care este structura lor. <\/p>\n<p><\/p>\n<p>Fiecare tip logic de eveniment se afl\u0103, de obicei, \u00eentr-un topic separat (topic). De exemplu, un eveniment de creare a unei anun\u021buri poate ajunge \u00een topicul item.created, iar un eveniment de modificare a acestuia \u2013 \u00een item.changed. Topicurile pot fi considerate clasificatori de evenimente. La nivel de topic, se pot seta urm\u0103torii parametri de configurare: <\/p>\n<p><\/p>\n<ul>\n<li>volumul datelor stocate \u0219i\/sau vechimea acestora (retention.bytes, retention.ms); <\/li>\n<li>factorul de redundan\u021b\u0103 a datelor (replication factor);<\/li>\n<li>dimensiunea maxim\u0103 a unui mesaj (max.message.bytes);<\/li>\n<li>num\u0103rul minim de replici sincronizate, \u00een care se pot scrie date \u00een topic (min.insync.replicas);<\/li>\n<li>posibilitatea de a efectua failover pe o replic\u0103 \u00eent\u00e2rziat\u0103 nesincronizat\u0103 cu o poten\u021bial\u0103 pierdere de date (unclean.leader.election.enable);<\/li>\n<li>\u0219i multe altele (<noindex><a rel=\"nofollow\" href=\"https:\/\/kafka.apache.org\/documentation\/#topicconfigs\">https:\/\/kafka.apache.org\/documentation\/#topicconfigs<\/a><\/noindex>).<\/li>\n<\/ul>\n<p><\/p>\n<p>La r\u00e2ndul s\u0103u, fiecare topic este \u00eemp\u0103r\u021bit \u00een una sau mai multe parti\u021bii (partition). Evenimentele ajung \u00een cele din urm\u0103 \u00een parti\u021bii. Dac\u0103 \u00een cluster exist\u0103 mai mult de un broker, parti\u021biile vor fi distribuite uniform \u00eentre to\u021bi brokerii (c\u00e2t mai mult posibil), ceea ce va permite scalarea sarcinii de scriere \u0219i citire pe un topic simultan pe mai mul\u021bi brokeri.<\/p>\n<p><\/p>\n<p>Pe disc, datele pentru fiecare parti\u021bie sunt stocate sub form\u0103 de fi\u0219iere de segmente, care, \u00een mod implicit, au dimensiunea de un gigabyte (controlat prin log.segment.bytes). O caracteristic\u0103 important\u0103 este c\u0103 \u0219tergerea datelor din parti\u021bii (la activarea retention) se face exact pe segmente (nu se poate \u0219terge un singur eveniment dintr-o parti\u021bie, se poate \u0219terge doar un \u00eentreg segment, \u0219i anume unul neactiv).<\/p>\n<p><\/p>\n<h3 id=\"zookeeper\">Zookeeper<\/h3>\n<p><\/p>\n<p>Zookeeper joac\u0103 rolul de depozit pentru metadate \u0219i coordonator. El este capabil s\u0103 spun\u0103 dac\u0103 brokerii sunt activi (o pute\u021bi observa din perspectiva zookeeper-ului utiliz\u00e2nd comanda zookeeper-shell <code>ls \/brokers\/ids<\/code>), care dintre brokeri este controller (<code>get \/controller<\/code>), sunt partitiile \u00een stare sincronizat\u0103 cu replicile lor (<code>get \/brokers\/topics\/topic_name\/partitions\/partition_number\/state<\/code>). De asemenea, produc\u0103torul \u0219i consumatorul vor comunica \u00eent\u00e2i cu zookeeper pentru a afla pe care broker sunt stocate anumite topicuri \u0219i parti\u021bii. \u00cen cazurile \u00een care un topic are un factor de replicare mai mare de 1, zookeeper va indica care parti\u021bii sunt lideri (\u00een care se va efectua scrierea \u0219i din care se va citi). \u00cen cazul c\u0103derii broker-ului, informa\u021biile despre noile parti\u021bii lider vor fi scrise exact \u00een zookeeper (\u00eencep\u00e2nd din versiunea 1.1.0, asincron, <noindex><a rel=\"nofollow\" href=\"https:\/\/www.confluent.io\/blog\/apache-kafka-supports-200k-partitions-per-cluster\">\u0219i este important<\/a><\/noindex>).<\/p>\n<p><\/p>\n<p>\u00cen versiunile mai vechi ale Kafka, zookeeper era responsabil \u0219i pentru stocarea offset-urilor, dar acum acestea sunt stocate \u00eentr-un topic special <code>__consumer_offsets<\/code> pe broker (de\u0219i pute\u021bi utiliza \u00een continuare zookeeper \u00een aceste scopuri). <\/p>\n<p><\/p>\n<p>Cel mai simplu mod de a transforma datele dvs. \u00een dovleci este chiar pierderea informa\u021biilor din zookeeper. \u00centr-un astfel de scenariu, va fi foarte greu s\u0103 \u00een\u021belege\u021bi ce \u0219i de unde trebuie s\u0103 citi\u021bi. <\/p>\n<p><\/p>\n<h3 id=\"producer\">Producer<\/h3>\n<p><\/p>\n<p>Producer-ul este cel mai adesea un serviciu care efectueaz\u0103 scrierea direct\u0103 a datelor \u00een Apache Kafka. Producer-ul alege topic-ul \u00een care vor fi stocate mesajele sale tematice \u0219i \u00eencepe s\u0103 scrie informa\u021bii \u00een acesta. De exemplu, producer-ul poate fi un serviciu de anun\u021buri. \u00cen acest caz, va trimite \u00een topicurile tematice evenimente precum \u201eanun\u021b creat\u201d, \u201eanun\u021b actualizat\u201d, \u201eanun\u021b \u0219ters\u201d etc. Fiecare eveniment reprezint\u0103 o pereche cheie-valoare.<\/p>\n<p><\/p>\n<p>Implicit, toate evenimentele sunt distribuite pe parti\u021biile topic-ului folosind round-robin dac\u0103 cheia nu este specificat\u0103 (pierd\u00e2nd ordonarea), \u0219i prin MurmurHash (cheie), dac\u0103 cheia este prezent\u0103 (ordonarea \u00een cadrul unei singure parti\u021bii).<\/p>\n<p><\/p>\n<p>Aici trebuie men\u021bionat c\u0103 Kafka garanteaz\u0103 ordinea evenimentelor doar \u00een cadrul unei singure parti\u021bii. Totu\u0219i, de multe ori acest lucru nu reprezint\u0103 o problem\u0103. De exemplu, este posibil s\u0103 se adauge cu certitudine toate modific\u0103rile unui singur anun\u021b \u00eentr-o parti\u021bie (men\u021bin\u00e2nd astfel ordinea acestor modific\u0103ri \u00een cadrul anun\u021bului). De asemenea, se pot transmite numere de ordine \u00eentr-unul dintre c\u00e2mpurile evenimentului.<\/p>\n<p><\/p>\n<h3 id=\"consumer\">Consumer<\/h3>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Kafka \u0219i microservicii: o prezentare general\u0103\" src=\"\/wp-content\/uploads\/2019\/09\/1f69a53dc19bb6c587883b8754cde29f.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Consumerul este responsabil pentru ob\u021binerea datelor din Apache Kafka. Dac\u0103 revenim la exemplul de mai sus, consumerul poate fi un serviciu de moderare. Acest serviciu se va abona la topicul serviciului de anun\u021buri \u0219i, atunci c\u00e2nd apare un nou anun\u021b, \u00eel va primi \u0219i va analiza pentru a verifica dac\u0103 respect\u0103 anumite politici stabilite.<\/p>\n<p><\/p>\n<p>Apache Kafka re\u021bine ultimele evenimente primite de consumer (pentru aceasta se folose\u0219te un topic de servicii. <code>__consumer__offsets<\/code>), asigur\u00e2nd astfel c\u0103, la citirea cu succes, consumerul nu va primi acela\u0219i mesaj de dou\u0103 ori. Cu toate acestea, dac\u0103 se folose\u0219te op\u021biunea enable.auto.commit = true \u0219i se las\u0103 complet monitorizarea pozi\u021biei consumer-ului pe seama Kafka, este posibil s\u0103 <noindex><a rel=\"nofollow\" href=\"https:\/\/blog.newrelic.com\/engineering\/kafka-consumer-config-auto-commit-data-loss\/\">se piard\u0103 datele<\/a><\/noindex>. \u00cen codul de produc\u021bie, de obicei, pozi\u021bia consumer-ului este controlat\u0103 manual (dezvoltatorul gestioneaz\u0103 momentul \u00een care trebuie s\u0103 aib\u0103 loc un commit al evenimentului citit).<\/p>\n<p><\/p>\n<p>\u00cen cazurile \u00een care un singur consumer nu este suficient (de exemplu, fluxul de evenimente noi este foarte mare), se pot ad\u0103uga \u00eenc\u0103 c\u00e2\u021biva consumatori, leg\u00e2ndu-i \u00eempreun\u0103 \u00eentr-un consumer group. Consumer group reprezint\u0103 logic acela\u0219i consumer, dar cu distribuirea datelor \u00eentre participan\u021bii grupului. Aceasta permite fiec\u0103rui participant s\u0103 preia o parte din mesaje, astfel scal\u00e2nd viteza de citire. <\/p>\n<p><\/p>\n<h1 id=\"rezultaty-testirovaniya\">Rezultatele test\u0103rii<\/h1>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Kafka \u0219i microservicii: o prezentare general\u0103\" src=\"\/wp-content\/uploads\/2019\/09\/f7abe4bcf34fdb0d762c8a928bd873ee.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Aici nu voi scrie multe explica\u021bii, ci voi \u00eemp\u0103rt\u0103\u0219i doar rezultatele ob\u021binute. Testarea a fost realizat\u0103 pe 3 ma\u0219ini fizice (12 CPU, 384GB RAM, 15k SAS DISK, 10GBit\/s Net), brokerii \u0219i zookeeper au fost desf\u0103\u0219ura\u021bi \u00een lxc. <\/p>\n<p><\/p>\n<p><strong>Testarea performan\u021bei<\/strong><\/p>\n<p><\/p>\n<p>\u00cen timpul test\u0103rii au fost ob\u021binute urm\u0103toarele rezultate.<\/p>\n<p><\/p>\n<ul>\n<li>Viteza de scriere a mesajelor de 1KB simultan de c\u0103tre 9 produc\u0103tori \u2014 1300000 evenimente pe secund\u0103.<\/li>\n<li>Viteza de citire a mesajelor de 1KB simultan de c\u0103tre 9 consumatori \u2014 1500000 evenimente pe secund\u0103.<\/li>\n<\/ul>\n<p><\/p>\n<p><strong>Testarea rezilien\u021bei<\/strong><\/p>\n<p><\/p>\n<p>\u00cen timpul test\u0103rii au fost ob\u021binute urm\u0103toarele rezultate (3 brokeri, 3 zookeeper).<\/p>\n<p><\/p>\n<ul>\n<li>Defec\u021biunea unui broker nu duce la oprirea sau indisponibilitatea clusterului. Func\u021bionarea continu\u0103 \u00een mod normal, dar pe brokerii r\u0103ma\u0219i exist\u0103 o sarcin\u0103 mai mare.<\/li>\n<li>Terminarea neplanificat\u0103 a dou\u0103 brokeri \u00eentr-un cluster format din trei brokeri \u0219i min.isr = 2 duce la indisponibilitatea cluster-ului pentru scriere, dar r\u0103m\u00e2ne disponibil pentru citire. \u00cen cazul \u00een care min.isr = 1, cluster-ul r\u0103m\u00e2ne disponibil at\u00e2t pentru citire, c\u00e2t \u0219i pentru scriere. Totu\u0219i, acest mod contravine cerin\u021bei de \u00eenalt\u0103 p\u0103strare a datelor.<\/li>\n<li>Terminarea neplanificat\u0103 a unuia dintre serverele Zookeeper nu duce la oprirea sau indisponibilitatea cluster-ului. Func\u021bionarea continu\u0103 \u00een mod normal.<\/li>\n<li>Terminarea neplanificat\u0103 a dou\u0103 servere Zookeeper duce la indisponibilitatea cluster-ului p\u00e2n\u0103 la restabilirea func\u021bion\u0103rii cel pu\u021bin unui server Zookeeper. Aceast\u0103 afirma\u021bie este adev\u0103rat\u0103 pentru un cluster Zookeeper format din 3 servere. Ca urmare a cercet\u0103rilor, s-a decis extinderea cluster-ului Zookeeper la 5 servere pentru a cre\u0219te rezilien\u021ba. <br clear=\"left\">\n <\/li>\n<\/ul>\n<p><\/p>\n<h1 id=\"kafka-as-a-service\">Kafka ca serviciu<\/h1>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Kafka \u0219i microservicii: o prezentare general\u0103\" src=\"\/wp-content\/uploads\/2019\/09\/19b2356131203cefbadfd90a25ae9674.png\" style=\"display:block;margin: 0 auto;\" \/> <\/p>\n<p><\/p>\n<p>Am constatat c\u0103 Kafka este o tehnologie excelent\u0103 care ne permite s\u0103 rezolv\u0103m sarcina propus\u0103 (implementarea unui broker de mesaje). Cu toate acestea, am decis s\u0103 interzicem serviciilor accesul direct la Kafka \u0219i am \u00eenchis-o deasupra cu un serviciu data-bus. De ce am f\u0103cut acest lucru? De fapt, sunt c\u00e2teva motive.<\/p>\n<p><\/p>\n<ul>\n<li>\n<p>Data-bus a preluat toate sarcinile legate de integrarea cu Kafka (implementarea \u0219i configurarea consumatorilor \u0219i produc\u0103torilor, monitorizarea, alertarea, logarea, scalarea etc.). Astfel, integrarea cu brokerul de mesaje se face c\u00e2t mai simplu. <\/p>\n<p>\n<\/li>\n<li>\n<p>Data-bus a permis abstractizarea de un anumit limbaj sau bibliotec\u0103 pentru lucrul cu Kafka. <\/p>\n<p>\n<\/li>\n<li>\n<p>Data-bus a permis altor servicii s\u0103 se abstreze de stratul de stocare. Poate c\u0103, la un moment dat, vom schimba Kafka cu Pulsar, \u0219i nimeni nu va observa (toate serviciile cunosc doar API-ul data-bus).<\/p>\n<p>\n<\/li>\n<li>\n<p>Data-bus a preluat validarea schemelor de evenimente.<\/p>\n<p>\n<\/li>\n<li>\n<p>Cu ajutorul data-bus-ului a fost implementat\u0103 autentificarea.<\/p>\n<p>\n<\/li>\n<li>\n<p>Sub umbrela data-bus-ului, putem actualiza versiunile Kafka f\u0103r\u0103 downtime, neobservat, \u0219i gestiona centralizat configura\u021biile produc\u0103torilor, consumatorilor, brokerilor etc.<\/p>\n<p>\n<\/li>\n<li>\n<p>Data-bus ne-a permis s\u0103 ad\u0103ug\u0103m func\u021bionalit\u0103\u021bile necesare care nu sunt disponibile \u00een Kafka (precum auditul topicurilor, monitorizarea anomaliilor din cluster, crearea DLQ etc.).<\/p>\n<p>\n<\/li>\n<li>\n<p>Data-bus permite implementarea failover-ului \u00eentr-un mod centralizat pentru toate serviciile.<\/p>\n<p>\n<\/li>\n<\/ul>\n<p><\/p>\n<p>\u00cen prezent, pentru a \u00eencepe trimiterea de evenimente c\u0103tre brokerul de mesaje, este suficient s\u0103 conecta\u021bi o mic\u0103 bibliotec\u0103 \u00een codul serviciului dumneavoastr\u0103. At\u00e2t. Ave\u021bi posibilitatea de a scrie, citi \u0219i scala cu o singur\u0103 linie de cod. Toat\u0103 implementarea este ascuns\u0103 de ochii dumneavoastr\u0103, l\u0103s\u00e2nd la suprafa\u021b\u0103 doar c\u00e2teva manete de tipul dimensiunii batch-ului. Sub capot\u0103, serviciul data-bus ridic\u0103 \u00een Kubernetes num\u0103rul necesar de instan\u021be de produc\u0103tori \u0219i consumatori \u0219i le furnizeaz\u0103 configura\u021bia necesar\u0103, dar toate acestea sunt transparente pentru serviciul dumneavoastr\u0103. <\/p>\n<p><\/p>\n<p>Desigur, nu exist\u0103 o solu\u021bie universal\u0103 \u0219i aceast\u0103 abordare are propriile sale limit\u0103ri.<\/p>\n<p><\/p>\n<ul>\n<li>Data-bus trebuie sus\u021binut de sine, spre deosebire de bibliotecile externe.<\/li>\n<li>Data-bus cre\u0219te num\u0103rul de interac\u021biuni \u00eentre servicii \u0219i brokerul de mesaje, ceea ce duce la o sc\u0103dere a performan\u021bei \u00een compara\u021bie cu Kafka gol.<\/li>\n<li>Nu totul poate fi ascuns at\u00e2t de simplu de servicii, nu ne dorim s\u0103 duplic\u0103m func\u021bionalitatea KSQL sau Kafka Streams \u00een data-bus, de aceea uneori trebuie s\u0103 permitem serviciilor s\u0103 acceseze direct. <\/li>\n<\/ul>\n<p><\/p>\n<p>\u00cen cazul nostru, avantajele au dep\u0103\u0219it dezavantajele, iar solu\u021bia de a ascunde brokerul de mesaje sub un serviciu separat s-a dovedit a fi justificat\u0103. \u00centr-un an de utilizare, nu am avut accidente sau probleme grave.<\/p>\n<p><\/p>\n<p>P.S. Mul\u021bumesc iubitei mele, Ekaterina Obalaya, pentru imaginile grozave folosite \u00een acest articol. Dac\u0103 v-au pl\u0103cut, <noindex><a rel=\"nofollow\" href=\"https:\/\/www.instagram.com\/o.k_o.k_\/\">aici<\/a><\/noindex> vor mai exista \u0219i alte ilustra\u021bii.<\/p>\n<p>Sursa: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/avito\/blog\/465315\/\">habr.com<\/a><\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u0412\u0441\u0435\u043c \u043f\u0440\u0438\u0432\u0435\u0442. \u0412 \u044d\u0442\u043e\u0439 \u0441\u0442\u0430\u0442\u044c\u0435 \u044f \u0440\u0430\u0441\u0441\u043a\u0430\u0436\u0443, \u043f\u043e\u0447\u0435\u043c\u0443 \u043c\u044b \u0432 \u0410\u0432\u0438\u0442\u043e \u0434\u0435\u0432\u044f\u0442\u044c \u043c\u0435\u0441\u044f\u0446\u0435\u0432 \u043d\u0430\u0437\u0430\u0434 \u0432\u044b\u0431\u0440\u0430\u043b\u0438 Kafka, \u0438 \u0447\u0442\u043e \u043e\u043d\u0430 \u0438\u0437 \u0441\u0435\u0431\u044f \u043f\u0440\u0435\u0434\u0441\u0442\u0430\u0432\u043b\u044f\u0435\u0442. \u041f\u043e\u0434\u0435\u043b\u044e\u0441\u044c \u043e\u0434\u043d\u0438\u043c \u0438\u0437 \u043a\u0435\u0439\u0441\u043e\u0432 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u043d\u0438\u044f \u2014 \u0431\u0440\u043e\u043a\u0435\u0440 \u0441\u043e\u043e\u0431\u0449\u0435\u043d\u0438\u0439. \u0418 \u043d\u0430\u043f\u043e\u0441\u043b\u0435\u0434\u043e\u043a \u043f\u043e\u0433\u043e\u0432\u043e\u0440\u0438\u043c \u043e \u0442\u043e\u043c, \u043a\u0430\u043a\u0438\u0435 \u043f\u043b\u044e\u0441\u044b \u043c\u044b \u043f\u043e\u043b\u0443\u0447\u0438\u043b\u0438 \u043e\u0442 \u043f\u0440\u0438\u043c\u0435\u043d\u0435\u043d\u0438\u044f \u043f\u043e\u0434\u0445\u043e\u0434\u0430 Kafka as a Service. \u041f\u0440\u043e\u0431\u043b\u0435\u043c\u0430 \u0414\u043b\u044f \u043d\u0430\u0447\u0430\u043b\u0430 \u043d\u0435\u043c\u043d\u043e\u0433\u043e \u043a\u043e\u043d\u0442\u0435\u043a\u0441\u0442\u0430. \u041d\u0435\u043a\u043e\u0442\u043e\u0440\u043e\u0435 \u0432\u0440\u0435\u043c\u044f \u043d\u0430\u0437\u0430\u0434 \u043c\u044b [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":28406,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-37849","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.2.1 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u0412\u0441\u0435\u043c \u043f\u0440\u0438\u0432\u0435\u0442.\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/ro\/blog\/administrirovanie\/kafka-i-mikroservisy-obzor\" \/>\n\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.2.1\" \/>\n\t\t<meta property=\"og:locale\" content=\"ro_RO\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47Kafka \u0438 \u043c\u0438\u043a\u0440\u043e\u0441\u0435\u0440\u0432\u0438\u0441\u044b: \u043e\u0431\u0437\u043e\u0440 | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u0412\u0441\u0435\u043c \u043f\u0440\u0438\u0432\u0435\u0442.\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/ro\/blog\/administrirovanie\/kafka-i-mikroservisy-obzor\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2019-10-31T19:20:09+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2019-10-31T19:20:09+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47Kafka \u0219i microserviciile: recenzie | ProHoster","description":"Salutare tuturor.","canonical_url":"https:\/\/prohoster.info\/ro\/blog\/administrirovanie\/kafka-i-mikroservisy-obzor","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"ro_RO","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47Kafka \u0438 \u043c\u0438\u043a\u0440\u043e\u0441\u0435\u0440\u0432\u0438\u0441\u044b: \u043e\u0431\u0437\u043e\u0440 | ProHoster","og:description":"\u0412\u0441\u0435\u043c \u043f\u0440\u0438\u0432\u0435\u0442.","og:url":"https:\/\/prohoster.info\/ro\/blog\/administrirovanie\/kafka-i-mikroservisy-obzor","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2019-10-31T19:20:09+00:00","article:modified_time":"2019-10-31T19:20:09+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"37849","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":"2026-01-23 19:31:03","breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-03-01 01:19:23","updated":"2026-01-23 19:31:03","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/posts\/37849","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/comments?post=37849"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/posts\/37849\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/media\/28406"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/media?parent=37849"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/categories?post=37849"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/tags?post=37849"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}