{"id":37849,"date":"2019-10-31T22:20:09","date_gmt":"2019-10-31T19:20:09","guid":{"rendered":"https:\/\/prohoster.info\/blog\/kafka-i-mikroservisy-obzor\/"},"modified":"2019-10-31T22:20:09","modified_gmt":"2019-10-31T19:20:09","slug":"kafka-i-mikroservisy-obzor","status":"publish","type":"post","link":"https:\/\/prohoster.info\/lv\/blog\/administrirovanie\/kafka-i-mikroservisy-obzor","title":{"rendered":"Kafka un mikroservisi: p\u0101rskats","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p><img decoding=\"async\" alt=\"Kafka un mikroservisi: p\u0101rskats\" src=\"\/wp-content\/uploads\/2019\/09\/ab0534ad5d28b3c3aa03e4e0b2d3101b.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Sveiki visiem. \u0160aj\u0101 rakst\u0101 es izst\u0101st\u012b\u0161u, k\u0101p\u0113c m\u0113s Avito pirms devi\u0146iem m\u0113ne\u0161iem izv\u0113l\u0113j\u0101mies Kafka un ko t\u0101 sev\u012b ietver. Dal\u012b\u0161os ar vienu no lieto\u0161anas gad\u012bjumiem \u2014 zi\u0146ojumu brokeris. Un visbeidzot parun\u0101sim par to, k\u0101dus ieguvumus m\u0113s guv\u0101m, izmantojot Kafka k\u0101 servisu.<\/p>\n<p><noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><\/p>\n<h1 id=\"problema\">Probl\u0113ma<\/h1>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Kafka un mikroservisi: p\u0101rskats\" src=\"\/wp-content\/uploads\/2019\/09\/dfa5e231847cee67259da17867a11812.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Vispirms nedaudz konteksta. Pirms k\u0101da laika m\u0113s s\u0101k\u0101m atk\u0101pties no monol\u012btisk\u0101s arhitekt\u016bras, un pa\u0161laik Avito jau ir vair\u0101ki simti da\u017e\u0101du pakalpojumu. Katram no tiem ir savas datu glab\u0101\u0161anas sist\u0113mas, tehnolo\u0123iju kaudzes un tie atbild par savu biznesa lo\u0123ikas da\u013cu. <\/p>\n<p><\/p>\n<p>Viena no probl\u0113m\u0101m ar lielu pakalpojumu skaitu ir komunik\u0101cija. Pakalpojums A bie\u017ei v\u0113las uzzin\u0101t inform\u0101ciju, kas ir pakalpojum\u0101 B. \u0160aj\u0101 gad\u012bjum\u0101 pakalpojums A v\u0113r\u0161as pie pakalpojuma B caur sinhrono API. Pakalpojums B v\u0113las zin\u0101t, kas notiek pakalpojumos G un D, un tie, savuk\u0101rt, interes\u0113jas par pakalpojumiem A un B. Kad t\u0101du \"zi\u0146k\u0101r\u012bgu\" pakalpojumu k\u013c\u016bst daudz, saites starp tiem p\u0101rv\u0113r\u0161as par muls\u012bgu mezglu.<\/p>\n<p><\/p>\n<p>Turkl\u0101t jebkur\u0101 br\u012bd\u012b pakalpojums A var k\u013c\u016bt nepieejams. Un ko \u0161aj\u0101 gad\u012bjum\u0101 dar\u012bt pakalpojumam B un visiem citiem ar to saist\u012btajiem pakalpojumiem? Bet, ja biznesa oper\u0101cijas izpildei ir j\u0101veic sec\u012bga sinhrono izsaukumu \u0137\u0113de, visp\u0101r\u0113j\u0101s oper\u0101cijas atteik\u0161an\u0101s iesp\u0113ja k\u013c\u016bst v\u0113l augst\u0101ka (un t\u0101 ir jo augst\u0101ka, jo gar\u0101ka \u0161\u012b \u0137\u0113de). <\/p>\n<p><\/p>\n<h1 id=\"vybor-tehnologii\">Tehnolo\u0123ijas izv\u0113le<\/h1>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Kafka un mikroservisi: p\u0101rskats\" src=\"\/wp-content\/uploads\/2019\/09\/3acd7bcf3dded8093bdef6d627eb4071.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Labi, probl\u0113mas ir skaidras. Tos var nov\u0113rst, izveidojot centraliz\u0113tu zi\u0146ojumu apmai\u0146as sist\u0113mu starp pakalpojumiem. Tagad katram pakalpojumam ir j\u0101zina tikai par \u0161o zi\u0146ojumu apmai\u0146as sist\u0113mu. Turkl\u0101t pa\u0161ai sist\u0113mai j\u0101b\u016bt notur\u012bgai pret k\u013c\u016bd\u0101m un horizont\u0101li papla\u0161in\u0101mai, k\u0101 ar\u012b av\u0101rijas gad\u012bjum\u0101 j\u0101kopj sev\u012b piepras\u012bjumu buferis turpm\u0101kai apstr\u0101dei. <\/p>\n<p><\/p>\n<p>Tagad izv\u0113l\u0113simies tehnolo\u0123iju, uz kuras tiks \u012bstenota zi\u0146ojumu pieg\u0101de. Vispirms saprat\u012bsim, ko m\u0113s no t\u0101s sagaid\u0101m:<\/p>\n<p><\/p>\n<ul>\n<li>zi\u0146ojumi starp pakalpojumiem nedr\u012bkst pazust;<\/li>\n<li>zi\u0146ojumi var b\u016bt dubl\u0113ti;<\/li>\n<li>zi\u0146ojumus var uzglab\u0101t un las\u012bt vair\u0101kus dienas (notur\u012bgs buferis);<\/li>\n<li>pakalpojumi var parakst\u012bties uz interes\u0113jo\u0161iem datiem;<\/li>\n<li>vair\u0101ki pakalpojumi var las\u012bt tos pa\u0161us datus;<\/li>\n<li>zi\u0146ojumi var satur\u0113t detaliz\u0113tu, apjom\u012bgu payload (notikumiem pies\u0101tin\u0101ta st\u0101vok\u013ca p\u0101rvade);<\/li>\n<li>da\u017ereiz nepiecie\u0161ama zi\u0146ojumu sec\u012bbas garantija.<\/li>\n<\/ul>\n<p><\/p>\n<p>T\u0101pat mums bija kritiski svar\u012bgi izv\u0113l\u0113ties p\u0113c iesp\u0113jas m\u0113rogojam\u0101ku un uzticam\u0101ku sist\u0113mu ar augstu caurlaidsp\u0113ju (vismaz 100 t\u016bksto\u0161i zi\u0146ojumu, pa da\u017eiem kilobaitiem sekund\u0113).<\/p>\n<p><\/p>\n<p>\u0160aj\u0101 posm\u0101 m\u0113s \u0161\u0137\u012br\u0101mies no RabbitMQ (gr\u016bti notur\u0113t stabilu pie augstiem RPS), PGQ no SkyTools (pietiekami \u0101trs un slikti m\u0113rogojams) un NSQ (nepersist\u0113jo\u0161s). Visas \u0161\u012bs tehnolo\u0123ijas ir izmantotas m\u016bsu uz\u0146\u0113mum\u0101, bet \u0161ai uzdevumam t\u0101s nebija piem\u0113rotas. <\/p>\n<p><\/p>\n<p>T\u0101l\u0101k m\u0113s s\u0101k\u0101m apl\u016bkot mums jaunus tehnolo\u0123iskus risin\u0101jumus \u2014 Apache Kafka, Apache Pulsar un NATS Streaming.<\/p>\n<p><\/p>\n<p>Pulsar m\u0113s noraid\u012bj\u0101m pirmaj\u0101 k\u0101rt\u0101. M\u0113s nol\u0113m\u0101m, ka Kafka un Pulsar ir diezgan l\u012bdz\u012bgas sist\u0113mas. Un, lai gan Pulsar ir p\u0101rbaud\u012bts lielos uz\u0146\u0113mumos, ir jaun\u0101ks un teor\u0113tiski pied\u0101v\u0101 zem\u0101ku latentumu, m\u0113s no \u0161iem diviem izv\u0113l\u0113j\u0101mies Kafka, jo t\u0101 ir de facto standarts \u0161\u0101d\u0101m uzdevumu izpildei. Iesp\u0113jams, n\u0101kotn\u0113 m\u0113s atgriez\u012bsimies pie Apache Pulsar. <\/p>\n<p><\/p>\n<p>T\u0101d\u0113j\u0101di palika divi kandid\u0101ti: NATS Streaming un Apache Kafka. M\u0113s diezgan detaliz\u0113ti izp\u0113t\u012bj\u0101m abus risin\u0101jumus, un abi tie atbilda uzdevumam. Bet beig\u0101s m\u0113s baid\u012bj\u0101mies no attiec\u012bg\u0101s jaun\u012bbas NATS Streaming (un no t\u0101, ka viens no galvenajiem izstr\u0101d\u0101t\u0101jiem, Tyler Treat, nol\u0113ma izst\u0101ties no projekta un uzs\u0101kt savu \u2014 Liftbridge). Turkl\u0101t NATS Streaming klasteriz\u0101cijas re\u017e\u012bms nepied\u0101v\u0101ja iesp\u0113ju sp\u0113c\u012bgai horizont\u0101lai m\u0113rogai (iesp\u0113jams, tas jau nav probl\u0113ma p\u0113c da\u013cu re\u017e\u012bma pievieno\u0161anas 2017. gad\u0101).<\/p>\n<p><\/p>\n<p>Tom\u0113r NATS Streaming ir lieliska tehnolo\u0123ija, uzrakst\u012bta valod\u0101 Go un apstiprin\u0101ta Cloud Native Computing Foundation. At\u0161\u0137ir\u012bb\u0101 no Apache Kafka, tai nav nepiecie\u0161ams Zookeeper darb\u012bbai (iesp\u0113jams, <noindex><a rel=\"nofollow\" href=\"https:\/\/cwiki.apache.org\/confluence\/display\/KAFKA\/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum\">dr\u012bz var\u0113s teikt to pa\u0161u ar\u012b par Kafka<\/a><\/noindex>), jo iek\u0161\u0113ji t\u0101 izmanto RAFT. Turkl\u0101t NATS Streaming ir viegl\u0101k administr\u0113t. M\u0113s neizsl\u0113dzam iesp\u0113ju, ka turpm\u0101k m\u0113s v\u0113lreiz atgriez\u012bsimies pie \u0161\u012bs tehnolo\u0123ijas. <\/p>\n<p><\/p>\n<p>Tom\u0113r \u0161obr\u012bd m\u016bsu uzvar\u0113t\u0101js ir Apache Kafka. M\u0113s testos t\u0101 par\u0101d\u012bj\u0101s pietiekami \u0101tra (vair\u0101k nek\u0101 miljons zi\u0146ojumu sekund\u0113 gan lasot, gan rakstot ar zi\u0146ojumu apjomu 1 kilobaits), pietiekami uzticama, labi m\u0113rogojama un pier\u0101d\u012bta lielos uz\u0146\u0113mumos. Turkl\u0101t Kafka atbalsta vismaz da\u017eas lielas komerci\u0101las komp\u0101nijas (m\u0113s, piem\u0113ram, izmantojam Confluent versiju), k\u0101 ar\u012b Kafka ir att\u012bst\u012bta ekosist\u0113ma.<br \/>\n<br clear=\"left\">\n <\/p>\n<p><\/p>\n<h1 id=\"obzor-kafka\">Kafka apskats<\/h1>\n<p><\/p>\n<p>Pirms uzs\u0101kt, nekav\u0113joties rekomend\u0113ju lielisku gr\u0101matu \u2014 <em>\u00abKafka: The Definitive Guide\u00bb<\/em> (ir ir krievu tulkojum\u0101, bet termini nedaudz mulsina). Taj\u0101 var atrast inform\u0101ciju, kas nepiecie\u0161ama pamata izpratnei par Kafka un pat nedaudz vair\u0101k. Pa\u0161ai dokument\u0101cijai no Apache un Confluent emu\u0101ram ir lieliski uzrakst\u012bts un viegli las\u0101ms. <\/p>\n<p><\/p>\n<p>T\u0101tad, apl\u016bkosim, k\u0101 Kafka izskat\u0101s no putna lidojuma. Pamata Kafka topolo\u0123ija sast\u0101v no producer, consumer, broker un zookeeper.<\/p>\n<p><\/p>\n<h3 id=\"broker\">Broker<\/h3>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Kafka un mikroservisi: p\u0101rskats\" src=\"\/wp-content\/uploads\/2019\/09\/82f34b5a1aadcdc07eef58eb5414b553.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>J\u016bsu datu glab\u0101\u0161anu nodro\u0161ina brokeris (broker). Visi dati tiek glab\u0101ti bin\u0101r\u0101 veid\u0101, un brokeris maz zina par to, kas tie ir un k\u0101da ir to strukt\u016bra. <\/p>\n<p><\/p>\n<p>Katrs lo\u0123iskais notikumu veids parasti atrodas sav\u0101 atsevi\u0161\u0137\u0101 topik\u0101 (topic). Piem\u0113ram, pazi\u0146ojuma izveido\u0161anas notikums var non\u0101kt topik\u0101 item.created, bet t\u0101 izmai\u0146u notikums \u2014 item.changed. Topikus var uzskat\u012bt par notikumu klasifikatoriem. Topika l\u012bmen\u012b ir iesp\u0113jams iestat\u012bt t\u0101das konfigur\u0101cijas parametrus k\u0101: <\/p>\n<p><\/p>\n<ul>\n<li>uzglab\u0101jamo datu apjomu un\/vai to vecumu (retention.bytes, retention.ms); <\/li>\n<li>datu redundances faktoru (replication factor);<\/li>\n<li>maksim\u0101lo viena zi\u0146ojuma izm\u0113ru (max.message.bytes);<\/li>\n<li>minim\u0101lo skaitu saska\u0146oto repliku, kur\u0101 var ierakst\u012bt datus topik\u0101 (min.insync.replicas);<\/li>\n<li>iesp\u0113ju veikt failover uz nesakr\u012bto atpaliku\u0161o repliku ar potenci\u0101lu datu zudumu (unclean.leader.election.enable);<\/li>\n<li>un v\u0113l daudz ko citu (<noindex><a rel=\"nofollow\" href=\"https:\/\/kafka.apache.org\/documentation\/#topicconfigs\">https:\/\/kafka.apache.org\/documentation\/#topicconfigs<\/a><\/noindex>).<\/li>\n<\/ul>\n<p><\/p>\n<p>Savuk\u0101rt katrs topiks tiek sadal\u012bts vien\u0101 vai vair\u0101k\u0101s partitij\u0101s (partition). Tie\u0161i partitij\u0101s beig\u0101s non\u0101k notikumi. Ja klaster\u012b ir vair\u0101k nek\u0101 viens brokers, partitijas tiks vienm\u0113r\u012bgi sadal\u012btas pa visiem brokeriem, cik iesp\u0113jams, kas \u013caus skalot slodzi uz rakst\u012b\u0161anu un las\u012b\u0161anu tikai vien\u0101 topik\u0101 uz vair\u0101kiem brokeriem.<\/p>\n<p><\/p>\n<p>Uz diska dati katrai partitijai tiek glab\u0101ti segmentu failu veid\u0101, kuru noklus\u0113jums ir viens gigabaits (to kontrol\u0113 caur log.segment.bytes). Svar\u012bga iez\u012bme ir t\u0101, ka datu dz\u0113\u0161ana no partitij\u0101m (aktualiz\u0113jot retention) notiek tie\u0161i pa segmentiem (nav iesp\u0113jams dz\u0113st vienu notikumu no partitijas, var dz\u0113st tikai veselu segmentu, turkl\u0101t tikai neakt\u012bvo).<\/p>\n<p><\/p>\n<h3 id=\"zookeeper\">Zookeeper<\/h3>\n<p><\/p>\n<p>Zookeeper darbojas k\u0101 metadatu kr\u0101tuve un koordinators. Tie\u0161i tas sp\u0113j pateikt, vai brokeri ir dz\u012bvi (uz to var paskat\u012bties caur zookeeper ac\u012bm, izmantojot zookeeper-shell komandu <code>ls \/brokers\/ids<\/code>), kur\u0161 brokers ir kontrolieris (<code>get \/controller<\/code>), vai sada\u013cas ir sinhron\u0101 st\u0101vokl\u012b ar sav\u0101m replik\u0101m (<code>get \/brokers\/topics\/topic_name\/partitions\/partition_number\/state<\/code>). T\u0101pat tie\u0161i uz zookeeper vispirms dosies ra\u017eot\u0101js un pat\u0113r\u0113t\u0101js, lai uzzin\u0101tu, kur\u0101 broker\u012b k\u0101di temati un sada\u013cas tiek glab\u0101tas. Gad\u012bjumos, kad t\u0113mai ir noteikts replication factor, kas p\u0101rsniedz 1, zookeeper nor\u0101d\u012bs, kuras sada\u013cas ir l\u012bderi (t\u0101s tiks ierakst\u012btas un no t\u0101m tiks las\u012bts). Ja brokers avar\u0113, tie\u0161i zookeeper tiks ierakst\u012bta inform\u0101cija par jauniem l\u012bderu sada\u013c\u0101m (s\u0101kot no versijas 1.1.0 asinkron\u0101 re\u017e\u012bm\u0101, <noindex><a rel=\"nofollow\" href=\"https:\/\/www.confluent.io\/blog\/apache-kafka-supports-200k-partitions-per-cluster\">un tas ir svar\u012bgi<\/a><\/noindex>).<\/p>\n<p><\/p>\n<p>Vec\u0101k\u0101s Kafka versij\u0101s zookeeper atbild\u0113ja ar\u012b par offsetu glab\u0101\u0161anu, bet tagad tie tiek glab\u0101ti \u012bpa\u0161\u0101 t\u0113m\u0101 <code>__consumer_offsets<\/code> broker\u012b (lai gan j\u016bs joproj\u0101m varat izmantot zookeeper \u0161iem m\u0113r\u0137iem). <\/p>\n<p><\/p>\n<p>Visvienk\u0101r\u0161\u0101kais veids, k\u0101 p\u0101rv\u0113rst j\u016bsu datus par \u0137irbi, ir tie\u0161i inform\u0101cijas zudums no zookeeper. \u0160\u0101d\u0101 scen\u0101rij\u0101 b\u016bs \u013coti gr\u016bti saprast, ko un no kurienes ir j\u0101lasa. <\/p>\n<p><\/p>\n<h3 id=\"producer\">Ra\u017eot\u0101js<\/h3>\n<p><\/p>\n<p>Ra\u017eot\u0101js \u2014 visbie\u017e\u0101k ir serviss, kas veic tie\u0161u datu ierakstu Apache Kafka. Ra\u017eot\u0101js izv\u0113las t\u0113mu, kur\u0101 tiks glab\u0101ti t\u0101 tematiskie zi\u0146ojumi, un s\u0101k taj\u0101 ierakst\u012bt inform\u0101ciju. Piem\u0113ram, ra\u017eot\u0101js var b\u016bt sludin\u0101jumu serviss. \u0160\u0101d\u0101 gad\u012bjum\u0101 tas nos\u016bt\u012bs \u0161\u0101dus pas\u0101kumus tematiskaj\u0101s t\u0113m\u0101s, k\u0101 \"sludin\u0101jums izveidots\", \"sludin\u0101jums atjaunots\", \"sludin\u0101jums dz\u0113sts\" utt. Katrs pas\u0101kums \u0161aj\u0101 gad\u012bjum\u0101 ir atsl\u0113gas-v\u0113rt\u012bbas p\u0101ris.<\/p>\n<p><\/p>\n<p>P\u0113c noklus\u0113juma visi pas\u0101kumi tiek sadal\u012bti pa t\u0113mas sada\u013c\u0101m, izmantojot round-robin, ja atsl\u0113ga nav nor\u0101d\u012bta (zaud\u0113jot sec\u012bbu), un ar MurmurHash (atsl\u0113ga), ja atsl\u0113ga ir kl\u0101t (sec\u012bba vien\u0101 sada\u013c\u0101).<\/p>\n<p><\/p>\n<p>\u0160eit uzreiz j\u0101atz\u012bm\u0113, ka Kafka garant\u0113 pas\u0101kumu sec\u012bbu tikai vien\u0101 sada\u013c\u0101. Bet paties\u012bb\u0101 bie\u017ei tas nav probl\u0113ma. Piem\u0113ram, var dro\u0161i pievienot visas izmai\u0146as viena un t\u0101 pa\u0161a sludin\u0101juma vien\u0101 sada\u013c\u0101 (t\u0101d\u0113j\u0101di saglab\u0101jot \u0161o izmai\u0146u sec\u012bbu attiec\u012bb\u0101 uz sludin\u0101jumu). T\u0101pat var p\u0101rnest sec\u012bbas numuru vien\u0101 no pas\u0101kuma laukiem.<\/p>\n<p><\/p>\n<h3 id=\"consumer\">Pat\u0113r\u0113t\u0101js<\/h3>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Kafka un mikroservisi: p\u0101rskats\" src=\"\/wp-content\/uploads\/2019\/09\/1f69a53dc19bb6c587883b8754cde29f.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Pat\u0113r\u0113t\u0101js atbild par datu sa\u0146em\u0161anu no Apache Kafka. Atgrie\u017eoties pie iepriek\u0161 min\u0113t\u0101 piem\u0113ra, k\u0101 pat\u0113r\u0113t\u0101js var\u0113tu b\u016bt moder\u0113\u0161anas pakalpojums. \u0160is pakalpojums b\u016bs parakst\u012bjies uz sludin\u0101jumu pakalpojuma t\u0113mu un, kad par\u0101d\u012bsies jauns sludin\u0101jums, to sa\u0146ems un analiz\u0113s saska\u0146\u0101 ar noteikt\u0101m politik\u0101m.<\/p>\n<p><\/p>\n<p>Apache Kafka atceras, k\u0101dus p\u0113d\u0113jos notikumus pat\u0113r\u0113t\u0101js sa\u0146\u0113mis (tam tiek izmantota pal\u012bdz\u012bbas t\u0113ma <code>__consumer__offsets<\/code>), t\u0101d\u0113j\u0101di garant\u0113jot, ka, veiksm\u012bgi nolasot, pat\u0113r\u0113t\u0101js nesa\u0146ems to pa\u0161u zi\u0146ojumu divreiz. Tom\u0113r, ja iesp\u0113jot opciju enable.auto.commit = true un piln\u012bb\u0101 uztic\u0113t pat\u0113r\u0113t\u0101ja poz\u012bcijas izseko\u0161anu Kafka, var <noindex><a rel=\"nofollow\" href=\"https:\/\/blog.newrelic.com\/engineering\/kafka-consumer-config-auto-commit-data-loss\/\">zaud\u0113t datus<\/a><\/noindex>. Ra\u017eo\u0161anas kod\u0101 visbie\u017e\u0101k pat\u0113r\u0113t\u0101ja poz\u012bciju kontrol\u0113 manu\u0101li (izstr\u0101d\u0101t\u0101js p\u0101rvalda br\u012bdi, kad oblig\u0101ti j\u0101veic nolas\u012bt\u0101 notikuma apstiprin\u0101\u0161ana).<\/p>\n<p><\/p>\n<p>Tajos gad\u012bjumos, kad viena pat\u0113r\u0113t\u0101ja nepietiek (piem\u0113ram, ja jauno notikumu pl\u016bsma ir \u013coti liela), var pievienot v\u0113l da\u017eus pat\u0113r\u0113t\u0101jus, apvienojot tos pat\u0113r\u0113t\u0101ju grup\u0101. Pat\u0113r\u0113t\u0101ju grupa lo\u0123iski p\u0101rst\u0101v tie\u0161i t\u0101du pa\u0161u pat\u0113r\u0113t\u0101ju, bet ar datu sadal\u012bjumu starp grupas dal\u012bbniekiem. Tas \u013cauj katram dal\u012bbniekam \u0146emt savu zi\u0146ojumu da\u013cu, t\u0101d\u0113j\u0101di palielinot las\u012b\u0161anas \u0101trumu. <\/p>\n<p><\/p>\n<h1 id=\"rezultaty-testirovaniya\">Test\u0113\u0161anas rezult\u0101ti<\/h1>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Kafka un mikroservisi: p\u0101rskats\" src=\"\/wp-content\/uploads\/2019\/09\/f7abe4bcf34fdb0d762c8a928bd873ee.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>\u0160eit nev\u0113los rakst\u012bt daudz paskaidrojo\u0161a teksta, vienk\u0101r\u0161i dal\u012b\u0161os ar ieg\u016btajiem rezult\u0101tiem. Test\u0113\u0161ana tika veikta uz 3 fiziskaj\u0101m ma\u0161\u012bn\u0101m (12 CPU, 384GB RAM, 15k SAS DISK, 10GBit\/s t\u012bkls), brokeri un zookeeper tika izvietoti lxc. <\/p>\n<p><\/p>\n<p><strong>Veiktsp\u0113jas test\u0113\u0161ana<\/strong><\/p>\n<p><\/p>\n<p>Test\u0113\u0161anas laik\u0101 tika ieg\u016bti sekojo\u0161i rezult\u0101ti.<\/p>\n<p><\/p>\n<ul>\n<li>Zi\u0146ojumu rakst\u012b\u0161anas \u0101trums ar 1KB lieliem zi\u0146ojumiem vienlaikus izmantojot 9 ra\u017eot\u0101jus \u2014 1300000 notikumu sekund\u0113.<\/li>\n<li>Zi\u0146ojumu las\u012b\u0161anas \u0101trums ar 1KB lieliem zi\u0146ojumiem vienlaikus izmantojot 9 pat\u0113r\u0113t\u0101jus \u2014 1500000 notikumu sekund\u0113.<\/li>\n<\/ul>\n<p><\/p>\n<p><strong>Nepiecie\u0161am\u012bbas test\u0113\u0161ana<\/strong><\/p>\n<p><\/p>\n<p>Test\u0113\u0161anas laik\u0101 tika ieg\u016bti sekojo\u0161i rezult\u0101ti (3 brokeri, 3 zookeeper).<\/p>\n<p><\/p>\n<ul>\n<li>Vienas no brokeru neparastas izsl\u0113g\u0161an\u0101s neizn\u012bcina vai nepadara pieejamu klasteri. Darbs turpin\u0101s norm\u0101l\u0101 re\u017e\u012bm\u0101, ta\u010du atliku\u0161ajiem brokeriem ir liela slodze.<\/li>\n<li>Brokera nepareizs izn\u0101kums divu brokera gad\u012bjum\u0101 tr\u012bs brokeru klaster\u012b ar min.isr = 2 noved pie klastera nepieejam\u012bbas rakst\u012b\u0161anai, bet pieejam\u012bbas las\u012b\u0161anai. Ja min.isr = 1, klasteris turpina b\u016bt pieejams gan las\u012b\u0161anai, gan rakst\u012b\u0161anai. Tom\u0113r \u0161is re\u017e\u012bms ir pretrun\u0101 augstiem datu dro\u0161\u012bbas pras\u012bb\u0101m.<\/li>\n<li>Viena no Zookeeper serveriem nepareizs izn\u0101kums nep\u0101rtrauc vai nepadara klasteri nepieejamu. Darbs turpin\u0101s norm\u0101li.<\/li>\n<li>Divu Zookeeper serveru nepareizs izn\u0101kums noved pie klastera nepieejam\u012bbas l\u012bdz br\u012bdim, kad viens no Zookeeper serveriem atjauno savu darb\u012bbu. \u0160is apgalvojums ir pareizs attiec\u012bb\u0101 uz Zookeeper klasteri, kas sast\u0101v no 3 serveriem. P\u0113c izp\u0113tes tika nolemts palielin\u0101t Zookeeper klasteri l\u012bdz 5 serveriem, lai palielin\u0101tu pieejam\u012bbu. <br clear=\"left\">\n <\/li>\n<\/ul>\n<p><\/p>\n<h1 id=\"kafka-as-a-service\">Kafka k\u0101 pakalpojums<\/h1>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Kafka un mikroservisi: p\u0101rskats\" src=\"\/wp-content\/uploads\/2019\/09\/19b2356131203cefbadfd90a25ae9674.png\" style=\"display:block;margin: 0 auto;\" \/> <\/p>\n<p><\/p>\n<p>M\u0113s esam p\u0101rliecin\u0101ju\u0161ies, ka Kafka ir lieliska tehnolo\u0123ija, kas pal\u012bdz mums atrisin\u0101t uzdevumu (zi\u0146ojumu brokera izveido\u0161ana). Tom\u0113r m\u0113s nol\u0113m\u0101m aizliegt pakalpojumiem tie\u0161i piek\u013c\u016bt Kafka un sl\u0113dz\u0101m to virs\u016b ar data-bus pakalpojumu. K\u0101p\u0113c m\u0113s to izdar\u012bj\u0101m? Paties\u012bb\u0101 ir vair\u0101kas iemeslu.<\/p>\n<p><\/p>\n<ul>\n<li>\n<p>Data-bus ir uz\u0146\u0113mis visas ar Kafka integr\u0101ciju saist\u012bt\u0101s funkcijas (pat\u0113ri\u0146a un ra\u017eot\u0101ju iestat\u012b\u0161ana un konfigur\u0113\u0161ana, uzraudz\u012bba, br\u012bdin\u0101\u0161ana, \u017eurn\u0101lfailu glab\u0101\u0161ana, m\u0113rogo\u0161ana utt.). T\u0101d\u0113j\u0101di integr\u0101cija ar zi\u0146ojumu brokeri notiek maksim\u0101li vienk\u0101r\u0161i. <\/p>\n<p>\n<\/li>\n<li>\n<p>Data-bus \u013c\u0101va abstrah\u0113ties no konkr\u0113t\u0101s valodas vai bibliot\u0113kas savienojumam ar Kafka. <\/p>\n<p>\n<\/li>\n<li>\n<p>Data-bus \u013c\u0101va citiem pakalpojumiem abstrah\u0113ties no datu glab\u0101\u0161anas sl\u0101\u0146a. Iesp\u0113jams, k\u0101d\u0101 br\u012bd\u012b m\u0113s aizvietosim Kafka ar Pulsar, un neviens pat neman\u012bs (visi pakalpojumi zina tikai par data-bus API).<\/p>\n<p>\n<\/li>\n<li>\n<p>Data-bus uz\u0146\u0113ma sev notikumu sh\u0113mu valid\u0101ciju.<\/p>\n<p>\n<\/li>\n<li>\n<p>Ar data-bus ir ieviesta autentifik\u0101cija.<\/p>\n<p>\n<\/li>\n<li>\n<p>Data-bus ietvaros m\u0113s varam bez d\u012bkst\u0101ves, neman\u0101mi atjaunin\u0101t Kafka versijas, centraliz\u0113ti p\u0101rvald\u012bt ra\u017eot\u0101jus, pat\u0113ri\u0146us, brokerus utt.<\/p>\n<p>\n<\/li>\n<li>\n<p>Data-bus \u013c\u0101va pievienot nepiecie\u0161am\u0101s funkcijas, kuras tr\u016bkst Kafka (k\u0101 piem\u0113ram, t\u0113mu audita veik\u0161ana, anom\u0101liju kontrole klaster\u012b, DLQ izveido\u0161ana utt.).<\/p>\n<p>\n<\/li>\n<li>\n<p>Data-bus \u013cauj realiz\u0113t failover centraliz\u0113ti visiem pakalpojumiem.<\/p>\n<p>\n<\/li>\n<\/ul>\n<p><\/p>\n<p>Pa\u0161laik, lai s\u0101ktu s\u016bt\u012bt notikumus uz zi\u0146ojumu brokeri, pietiek ar to, lai sav\u0101 pakalpojuma kod\u0101 pievienotu nelielu bibliot\u0113ku. Tas ir viss. Jums rodas iesp\u0113ja rakst\u012bt, las\u012bt un skalot ar vienu koda rindu. Visa realiz\u0101cija ir sl\u0113pta no jums, redzam\u0101 da\u013ca ir tikai da\u017eas rokasgr\u0101matas ar partijas izm\u0113ru. Zem v\u0101ka pakalpojums data-bus palai\u017e Kubernetes nepiecie\u0161amo instan\u010du skaitu ra\u017eot\u0101jiem un pat\u0113r\u0113t\u0101jiem un nodro\u0161ina vi\u0146iem nepiecie\u0161amo konfigur\u0101ciju, ta\u010du viss tas j\u016bsu pakalpojumam ir caursp\u012bd\u012bgs. <\/p>\n<p><\/p>\n<p>Protams, nav sudraba lodes, un \u0161\u0101dam pieejai ir savi ierobe\u017eojumi.<\/p>\n<p><\/p>\n<ul>\n<li>Data-bus ir j\u0101atbalsta pa\u0161iem, at\u0161\u0137ir\u012bb\u0101 no tre\u0161o pu\u0161u bibliot\u0113k\u0101m.<\/li>\n<li>Data-bus palielina mijiedarb\u012bbu starp pakalpojumiem un zi\u0146ojumu brokeri, kas samazina veiktsp\u0113ju sal\u012bdzin\u0101jum\u0101 ar t\u012bru Kafka.<\/li>\n<li>Ne viss var tikt t\u0101 vienk\u0101r\u0161i nosl\u0113pts no pakalpojumiem, dubl\u0113t KSQL vai Kafka Streams funkcionalit\u0101ti data-bus mums negribas, t\u0101d\u0113\u013c da\u017ereiz n\u0101kas at\u013caut pakalpojumiem savienoties tie\u0161i. <\/li>\n<\/ul>\n<p><\/p>\n<p>M\u016bsu gad\u012bjum\u0101 plusi p\u0101rsita m\u012bnusus, un l\u0113mums sl\u0113pt zi\u0146ojumu brokeri saska\u0146ot\u0101 pakalpojum\u0101 izr\u0101d\u012bj\u0101s pamatots. Gada laik\u0101 ekspluat\u0101cijas laik\u0101 mums nebija nek\u0101du nopietnu av\u0101riju un probl\u0113mu.<\/p>\n<p><\/p>\n<p>P.S. Paldies manai draudzenei, Ekaterinai Obalajevai, par lieliskaj\u0101m ilustr\u0101cij\u0101m \u0161im rakstam. Ja jums t\u0101s patika, <noindex><a rel=\"nofollow\" href=\"https:\/\/www.instagram.com\/o.k_o.k_\/\">\u0161eit<\/a><\/noindex> atrad\u012bsies v\u0113l vair\u0101k ilustr\u0101ciju.<\/p>\n<p>Avots: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/avito\/blog\/465315\/\">habr.com<\/a><\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u0412\u0441\u0435\u043c \u043f\u0440\u0438\u0432\u0435\u0442. \u0412 \u044d\u0442\u043e\u0439 \u0441\u0442\u0430\u0442\u044c\u0435 \u044f \u0440\u0430\u0441\u0441\u043a\u0430\u0436\u0443, \u043f\u043e\u0447\u0435\u043c\u0443 \u043c\u044b \u0432 \u0410\u0432\u0438\u0442\u043e \u0434\u0435\u0432\u044f\u0442\u044c \u043c\u0435\u0441\u044f\u0446\u0435\u0432 \u043d\u0430\u0437\u0430\u0434 \u0432\u044b\u0431\u0440\u0430\u043b\u0438 Kafka, \u0438 \u0447\u0442\u043e \u043e\u043d\u0430 \u0438\u0437 \u0441\u0435\u0431\u044f \u043f\u0440\u0435\u0434\u0441\u0442\u0430\u0432\u043b\u044f\u0435\u0442. \u041f\u043e\u0434\u0435\u043b\u044e\u0441\u044c \u043e\u0434\u043d\u0438\u043c \u0438\u0437 \u043a\u0435\u0439\u0441\u043e\u0432 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u043d\u0438\u044f \u2014 \u0431\u0440\u043e\u043a\u0435\u0440 \u0441\u043e\u043e\u0431\u0449\u0435\u043d\u0438\u0439. \u0418 \u043d\u0430\u043f\u043e\u0441\u043b\u0435\u0434\u043e\u043a \u043f\u043e\u0433\u043e\u0432\u043e\u0440\u0438\u043c \u043e \u0442\u043e\u043c, \u043a\u0430\u043a\u0438\u0435 \u043f\u043b\u044e\u0441\u044b \u043c\u044b \u043f\u043e\u043b\u0443\u0447\u0438\u043b\u0438 \u043e\u0442 \u043f\u0440\u0438\u043c\u0435\u043d\u0435\u043d\u0438\u044f \u043f\u043e\u0434\u0445\u043e\u0434\u0430 Kafka as a Service. \u041f\u0440\u043e\u0431\u043b\u0435\u043c\u0430 \u0414\u043b\u044f \u043d\u0430\u0447\u0430\u043b\u0430 \u043d\u0435\u043c\u043d\u043e\u0433\u043e \u043a\u043e\u043d\u0442\u0435\u043a\u0441\u0442\u0430. \u041d\u0435\u043a\u043e\u0442\u043e\u0440\u043e\u0435 \u0432\u0440\u0435\u043c\u044f \u043d\u0430\u0437\u0430\u0434 \u043c\u044b [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":28406,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-37849","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.5 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u0412\u0441\u0435\u043c \u043f\u0440\u0438\u0432\u0435\u0442.\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/lv\/blog\/administrirovanie\/kafka-i-mikroservisy-obzor\" \/>\n\t\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.5\" \/>\n\t\t<meta property=\"og:locale\" content=\"lv_LV\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47Kafka \u0438 \u043c\u0438\u043a\u0440\u043e\u0441\u0435\u0440\u0432\u0438\u0441\u044b: \u043e\u0431\u0437\u043e\u0440 | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u0412\u0441\u0435\u043c \u043f\u0440\u0438\u0432\u0435\u0442.\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/lv\/blog\/administrirovanie\/kafka-i-mikroservisy-obzor\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2019-10-31T19:20:09+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2019-10-31T19:20:09+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47Kafka un mikroservisi: p\u0101rskats | ProHoster","description":"Visiem sveiciens.","canonical_url":"https:\/\/prohoster.info\/lv\/blog\/administrirovanie\/kafka-i-mikroservisy-obzor","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"lv_LV","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47Kafka \u0438 \u043c\u0438\u043a\u0440\u043e\u0441\u0435\u0440\u0432\u0438\u0441\u044b: \u043e\u0431\u0437\u043e\u0440 | ProHoster","og:description":"\u0412\u0441\u0435\u043c \u043f\u0440\u0438\u0432\u0435\u0442.","og:url":"https:\/\/prohoster.info\/lv\/blog\/administrirovanie\/kafka-i-mikroservisy-obzor","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2019-10-31T19:20:09+00:00","article:modified_time":"2019-10-31T19:20:09+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"37849","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":"2026-01-23 19:31:03","breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-03-01 01:19:23","updated":"2026-01-23 19:31:03","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/lv\/wp-json\/wp\/v2\/posts\/37849","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/lv\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/lv\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/lv\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/lv\/wp-json\/wp\/v2\/comments?post=37849"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/lv\/wp-json\/wp\/v2\/posts\/37849\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/lv\/wp-json\/wp\/v2\/media\/28406"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/lv\/wp-json\/wp\/v2\/media?parent=37849"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/lv\/wp-json\/wp\/v2\/categories?post=37849"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/lv\/wp-json\/wp\/v2\/tags?post=37849"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}