{"id":71213,"date":"2020-02-24T16:09:17","date_gmt":"2020-02-24T13:09:17","guid":{"rendered":"https:\/\/prohoster.info\/blog\/opredelyaem-podhodyashhij-razmer-dlya-klastera-kafka-v-kubernetes"},"modified":"2020-03-03T16:14:29","modified_gmt":"2020-03-03T13:14:29","slug":"opredelyaem-podhodyashhij-razmer-dlya-klastera-kafka-v-kubernetes","status":"publish","type":"post","link":"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/opredelyaem-podhodyashhij-razmer-dlya-klastera-kafka-v-kubernetes","title":{"rendered":"Ur\u010dujeme vhodn\u00fa ve\u013ekos\u0165 pre Kafka klaster v Kubernetes","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p><i><b>Pozn. prekl.<\/b>: V tomto \u010dl\u00e1nku spolo\u010dnos\u0165 Banzai Cloud zdie\u013ea pr\u00edklad pou\u017eitia svojich \u0161peci\u00e1lnych n\u00e1strojov na zjednodu\u0161enie prev\u00e1dzky Kafka v r\u00e1mci Kubernetes. Poskytnut\u00e9 pokyny ilustruj\u00fa, ako ur\u010di\u0165 optim\u00e1lnu ve\u013ekos\u0165 infra\u0161trukt\u00fary a nakonfigurova\u0165 samotn\u00fa Kafka pre dosiahnutie po\u017eadovanej priepustnosti.<\/i><\/p>\n<p><img decoding=\"async\" alt=\"Ur\u010dujeme vhodn\u00fa ve\u013ekos\u0165 pre Kafka klaster v Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/8a598ec7db091c2944a6442f9bc124da.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nApache Kafka je distribuovan\u00e1 streamovacia platforma na vytv\u00e1ranie spo\u013eahliv\u00fdch, \u0161k\u00e1lovate\u013en\u00fdch a vysokov\u00fdkonn\u00fdch syst\u00e9mov spracovania \u00fadajov v re\u00e1lnom \u010dase. Jej p\u00f4sobiv\u00e9 mo\u017enosti m\u00f4\u017eeme roz\u0161\u00edri\u0165 pomocou Kubernetes. Na tento \u00fa\u010del sme vyvinuli <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/banzaicloud\/kafka-operator\">Open Source oper\u00e1tor Kafka<\/a><\/noindex> a n\u00e1stroj s n\u00e1zvom <noindex><a rel=\"nofollow\" href=\"https:\/\/banzaicloud.com\/docs\/supertubes\/overview\/\">Supertubes<\/a><\/noindex>. Tieto n\u00e1stroje umo\u017e\u0148uj\u00fa spusti\u0165 Kafka v Kubernetes a vyu\u017ei\u0165 jej r\u00f4zne funkcie, ako je jemn\u00e9 ladenie konfigur\u00e1cie brokerov, \u0161k\u00e1lovanie na z\u00e1klade metr\u00edk s rebalance, rack awareness, \u201ejemn\u00e9\u201c <i>(graceful)<\/i> uv\u00e1dzanie aktualiz\u00e1ci\u00ed at\u010f.<noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><\/p>\n<blockquote><p>Vysk\u00fa\u0161ajte Supertubes vo svojom klastri:<\/p>\n<pre><code class=\"bash\">curl https:\/\/getsupertubes.sh | sh a supertubes install -a --no-democluster --kubeconfig<\/code><\/pre>\n<p>\nAlebo sa obr\u00e1\u0165te na <noindex><a rel=\"nofollow\" href=\"https:\/\/banzaicloud.com\/docs\/supertubes\/cli\/reference\/\">dokumen\u00e1cie<\/a><\/noindex>. M\u00f4\u017eete si tie\u017e pre\u010d\u00edta\u0165 o niektor\u00fdch funkci\u00e1ch Kafka, ktor\u00fdch spr\u00e1va je automatizovan\u00e1 pomocou Supertubes a Kafka oper\u00e1tora. O nich sme u\u017e p\u00edsali na blogu:<\/p>\n<ul>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/banzaicloud.com\/blog\/kafka-operator\/\">Oh no! E\u0161te jeden Kafka oper\u00e1tor pre Kubernetes<\/a><\/noindex>;<\/li>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/banzaicloud.com\/blog\/kafka-alert\/\">Monitorujte a spravujte Kafka na z\u00e1klade Prometheus metr\u00edk<\/a><\/noindex>;<\/li>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/banzaicloud.com\/blog\/kafka-rack-awareness\/\">Kafka rack awareness na Kubernetes<\/a><\/noindex>;<\/li>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/banzaicloud.com\/blog\/kafka-on-istio-performance\/\">Sp\u00fa\u0161\u0165anie Apache Kafka cez Istio \u2014 benchmark<\/a><\/noindex>;<\/li>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/banzaicloud.com\/blog\/kafka-topic-user-management\/\">U\u017e\u00edvate\u013emi autentifikovan\u00e9 a pr\u00edstupov\u00e9 kontrolovan\u00e9 klastry s Kafka oper\u00e1torom<\/a><\/noindex>;<\/li>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/banzaicloud.com\/blog\/kafka-rolling-upgrade\/\">Kafka rolling upgrade a dynamick\u00e1 konfigur\u00e1cia na Kubernetes<\/a><\/noindex>;<\/li>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/banzaicloud.com\/blog\/kafka-envoy-protocol-filter\/\">Envoy protokolov\u00fd filter pre Kafka, zaran\u017eovan\u00fd<\/a><\/noindex>.<\/li>\n<\/ul>\n<\/blockquote>\n<p>\nKe\u010f sa rozhodnete nasadi\u0165 klaster Kafka v Kubernetes, ur\u010dite naraz\u00edte na probl\u00e9m ur\u010denia optim\u00e1lnej ve\u013ekosti z\u00e1kladnej infra\u0161trukt\u00fary a nutnosti jemn\u00e9ho doladenia konfigur\u00e1cie Kafka, aby splnila po\u017eiadavky na priepustnos\u0165. Maxim\u00e1lny v\u00fdkon ka\u017ed\u00e9ho brokera je ovplyvnen\u00fd v\u00fdkonom komponentov infra\u0161trukt\u00fary pod n\u00edm, ako s\u00fa pam\u00e4\u0165, procesor, r\u00fdchlos\u0165 disku, priepustnos\u0165 siete at\u010f.<\/p>\n<p>Ide\u00e1lne by mala by\u0165 konfigur\u00e1cia brokerov tak\u00e1, aby v\u0161etky prvky infra\u0161trukt\u00fary boli vyu\u017eit\u00e9 na maximum svojich mo\u017enost\u00ed. V re\u00e1lnom \u017eivote je v\u0161ak tak\u00e1to nastavenie ve\u013emi zlo\u017eit\u00e1. Je pravdepodobnej\u0161ie, \u017ee pou\u017e\u00edvatelia nastavia konfigur\u00e1ciu brokerov tak, aby maximalizovali vyu\u017eitie jedn\u00e9ho alebo dvoch komponentov (disku, pam\u00e4ti alebo procesora). V\u0161eobecne plat\u00ed, \u017ee broker dosahuje maxim\u00e1lny v\u00fdkon, ke\u010f jeho konfigur\u00e1cia umo\u017e\u0148uje plne vyu\u017ei\u0165 najpomal\u0161\u00ed komponent. Takto m\u00f4\u017eeme z\u00edska\u0165 orienta\u010dn\u00fd preh\u013ead o z\u00e1\u0165a\u017ei, s ktorou si dok\u00e1\u017ee jeden broker poradi\u0165.<\/p>\n<p>Teoreticky m\u00f4\u017eeme tie\u017e odhadn\u00fa\u0165 po\u010det brokerov potrebn\u00fdch na spracovanie ur\u010ditej z\u00e1\u0165a\u017ee. V praxi v\u0161ak existuje to\u013eko variantov nastavenia na r\u00f4znych \u00farovniach, \u017ee nie je \u013eahk\u00e9 odhadn\u00fa\u0165 potenci\u00e1lny v\u00fdkon ur\u010ditej konfigur\u00e1cie (ak nie je to nemo\u017en\u00e9). In\u00fdmi slovami, \u0165a\u017eko sa pl\u00e1nuje konfigur\u00e1cia na z\u00e1klade ur\u010ditej zadanej v\u00fdkonnosti.<\/p>\n<p>Pre pou\u017e\u00edvate\u013eov Supertubes zvy\u010dajne uplat\u0148ujeme nasleduj\u00faci pr\u00edstup: za\u010d\u00edname s ur\u010ditou konfigur\u00e1ciou (infra\u0161trukt\u00fara + nastavenia), potom mer\u00e1me jej v\u00fdkon, upravujeme nastavenia brokerov a proces opakujeme. To trv\u00e1, k\u00fdm nie je \u00faplne vyu\u017eit\u00fd potenci\u00e1l najpomal\u0161ieho komponentu infra\u0161trukt\u00fary.<\/p>\n<p>T\u00fdmto sp\u00f4sobom z\u00edskavame jasnej\u0161iu predstavu o tom, ko\u013eko brokerov je potrebn\u00e9 klasteru na zvl\u00e1dnutie ur\u010ditej z\u00e1\u0165a\u017ee (po\u010det brokerov z\u00e1vis\u00ed aj od in\u00fdch faktorov, ako je minim\u00e1lny po\u010det replik spr\u00e1v na zabezpe\u010denie robustnosti, po\u010det partition l\u00eddrov at\u010f.). Okrem toho z\u00edskavame predstavu o tom, pre ktor\u00fd infra\u0161trukt\u00farny komponent by bolo \u017eiaduce vertik\u00e1lne \u0161k\u00e1lovanie.<\/p>\n<p>V tomto \u010dl\u00e1nku p\u00f4jde o kroky, ktor\u00e9 podnik\u00e1me, aby sme 'vy\u017em\u00fdkali v\u0161etko' z najpomal\u0161\u00edch komponentov v po\u010diato\u010dn\u00fdch konfigur\u00e1ci\u00e1ch a merali priepustnos\u0165 klastera Kafka. Vysoko odoln\u00e1 konfigur\u00e1cia vy\u017eaduje pr\u00edtomnos\u0165 aspo\u0148 troch funk\u010dn\u00fdch brokerov (<code>min.insync.replicas=3<\/code>), rozdelen\u00fdch do troch r\u00f4znych z\u00f3n dostupnosti. Na konfigur\u00e1ciu, \u0161k\u00e1lovanie a monitorovanie infra\u0161trukt\u00fary Kubernetes pou\u017e\u00edvame na\u0161u vlastn\u00fa platformu na spr\u00e1vu kontajnerov pre hybridn\u00e9 cloudov\u00e9 rie\u0161enia \u2014 <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/banzaicloud\/pipeline\">Pipeline<\/a><\/noindex>. Podporuje on-premise (bare metal, VMware) a p\u00e4\u0165 typov cloudu (Alibaba, AWS, Azure, Google, Oracle), ako aj ak\u00e9ko\u013evek ich kombin\u00e1cie.<\/p>\n<h2>N\u00e1zory na infra\u0161trukt\u00faru a konfigur\u00e1ciu klastru Kafka<\/h2>\n<p>\nPre ni\u017e\u0161ie uveden\u00e9 pr\u00edklady sme vybrali AWS ako poskytovate\u013ea cloudov\u00fdch slu\u017eieb a EKS ako distrib\u00faciu Kubernetes. Podobn\u00fa konfigur\u00e1ciu je mo\u017en\u00e9 implementova\u0165 pomocou <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/banzaicloud\/pke\">PKE<\/a><\/noindex> \u2014 distrib\u00facie Kubernetes od Banzai Cloud, certifikovan\u00e9ho CNCF.<\/p>\n<h3>Disk<\/h3>\n<p>\nAmazon pon\u00faka r\u00f4zne <noindex><a rel=\"nofollow\" href=\"https:\/\/docs.aws.amazon.com\/AWSEC2\/latest\/UserGuide\/ebs-volume-types.html\">typy objemov EBS<\/a><\/noindex>. V z\u00e1klade <i>gp2<\/i> a <i>io1<\/i> s\u00fa SSD disky, av\u0161ak pre zabezpe\u010denie vysokej priepustnosti <i>gp2<\/i> spotrebuje nahromaden\u00e9 kredity <i>(I\/O kredity)<\/i>, preto sme uprednostnili typ <i>io1<\/i>, ktor\u00fd pon\u00faka stabiln\u00fa vysok\u00fa priepustnos\u0165.<\/p>\n<h3>Typy in\u0161tanci\u00ed<\/h3>\n<p>\nV\u00fdkon Kafka ve\u013emi z\u00e1vis\u00ed od str\u00e1nkovacieho cache opera\u010dn\u00e9ho syst\u00e9mu, a preto potrebujeme in\u0161tancie s dostato\u010dn\u00fdm mno\u017estvom pam\u00e4te pre brokerov (JVM) a str\u00e1nkovac\u00ed cache. In\u0161tancia <i>c5.2xlarge<\/i> \u2014 nie je zl\u00fd za\u010diatok, ke\u010f\u017ee m\u00e1 16 GB pam\u00e4te a <noindex><a rel=\"nofollow\" href=\"https:\/\/docs.aws.amazon.com\/AWSEC2\/latest\/UserGuide\/ebs-optimized.html\">je optimalizovan\u00e1 pre pr\u00e1cu s EBS<\/a><\/noindex>. Jej nev\u00fdhodou je, \u017ee dok\u00e1\u017ee zabezpe\u010di\u0165 maxim\u00e1lny v\u00fdkon maxim\u00e1lne 30 min\u00fat ka\u017ed\u00fdch 24 hod\u00edn. Ak pracovn\u00e1 z\u00e1\u0165a\u017e vy\u017eaduje maxim\u00e1lny v\u00fdkon po\u010das dlh\u0161ej doby, malo by sa pozrie\u0165 na in\u00e9 typy in\u0161tanci\u00ed. My sme presne tak urobili a rozhodli sa pre <i>c5.4xlarge<\/i>. Zabezpe\u010duje maxim\u00e1lnu priepustnos\u0165 v <b>593,75 MB\/s<\/b>. Maxim\u00e1lna priepustnos\u0165 objemu EBS <i>io1<\/i> je vy\u0161\u0161ia ako u in\u0161tancie <i>c5.4xlarge<\/i>, a preto sa zd\u00e1, \u017ee najpomal\u0161\u00edm prvkom infra\u0161trukt\u00fary je priepustnos\u0165 I\/O tohto typu in\u0161tancie (\u010do by mali potvrdi\u0165 aj v\u00fdsledky na\u0161ich z\u00e1\u0165a\u017eov\u00fdch testov).<\/p>\n<h3>Sie\u0165<\/h3>\n<p>\nPriepustnos\u0165 siete by mala by\u0165 dostato\u010dne ve\u013ek\u00e1 v porovnan\u00ed s v\u00fdkonom in\u0161tancie VM a disku, inak sa sie\u0165 st\u00e1va \u00fazkym hrdlom. V na\u0161om pr\u00edpade sie\u0165ov\u00e9 rozhranie <i>c5.4xlarge<\/i> podporuje r\u00fdchlos\u0165 do 10 Gb\/s, \u010do je v\u00fdrazne viac ako priepustnos\u0165 I\/O in\u0161tancie VM.<\/p>\n<h3>Nasadenie brokerov<\/h3>\n<p>\nBrokery by mali by\u0165 nasaden\u00e9 (pl\u00e1novan\u00e9 v Kubernetes) na vyhraden\u00e9 uzly, aby sa predi\u0161lo s\u00fa\u0165a\u017eeniu s in\u00fdmi procesmi o zdroje procesora, pam\u00e4te, siete a disku.<\/p>\n<h3>Verzia Java<\/h3>\n<p>\nLogickou vo\u013ebou je Java 11, preto\u017ee je kompatibiln\u00e1 s Dockerom v tom zmysle, \u017ee JVM spr\u00e1vne ur\u010duje procesory a pam\u00e4\u0165 dostupn\u00e9 pre kontajner, v ktorom be\u017e\u00ed broker. Ke\u010f\u017ee limity procesorov s\u00fa d\u00f4le\u017eit\u00e9, JVM vn\u00fatorn\u00e9 a transparentne nastavuje po\u010det GC vl\u00e1kien a vl\u00e1kien JIT kompil\u00e1tora. Pou\u017eili sme obraz Kafka <code>banzaicloud\/kafka:2.13-2.4.0<\/code>, vr\u00e1tane verzie Kafka 2.4.0 (Scala 2.13) na Java 11.<\/p>\n<blockquote><p>Ak sa chcete dozvedie\u0165 viac o Java\/JVM na Kubernetes, pozrite si na\u0161e nasleduj\u00face publik\u00e1cie:<\/p>\n<ul>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/banzaicloud.com\/blog\/java-resource-limits\/\">Pre\u010do moja Java aplik\u00e1cia bola OOMKilled<\/a><\/noindex>;<\/li>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/banzaicloud.com\/blog\/java10-container-sizing\/\">Ako spr\u00e1vne dimenzova\u0165 kontajnery pre Java 10 aplik\u00e1cie<\/a><\/noindex>.<\/li>\n<\/ul>\n<\/blockquote>\n<p><\/p>\n<h3>Nastavenia pam\u00e4te brokera<\/h3>\n<p>\nExistuj\u00fa dva k\u013e\u00fa\u010dov\u00e9 aspekty pri nastaven\u00ed pam\u00e4te brokera: nastavenia pre JVM a pre pod Kubernetes. Limit pam\u00e4te nastaven\u00fd pre pod by mal by\u0165 v\u00e4\u010d\u0161\u00ed ako maxim\u00e1lna ve\u013ekos\u0165 heap, aby JVM malo miesto pre metapriestor Java, ktor\u00fd sa nach\u00e1dza v vlastnej pam\u00e4ti, a pre str\u00e1nkovac\u00ed cache opera\u010dn\u00e9ho syst\u00e9mu, ktor\u00fd Kafka akt\u00edvne vyu\u017e\u00edva. V na\u0161ich testoch sme sp\u00fa\u0161\u0165ali brokery Kafka s parametrami <code>-Xmx4G -Xms2G<\/code>, pri\u010dom limit pam\u00e4te pre pod bol <code>10 Gi<\/code>. Upozor\u0148ujeme, \u017ee nastavenia pam\u00e4te pre JVM je mo\u017en\u00e9 automaticky z\u00edskava\u0165 pomocou <code>-XX:MaxRAMPercentage<\/code> a <code>-X:MinRAMPercentage<\/code>, na z\u00e1klade limitu pam\u00e4te pre pod.<\/p>\n<h3>Procesorov\u00e9 nastavenia brokera<\/h3>\n<p>\nV\u0161eobecne plat\u00ed, \u017ee m\u00f4\u017eete zv\u00fd\u0161i\u0165 v\u00fdkon zv\u00fd\u0161en\u00edm paralelizmu zv\u00fd\u0161en\u00edm po\u010dtu vl\u00e1kien pou\u017e\u00edvan\u00fdch Kafka. \u010c\u00edm viac procesorov je dostupn\u00fdch pre Kafka, t\u00fdm lep\u0161ie. V na\u0161om teste sme za\u010dali s limitom 6 procesorov a postupne (iter\u00e1ciami) sme ich po\u010det zv\u00fd\u0161ili na 15. Okrem toho sme nastavili <code>num.network.threads=12<\/code> v nastaveniach brokera, aby sme zv\u00fd\u0161ili po\u010det vl\u00e1kien, ktor\u00e9 prij\u00edmaj\u00fa d\u00e1ta zo siete a posielaj\u00fa ich. Hne\u010f ako sme zistili, \u017ee brokery-n\u00e1sledovn\u00edci nem\u00f4\u017eu dost\u00e1va\u0165 repliky dostato\u010dne r\u00fdchlo, zv\u00fd\u0161ili sme <code>num.replica.fetchers<\/code> na 4, aby sme zv\u00fd\u0161ili r\u00fdchlos\u0165, s ktorou brokery-n\u00e1sledovn\u00edci replikovali spr\u00e1vy od l\u00eddrov.<\/p>\n<h3>N\u00e1stroj na generovanie z\u00e1\u0165a\u017ee<\/h3>\n<p>\nJe potrebn\u00e9 zabezpe\u010di\u0165, aby potenci\u00e1l vybran\u00e9ho gener\u00e1tora za\u0165a\u017eenia nevyschol sk\u00f4r, ne\u017e dosiahne klaster Kafka (pre ktor\u00fd sa vykon\u00e1va benchmark) svoju maxim\u00e1lnu z\u00e1\u0165a\u017e. In\u00fdmi slovami, je potrebn\u00e9 vykona\u0165 predbe\u017en\u00e9 hodnotenie mo\u017enost\u00ed n\u00e1stroja na generovanie za\u0165a\u017eenia a vybra\u0165 pre\u0148 typy in\u0161tanci\u00ed s dostato\u010dn\u00fdm po\u010dtom procesorov a pam\u00e4\u0165ou. V tomto pr\u00edpade n\u00e1\u0161 n\u00e1stroj vytvor\u00ed viac za\u0165a\u017eenia, ne\u017e dok\u00e1\u017ee klaster Kafka spracova\u0165. Po mnoh\u00fdch experimentoch sme sa zastavili na troch in\u0161tanci\u00e1ch <i>c5.4xlarge<\/i>, pri\u010dom ka\u017ed\u00e1 z nich mala spusten\u00fd gener\u00e1tor.<\/p>\n<h2>Benchmarking<\/h2>\n<p>\nMeranie v\u00fdkonu je iterat\u00edvny proces, ktor\u00fd zah\u0155\u0148a nasleduj\u00face f\u00e1zy:<\/p>\n<ul>\n<li> nastavenie infra\u0161trukt\u00fary (klaster EKS, klaster Kafka, n\u00e1stroj na generovanie z\u00e1\u0165a\u017ee, ako aj Prometheus a Grafana);<\/li>\n<li> generovanie z\u00e1\u0165a\u017ee po\u010das ur\u010dit\u00e9ho obdobia na filtrovanie n\u00e1hodn\u00fdch odch\u00fdlok v zhroma\u017eden\u00fdch ukazovate\u013eoch v\u00fdkonu;<\/li>\n<li> \u00faprava infra\u0161trukt\u00fary a konfigur\u00e1cie brokera na z\u00e1klade pozorovan\u00fdch ukazovate\u013eov v\u00fdkonu;<\/li>\n<li> opakovanie procesu, k\u00fdm nie je dosiahnut\u00e1 po\u017eadovan\u00e1 \u00farove\u0148 priepustnosti klastera Kafka. T\u00e1 by mala by\u0165 stabilne reprodukovate\u013en\u00e1 a vykazova\u0165 minim\u00e1lne vari\u00e1cie priepustnosti.<\/li>\n<\/ul>\n<p>\nV nasleduj\u00facej \u010dasti s\u00fa pop\u00edsan\u00e9 kroky, ktor\u00e9 sa vykonali v procese benchmarkovania testovacieho klastera.<\/p>\n<h3>N\u00e1stroje<\/h3>\n<p>\nNa r\u00fdchle nasadenie z\u00e1kladnej konfigur\u00e1cie, generovanie z\u00e1\u0165a\u017ee a meranie v\u00fdkonu boli pou\u017eit\u00e9 nasleduj\u00face n\u00e1stroje:<\/p>\n<ul>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/banzaicloud.com\/docs\/pipeline\/overview\/\">Banzai Cloud Pipeline<\/a><\/noindex> na zorganizovanie klastera EKS od Amazonu s <noindex><a rel=\"nofollow\" href=\"https:\/\/prometheus.io\/\">Prometheus<\/a><\/noindex> (na zhroma\u017e\u010fovanie metr\u00edk Kafka a infra\u0161trukt\u00fary) a <noindex><a rel=\"nofollow\" href=\"https:\/\/grafana.com\/\">Grafana<\/a><\/noindex> (na vizualiz\u00e1ciu t\u00fdchto metr\u00edk). Vyu\u017eili sme <b>integrovan\u00e9<\/b> v <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/banzaicloud\/pipeline\">Pipeline<\/a><\/noindex> slu\u017eby, ktor\u00e9 zabezpe\u010duj\u00fa federat\u00edvne monitorovanie, centralizovan\u00e9 zhroma\u017e\u010fovanie logov, skenovanie zranite\u013enost\u00ed, obnovenie po v\u00fdpadkoch, bezpe\u010dnos\u0165 na \u00farovni podniku a mnoho \u010fal\u0161ieho.<\/li>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/jamiealquiza\/sangrenel\">Sangrenel<\/a><\/noindex> je n\u00e1stroj na z\u00e1\u0165a\u017eov\u00e9 testovanie klastera Kafka.<\/li>\n<li> Panely Grafana na vizualiz\u00e1ciu metr\u00edk Kafka a infra\u0161trukt\u00fary: <noindex><a rel=\"nofollow\" href=\"https:\/\/grafana.com\/grafana\/dashboards\/10123\">Kubernetes Kafka<\/a><\/noindex>, <noindex><a rel=\"nofollow\" href=\"https:\/\/grafana.com\/grafana\/dashboards\/1860\">Node Exporter<\/a><\/noindex>.<\/li>\n<li> Supertubes CLI pre maxim\u00e1lne jednoduch\u00e9 nastavenie klastru Kafka v Kubernetes. Zookeeper, Kafka operator, Envoy a mno\u017estvo \u010fal\u0161\u00edch komponentov s\u00fa nain\u0161talovan\u00e9 a spr\u00e1vne nakonfigurovan\u00e9 pre spustenie klastru Kafka pripraven\u00e9ho na produkciu v Kubernetes.\n<ul>\n<li> Na in\u0161tal\u00e1ciu <i>supertubes CLI<\/i> vyu\u017eite pokyny uveden\u00e9 <noindex><a rel=\"nofollow\" href=\"https:\/\/banzaicloud.com\/docs\/supertubes\/cli\/install\/\">tu<\/a><\/noindex>.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p>\n<img decoding=\"async\" alt=\"Ur\u010dujeme vhodn\u00fa ve\u013ekos\u0165 pre Kafka klaster v Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/23ecf423b1ea66813406943ae7eabf61.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<\/p>\n<h3>Klastor EKS<\/h3>\n<p>\nPripravte klaster EKS s vyhraden\u00fdmi pracovn\u00fdmi uzlami <i>c5.4xlarge<\/i> v r\u00f4znych z\u00f3nach dostupnosti pre pod s brokeri Kafka, rovnako ako vyhraden\u00e9 uzly pre gener\u00e1tor za\u0165a\u017eenia a monitorovaciu infra\u0161trukt\u00faru.<\/p>\n<pre><code class=\"bash\">banzai cluster create -f https:\/\/raw.githubusercontent.com\/banzaicloud\/kafka-operator\/master\/docs\/benchmarks\/infrastructure\/cluster_eks_202001.json<\/code><\/pre>\n<p>\nKe\u010f klaster EKS za\u010dne fungova\u0165, aktivujte jeho integrovan\u00fa <noindex><a rel=\"nofollow\" href=\"https:\/\/banzaicloud.com\/docs\/pipeline\/features\/integrated-services\/\">monitorovaciu slu\u017ebu<\/a><\/noindex> \u2014  t\u00e1 sa rozvinie s Prometheus a Grafana v klastri.<\/p>\n<h3>Syst\u00e9mov\u00e9 komponenty Kafka<\/h3>\n<p>\nNain\u0161talujte syst\u00e9mov\u00e9 komponenty Kafka (Zookeeper, kafka-operator) v EKS pomocou supertubes CLI:<\/p>\n<pre><code class=\"bash\">supertubes install -a --no-democluster --kubeconfig<\/code><\/pre>\n<p><\/p>\n<h3>Klastor Kafka<\/h3>\n<p>\nVo v\u00fdchozom nastaven\u00ed sa v EKS pou\u017e\u00edvaj\u00fa EBS objemy typu <i>gp2<\/i>, preto je potrebn\u00e9 vytvori\u0165 samostatn\u00fa triedu \u00falo\u017eiska zalo\u017een\u00fa na objemoch <i>io1<\/i> pre klaster Kafka:<\/p>\n<pre><code class=\"plaintext\">kubectl create -f - &lt;&lt;EOF\napiVersion: storage.k8s.io\/v1\nkind: StorageClass\nmetadata:\n  name: fast-ssd\nprovisioner: kubernetes.io\/aws-ebs\nparameters:\n  type: io1\n  iopsPerGB: &quot;50&quot;\n  fsType: ext4\nvolumeBindingMode: WaitForFirstConsumer\nEOF<\/code><\/pre>\n<p>\nNastavte pre brokerov parameter <code>min.insync.replicas=3<\/code> a nasadte pody brokerov na uzloch v troch r\u00f4znych z\u00f3nach dostupnosti:<\/p>\n<pre><code class=\"bash\">supertubes cluster create -n kafka --kubeconfig  -f https:\/\/raw.githubusercontent.com\/banzaicloud\/kafka-operator\/master\/docs\/benchmarks\/infrastructure\/kafka_202001_3brokers.yaml --wait --timeout 600<\/code><\/pre>\n<p><\/p>\n<h3>T\u00e9my<\/h3>\n<p>\nS\u00fa\u010dasne sme spustili tri in\u0161tancie gener\u00e1tora z\u00e1\u0165a\u017ee. Ka\u017ed\u00fd z nich p\u00ed\u0161e do svojho vlastn\u00e9ho t\u00e9matu, to znamen\u00e1, \u017ee potrebujeme celkovo tri t\u00e9my:<\/p>\n<pre><code class=\"bash\">supertubes cluster topic create -n kafka --kubeconfig  -f -&lt;&lt;EOF\napiVersion: kafka.banzaicloud.io\/v1alpha1\nkind: KafkaTopic\nmetadata:\n  name: perftest1\nspec:\n  name: perftest1\n  partitions: 12\n  replicationFactor: 3\n  retention.ms: &#039;28800000&#039;\n  cleanup.policy: delete\nEOF\n\nsupertubes cluster topic create -n kafka --kubeconfig  -f -&lt;&lt;EOF\napiVersion: kafka.banzaicloud.io\/v1alpha1\nkind: KafkaTopic\nmetadata:\n    name: perftest2\nspec:\n  name: perftest2\n  partitions: 12\n  replicationFactor: 3\n  retention.ms: &#039;28800000&#039;\n  cleanup.policy: delete\nEOF\n\nsupertubes cluster topic create -n kafka --kubeconfig  -f -&lt;&lt;EOF\napiVersion: kafka.banzaicloud.io\/v1alpha1\nkind: KafkaTopic\nmetadata:\n  name: perftest3\nspec:\n  name: perftest3\n  partitions: 12\n  replicationFactor: 3\n  retention.ms: &#039;28800000&#039;\n  cleanup.policy: delete\nEOF<\/code><\/pre>\n<p>\nPre ka\u017ed\u00e9 t\u00e9ma je faktor replik\u00e1cie 3 \u2014 minim\u00e1lna odpor\u00fa\u010dan\u00e1 hodnota pre vysoko dostupn\u00e9 produk\u010dn\u00e9 syst\u00e9my.<\/p>\n<h3>N\u00e1stroj na generovanie z\u00e1\u0165a\u017ee<\/h3>\n<p>\nSpustili sme tri in\u0161tancie gener\u00e1tora za\u0165a\u017eenia (ka\u017ed\u00e1 zapisovala do samostatn\u00e9ho topicu). Pre pody gener\u00e1tora za\u0165a\u017eenia je potrebn\u00e9 nastavi\u0165 node affinity, aby boli pl\u00e1novan\u00e9 iba na vyhraden\u00e9 uzly:<\/p>\n<pre><code class=\"plaintext\">apiVersion: extensions\/v1beta1\nkind: Deployment\nmetadata:\n  labels:\n    app: loadtest\n  name: perf-load1\n  namespace: kafka\nspec:\n  progressDeadlineSeconds: 600\n  replicas: 1\n  revisionHistoryLimit: 10\n  selector:\n    matchLabels:\n      app: loadtest\n  strategy:\n    rollingUpdate:\n      maxSurge: 25%\n      maxUnavailable: 25%\n    type: RollingUpdate\n  template:\n    metadata:\n      creationTimestamp: null\n      labels:\n        app: loadtest\n    spec:\n      affinity:\n        nodeAffinity:\n          requiredDuringSchedulingIgnoredDuringExecution:\n            nodeSelectorTerms:\n            - matchExpressions:\n              - key: nodepool.banzaicloud.io\/name\n                operator: In\n                values:\n                - loadgen\n      containers:\n      - args:\n        - -brokers=kafka-0:29092,kafka-1:29092,kafka-2:29092,kafka-3:29092\n        - -topic=perftest1\n        - -required-acks=all\n        - -message-size=512\n        - -workers=20\n        image: banzaicloud\/perfload:0.1.0-blog\n        imagePullPolicy: Always\n        name: sangrenel\n        resources:\n          limits:\n            cpu: 2\n            memory: 1Gi\n          requests:\n            cpu: 2\n            memory: 1Gi\n        terminationMessagePath: \/dev\/termination-log\n        terminationMessagePolicy: File\n      dnsPolicy: ClusterFirst\n      restartPolicy: Always\n      schedulerName: default-scheduler\n      securityContext: {}\n      terminationGracePeriodSeconds: 30<\/code><\/pre>\n<p>\nNieko\u013eko aspektov, na ktor\u00e9 treba pouk\u00e1za\u0165:<\/p>\n<ul>\n<li> Gener\u00e1tor z\u00e1\u0165a\u017ee generuje spr\u00e1vy s d\u013a\u017ekou 512 bajtov a publikuje ich v Kafka v d\u00e1vkach po 500 spr\u00e1v.<\/li>\n<li> Pomocou argumentu <code>-required-acks=all<\/code> sa publik\u00e1cia pova\u017euje za \u00faspe\u0161n\u00fa, ke\u010f s\u00fa v\u0161etky synchronizovan\u00e9 repliky spr\u00e1vy prijat\u00e9 a potvrden\u00e9 brokerami Kafka. To znamen\u00e1, \u017ee v benchmarku sme merali nielen r\u00fdchlos\u0165 pr\u00e1ce l\u00eddrov, ktor\u00ed prij\u00edmaj\u00fa spr\u00e1vy, ale aj ich n\u00e1sledn\u00edkov, ktor\u00ed replikuj\u00fa spr\u00e1vy. Cie\u013eom tohto testu nie je oceni\u0165 r\u00fdchlos\u0165 \u010d\u00edtania spotrebite\u013emi <i>(consumers)<\/i> ned\u00e1vno prijat\u00fdch spr\u00e1v, ktor\u00e9 zatia\u013e zost\u00e1vaj\u00fa v str\u00e1nkovom cache syst\u00e9mu a jej porovnanie s r\u00fdchlos\u0165ou \u010d\u00edtania spr\u00e1v, ktor\u00e9 s\u00fa ulo\u017een\u00e9 na disku.<\/li>\n<li> Gener\u00e1tor za\u0165a\u017eenia parallelne sp\u00fa\u0161\u0165a 20 pracovn\u00edkov (<code>-workers=20<\/code>). Ka\u017ed\u00fd pracovn\u00edk obsahuje 5 producentov, ktor\u00ed spolo\u010dne vyu\u017e\u00edvaj\u00fa pripojenie pracovn\u00edka k klastru Kafka. V d\u00f4sledku toho m\u00e1 ka\u017ed\u00fd gener\u00e1tor 100 producentov, ktor\u00ed v\u0161etci posielaj\u00fa spr\u00e1vy do klastru Kafka.<\/li>\n<\/ul>\n<p><\/p>\n<h3>Pozorovanie stavu klasteru<\/h3>\n<p>\nPo\u010das z\u00e1\u0165a\u017eov\u00e9ho testovania klastru Kafka sme tie\u017e sledovali jeho zdravie, aby sme sa uistili o absencii re\u0161tartov podov, nesynchr\u00f3nnych replik a maxim\u00e1lnej priepustnosti s minim\u00e1lnymi v\u00fdkyvmi:<\/p>\n<ul>\n<li> Gener\u00e1tor za\u0165a\u017eenia p\u00ed\u0161e \u0161tandardn\u00fa \u0161tatistiku o po\u010dte publikovan\u00fdch spr\u00e1v a \u00farovni ch\u00fdb. Percento ch\u00fdb by malo zosta\u0165 na hodnote <code>0,00%<\/code>.<\/li>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/linkedin\/cruise-control\">Cruise Control<\/a><\/noindex>, roz\u0161\u00edren\u00fd kafka-operatorom, poskytuje monitorovacie rozhranie, na ktorom m\u00f4\u017eeme sledova\u0165 stav klasteru. Pre zobrazenie tohto rozhrania vykonajte:\n<pre><code class=\"bash\">supertubes cluster cruisecontrol show -n kafka --kubeconfig<\/code><\/pre>\n<\/li>\n<li> \u00darove\u0148 ISR <i>(po\u010det replik \u201esynchronizovan\u00fdch\u201c)<\/i> zmr\u0161\u0165ovanie a roz\u0161irovanie s\u00fa rovn\u00e9 0.<\/li>\n<\/ul>\n<p><\/p>\n<h2>V\u00fdsledky merania<\/h2>\n<p><\/p>\n<h3>3 brokery, ve\u013ekos\u0165 spr\u00e1v \u2014 512 byte<\/h3>\n<p>\nS partitionami, rovnomerne rozdelen\u00fdmi medzi tromi brokermi, sme dosiahli v\u00fdkon <i>~500 MB\/s (pribli\u017ene 990 tis\u00edc spr\u00e1v za sekundu)<\/i>:<\/p>\n<p><img decoding=\"async\" alt=\"Ur\u010dujeme vhodn\u00fa ve\u013ekos\u0165 pre Kafka klaster v Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/df65ff6a1a26e28d5e5a7a5ab1a20d98.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"Ur\u010dujeme vhodn\u00fa ve\u013ekos\u0165 pre Kafka klaster v Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/15c5883c0056c651f0a0bd14e9991967.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"Ur\u010dujeme vhodn\u00fa ve\u013ekos\u0165 pre Kafka klaster v Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/9b708fa5ce809b0d059cba9f5edfbb2f.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nSpotreba pam\u00e4te virtu\u00e1lnym strojom JVM nepresiahla 2 GB:<\/p>\n<p><img decoding=\"async\" alt=\"Ur\u010dujeme vhodn\u00fa ve\u013ekos\u0165 pre Kafka klaster v Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/3320e2eea891055cc0935da492d18ee9.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"Ur\u010dujeme vhodn\u00fa ve\u013ekos\u0165 pre Kafka klaster v Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/6e3976c310309aa8be45fa3fdf547750.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"Ur\u010dujeme vhodn\u00fa ve\u013ekos\u0165 pre Kafka klaster v Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/94baa1b689ae854693fb7cc4ed6e105e.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nPriepustnos\u0165 disku dosiahla maxim\u00e1lnu priepustnos\u0165 I\/O uzla na v\u0161etk\u00fdch troch in\u0161tanci\u00e1ch, na ktor\u00fdch pracovn\u00e9 brokery:<\/p>\n<p><img decoding=\"async\" alt=\"Ur\u010dujeme vhodn\u00fa ve\u013ekos\u0165 pre Kafka klaster v Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/408139ad33cda1294755660df1c128a8.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"Ur\u010dujeme vhodn\u00fa ve\u013ekos\u0165 pre Kafka klaster v Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/391d2df0ddfe82f94abdb200370ff775.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"Ur\u010dujeme vhodn\u00fa ve\u013ekos\u0165 pre Kafka klaster v Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/e7e9fd021a2cdc71992ce672363a78ec.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nZ \u00fadajov o vyu\u017eit\u00ed pam\u00e4te uzlami vypl\u00fdva, \u017ee syst\u00e9mov\u00e9 vyrovn\u00e1vanie a caching zaberali ~10-15 GB:<\/p>\n<p><img decoding=\"async\" alt=\"Ur\u010dujeme vhodn\u00fa ve\u013ekos\u0165 pre Kafka klaster v Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/781887507e01a4903b86871961ff9260.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"Ur\u010dujeme vhodn\u00fa ve\u013ekos\u0165 pre Kafka klaster v Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/63f10177f01342039f04192ea93c8bca.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"Ur\u010dujeme vhodn\u00fa ve\u013ekos\u0165 pre Kafka klaster v Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/48b992cc73a968b3d3b1442f96562c4e.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<\/p>\n<h3>3 brokery, ve\u013ekos\u0165 spr\u00e1v \u2014 100 byte<\/h3>\n<p>\nSo zmenou ve\u013ekosti spr\u00e1v kles\u00e1 priepustnos\u0165 pribli\u017ene o 15-20%: prejavuje sa to \u010dasom str\u00e1ven\u00fdm spracovan\u00edm ka\u017edej spr\u00e1vy. Okrem toho sa z\u00e1\u0165a\u017e procesora zv\u00fd\u0161ila takmer dvojn\u00e1sobne.<\/p>\n<p><img decoding=\"async\" alt=\"Ur\u010dujeme vhodn\u00fa ve\u013ekos\u0165 pre Kafka klaster v Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/2f7dc5a94da371cfe17c9e0fc0ec1941.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"Ur\u010dujeme vhodn\u00fa ve\u013ekos\u0165 pre Kafka klaster v Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/986160a3a4e099a0dbb8aeefaf6f1c2f.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"Ur\u010dujeme vhodn\u00fa ve\u013ekos\u0165 pre Kafka klaster v Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/c08244457bf4dbfadf52838d4ea2969e.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nKe\u010f\u017ee na uzloch brokerov s\u00fa st\u00e1le nevyu\u017eit\u00e9 jadra, v\u00fdkon je mo\u017en\u00e9 zv\u00fd\u0161i\u0165 zmenou konfigur\u00e1cie Kafka. To nie je jednoduch\u00e1 \u00faloha, preto je lep\u0161ie pracova\u0165 so spr\u00e1vami v\u00e4\u010d\u0161ej ve\u013ekosti na zv\u00fd\u0161enie priepustnosti.<\/p>\n<h3>4 brokery, ve\u013ekos\u0165 spr\u00e1v \u2014 512 byte<\/h3>\n<p>\nV\u00fdkon klasteru Kafka je mo\u017en\u00e9 \u013eahko zv\u00fd\u0161i\u0165 jednoducho pridan\u00edm nov\u00fdch brokerov a udr\u017eiavan\u00edm vyv\u00e1\u017eenia partitionov (to zabezpe\u010duje rovnomern\u00e9 rozdelenie z\u00e1\u0165a\u017ee medzi brokermi). V na\u0161om pr\u00edpade po pridan\u00ed brokera sa priepustnos\u0165 klasteru zv\u00fd\u0161ila na <i>~580 MB\/s (~1,1 mili\u00f3na spr\u00e1v za sekundu)<\/i>. R\u0430\u0441\u0442 na\u0161ich o\u010dak\u00e1van\u00ed bol men\u0161\u00ed, ne\u017e sa o\u010dak\u00e1valo: predov\u0161etk\u00fdm je to sp\u00f4soben\u00e9 nerovnov\u00e1hou partitionov (nie v\u0161etci brok\u00e9ri pracuj\u00fa na plnej kapacite).<\/p>\n<p><img decoding=\"async\" alt=\"Ur\u010dujeme vhodn\u00fa ve\u013ekos\u0165 pre Kafka klaster v Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/8c8d2e70739add3c3f88fe42acf98c46.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"Ur\u010dujeme vhodn\u00fa ve\u013ekos\u0165 pre Kafka klaster v Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/a6e4812270c04487de0ad39b996a13f8.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"Ur\u010dujeme vhodn\u00fa ve\u013ekos\u0165 pre Kafka klaster v Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/6162c04cfb87b9d9cd0f3147a4ae1647.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"Ur\u010dujeme vhodn\u00fa ve\u013ekos\u0165 pre Kafka klaster v Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/dae5de3b14be212c679f3627a3b77d7b.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nSpotreba pam\u00e4te JVM zostala pod 2 GB:<\/p>\n<p><img decoding=\"async\" alt=\"Ur\u010dujeme vhodn\u00fa ve\u013ekos\u0165 pre Kafka klaster v Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/0bca07a817d7da3104b08de10979f049.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"Ur\u010dujeme vhodn\u00fa ve\u013ekos\u0165 pre Kafka klaster v Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/7e1e64f80ec140f13e171cfe3be930ce.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"Ur\u010dujeme vhodn\u00fa ve\u013ekos\u0165 pre Kafka klaster v Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/5e9700b1e54e45f9cd5fa018ec07dba1.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"Ur\u010dujeme vhodn\u00fa ve\u013ekos\u0165 pre Kafka klaster v Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/4ba0c4b59bfe72bcb0d2f22056c47d79.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nNa pr\u00e1cu brokerov s \u00falo\u017eiskami vpl\u00fdvala nerovnov\u00e1ha partitionov:<\/p>\n<p><img decoding=\"async\" alt=\"Ur\u010dujeme vhodn\u00fa ve\u013ekos\u0165 pre Kafka klaster v Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/2b8ed3c21adf70c751d1daa6d4c41895.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"Ur\u010dujeme vhodn\u00fa ve\u013ekos\u0165 pre Kafka klaster v Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/77b980869b99903d9de5cd7d43d79291.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"Ur\u010dujeme vhodn\u00fa ve\u013ekos\u0165 pre Kafka klaster v Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/f2f9569a3a1ef0ce35e89b2e52e4a8b2.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"Ur\u010dujeme vhodn\u00fa ve\u013ekos\u0165 pre Kafka klaster v Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/191d6d97b32b2128c5cfd3914171de15.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<\/p>\n<h2>Zhrnutie<\/h2>\n<p>\nPredlo\u017een\u00fd iterat\u00edvny pr\u00edstup m\u00f4\u017ee by\u0165 roz\u0161\u00edren\u00fd na pokrytie zlo\u017eitej\u0161\u00edch scen\u00e1rov, ktor\u00e9 zah\u0155\u0148aj\u00fa stovky consumerov, repartitioning, aktualiz\u00e1cie, re\u0161tarty podov at\u010f. To v\u0161etko n\u00e1m umo\u017e\u0148uje zhodnoti\u0165 limity schopnost\u00ed klasteru Kafka v r\u00f4znych podmienkach, odhali\u0165 \u00fazke miesta v jeho pracovnom procese a n\u00e1js\u0165 sp\u00f4soby, ako sa s nimi vyrovna\u0165.<\/p>\n<p>Vyvinuli sme Supertubes pre r\u00fdchle a jednoduch\u00e9 nasadenie klasteru, jeho konfigur\u00e1ciu, prid\u00e1vanie\/odstra\u0148ovanie brokerov a t\u00e9m, reagovanie na upozornenia a zabezpe\u010denie spr\u00e1vneho fungovania Kafka v Kubernetes ako celku. Na\u0161\u00edm cie\u013eom je pom\u00f4c\u0165 s\u00fastredi\u0165 sa na hlavn\u00fd \u00fakol ('generova\u0165' a 'spotrebov\u00e1va\u0165' spr\u00e1vy Kafka), zatia\u013e \u010do cel\u00fa \u0165a\u017ek\u00fa pr\u00e1cu nech\u00e1me Supertubes a Kafka operatorovi.<\/p>\n<p>Ak v\u00e1s zauj\u00edmaj\u00fa technol\u00f3gie a Open Source projekty Banzai Cloud, sledujte spolo\u010dnos\u0165 na <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/banzaicloud\">GitHub<\/a><\/noindex>, <noindex><a rel=\"nofollow\" href=\"https:\/\/www.linkedin.com\/company\/banzaicloud\">LinkedIn<\/a><\/noindex> alebo <noindex><a rel=\"nofollow\" href=\"https:\/\/twitter.com\/BanzaiCloud\">Twitter<\/a><\/noindex>.<\/p>\n<h2>P.S. od prekladate\u013ea<\/h2>\n<p>\nPre\u010d\u00edtajte si tie\u017e na na\u0161om blogu:<\/p>\n<ul>\n<li> \u00ab<noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/480722\/\">Jedna hist\u00f3ria s oper\u00e1torom Redis v K8s a mini-preh\u013ead n\u00e1strojov na anal\u00fdzu d\u00e1t tejto datab\u00e1zy<\/a><\/noindex>\u00bb;<\/li>\n<li> \u00ab<noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/450662\/\">Zlo\u017eit\u00e1 migr\u00e1cia RabbitMQ do Kubernetes<\/a><\/noindex>\u00bb;<\/li>\n<li> \u00ab<noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/329224\/\">zetcd od CoreOS: Nahradenie ZooKeepera\u2026 \u00falo\u017eiskom etcd<\/a><\/noindex>\u00bb.<\/li>\n<\/ul>\n<p>Zdroj: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/488920\/\">habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u041f\u0440\u0438\u043c. \u043f\u0435\u0440\u0435\u0432.: \u0412 \u044d\u0442\u043e\u0439 \u0441\u0442\u0430\u0442\u044c\u0435 \u043a\u043e\u043c\u043f\u0430\u043d\u0438\u044f Banzai Cloud \u0434\u0435\u043b\u0438\u0442\u0441\u044f \u043f\u0440\u0438\u043c\u0435\u0440\u043e\u043c \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u043d\u0438\u044f \u0435\u0451 \u0441\u043f\u0435\u0446\u0438\u0430\u043b\u044c\u043d\u044b\u0445 \u0443\u0442\u0438\u043b\u0438\u0442 \u0434\u043b\u044f \u043e\u0431\u043b\u0435\u0433\u0447\u0435\u043d\u0438\u044f \u044d\u043a\u0441\u043f\u043b\u0443\u0430\u0442\u0430\u0446\u0438\u0438 Kafka \u0432 \u0440\u0430\u043c\u043a\u0430\u0445 Kubernetes. \u041f\u0440\u0438\u0432\u043e\u0434\u0438\u043c\u044b\u0435 \u0438\u043d\u0441\u0442\u0440\u0443\u043a\u0446\u0438\u0438 \u0438\u043b\u043b\u044e\u0441\u0442\u0440\u0438\u0440\u0443\u044e\u0442, \u043a\u0430\u043a \u043c\u043e\u0436\u043d\u043e \u043e\u043f\u0440\u0435\u0434\u0435\u043b\u0438\u0442\u044c \u043e\u043f\u0442\u0438\u043c\u0430\u043b\u044c\u043d\u044b\u0439 \u0440\u0430\u0437\u043c\u0435\u0440 \u0438\u043d\u0444\u0440\u0430\u0441\u0442\u0440\u0443\u043a\u0442\u0443\u0440\u044b \u0438 \u043d\u0430\u0441\u0442\u0440\u043e\u0438\u0442\u044c \u0441\u0430\u043c\u0443 Kafka \u0434\u043b\u044f \u0434\u043e\u0441\u0442\u0438\u0436\u0435\u043d\u0438\u044f \u0442\u0440\u0435\u0431\u0443\u0435\u043c\u043e\u0439 \u043f\u0440\u043e\u043f\u0443\u0441\u043a\u043d\u043e\u0439 \u0441\u043f\u043e\u0441\u043e\u0431\u043d\u043e\u0441\u0442\u0438. Apache Kafka \u2014 \u0440\u0430\u0441\u043f\u0440\u0435\u0434\u0435\u043b\u0451\u043d\u043d\u0430\u044f \u0441\u0442\u0440\u0438\u043c\u0438\u043d\u0433\u043e\u0432\u0430\u044f \u043f\u043b\u0430\u0442\u0444\u043e\u0440\u043c\u0430 \u0434\u043b\u044f \u0441\u043e\u0437\u0434\u0430\u043d\u0438\u044f \u043d\u0430\u0434\u0451\u0436\u043d\u044b\u0445, \u043c\u0430\u0441\u0448\u0442\u0430\u0431\u0438\u0440\u0443\u0435\u043c\u044b\u0445 \u0438 \u0432\u044b\u0441\u043e\u043a\u043e\u043f\u0440\u043e\u0438\u0437\u0432\u043e\u0434\u0438\u0442\u0435\u043b\u044c\u043d\u044b\u0445 \u043f\u043e\u0442\u043e\u043a\u043e\u0432\u044b\u0445 \u0441\u0438\u0441\u0442\u0435\u043c \u0440\u0435\u0430\u043b\u044c\u043d\u043e\u0433\u043e \u0432\u0440\u0435\u043c\u0435\u043d\u0438. [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":71214,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-71213","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.4 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u041f\u0440\u0438\u043c. \u043f\u0435\u0440\u0435\u0432.: \u0412 \u044d\u0442\u043e\u0439 \u0441\u0442\u0430\u0442\u044c\u0435 \u043a\u043e\u043c\u043f\u0430\u043d\u0438\u044f Banzai Cloud \u0434\u0435\u043b\u0438\u0442\u0441\u044f \u043f\u0440\u0438\u043c\u0435\u0440\u043e\u043c \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u043d\u0438\u044f \u0435\u0451 \u0441\u043f\u0435\u0446\u0438\u0430\u043b\u044c\u043d\u044b\u0445 \u0443\u0442\u0438\u043b\u0438\u0442 \u0434\u043b\u044f \u043e\u0431\u043b\u0435\u0433\u0447\u0435\u043d\u0438\u044f \u044d\u043a\u0441\u043f\u043b\u0443\u0430\u0442\u0430\u0446\u0438\u0438 Kafka \u0432 \u0440\u0430\u043c\u043a\u0430\u0445 Kubernetes.\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/opredelyaem-podhodyashhij-razmer-dlya-klastera-kafka-v-kubernetes\" \/>\n\t\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.4\" \/>\n\t\t<meta property=\"og:locale\" content=\"sk_SK\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47\u041e\u043f\u0440\u0435\u0434\u0435\u043b\u044f\u0435\u043c \u043f\u043e\u0434\u0445\u043e\u0434\u044f\u0449\u0438\u0439 \u0440\u0430\u0437\u043c\u0435\u0440 \u0434\u043b\u044f \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0430 Kafka \u0432 Kubernetes | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u041f\u0440\u0438\u043c. \u043f\u0435\u0440\u0435\u0432.: \u0412 \u044d\u0442\u043e\u0439 \u0441\u0442\u0430\u0442\u044c\u0435 \u043a\u043e\u043c\u043f\u0430\u043d\u0438\u044f Banzai Cloud \u0434\u0435\u043b\u0438\u0442\u0441\u044f \u043f\u0440\u0438\u043c\u0435\u0440\u043e\u043c \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u043d\u0438\u044f \u0435\u0451 \u0441\u043f\u0435\u0446\u0438\u0430\u043b\u044c\u043d\u044b\u0445 \u0443\u0442\u0438\u043b\u0438\u0442 \u0434\u043b\u044f \u043e\u0431\u043b\u0435\u0433\u0447\u0435\u043d\u0438\u044f \u044d\u043a\u0441\u043f\u043b\u0443\u0430\u0442\u0430\u0446\u0438\u0438 Kafka \u0432 \u0440\u0430\u043c\u043a\u0430\u0445 Kubernetes.\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/opredelyaem-podhodyashhij-razmer-dlya-klastera-kafka-v-kubernetes\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2020-02-24T13:09:17+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-03-03T13:14:29+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47 Ur\u010dujeme vhodn\u00fa ve\u013ekos\u0165 pre klaster Kafka v Kubernetes | ProHoster","description":"Pozn. prekl.: V tomto \u010dl\u00e1nku spolo\u010dnos\u0165 Banzai Cloud zdie\u013ea pr\u00edklad pou\u017eitia svojich \u0161pecializovan\u00fdch n\u00e1strojov na u\u013eah\u010denie prev\u00e1dzky Kafka v r\u00e1mci Kubernetes.","canonical_url":"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/opredelyaem-podhodyashhij-razmer-dlya-klastera-kafka-v-kubernetes","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"sk_SK","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u041e\u043f\u0440\u0435\u0434\u0435\u043b\u044f\u0435\u043c \u043f\u043e\u0434\u0445\u043e\u0434\u044f\u0449\u0438\u0439 \u0440\u0430\u0437\u043c\u0435\u0440 \u0434\u043b\u044f \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0430 Kafka \u0432 Kubernetes | ProHoster","og:description":"\u041f\u0440\u0438\u043c. \u043f\u0435\u0440\u0435\u0432.: \u0412 \u044d\u0442\u043e\u0439 \u0441\u0442\u0430\u0442\u044c\u0435 \u043a\u043e\u043c\u043f\u0430\u043d\u0438\u044f Banzai Cloud \u0434\u0435\u043b\u0438\u0442\u0441\u044f \u043f\u0440\u0438\u043c\u0435\u0440\u043e\u043c \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u043d\u0438\u044f \u0435\u0451 \u0441\u043f\u0435\u0446\u0438\u0430\u043b\u044c\u043d\u044b\u0445 \u0443\u0442\u0438\u043b\u0438\u0442 \u0434\u043b\u044f \u043e\u0431\u043b\u0435\u0433\u0447\u0435\u043d\u0438\u044f \u044d\u043a\u0441\u043f\u043b\u0443\u0430\u0442\u0430\u0446\u0438\u0438 Kafka \u0432 \u0440\u0430\u043c\u043a\u0430\u0445 Kubernetes.","og:url":"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/opredelyaem-podhodyashhij-razmer-dlya-klastera-kafka-v-kubernetes","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-02-24T13:09:17+00:00","article:modified_time":"2020-03-03T13:14:29+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"71213","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 19:06:22","updated":"2022-10-04 20:58:00","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/posts\/71213","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/comments?post=71213"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/posts\/71213\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/media\/71214"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/media?parent=71213"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/categories?post=71213"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/tags?post=71213"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}