{"id":71213,"date":"2020-02-24T16:09:17","date_gmt":"2020-02-24T13:09:17","guid":{"rendered":"https:\/\/prohoster.info\/blog\/opredelyaem-podhodyashhij-razmer-dlya-klastera-kafka-v-kubernetes"},"modified":"2020-03-03T16:14:29","modified_gmt":"2020-03-03T13:14:29","slug":"opredelyaem-podhodyashhij-razmer-dlya-klastera-kafka-v-kubernetes","status":"publish","type":"post","link":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/opredelyaem-podhodyashhij-razmer-dlya-klastera-kafka-v-kubernetes","title":{"rendered":"Ur\u010den\u00ed vhodn\u00e9 velikosti pro Kafka cluster v Kubernetes","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p><i><b>Pozn. p\u0159ek.<\/b>: V tomto \u010dl\u00e1nku spole\u010dnost Banzai Cloud sd\u00edl\u00ed p\u0159\u00edklad pou\u017eit\u00ed sv\u00fdch speci\u00e1ln\u00edch n\u00e1stroj\u016f pro usnadn\u011bn\u00ed spr\u00e1vy Kafka v r\u00e1mci Kubernetes. Uveden\u00e9 pokyny ilustruj\u00ed, jak lze ur\u010dit optim\u00e1ln\u00ed velikost infrastruktury a nastavit samotnou Kafka pro dosa\u017een\u00ed po\u017eadovan\u00e9 propustnosti.<\/i><\/p>\n<p><img decoding=\"async\" alt=\"Ur\u010den\u00ed vhodn\u00e9 velikosti pro Kafka cluster v Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/8a598ec7db091c2944a6442f9bc124da.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nApache Kafka je distribuovan\u00e1 streamovac\u00ed platforma pro vytv\u00e1\u0159en\u00ed spolehliv\u00fdch, \u0161k\u00e1lovateln\u00fdch a vysoce v\u00fdkonn\u00fdch syst\u00e9m\u016f zpracov\u00e1n\u00ed re\u00e1ln\u00e9ho \u010dasu. Jej\u00ed \u00fa\u017easn\u00e9 mo\u017enosti lze roz\u0161\u00ed\u0159it pomoc\u00ed Kubernetes. K tomu jsme vyvinuli <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/banzaicloud\/kafka-operator\">Open Source oper\u00e1tor Kafka<\/a><\/noindex> a n\u00e1stroj s n\u00e1zvem <noindex><a rel=\"nofollow\" href=\"https:\/\/banzaicloud.com\/docs\/supertubes\/overview\/\">Supertubes<\/a><\/noindex>. Tyto n\u00e1stroje umo\u017e\u0148uj\u00ed spou\u0161t\u011bt Kafka v Kubernetes a vyu\u017e\u00edvat jej\u00ed r\u016fzn\u00e9 funkce, jako je jemn\u00e9 lad\u011bn\u00ed konfigurace brokera, \u0161k\u00e1lov\u00e1n\u00ed na z\u00e1klad\u011b metrik s p\u0159erozd\u011blen\u00edm, rack awareness (v\u011bdom\u00ed o hardwarov\u00fdch zdroj\u00edch), \"pohodln\u00e9\" <i>(graceful)<\/i> nasazen\u00ed aktualizac\u00ed atd.<noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><\/p>\n<blockquote><p>Vyzkou\u0161ejte Supertubes ve sv\u00e9m klastru:<\/p>\n<pre><code class=\"bash\">curl https:\/\/getsupertubes.sh | sh a supertubes install -a --no-democluster --kubeconfig<\/code><\/pre>\n<p>\nNebo se obra\u0165te na <noindex><a rel=\"nofollow\" href=\"https:\/\/banzaicloud.com\/docs\/supertubes\/cli\/reference\/\">dokumentaci<\/a><\/noindex>. Tak\u00e9 si m\u016f\u017eete p\u0159e\u010d\u00edst o n\u011bkter\u00fdch funkc\u00edch Kafka, jejich\u017e pou\u017eit\u00ed je automatizov\u00e1no pomoc\u00ed Supertubes a Kafka oper\u00e1tora. O t\u011bchto funkc\u00edch jsme ji\u017e psali v blogu:<\/p>\n<ul>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/banzaicloud.com\/blog\/kafka-operator\/\">Oh ne! Je\u0161t\u011b jeden Kafka oper\u00e1tor pro Kubernetes<\/a><\/noindex>;<\/li>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/banzaicloud.com\/blog\/kafka-alert\/\">Monitorov\u00e1n\u00ed a provoz Kafka na z\u00e1klad\u011b metrik Prometheus<\/a><\/noindex>;<\/li>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/banzaicloud.com\/blog\/kafka-rack-awareness\/\">Kafka rack awareness na Kubernetes<\/a><\/noindex>;<\/li>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/banzaicloud.com\/blog\/kafka-on-istio-performance\/\">Spu\u0161t\u011bn\u00ed Apache Kafka p\u0159es Istio \u2014 benchmark<\/a><\/noindex>;<\/li>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/banzaicloud.com\/blog\/kafka-topic-user-management\/\">U\u017eivatel autentizovan\u00fd a p\u0159\u00edstup kontrolovan\u00e9 klastry s Kafka oper\u00e1torem<\/a><\/noindex>;<\/li>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/banzaicloud.com\/blog\/kafka-rolling-upgrade\/\">Rolling upgrade Kafka a dynamick\u00e1 konfigurace na Kubernetes<\/a><\/noindex>;<\/li>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/banzaicloud.com\/blog\/kafka-envoy-protocol-filter\/\">Envoy protokolov\u00fd filtr pro Kafka, ve meshovan\u00e9 s\u00edti<\/a><\/noindex>.<\/li>\n<\/ul>\n<\/blockquote>\n<p>\nPokud se rozhodnete nasadit klastr Kafka v Kubernetes, pravd\u011bpodobn\u011b se setk\u00e1te s probl\u00e9mem ur\u010den\u00ed optim\u00e1ln\u00ed velikosti z\u00e1kladn\u00ed infrastruktury a nutnost\u00ed jemn\u011b ladit konfiguraci Kafka tak, aby vyhovovala po\u017eadavk\u016fm na propustnost. Maxim\u00e1ln\u00ed v\u00fdkon ka\u017ed\u00e9ho brokera je ur\u010den v\u00fdkonem komponent\u016f infrastruktury, kter\u00e9 ho tvo\u0159\u00ed, jako je pam\u011b\u0165, procesor, rychlost disku, \u0161\u00ed\u0159ka p\u00e1sma s\u00edt\u011b atd.<\/p>\n<p>Ide\u00e1ln\u011b by m\u011bla b\u00fdt konfigurace brokera nastavena tak, aby v\u0161echny prvky infrastruktury pracovaly na maxim\u00e1ln\u00ed v\u00fdkon. Ve skute\u010dnosti je v\u0161ak takov\u00e9 nastaven\u00ed velmi slo\u017eit\u00e9. Je pravd\u011bpodobn\u011bj\u0161\u00ed, \u017ee u\u017eivatel\u00e9 budou konfigurovat brokery tak, aby maximalizovali vyu\u017eit\u00ed jednoho nebo dvou komponent (disku, pam\u011bti nebo procesoru). Obecn\u011b plat\u00ed, \u017ee broker dosahuje maxim\u00e1ln\u00edho v\u00fdkonu, kdy\u017e jeho konfigurace umo\u017e\u0148uje vyu\u017eit\u00ed nejpomalej\u0161\u00edho komponentu \"naplno\". Tak m\u016f\u017eeme z\u00edskat p\u0159edstavu o z\u00e1t\u011b\u017ei, se kterou si jeden broker dok\u00e1\u017ee poradit.<\/p>\n<p>Teoreticky m\u016f\u017eeme tak\u00e9 odhadnout po\u010det broker\u016f pot\u0159ebn\u00fdch pro pr\u00e1ci se stanovenou z\u00e1t\u011b\u017e\u00ed. V praxi je v\u0161ak mo\u017enost\u00ed nastaven\u00ed na r\u016fzn\u00fdch \u00farovn\u00edch tolik, \u017ee je velmi obt\u00ed\u017en\u00e9 (pokud ne nemo\u017en\u00e9) posoudit potenci\u00e1ln\u00ed v\u00fdkon ur\u010dit\u00e9 konfigurace. Jin\u00fdmi slovy, je velmi slo\u017eit\u00e9 napl\u00e1novat konfiguraci na z\u00e1klad\u011b stanoven\u00e9ho v\u00fdkonu.<\/p>\n<p>Pro u\u017eivatele Supertubes obvykle uplat\u0148ujeme n\u00e1sleduj\u00edc\u00ed postup: za\u010d\u00edn\u00e1me s ur\u010ditou konfigurac\u00ed (infrastruktura + nastaven\u00ed), pot\u00e9 m\u011b\u0159\u00edme jej\u00ed v\u00fdkon, korigujeme nastaven\u00ed brokera a proces opakujeme. Toto se d\u011bje, dokud nen\u00ed pln\u011b vyu\u017eit potenci\u00e1l nejpomalej\u0161\u00edho komponentu infrastruktury.<\/p>\n<p>T\u00edmto zp\u016fsobem z\u00edsk\u00e1v\u00e1me jasn\u011bj\u0161\u00ed p\u0159edstavu o tom, kolik broker\u016f je pot\u0159ebn\u00fdch pro cluster, aby si poradil s ur\u010ditou z\u00e1t\u011b\u017e\u00ed (po\u010det broker\u016f tak\u00e9 z\u00e1vis\u00ed na dal\u0161\u00edch faktorech, jako je minim\u00e1ln\u00ed po\u010det replik zpr\u00e1v pro zaji\u0161t\u011bn\u00ed odolnosti, po\u010det leader\u016f partition atd.). Krom\u011b toho z\u00edsk\u00e1v\u00e1me p\u0159edstavu o tom, pro kter\u00fd infrastrukturn\u00ed komponent je \u017e\u00e1douc\u00ed vertik\u00e1ln\u00ed \u0161k\u00e1lov\u00e1n\u00ed.<\/p>\n<p>V tomto \u010dl\u00e1nku se budeme v\u011bnovat krok\u016fm, kter\u00e9 podnik\u00e1me, abychom \u201evyt\u011b\u017eili maximum\u201c z nejpomalej\u0161\u00edch komponent v po\u010d\u00e1te\u010dn\u00edch konfigurac\u00edch a zm\u011b\u0159ili propustnost clusteru Kafka. Vysoce odoln\u00e1 konfigurace vy\u017eaduje alespo\u0148 t\u0159i funguj\u00edc\u00ed brokery (<code>min.insync.replicas=3<\/code>), rozlo\u017een\u00fdch do t\u0159\u00ed r\u016fzn\u00fdch z\u00f3n dostupnosti. K nastaven\u00ed, \u0161k\u00e1lov\u00e1n\u00ed a monitorov\u00e1n\u00ed infrastruktury Kubernetes pou\u017e\u00edv\u00e1me na\u0161i vlastn\u00ed platformu pro spr\u00e1vu kontejner\u016f pro hybridn\u00ed cloudy \u2014 <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/banzaicloud\/pipeline\">Pipeline<\/a><\/noindex>. Podporuje on-premise (bare metal, VMware) a p\u011bt typ\u016f cloud\u016f (Alibaba, AWS, Azure, Google, Oracle), jako\u017e i libovoln\u00e9 jejich kombinace.<\/p>\n<h2>My\u0161lenky na infrastrukturu a konfiguraci klastru Kafka<\/h2>\n<p>\nPro n\u00ed\u017ee uveden\u00e9 p\u0159\u00edklady jsme vybrali AWS jako poskytovatele cloudov\u00fdch slu\u017eeb a EKS jako distribuci Kubernetes. Podobnou konfiguraci lze realizovat pomoc\u00ed <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/banzaicloud\/pke\">PKE<\/a><\/noindex> \u2014 distribuce Kubernetes od Banzai Cloud, certifikovan\u00e1 CNCF.<\/p>\n<h3>Disk<\/h3>\n<p>\nAmazon nab\u00edz\u00ed r\u016fzn\u00e9 <noindex><a rel=\"nofollow\" href=\"https:\/\/docs.aws.amazon.com\/AWSEC2\/latest\/UserGuide\/ebs-volume-types.html\">typy svazk\u016f EBS<\/a><\/noindex>. Z\u00e1kladem <i>gp2<\/i> a <i>io1<\/i> jsou SSD disky, av\u0161ak pro zaji\u0161t\u011bn\u00ed vysok\u00e9 propustnosti <i>gp2<\/i> spot\u0159ebov\u00e1v\u00e1 nasb\u00edran\u00e9 kredity <i>I\/O credits<\/i>, proto jsme preferovali typ <i>io1<\/i>, kter\u00fd nab\u00edz\u00ed stabiln\u00ed vysokou propustnost.<\/p>\n<h3>Typy instanc\u00ed<\/h3>\n<p>\nV\u00fdkon Kafka siln\u011b z\u00e1vis\u00ed na str\u00e1nkov\u00e9m cache opera\u010dn\u00edho syst\u00e9mu, proto pot\u0159ebujeme instance se dostate\u010dn\u00fdm mno\u017estv\u00edm pam\u011bti pro brokery (JVM) a str\u00e1nkov\u00fd cache. Instance <i>c5.2xlarge<\/i> je dobr\u00fdm za\u010d\u00e1tkem, nebo\u0165 m\u00e1 16 GB pam\u011bti a <noindex><a rel=\"nofollow\" href=\"https:\/\/docs.aws.amazon.com\/AWSEC2\/latest\/UserGuide\/ebs-optimized.html\">je optimalizov\u00e1na pro pr\u00e1ci s EBS.<\/a><\/noindex>Jej\u00ed nev\u00fdhodou je, \u017ee dok\u00e1\u017ee poskytovat maxim\u00e1ln\u00ed v\u00fdkon pouze po dobu 30 minut ka\u017ed\u00fdch 24 hodin. Pokud pracovn\u00ed z\u00e1t\u011b\u017e vy\u017eaduje maxim\u00e1ln\u00ed v\u00fdkon po del\u0161\u00ed dobu, je t\u0159eba se poohl\u00e9dnout po jin\u00fdch typech instanc\u00ed. To jsme tak u\u010dinili, a zastavili se na <i>c5.4xlarge.<\/i>Zaji\u0161\u0165uje maxim\u00e1ln\u00ed propustnost ve <b>593,75 MB\/s.<\/b>Maxim\u00e1ln\u00ed propustnost svazku EBS <i>io1<\/i> je vy\u0161\u0161\u00ed ne\u017e u instance <i>c5.4xlarge.<\/i>, proto je nejpomalej\u0161\u00edm prvkem infrastruktury propustnost I\/O tohoto typu instance (co\u017e by m\u011bly tak\u00e9 potvrdit v\u00fdsledky na\u0161ich z\u00e1t\u011b\u017eov\u00fdch test\u016f).<\/p>\n<h3>S\u00ed\u0165<\/h3>\n<p>\nPropustnost s\u00edt\u011b by m\u011bla b\u00fdt dostate\u010dn\u011b vysok\u00e1 ve srovn\u00e1n\u00ed s v\u00fdkonem instance VM a disku, jinak se s\u00ed\u0165 st\u00e1v\u00e1 \u00fazk\u00fdm m\u00edstem. V na\u0161em p\u0159\u00edpad\u011b s\u00ed\u0165ov\u00e9 rozhran\u00ed <i>c5.4xlarge.<\/i> podporuje rychlost a\u017e 10 Gb\/s, co\u017e je v\u00fdrazn\u011b vy\u0161\u0161\u00ed ne\u017e propustnost I\/O instance VM.<\/p>\n<h3>Nasazen\u00ed broker\u016f<\/h3>\n<p>\nBroke\u0159i by m\u011bli b\u00fdt nasazeni (pl\u00e1nov\u00e1ni v Kubernetes) na dedikovan\u00e9 uzly, aby se vyhnuli sout\u011b\u017een\u00ed s jin\u00fdmi procesy o zdroje CPU, pam\u011bti, s\u00edt\u011b a disku.<\/p>\n<h3>Verze Java<\/h3>\n<p>\nLogickou volbou je Java 11, proto\u017ee je kompatibiln\u00ed s Dockerem, co\u017e znamen\u00e1, \u017ee JVM spr\u00e1vn\u011b identifikuje dostupn\u00e9 procesory a pam\u011b\u0165 pro kontejner, ve kter\u00e9m broker b\u011b\u017e\u00ed. S v\u011bdom\u00edm, \u017ee limity CPU jsou d\u016fle\u017eit\u00e9, JVM intern\u011b a transparentn\u011b nastavuje po\u010det GC a JIT-kompil\u00e1torov\u00fdch vl\u00e1ken. Pou\u017eili jsme obraz Kafka <code>banzaicloud\/kafka:2.13-2.4.0<\/code>, kter\u00fd zahrnuje verzi Kafka 2.4.0 (Scala 2.13) na Java 11.<\/p>\n<blockquote><p>Pokud se chcete dozv\u011bd\u011bt v\u00edce o Java\/JVM na Kubernetes, pod\u00edvejte se na na\u0161e n\u00e1sleduj\u00edc\u00ed publikace:<\/p>\n<ul>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/banzaicloud.com\/blog\/java-resource-limits\/\">Pro\u010d byla m\u00e1 Java aplikace OOMKilled<\/a><\/noindex>;<\/li>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/banzaicloud.com\/blog\/java10-container-sizing\/\">Jak spr\u00e1vn\u011b dimenzovat kontejnery pro aplikace Java 10<\/a><\/noindex>.<\/li>\n<\/ul>\n<\/blockquote>\n<p><\/p>\n<h3>Nastaven\u00ed pam\u011bti brokera<\/h3>\n<p>\nExistuj\u00ed dva kl\u00ed\u010dov\u00e9 aspekty p\u0159i konfiguraci pam\u011bti brokera: nastaven\u00ed pro JVM a pro pod Kubernetes. Limit pam\u011bti nastaven\u00fd pro pod by m\u011bl b\u00fdt v\u011bt\u0161\u00ed ne\u017e maxim\u00e1ln\u00ed heap size, aby m\u011bla JVM prostor pro metaprostor Java, kter\u00fd je ulo\u017een v samostatn\u00e9 pam\u011bti, a pro str\u00e1nkovac\u00ed cache opera\u010dn\u00edho syst\u00e9mu, kterou Kafka aktivn\u011b vyu\u017e\u00edv\u00e1. V na\u0161ich testech jsme spou\u0161t\u011bli brokery Kafka s parametry <code>-Xmx4G -Xms2G<\/code>, p\u0159i\u010dem\u017e limit pam\u011bti pro pod byl <code>10 Gi<\/code>. Vezm\u011bte pros\u00edm na v\u011bdom\u00ed, \u017ee nastaven\u00ed pam\u011bti pro JVM lze automaticky z\u00edskat pomoc\u00ed <code>-XX:MaxRAMPercentage<\/code> a <code>-X:MinRAMPercentage<\/code>, vych\u00e1zej\u00edc z limitu pam\u011bti pro pod.<\/p>\n<h3>Nastaven\u00ed CPU brokera<\/h3>\n<p>\nObecn\u011b lze zv\u00fd\u0161it v\u00fdkon zvy\u0161ov\u00e1n\u00edm paralelismu pomoc\u00ed zvy\u0161ov\u00e1n\u00ed po\u010dtu vl\u00e1ken, kter\u00e9 Kafka pou\u017e\u00edv\u00e1. \u010c\u00edm v\u00edce procesor\u016f je pro Kafka k dispozici, t\u00edm l\u00e9pe. V na\u0161em testu jsme za\u010dali s limitem 6 procesor\u016f a postupn\u011b (iteracemi) jsme zvy\u0161ovali jejich po\u010det na 15. Krom\u011b toho jsme nastavili <code>num.network.threads=12<\/code> v nastaven\u00ed brokera, abychom zv\u00fd\u0161ili po\u010det vl\u00e1ken, kter\u00e9 p\u0159ij\u00edmaj\u00ed data ze s\u00edt\u011b a pos\u00edlaj\u00ed je. Hned jsme zjistili, \u017ee n\u00e1slednick\u00e9 brokery nedok\u00e1\u017eou rychle z\u00edsk\u00e1vat repliky, a tak jsme zv\u00fd\u0161ili <code>num.replica.fetchers<\/code> na 4, abychom zv\u00fd\u0161ili rychlost, s jakou n\u00e1slednick\u00e9 brokery replicovaly zpr\u00e1vy od leader\u016f.<\/p>\n<h3>N\u00e1stroj pro generov\u00e1n\u00ed z\u00e1t\u011b\u017ee<\/h3>\n<p>\nJe t\u0159eba zajistit, aby potenci\u00e1l vybran\u00e9ho gener\u00e1toru z\u00e1t\u011b\u017ee nebyl vy\u010derp\u00e1n d\u0159\u00edve, ne\u017e dos\u00e1hne cluster Kafka (jeho benchmark) sv\u00e9 maxim\u00e1ln\u00ed z\u00e1t\u011b\u017ee. Jin\u00fdmi slovy, je nutn\u00e9 prov\u00e9st p\u0159edb\u011b\u017en\u00e9 posouzen\u00ed schopnost\u00ed n\u00e1stroje generov\u00e1n\u00ed z\u00e1t\u011b\u017ee a tak\u00e9 vyb\u00edrat pro n\u011bj typy instanc\u00ed s dostate\u010dn\u00fdm po\u010dtem procesor\u016f a pam\u011bti. V tomto p\u0159\u00edpad\u011b n\u00e1\u0161 n\u00e1stroj bude produkovat v\u00edce z\u00e1t\u011b\u017ee, ne\u017e je cluster Kafka schopen zpracovat. Po mnoha pokusech jsme se rozhodli pro t\u0159i instance <i>c5.4xlarge.<\/i>, na ka\u017ed\u00e9 z nich byl spu\u0161t\u011bn gener\u00e1tor.<\/p>\n<h2>Benchmarking<\/h2>\n<p>\nM\u011b\u0159en\u00ed v\u00fdkonnosti je iterativn\u00ed proces zahrnuj\u00edc\u00ed n\u00e1sleduj\u00edc\u00ed f\u00e1ze:<\/p>\n<ul>\n<li> nastaven\u00ed infrastruktury (klastr EKS, klastr Kafka, n\u00e1stroj pro generov\u00e1n\u00ed z\u00e1t\u011b\u017ee a tak\u00e9 Prometheus a Grafana);<\/li>\n<li> generov\u00e1n\u00ed z\u00e1t\u011b\u017ee po ur\u010dit\u00e9 obdob\u00ed pro filtrov\u00e1n\u00ed n\u00e1hodn\u00fdch odchylek ve shrom\u00e1\u017ed\u011bn\u00fdch v\u00fdkonnostn\u00edch ukazatel\u00edch;<\/li>\n<li> dolad\u011bn\u00ed infrastruktury a konfigurace brokera na z\u00e1klad\u011b pozorovan\u00fdch v\u00fdkonnostn\u00edch ukazatel\u016f;<\/li>\n<li> opakov\u00e1n\u00ed procesu, dokud nen\u00ed dosa\u017eeno po\u017eadovan\u00e9 \u00farovn\u011b propustnosti klastru Kafka. Ta by m\u011bla b\u00fdt stabiln\u011b reprodukovateln\u00e1 a vykazovat minim\u00e1ln\u00ed variace propustnosti.<\/li>\n<\/ul>\n<p>\nV n\u00e1sleduj\u00edc\u00ed \u010d\u00e1sti jsou pops\u00e1ny kroky, kter\u00e9 byly provedeny b\u011bhem benchmarku testovac\u00edho klastru.<\/p>\n<h3>N\u00e1stroje<\/h3>\n<p>\nPro rychl\u00e9 uveden\u00ed do provozu z\u00e1kladn\u00ed konfigurace, generov\u00e1n\u00ed z\u00e1t\u011b\u017ee a m\u011b\u0159en\u00ed v\u00fdkonnosti byly pou\u017eity n\u00e1sleduj\u00edc\u00ed n\u00e1stroje:<\/p>\n<ul>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/banzaicloud.com\/docs\/pipeline\/overview\/\">Banzai Cloud Pipeline<\/a><\/noindex> pro organizaci klastru EKS od Amazonu s <noindex><a rel=\"nofollow\" href=\"https:\/\/prometheus.io\/\">Prometheus<\/a><\/noindex> (pro sb\u011br metrik Kafka a infrastruktury) a <noindex><a rel=\"nofollow\" href=\"https:\/\/grafana.com\/\">Grafana<\/a><\/noindex> (pro vizualizaci t\u011bchto metrik). Vyu\u017eili jsme <b>integrovan\u00e9<\/b> v <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/banzaicloud\/pipeline\">Pipeline<\/a><\/noindex> slu\u017eby, kter\u00e9 zaji\u0161\u0165uj\u00ed federativn\u00ed monitorov\u00e1n\u00ed, centralizovan\u00e9 shroma\u017e\u010fov\u00e1n\u00ed log\u016f, skenov\u00e1n\u00ed zranitelnost\u00ed, obnoven\u00ed po hav\u00e1rii, podnikov\u00e9 zabezpe\u010den\u00ed a mnoho dal\u0161\u00edho.<\/li>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/jamiealquiza\/sangrenel\">Sangrenel<\/a><\/noindex> \u2014 n\u00e1stroj pro z\u00e1t\u011b\u017eov\u00e9 testov\u00e1n\u00ed klastru Kafka.<\/li>\n<li> Panely Grafana pro vizualizaci metrik Kafka a infrastruktury: <noindex><a rel=\"nofollow\" href=\"https:\/\/grafana.com\/grafana\/dashboards\/10123\">Kubernetes Kafka<\/a><\/noindex>, <noindex><a rel=\"nofollow\" href=\"https:\/\/grafana.com\/grafana\/dashboards\/1860\">Node Exporter<\/a><\/noindex>.<\/li>\n<li> Supertubes CLI pro maxim\u00e1ln\u011b jednoduch\u00e9 nastaven\u00ed klastry Kafka v Kubernetes. Zookeeper, Kafka operator, Envoy a mnoho dal\u0161\u00edch komponent je nainstalov\u00e1no a spr\u00e1vn\u011b nakonfigurov\u00e1no pro spu\u0161t\u011bn\u00ed produk\u010dn\u00ed klastry Kafka v Kubernetes.\n<ul>\n<li> Pro instalaci <i>supertubes CLI<\/i> vyu\u017eijte pokyny uveden\u00e9 <noindex><a rel=\"nofollow\" href=\"https:\/\/banzaicloud.com\/docs\/supertubes\/cli\/install\/\">zde<\/a><\/noindex>.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p>\n<img decoding=\"async\" alt=\"Ur\u010den\u00ed vhodn\u00e9 velikosti pro Kafka cluster v Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/23ecf423b1ea66813406943ae7eabf61.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<\/p>\n<h3>Klastr EKS<\/h3>\n<p>\nP\u0159ipravte klastr EKS s dedikovan\u00fdmi pracovn\u00edmi uzly <i>c5.4xlarge.<\/i> v r\u016fzn\u00fdch z\u00f3n\u00e1ch dostupnosti pro pody s brokery Kafka a tak\u00e9 vyhrazen\u00e9 uzly pro gener\u00e1tor z\u00e1t\u011b\u017ee a monitorovac\u00ed infrastrukturu.<\/p>\n<pre><code class=\"bash\">banzai cluster create -f https:\/\/raw.githubusercontent.com\/banzaicloud\/kafka-operator\/master\/docs\/benchmarks\/infrastructure\/cluster_eks_202001.json<\/code><\/pre>\n<p>\nJakmile klastr EKS za\u010dne fungovat, zapn\u011bte jeho integrovanou <noindex><a rel=\"nofollow\" href=\"https:\/\/banzaicloud.com\/docs\/pipeline\/features\/integrated-services\/\">monitorovac\u00ed slu\u017ebu<\/a><\/noindex> \u2014 ta nasad\u00ed Prometheus a Grafana do klastru.<\/p>\n<h3>Syst\u00e9mov\u00e9 komponenty Kafka<\/h3>\n<p>\nNainstalujte syst\u00e9mov\u00e9 komponenty Kafka (Zookeeper, kafka-operator) v EKS pomoc\u00ed supertubes CLI:<\/p>\n<pre><code class=\"bash\">supertubes install -a --no-democluster --kubeconfig<\/code><\/pre>\n<p><\/p>\n<h3>Klastr Kafka<\/h3>\n<p>\nVe v\u00fdchoz\u00edm nastaven\u00ed se v EKS pou\u017e\u00edvaj\u00ed svazky EBS typu <i>gp2<\/i>, proto je nutn\u00e9 vytvo\u0159it samostatnou t\u0159\u00eddu \u00falo\u017ei\u0161t\u011b zalo\u017eenou na svazc\u00edch <i>io1<\/i> pro klastr Kafka:<\/p>\n<pre><code class=\"plaintext\">kubectl create -f - &lt;&lt;EOF\napiVersion: storage.k8s.io\/v1\nkind: StorageClass\nmetadata:\n  name: fast-ssd\nprovisioner: kubernetes.io\/aws-ebs\nparameters:\n  type: io1\n  iopsPerGB: &quot;50&quot;\n  fsType: ext4\nvolumeBindingMode: WaitForFirstConsumer\nEOF<\/code><\/pre>\n<p>\nNastavte pro brokery parametr <code>min.insync.replicas=3<\/code> a nasadit pody broker\u016f na uzlech ve t\u0159ech r\u016fzn\u00fdch z\u00f3n\u00e1ch dostupnosti:<\/p>\n<pre><code class=\"bash\">supertubes cluster create -n kafka --kubeconfig  -f https:\/\/raw.githubusercontent.com\/banzaicloud\/kafka-operator\/master\/docs\/benchmarks\/infrastructure\/kafka_202001_3brokers.yaml --wait --timeout 600<\/code><\/pre>\n<p><\/p>\n<h3>T\u00e9mata<\/h3>\n<p>\nSou\u010dasn\u011b jsme spou\u0161t\u011bli t\u0159i instance gener\u00e1toru z\u00e1t\u011b\u017ee. Ka\u017ed\u00e1 z nich zapisuje do sv\u00e9ho t\u00e9matu, co\u017e znamen\u00e1, \u017ee pot\u0159ebujeme celkem t\u0159i t\u00e9mata:<\/p>\n<pre><code class=\"bash\">supertubes cluster topic create -n kafka --kubeconfig  -f -&lt;&lt;EOF\napiVersion: kafka.banzaicloud.io\/v1alpha1\nkind: KafkaTopic\nmetadata:\n  name: perftest1\nspec:\n  name: perftest1\n  partitions: 12\n  replicationFactor: 3\n  retention.ms: &#039;28800000&#039;\n  cleanup.policy: delete\nEOF\n\nsupertubes cluster topic create -n kafka --kubeconfig  -f -&lt;&lt;EOF\napiVersion: kafka.banzaicloud.io\/v1alpha1\nkind: KafkaTopic\nmetadata:\n    name: perftest2\nspec:\n  name: perftest2\n  partitions: 12\n  replicationFactor: 3\n  retention.ms: &#039;28800000&#039;\n  cleanup.policy: delete\nEOF\n\nsupertubes cluster topic create -n kafka --kubeconfig  -f -&lt;&lt;EOF\napiVersion: kafka.banzaicloud.io\/v1alpha1\nkind: KafkaTopic\nmetadata:\n  name: perftest3\nspec:\n  name: perftest3\n  partitions: 12\n  replicationFactor: 3\n  retention.ms: &#039;28800000&#039;\n  cleanup.policy: delete\nEOF<\/code><\/pre>\n<p>\nPro ka\u017ed\u00e9 t\u00e9ma je faktor replikace 3 \u2014 minim\u00e1ln\u011b doporu\u010dovan\u00e1 hodnota pro vysoce dostupn\u00e9 produk\u010dn\u00ed syst\u00e9my.<\/p>\n<h3>N\u00e1stroj pro generov\u00e1n\u00ed z\u00e1t\u011b\u017ee<\/h3>\n<p>\nSpustili jsme t\u0159i instance gener\u00e1toru z\u00e1t\u011b\u017ee (ka\u017ed\u00e1 zapisuje do samostatn\u00e9ho t\u00e9matu). Pro pody gener\u00e1toru z\u00e1t\u011b\u017ee je t\u0159eba nastavit node affinity, aby byly pl\u00e1nov\u00e1ny pouze na vyhrazen\u00e9 uzly:<\/p>\n<pre><code class=\"plaintext\">apiVersion: extensions\/v1beta1\nkind: Deployment\nmetadata:\n  labels:\n    app: loadtest\n  name: perf-load1\n  namespace: kafka\nspec:\n  progressDeadlineSeconds: 600\n  replicas: 1\n  revisionHistoryLimit: 10\n  selector:\n    matchLabels:\n      app: loadtest\n  strategy:\n    rollingUpdate:\n      maxSurge: 25%\n      maxUnavailable: 25%\n    type: RollingUpdate\n  template:\n    metadata:\n      creationTimestamp: null\n      labels:\n        app: loadtest\n    spec:\n      affinity:\n        nodeAffinity:\n          requiredDuringSchedulingIgnoredDuringExecution:\n            nodeSelectorTerms:\n            - matchExpressions:\n              - key: nodepool.banzaicloud.io\/name\n                operator: In\n                values:\n                - loadgen\n      containers:\n      - args:\n        - -brokers=kafka-0:29092,kafka-1:29092,kafka-2:29092,kafka-3:29092\n        - -topic=perftest1\n        - -required-acks=all\n        - -message-size=512\n        - -workers=20\n        image: banzaicloud\/perfload:0.1.0-blog\n        imagePullPolicy: Always\n        name: sangrenel\n        resources:\n          limits:\n            cpu: 2\n            memory: 1Gi\n          requests:\n            cpu: 2\n            memory: 1Gi\n        terminationMessagePath: \/dev\/termination-log\n        terminationMessagePolicy: File\n      dnsPolicy: ClusterFirst\n      restartPolicy: Always\n      schedulerName: default-scheduler\n      securityContext: {}\n      terminationGracePeriodSeconds: 30<\/code><\/pre>\n<p>\nN\u011bkolik bod\u016f, na kter\u00e9 je t\u0159eba d\u00e1vat pozor:<\/p>\n<ul>\n<li> Z\u00e1t\u011b\u017eov\u00fd gener\u00e1tor vytv\u00e1\u0159\u00ed zpr\u00e1vy o velikosti 512 bajt\u016f a publikuje je v Kafka po d\u00e1vk\u00e1ch po 500 zpr\u00e1v\u00e1ch.<\/li>\n<li> Pomoc\u00ed argumentu <code>-required-acks=all<\/code> je publikace pova\u017eov\u00e1na za \u00fasp\u011b\u0161nou, kdy\u017e v\u0161echny synchronizovan\u00e9 repliky zpr\u00e1vy byly p\u0159ijaty a potvrzeny brokery Kafka. To znamen\u00e1, \u017ee v benchmarku m\u011b\u0159\u00edme nejen rychlost pr\u00e1ce lead\u016f, kte\u0159\u00ed p\u0159ij\u00edmaj\u00ed zpr\u00e1vy, ale i jejich n\u00e1sledovn\u00edk\u016f, kte\u0159\u00ed replikuj\u00ed zpr\u00e1vy. \u00dakol t\u00e9to testu neobsahuje hodnocen\u00ed rychlosti \u010dten\u00ed spot\u0159ebiteli <i>(consumers)<\/i> ned\u00e1vno p\u0159ijat\u00fdch zpr\u00e1v, kter\u00e9 zat\u00edm z\u016fst\u00e1vaj\u00ed v pam\u011b\u0165ov\u00e9 cache OS, a jej\u00ed srovn\u00e1n\u00ed s rychlost\u00ed \u010dten\u00ed zpr\u00e1v ulo\u017een\u00fdch na disku.<\/li>\n<li> Gener\u00e1tor z\u00e1t\u011b\u017ee paraleln\u011b spou\u0161t\u00ed 20 worker\u016f (<code>-workers=20<\/code>). Ka\u017ed\u00fd worker obsahuje 5 producer\u016f, kte\u0159\u00ed spole\u010dn\u011b vyu\u017e\u00edvaj\u00ed p\u0159ipojen\u00ed workera k clusteru Kafka. V\u00fdsledkem je, \u017ee ka\u017ed\u00fd gener\u00e1tor m\u00e1 100 producer\u016f, kte\u0159\u00ed v\u0161echny odes\u00edlaj\u00ed zpr\u00e1vy do clusteru Kafka.<\/li>\n<\/ul>\n<p><\/p>\n<h3>Sledov\u00e1n\u00ed stavu clusteru<\/h3>\n<p>\nB\u011bhem z\u00e1t\u011b\u017eov\u00e9ho testov\u00e1n\u00ed clusteru Kafka jsme tak\u00e9 sledovali jeho zdrav\u00ed, abychom se ujistili, \u017ee nedoch\u00e1z\u00ed k restart\u016fm pod\u016f, nesynchronizovan\u00fdm replik\u00e1m a maxim\u00e1ln\u00ed propustnosti s minim\u00e1ln\u00edmi v\u00fdkyvy:<\/p>\n<ul>\n<li> Gener\u00e1tor z\u00e1t\u011b\u017ee zapisuje standardn\u00ed statistiky o po\u010dtu publikovan\u00fdch zpr\u00e1v a \u00farovni chyb. Procento chyb by m\u011blo z\u016fstat na hodnot\u011b <code>0,00%<\/code>.<\/li>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/linkedin\/cruise-control\">Cruise Control<\/a><\/noindex>, roz\u0161\u00ed\u0159en\u00fd kafka-operator, poskytuje monitorovac\u00ed panel, na kter\u00e9m m\u016f\u017eeme tak\u00e9 sledovat stav klastra. Pro zobrazen\u00ed tohoto panelu prove\u010fte:\n<pre><code class=\"bash\">supertubes cluster cruisecontrol show -n kafka --kubeconfig<\/code><\/pre>\n<\/li>\n<li> \u00darove\u0148 ISR <i>(po\u010det replik \u201ein-sync\u201c)<\/i> shrink a expansion je 0.<\/li>\n<\/ul>\n<p><\/p>\n<h2>V\u00fdsledky m\u011b\u0159en\u00ed<\/h2>\n<p><\/p>\n<h3>3 brokery, velikost zpr\u00e1vy \u2014 512 byt\u016f<\/h3>\n<p>\nS partitiony rovnom\u011brn\u011b rozlo\u017een\u00fdmi na t\u0159ech brokerech jsme dok\u00e1zali dos\u00e1hnout v\u00fdkonu <i>~500 Mb\/s (p\u0159ibli\u017en\u011b 990 tis\u00edc zpr\u00e1v za sekundu)<\/i>:<\/p>\n<p><img decoding=\"async\" alt=\"Ur\u010den\u00ed vhodn\u00e9 velikosti pro Kafka cluster v Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/df65ff6a1a26e28d5e5a7a5ab1a20d98.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"Ur\u010den\u00ed vhodn\u00e9 velikosti pro Kafka cluster v Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/15c5883c0056c651f0a0bd14e9991967.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"Ur\u010den\u00ed vhodn\u00e9 velikosti pro Kafka cluster v Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/9b708fa5ce809b0d059cba9f5edfbb2f.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nSpot\u0159eba pam\u011bti virtu\u00e1ln\u00edm strojem JVM nep\u0159ekro\u010dila 2 Gb:<\/p>\n<p><img decoding=\"async\" alt=\"Ur\u010den\u00ed vhodn\u00e9 velikosti pro Kafka cluster v Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/3320e2eea891055cc0935da492d18ee9.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"Ur\u010den\u00ed vhodn\u00e9 velikosti pro Kafka cluster v Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/6e3976c310309aa8be45fa3fdf547750.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"Ur\u010den\u00ed vhodn\u00e9 velikosti pro Kafka cluster v Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/94baa1b689ae854693fb7cc4ed6e105e.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nPropustnost disku dos\u00e1hla maxim\u00e1ln\u00ed propustnosti I\/O uzlu na v\u0161ech t\u0159ech instanc\u00edch, na kter\u00fdch pracovaly brokery:<\/p>\n<p><img decoding=\"async\" alt=\"Ur\u010den\u00ed vhodn\u00e9 velikosti pro Kafka cluster v Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/408139ad33cda1294755660df1c128a8.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"Ur\u010den\u00ed vhodn\u00e9 velikosti pro Kafka cluster v Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/391d2df0ddfe82f94abdb200370ff775.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"Ur\u010den\u00ed vhodn\u00e9 velikosti pro Kafka cluster v Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/e7e9fd021a2cdc71992ce672363a78ec.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nZ dat o vyu\u017eit\u00ed pam\u011bti uzly vypl\u00fdv\u00e1, \u017ee syst\u00e9mov\u00e9 vyrovn\u00e1v\u00e1n\u00ed a cachov\u00e1n\u00ed zabralo ~10-15 Gb:<\/p>\n<p><img decoding=\"async\" alt=\"Ur\u010den\u00ed vhodn\u00e9 velikosti pro Kafka cluster v Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/781887507e01a4903b86871961ff9260.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"Ur\u010den\u00ed vhodn\u00e9 velikosti pro Kafka cluster v Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/63f10177f01342039f04192ea93c8bca.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"Ur\u010den\u00ed vhodn\u00e9 velikosti pro Kafka cluster v Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/48b992cc73a968b3d3b1442f96562c4e.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<\/p>\n<h3>3 brokery, velikost zpr\u00e1vy \u2014 100 byt\u016f<\/h3>\n<p>\nSe sn\u00ed\u017een\u00edm velikosti zpr\u00e1v propustnost kles\u00e1 p\u0159ibli\u017en\u011b o 15-20%: projevuje se \u010das str\u00e1ven\u00fd zpracov\u00e1n\u00edm ka\u017ed\u00e9 zpr\u00e1vy. D\u00e1le se zat\u00ed\u017een\u00ed procesoru t\u00e9m\u011b\u0159 zdvojn\u00e1sobilo.<\/p>\n<p><img decoding=\"async\" alt=\"Ur\u010den\u00ed vhodn\u00e9 velikosti pro Kafka cluster v Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/2f7dc5a94da371cfe17c9e0fc0ec1941.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"Ur\u010den\u00ed vhodn\u00e9 velikosti pro Kafka cluster v Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/986160a3a4e099a0dbb8aeefaf6f1c2f.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"Ur\u010den\u00ed vhodn\u00e9 velikosti pro Kafka cluster v Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/c08244457bf4dbfadf52838d4ea2969e.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nProto\u017ee na uzlech broker\u016f st\u00e1le existuj\u00ed nevyu\u017eit\u00e1 j\u00e1dra, lze v\u00fdkon zv\u00fd\u0161it zm\u011bnou konfigurace Kafka. To je slo\u017eit\u00fd \u00fakol, proto je lep\u0161\u00ed pro zv\u00fd\u0161en\u00ed propustnosti pracovat s v\u011bt\u0161\u00edmi zpr\u00e1vami.<\/p>\n<h3>4 brokery, velikost zpr\u00e1vy \u2014 512 byt\u016f<\/h3>\n<p>\nV\u00fdkon klastra Kafka lze snadno zv\u00fd\u0161it jednodu\u0161e p\u0159id\u00e1n\u00edm nov\u00fdch broker\u016f a zachov\u00e1n\u00edm rovnov\u00e1hy partition\u016f (to zaji\u0161\u0165uje rovnom\u011brn\u00e9 rozlo\u017een\u00ed z\u00e1t\u011b\u017ee mezi brokery). V na\u0161em p\u0159\u00edpad\u011b po p\u0159id\u00e1n\u00ed brokera vzrostla propustnost klastra na <i>~580 Mb\/s (~1,1 milionu zpr\u00e1v za sekundu)<\/i>. R\u016fst byl men\u0161\u00ed, ne\u017e se o\u010dek\u00e1valo: p\u0159ev\u00e1\u017en\u011b to zp\u016fsobuje nerovnov\u00e1ha partition\u016f (ne v\u0161ichni broke\u0159i pracuj\u00ed na sv\u00fdch maxim\u00e1ln\u00edch mo\u017enostech).<\/p>\n<p><img decoding=\"async\" alt=\"Ur\u010den\u00ed vhodn\u00e9 velikosti pro Kafka cluster v Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/8c8d2e70739add3c3f88fe42acf98c46.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"Ur\u010den\u00ed vhodn\u00e9 velikosti pro Kafka cluster v Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/a6e4812270c04487de0ad39b996a13f8.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"Ur\u010den\u00ed vhodn\u00e9 velikosti pro Kafka cluster v Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/6162c04cfb87b9d9cd0f3147a4ae1647.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"Ur\u010den\u00ed vhodn\u00e9 velikosti pro Kafka cluster v Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/dae5de3b14be212c679f3627a3b77d7b.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nSpot\u0159eba pam\u011bti stroj\u016fm JVM z\u016fstala pod 2 GB:<\/p>\n<p><img decoding=\"async\" alt=\"Ur\u010den\u00ed vhodn\u00e9 velikosti pro Kafka cluster v Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/0bca07a817d7da3104b08de10979f049.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"Ur\u010den\u00ed vhodn\u00e9 velikosti pro Kafka cluster v Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/7e1e64f80ec140f13e171cfe3be930ce.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"Ur\u010den\u00ed vhodn\u00e9 velikosti pro Kafka cluster v Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/5e9700b1e54e45f9cd5fa018ec07dba1.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"Ur\u010den\u00ed vhodn\u00e9 velikosti pro Kafka cluster v Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/4ba0c4b59bfe72bcb0d2f22056c47d79.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nNa pr\u00e1ci broker\u016f s \u00falo\u017ei\u0161ti m\u011bla vliv nerovnov\u00e1ha partition\u016f:<\/p>\n<p><img decoding=\"async\" alt=\"Ur\u010den\u00ed vhodn\u00e9 velikosti pro Kafka cluster v Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/2b8ed3c21adf70c751d1daa6d4c41895.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"Ur\u010den\u00ed vhodn\u00e9 velikosti pro Kafka cluster v Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/77b980869b99903d9de5cd7d43d79291.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"Ur\u010den\u00ed vhodn\u00e9 velikosti pro Kafka cluster v Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/f2f9569a3a1ef0ce35e89b2e52e4a8b2.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"Ur\u010den\u00ed vhodn\u00e9 velikosti pro Kafka cluster v Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/191d6d97b32b2128c5cfd3914171de15.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<\/p>\n<h2>Z\u00e1v\u011bry<\/h2>\n<p>\nV\u00fd\u0161e uveden\u00fd iterativn\u00ed p\u0159\u00edstup lze roz\u0161\u00ed\u0159it o slo\u017eit\u011bj\u0161\u00ed sc\u00e9n\u00e1\u0159e, kter\u00e9 zahrnuj\u00ed stovky consumer\u016f, repartitioning, nasazov\u00e1n\u00ed aktualizac\u00ed, restartov\u00e1n\u00ed pod\u016f atd. To n\u00e1m umo\u017e\u0148uje posoudit hranice schopnost\u00ed klastra Kafka za r\u016fzn\u00fdch podm\u00ednek, identifikovat \u00fazk\u00e1 m\u00edsta v jeho provozu a naj\u00edt zp\u016fsoby, jak se s nimi vypo\u0159\u00e1dat.<\/p>\n<p>Vyvinuli jsme Supertubes pro rychl\u00e9 a snadn\u00e9 nasazen\u00ed klastra, jeho konfiguraci, p\u0159id\u00e1v\u00e1n\u00ed\/odeb\u00edr\u00e1n\u00ed broker\u016f a t\u00e9mat, reagov\u00e1n\u00ed na upozorn\u011bn\u00ed a zaji\u0161t\u011bn\u00ed spr\u00e1vn\u00e9ho fungov\u00e1n\u00ed Kafka v Kubernetes jako celku. Na\u0161\u00edm c\u00edlem je pomoci soust\u0159edit se na hlavn\u00ed \u00fakol (\"generovat\" a \"spot\u0159ebov\u00e1vat\" zpr\u00e1vy Kafka), zat\u00edmco ve\u0161kerou t\u011b\u017ekou pr\u00e1ci p\u0159evezmou Supertubes a kafka-operator.<\/p>\n<p>Pokud v\u00e1s zaj\u00edmaj\u00ed technologie a projekty Open Source Banzai Cloud, sledujte na\u0161i spole\u010dnost na <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/banzaicloud\">GitHub<\/a><\/noindex>, <noindex><a rel=\"nofollow\" href=\"https:\/\/www.linkedin.com\/company\/banzaicloud\">LinkedIn<\/a><\/noindex> nebo <noindex><a rel=\"nofollow\" href=\"https:\/\/twitter.com\/BanzaiCloud\">Twitter<\/a><\/noindex>.<\/p>\n<h2>P.S. od p\u0159ekladatele<\/h2>\n<p>\nP\u0159e\u010dt\u011bte si tak\u00e9 n\u00e1\u0161 blog:<\/p>\n<ul>\n<li> \u00ab<noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/480722\/\">Jedna historie s oper\u00e1torem Redis v K8s a mini p\u0159ehled utility pro anal\u00fdzu dat t\u00e9to datab\u00e1ze<\/a><\/noindex>\u00bb;<\/li>\n<li> \u00ab<noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/450662\/\">N\u00e1ro\u010dn\u00e1 migrace RabbitMQ do Kubernetes<\/a><\/noindex>\u00bb;<\/li>\n<li> \u00ab<noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/329224\/\">zetcd od CoreOS: Nahrazen\u00ed ZooKeeper\u2026 \u00falo\u017ei\u0161t\u011bm etcd<\/a><\/noindex>\u00bb.<\/li>\n<\/ul>\n<p>Zdroj: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/488920\/\">habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u041f\u0440\u0438\u043c. \u043f\u0435\u0440\u0435\u0432.: \u0412 \u044d\u0442\u043e\u0439 \u0441\u0442\u0430\u0442\u044c\u0435 \u043a\u043e\u043c\u043f\u0430\u043d\u0438\u044f Banzai Cloud \u0434\u0435\u043b\u0438\u0442\u0441\u044f \u043f\u0440\u0438\u043c\u0435\u0440\u043e\u043c \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u043d\u0438\u044f \u0435\u0451 \u0441\u043f\u0435\u0446\u0438\u0430\u043b\u044c\u043d\u044b\u0445 \u0443\u0442\u0438\u043b\u0438\u0442 \u0434\u043b\u044f \u043e\u0431\u043b\u0435\u0433\u0447\u0435\u043d\u0438\u044f \u044d\u043a\u0441\u043f\u043b\u0443\u0430\u0442\u0430\u0446\u0438\u0438 Kafka \u0432 \u0440\u0430\u043c\u043a\u0430\u0445 Kubernetes. \u041f\u0440\u0438\u0432\u043e\u0434\u0438\u043c\u044b\u0435 \u0438\u043d\u0441\u0442\u0440\u0443\u043a\u0446\u0438\u0438 \u0438\u043b\u043b\u044e\u0441\u0442\u0440\u0438\u0440\u0443\u044e\u0442, \u043a\u0430\u043a \u043c\u043e\u0436\u043d\u043e \u043e\u043f\u0440\u0435\u0434\u0435\u043b\u0438\u0442\u044c \u043e\u043f\u0442\u0438\u043c\u0430\u043b\u044c\u043d\u044b\u0439 \u0440\u0430\u0437\u043c\u0435\u0440 \u0438\u043d\u0444\u0440\u0430\u0441\u0442\u0440\u0443\u043a\u0442\u0443\u0440\u044b \u0438 \u043d\u0430\u0441\u0442\u0440\u043e\u0438\u0442\u044c \u0441\u0430\u043c\u0443 Kafka \u0434\u043b\u044f \u0434\u043e\u0441\u0442\u0438\u0436\u0435\u043d\u0438\u044f \u0442\u0440\u0435\u0431\u0443\u0435\u043c\u043e\u0439 \u043f\u0440\u043e\u043f\u0443\u0441\u043a\u043d\u043e\u0439 \u0441\u043f\u043e\u0441\u043e\u0431\u043d\u043e\u0441\u0442\u0438. Apache Kafka \u2014 \u0440\u0430\u0441\u043f\u0440\u0435\u0434\u0435\u043b\u0451\u043d\u043d\u0430\u044f \u0441\u0442\u0440\u0438\u043c\u0438\u043d\u0433\u043e\u0432\u0430\u044f \u043f\u043b\u0430\u0442\u0444\u043e\u0440\u043c\u0430 \u0434\u043b\u044f \u0441\u043e\u0437\u0434\u0430\u043d\u0438\u044f \u043d\u0430\u0434\u0451\u0436\u043d\u044b\u0445, \u043c\u0430\u0441\u0448\u0442\u0430\u0431\u0438\u0440\u0443\u0435\u043c\u044b\u0445 \u0438 \u0432\u044b\u0441\u043e\u043a\u043e\u043f\u0440\u043e\u0438\u0437\u0432\u043e\u0434\u0438\u0442\u0435\u043b\u044c\u043d\u044b\u0445 \u043f\u043e\u0442\u043e\u043a\u043e\u0432\u044b\u0445 \u0441\u0438\u0441\u0442\u0435\u043c \u0440\u0435\u0430\u043b\u044c\u043d\u043e\u0433\u043e \u0432\u0440\u0435\u043c\u0435\u043d\u0438. [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":71214,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-71213","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.3 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u041f\u0440\u0438\u043c. \u043f\u0435\u0440\u0435\u0432.: \u0412 \u044d\u0442\u043e\u0439 \u0441\u0442\u0430\u0442\u044c\u0435 \u043a\u043e\u043c\u043f\u0430\u043d\u0438\u044f Banzai Cloud \u0434\u0435\u043b\u0438\u0442\u0441\u044f \u043f\u0440\u0438\u043c\u0435\u0440\u043e\u043c \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u043d\u0438\u044f \u0435\u0451 \u0441\u043f\u0435\u0446\u0438\u0430\u043b\u044c\u043d\u044b\u0445 \u0443\u0442\u0438\u043b\u0438\u0442 \u0434\u043b\u044f \u043e\u0431\u043b\u0435\u0433\u0447\u0435\u043d\u0438\u044f \u044d\u043a\u0441\u043f\u043b\u0443\u0430\u0442\u0430\u0446\u0438\u0438 Kafka \u0432 \u0440\u0430\u043c\u043a\u0430\u0445 Kubernetes.\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/opredelyaem-podhodyashhij-razmer-dlya-klastera-kafka-v-kubernetes\" \/>\n\t\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.3\" \/>\n\t\t<meta property=\"og:locale\" content=\"cs_CZ\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47\u041e\u043f\u0440\u0435\u0434\u0435\u043b\u044f\u0435\u043c \u043f\u043e\u0434\u0445\u043e\u0434\u044f\u0449\u0438\u0439 \u0440\u0430\u0437\u043c\u0435\u0440 \u0434\u043b\u044f \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0430 Kafka \u0432 Kubernetes | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u041f\u0440\u0438\u043c. \u043f\u0435\u0440\u0435\u0432.: \u0412 \u044d\u0442\u043e\u0439 \u0441\u0442\u0430\u0442\u044c\u0435 \u043a\u043e\u043c\u043f\u0430\u043d\u0438\u044f Banzai Cloud \u0434\u0435\u043b\u0438\u0442\u0441\u044f \u043f\u0440\u0438\u043c\u0435\u0440\u043e\u043c \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u043d\u0438\u044f \u0435\u0451 \u0441\u043f\u0435\u0446\u0438\u0430\u043b\u044c\u043d\u044b\u0445 \u0443\u0442\u0438\u043b\u0438\u0442 \u0434\u043b\u044f \u043e\u0431\u043b\u0435\u0433\u0447\u0435\u043d\u0438\u044f \u044d\u043a\u0441\u043f\u043b\u0443\u0430\u0442\u0430\u0446\u0438\u0438 Kafka \u0432 \u0440\u0430\u043c\u043a\u0430\u0445 Kubernetes.\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/opredelyaem-podhodyashhij-razmer-dlya-klastera-kafka-v-kubernetes\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2020-02-24T13:09:17+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-03-03T13:14:29+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47 Ur\u010den\u00ed vhodn\u00e9 velikosti pro cluster Kafka v Kubernetes | ProHoster","description":"Pozn. p\u0159ekl.: V tomto \u010dl\u00e1nku Banzai Cloud sd\u00edl\u00ed p\u0159\u00edklad pou\u017eit\u00ed sv\u00fdch speci\u00e1ln\u00edch n\u00e1stroj\u016f pro usnadn\u011bn\u00ed provozu Kafka v r\u00e1mci Kubernetes.","canonical_url":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/opredelyaem-podhodyashhij-razmer-dlya-klastera-kafka-v-kubernetes","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"cs_CZ","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u041e\u043f\u0440\u0435\u0434\u0435\u043b\u044f\u0435\u043c \u043f\u043e\u0434\u0445\u043e\u0434\u044f\u0449\u0438\u0439 \u0440\u0430\u0437\u043c\u0435\u0440 \u0434\u043b\u044f \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0430 Kafka \u0432 Kubernetes | ProHoster","og:description":"\u041f\u0440\u0438\u043c. \u043f\u0435\u0440\u0435\u0432.: \u0412 \u044d\u0442\u043e\u0439 \u0441\u0442\u0430\u0442\u044c\u0435 \u043a\u043e\u043c\u043f\u0430\u043d\u0438\u044f Banzai Cloud \u0434\u0435\u043b\u0438\u0442\u0441\u044f \u043f\u0440\u0438\u043c\u0435\u0440\u043e\u043c \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u043d\u0438\u044f \u0435\u0451 \u0441\u043f\u0435\u0446\u0438\u0430\u043b\u044c\u043d\u044b\u0445 \u0443\u0442\u0438\u043b\u0438\u0442 \u0434\u043b\u044f \u043e\u0431\u043b\u0435\u0433\u0447\u0435\u043d\u0438\u044f \u044d\u043a\u0441\u043f\u043b\u0443\u0430\u0442\u0430\u0446\u0438\u0438 Kafka \u0432 \u0440\u0430\u043c\u043a\u0430\u0445 Kubernetes.","og:url":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/opredelyaem-podhodyashhij-razmer-dlya-klastera-kafka-v-kubernetes","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-02-24T13:09:17+00:00","article:modified_time":"2020-03-03T13:14:29+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"71213","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 19:06:22","updated":"2022-10-04 20:58:00","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts\/71213","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/comments?post=71213"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts\/71213\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/media\/71214"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/media?parent=71213"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/categories?post=71213"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/tags?post=71213"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}