{"id":71213,"date":"2020-02-24T16:09:17","date_gmt":"2020-02-24T13:09:17","guid":{"rendered":"https:\/\/prohoster.info\/blog\/opredelyaem-podhodyashhij-razmer-dlya-klastera-kafka-v-kubernetes"},"modified":"2020-03-03T16:14:29","modified_gmt":"2020-03-03T13:14:29","slug":"opredelyaem-podhodyashhij-razmer-dlya-klastera-kafka-v-kubernetes","status":"publish","type":"post","link":"https:\/\/prohoster.info\/et\/blog\/administrirovanie\/opredelyaem-podhodyashhij-razmer-dlya-klastera-kafka-v-kubernetes","title":{"rendered":"M\u00e4\u00e4rame sobiva suuruse Kafka klastri jaoks Kuberneteses.","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p><i><b>M\u00e4rk. t\u00f5lge.<\/b>: Selles artiklis jagab Banzai Cloud n\u00e4idet, kuidas kasutada oma spetsiaalseid utiliite, et lihtsustada Kafka haldamist Kuberneteses. Esitatud juhised illustreerivad, kuidas leida optimaalse infrastruktuuri suuruse ja seadistada Kafka n\u00f5utava l\u00e4bilaskev\u00f5ime saavutamiseks.<\/i><\/p>\n<p><img decoding=\"async\" alt=\"M\u00e4\u00e4rame sobiva suuruse Kafka klastri jaoks Kuberneteses.\" src=\"\/wp-content\/uploads\/2020\/02\/8a598ec7db091c2944a6442f9bc124da.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nApache Kafka on jaotatud voogedastuse platvorm, mis v\u00f5imaldab luua usaldusv\u00e4\u00e4rseid, skaleeritavaid ja k\u00f5rge j\u00f5udlusega reaalajas voogesituss\u00fcsteeme. Selle muljetavaldavaid v\u00f5imalusi saab laiendada Kubernetesega. Selleks oleme v\u00e4lja t\u00f6\u00f6tanud <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/banzaicloud\/kafka-operator\">Open Source-Kafka operaatori<\/a><\/noindex> ja t\u00f6\u00f6riista nimega <noindex><a rel=\"nofollow\" href=\"https:\/\/banzaicloud.com\/docs\/supertubes\/overview\/\">Supertubes<\/a><\/noindex>. Need v\u00f5imaldavad k\u00e4ivitada Kafka Kuberneteses ja kasutada selle erinevaid funktsioone, nagu h\u00e4\u00e4lestamine, automaatne skaleerimine, p\u00f5hivarade teadlikkus, \"pehme\" <i>(graceful)<\/i> uuenduste juurutamine jne.<noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><\/p>\n<blockquote><p>Proovige Supertubes oma klastris:<\/p>\n<pre><code class=\"bash\">curl https:\/\/getsupertubes.sh | sh ja supertubes install -a --no-democluster --kubeconfig<\/code><\/pre>\n<p>\nV\u00f5i v\u00f5tke \u00fchendust <noindex><a rel=\"nofollow\" href=\"https:\/\/banzaicloud.com\/docs\/supertubes\/cli\/reference\/\">dokumentatsioonis<\/a><\/noindex>. Samuti v\u00f5ib lugeda m\u00f5ningatest Kafka v\u00f5imalustest, mille t\u00f6\u00f6 on automatiseeritud Supertubes'i ja Kafka operaatori abil. Me oleme neist juba blogis kirjutanud:<\/p>\n<ul>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/banzaicloud.com\/blog\/kafka-operator\/\">Oi ei! J\u00e4lle \u00fcks Kafka operaator Kubernetes'ele<\/a><\/noindex>;<\/li>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/banzaicloud.com\/blog\/kafka-alert\/\">Monitoori ja halda Kafka't, tuginedes Prometheuse m\u00f5\u00f5tmistele<\/a><\/noindex>;<\/li>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/banzaicloud.com\/blog\/kafka-rack-awareness\/\">Kafka rack awareness Kubernetes'es<\/a><\/noindex>;<\/li>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/banzaicloud.com\/blog\/kafka-on-istio-performance\/\">Apache Kafka p\u00f5hjal Istio \u2014 v\u00f5rdlus<\/a><\/noindex>;<\/li>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/banzaicloud.com\/blog\/kafka-topic-user-management\/\">Kasutajate autentimine ja juurdep\u00e4\u00e4sukontrolliga klastri haldamine Kafka operaatori kaudu<\/a><\/noindex>;<\/li>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/banzaicloud.com\/blog\/kafka-rolling-upgrade\/\">Kafka sujuv uuendamine ja d\u00fcnaamiline konfiguratsioon Kubernetes'es<\/a><\/noindex>;<\/li>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/banzaicloud.com\/blog\/kafka-envoy-protocol-filter\/\">Envoy protokollifilter Kafka jaoks, mesh'itud<\/a><\/noindex>.<\/li>\n<\/ul>\n<\/blockquote>\n<p>\nKui otsustate k\u00e4itada Kafka klastrit Kubernetes'es, seisate t\u00f5en\u00e4oliselt silmitsi k\u00fcsimusega, kuidas m\u00e4\u00e4rata baas infrastruktuuri optimaalne suurus ja kohandada Kafka konfiguratsiooni vastavalt l\u00e4bilaskev\u00f5ime n\u00f5udmistele. Iga brokera maksimaalne j\u00f5udlus s\u00f5ltub selle aluseks olevate infrastruktuuri komponentide, nagu m\u00e4lu, protsessor, ketta kiirus, v\u00f5rgu l\u00e4bilaskev\u00f5ime jne, j\u00f5udlusest.<\/p>\n<p>Ideaalis peaks maakleri konfiguratsioon olema selline, et k\u00f5ik infrastruktuuri elemendid t\u00f6\u00f6tavad oma maksimaalses v\u00f5imekuses. Kuid reaalses elus on sellise seadistuse saavutamine \u00fcsna keeruline. T\u00f5en\u00e4olisem on, et kasutajad seadistavad maakleri konfiguratsiooni nii, et maksimeerivad \u00fche v\u00f5i kahe komponendi (ketta, m\u00e4lu v\u00f5i protsessori) kasutamist. \u00dcldiselt n\u00e4itab maakler maksimaalset j\u00f5udlust, kui tema konfiguratsioon v\u00f5imaldab k\u00f5ige aeglasemal komponendil t\u00f6\u00f6tada t\u00e4ielikult. Nii saame umbkaudse ettekujutuse koormusest, millega \u00fcks maakler toime tulla suudab.<\/p>\n<p>Teoreetiliselt saame samuti hinnata nende maaklerite arvu, mis on vajalik antud koormuse talumiseks. Kuid praktikas on erinevate tasemete seadistamiseks nii palju variante, et mingi konfiguratsiooni potentsiaalset j\u00f5udlust on v\u00e4ga keeruline (kui mitte v\u00f5imatu) hinnata. Teisis\u00f5nu, on v\u00e4ga keeruline planeerida konfiguratsiooni, tuginedes mingile kindlale j\u00f5udlusele.<\/p>\n<p>Supertubes kasutajatele rakendame tavaliselt j\u00e4rgmist l\u00e4henemist: alustame teatud konfiguratsioonist (infrastruktuur + seaded), seej\u00e4rel m\u00f5\u00f5dame selle j\u00f5udlust, kohandame maakleri seadeid ja korrame protsessi veel kord. See kestab seni, kuni k\u00f5ige aeglasema infrastruktuuri komponendi potentsiaal on t\u00e4ielikult \u00e4ra kasutatud.<\/p>\n<p>Nii saame selgema \u00fclevaate sellest, kui palju maaklereid on klastril vaja, et teatud koormusega toime tulla (maaklerite arv s\u00f5ltub ka teistest teguritest, nagu minimaalne s\u00f5numite replikate arv stabiilsuse tagamiseks, partition-juhtide arv jne). Lisaks saame aimu, millist infrastruktuuri komponenti on soovitatav vertikaalselt skaleerida.<\/p>\n<p>Selles artiklis r\u00e4\u00e4gitakse sammudest, mida astume, et 'v\u00e4lja pigistada k\u00f5ik' k\u00f5ige aeglasematest komponentidest algkonfiguratsioonides ja m\u00f5\u00f5ta Kafka klastrite l\u00e4bilaskev\u00f5imet. K\u00f5rge usaldusv\u00e4\u00e4rsusega konfiguratsioon vajab v\u00e4hemalt kolme t\u00f6\u00f6tavat maaklerit (<code>min.insync.replicas=3<\/code>), kolm erineva k\u00e4ttesaadavuspiirkonna vahel. Kubernetes'i infrastruktuuri konfigureerimiseks, skaleerimiseks ja j\u00e4lgimiseks kasutame meie enda konteinerihaldusplatvormi h\u00fcbriidpilvede jaoks \u2014 <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/banzaicloud\/pipeline\">Pipeline<\/a><\/noindex>. See toetab on-premise (bare metal, VMware) ja viit erinevat pilve (Alibaba, AWS, Azure, Google, Oracle), samuti nende mis tahes kombinatsioone.<\/p>\n<h2>M\u00f5tted infrastruktuuri ja Kafka klastrite konfiguratsiooni kohta<\/h2>\n<p>\nAllpool toodud n\u00e4idete jaoks oleme valinud AWS pilveteenuse pakkujaks ja EKS-i Kubernetes'i jaotuseks. Sarnast konfiguratsiooni saab rakendada ka <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/banzaicloud\/pke\">PKE<\/a><\/noindex> \u2014 Banzai Cloud'i Kubernetes'i jaotus, mis on CNCF'i sertifitseeritud.<\/p>\n<h3>Kett<\/h3>\n<p>\nAmazon pakub erinevaid <noindex><a rel=\"nofollow\" href=\"https:\/\/docs.aws.amazon.com\/AWSEC2\/latest\/UserGuide\/ebs-volume-types.html\">EBS-i m\u00e4lut\u00fc\u00fcpe<\/a><\/noindex>. Aluseks on <i>gp2<\/i> ja <i>io1<\/i> SSD-diskid, kuid k\u00f5rge l\u00e4bilaskev\u00f5ime tagamiseks <i>gp2<\/i> kasutab kogutud krediite <i>(I\/O krediidid)<\/i>, seet\u00f5ttu eelistasime t\u00fc\u00fcpi <i>io1<\/i>, mis pakub stabiilset k\u00f5rget l\u00e4bilaskev\u00f5imet.<\/p>\n<h3>Instantsit\u00fc\u00fcbid<\/h3>\n<p>\nKafka j\u00f5udlus s\u00f5ltub tugevalt operatsioonis\u00fcsteemi lehek\u00fcljekattest, seega vajame instantsse, millel on piisavalt m\u00e4lu brokera (JVM) ja lehek\u00fcljekatte jaoks. Instants <i>c5.2xlarge<\/i> \u2014 hea algus, kuna sellel on 16 GB RAM-i ja <noindex><a rel=\"nofollow\" href=\"https:\/\/docs.aws.amazon.com\/AWSEC2\/latest\/UserGuide\/ebs-optimized.html\">see on optimeeritud EBS-i t\u00f6\u00f6tamiseks<\/a><\/noindex>. Selle miinus on see, et see suudab tagada maksimaalset j\u00f5udlust mitte rohkem kui 30 minutit iga 24 tunni jooksul. Kui t\u00f6\u00f6koormus n\u00f5uab maksimaalset j\u00f5udlust pikema aja jooksul, tasub kaaluda teisi t\u00fc\u00fcpe instantsse. Just nii me tegime, valides <i>c5.4xlarge<\/i>. See tagab maksimaalse l\u00e4bilaskev\u00f5ime <b>593,75 MB\/s<\/b>. EBS-i mahuti maksimaalne l\u00e4bilaskev\u00f5ime <i>io1<\/i> on suurem kui selle instantsi <i>c5.4xlarge<\/i>, seega tundub, et infrastruktuuri aeglasem komponent on selle t\u00fc\u00fcbi instantsi I\/O l\u00e4bilaskev\u00f5ime (mida peaksid kinnitama ka meie koormustestide tulemused).<\/p>\n<h3>V\u00f5rk<\/h3>\n<p>\nV\u00f5rgu l\u00e4bilaskev\u00f5ime peaks olema piisavalt suur v\u00f5rreldes VM-i instantsi ja ketta j\u00f5udlusega, vastasel juhul muutub v\u00f5rk kitsaskohaks. Meie juhul toetab v\u00f5rgu liides <i>c5.4xlarge<\/i> kiirus kuni 10 Gb\/s, mis on oluliselt k\u00f5rgem VM-i instantsi I\/O l\u00e4bilaskev\u00f5imest.<\/p>\n<h3>Brokereid juurutamine<\/h3>\n<p>\nBrokereid tuleb k\u00e4ivitada (planeerida Kuberneteses) p\u00fchendatud s\u00f5lmedes, et v\u00e4ltida konkurentsi teiste protsesside vahel CPU, m\u00e4lu, v\u00f5rgu ja kettaressursside p\u00e4rast.<\/p>\n<h3>Java versioon<\/h3>\n<p>\nLoogiline valik on Java 11, kuna see on Dockeriga \u00fchilduv, tagades, et JVM m\u00e4\u00e4rab \u00f5igesti protsessorid ja m\u00e4lu, mis on saadaval konteineris, kus broker t\u00f6\u00f6tab. Teades, et protsessorite piirangud on olulised, m\u00e4\u00e4rab JVM sisemiselt ja l\u00e4bipaistvalt GC ja JIT-kompilaatori l\u00f5ime arvu. Kasutasime Kafka pilti <code>banzaicloud\/kafka:2.13-2.4.0<\/code>, mis sisaldab Kafka versiooni 2.4.0 (Scala 2.13) Java 11 peal.<\/p>\n<blockquote><p>Kui soovite Java\/JVM kohta Kuberneteses rohkem teada saada, vaadake meie j\u00e4rgmisi publikatsioone:<\/p>\n<ul>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/banzaicloud.com\/blog\/java-resource-limits\/\">Miks minu Java rakendus on OOMKilled<\/a><\/noindex>;<\/li>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/banzaicloud.com\/blog\/java10-container-sizing\/\">Kuidas \u00f5igesti suurendada Java 10 rakenduste konteinereid<\/a><\/noindex>.<\/li>\n<\/ul>\n<\/blockquote>\n<p><\/p>\n<h3>Brok\u0435\u0440\u0438 m\u00e4lu seaded<\/h3>\n<p>\nBrokkeri m\u00e4luhalduses on kaks peamist aspekti: JVM-i seadistused ja Kubernetes'e pod'i seadistused. Pod'i jaoks m\u00e4\u00e4ratud m\u00e4lulimiit peab olema suurem kui maksimaalne heap size, et JVM-il j\u00e4\u00e4ks ruumi Java metaruumi jaoks, mis asub oma m\u00e4lus, ja operatsioonis\u00fcsteemi lehevahe m\u00e4luks, mida Kafka aktiivselt kasutab. Testides oleme k\u00e4ivitanud Kafka brokkereid j\u00e4rgmiste seadistustega <code>-Xmx4G -Xms2G<\/code>, samas kui pod'i m\u00e4lulimiit oli <code>10 Gi<\/code>. Pidage meeles, et JVM-i m\u00e4lu seadistusi saab automaatselt hankida, kasutades <code>-XX:MaxRAMPercentage<\/code> ja <code>-X:MinRAMPercentage<\/code>, tuginedes pod'i m\u00e4lulimiidile.<\/p>\n<h3>Brokri protsessoriseaded<\/h3>\n<p>\n\u00dcldiselt saab t\u00f5husust suurendada, suurendades paralleelsust, suurendades Kafka kasutatavate l\u00f5ime arvu. Mida rohkem protsessoreid Kafka jaoks saadaval on, seda parem. Meie testis alustasime 6 protsessori piiranguga ja suurendasime nende arvu j\u00e4rk-j\u00e4rgult (iteratsioonide kaupa) kuni 15. Samuti seadsime <code>num.network.threads=12<\/code> brokerite seadetes, et suurendada voogude arvu, mis v\u00f5tab andmeid v\u00f5rgust ja saadab need edasi. Kohe p\u00e4rast avastamist, et j\u00e4relevalve-brokerid ei suuda piisavalt kiiresti replikate saada, t\u00f5steti <code>num.replica.fetchers<\/code> kuni 4, et suurendada kiirust, millega j\u00e4relevalve-brokerid replitseerivad s\u00f5numeid liidritelt.<\/p>\n<h3>Koormuse genereerimise t\u00f6\u00f6riist<\/h3>\n<p>\nTuleb tagada, et valitud koormuse generaatori potentsiaal ei l\u00f5pe enne, kui Kafka klaster (mida testitakse) saavutab oma maksimaalse koormuse. Teisis\u00f5nu, tuleb eelnevalt hinnata koormuse genereerimise t\u00f6\u00f6riista v\u00f5imeid ning valida selle jaoks piisava arvu protsessorite ja m\u00e4lu t\u00fc\u00fcbiga instantsid. Sel juhul toob meie t\u00f6\u00f6riist genereerida rohkem koormust, kui Kafka klaster suudab taluda. P\u00e4rast mitmeid katsetusi otsustasime kolme instantsi kasuks <i>c5.4xlarge<\/i>milles igas oli k\u00e4ivitatud generaator.<\/p>\n<h2>Benkmarkimine<\/h2>\n<p>\nSuuruse m\u00f5\u00f5tmine on iteratiivne protsess, mis h\u00f5lmab j\u00e4rgmisi etappe:<\/p>\n<ul>\n<li> infrastruktuuri seadistamine (EKS klaster, Kafka klaster, koormuse genereerimise t\u00f6\u00f6riist, samuti Prometheus ja Grafana);<\/li>\n<li> koormuse genereerimine kindlaksm\u00e4\u00e4ratud ajavahemiku jooksul, et filtreerida juhuslikke k\u00f5rvalekaldeid kogutud j\u00f5udlusn\u00e4itajates;<\/li>\n<li> infrastruktuuri ja vahendite konfigureerimise kohandamine vaatatud j\u00f5udlusn\u00e4itajate p\u00f5hjal;<\/li>\n<li> protsessi kordamine seni, kuni saavutatakse n\u00f5utav Kafka klastrite l\u00e4bivoolu tase. See peab olema stabiilselt reprodutseeritav ja n\u00e4itama minimaalset variatsiooni l\u00e4bivoolust.<\/li>\n<\/ul>\n<p>\nJ\u00e4rgmises jaos kirjeldatakse samme, mida tehti testklastri b\u00e4nkm\u00e4rkimise protsessi k\u00e4igus.<\/p>\n<h3>T\u00f6\u00f6riistad<\/h3>\n<p>\nKiireks p\u00f5hikonfiguratsiooni seadistamiseks, koormuse genereerimiseks ja j\u00f5udluse m\u00f5\u00f5tmiseks kasutati j\u00e4rgmisi t\u00f6\u00f6riistu:<\/p>\n<ul>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/banzaicloud.com\/docs\/pipeline\/overview\/\">Banzai Cloud Pipeline<\/a><\/noindex> EKS klastri korraldamiseks Amazonilt c <noindex><a rel=\"nofollow\" href=\"https:\/\/prometheus.io\/\">Prometheus<\/a><\/noindex> (Kafka ja infrastruktuuri meetrikate kogumiseks) ja <noindex><a rel=\"nofollow\" href=\"https:\/\/grafana.com\/\">Grafana<\/a><\/noindex> (selle meetrikate visualiseerimiseks). Kasutasime <b>integreeritud<\/b> \u00fches <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/banzaicloud\/pipeline\">Pipeline<\/a><\/noindex> teenuseid, mis pakuvad f\u00f6deraalselt j\u00e4lgimist, keskset logide kogumist, haavatavuste skaneerimist, t\u00f5rke taastamist, ettev\u00f5tte tasemel turvalisust ja palju muud.<\/li>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/jamiealquiza\/sangrenel\">Sangrenel<\/a><\/noindex> \u2014 t\u00f6\u00f6riist Kafka klasteri koormustestimiseks.<\/li>\n<li> Grafana paneelid Kafka ja infrastruktuuri metrikate visualiseerimiseks: <noindex><a rel=\"nofollow\" href=\"https:\/\/grafana.com\/grafana\/dashboards\/10123\">Kubernetes Kafka<\/a><\/noindex>, <noindex><a rel=\"nofollow\" href=\"https:\/\/grafana.com\/grafana\/dashboards\/1860\">Node Exporter<\/a><\/noindex>.<\/li>\n<li> Supertubes CLI maksimaalselt lihtsaks Kafka klasteri seadistamiseks Kuberneteses. Zookeeper, Kafka operator, Envoy ja palju muid komponente on installitud ja \u00f5igesti konfigureeritud, et k\u00e4ivitada production-ready Kafka klaster Kuberneteses.\n<ul>\n<li> Paigaldamiseks <i>supertubes CLI<\/i> kasutage juhiseid, mis on toodud <noindex><a rel=\"nofollow\" href=\"https:\/\/banzaicloud.com\/docs\/supertubes\/cli\/install\/\">siit<\/a><\/noindex>.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p>\n<img decoding=\"async\" alt=\"M\u00e4\u00e4rame sobiva suuruse Kafka klastri jaoks Kuberneteses.\" src=\"\/wp-content\/uploads\/2020\/02\/23ecf423b1ea66813406943ae7eabf61.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<\/p>\n<h3>EKS klaster<\/h3>\n<p>\nValmistage EKS klaster, kus on eraldatud t\u00f6\u00f6v\u00f5lurid <i>c5.4xlarge<\/i> erinevates k\u00e4tte saadavustes Kafka brokkeritega pod'ide jaoks, samuti eraldatud s\u00f5lmed koormuse generaatori ja j\u00e4lgimisinfote jaoks.<\/p>\n<pre><code class=\"bash\">banzai cluster create -f https:\/\/raw.githubusercontent.com\/banzaicloud\/kafka-operator\/master\/docs\/benchmarks\/infrastructure\/cluster_eks_202001.json<\/code><\/pre>\n<p>\nKui EKS klaster on \u00fcles pandud, aktiveerige selle sisseehitatud <noindex><a rel=\"nofollow\" href=\"https:\/\/banzaicloud.com\/docs\/pipeline\/features\/integrated-services\/\">j\u00e4lgimisteenus<\/a><\/noindex> \u2014 see paigaldab Prometheuse ja Grafana klastrisse.<\/p>\n<h3>Kafka s\u00fcsteemikomponendid<\/h3>\n<p>\nInstallige Kafka s\u00fcsteemikomponendid (Zookeeper, kafka-operator) EKS-is supertubes CLI abil:<\/p>\n<pre><code class=\"bash\">supertubes install -a --no-democluster --kubeconfig<\/code><\/pre>\n<p><\/p>\n<h3>Kafka klaster<\/h3>\n<p>\nEKS kasutab vaikimisi EBS t\u00fc\u00fcpi mahuteid <i>gp2<\/i>, seet\u00f5ttu tuleb luua eraldi salvestusklass mahtude p\u00f5hjal <i>io1<\/i> Kafka klastri jaoks:<\/p>\n<pre><code class=\"plaintext\">kubectl create -f - &lt;&lt;EOF\napiVersion: storage.k8s.io\/v1\nkind: StorageClass\nmetadata:\n  name: fast-ssd\nprovisioner: kubernetes.io\/aws-ebs\nparameters:\n  type: io1\n  iopsPerGB: &quot;50&quot;\n  fsType: ext4\nvolumeBindingMode: WaitForFirstConsumer\nEOF<\/code><\/pre>\n<p>\nSeadke brokerite jaoks parameeter <code>min.insync.replicas=3<\/code> ja juurutage brokkereid pod'ides kolme erineva k\u00e4tte saadavuse s\u00f5lmedes:<\/p>\n<pre><code class=\"bash\">supertubes cluster create -n kafka --kubeconfig  -f https:\/\/raw.githubusercontent.com\/banzaicloud\/kafka-operator\/master\/docs\/benchmarks\/infrastructure\/kafka_202001_3brokers.yaml --wait --timeout 600<\/code><\/pre>\n<p><\/p>\n<h3>Teemad<\/h3>\n<p>\nK\u00e4itasime samaaegselt kolme koormusgeneraatori eksemplari. Iga\u00fcks neist kirjutab oma teemasse, st meil on kokku vaja kolme teemat:<\/p>\n<pre><code class=\"bash\">supertubes cluster topic create -n kafka --kubeconfig  -f -&lt;&lt;EOF\napiVersion: kafka.banzaicloud.io\/v1alpha1\nkind: KafkaTopic\nmetadata:\n  name: perftest1\nspec:\n  name: perftest1\n  partitions: 12\n  replicationFactor: 3\n  retention.ms: &#039;28800000&#039;\n  cleanup.policy: delete\nEOF\n\nsupertubes cluster topic create -n kafka --kubeconfig  -f -&lt;&lt;EOF\napiVersion: kafka.banzaicloud.io\/v1alpha1\nkind: KafkaTopic\nmetadata:\n    name: perftest2\nspec:\n  name: perftest2\n  partitions: 12\n  replicationFactor: 3\n  retention.ms: &#039;28800000&#039;\n  cleanup.policy: delete\nEOF\n\nsupertubes cluster topic create -n kafka --kubeconfig  -f -&lt;&lt;EOF\napiVersion: kafka.banzaicloud.io\/v1alpha1\nkind: KafkaTopic\nmetadata:\n  name: perftest3\nspec:\n  name: perftest3\n  partitions: 12\n  replicationFactor: 3\n  retention.ms: &#039;28800000&#039;\n  cleanup.policy: delete\nEOF<\/code><\/pre>\n<p>\nIga teema replikatsioonifaktor on 3 \u2014 minimaalne soovitatav v\u00e4\u00e4rtus k\u00f5rge k\u00e4ttesaadavuse tootmiss\u00fcsteemide jaoks.<\/p>\n<h3>Koormuse genereerimise t\u00f6\u00f6riist<\/h3>\n<p>\nK\u00e4ivitasime kolm koormuse generaatori instantsi (iga\u00fcks kirjutas eraldi teemas). Koormuse generaatori pod'ide jaoks on vajalik node affinity m\u00e4\u00e4ramine, et need planeeritaks ainult neile etten\u00e4htud s\u00f5lmedele:<\/p>\n<pre><code class=\"plaintext\">apiVersion: extensions\/v1beta1\nkind: Deployment\nmetadata:\n  labels:\n    app: loadtest\n  name: perf-load1\n  namespace: kafka\nspec:\n  progressDeadlineSeconds: 600\n  replicas: 1\n  revisionHistoryLimit: 10\n  selector:\n    matchLabels:\n      app: loadtest\n  strategy:\n    rollingUpdate:\n      maxSurge: 25%\n      maxUnavailable: 25%\n    type: RollingUpdate\n  template:\n    metadata:\n      creationTimestamp: null\n      labels:\n        app: loadtest\n    spec:\n      affinity:\n        nodeAffinity:\n          requiredDuringSchedulingIgnoredDuringExecution:\n            nodeSelectorTerms:\n            - matchExpressions:\n              - key: nodepool.banzaicloud.io\/name\n                operator: In\n                values:\n                - loadgen\n      containers:\n      - args:\n        - -brokers=kafka-0:29092,kafka-1:29092,kafka-2:29092,kafka-3:29092\n        - -topic=perftest1\n        - -required-acks=all\n        - -message-size=512\n        - -workers=20\n        image: banzaicloud\/perfload:0.1.0-blog\n        imagePullPolicy: Always\n        name: sangrenel\n        resources:\n          limits:\n            cpu: 2\n            memory: 1Gi\n          requests:\n            cpu: 2\n            memory: 1Gi\n        terminationMessagePath: \/dev\/termination-log\n        terminationMessagePolicy: File\n      dnsPolicy: ClusterFirst\n      restartPolicy: Always\n      schedulerName: default-scheduler\n      securityContext: {}\n      terminationGracePeriodSeconds: 30<\/code><\/pre>\n<p>\nM\u00f5ned punktid, millele t\u00e4helepanu p\u00f6\u00f6rata:<\/p>\n<ul>\n<li> Koormuse genereerija genereerib 512-baidiseid s\u00f5numeid ja avaldab need Kafka-s 500-s\u00f5numiliste gruppidena.<\/li>\n<li> Argumenti kasutades <code>-required-acks=all<\/code> publikatsioon loetakse edukaks, kui k\u00f5ik s\u00fcnkroniseeritud s\u00f5numi koopiad on saadud ja Kafka brokerid on need kinnitanud. See t\u00e4hendab, et meie m\u00f5\u00f5tmises hindasime mitte ainult liidri kiirusest, kes s\u00f5numeid vastu v\u00f5tab, vaid ka nende j\u00e4rgijate kiirusest, kes s\u00f5numeid replitseerivad. Selle testi eesm\u00e4rk ei ole hinnata tarbijate (consumers) kiiruset, kes loevad uusimaid s\u00f5numeid, mis j\u00e4\u00e4vad veel operatsioonis\u00fcsteemi lehekausta ja v\u00f5rrelda seda s\u00f5numite lugemise kiirusest, mis on salvestatud kettale. <i>(consumers)<\/i> Hiljuti vastuv\u00f5etud s\u00f5numeid, mis j\u00e4\u00e4vad veel operatsioonis\u00fcsteemi lehekausta, ja selle v\u00f5rreldavust s\u00f5numite lugemise kiirusest, mis on salvestatud kettale.<\/li>\n<li> Koormuse generaator k\u00e4ivitab paralleelselt 20 t\u00f6\u00f6tajat (<code>-workers=20<\/code>). Iga t\u00f6\u00f6taja sisaldab 5 produtsenti, kes jagavad \u00fchendust t\u00f6\u00f6tajaga Kafka klastrisse. Kokku on igas koormuse generaatoris 100 produtsenti, kes k\u00f5ik saadavad s\u00f5numeid Kafka klastrisse.<\/li>\n<\/ul>\n<p><\/p>\n<h3>Klastri oleku j\u00e4lgimine<\/h3>\n<p>\nKoormustestimise k\u00e4igus j\u00e4lgisime ka Kafka klastrit, et veenduda, et pod'id ei taask\u00e4ivitu, replikad ei ole des\u00fcnkroniseeritud ning maksimaalne l\u00e4bilaskev\u00f5ime on minimaalsete k\u00f5ikumistega:<\/p>\n<ul>\n<li> Koormuse generaator annab standardstatistikat avaldatud s\u00f5numite arvu ja vigade taseme kohta. Vigade protsent peaks j\u00e4\u00e4ma v\u00e4\u00e4rtusele <code>0,00%<\/code>.<\/li>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/linkedin\/cruise-control\">Cruise Control<\/a><\/noindex>, mille on k\u00e4ivitanud kafka-operator, pakub j\u00e4lgimispaneeli, kus saame j\u00e4lgida klastrit. Selle paneeli vaatamiseks k\u00e4ivitage:\n<pre><code class=\"bash\">supertubes cluster cruisecontrol show -n kafka --kubeconfig<\/code><\/pre>\n<\/li>\n<li> ISR tase <i>(in-sync replikate arv)<\/i> shrink ja expansion on 0.<\/li>\n<\/ul>\n<p><\/p>\n<h2>M\u00f5\u00f5tmis tulemused<\/h2>\n<p><\/p>\n<h3>3 brokerit, s\u00f5numite suurus \u2014 512 baiti<\/h3>\n<p>\n3 brokkerile \u00fchtlaselt jaotatud partition'idega \u00f5nnestus meil saavutada l\u00e4bilaskev\u00f5ime <i>~500 MB\/s (umbes 990 tuhat s\u00f5numit sekundis)<\/i>:<\/p>\n<p><img decoding=\"async\" alt=\"M\u00e4\u00e4rame sobiva suuruse Kafka klastri jaoks Kuberneteses.\" src=\"\/wp-content\/uploads\/2020\/02\/df65ff6a1a26e28d5e5a7a5ab1a20d98.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"M\u00e4\u00e4rame sobiva suuruse Kafka klastri jaoks Kuberneteses.\" src=\"\/wp-content\/uploads\/2020\/02\/15c5883c0056c651f0a0bd14e9991967.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"M\u00e4\u00e4rame sobiva suuruse Kafka klastri jaoks Kuberneteses.\" src=\"\/wp-content\/uploads\/2020\/02\/9b708fa5ce809b0d059cba9f5edfbb2f.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nJVM-i virtuaalmasina m\u00e4lu kasutamine ei \u00fcletanud 2 GB:<\/p>\n<p><img decoding=\"async\" alt=\"M\u00e4\u00e4rame sobiva suuruse Kafka klastri jaoks Kuberneteses.\" src=\"\/wp-content\/uploads\/2020\/02\/3320e2eea891055cc0935da492d18ee9.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"M\u00e4\u00e4rame sobiva suuruse Kafka klastri jaoks Kuberneteses.\" src=\"\/wp-content\/uploads\/2020\/02\/6e3976c310309aa8be45fa3fdf547750.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"M\u00e4\u00e4rame sobiva suuruse Kafka klastri jaoks Kuberneteses.\" src=\"\/wp-content\/uploads\/2020\/02\/94baa1b689ae854693fb7cc4ed6e105e.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nKetta l\u00e4bilaskev\u00f5ime saavutas igal kolmel instantsil maksimaalse I\/O l\u00e4bilaskev\u00f5ime, kus brookerid t\u00f6\u00f6tasid:<\/p>\n<p><img decoding=\"async\" alt=\"M\u00e4\u00e4rame sobiva suuruse Kafka klastri jaoks Kuberneteses.\" src=\"\/wp-content\/uploads\/2020\/02\/408139ad33cda1294755660df1c128a8.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"M\u00e4\u00e4rame sobiva suuruse Kafka klastri jaoks Kuberneteses.\" src=\"\/wp-content\/uploads\/2020\/02\/391d2df0ddfe82f94abdb200370ff775.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"M\u00e4\u00e4rame sobiva suuruse Kafka klastri jaoks Kuberneteses.\" src=\"\/wp-content\/uploads\/2020\/02\/e7e9fd021a2cdc71992ce672363a78ec.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nS\u00f5lmede m\u00e4lukasutuse andmetest selgub, et s\u00fcsteemne puhversalvestus ja vahem\u00e4lu olid ~10-15 GB:<\/p>\n<p><img decoding=\"async\" alt=\"M\u00e4\u00e4rame sobiva suuruse Kafka klastri jaoks Kuberneteses.\" src=\"\/wp-content\/uploads\/2020\/02\/781887507e01a4903b86871961ff9260.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"M\u00e4\u00e4rame sobiva suuruse Kafka klastri jaoks Kuberneteses.\" src=\"\/wp-content\/uploads\/2020\/02\/63f10177f01342039f04192ea93c8bca.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"M\u00e4\u00e4rame sobiva suuruse Kafka klastri jaoks Kuberneteses.\" src=\"\/wp-content\/uploads\/2020\/02\/48b992cc73a968b3d3b1442f96562c4e.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<\/p>\n<h3>3 brokerit, s\u00f5numite suurus \u2014 100 baiti<\/h3>\n<p>\nS\u00f5numite suuruse v\u00e4henemisega langeb l\u00e4bilaskev\u00f5ime umbes 15-20%: see m\u00f5jutab iga s\u00f5numi t\u00f6\u00f6tlemiseks kuluvat aega. Samuti on protsessori koormus peaaegu kahekordistunud.<\/p>\n<p><img decoding=\"async\" alt=\"M\u00e4\u00e4rame sobiva suuruse Kafka klastri jaoks Kuberneteses.\" src=\"\/wp-content\/uploads\/2020\/02\/2f7dc5a94da371cfe17c9e0fc0ec1941.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"M\u00e4\u00e4rame sobiva suuruse Kafka klastri jaoks Kuberneteses.\" src=\"\/wp-content\/uploads\/2020\/02\/986160a3a4e099a0dbb8aeefaf6f1c2f.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"M\u00e4\u00e4rame sobiva suuruse Kafka klastri jaoks Kuberneteses.\" src=\"\/wp-content\/uploads\/2020\/02\/c08244457bf4dbfadf52838d4ea2969e.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nKuna brokeri s\u00f5lmedes on endiselt kasutamata tuumasid, saab j\u00f5udlust parandada Kafka konfiguratsiooni muutmisega. See pole lihtne \u00fclesanne, seet\u00f5ttu on parem t\u00f6\u00f6tada suuremate s\u00f5numitega, et suurendada l\u00e4bilaskev\u00f5imet.<\/p>\n<h3>4 brok silent, s\u00f5numi suurus \u2014 512 baiti<\/h3>\n<p>\nKafka klastrite l\u00e4bilaskev\u00f5imet on lihtne suurendada, lisan uusi brokkereid ja hoides partition'ide tasakaalu (see tagab koormuse \u00fchtlase jaotumise brokkerite vahel). Meie juhul suurenes klastrite l\u00e4bilaskev\u00f5ime brokkereid lisades <i>~580 Mb\/s (~1,1 miljonit s\u00f5numit sekundis)<\/i>. T\u00f5us oli v\u00e4iksem, kui oodata oli: see on peamiselt seletatav partition'ide tasakaalu puudumisega (k\u00f5ik brokkerid ei t\u00f6\u00f6ta oma maksimaalsetes v\u00f5imetes).<\/p>\n<p><img decoding=\"async\" alt=\"M\u00e4\u00e4rame sobiva suuruse Kafka klastri jaoks Kuberneteses.\" src=\"\/wp-content\/uploads\/2020\/02\/8c8d2e70739add3c3f88fe42acf98c46.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"M\u00e4\u00e4rame sobiva suuruse Kafka klastri jaoks Kuberneteses.\" src=\"\/wp-content\/uploads\/2020\/02\/a6e4812270c04487de0ad39b996a13f8.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"M\u00e4\u00e4rame sobiva suuruse Kafka klastri jaoks Kuberneteses.\" src=\"\/wp-content\/uploads\/2020\/02\/6162c04cfb87b9d9cd0f3147a4ae1647.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"M\u00e4\u00e4rame sobiva suuruse Kafka klastri jaoks Kuberneteses.\" src=\"\/wp-content\/uploads\/2020\/02\/dae5de3b14be212c679f3627a3b77d7b.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nJVM-i masina m\u00e4lu tarbimine j\u00e4i alla 2 Gb:<\/p>\n<p><img decoding=\"async\" alt=\"M\u00e4\u00e4rame sobiva suuruse Kafka klastri jaoks Kuberneteses.\" src=\"\/wp-content\/uploads\/2020\/02\/0bca07a817d7da3104b08de10979f049.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"M\u00e4\u00e4rame sobiva suuruse Kafka klastri jaoks Kuberneteses.\" src=\"\/wp-content\/uploads\/2020\/02\/7e1e64f80ec140f13e171cfe3be930ce.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"M\u00e4\u00e4rame sobiva suuruse Kafka klastri jaoks Kuberneteses.\" src=\"\/wp-content\/uploads\/2020\/02\/5e9700b1e54e45f9cd5fa018ec07dba1.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"M\u00e4\u00e4rame sobiva suuruse Kafka klastri jaoks Kuberneteses.\" src=\"\/wp-content\/uploads\/2020\/02\/4ba0c4b59bfe72bcb0d2f22056c47d79.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nBrokkereid m\u00f5jutab partition'ide tasakaalu puudumine:<\/p>\n<p><img decoding=\"async\" alt=\"M\u00e4\u00e4rame sobiva suuruse Kafka klastri jaoks Kuberneteses.\" src=\"\/wp-content\/uploads\/2020\/02\/2b8ed3c21adf70c751d1daa6d4c41895.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"M\u00e4\u00e4rame sobiva suuruse Kafka klastri jaoks Kuberneteses.\" src=\"\/wp-content\/uploads\/2020\/02\/77b980869b99903d9de5cd7d43d79291.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"M\u00e4\u00e4rame sobiva suuruse Kafka klastri jaoks Kuberneteses.\" src=\"\/wp-content\/uploads\/2020\/02\/f2f9569a3a1ef0ce35e89b2e52e4a8b2.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"M\u00e4\u00e4rame sobiva suuruse Kafka klastri jaoks Kuberneteses.\" src=\"\/wp-content\/uploads\/2020\/02\/191d6d97b32b2128c5cfd3914171de15.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<\/p>\n<h2>J\u00e4reldused<\/h2>\n<p>\n\u00dclaltoodud iteratiivset l\u00e4henemist saab laiendada, et katta keerukamaid stsenaariume, sealhulgas sadu tarbijaid, repartitioning'ut, uuendusi, pod'ide taask\u00e4ivitamist jne. See k\u00f5ik v\u00f5imaldab meil hinnata Kafka klastrite v\u00f5imeid erinevates tingimustes, tuvastada kitsaskohti ja leida meetodeid nende lahendamiseks.<\/p>\n<p>Oleme loonud Supertubes'i kiireks ja lihtsaks klastrite juurutamiseks, seadistamiseks, brokkerite ja teemade lisamiseks\/eemaldamiseks, h\u00e4iretele reageerimiseks ja Kafka \u00f5ige toimimise tagamiseks Kuberneteses. Meie eesm\u00e4rk on aidata keskenduda p\u00f5hi\u00fclesandele ('saata' ja 'tarbida' Kafka s\u00f5numeid) ning kogu keeruline t\u00f6\u00f6 usaldada Supertubes'ile ja Kafka operator'ile.<\/p>\n<p>Kui olete huvitatud tehnikast ja Banzai Cloudi avatud l\u00e4htekoodiga projektidest, j\u00e4lgige ettev\u00f5tet <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/banzaicloud\">GitHub<\/a><\/noindex>, <noindex><a rel=\"nofollow\" href=\"https:\/\/www.linkedin.com\/company\/banzaicloud\">LinkedIn<\/a><\/noindex> v\u00f5i <noindex><a rel=\"nofollow\" href=\"https:\/\/twitter.com\/BanzaiCloud\">Twitter<\/a><\/noindex>.<\/p>\n<h2>P.S. t\u00f5lkija m\u00e4rkused<\/h2>\n<p>\nLugege ka meie blogist:<\/p>\n<ul>\n<li> \u00ab<noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/480722\/\">Lugu Redis- operaatorist K8s-is ja l\u00fchike \u00fclevaade selle andmebaasi anal\u00fc\u00fcsit\u00f6\u00f6riistadest.<\/a><\/noindex>\u00bb;<\/li>\n<li> \u00ab<noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/450662\/\">Ainulaadne RabbitMQ migreerimine Kubernetesesse<\/a><\/noindex>\u00bb;<\/li>\n<li> \u00ab<noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/329224\/\">zetcd CoreOS'ilt: Asendades ZooKeeperi\u2026 etcd salvestusega<\/a><\/noindex>\u00bb.<\/li>\n<\/ul>\n<p>Allikas: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/488920\/\">habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u041f\u0440\u0438\u043c. \u043f\u0435\u0440\u0435\u0432.: \u0412 \u044d\u0442\u043e\u0439 \u0441\u0442\u0430\u0442\u044c\u0435 \u043a\u043e\u043c\u043f\u0430\u043d\u0438\u044f Banzai Cloud \u0434\u0435\u043b\u0438\u0442\u0441\u044f \u043f\u0440\u0438\u043c\u0435\u0440\u043e\u043c \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u043d\u0438\u044f \u0435\u0451 \u0441\u043f\u0435\u0446\u0438\u0430\u043b\u044c\u043d\u044b\u0445 \u0443\u0442\u0438\u043b\u0438\u0442 \u0434\u043b\u044f \u043e\u0431\u043b\u0435\u0433\u0447\u0435\u043d\u0438\u044f \u044d\u043a\u0441\u043f\u043b\u0443\u0430\u0442\u0430\u0446\u0438\u0438 Kafka \u0432 \u0440\u0430\u043c\u043a\u0430\u0445 Kubernetes. \u041f\u0440\u0438\u0432\u043e\u0434\u0438\u043c\u044b\u0435 \u0438\u043d\u0441\u0442\u0440\u0443\u043a\u0446\u0438\u0438 \u0438\u043b\u043b\u044e\u0441\u0442\u0440\u0438\u0440\u0443\u044e\u0442, \u043a\u0430\u043a \u043c\u043e\u0436\u043d\u043e \u043e\u043f\u0440\u0435\u0434\u0435\u043b\u0438\u0442\u044c \u043e\u043f\u0442\u0438\u043c\u0430\u043b\u044c\u043d\u044b\u0439 \u0440\u0430\u0437\u043c\u0435\u0440 \u0438\u043d\u0444\u0440\u0430\u0441\u0442\u0440\u0443\u043a\u0442\u0443\u0440\u044b \u0438 \u043d\u0430\u0441\u0442\u0440\u043e\u0438\u0442\u044c \u0441\u0430\u043c\u0443 Kafka \u0434\u043b\u044f \u0434\u043e\u0441\u0442\u0438\u0436\u0435\u043d\u0438\u044f \u0442\u0440\u0435\u0431\u0443\u0435\u043c\u043e\u0439 \u043f\u0440\u043e\u043f\u0443\u0441\u043a\u043d\u043e\u0439 \u0441\u043f\u043e\u0441\u043e\u0431\u043d\u043e\u0441\u0442\u0438. Apache Kafka \u2014 \u0440\u0430\u0441\u043f\u0440\u0435\u0434\u0435\u043b\u0451\u043d\u043d\u0430\u044f \u0441\u0442\u0440\u0438\u043c\u0438\u043d\u0433\u043e\u0432\u0430\u044f \u043f\u043b\u0430\u0442\u0444\u043e\u0440\u043c\u0430 \u0434\u043b\u044f \u0441\u043e\u0437\u0434\u0430\u043d\u0438\u044f \u043d\u0430\u0434\u0451\u0436\u043d\u044b\u0445, \u043c\u0430\u0441\u0448\u0442\u0430\u0431\u0438\u0440\u0443\u0435\u043c\u044b\u0445 \u0438 \u0432\u044b\u0441\u043e\u043a\u043e\u043f\u0440\u043e\u0438\u0437\u0432\u043e\u0434\u0438\u0442\u0435\u043b\u044c\u043d\u044b\u0445 \u043f\u043e\u0442\u043e\u043a\u043e\u0432\u044b\u0445 \u0441\u0438\u0441\u0442\u0435\u043c \u0440\u0435\u0430\u043b\u044c\u043d\u043e\u0433\u043e \u0432\u0440\u0435\u043c\u0435\u043d\u0438. [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":71214,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-71213","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 4.9.10 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u041f\u0440\u0438\u043c. \u043f\u0435\u0440\u0435\u0432.: \u0412 \u044d\u0442\u043e\u0439 \u0441\u0442\u0430\u0442\u044c\u0435 \u043a\u043e\u043c\u043f\u0430\u043d\u0438\u044f Banzai Cloud \u0434\u0435\u043b\u0438\u0442\u0441\u044f \u043f\u0440\u0438\u043c\u0435\u0440\u043e\u043c \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u043d\u0438\u044f \u0435\u0451 \u0441\u043f\u0435\u0446\u0438\u0430\u043b\u044c\u043d\u044b\u0445 \u0443\u0442\u0438\u043b\u0438\u0442 \u0434\u043b\u044f \u043e\u0431\u043b\u0435\u0433\u0447\u0435\u043d\u0438\u044f \u044d\u043a\u0441\u043f\u043b\u0443\u0430\u0442\u0430\u0446\u0438\u0438 Kafka \u0432 \u0440\u0430\u043c\u043a\u0430\u0445 Kubernetes. \u041f\u0440\u0438\u0432\u043e\u0434\u0438\u043c\u044b\u0435 \u0438\u043d\u0441\u0442\u0440\u0443\u043a\u0446\u0438\u0438 \u0438\u043b\u043b\u044e\u0441\u0442\u0440\u0438\u0440\u0443\u044e\u0442, \u043a\u0430\u043a \u043c\u043e\u0436\u043d\u043e \u043e\u043f\u0440\u0435\u0434\u0435\u043b\u0438\u0442\u044c \u043e\u043f\u0442\u0438\u043c\u0430\u043b\u044c\u043d\u044b\u0439 \u0440\u0430\u0437\u043c\u0435\u0440 \u0438\u043d\u0444\u0440\u0430\u0441\u0442\u0440\u0443\u043a\u0442\u0443\u0440\u044b \u0438 \u043d\u0430\u0441\u0442\u0440\u043e\u0438\u0442\u044c \u0441\u0430\u043c\u0443 Kafka \u0434\u043b\u044f \u0434\u043e\u0441\u0442\u0438\u0436\u0435\u043d\u0438\u044f \u0442\u0440\u0435\u0431\u0443\u0435\u043c\u043e\u0439 \u043f\u0440\u043e\u043f\u0443\u0441\u043a\u043d\u043e\u0439 \u0441\u043f\u043e\u0441\u043e\u0431\u043d\u043e\u0441\u0442\u0438. Apache Kafka \u2014 \u0440\u0430\u0441\u043f\u0440\u0435\u0434\u0435\u043b\u0451\u043d\u043d\u0430\u044f \u0441\u0442\u0440\u0438\u043c\u0438\u043d\u0433\u043e\u0432\u0430\u044f \u043f\u043b\u0430\u0442\u0444\u043e\u0440\u043c\u0430 \u0434\u043b\u044f \u0441\u043e\u0437\u0434\u0430\u043d\u0438\u044f \u043d\u0430\u0434\u0451\u0436\u043d\u044b\u0445, \u043c\u0430\u0441\u0448\u0442\u0430\u0431\u0438\u0440\u0443\u0435\u043c\u044b\u0445 \u0438 \u0432\u044b\u0441\u043e\u043a\u043e\u043f\u0440\u043e\u0438\u0437\u0432\u043e\u0434\u0438\u0442\u0435\u043b\u044c\u043d\u044b\u0445 \u043f\u043e\u0442\u043e\u043a\u043e\u0432\u044b\u0445 \u0441\u0438\u0441\u0442\u0435\u043c \u0440\u0435\u0430\u043b\u044c\u043d\u043e\u0433\u043e \u0432\u0440\u0435\u043c\u0435\u043d\u0438.\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/et\/blog\/administrirovanie\/opredelyaem-podhodyashhij-razmer-dlya-klastera-kafka-v-kubernetes\" \/>\n\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 4.9.10\" \/>\n\t\t<meta property=\"og:locale\" content=\"et_EE\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47\u041e\u043f\u0440\u0435\u0434\u0435\u043b\u044f\u0435\u043c \u043f\u043e\u0434\u0445\u043e\u0434\u044f\u0449\u0438\u0439 \u0440\u0430\u0437\u043c\u0435\u0440 \u0434\u043b\u044f \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0430 Kafka \u0432 Kubernetes | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u041f\u0440\u0438\u043c. \u043f\u0435\u0440\u0435\u0432.: \u0412 \u044d\u0442\u043e\u0439 \u0441\u0442\u0430\u0442\u044c\u0435 \u043a\u043e\u043c\u043f\u0430\u043d\u0438\u044f Banzai Cloud \u0434\u0435\u043b\u0438\u0442\u0441\u044f \u043f\u0440\u0438\u043c\u0435\u0440\u043e\u043c \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u043d\u0438\u044f \u0435\u0451 \u0441\u043f\u0435\u0446\u0438\u0430\u043b\u044c\u043d\u044b\u0445 \u0443\u0442\u0438\u043b\u0438\u0442 \u0434\u043b\u044f \u043e\u0431\u043b\u0435\u0433\u0447\u0435\u043d\u0438\u044f \u044d\u043a\u0441\u043f\u043b\u0443\u0430\u0442\u0430\u0446\u0438\u0438 Kafka \u0432 \u0440\u0430\u043c\u043a\u0430\u0445 Kubernetes. \u041f\u0440\u0438\u0432\u043e\u0434\u0438\u043c\u044b\u0435 \u0438\u043d\u0441\u0442\u0440\u0443\u043a\u0446\u0438\u0438 \u0438\u043b\u043b\u044e\u0441\u0442\u0440\u0438\u0440\u0443\u044e\u0442, \u043a\u0430\u043a \u043c\u043e\u0436\u043d\u043e \u043e\u043f\u0440\u0435\u0434\u0435\u043b\u0438\u0442\u044c \u043e\u043f\u0442\u0438\u043c\u0430\u043b\u044c\u043d\u044b\u0439 \u0440\u0430\u0437\u043c\u0435\u0440 \u0438\u043d\u0444\u0440\u0430\u0441\u0442\u0440\u0443\u043a\u0442\u0443\u0440\u044b \u0438 \u043d\u0430\u0441\u0442\u0440\u043e\u0438\u0442\u044c \u0441\u0430\u043c\u0443 Kafka \u0434\u043b\u044f \u0434\u043e\u0441\u0442\u0438\u0436\u0435\u043d\u0438\u044f \u0442\u0440\u0435\u0431\u0443\u0435\u043c\u043e\u0439 \u043f\u0440\u043e\u043f\u0443\u0441\u043a\u043d\u043e\u0439 \u0441\u043f\u043e\u0441\u043e\u0431\u043d\u043e\u0441\u0442\u0438. Apache Kafka \u2014 \u0440\u0430\u0441\u043f\u0440\u0435\u0434\u0435\u043b\u0451\u043d\u043d\u0430\u044f \u0441\u0442\u0440\u0438\u043c\u0438\u043d\u0433\u043e\u0432\u0430\u044f \u043f\u043b\u0430\u0442\u0444\u043e\u0440\u043c\u0430 \u0434\u043b\u044f \u0441\u043e\u0437\u0434\u0430\u043d\u0438\u044f \u043d\u0430\u0434\u0451\u0436\u043d\u044b\u0445, \u043c\u0430\u0441\u0448\u0442\u0430\u0431\u0438\u0440\u0443\u0435\u043c\u044b\u0445 \u0438 \u0432\u044b\u0441\u043e\u043a\u043e\u043f\u0440\u043e\u0438\u0437\u0432\u043e\u0434\u0438\u0442\u0435\u043b\u044c\u043d\u044b\u0445 \u043f\u043e\u0442\u043e\u043a\u043e\u0432\u044b\u0445 \u0441\u0438\u0441\u0442\u0435\u043c \u0440\u0435\u0430\u043b\u044c\u043d\u043e\u0433\u043e \u0432\u0440\u0435\u043c\u0435\u043d\u0438.\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/et\/blog\/administrirovanie\/opredelyaem-podhodyashhij-razmer-dlya-klastera-kafka-v-kubernetes\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2020-02-24T13:09:17+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-03-03T13:14:29+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47M\u00e4\u00e4rame sobiva suuruse Kafka klastri jaoks Kuberneteses | ProHoster","description":"M\u00e4rkus: Selles artiklis jagab ettev\u00f5te Banzai Cloud n\u00e4idet oma spetsiaalsete utiliitide kasutamisest Kafka haldamiseks Kuberneteses. Antud juhised illustreerivad, kuidas m\u00e4\u00e4rata infrastruktuuri optimaalne suurus ning seadistada Kafka, et saavutada soovitud l\u00e4bilaskev\u00f5ime. Apache Kafka on jaotatud voogedastusplatvorm usaldusv\u00e4\u00e4rsete, skaleeritavate ja suure j\u00f5udlusega reaalajas voogesituss\u00fcsteemide loomiseks.","canonical_url":"https:\/\/prohoster.info\/et\/blog\/administrirovanie\/opredelyaem-podhodyashhij-razmer-dlya-klastera-kafka-v-kubernetes","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"et_EE","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u041e\u043f\u0440\u0435\u0434\u0435\u043b\u044f\u0435\u043c \u043f\u043e\u0434\u0445\u043e\u0434\u044f\u0449\u0438\u0439 \u0440\u0430\u0437\u043c\u0435\u0440 \u0434\u043b\u044f \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0430 Kafka \u0432 Kubernetes | ProHoster","og:description":"\u041f\u0440\u0438\u043c. \u043f\u0435\u0440\u0435\u0432.: \u0412 \u044d\u0442\u043e\u0439 \u0441\u0442\u0430\u0442\u044c\u0435 \u043a\u043e\u043c\u043f\u0430\u043d\u0438\u044f Banzai Cloud \u0434\u0435\u043b\u0438\u0442\u0441\u044f \u043f\u0440\u0438\u043c\u0435\u0440\u043e\u043c \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u043d\u0438\u044f \u0435\u0451 \u0441\u043f\u0435\u0446\u0438\u0430\u043b\u044c\u043d\u044b\u0445 \u0443\u0442\u0438\u043b\u0438\u0442 \u0434\u043b\u044f \u043e\u0431\u043b\u0435\u0433\u0447\u0435\u043d\u0438\u044f \u044d\u043a\u0441\u043f\u043b\u0443\u0430\u0442\u0430\u0446\u0438\u0438 Kafka \u0432 \u0440\u0430\u043c\u043a\u0430\u0445 Kubernetes. \u041f\u0440\u0438\u0432\u043e\u0434\u0438\u043c\u044b\u0435 \u0438\u043d\u0441\u0442\u0440\u0443\u043a\u0446\u0438\u0438 \u0438\u043b\u043b\u044e\u0441\u0442\u0440\u0438\u0440\u0443\u044e\u0442, \u043a\u0430\u043a \u043c\u043e\u0436\u043d\u043e \u043e\u043f\u0440\u0435\u0434\u0435\u043b\u0438\u0442\u044c \u043e\u043f\u0442\u0438\u043c\u0430\u043b\u044c\u043d\u044b\u0439 \u0440\u0430\u0437\u043c\u0435\u0440 \u0438\u043d\u0444\u0440\u0430\u0441\u0442\u0440\u0443\u043a\u0442\u0443\u0440\u044b \u0438 \u043d\u0430\u0441\u0442\u0440\u043e\u0438\u0442\u044c \u0441\u0430\u043c\u0443 Kafka \u0434\u043b\u044f \u0434\u043e\u0441\u0442\u0438\u0436\u0435\u043d\u0438\u044f \u0442\u0440\u0435\u0431\u0443\u0435\u043c\u043e\u0439 \u043f\u0440\u043e\u043f\u0443\u0441\u043a\u043d\u043e\u0439 \u0441\u043f\u043e\u0441\u043e\u0431\u043d\u043e\u0441\u0442\u0438. Apache Kafka \u2014 \u0440\u0430\u0441\u043f\u0440\u0435\u0434\u0435\u043b\u0451\u043d\u043d\u0430\u044f \u0441\u0442\u0440\u0438\u043c\u0438\u043d\u0433\u043e\u0432\u0430\u044f \u043f\u043b\u0430\u0442\u0444\u043e\u0440\u043c\u0430 \u0434\u043b\u044f \u0441\u043e\u0437\u0434\u0430\u043d\u0438\u044f \u043d\u0430\u0434\u0451\u0436\u043d\u044b\u0445, \u043c\u0430\u0441\u0448\u0442\u0430\u0431\u0438\u0440\u0443\u0435\u043c\u044b\u0445 \u0438 \u0432\u044b\u0441\u043e\u043a\u043e\u043f\u0440\u043e\u0438\u0437\u0432\u043e\u0434\u0438\u0442\u0435\u043b\u044c\u043d\u044b\u0445 \u043f\u043e\u0442\u043e\u043a\u043e\u0432\u044b\u0445 \u0441\u0438\u0441\u0442\u0435\u043c \u0440\u0435\u0430\u043b\u044c\u043d\u043e\u0433\u043e \u0432\u0440\u0435\u043c\u0435\u043d\u0438.","og:url":"https:\/\/prohoster.info\/et\/blog\/administrirovanie\/opredelyaem-podhodyashhij-razmer-dlya-klastera-kafka-v-kubernetes","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-02-24T13:09:17+00:00","article:modified_time":"2020-03-03T13:14:29+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"71213","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 19:06:22","updated":"2022-10-04 20:58:00"},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/et\/wp-json\/wp\/v2\/posts\/71213","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/et\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/et\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/et\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/et\/wp-json\/wp\/v2\/comments?post=71213"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/et\/wp-json\/wp\/v2\/posts\/71213\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/et\/wp-json\/wp\/v2\/media\/71214"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/et\/wp-json\/wp\/v2\/media?parent=71213"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/et\/wp-json\/wp\/v2\/categories?post=71213"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/et\/wp-json\/wp\/v2\/tags?post=71213"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}