Sarudza saizi yakakodzera yeboka reKafka muKubernetes

Cherechedza. transl.: Muchinyorwa chino, Banzai Cloud inogovera muenzaniso wekuti maturusi ayo echivanhu anogona kushandiswa sei kuita kuti Kafka ive nyore kushandisa mukati meKubernetes. Mirayiridzo inotevera inoratidza maitiro aunokwanisa kuona hukuru hwakakwana hwezvivakwa zvako uye kugadzirisa Kafka pachayo kuti uwane iyo inodiwa throughput.

Apache Kafka inzvimbo yakagoverwa yekushambadzira yekugadzira yakavimbika, inoshoreka uye yepamusoro-inoshanda chaiyo-nguva yekufambisa masisitimu. Kugona kwayo kunoshamisa kunogona kuwedzerwa uchishandisa Kubernetes. Nokuda kwaizvozvi takazvigadzira Open Source Kafka opareta uye mudziyo unonzi Supertubes. Ivo vanokutendera kuti umhanye Kafka paKubernetes uye shandisa ayo akasiyana maficha, akadai sekugadzirisa zvakanaka dhizaini dhizaini, metric-yakavakirwa kuyera nekuenzanisazve, rack kuziva, "nyoro" (nenyasha) kuburitsa zvinyorwa zvitsva, nezvimwe.

Edza Supertubes musumbu rako:

curl https://getsupertubes.sh | sh и supertubes install -a --no-democluster --kubeconfig <path-to-eks-cluster-kubeconfig-file>

Kana kuonana zvinyorwa. Iwe unogona zvakare kuverenga nezve zvimwe zvekugona kweKafka, iro basa rinoitwa otomatiki uchishandisa Supertubes uye Kafka opareta. Takatonyora nezvavo pablog:

• Kwete! Imwezve Kafka opareta yeKubernetes;
• Tarisa uye shandisa Kafka zvichibva paPrometheus metrics;
• Kafka rack kuziva paKubernetes;
• Kumhanya Apache Kafka pamusoro peIstio - bhenji;
• Mushandisi akatenderwa uye kuwana anodzorwa masumbu ane Kafka opareta;
• Kafka rolling kukwidziridzwa uye inoshanduka gadziriso paKubernetes;
• Envoy protocol filter yeKafka, meshed.

Paunosarudza kuendesa boka reKafka paKubernetes, ungangove wakatarisana nedambudziko rekuona saizi yakakwana yezvivakwa zvepasi uye kudiwa kwekugadzirisa Kafka yako kumisikidzwa kuti isangane nezvinodiwa. Kunyanya kushanda kwemutengesi wega wega kunotarirwa nekushanda kwezvikamu zvezvivakwa zvemukati, senge memory, processor, disk speed, network bandwidth, nezvimwe.

Sezvineiwo, iyo bhuroka dhizaini inofanirwa kuve yekuti zvese zvivakwa zvishandiswa kune yavo yakanyanya kugona. Nekudaro, muhupenyu chaihwo kuseta uku kwakaoma. Zvinonyanya kuitika kuti vashandisi vachagadzirisa vatengesi kuti vawedzere kushandiswa kwechimwe kana zviviri zvikamu (disk, memory, kana processor). Kazhinji kutaura, bhuroka rinoratidza hukuru hwekuita kana kumisikidzwa kwayo ichibvumira iyo inononoka chikamu kuti ishandiswe kusvika pakuzara kwayo. Nenzira iyi tinogona kuwana yakaomesesa pfungwa yemutoro uyo mumwe bhuroka anogona kubata.

Theoretically, isu tinogona zvakare kufungidzira huwandu hwemabhuroka anodiwa kubata mutoro wakapihwa. Zvisinei, mukuita kune zvakawanda zvingasarudzwa zvekugadzirisa pamatanho akasiyana-siyana zvokuti zvakaoma zvikuru (kana zvisingabviri) kuongorora kukwanisa kushanda kweimwe gadziriro. Mune mamwe mazwi, zvakanyanya kuoma kuronga gadziriso zvichienderana nekumwe kuita kwakapihwa.

Kune vashandisi veSupertubes, isu tinowanzo tora nzira inotevera: isu tinotanga neimwe gadziriso (infrastructure + marongero), tobva tayera kuita kwayo, gadzirisa marongero ebroker uye dzokorora maitiro zvakare. Izvi zvinoitika kusvikira chikamu chinononoka chezvivakwa chashandiswa zvizere.

Nenzira iyi, tinowana pfungwa yakajeka yekuti vangani mabroker sumbu rinoda kubata mumwe mutoro (huwandu hwemabhuroka zvakare hunoenderana nezvimwe zvinhu, senge hushoma nhamba yemeseji replicas kuona kusimba, huwandu hwechikamu. vatungamiri, nezvimwewo). Pamusoro pezvo, tinowana nzwisiso yekuti ndezvipi zvikamu zvezvivakwa zvinoda vertical scaling.

Ichi chinyorwa chichataura pamusoro pematanho atinotora kuti tiwane zvakanyanya kubva pane zvishoma zvishoma muzvikamu zvekutanga uye kuyera kubuda kweboka reKafka. Kurongeka kwakasimba kunoda kanenge matatu anomhanya bhuroka (min.insync.replicas=3), yakagoverwa munzvimbo nhatu dzakasiyana dzekusvikika. Kugadzirisa, kuyera uye kutarisa Kubernetes zvivakwa, isu tinoshandisa yedu pachedu manejimendi chikuva chemakore akasanganiswa - Pipeline. Inotsigira pane-nzvimbo (simbi isina chinhu, VMware) uye marudzi mashanu emakore (Alibaba, AWS, Azure, Google, Oracle), pamwe nechero musanganiswa wavo.

Pfungwa paKafka cluster zvivakwa uye kumisikidzwa

Pamienzaniso iri pazasi, takasarudza AWS semupi wegore uye EKS sekugovera Kubernetes. Chimiro chakafanana chinogona kushandiswa uchishandisa P.K.E. -Kubernetes kugovera kubva kuBanzai Cloud, yakasimbiswa neCNCF.

dhisiki

Amazon inopa zvakasiyana-siyana EBS vhoriyamu mhando. Pakati pepakati gp2 и io1 kune maSSD madhiraivha, zvisinei, kuve nechokwadi chepamusoro gp2 inoshandisa zvikwereti zvakaunganidzwa (I/O zvikwereti), saka takasarudza mhando io1, iyo inopa inowirirana yakakwirira throughput.

Mhando dzemienzaniso

Kuita kwaKafka kunonyanya kutsamira pacheshi yepeji rekushandisa, saka isu tinoda zviitiko zvine ndangariro dzakakwana kune vatengesi (JVM) uye peji cache. Muenzaniso c5.2 yakakura - kutanga kwakanaka, sezvo ine 16 GB yekuyeuka uye yakagadziridzwa kushanda neEBS. Kukanganisa kwayo ndeyekuti inongokwanisa kupa huwandu hwekuita kweasingapfuure maminetsi makumi matatu ega ega maawa makumi maviri nemana. Kana basa rako richida kushanda kwepamusoro kwenguva yakareba, ungada kufunga nezvemamwe marudzi emuenzaniso. Ndizvo chaizvo zvatakaita, takamira c5.4 yakakura. Inopa huwandu hwepamusoro mukati 593,75 Mb/s. Yakanyanya kudarika yevhoriyamu yeEBS io1 yakakwirira kupfuura muenzaniso c5.4 yakakura, saka chinononoka chinhu chezvivakwa chinogona kunge chiri iyo I/O yekufambisa yeiyi muenzaniso mhando (iyo yedu miyedzo yemutoro inofanirawo kusimbisa).

Network

Iyo network throughput inofanirwa kuve yakakura zvakakwana kana ichienzaniswa nekuita kweVM muenzaniso uye diski, zvikasadaro network inova bhodhoro. Muchiitiko chedu, iyo network interface c5.4 yakakura inotsigira kumhanya kweanosvika gumi Gb/s, iyo yakakwira zvakanyanya kupfuura iyo I/O yekubuda kwechiitiko cheVM.

Broker Deployment

Mabhuroka anofanirwa kuendeswa (akarongwa muKubernetes) kunzvimbo dzakatsaurirwa kudzivirira kukwikwidza nemamwe maitiro eCPU, ndangariro, network, uye disk zviwanikwa.

Java version

Sarudzo ine musoro iJava 11 nekuti inoenderana neDocker mupfungwa yekuti JVM inonyatso sarudza ma processor uye ndangariro inowanikwa kune mudziyo uri kushanda nebroker. Kuziva kuti CPU miganhu yakakosha, iyo JVM mukati uye pachena inoisa huwandu hweGC tambo uye JIT tambo. Takashandisa mufananidzo weKafka banzaicloud/kafka:2.13-2.4.0, iyo inosanganisira Kafka vhezheni 2.4.0 (Scala 2.13) paJava 11.

Kana iwe uchida kudzidza zvakawanda nezveJava/JVM paKubernetes, tarisa zvinotevera mameseji:

• Nei yangu Java application iri OOMKilled;
• Maitiro ekuita sei saizi midziyo yeJava 10 application.

Broker memory marongero

Pane zvinhu zviviri zvakakosha pakugadzirisa bhuroka ndangariro: zvigadziriso zveJVM uye zveKubernetes pod. Muganho wekurangarira wakaseterwa podhi unofanirwa kuve wakakura kupfuura saizi yepamusoro yemurwi kuitira kuti JVM ive nenzvimbo yeJava metaspace inogara mundangariro yayo uye neyekushandisa peji cache iyo Kafka anoshandisa nesimba. Mumiyedzo yedu takatangisa Kafka vatengesi vane parameter -Xmx4G -Xms2G, uye muganho wendangariro wepodhi waiva 10 Gi. Ndokumbira utarise kuti ndangariro marongero eJVM anogona kuwanikwa otomatiki uchishandisa -XX:MaxRAMPercentage и -X:MinRAMPercentage, zvichibva pamuganho wendangariro wepodhi.

Broker processor marongero

Kazhinji kutaura, unogona kuvandudza mashandiro nekuwedzera parallelism nekuwedzera huwandu hwetambo dzinoshandiswa naKafka. Iyo yakawanda processors inowanikwa yeKafka, zviri nani. Mukuedza kwedu, takatanga nemuganhu we 6 processors uye zvishoma nezvishoma (kuburikidza neiterations) vakasimudza nhamba yavo kusvika 15. Mukuwedzera, tinoisa. num.network.threads=12 mune zvigadziriso zvebhuroka kuti uwedzere huwandu hwetambo dzinogamuchira data kubva kunetiweki uye kuitumira. Pavakangoona kuti vateveri vatengesi vaisakwanisa kugamuchira replicas nekukurumidza, vakasimudza num.replica.fetchers kusvika ku4 kuwedzera kumhanya uko vateveri vatengesi vakadzokorora mameseji kubva kuvatungamiriri.

Rodha Generation Tool

Iwe unofanirwa kuve nechokwadi kuti iyo yakasarudzwa mutoro jenareta haipere simba pamberi peKafka cluster (iyo iri kumisikidzwa) yasvika pakuremerwa kwayo. Mune mamwe mazwi, zvinodikanwa kuita ongororo yekutanga yekugona kwechigadzirwa chekutakura mutoro, uye zvakare sarudza mhando dzemhando dzayo ine nhamba yakakwana yema processor uye ndangariro. Muchiitiko ichi, chishandiso chedu chinoburitsa mutoro wakawanda kupfuura iyo Kafka cluster inogona kubata. Pashure pekuedza kwakawanda, takagara pamakopi matatu c5.4 yakakura, imwe neimwe yaive nejenareta raimhanya.

Benchmarking

Performance kuyerwa iterative process inosanganisira anotevera matanho:

• kumisikidza zvivakwa (EKS cluster, Kafka cluster, load generation tool, pamwe nePrometheus neGrafana);
• kugadzira mutoro kune imwe nguva yekusefa zvisina tsarukano kutsauka mune yakaunganidzwa maitiro zviratidzo;
• kugadzirisa zvigadziriso zvemubhuroka uye gadziriso zvichibva pane zvakacherechedzwa kuita zviratidzo;
• kudzokorora nzira yacho kusvikira chiyero chinodiwa cheKafka cluster throughput chawanikwa. Panguva imwecheteyo, inofanira kunge ichidzokororwa nguva dzose uye kuratidza misiyano mishoma mukubuda.

Chikamu chinotevera chinotsanangura matanho akaitwa panguva yebvunzo cluster benchmarking process.

Tools

Zvishandiso zvinotevera zvakashandiswa kukurumidza kuendesa gadziriro yekutanga, kugadzira mitoro, uye kuyera kuita:

• Banzai Cloud Pipeline yekuronga boka re EKS kubva kuAmazon c Prometheus (kuunganidza Kafka uye zvivakwa metrics) uye grafana (kuona ma metrics aya). Takatora mukana integrated в Pipeline masevhisi anopa federated monitoring, centralized log collection, vulnerability scanning, kudzoreredza njodzi, bhizinesi-giredhi chengetedzo nezvimwe zvakawanda.
• Sangrenel - chishandiso chekuyedza kurodha boka reKafka.
• Grafana dashboards ekuona Kafka metrics uye zvivakwa: Kubernetes Kafka, Node Exporter.
• Supertubes CLI yenzira iri nyore yekumisikidza Kafka cluster paKubernetes. Zookeeper, Kafka opareta, Envoy uye zvimwe zvinhu zvakawanda zvakaiswa uye zvakagadzirirwa zvakanaka kuti zvimhanye-yakagadzirira Kafka cluster paKubernetes.
  • Zvekugadzwa supertubes CLI shandisa mirairo yakapihwa pano.

EKS cluster

Gadzirira gungano reEKS rine node dzakazvipira dzevashandi c5.4 yakakura munzvimbo dzakasiyana dzekuwanikwa dzepods neKafka broker, pamwe nenzvimbo dzakatsaurirwa dzejenareta remutoro uye yekutarisa zvivakwa.

banzai cluster create -f https://raw.githubusercontent.com/banzaicloud/kafka-operator/master/docs/benchmarks/infrastructure/cluster_eks_202001.json

Kana iyo EKS cluster yasimuka uye ichimhanya, ita kuti ibatanidzwe monitoring service - achaendesa Prometheus naGrafana kuita sumbu.

Kafka system components

Isa Kafka system zvikamu (Zookeeper, kafka-operator) muEKS uchishandisa supertubes CLI:

supertubes install -a --no-democluster --kubeconfig <path-to-eks-cluster-kubeconfig-file>

Kafka cluster

Nekusagadzikana, EKS inoshandisa EBS mavhoriyamu emhando gp2, saka iwe unofanirwa kugadzira kirasi yakaparadzana yekuchengetedza zvichienderana nemavhoriyamu io1 yeKafka cluster:

kubectl create -f - <<EOF
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd
provisioner: kubernetes.io/aws-ebs
parameters:
  type: io1
  iopsPerGB: "50"
  fsType: ext4
volumeBindingMode: WaitForFirstConsumer
EOF

Seta iyo parameter kune vatengesi min.insync.replicas=3 uye tumira mabroker pod pane node munzvimbo nhatu dzakasiyana dzekuwanikwa:

supertubes cluster create -n kafka --kubeconfig <path-to-eks-cluster-kubeconfig-file> -f https://raw.githubusercontent.com/banzaicloud/kafka-operator/master/docs/benchmarks/infrastructure/kafka_202001_3brokers.yaml --wait --timeout 600

Misoro

Isu takamhanyisa matatu majenareta majenareta akafanana. Mumwe nemumwe wavo anonyora kune yavo nyaya, kureva kuti, isu tinoda misoro mitatu yakazara:

supertubes cluster topic create -n kafka --kubeconfig <path-to-eks-cluster-kubeconfig-file> -f -<<EOF
apiVersion: kafka.banzaicloud.io/v1alpha1
kind: KafkaTopic
metadata:
  name: perftest1
spec:
  name: perftest1
  partitions: 12
  replicationFactor: 3
  retention.ms: '28800000'
  cleanup.policy: delete
EOF

supertubes cluster topic create -n kafka --kubeconfig <path-to-eks-cluster-kubeconfig-file> -f -<<EOF
apiVersion: kafka.banzaicloud.io/v1alpha1
kind: KafkaTopic
metadata:
    name: perftest2
spec:
  name: perftest2
  partitions: 12
  replicationFactor: 3
  retention.ms: '28800000'
  cleanup.policy: delete
EOF

supertubes cluster topic create -n kafka --kubeconfig <path-to-eks-cluster-kubeconfig-file> -f -<<EOF
apiVersion: kafka.banzaicloud.io/v1alpha1
kind: KafkaTopic
metadata:
  name: perftest3
spec:
  name: perftest3
  partitions: 12
  replicationFactor: 3
  retention.ms: '28800000'
  cleanup.policy: delete
EOF

Pamusoro wega wega, iyo yekudzokorodza chinhu ndeye 3-hushoma hunokurudzirwa kukosha kune anowanikwa zvakanyanya masisitimu ekugadzira.

Rodha Generation Tool

Takatanga makopi matatu ejenareta rekutakura (rimwe nerimwe rakanyora mune imwe nyaya). Kune mapodhi ejenareta, unofanirwa kuseta node affinity kuitira kuti varongerwe chete pane node dzakagoverwa kwavari:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: loadtest
  name: perf-load1
  namespace: kafka
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: loadtest
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: loadtest
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: nodepool.banzaicloud.io/name
                operator: In
                values:
                - loadgen
      containers:
      - args:
        - -brokers=kafka-0:29092,kafka-1:29092,kafka-2:29092,kafka-3:29092
        - -topic=perftest1
        - -required-acks=all
        - -message-size=512
        - -workers=20
        image: banzaicloud/perfload:0.1.0-blog
        imagePullPolicy: Always
        name: sangrenel
        resources:
          limits:
            cpu: 2
            memory: 1Gi
          requests:
            cpu: 2
            memory: 1Gi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30

Mapoinzi mashoma ekucherechedza:

• Iyo jenareta inoburitsa mameseji e512 byte pakureba uye inoashambadzira kuKafka mumabheji e500 meseji.
• Kushandisa nharo -required-acks=all Kudhindwa kwacho kunoonekwa sekwakabudirira kana zvese zvakawiriraniswa zvemeseji zvagamuchirwa uye zvinosimbiswa neKafka vatengesi. Izvi zvinoreva kuti mubenchmark hatina kuyera chete kumhanya kwevatungamiriri vanogamuchira mameseji, asiwo vateveri vavo vachidzokorora mameseji. Chinangwa chebvunzo iyi hachisi chekuongorora kukurumidza kuverenga kwevatengi (vatengi) nguva pfupi yadarika yakagamuchira meseji ichiri mu OS peji cache, uye kuenzanisa kwayo nekumhanya kwekuverenga mameseji akachengetwa pa diski.
• Iyo jenareta inotakura vashandi makumi maviri mukufanana (-workers=20) Mushandi wega wega ane vagadziri vashanu vanogovera kubatana kwevashandi kune Kafka cluster. Nekuda kweizvozvo, jenareta yega yega ine vagadziri ve5, uye vese vanotumira meseji kuboka reKafka.

Kuongorora hutano hwesumbu

Munguva yekuyedzwa kwemutoro weboka reKafka, isu takatarisawo hutano hwayo kuti ive nechokwadi chekuti pakanga pasina podhi yekutanga, hapana kunze-kwe-sync replicas, uye yakanyanya kubuda nekushanduka kushoma:

• Iyo jenareta inotakura inonyora yakajairwa nhamba nezvehuwandu hwemeseji dzakaburitswa uye mwero wekukanganisa. Chiyero chekukanganisa chinofanira kuramba chakafanana 0,00%.
• Cruise Kudzora, yakashandiswa nekafka-operator, inopa dashboard apo isu tinogonawo kutarisa mamiriro echikwata. Kuti uone pani iyi ita:

  supertubes cluster cruisecontrol show -n kafka --kubeconfig <path-to-eks-cluster-kubeconfig-file>

• ISR level (nhamba ye "in-sync" replicas) kuderera uye kuwedzera zvakaenzana ne0.

Zviyero zvabuda

3 vatengesi, saizi yemeseji - 512 bytes

Nezvikamu zvakagoverwa zvakaenzana kune vatatu vatengesi, takakwanisa kuita basa ~500 Mb/s (inoda kusvika 990 zviuru zvemashoko pasekondi):

Iko kushandiswa kwekuyeuka kweJVM chaiyo muchina hakuna kupfuura 2 GB:

Disk throughput yakasvika pahupamhi hweI/O node throughput pazviitiko zvese zvitatu izvo mabroker aimhanya:

Kubva pane data pakushandiswa kwendangariro nemanodhi, zvinotevera kuti system buffering uye caching yakatora ~ 10-15 GB:

3 vatengesi, saizi yemeseji - 100 bytes

Sezvo saizi yemeseji inodzikira, kubuda kunodonha neanenge 15-20%: iyo nguva inoshandiswa kugadzirisa meseji yega yega inoibata. Mukuwedzera, iyo processor mutoro wakapetwa kaviri.

Sezvo bhuroka node dzichine dzisina kushandiswa cores, kuita kunogona kuvandudzwa nekushandura Kafka kumisikidzwa. Iri harisi basa riri nyore, saka kuti uwedzere throughput zviri nani kushanda nemameseji makuru.

4 vatengesi, saizi yemeseji - 512 bytes

Iwe unogona kuwedzera zviri nyore kuita kweboka reKafka nekungowedzera mabhuroka matsva uye kuchengetedza chiyero chezvikamu (izvi zvinovimbisa kuti mutoro unogovaniswa zvakaenzana pakati pemabhuroka). Muchiitiko chedu, mushure mekuwedzera bhuroka, iyo cluster throughput yakawedzera kusvika ~580 Mb/s (~1,1 miriyoni meseji pasekondi). Kukura kwacho kwakave kushoma pane zvaitarisirwa: izvi zvinonyanya kutsanangurwa nekusaenzana kwezvikamu (kwete vese vatengesi vanoshanda pakakwirira kwekugona kwavo).

Memory kushandiswa kweJVM muchina wakaramba uri pazasi 2 GB:

Basa remabroker ane madhiraivha rakakanganiswa nekusaenzana kwezvikamu:

zvakawanikwa

Iyo iterative nzira yaunzwa pamusoro inogona kukwidziridzwa kuti ivhare zvakanyanya kuomarara mamiriro anosanganisira mazana evatengi, repartition, rolling updates, pod restarts, nezvimwe. Zvose izvi zvinotibvumira kuongorora miganhu yekukwanisa kweKafka cluster mumamiriro akasiyana-siyana, kuziva mabhodhoro mukushanda kwayo uye kutsvaga nzira dzekuzvirwisa.

Isu takagadzira Supertubes kuti ikurumidze uye nyore kuendesa sumbu, kurigadzirisa, kuwedzera/kubvisa mabhuroka uye misoro, kupindura chenjedzo, uye kuona kuti Kafka kazhinji inoshanda nemazvo paKubernetes. Chinangwa chedu ndechekukubatsira kuti uise pfungwa pabasa guru ("gadzira" uye "kudya" mameseji eKafka), uye kusiya basa rese rakaoma kune Supertubes uye Kafka opareta.

Kana iwe uchifarira Banzai Cloud tekinoroji uye Open Source mapurojekiti, nyorera kukambani pa GitHub, LinkedIn kana Twitter.

PS kubva kumushanduri

Verenga zvakare pablog yedu:

• «Imwe nyaya ine Redis opareta muK8s uye mini-ongororo yezvishandiso zvekuongorora data kubva pane iyi database.";
• «Seamless kutama kweRabbitMQ kuenda Kubernetes";
• «zetcd kubva kuCoreOS: Kutsiva ZooKeeper ne...etcd kuchengetedza".

Source: www.habr.com