{"id":71213,"date":"2020-02-24T16:09:17","date_gmt":"2020-02-24T13:09:17","guid":{"rendered":"https:\/\/prohoster.info\/blog\/opredelyaem-podhodyashhij-razmer-dlya-klastera-kafka-v-kubernetes"},"modified":"2020-03-03T16:14:29","modified_gmt":"2020-03-03T13:14:29","slug":"opredelyaem-podhodyashhij-razmer-dlya-klastera-kafka-v-kubernetes","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/opredelyaem-podhodyashhij-razmer-dlya-klastera-kafka-v-kubernetes","title":{"rendered":"Determining the Appropriate Size for a Kafka Cluster in Kubernetes","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p><i><b>Note: translation.<\/b>: In this article, Banzai Cloud shares an example of using its special utilities to facilitate the management of Kafka within Kubernetes. The provided instructions illustrate how to determine the optimal size of infrastructure and configure Kafka itself to achieve the required throughput.<\/i><\/p>\n<p><img decoding=\"async\" alt=\"Determining the Appropriate Size for a Kafka Cluster in Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/8a598ec7db091c2944a6442f9bc124da.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nApache Kafka is a distributed streaming platform for building reliable, scalable, and high-performance real-time streaming systems. Its impressive capabilities can be extended with Kubernetes. To this end, we have developed <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/banzaicloud\/kafka-operator\">an Open Source Kafka operator<\/a><\/noindex> and a tool called <noindex><a rel=\"nofollow\" href=\"https:\/\/banzaicloud.com\/docs\/supertubes\/overview\/\">Supertubes<\/a><\/noindex>. They allow you to run Kafka in Kubernetes and utilize its various features, such as fine-tuning broker configuration, metric-based scaling with rebalancing, rack awareness, graceful <i>(graceful)<\/i> rolling updates, etc.<noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><\/p>\n<blockquote><p>Try Supertubes in your cluster:<\/p>\n<pre><code class=\"bash\">curl https:\/\/getsupertubes.sh | sh and supertubes install -a --no-democluster --kubeconfig<\/code><\/pre>\n<p>\nOr consult <noindex><a rel=\"nofollow\" href=\"https:\/\/banzaicloud.com\/docs\/supertubes\/cli\/reference\/\">the documentation<\/a><\/noindex>. You can also read about some of the Kafka features that are automated using Supertubes and the Kafka operator. We have already written about them in our blog:<\/p>\n<ul>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/banzaicloud.com\/blog\/kafka-operator\/\">Oh no! Yet another Kafka operator for Kubernetes<\/a><\/noindex>;<\/li>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/banzaicloud.com\/blog\/kafka-alert\/\">Monitor and operate Kafka based on Prometheus metrics<\/a><\/noindex>;<\/li>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/banzaicloud.com\/blog\/kafka-rack-awareness\/\">Kafka rack awareness on Kubernetes<\/a><\/noindex>;<\/li>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/banzaicloud.com\/blog\/kafka-on-istio-performance\/\">Running Apache Kafka over Istio \u2014 benchmark<\/a><\/noindex>;<\/li>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/banzaicloud.com\/blog\/kafka-topic-user-management\/\">User authenticated and access controlled clusters with the Kafka operator<\/a><\/noindex>;<\/li>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/banzaicloud.com\/blog\/kafka-rolling-upgrade\/\">Kafka rolling upgrade and dynamic configuration on Kubernetes<\/a><\/noindex>;<\/li>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/banzaicloud.com\/blog\/kafka-envoy-protocol-filter\/\">Envoy protocol filter for Kafka, meshed<\/a><\/noindex>.<\/li>\n<\/ul>\n<\/blockquote>\n<p>\nWhen deciding to deploy a Kafka cluster in Kubernetes, you will likely encounter the challenge of determining the optimal size of the underlying infrastructure and the need to fine-tune the Kafka configuration to meet throughput requirements. The maximum performance of each broker is determined by the performance of the underlying infrastructure components, such as memory, CPU, disk speed, network bandwidth, etc.<\/p>\n<p>Ideally, the broker configuration should be such that all infrastructure elements are utilized to their maximum potential. However, in real life, such a setup is quite complex. It is more likely that users will configure broker settings to maximize the usage of one or two components (disk, memory, or processor). Generally speaking, a broker demonstrates its peak performance when its configuration allows the slowest component to be fully engaged. This way, we can get an approximate understanding of the load that a single broker can handle.<\/p>\n<p>In theory, we can also estimate the number of brokers needed to handle a given load. However, in practice, the configuration options at various levels are so numerous that assessing the potential performance of a certain setup is quite challenging (if not impossible). In other words, it is very difficult to plan a configuration based on a given performance metric.<\/p>\n<p>For Supertubes users, we typically apply the following approach: we start with a certain configuration (infrastructure + settings), then measure its performance, adjust the broker settings, and repeat the process again. This continues until the potential of the slowest component of the infrastructure is fully utilized.<\/p>\n<p>In this way, we gain a clearer understanding of how many brokers the cluster needs to manage a specific load (the number of brokers also depends on other factors, such as the minimum number of message replicas for resilience, the number of partition leaders, and so on). Additionally, we get insights into which infrastructure component would benefit from vertical scaling.<\/p>\n<p>This article will discuss the steps we take to 'squeeze everything' out of the slowest components in initial configurations and measure the throughput of the Kafka cluster. A highly resilient configuration requires at least three operational brokers (<code>min.insync.replicas=3<\/code>), distributed across three different availability zones. To configure, scale, and monitor the Kubernetes infrastructure, we use our own container management platform for hybrid clouds \u2014 <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/banzaicloud\/pipeline\">Pipeline<\/a><\/noindex>. It supports on-premise (bare metal, VMware) and five types of clouds (Alibaba, AWS, Azure, Google, Oracle), as well as any combinations of them.<\/p>\n<h2>Thoughts on the infrastructure and configuration of the Kafka cluster<\/h2>\n<p>\nFor the examples provided below, we chose AWS as the cloud service provider and EKS as the Kubernetes distribution. A similar configuration can be implemented using <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/banzaicloud\/pke\">PKE<\/a><\/noindex> \u2014 a Kubernetes distribution from Banzai Cloud, certified by CNCF.<\/p>\n<h3>Disk<\/h3>\n<p>\nAmazon offers various <noindex><a rel=\"nofollow\" href=\"https:\/\/docs.aws.amazon.com\/AWSEC2\/latest\/UserGuide\/ebs-volume-types.html\">types of EBS volumes<\/a><\/noindex>. At the core of <i>gp2<\/i> and <i>io1<\/i> are SSDs, however, to ensure high throughput, <i>gp2<\/i> it consumes accrued credits <i>(I\/O credits)<\/i>, which is why we preferred the type <i>io1<\/i>, that offers stable high throughput.<\/p>\n<h3>Instance types<\/h3>\n<p>\nKafka performance is heavily dependent on the operating system's page cache, so we need instances with sufficient memory for brokers (JVM) and page cache. The instance <i>c5.2xlarge<\/i> is a decent start since it has 16 GB of memory and <noindex><a rel=\"nofollow\" href=\"https:\/\/docs.aws.amazon.com\/AWSEC2\/latest\/UserGuide\/ebs-optimized.html\">is optimized for EBS<\/a><\/noindex>. Its drawback is that it can provide maximum performance for no more than 30 minutes every 24 hours. If the workload requires maximum performance for a longer period, other instance types should be considered. That's exactly what we did, settling on <i>c5.4xlarge<\/i>. It provides maximum throughput of <b>593.75 MB\/s<\/b>. The maximum throughput of an EBS volume <i>io1<\/i> is higher than that of the instance <i>c5.4xlarge<\/i>, so the slowest component of the infrastructure is presumably the I\/O throughput of this instance type (which should also be confirmed by the results of our load tests).<\/p>\n<h3>Network<\/h3>\n<p>\nNetwork bandwidth should be sufficiently large compared to the performance of the VM instance and disk; otherwise, the network becomes a bottleneck. In our case, the network interface <i>c5.4xlarge<\/i> supports speeds up to 10 Gb\/s, which is significantly higher than the I\/O throughput of the VM instance.<\/p>\n<h3>Deployment of brokers<\/h3>\n<p>\nBrokers should be deployed (planned in Kubernetes) on dedicated nodes to avoid competing with other processes for CPU, memory, network, and disk resources.<\/p>\n<h3>Java Version<\/h3>\n<p>\nJava 11 is a logical choice as it is compatible with Docker in that the JVM correctly identifies the processors and memory available to the container in which the broker runs. Knowing that CPU limits are important, the JVM internally and transparently sets the number of GC threads and JIT compiler threads. We used the Kafka image <code>banzaicloud\/kafka:2.13-2.4.0<\/code>, which includes Kafka version 2.4.0 (Scala 2.13) on Java 11.<\/p>\n<blockquote><p>If you want to learn more about Java\/JVM on Kubernetes, check out the following publications:<\/p>\n<ul>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/banzaicloud.com\/blog\/java-resource-limits\/\">Why my Java application is OOMKilled<\/a><\/noindex>;<\/li>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/banzaicloud.com\/blog\/java10-container-sizing\/\">How to correctly size containers for Java 10 applications<\/a><\/noindex>.<\/li>\n<\/ul>\n<\/blockquote>\n<p><\/p>\n<h3>Broker memory settings<\/h3>\n<p>\nThere are two key aspects in configuring the broker's memory: settings for the JVM and for the Kubernetes pod. The memory limit set for the pod must be greater than the maximum heap size, so that there is room for Java's metaspace, which resides in its own memory, and for the operating system's page cache that Kafka actively uses. In our tests, we ran Kafka brokers with parameters <code>-Xmx4G -Xms2G<\/code>, while the memory limit for the pod was <code>10 Gi<\/code>. Note that JVM memory settings can be obtained automatically using <code>-XX:MaxRAMPercentage<\/code> and <code>-X:MinRAMPercentage<\/code>, based on the memory limit for the pod.<\/p>\n<h3>Broker CPU settings<\/h3>\n<p>\nIn general, performance can be improved by increasing parallelism through an increase in the number of threads used by Kafka. The more processors available to Kafka, the better. In our test, we started with a limit of 6 processors and gradually (in iterations) increased that number to 15. Additionally, we set <code>num.network.threads=12<\/code> in the broker settings to increase the number of threads handling incoming network data and sending it out. Immediately noticing that follower brokers could not fetch replicas quickly enough, we raised <code>num.replica.fetchers<\/code> to 4 to increase the speed at which follower brokers replicated messages from leaders.<\/p>\n<h3>Load generation tool<\/h3>\n<p>\nIt's important to ensure that the capacity of the chosen load generator does not run out before the Kafka cluster (which is being benchmarked) reaches its maximum load. In other words, a preliminary assessment of the load generation tool's capabilities should be conducted, as well as selecting instance types with sufficient processors and memory. In this case, our tool will produce more load than the Kafka cluster can handle. After numerous experiments, we settled on three instances <i>c5.4xlarge<\/i>, each of which ran the generator.<\/p>\n<h2>Benchmarking<\/h2>\n<p>\nPerformance measurement is an iterative process that includes the following stages:<\/p>\n<ul>\n<li> setting up the infrastructure (EKS cluster, Kafka cluster, load generation tool, as well as Prometheus and Grafana);<\/li>\n<li> generating load over a specified period to filter out random fluctuations in the collected performance metrics;<\/li>\n<li> adjusting the infrastructure and broker configuration based on observed performance metrics;<\/li>\n<li> repeating the process until the required throughput level of the Kafka cluster is achieved. During this, it must be consistently reproducible and demonstrate minimal variations in throughput.<\/li>\n<\/ul>\n<p>\nThe following section describes the steps performed during the benchmarking of the test cluster.<\/p>\n<h3>Tools<\/h3>\n<p>\nTo quickly deploy the basic configuration, generate load, and measure performance, the following tools were used:<\/p>\n<ul>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/banzaicloud.com\/docs\/pipeline\/overview\/\">Banzai Cloud Pipeline<\/a><\/noindex> to organize the EKS cluster from Amazon with <noindex><a rel=\"nofollow\" href=\"https:\/\/prometheus.io\/\">Prometheus<\/a><\/noindex> (for collecting Kafka and infrastructure metrics) and <noindex><a rel=\"nofollow\" href=\"https:\/\/grafana.com\/\">Grafana<\/a><\/noindex> (for visualizing these metrics). We utilized <b>integrated<\/b> downward API support (simultaneously with this in <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/banzaicloud\/pipeline\">Pipeline<\/a><\/noindex> services that provide federated monitoring, centralized log collection, vulnerability scanning, disaster recovery, enterprise-grade security, and much more.<\/li>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/jamiealquiza\/sangrenel\">Sangrenel<\/a><\/noindex> is a load testing tool for the Kafka cluster.<\/li>\n<li> Grafana panels for visualizing Kafka and infrastructure metrics: <noindex><a rel=\"nofollow\" href=\"https:\/\/grafana.com\/grafana\/dashboards\/10123\">Kubernetes Kafka<\/a><\/noindex>, <noindex><a rel=\"nofollow\" href=\"https:\/\/grafana.com\/grafana\/dashboards\/1860\">Node Exporter<\/a><\/noindex>.<\/li>\n<li> Supertubes CLI for easily configuring a Kafka cluster in Kubernetes. Zookeeper, Kafka operator, Envoy, and many other components are installed and properly configured to run a production-ready Kafka cluster in Kubernetes.\n<ul>\n<li> To install <i>supertubes CLI<\/i> follow the instructions provided <noindex><a rel=\"nofollow\" href=\"https:\/\/banzaicloud.com\/docs\/supertubes\/cli\/install\/\">here<\/a><\/noindex>.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p>\n<img decoding=\"async\" alt=\"Determining the Appropriate Size for a Kafka Cluster in Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/23ecf423b1ea66813406943ae7eabf61.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<\/p>\n<h3>EKS Cluster<\/h3>\n<p>\nPrepare the EKS cluster with dedicated worker nodes <i>c5.4xlarge<\/i> in different availability zones for the Kafka broker pods, as well as dedicated nodes for the load generator and monitoring infrastructure.<\/p>\n<pre><code class=\"bash\">banzai cluster create -f https:\/\/raw.githubusercontent.com\/banzaicloud\/kafka-operator\/master\/docs\/benchmarks\/infrastructure\/cluster_eks_202001.json<\/code><\/pre>\n<p>\nOnce the EKS cluster is up and running, enable its integrated <noindex><a rel=\"nofollow\" href=\"https:\/\/banzaicloud.com\/docs\/pipeline\/features\/integrated-services\/\">monitoring service<\/a><\/noindex> \u2014 it will deploy Prometheus and Grafana into the cluster.<\/p>\n<h3>Kafka System Components<\/h3>\n<p>\nInstall Kafka system components (Zookeeper, kafka-operator) in EKS using the supertubes CLI:<\/p>\n<pre><code class=\"bash\">supertubes install -a --no-democluster --kubeconfig<\/code><\/pre>\n<p><\/p>\n<h3>Kafka Cluster<\/h3>\n<p>\nBy default, EKS uses EBS volumes of type <i>gp2<\/i>, so a separate storage class based on volumes needs to be created <i>io1<\/i> for the Kafka cluster:<\/p>\n<pre><code class=\"plaintext\">kubectl create -f - &lt;&lt;EOF\napiVersion: storage.k8s.io\/v1\nkind: StorageClass\nmetadata:\n  name: fast-ssd\nprovisioner: kubernetes.io\/aws-ebs\nparameters:\n  type: io1\n  iopsPerGB: &quot;50&quot;\n  fsType: ext4\nvolumeBindingMode: WaitForFirstConsumer\nEOF<\/code><\/pre>\n<p>\nSet the parameter for brokers <code>min.insync.replicas=3<\/code> and deploy the broker pods on nodes in three different availability zones:<\/p>\n<pre><code class=\"bash\">supertubes cluster create -n kafka --kubeconfig  -f https:\/\/raw.githubusercontent.com\/banzaicloud\/kafka-operator\/master\/docs\/benchmarks\/infrastructure\/kafka_202001_3brokers.yaml --wait --timeout 600<\/code><\/pre>\n<p><\/p>\n<h3>Topics<\/h3>\n<p>\nWe ran three instances of the load generator in parallel. Each writes to its topic, so we need a total of three topics:<\/p>\n<pre><code class=\"bash\">supertubes cluster topic create -n kafka --kubeconfig  -f -&lt;&lt;EOF\napiVersion: kafka.banzaicloud.io\/v1alpha1\nkind: KafkaTopic\nmetadata:\n  name: perftest1\nspec:\n  name: perftest1\n  partitions: 12\n  replicationFactor: 3\n  retention.ms: &#039;28800000&#039;\n  cleanup.policy: delete\nEOF\n\nsupertubes cluster topic create -n kafka --kubeconfig  -f -&lt;&lt;EOF\napiVersion: kafka.banzaicloud.io\/v1alpha1\nkind: KafkaTopic\nmetadata:\n    name: perftest2\nspec:\n  name: perftest2\n  partitions: 12\n  replicationFactor: 3\n  retention.ms: &#039;28800000&#039;\n  cleanup.policy: delete\nEOF\n\nsupertubes cluster topic create -n kafka --kubeconfig  -f -&lt;&lt;EOF\napiVersion: kafka.banzaicloud.io\/v1alpha1\nkind: KafkaTopic\nmetadata:\n  name: perftest3\nspec:\n  name: perftest3\n  partitions: 12\n  replicationFactor: 3\n  retention.ms: &#039;28800000&#039;\n  cleanup.policy: delete\nEOF<\/code><\/pre>\n<p>\nFor each topic, the replication factor is 3 \u2014 the minimum recommended value for high-availability production systems.<\/p>\n<h3>Load generation tool<\/h3>\n<p>\nWe launched three instances of the load generator (each writing to a separate topic). For the load generator pods, node affinity needs to be specified so that they are scheduled only on the dedicated nodes for them:<\/p>\n<pre><code class=\"plaintext\">apiVersion: extensions\/v1beta1\nkind: Deployment\nmetadata:\n  labels:\n    app: loadtest\n  name: perf-load1\n  namespace: kafka\nspec:\n  progressDeadlineSeconds: 600\n  replicas: 1\n  revisionHistoryLimit: 10\n  selector:\n    matchLabels:\n      app: loadtest\n  strategy:\n    rollingUpdate:\n      maxSurge: 25%\n      maxUnavailable: 25%\n    type: RollingUpdate\n  template:\n    metadata:\n      creationTimestamp: null\n      labels:\n        app: loadtest\n    spec:\n      affinity:\n        nodeAffinity:\n          requiredDuringSchedulingIgnoredDuringExecution:\n            nodeSelectorTerms:\n            - matchExpressions:\n              - key: nodepool.banzaicloud.io\/name\n                operator: In\n                values:\n                - loadgen\n      containers:\n      - args:\n        - -brokers=kafka-0:29092,kafka-1:29092,kafka-2:29092,kafka-3:29092\n        - -topic=perftest1\n        - -required-acks=all\n        - -message-size=512\n        - -workers=20\n        image: banzaicloud\/perfload:0.1.0-blog\n        imagePullPolicy: Always\n        name: sangrenel\n        resources:\n          limits:\n            cpu: 2\n            memory: 1Gi\n          requests:\n            cpu: 2\n            memory: 1Gi\n        terminationMessagePath: \/dev\/termination-log\n        terminationMessagePolicy: File\n      dnsPolicy: ClusterFirst\n      restartPolicy: Always\n      schedulerName: default-scheduler\n      securityContext: {}\n      terminationGracePeriodSeconds: 30<\/code><\/pre>\n<p>\nSeveral points to consider:<\/p>\n<ul>\n<li> The load generator generates messages of 512 bytes each and publishes them to Kafka in batches of 500 messages.<\/li>\n<li> With the argument <code>-required-acks=all<\/code> publication is considered successful when all synchronized replicas of the message are received and acknowledged by the Kafka brokers. This means that in the benchmark we measured not only the speed of the leaders receiving messages, but also their followers replicating the messages. The task of this test does not include evaluating the read speed of consumers <i>(consumers)<\/i> of recently received messages that remain in the OS page cache, and comparing it with the read speed of messages stored on disk.<\/li>\n<li> The load generator runs 20 workers in parallel (<code>-workers=20<\/code>). Each worker contains 5 producers that share the worker's connection to the Kafka cluster. As a result, each generator counts 100 producers, and they all send messages to the Kafka cluster.<\/li>\n<\/ul>\n<p><\/p>\n<h3>Monitoring the status of the cluster<\/h3>\n<p>\nDuring the load testing of the Kafka cluster, we also monitored its health to ensure there were no pod restarts, out-of-sync replicas, and maximum throughput with minimal fluctuations:<\/p>\n<ul>\n<li> The load generator records standard statistics on the number of published messages and the error rate. The error rate should remain at <code>0,00%<\/code>.<\/li>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/linkedin\/cruise-control\">Cruise Control<\/a><\/noindex>, deployed by the kafka operator, provides a monitoring panel where we can also observe the state of the cluster. To view this panel, execute:\n<pre><code class=\"bash\">supertubes cluster cruisecontrol show -n kafka --kubeconfig<\/code><\/pre>\n<\/li>\n<li> ISR level <i>(number of \"in-sync\" replicas)<\/i> shrink and expansion equals 0.<\/li>\n<\/ul>\n<p><\/p>\n<h2>Measurement results<\/h2>\n<p><\/p>\n<h3>3 brokers, message size \u2014 512 bytes<\/h3>\n<p>\nWith partitions evenly distributed across three brokers, we managed to achieve performance <i>~500 Mb\/s (approximately 990 thousand messages per second)<\/i>:<\/p>\n<p><img decoding=\"async\" alt=\"Determining the Appropriate Size for a Kafka Cluster in Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/df65ff6a1a26e28d5e5a7a5ab1a20d98.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"Determining the Appropriate Size for a Kafka Cluster in Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/15c5883c0056c651f0a0bd14e9991967.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"Determining the Appropriate Size for a Kafka Cluster in Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/9b708fa5ce809b0d059cba9f5edfbb2f.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nThe JVM virtual machine's memory consumption did not exceed 2 GB:<\/p>\n<p><img decoding=\"async\" alt=\"Determining the Appropriate Size for a Kafka Cluster in Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/3320e2eea891055cc0935da492d18ee9.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"Determining the Appropriate Size for a Kafka Cluster in Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/6e3976c310309aa8be45fa3fdf547750.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"Determining the Appropriate Size for a Kafka Cluster in Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/94baa1b689ae854693fb7cc4ed6e105e.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nDisk throughput reached the maximum I\/O capacity of the node on all three instances where the brokers were running:<\/p>\n<p><img decoding=\"async\" alt=\"Determining the Appropriate Size for a Kafka Cluster in Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/408139ad33cda1294755660df1c128a8.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"Determining the Appropriate Size for a Kafka Cluster in Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/391d2df0ddfe82f94abdb200370ff775.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"Determining the Appropriate Size for a Kafka Cluster in Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/e7e9fd021a2cdc71992ce672363a78ec.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nFrom the node memory usage data, it appears that system buffering and caching took ~10-15 GB:<\/p>\n<p><img decoding=\"async\" alt=\"Determining the Appropriate Size for a Kafka Cluster in Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/781887507e01a4903b86871961ff9260.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"Determining the Appropriate Size for a Kafka Cluster in Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/63f10177f01342039f04192ea93c8bca.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"Determining the Appropriate Size for a Kafka Cluster in Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/48b992cc73a968b3d3b1442f96562c4e.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<\/p>\n<h3>3 brokers, message size \u2014 100 bytes<\/h3>\n<p>\nWith a reduction in message size, throughput decreases by approximately 15-20%: this is affected by the time spent processing each message. Additionally, CPU load nearly doubled.<\/p>\n<p><img decoding=\"async\" alt=\"Determining the Appropriate Size for a Kafka Cluster in Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/2f7dc5a94da371cfe17c9e0fc0ec1941.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"Determining the Appropriate Size for a Kafka Cluster in Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/986160a3a4e099a0dbb8aeefaf6f1c2f.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"Determining the Appropriate Size for a Kafka Cluster in Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/c08244457bf4dbfadf52838d4ea2969e.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nSince there are still unused cores on the broker nodes, performance can be improved by adjusting Kafka's configuration. This is a tricky task, so to increase throughput, it's better to work with larger messages.<\/p>\n<h3>4 brokers, message size \u2014 512 bytes<\/h3>\n<p>\nIt is easy to increase the performance of the Kafka cluster simply by adding new brokers and maintaining balance of partitions (this ensures an even load distribution among brokers). In our case, after adding a broker, the cluster's bandwidth increased to <i>~580 Mb\/s (~1.1 million messages per second)<\/i>. The growth was less than expected; this is primarily explained by the imbalance of partitions (not all brokers are operating at peak capacity).<\/p>\n<p><img decoding=\"async\" alt=\"Determining the Appropriate Size for a Kafka Cluster in Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/8c8d2e70739add3c3f88fe42acf98c46.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"Determining the Appropriate Size for a Kafka Cluster in Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/a6e4812270c04487de0ad39b996a13f8.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"Determining the Appropriate Size for a Kafka Cluster in Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/6162c04cfb87b9d9cd0f3147a4ae1647.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"Determining the Appropriate Size for a Kafka Cluster in Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/dae5de3b14be212c679f3627a3b77d7b.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nThe memory usage of the JVM machine remained below 2 GB:<\/p>\n<p><img decoding=\"async\" alt=\"Determining the Appropriate Size for a Kafka Cluster in Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/0bca07a817d7da3104b08de10979f049.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"Determining the Appropriate Size for a Kafka Cluster in Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/7e1e64f80ec140f13e171cfe3be930ce.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"Determining the Appropriate Size for a Kafka Cluster in Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/5e9700b1e54e45f9cd5fa018ec07dba1.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"Determining the Appropriate Size for a Kafka Cluster in Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/4ba0c4b59bfe72bcb0d2f22056c47d79.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nThe operation of the brokers with storage was affected by the imbalance of partitions:<\/p>\n<p><img decoding=\"async\" alt=\"Determining the Appropriate Size for a Kafka Cluster in Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/2b8ed3c21adf70c751d1daa6d4c41895.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"Determining the Appropriate Size for a Kafka Cluster in Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/77b980869b99903d9de5cd7d43d79291.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"Determining the Appropriate Size for a Kafka Cluster in Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/f2f9569a3a1ef0ce35e89b2e52e4a8b2.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"Determining the Appropriate Size for a Kafka Cluster in Kubernetes\" src=\"\/wp-content\/uploads\/2020\/02\/191d6d97b32b2128c5cfd3914171de15.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<\/p>\n<h2>Conclusions<\/h2>\n<p>\nThe iterative approach presented above can be expanded to cover more complex scenarios involving hundreds of consumers, repartitioning, rolling updates, pod restarts, etc. All of this allows us to assess the limits of the Kafka cluster's capabilities under various conditions, identify bottlenecks in its operation, and find ways to address them.<\/p>\n<p>We developed Supertubes for rapid and easy deployment of the cluster, its configuration, adding\/removing brokers and topics, responding to alerts, and ensuring the proper operation of Kafka in Kubernetes overall. Our goal is to help you focus on the main task (to 'produce' and 'consume' Kafka messages), while leaving all the heavy lifting to Supertubes and the Kafka operator.<\/p>\n<p>If you are interested in the technologies and open-source projects by Banzai Cloud, follow the company on <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/banzaicloud\">GitHub<\/a><\/noindex>, <noindex><a rel=\"nofollow\" href=\"https:\/\/www.linkedin.com\/company\/banzaicloud\">LinkedIn<\/a><\/noindex> or <noindex><a rel=\"nofollow\" href=\"https:\/\/twitter.com\/BanzaiCloud\">Twitter<\/a><\/noindex>.<\/p>\n<h2>P.S. from the translator<\/h2>\n<p>\nAlso read in our blog:<\/p>\n<ul>\n<li> \u00ab<noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/480722\/\">A Story about the Redis Operator in K8s and a Brief Overview of Utilities for Analyzing Data from This Database<\/a><\/noindex>\u00bb;<\/li>\n<li> \u00ab<noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/450662\/\">Challenging migration of RabbitMQ to Kubernetes<\/a><\/noindex>\u00bb;<\/li>\n<li> \u00ab<noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/329224\/\">zetcd from CoreOS: Replacing ZooKeeper with... etcd storage<\/a><\/noindex>\u00bb.<\/li>\n<\/ul>\n<p>Source: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/488920\/\">habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u041f\u0440\u0438\u043c. \u043f\u0435\u0440\u0435\u0432.: \u0412 \u044d\u0442\u043e\u0439 \u0441\u0442\u0430\u0442\u044c\u0435 \u043a\u043e\u043c\u043f\u0430\u043d\u0438\u044f Banzai Cloud \u0434\u0435\u043b\u0438\u0442\u0441\u044f \u043f\u0440\u0438\u043c\u0435\u0440\u043e\u043c \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u043d\u0438\u044f \u0435\u0451 \u0441\u043f\u0435\u0446\u0438\u0430\u043b\u044c\u043d\u044b\u0445 \u0443\u0442\u0438\u043b\u0438\u0442 \u0434\u043b\u044f \u043e\u0431\u043b\u0435\u0433\u0447\u0435\u043d\u0438\u044f \u044d\u043a\u0441\u043f\u043b\u0443\u0430\u0442\u0430\u0446\u0438\u0438 Kafka \u0432 \u0440\u0430\u043c\u043a\u0430\u0445 Kubernetes. \u041f\u0440\u0438\u0432\u043e\u0434\u0438\u043c\u044b\u0435 \u0438\u043d\u0441\u0442\u0440\u0443\u043a\u0446\u0438\u0438 \u0438\u043b\u043b\u044e\u0441\u0442\u0440\u0438\u0440\u0443\u044e\u0442, \u043a\u0430\u043a \u043c\u043e\u0436\u043d\u043e \u043e\u043f\u0440\u0435\u0434\u0435\u043b\u0438\u0442\u044c \u043e\u043f\u0442\u0438\u043c\u0430\u043b\u044c\u043d\u044b\u0439 \u0440\u0430\u0437\u043c\u0435\u0440 \u0438\u043d\u0444\u0440\u0430\u0441\u0442\u0440\u0443\u043a\u0442\u0443\u0440\u044b \u0438 \u043d\u0430\u0441\u0442\u0440\u043e\u0438\u0442\u044c \u0441\u0430\u043c\u0443 Kafka \u0434\u043b\u044f \u0434\u043e\u0441\u0442\u0438\u0436\u0435\u043d\u0438\u044f \u0442\u0440\u0435\u0431\u0443\u0435\u043c\u043e\u0439 \u043f\u0440\u043e\u043f\u0443\u0441\u043a\u043d\u043e\u0439 \u0441\u043f\u043e\u0441\u043e\u0431\u043d\u043e\u0441\u0442\u0438. Apache Kafka \u2014 \u0440\u0430\u0441\u043f\u0440\u0435\u0434\u0435\u043b\u0451\u043d\u043d\u0430\u044f \u0441\u0442\u0440\u0438\u043c\u0438\u043d\u0433\u043e\u0432\u0430\u044f \u043f\u043b\u0430\u0442\u0444\u043e\u0440\u043c\u0430 \u0434\u043b\u044f \u0441\u043e\u0437\u0434\u0430\u043d\u0438\u044f \u043d\u0430\u0434\u0451\u0436\u043d\u044b\u0445, \u043c\u0430\u0441\u0448\u0442\u0430\u0431\u0438\u0440\u0443\u0435\u043c\u044b\u0445 \u0438 \u0432\u044b\u0441\u043e\u043a\u043e\u043f\u0440\u043e\u0438\u0437\u0432\u043e\u0434\u0438\u0442\u0435\u043b\u044c\u043d\u044b\u0445 \u043f\u043e\u0442\u043e\u043a\u043e\u0432\u044b\u0445 \u0441\u0438\u0441\u0442\u0435\u043c \u0440\u0435\u0430\u043b\u044c\u043d\u043e\u0433\u043e \u0432\u0440\u0435\u043c\u0435\u043d\u0438. [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":71214,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-71213","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.2 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u041f\u0440\u0438\u043c. \u043f\u0435\u0440\u0435\u0432.: \u0412 \u044d\u0442\u043e\u0439 \u0441\u0442\u0430\u0442\u044c\u0435 \u043a\u043e\u043c\u043f\u0430\u043d\u0438\u044f Banzai Cloud \u0434\u0435\u043b\u0438\u0442\u0441\u044f \u043f\u0440\u0438\u043c\u0435\u0440\u043e\u043c \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u043d\u0438\u044f \u0435\u0451 \u0441\u043f\u0435\u0446\u0438\u0430\u043b\u044c\u043d\u044b\u0445 \u0443\u0442\u0438\u043b\u0438\u0442 \u0434\u043b\u044f \u043e\u0431\u043b\u0435\u0433\u0447\u0435\u043d\u0438\u044f \u044d\u043a\u0441\u043f\u043b\u0443\u0430\u0442\u0430\u0446\u0438\u0438 Kafka \u0432 \u0440\u0430\u043c\u043a\u0430\u0445 Kubernetes.\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/opredelyaem-podhodyashhij-razmer-dlya-klastera-kafka-v-kubernetes\" \/>\n\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.2\" \/>\n\t\t<meta property=\"og:locale\" content=\"en_US\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47\u041e\u043f\u0440\u0435\u0434\u0435\u043b\u044f\u0435\u043c \u043f\u043e\u0434\u0445\u043e\u0434\u044f\u0449\u0438\u0439 \u0440\u0430\u0437\u043c\u0435\u0440 \u0434\u043b\u044f \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0430 Kafka \u0432 Kubernetes | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u041f\u0440\u0438\u043c. \u043f\u0435\u0440\u0435\u0432.: \u0412 \u044d\u0442\u043e\u0439 \u0441\u0442\u0430\u0442\u044c\u0435 \u043a\u043e\u043c\u043f\u0430\u043d\u0438\u044f Banzai Cloud \u0434\u0435\u043b\u0438\u0442\u0441\u044f \u043f\u0440\u0438\u043c\u0435\u0440\u043e\u043c \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u043d\u0438\u044f \u0435\u0451 \u0441\u043f\u0435\u0446\u0438\u0430\u043b\u044c\u043d\u044b\u0445 \u0443\u0442\u0438\u043b\u0438\u0442 \u0434\u043b\u044f \u043e\u0431\u043b\u0435\u0433\u0447\u0435\u043d\u0438\u044f \u044d\u043a\u0441\u043f\u043b\u0443\u0430\u0442\u0430\u0446\u0438\u0438 Kafka \u0432 \u0440\u0430\u043c\u043a\u0430\u0445 Kubernetes.\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/opredelyaem-podhodyashhij-razmer-dlya-klastera-kafka-v-kubernetes\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2020-02-24T13:09:17+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-03-03T13:14:29+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47Determining the appropriate size for a Kafka cluster in Kubernetes | ProHoster","description":"Note from the translator: In this article, Banzai Cloud shares an example of using its special utilities to simplify the operation of Kafka within Kubernetes.","canonical_url":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/opredelyaem-podhodyashhij-razmer-dlya-klastera-kafka-v-kubernetes","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"en_US","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u041e\u043f\u0440\u0435\u0434\u0435\u043b\u044f\u0435\u043c \u043f\u043e\u0434\u0445\u043e\u0434\u044f\u0449\u0438\u0439 \u0440\u0430\u0437\u043c\u0435\u0440 \u0434\u043b\u044f \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0430 Kafka \u0432 Kubernetes | ProHoster","og:description":"\u041f\u0440\u0438\u043c. \u043f\u0435\u0440\u0435\u0432.: \u0412 \u044d\u0442\u043e\u0439 \u0441\u0442\u0430\u0442\u044c\u0435 \u043a\u043e\u043c\u043f\u0430\u043d\u0438\u044f Banzai Cloud \u0434\u0435\u043b\u0438\u0442\u0441\u044f \u043f\u0440\u0438\u043c\u0435\u0440\u043e\u043c \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u043d\u0438\u044f \u0435\u0451 \u0441\u043f\u0435\u0446\u0438\u0430\u043b\u044c\u043d\u044b\u0445 \u0443\u0442\u0438\u043b\u0438\u0442 \u0434\u043b\u044f \u043e\u0431\u043b\u0435\u0433\u0447\u0435\u043d\u0438\u044f \u044d\u043a\u0441\u043f\u043b\u0443\u0430\u0442\u0430\u0446\u0438\u0438 Kafka \u0432 \u0440\u0430\u043c\u043a\u0430\u0445 Kubernetes.","og:url":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/opredelyaem-podhodyashhij-razmer-dlya-klastera-kafka-v-kubernetes","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-02-24T13:09:17+00:00","article:modified_time":"2020-03-03T13:14:29+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"71213","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 19:06:22","updated":"2022-10-04 20:58:00","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/71213","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/comments?post=71213"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/71213\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/media\/71214"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/media?parent=71213"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/categories?post=71213"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/tags?post=71213"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}