{"id":34366,"date":"2019-10-31T21:57:56","date_gmt":"2019-10-31T18:57:56","guid":{"rendered":"https:\/\/prohoster.info\/blog\/benchmark-potrebleniya-tsp-dlya-istio-i-linkerd\/"},"modified":"2019-10-31T21:57:56","modified_gmt":"2019-10-31T18:57:56","slug":"benchmark-potrebleniya-tsp-dlya-istio-i-linkerd","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/benchmark-potrebleniya-tsp-dlya-istio-i-linkerd","title":{"rendered":"CPU Consumption Benchmark for Istio and Linkerd","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

\"CPU<\/p>\n

<\/p>\n

Introduction<\/h2>\n

<\/p>\n

We at Shopify<\/a><\/noindex> we focused on deploying Istio as a service mesh. Generally, everything is fine except for one thing: it's expensive<\/strong>.<\/p>\n

<\/p>\n

In published benchmarks<\/a><\/noindex> it is stated for Istio:<\/p>\n

<\/p>\n

With Istio 1.1, the proxy consumes approximately 0.6 vCPU (virtual cores) per 1000 requests per second.<\/p><\/blockquote>\n

For the first region in the service mesh (2 proxies on each side of the connection), we will need 1200 cores just for the proxies, based on one million requests per second. According to Google's cost calculator, that amounts to about $40\/month\/core for the configuration n1-standard-64<\/code>, which means this single region will cost us over 50 thousand dollars a month for 1 million requests per second.<\/p>\n

<\/p>\n

Ivan Sim (Ivan Sim<\/a><\/noindex>) visually compared<\/a><\/noindex> service mesh latencies from last year and promised the same for memory and CPU, but it didn't work out:<\/p>\n

<\/p>\n

Apparently, values-istio-test.yaml will significantly increase CPU requests. If I calculated correctly, we need about 24 CPU cores for the control panel and 0.5 CPU for each proxy. I don't have that many. I will repeat the tests when I am allocated more resources.<\/p><\/blockquote>\n

I wanted to see for myself how Istio's metrics compare to another open-source service mesh: Linkerd<\/a><\/noindex>.<\/p>\n

<\/a><\/noindex><\/p>\n

Installing the service mesh<\/h3>\n

<\/p>\n

First, I installed in the cluster SuperGloo<\/a><\/noindex>:<\/p>\n

<\/p>\n

$ supergloo init\ninstalling supergloo version 0.3.12\nusing chart uri https:\/\/storage.googleapis.com\/supergloo-helm\/charts\/supergloo-0.3.12.tgz\nconfigmap\/sidecar-injection-resources created\nserviceaccount\/supergloo created\nserviceaccount\/discovery created\nserviceaccount\/mesh-discovery created\nclusterrole.rbac.authorization.k8s.io\/discovery created\nclusterrole.rbac.authorization.k8s.io\/mesh-discovery created\nclusterrolebinding.rbac.authorization.k8s.io\/supergloo-role-binding created\nclusterrolebinding.rbac.authorization.k8s.io\/discovery-role-binding created\nclusterrolebinding.rbac.authorization.k8s.io\/mesh-discovery-role-binding created\ndeployment.extensions\/supergloo created\ndeployment.extensions\/discovery created\ndeployment.extensions\/mesh-discovery created\ninstall successful!<\/code><\/pre>\n
<\/p>\n
I used SuperGloo because it significantly simplifies the initial setup of a service mesh. I hardly had to do anything. In production, we do not use SuperGloo, but it is perfect for such tasks. I had to apply just a couple of commands for each service mesh. I used two clusters for isolation \u2014 one each for Istio and Linkerd.<\/p>\n
<\/p>\n
The experiment was conducted on Google Kubernetes Engine. I used Kubernetes 1.12.7-gke.7<\/code> and a node pool n1-standard-4<\/code> with automatic node scaling (minimum 4, maximum 16).<\/p>\n
<\/p>\n
Then I installed both service meshes from the command line.<\/p>\n
<\/p>\n
First, Linkerd:<\/p>\n
<\/p>\n
$ supergloo install linkerd --name linkerd\n+---------+--------------+---------+---------------------------+\n| INSTALL |     TYPE     | STATUS  |          DETAILS          |\n+---------+--------------+---------+---------------------------+\n| linkerd | Linkerd Mesh | Pending | enabled: true             |\n|         |              |         | version: stable-2.3.0     |\n|         |              |         | namespace: linkerd        |\n|         |              |         | mtls enabled: true        |\n|         |              |         | auto inject enabled: true |\n+---------+--------------+---------+---------------------------+<\/code><\/pre>\n
<\/p>\n
Next is Istio:<\/p>\n
<\/p>\n
$ supergloo install istio --name istio --installation-namespace istio-system --mtls=true --auto-inject=true\n+---------+------------+---------+---------------------------+\n| INSTALL |    TYPE    | STATUS  |          DETAILS          |\n+---------+------------+---------+---------------------------+\n| istio   | Istio Mesh | Pending | enabled: true             |\n|         |            |         | version: 1.0.6            |\n|         |            |         | namespace: istio-system   |\n|         |            |         | mtls enabled: true        |\n|         |            |         | auto inject enabled: true |\n|         |            |         | grafana enabled: true     |\n|         |            |         | prometheus enabled: true  |\n|         |            |         | jaeger enabled: true      |\n+---------+------------+---------+---------------------------+<\/code><\/pre>\n
<\/p>\n
The crash-loop took several minutes, and then the control panels stabilized.<\/p>\n
<\/p>\n
(Note: SuperGloo currently supports only Istio 1.0.x. I repeated the experiment with Istio 1.1.3, but I did not notice any significant difference.)<\/em><\/p>\n
<\/p>\n
Configuring Automatic Istio Injection<\/h3>\n
<\/p>\n
To have Istio install the Envoy sidecar, we use the sidecar injector \u2014 MutatingAdmissionWebhook<\/code>. We won't discuss it in this article. I'll just say that it's a controller that monitors access for all new pods and dynamically adds the sidecar and initContainer responsible for tasks. iptables<\/code>.<\/p>\n
<\/p>\n
At Shopify, we wrote our own access controller for injecting sidecars, but in this benchmark, I took the controller that comes with Istio. The default controller injects sidecars when there's a label in the namespace. istio-injection: enabled<\/code>:<\/p>\n
<\/p>\n
$ kubectl label namespace irs-client-dev istio-injection=enabled\nnamespace\/irs-client-dev labeled\n\n$ kubectl label namespace irs-server-dev istio-injection=enabled\nnamespace\/irs-server-dev labeled<\/code><\/pre>\n
<\/p>\n
Configuring Automatic Linkerd Injection<\/h3>\n
<\/p>\n
To set up the injection of Linkerd sidecars, we use annotations (I added them manually through kubectl edit<\/code>):<\/p>\n
<\/p>\n
metadata:\n  annotations:\n    linkerd.io\/inject: enabled<\/code><\/pre>\n
<\/p>\n
$ k edit ns irs-server-dev \nnamespace\/irs-server-dev edited\n\n$ k get ns irs-server-dev -o yaml\napiVersion: v1\nkind: Namespace\nmetadata:\n  annotations:\n    linkerd.io\/inject: enabled\n  name: irs-server-dev\nspec:\n  finalizers:\n  - kubernetes\nstatus:\n  phase: Active<\/code><\/pre>\n
<\/p>\n
Istio Failover Simulator<\/h3>\n
<\/p>\n
We created an Istio fault tolerance simulator to experiment with traffic unique to Shopify. We needed a tool to create arbitrary topologies that represent a specific part of our service graph with dynamic configurations for modeling specific workloads.<\/p>\n
<\/p>\n
Shopify's infrastructure experiences a heavy load during flash sales. In this context, Shopify recommends that sellers hold such sales more frequently<\/a><\/noindex>. Large clients sometimes give us a heads-up about scheduled flash sales. Others conduct them unexpectedly at any time of day or night.<\/p>\n
<\/p>\n
We wanted our fault tolerance simulator to model workflows corresponding to the topologies and workloads that previously overloaded Shopify's infrastructure. The primary goal of using a service mesh is that we need reliability and fault tolerance at the network level, and it is crucial for us that the service mesh effectively handles workloads that had previously disrupted service operations.<\/p>\n
<\/p>\n
At the core of the fault tolerance simulator is a worker node acting as a service mesh node. The worker node can be configured statically at startup or dynamically via REST API. We use dynamic worker node configuration to create workflows in the form of regression tests.<\/p>\n
<\/p>\n
Here\u2019s an example of such a process:<\/p>\n
<\/p>\n