🥇CPU Consumption Benchmark for Istio and Linkerd

CPU Consumption Benchmark for Istio and Linkerd

Introduction

We at Shopify we focused on deploying Istio as a service mesh. Generally, everything is fine except for one thing: it's expensive.

In published benchmarks it is stated for Istio:

With Istio 1.1, the proxy consumes approximately 0.6 vCPU (virtual cores) per 1000 requests per second.

For the first region in the service mesh (2 proxies on each side of the connection), we will need 1200 cores just for the proxies, based on one million requests per second. According to Google's cost calculator, that amounts to about $40/month/core for the configuration n1-standard-64, which means this single region will cost us over 50 thousand dollars a month for 1 million requests per second.

Ivan Sim (Ivan Sim) visually compared service mesh latencies from last year and promised the same for memory and CPU, but it didn't work out:

Apparently, values-istio-test.yaml will significantly increase CPU requests. If I calculated correctly, we need about 24 CPU cores for the control panel and 0.5 CPU for each proxy. I don't have that many. I will repeat the tests when I am allocated more resources.

I wanted to see for myself how Istio's metrics compare to another open-source service mesh: Linkerd.

Installing the service mesh

First, I installed in the cluster SuperGloo:

$ supergloo init
installing supergloo version 0.3.12
using chart uri https://storage.googleapis.com/supergloo-helm/charts/supergloo-0.3.12.tgz
configmap/sidecar-injection-resources created
serviceaccount/supergloo created
serviceaccount/discovery created
serviceaccount/mesh-discovery created
clusterrole.rbac.authorization.k8s.io/discovery created
clusterrole.rbac.authorization.k8s.io/mesh-discovery created
clusterrolebinding.rbac.authorization.k8s.io/supergloo-role-binding created
clusterrolebinding.rbac.authorization.k8s.io/discovery-role-binding created
clusterrolebinding.rbac.authorization.k8s.io/mesh-discovery-role-binding created
deployment.extensions/supergloo created
deployment.extensions/discovery created
deployment.extensions/mesh-discovery created
install successful!

I used SuperGloo because it significantly simplifies the initial setup of a service mesh. I hardly had to do anything. In production, we do not use SuperGloo, but it is perfect for such tasks. I had to apply just a couple of commands for each service mesh. I used two clusters for isolation — one each for Istio and Linkerd.

The experiment was conducted on Google Kubernetes Engine. I used Kubernetes 1.12.7-gke.7 and a node pool n1-standard-4 with automatic node scaling (minimum 4, maximum 16).

Then I installed both service meshes from the command line.

First, Linkerd:

$ supergloo install linkerd --name linkerd
+---------+--------------+---------+---------------------------+
| INSTALL |     TYPE     | STATUS  |          DETAILS          |
+---------+--------------+---------+---------------------------+
| linkerd | Linkerd Mesh | Pending | enabled: true             |
|         |              |         | version: stable-2.3.0     |
|         |              |         | namespace: linkerd        |
|         |              |         | mtls enabled: true        |
|         |              |         | auto inject enabled: true |
+---------+--------------+---------+---------------------------+

Next is Istio:

$ supergloo install istio --name istio --installation-namespace istio-system --mtls=true --auto-inject=true
+---------+------------+---------+---------------------------+
| INSTALL |    TYPE    | STATUS  |          DETAILS          |
+---------+------------+---------+---------------------------+
| istio   | Istio Mesh | Pending | enabled: true             |
|         |            |         | version: 1.0.6            |
|         |            |         | namespace: istio-system   |
|         |            |         | mtls enabled: true        |
|         |            |         | auto inject enabled: true |
|         |            |         | grafana enabled: true     |
|         |            |         | prometheus enabled: true  |
|         |            |         | jaeger enabled: true      |
+---------+------------+---------+---------------------------+

The crash-loop took several minutes, and then the control panels stabilized.

(Note: SuperGloo currently supports only Istio 1.0.x. I repeated the experiment with Istio 1.1.3, but I did not notice any significant difference.)

Configuring Automatic Istio Injection

To have Istio install the Envoy sidecar, we use the sidecar injector — MutatingAdmissionWebhook. We won't discuss it in this article. I'll just say that it's a controller that monitors access for all new pods and dynamically adds the sidecar and initContainer responsible for tasks. iptables.

At Shopify, we wrote our own access controller for injecting sidecars, but in this benchmark, I took the controller that comes with Istio. The default controller injects sidecars when there's a label in the namespace. istio-injection: enabled:

$ kubectl label namespace irs-client-dev istio-injection=enabled
namespace/irs-client-dev labeled

$ kubectl label namespace irs-server-dev istio-injection=enabled
namespace/irs-server-dev labeled

Configuring Automatic Linkerd Injection

To set up the injection of Linkerd sidecars, we use annotations (I added them manually through kubectl edit):

metadata:
  annotations:
    linkerd.io/inject: enabled

$ k edit ns irs-server-dev 
namespace/irs-server-dev edited

$ k get ns irs-server-dev -o yaml
apiVersion: v1
kind: Namespace
metadata:
  annotations:
    linkerd.io/inject: enabled
  name: irs-server-dev
spec:
  finalizers:
  - kubernetes
status:
  phase: Active

Istio Failover Simulator

We created an Istio fault tolerance simulator to experiment with traffic unique to Shopify. We needed a tool to create arbitrary topologies that represent a specific part of our service graph with dynamic configurations for modeling specific workloads.

Shopify's infrastructure experiences a heavy load during flash sales. In this context, Shopify recommends that sellers hold such sales more frequently. Large clients sometimes give us a heads-up about scheduled flash sales. Others conduct them unexpectedly at any time of day or night.

We wanted our fault tolerance simulator to model workflows corresponding to the topologies and workloads that previously overloaded Shopify's infrastructure. The primary goal of using a service mesh is that we need reliability and fault tolerance at the network level, and it is crucial for us that the service mesh effectively handles workloads that had previously disrupted service operations.

At the core of the fault tolerance simulator is a worker node acting as a service mesh node. The worker node can be configured statically at startup or dynamically via REST API. We use dynamic worker node configuration to create workflows in the form of regression tests.

Here’s an example of such a process:

We launch 10 servers as a bar service that returns a response 200/OK after 100 ms.
We launch 10 clients — each sending 100 requests per second to bar.
Every 10 seconds, we remove 1 server, monitoring errors 5xx on the client.

At the end of the workflow, we examine logs and metrics to check if the test passed. This way, we learn about the performance of our service mesh and conduct regression testing to verify our assumptions about fault tolerance.

(Note: We are considering open-sourcing the Istio fault tolerance simulator, but we are not ready for that yet.)

Istio Fault Tolerance Simulator for Service Mesh Benchmarking

We configure several worker nodes of the simulator:

irs-client-loadgen: 3 replicas, each sending 100 requests per second to irs-client.
irs-client: 3 replicas that receive requests, wait 100 ms and forward the request to irs-server.
irs-server: 3 replicas that return 200/OK after 100 ms.

With this configuration, we can measure a steady traffic flow between 9 endpoints. Sidecars at irs-client-loadgen and irs-server receive 100 requests per second, while irs-client — 200 (incoming and outgoing).

We monitor resource usage through DataDog, because we do not have a Prometheus cluster.

Results

Control panels

First, we examined CPU consumption.

Linkerd dashboard ~22 milliwatts

Istio dashboard: ~750 milliwatts

The Istio dashboard uses about 35 times more CPU resources, than Linkerd. Of course, everything is set to defaults, and a significant amount of CPU resources here is consumed by istio-telemetry (which can be disabled by foregoing some features). Even without this component, it still results in over 100 milliwatts, meaning 4 times more, than Linkerd.

Sidecar proxies

Then we checked proxy usage. There should be a linear dependence on the number of requests, but each sidecar has some overhead that affects the curve.

Linkerd: ~100 milliwatts for irs-client, ~50 milliwatts for irs-client-loadgen

The results make sense since the client proxy receives twice as much traffic as the loadgen proxy: for each outgoing request from loadgen, the client has one incoming and one outgoing.

Istio/Envoy: ~155 milliwatts for irs-client, ~75 milliwatts for irs-client-loadgen

We see similar results for Istio sidecars.

But overall, the Istio/Envoy proxies consume approximately 50% more CPU resources, than Linkerd.

We see the same pattern on the server side:

Linkerd: ~50 milliwatts for irs-server

Istio/Envoy: ~80 milliwatts for irs-server

On the server side, the Istio/Envoy sidecar consumes approximately 60% more CPU resources, than Linkerd.

Conclusion

The Istio Envoy proxy consumes over 50% more CPU than Linkerd in our modeled workload. The Linkerd dashboard consumes significantly fewer resources than Istio, particularly concerning core components.

We are still considering how to reduce these costs. If you have ideas, please share!

Source: habr.com