ʻO ka pae hoʻohana CPU no Istio a me Linkerd

ʻO ka pae hoʻohana CPU no Istio a me Linkerd

Hōʻike

Aia mākou i loko Shopify hoʻomaka ka hoʻohana ʻana iā Istio ma ke ʻano he lawelawe lawelawe. Ma ke kumu, maikaʻi nā mea a pau, koe wale nō hoʻokahi mea: he pipiʻi.

В paʻi ʻia nā pae ʻāina no Istio ke olelo nei.

Me Istio 1.1, hoʻopau ka mea koho ma kahi o 0,6 vCPUs (virtual cores) no 1000 noi i kekona.

No ka māhele mua o ka mesh lawelawe (2 proxies ma kēlā me kēia ʻaoʻao o ka pilina), e loaʻa iā mākou he 1200 cores no ka mea koho wale nō, ma ka helu o hoʻokahi miliona noi i kekona. Wahi a ka helu helu kumukūʻai a Google, ʻoi aku ka maikaʻi ma kahi o $ 40 / mahina / kumu no ka hoʻonohonoho. n1-standard-64, ʻo ia hoʻi, ʻo kēia māhele wale nō e uku iā mākou ma mua o 50 tausani kālā i kēlā me kēia mahina no 1 miliona mau noi i kekona.

Ivan Sim (ʻO Ivan Sim) hoʻohālikelike ʻike Ua hoʻopaneʻe ka mesh mesh i ka makahiki i hala a ua hoʻohiki ʻo ia no ka hoʻomanaʻo a me ke kaʻina hana, akā ʻaʻole ia i hana:

ʻIke ʻia, e hoʻonui nui ʻo values-istio-test.yaml i nā noi CPU. Inā ua hana pololei wau i kaʻu makemakika, pono ʻoe ma kahi o 24 CPU cores no ka panel control a me 0,5 CPU no kēlā me kēia koho. ʻAʻole nui kaʻu. E hana hou au i nā hoʻokolohua ke hāʻawi ʻia nā kumuwaiwai hou aʻe iaʻu.

Makemake au e ʻike noʻu iho i ke ʻano like o ka hana a Istio me kekahi mesh service open source: Linkerd.

Hoʻokomo ʻia mesh lawelawe

ʻO ka mea mua, ua hoʻokomo wau i loko o kahi pūpū SuperGloo:

$ supergloo init
installing supergloo version 0.3.12
using chart uri https://storage.googleapis.com/supergloo-helm/charts/supergloo-0.3.12.tgz
configmap/sidecar-injection-resources created
serviceaccount/supergloo created
serviceaccount/discovery created
serviceaccount/mesh-discovery created
clusterrole.rbac.authorization.k8s.io/discovery created
clusterrole.rbac.authorization.k8s.io/mesh-discovery created
clusterrolebinding.rbac.authorization.k8s.io/supergloo-role-binding created
clusterrolebinding.rbac.authorization.k8s.io/discovery-role-binding created
clusterrolebinding.rbac.authorization.k8s.io/mesh-discovery-role-binding created
deployment.extensions/supergloo created
deployment.extensions/discovery created
deployment.extensions/mesh-discovery created
install successful!

Ua hoʻohana au iā SuperGloo no ka mea ʻoi aku ka maʻalahi o ka bootstrap i ka mesh lawelawe. ʻAʻole nui kaʻu hana. ʻAʻole mākou e hoʻohana iā SuperGloo i ka hana ʻana, akā kūpono ia no ia hana. Pono wau e hoʻohana maoli i ʻelua mau kauoha no kēlā me kēia mesh lawelawe. Ua hoʻohana au i ʻelua mau puʻupuʻu no ka kaʻawale - hoʻokahi no Istio a me Linkerd.

Ua mālama ʻia ka hoʻokolohua ma Google Kubernetes Engine. Ua hoʻohana au i nā Kubernetes 1.12.7-gke.7 a me ka punawai o na node n1-standard-4 me ka scaling node aunoa (ka palena iki 4, ka nui 16).

A laila hoʻokomo wau i nā meshes lawelawe ʻelua mai ka laina kauoha.

Mea hoʻopili mua:

$ supergloo install linkerd --name linkerd
+---------+--------------+---------+---------------------------+
| INSTALL |     TYPE     | STATUS  |          DETAILS          |
+---------+--------------+---------+---------------------------+
| linkerd | Linkerd Mesh | Pending | enabled: true             |
|         |              |         | version: stable-2.3.0     |
|         |              |         | namespace: linkerd        |
|         |              |         | mtls enabled: true        |
|         |              |         | auto inject enabled: true |
+---------+--------------+---------+---------------------------+

A laila ʻo Istio:

$ supergloo install istio --name istio --installation-namespace istio-system --mtls=true --auto-inject=true
+---------+------------+---------+---------------------------+
| INSTALL |    TYPE    | STATUS  |          DETAILS          |
+---------+------------+---------+---------------------------+
| istio   | Istio Mesh | Pending | enabled: true             |
|         |            |         | version: 1.0.6            |
|         |            |         | namespace: istio-system   |
|         |            |         | mtls enabled: true        |
|         |            |         | auto inject enabled: true |
|         |            |         | grafana enabled: true     |
|         |            |         | prometheus enabled: true  |
|         |            |         | jaeger enabled: true      |
+---------+------------+---------+---------------------------+

He mau minuke ka lōʻihi o ka hāʻule ʻana, a laila hoʻopaʻa ʻia nā panela hoʻomalu.

(E hoʻomaopopo: kākoʻo wale ʻo SuperGloo iā Istio 1.0.x i kēia manawa. Ua hana hou au i ka hoʻokolohua me Istio 1.1.3, akā ʻaʻole i ʻike i kahi ʻokoʻa ʻike.)

Hoʻonohonoho ʻia ʻo Istio Automatic Deployment

No ka hoʻokomo ʻana iā Istio i ka Envoy sidecar, hoʻohana mākou i ka injector sidecar − MutatingAdmissionWebhook. ʻAʻole mākou e kamaʻilio e pili ana i kēia ʻatikala. E ʻōlelo wale wau he mea hoʻoponopono kēia e nānā i ke komo ʻana o nā pods hou a hoʻohui pū i kahi sidecar a me initContainer, nona ke kuleana no nā hana. iptables.

Ua kākau mākou ma Shopify i kā mākou mea hoʻokele pono e hoʻokō i nā sidecars, akā no kēia benchmark ua hoʻohana wau i ka mea hoʻokele e hele mai me Istio. Hoʻokomo ka mea hoʻoponopono i nā kaʻa ʻaoʻao ma ke ʻano maʻamau inā loaʻa kahi pōkole ma ka inoa inoa istio-injection: enabled:

$ kubectl label namespace irs-client-dev istio-injection=enabled
namespace/irs-client-dev labeled

$ kubectl label namespace irs-server-dev istio-injection=enabled
namespace/irs-server-dev labeled

Hoʻonohonoho ʻana i ka hoʻolaha ʻana o Linkerd

No ka hoʻonohonoho ʻana i ka hoʻokomo ʻana i ka sidecar Linkerd, hoʻohana mākou i nā annotations (hoʻohui wau iā lākou me ka lima ma o kubectl edit):

metadata:
  annotations:
    linkerd.io/inject: enabled

$ k edit ns irs-server-dev 
namespace/irs-server-dev edited

$ k get ns irs-server-dev -o yaml
apiVersion: v1
kind: Namespace
metadata:
  annotations:
    linkerd.io/inject: enabled
  name: irs-server-dev
spec:
  finalizers:
  - kubernetes
status:
  phase: Active

ʻO Istio Fault Tolerance Simulator

Ua kūkulu mākou i kahi simulator hoʻomanawanui hewa i kapa ʻia ʻo Istio e hoʻokolohua me nā kaʻa kūʻokoʻa i Shopify. Pono mākou i mea hana e hana i kahi topology maʻamau e hōʻike ana i kahi ʻāpana kikoʻī o kā mākou pakuhi lawelawe, i hoʻonohonoho ikaika ʻia e hoʻohālike i nā haʻahaʻa hana.

Aia ma lalo o ka ukana kaumaha o Shopify i ka wā kūʻai uila. Ma ka manawa like, Shopify paipai i nā mea kūʻai aku e hoʻopaʻa pinepine i kēlā mau kūʻai. Hoʻolaha nā mea kūʻai nui i kekahi manawa e pili ana i kahi kūʻai uila i hoʻolālā ʻia. ʻO kekahi poʻe e alakaʻi iā lākou me ka manaʻo ʻole no mākou i kēlā me kēia manawa o ke ao a i ka pō.

Ua makemake mākou i kā mākou resiliency simulator e hoʻohālike i nā kahe hana e kūlike i nā topologies a me nā haʻahaʻa hana i hoʻonui i ka ʻoihana Shopify i ka wā ma mua. ʻO ke kumu nui o ka hoʻohana ʻana i ka mesh lawelawe ʻo ia ka mea e pono ai mākou i ka hilinaʻi a me ka hoʻomanawanui hewa ma ka pae pūnaewele, a he mea nui iā mākou e hoʻokō pono ka mīkini lawelawe me nā ukana i hoʻopau i nā lawelawe.

Aia ma ka puʻuwai o ka simulator hoʻomanawanui hewa he node limahana, e hana ana ma ke ʻano he node mesh lawelawe. Hiki ke hoʻonohonoho pono ʻia ka node limahana ma ka hoʻomaka ʻana a i ʻole ma o ka REST API. Hoʻohana mākou i ka hoʻonohonoho ikaika o nā node limahana e hana i nā kahe hana ma ke ʻano o nā hoʻokolohua regression.

Eia kekahi laʻana o ia hana.

  • Hoʻomaka mākou i nā kikowaena 10 e like me bar lawelawe e hoihoi i ka pane 200/OK ma hope o 100 ms.
  • Hoʻomaka mākou i nā mea kūʻai aku 10 - hoʻouna kēlā me kēia i 100 noi i kēlā me kēia kekona i bar.
  • I kēlā me kēia 10 kekona, wehe mākou i kahi kikowaena 1 a nānā i nā hewa 5xx ma ka mea kūʻai aku.

I ka pau ʻana o ke kaʻina hana, nānā mākou i nā lāʻau a me nā metric a nānā inā ua hala ka hōʻike. Ma kēia ala mākou e aʻo ai e pili ana i ka hana o kā mākou lawelawe mesh a holo i kahi hoʻāʻo regression e hoʻāʻo i kā mākou mau manaʻo e pili ana i ka hoʻomanawanui hewa.

(E hoʻomaopopo: Ke noʻonoʻo nei mākou e pili ana i ka wehe ʻana i ka Istio fault tolerance simulator, akā ʻaʻole mākaukau e hana pēlā.)

ʻO Istio fault tolerance simulator no ka hōʻailona mesh service

Hoʻonohonoho mākou i kekahi mau nodes hana o ka simulator:

  • irs-client-loadgen: 3 replicas e hoʻouna i 100 noi i kēlā me kēia kekona irs-client.
  • irs-client: 3 replicas i loaʻa ka noi, kali 100ms a hoʻouna i ka noi i irs-server.
  • irs-server: 3 kope e hoʻi 200/OK ma hope o 100 ms.

Me kēia hoʻonohonoho ʻana, hiki iā mākou ke ana i kahi kahe kaʻa paʻa ma waena o 9 mau hopena. ʻO nā sidecars i loko irs-client-loadgen и irs-server loaa 100 noi i kekona, a irs-client — 200 (hiki mai a puka aku).

Mālama mākou i ka hoʻohana waiwai ma o DataDogno ka mea, ʻaʻohe o mākou pūʻulu Prometheus.

Nā hualoaʻa

Nā panela kaohi

ʻO ka mea mua, ua nānā mākou i ka hoʻohana CPU.

ʻO ka pae hoʻohana CPU no Istio a me Linkerd
Linkerd control panel ~22 millicore

ʻO ka pae hoʻohana CPU no Istio a me Linkerd
Istio control panel: ~750 millicore

Hoʻohana ka Istio control panel ma kahi o ʻO 35 mau manawa hou aku i nā kumuwaiwai CPUma mua o Linkerd. ʻOiaʻiʻo, ua hoʻokomo ʻia nā mea āpau ma ke ʻano maʻamau, a ʻo ka istio-telemetry e hoʻopau i ka nui o nā kumuwaiwai processor ma aneʻi (hiki ke hoʻopau ʻia ma ka hoʻopau ʻana i kekahi mau hana). Inā wehe mākou i kēia ʻāpana, loaʻa iā mākou ma mua o 100 millicores, ʻo ia hoʻi 4 mau manawa hou akuma mua o Linkerd.

ʻO ka mea koho kaʻa ʻaoʻao

A laila hoʻāʻo mākou i ka hoʻohana ʻana i kahi proxy. Pono e loaʻa kahi pilina laina me ka helu o nā noi, akā no kēlā me kēia kaʻa ʻaoʻao aia kekahi ma luna e pili ana i ka pihi.

ʻO ka pae hoʻohana CPU no Istio a me Linkerd
Linkerd: ~100 millicores no irs-client, ~50 millicores no irs-client-loadgen

Ua kūpono nā hualoaʻa, no ka mea, loaʻa ʻelua ka nui o ka hele ʻana o ka mea kūʻai aku ma mua o ka loadgen proxy: no kēlā me kēia noi puka mai loadgen, loaʻa i ka mea kūʻai aku hoʻokahi komo a hoʻokahi puka.

ʻO ka pae hoʻohana CPU no Istio a me Linkerd
Istio/Envoy: ~155 millicores no irs-client, ~75 millicores no irs-client-loadgen

ʻIke mākou i nā hopena like no nā sidecars Istio.

Akā ma ka laulā, ʻai nā proxies Istio/Envoy ma kahi o 50% mau kumuwaiwai CPUma mua o Linkerd.

ʻIke mākou i ka papahana like ma ka ʻaoʻao kikowaena:

ʻO ka pae hoʻohana CPU no Istio a me Linkerd
Linkerd: ~50 millicore no irs-server

ʻO ka pae hoʻohana CPU no Istio a me Linkerd
Istio/Envoy: ~80 millicore no irs-server

Ma ka ʻaoʻao kikowaena, hoʻopau ka sidecar Istio/Envoy ma kahi o 60% mau kumuwaiwai CPUma mua o Linkerd.

hopena

Hoʻopau ka Istio Envoy proxy i ka 50+% ʻoi aku ka CPU ma mua o Linkerd ma kā mākou hana hoʻohālikelike. ʻOi aku ka liʻiliʻi o nā kumuwaiwai i ka Linkerd control panel ma mua o Istio, ʻoi aku hoʻi no nā ʻāpana kumu.

Ke noʻonoʻo nei mākou pehea e hōʻemi ai i kēia mau kumukūʻai. Inā he manaʻo kāu, e ʻoluʻolu e kaʻana like!

Source: www.habr.com

Pākuʻi i ka manaʻo hoʻopuka