ProHoster > Блог > Kev tswj hwm > Kubernetes Network Plugin (CNI) Benchmark Results tshaj 10 Gbps Network (Hloov kho: Plaub Hlis 2019)

Kubernetes Network Plugin (CNI) Benchmark Results tshaj 10 Gbps Network (Hloov kho: Plaub Hlis 2019)

Kubernetes Network Plugin (CNI) Benchmark Results tshaj 10 Gbps Network (Hloov kho: Plaub Hlis 2019)
Nov yog kuv qhov hloov tshiab yav dhau los benchmark, uas tam sim no khiav ntawm Kubernetes 1.14 nrog qhov tseeb CNI version thaum lub Plaub Hlis 2019.

Ua ntej tshaj plaws, kuv xav ua tsaug rau pab pawg Cilium: cov txiv neej tau pab kuv tshawb xyuas thiab kho cov ntawv ntsuas ntsuas ntsuas.

Dab tsi tau hloov txij lub Kaum Ib Hlis 2018

Nov yog qhov hloov pauv txij thaum ntawd los (yog tias koj txaus siab):

Flannel tseem yog qhov ceev tshaj plaws thiab yooj yim tshaj plaws CNI interface, tab sis tseem tsis txhawb kev cai network thiab encryption.

Romana tsis tau txais kev txhawb nqa lawm, yog li peb tau tshem tawm ntawm qhov ntsuas.

WeaveNet tam sim no txhawb nqa cov cai hauv network rau Ingress thiab Egress! Tab sis kev tsim khoom tau txo qis.

Hauv Calico, koj tseem yuav tsum manually teeb tsa qhov siab tshaj plaws pob ntawv loj (MTU) rau kev ua tau zoo tshaj plaws. Calico muaj ob txoj hauv kev rau kev txhim kho CNI, yog li koj tuaj yeem ua yam tsis muaj ETCD repository cais:

  • khaws lub xeev hauv Kubernetes API ua cov ntaub ntawv khaws cia (pawg loj <50 nodes);
  • khaws cia lub xeev hauv Kubernetes API raws li cov ntaub ntawv khaws cia nrog Typha proxy los txo cov load ntawm K8S API (pawg loj> 50 nodes).

Calico tshaj tawm kev txhawb nqa daim ntawv thov qib kev cai nyob rau sab saum toj ntawm Istio rau daim ntawv thov-qib kev ruaj ntseg.

Cilium tam sim no txhawb kev encryption! Cilium muab encryption nrog IPSec tunnels thiab muaj lwm txoj hauv kev rau encrypted WeaveNet network. Tab sis WeaveNet tau nrawm dua Cilium nrog kev nkag mus encryption.

Cilium tam sim no yooj yim dua rau kev xa tawm ua tsaug rau tus neeg teb xov tooj built-in ETCD.

Pab pawg Cilium tau sim txiav qee qhov hnyav ntawm nws CNI los ntawm kev txo qis kev siv lub cim xeeb thiab CPU nqi, tab sis nws cov neeg sib tw tseem sib dua.

Benchmark ntsiab lus

Qhov ntsuas ntsuas tau khiav ntawm peb lub servers tsis yog supermicro virtual nrog 10 Gb Supermicro hloov. Cov servers txuas ncaj qha mus rau qhov hloov ntawm passive DAC SFP + cables thiab tau teeb tsa ntawm tib VLAN nrog jumbo thav ntawv (MTU 9000).

Kubernetes 1.14.0 tau nruab rau ntawm Ubuntu 18.04 LTS nrog Docker 18.09.2 (lub neej ntawd Docker version hauv qhov kev tso tawm no).

Txhawm rau txhim kho kev tsim tawm, peb tau txiav txim siab ib txwm teeb tsa tus tswv ntawm thawj qhov ntawm, tso tus neeg rau zaub mov ntawm qhov ntsuas ntawm tus thib ob server, thiab tus neeg siv khoom thib peb. Txhawm rau ua qhov no, peb siv NodeSelector hauv Kubernetes xa tawm.

Peb yuav piav qhia txog cov txiaj ntsig ntawm qhov ntsuas hauv qab no:

Kubernetes Network Plugin (CNI) Benchmark Results tshaj 10 Gbps Network (Hloov kho: Plaub Hlis 2019)

Xaiv CNI rau qhov ntsuas

Qhov no yog qhov ntsuas ntsuas tsuas yog rau CNI los ntawm cov npe hauv ntu txog kev tsim ib pawg tswv nrog kubeadm Saib cov ntaub ntawv Kubernetes official. Ntawm 9 CNIs, peb tsuas yog 6: peb yuav tshem tawm cov uas nyuaj rau nruab thiab / lossis tsis ua haujlwm yam tsis muaj kev teeb tsa raws li cov ntaub ntawv (Romana, Contiv-VPP thiab JuniperContrail / TungstenFabric).

Peb yuav muab piv rau CNIs hauv qab no:

  • Calico v3.6
  • Canal v3.6 (qhov tseem ceeb Flannel rau kev sib tham + Calico ua lub foob pob hluav taws)
  • Cilium 1.4.2
  • Flannel 0.11.0
  • Kube-router 0.2.5
  • WeaveNet 2.5.1

chaw

Qhov yooj yim dua lub CNI yog rau nruab, qhov zoo dua peb thawj qhov kev xav yuav. Txhua CNIs los ntawm qhov ntsuas tau yooj yim heev rau nruab (nrog ib lossis ob qho lus txib).

Raws li peb tau hais, cov servers thiab hloov pauv tau teeb tsa nrog jumbo thav duab qhib (peb teeb tsa MTU rau 9000). Peb yuav zoo siab yog tias CNI tau txiav txim siab MTU raws li tus neeg siv khoom hloov kho. Txawm li cas los xij, tsuas yog Cilium thiab Flannel tswj qhov no. Tus so ntawm CNIs muaj kev thov ntawm GitHub ntxiv qhov kev tshawb pom MTU tsis siv neeg, tab sis peb yuav teeb tsa nws manually los ntawm kev hloov ConfigMap rau Calico, Canal thiab Kube-router, lossis dhau ib puag ncig hloov pauv rau WeaveNet.

Qhov teeb meem nrog MTU tsis raug yog dab tsi? Daim duab no qhia qhov txawv ntawm WeaveNet nrog lub neej ntawd MTU thiab jumbo thav ntawv tau qhib:

Kubernetes Network Plugin (CNI) Benchmark Results tshaj 10 Gbps Network (Hloov kho: Plaub Hlis 2019)
MTU cuam tshuam li cas ntawm kev xa tawm?

Peb tau pom tias MTU tseem ceeb npaum li cas rau kev ua tau zoo, tam sim no cia saib seb peb CNIs txiav txim siab nws li cas:

Kubernetes Network Plugin (CNI) Benchmark Results tshaj 10 Gbps Network (Hloov kho: Plaub Hlis 2019)
CNI cia li ntes MTU

Daim duab qhia tau hais tias koj yuav tsum teeb tsa MTU rau Calico, Canal, Kube-router thiab WeaveNet rau kev ua haujlwm zoo. Cilium thiab Flannel tuaj yeem txiav txim siab MTU lawv tus kheej kom raug yam tsis muaj kev teeb tsa.

Kev ruaj ntseg

Peb yuav sib piv CNI kev ruaj ntseg nyob rau hauv ob yam: lub peev xwm los encrypt kis tau cov ntaub ntawv thiab kev siv ntawm Kubernetes network txoj cai (raws li kev ntsuam xyuas tiag tiag, tsis yog cov ntaub ntawv).

Tsuas yog ob lub CNIs encrypt cov ntaub ntawv: Cilium thiab WeaveNet. Encryption WeaveNet enabled los ntawm kev teeb tsa tus password encryption ua CNI ib puag ncig hloov pauv. IN cov ntaub ntawv WeaveNet piav qhia nws txoj hauv kev nyuaj, tab sis txhua yam ua tiav yooj yim. Encryption cilium teeb tsa los ntawm cov lus txib, los ntawm kev tsim Kubernetes zais cia, thiab los ntawm kev hloov kho ntawm daemonSet (me ntsis nyuaj dua li hauv WeaveNet, tab sis Cilium muaj cov kauj ruam dhau los. cov lus qhia).

Raws li kev siv txoj cai network, lawv tau ua tiav Calico, Canal, Cilium thiab WeaveNet, uas koj tuaj yeem teeb tsa Ingress thiab Egress cov cai. Rau Kub-router muaj cov cai tsuas yog rau Ingress, thiab Flannel Tsis muaj txoj cai network txhua.

Nov yog tag nrho cov txiaj ntsig:

Kubernetes Network Plugin (CNI) Benchmark Results tshaj 10 Gbps Network (Hloov kho: Plaub Hlis 2019)
Safety Performance Benchmark Results

Tsim tau

Qhov kev ntsuas no qhia tau hais tias qhov nruab nrab dhau los ntawm yam tsawg kawg peb zaug ntawm txhua qhov kev xeem. Peb sim ua haujlwm ntawm TCP thiab UDP (siv iperf3), cov ntawv thov tiag tiag xws li HTTP (nrog Nginx thiab curl) lossis FTP (nrog vsftpd thiab curl) thiab thaum kawg daim ntawv thov kev ua tau zoo siv SCP-based encryption (siv cov neeg siv khoom thiab server OpenSSH).

Rau txhua qhov kev ntsuam xyuas, peb tau ua tus qauv hlau liab qab (kab ntsuab) los sib piv CNI kev ua tau zoo nrog kev ua haujlwm hauv zej zog. Ntawm no peb siv tib lub teev, tab sis xim:

  • Yellow = zoo heev
  • Txiv kab ntxwv = zoo
  • Blue = yog
  • Liab = phem

Peb yuav tsis coj CNIs teeb tsa tsis raug thiab tsuas yog qhia cov txiaj ntsig rau CNIs nrog MTU raug. (Faj seeb: Cilium tsis raug xam qhov MTU yog tias koj qhib encryption, yog li koj yuav tsum manually txo MTU rau 8900 hauv version 1.4. Cov tom ntej version, 1.5, ua qhov no cia li.)

Nov yog cov txiaj ntsig:

Kubernetes Network Plugin (CNI) Benchmark Results tshaj 10 Gbps Network (Hloov kho: Plaub Hlis 2019)
TCP Performance

Txhua CNIs ua tau zoo nyob rau hauv TCP benchmark. CNI nrog encryption qeeb qeeb vim tias kev encryption kim.

Kubernetes Network Plugin (CNI) Benchmark Results tshaj 10 Gbps Network (Hloov kho: Plaub Hlis 2019)
UDP kev ua haujlwm

Ntawm no, ib yam nkaus, txhua tus CNIs ua tau zoo. CNI nrog encryption pom yuav luag tib yam tshwm sim. Cilium yog me ntsis tom qab kev sib tw, tab sis nws tsuas yog 2,3% ntawm cov hlau liab qab, yog li nws tsis yog qhov tshwm sim tsis zoo. Tsis txhob hnov ​​​​qab tias tsuas yog Cilium thiab Flannel txiav txim siab MTU raug lawv tus kheej, thiab cov no yog lawv cov txiaj ntsig yam tsis muaj kev teeb tsa ntxiv.

Kubernetes Network Plugin (CNI) Benchmark Results tshaj 10 Gbps Network (Hloov kho: Plaub Hlis 2019)

Yuav ua li cas txog daim ntawv thov tiag? Raws li koj tuaj yeem pom, tag nrho kev ua tau zoo rau HTTP yog me ntsis qis dua rau TCP. Txawm hais tias koj siv HTTP nrog TCP, peb tau teeb tsa iperf3 hauv TCP ntsuas ntsuas kom tsis txhob pib qeeb uas yuav cuam tshuam rau HTTP benchmark. Txhua tus tau ua haujlwm zoo ntawm no. Kube-router muaj qhov zoo dua qub, tab sis WeaveNet tsis ua haujlwm zoo: txog 20% ​​phem dua li cov hlau liab qab. Cilium thiab WeaveNet nrog encryption zoo li tu siab heev.

Kubernetes Network Plugin (CNI) Benchmark Results tshaj 10 Gbps Network (Hloov kho: Plaub Hlis 2019)

Nrog FTP, lwm TCP raws tu qauv, cov txiaj ntsig sib txawv. Flannel thiab Kube-router ua txoj haujlwm, tab sis Calico, Canal thiab Cilium yog me ntsis tom qab thiab yog li 10% qeeb dua li cov hlau liab qab. WeaveNet yog qab los ntawm ntau npaum li 17%, tab sis encrypted WeaveNet yog 40% ua ntej encrypted Cilium.

Kubernetes Network Plugin (CNI) Benchmark Results tshaj 10 Gbps Network (Hloov kho: Plaub Hlis 2019)

Nrog SCP peb tuaj yeem pom sai npaum li cas SSH encryption nqi peb. Yuav luag txhua tus CNIs ua tau zoo, tab sis WeaveNet tseem poob qis dua. Cilium thiab WeaveNet nrog encryption yuav tsum yog qhov phem tshaj vim yog ob chav encryption (SSH + CNI).

Ntawm no yog ib lub ntsiab lus nrog cov txiaj ntsig:

Kubernetes Network Plugin (CNI) Benchmark Results tshaj 10 Gbps Network (Hloov kho: Plaub Hlis 2019)

Kev siv peev txheej

Tam sim no cia peb sib piv li cas CNI siv cov peev txheej hauv qab hnyav (thaum TCP hloov pauv, 10 Gbps). Hauv kev ntsuas kev ua tau zoo peb piv CNI nrog cov hlau liab qab (kab ntsuab). Rau kev siv peev txheej, cia peb qhia Kubernetes dawb huv (kab liab) yam tsis muaj CNI thiab saib seb muaj pes tsawg cov peev txheej ntxiv CNI siv.

Cia peb pib nrog kev nco. Ntawm no yog tus nqi nruab nrab rau cov nodes 'RAM (tsis suav nrog buffers thiab cache) hauv MB thaum hloov tsheb.

Kubernetes Network Plugin (CNI) Benchmark Results tshaj 10 Gbps Network (Hloov kho: Plaub Hlis 2019)
Nco noj

Flannel thiab Kube-router pom tau zoo heev - tsuas yog 50 MB. Calico thiab Canal txhua tus muaj 70. WeaveNet kom meej meej siv ntau dua li lwm tus - 130 MB, thiab Cilium siv ntau npaum li 400.
Tam sim no cia peb xyuas lub sijhawm siv CPU. Nco ntsoov: daim duab qhia tsis feem pua, tab sis ppm, uas yog, 38 ppm rau "liab liab qab" yog 3,8%. Nov yog cov txiaj ntsig:

Kubernetes Network Plugin (CNI) Benchmark Results tshaj 10 Gbps Network (Hloov kho: Plaub Hlis 2019)
CPU siv

Calico, Canal, Flannel thiab Kube-router yog CPU heev - tsuas yog 2% ntau dua Kubernetes yam tsis muaj CNI. WeaveNet poob qis dua nrog ib qho ntxiv 5%, ua raws li Cilium ntawm 7%.

Nov yog cov ntsiab lus ntawm kev siv cov peev txheej:

Kubernetes Network Plugin (CNI) Benchmark Results tshaj 10 Gbps Network (Hloov kho: Plaub Hlis 2019)

Cov txiaj ntsim tau los

Table nrog tag nrho cov txiaj ntsig:

Kubernetes Network Plugin (CNI) Benchmark Results tshaj 10 Gbps Network (Hloov kho: Plaub Hlis 2019)
Cov txiaj ntsig kev ntsuas dav dav

xaus

Hauv ntu kawg kuv yuav qhia kuv qhov kev xav txog cov txiaj ntsig. Nco ntsoov tias qhov ntsuas ntsuas no tsuas yog ntsuas qhov kev sib txuas ntawm ib qho kev sib txuas ntawm ib pawg me me (3 nodes). Nws tsis siv rau pawg loj (<50 nodes) lossis kev sib txuas ua ke.

Kuv pom zoo kom siv CNIs hauv qab no nyob ntawm qhov xwm txheej:

  • Koj puas muaj nyob hauv koj pawg nodes nrog ob peb cov kev pab (ntau GB ntawm RAM, ob peb cores) thiab koj tsis xav tau kev ruaj ntseg nta - xaiv Flannel. Qhov no yog ib qho ntawm cov txiaj ntsig zoo tshaj plaws CNIs. Thiab nws yog sib xws nrog ntau yam architectures (amd64, caj npab, arm64, thiab lwm yam). Tsis tas li ntawd, qhov no yog ib qho ntawm ob (lwm yam yog Cilium) CNI uas tuaj yeem txiav txim siab MTU, yog li koj tsis tas yuav teeb tsa ib yam dab tsi. Kube-router kuj tsim nyog, tab sis nws tsis yog raws li tus qauv thiab koj yuav tau manually configure MTU.
  • Yog tias tsim nyog encrypt lub network rau kev nyab xeeb, coj WeaveNet. Tsis txhob hnov ​​​​qab qhia qhov loj me MTU yog tias koj siv jumbo thav duab, thiab pab kom encryption los ntawm kev qhia tus password ntawm ib puag ncig hloov pauv. Tab sis nws yog qhov zoo dua kom tsis nco qab txog kev ua tau zoo - qhov ntawd yog tus nqi ntawm encryption.
  • rau ib txwm siv советую Calico. Qhov CNI no tau siv dav hauv ntau yam Kubernetes xa cov cuab yeej (Kops, Kubespray, Rancher, thiab lwm yam). Raws li nrog WeaveNet, nco ntsoov teeb tsa MTU hauv ConfigMap yog tias siv jumbo thav duab. Nws yog cov cuab yeej siv ntau yam uas muaj txiaj ntsig zoo ntawm kev siv peev txheej, kev ua haujlwm thiab kev ruaj ntseg.

Thiab thaum kawg, kuv qhia koj ua raws li kev txhim kho cilium. Qhov CNI no muaj ib pab neeg ua haujlwm zoo heev uas ua haujlwm ntau rau lawv cov khoom (cov yam ntxwv, kev txuag nyiaj, kev ua haujlwm, kev ruaj ntseg, pawg ...) thiab lawv muaj cov phiaj xwm nthuav dav heev.

Kubernetes Network Plugin (CNI) Benchmark Results tshaj 10 Gbps Network (Hloov kho: Plaub Hlis 2019)
Visual daim duab rau CNI xaiv

Tau qhov twg los: www.hab.com

Ntxiv ib saib