I-Kubernetes Network Plugin (CNI) Benchmark Results over 10 Gbps Network (Ibuyekeziwe: April 2019)

Lesi isibuyekezo sami ibhentshimakhi yangaphambilini, manje esebenza ku-Kubernetes 1.14 ngenguqulo yakamuva ye-CNI kusukela ngo-Ephreli 2019.

Okokuqala, ngifuna ukubonga ithimba le-Cilium: abafana bangisize ukuthi ngihlole futhi ngilungise imibhalo yokuqapha amamethrikhi.

Yini eshintshile kusukela ngoNovemba 2018

Nakhu okushintshile kusukela lapho (uma uthanda):

I-Flannel isalokhu iyisixhumi esibonakalayo se-CNI esishesha kakhulu nesilula, kodwa namanje ayisekeli izinqubomgomo zenethiwekhi nokubethela.

I-Romana ayisasekelwa, ngakho siyisusile kubhentshimakhi.

I-WeaveNet manje isekela izinqubomgomo zenethiwekhi ze-Ingress ne-Egress! Kodwa ukukhiqiza kwehlile.

Ku-Calico, usadinga ukulungiselela mathupha usayizi wephakethe omkhulu (MTU) ukuze usebenze kahle kakhulu. I-Calico inikeza izinketho ezimbili zokufaka i-CNI, ukuze ukwazi ukwenza ngaphandle kwendawo yokugcina ye-ETCD:

• ukugcina isimo ku-Kubernetes API njengesitolo sedatha (usayizi weqoqo < 50 nodes);
• ukugcina isimo ku-Kubernetes API njengesitolo sedatha ngommeleli we-Typha ukuze kukhululwe umthwalo ku-K8S API (usayizi weqoqo > amanodi angu-50).

UCalico umemezele ukwesekwa izinqubomgomo zezinga lohlelo lokusebenza phezu kwe-Istio yokuphepha kwezinga lesicelo.

I-Cilium manje isekela ukubethela! I-Cilium inikeza ukubethela ngemihubhe ye-IPSec futhi inikeza enye indlela kunethiwekhi ye-WeaveNet ebethelwe. Kodwa i-WeaveNet ishesha kune-Cilium ngokubethela kunikwe amandla.

I-Cilium manje sekulula ukuyisebenzisa ngenxa yomqhubi owakhelwe ngaphakathi we-ETCD.

Ithimba le-Cilium lizamile ukunciphisa isisindo ku-CNI yalo ngokunciphisa ukusetshenziswa kwememori nezindleko ze-CPU, kodwa izimbangi zayo zisalula.

Ibhentshimakhi umongo

Ibhentshimakhi isetshenziswa kumaseva e-Supermicro angewona ama-virtualized ane-switch ye-10 Gb Supermicro. Amaseva axhunywe ngokuqondile kuswishi ngezintambo ze-DAC SFP+ ezingenzi lutho futhi alungiswa ku-VLAN efanayo namafreyimu e-jumbo (MTU 9000).

I-Kubernetes 1.14.0 ifakwe ku-Ubuntu 18.04 LTS nge-Docker 18.09.2 (inguqulo ye-Docker ezenzakalelayo kulokhu kukhululwa).

Ukuze sithuthukise ukukhiqizwa kabusha, sinqume ukumisa njalo okuyinhloko endaweni yokuqala, sibeke ingxenye yeseva yebhentshimakhi kuseva yesibili, nengxenye yeklayenti kweyesithathu. Ukwenza lokhu, sisebenzisa i-NodeSelector ku-Kubernetes deployments.

Sizochaza imiphumela yokuma esikalini esilandelayo:

Ukukhetha i-CNI yebhentshimakhi

Lena ibhentshimakhi ye-CNI kuphela ohlwini olusesigabeni mayelana nokudala iqoqo elilodwa eliyinhloko nge-kubeadm Bona imibhalo esemthethweni ye-Kubernetes. Kuma-CNI angu-9, sizothatha angu-6 kuphela: sizokhipha lawo okunzima ukuwafaka kanye/noma angasebenzi ngaphandle kokucushwa ngokuvumelana nemibhalo (Romana, Contiv-VPP kanye neJuniperContrail/TungstenFabric).

Sizoqhathanisa ama-CNI alandelayo:

• I-Calico v3.6
• I-Canal v3.6 (empeleni i-Flannel yokuxhumana + ne-Calico njenge-firewall)
• I-Cilium 1.4.2
• I-Flannel 0.11.0
• I-Kube-router 0.2.5
• I-WeaveNet 2.5.1

setting

Uma kulula ukufaka i-CNI, kuzoba ngcono umbono wethu wokuqala. Wonke ama-CNI asuka kubhentshimakhi alula kakhulu ukuwafaka (ngomyalo owodwa noma emibili).

Njengoba sishilo, amaseva neswishi kulungiselelwa ngamafreyimu we-jumbo anikwe amandla (setha i-MTU ibe ngu-9000). Singajabula uma i-CNI inquma ngokuzenzakalelayo i-MTU ngokusekelwe ekucushweni kwama-adaptha. Nokho, uCilium noFlannel kuphela abaphethe lokhu. Amanye ama-CNI anezicelo ku-GitHub zokungeza ukutholwa kwe-MTU okuzenzakalelayo, kodwa sizoyilungisa mathupha ngokushintsha i-ConfigMap ye-Calico, i-Canal ne-Kube-router, noma sidlulise okuguquguqukayo kwendawo ye-WeaveNet.

Iyini inkinga nge-MTU engalungile? Lo mdwebo ubonisa umehluko phakathi kwe-WeaveNet ene-MTU ezenzakalelayo kanye namafreyimu e-jumbo anikwe amandla:

Ngabe i-MTU iwuthinta kanjani umsebenzi owenziwayo?

Sibonile ukuthi i-MTU ibaluleke kangakanani ekusebenzeni, manje ake sibone ukuthi ama-CNIs ethu ayinquma kanjani ngokuzenzakalelayo:

I-CNI ithola i-MTU ngokuzenzakalelayo

Igrafu ibonisa ukuthi udinga ukulungisa i-MTU ye-Calico, Canal, Kube-router kanye ne-WeaveNet ukuze usebenze kahle. I-Cilium ne-Flannel bakwazile ukunquma kahle i-MTU ngokwabo ngaphandle kwanoma yiziphi izilungiselelo.

Ukuphepha

Sizoqhathanisa ukuphepha kwe-CNI ezicini ezimbili: ikhono lokubethela idatha edlulisiwe kanye nokuqaliswa kwezinqubomgomo zenethiwekhi ye-Kubernetes (ngokusekelwe ekuhloleni kwangempela, hhayi imibhalo).

Ama-CNI amabili kuphela abethela idatha: I-Cilium ne-WeaveNet. Ukubethela I-WeaveNet inikwe amandla ngokusetha iphasiwedi yokubethela njengokuguquguquka kwemvelo ye-CNI. IN imibhalo I-WeaveNet iyichaza ngendlela eyinkimbinkimbi, kodwa yonke into yenziwa kalula. Ukubethela I-Cilium ihlelwe ngemiyalo, ngokwakha izimfihlo ze-Kubernetes, nangokulungiswa kwe-daemonSet (iyinkimbinkimbi kakhulu kune-WeaveNet, kodwa i-Cilium inesinyathelo ngesinyathelo imiyalelo).

Mayelana nokuqaliswa kwenqubomgomo yenethiwekhi, baphumelele I-Calico, iCanal, i-Cilium ne-WeaveNet, lapho ungalungiselela khona imithetho ye-Ingress ne-Egress. Ngoba Kube-router kunemithetho ye-Ingress kuphela, futhi Flannel Azikho nhlobo izinqubomgomo zenethiwekhi.

Nansi yonke imiphumela:

Imiphumela Yebhentshimakhi Yokusebenza Kokuphepha

Ukukhiqiza

Leli benchmark libonisa isilinganiso sokuphuma kokungenani ngama-run amathathu ohlolo ngalunye. Sihlola ukusebenza kwe-TCP ne-UDP (sisebenzisa iperf3), izinhlelo zokusebenza zangempela ezifana ne-HTTP (ene-Nginx ne-curl) noma i-FTP (ene-vsftpd ne-curl) futhi ekugcineni ukusebenza kohlelo lokusebenza kusetshenziswa ukubethela okusekelwe ku-SCP (kusetshenziswa iklayenti neseva i-OpenSSH).

Kuzo zonke izivivinyo, senze ibhentshimakhi yensimbi engenalutho (umugqa oluhlaza) ukuze siqhathanise ukusebenza kwe-CNI nokusebenza kwenethiwekhi yomdabu. Lapha sisebenzisa isikali esifanayo, kodwa ngombala:

• Ophuzi = kuhle kakhulu
• Okuwolintshi = okuhle
• Okuluhlaza okwesibhakabhaka = ngakho-ke
• Okubomvu = okubi

Ngeke sithathe ama-CNI amiswe ngokungalungile futhi sizobonisa kuphela imiphumela yama-CNI nge-MTU elungile. (Qaphela: I-Cilium ayibali kahle i-MTU uma uvumela ukubethela, ngakho-ke kuzodingeka wehlise ngesandla i-MTU ibe ngu-8900 kunguqulo 1.4. Inguqulo elandelayo, 1.5, ikwenza lokhu ngokuzenzakalelayo.)

Nansi imiphumela:

Ukusebenza kwe-TCP

Wonke ama-CNI asebenze kahle kubhentshimakhi ye-TCP. I-CNI ene-encryption isalele emuva kakhulu ngoba ukubethela kuyabiza.

Ukusebenza kwe-UDP

Nalapha, wonke ama-CNI enza kahle. I-CNI ene-encryption ibonise cishe umphumela ofanayo. I-Cilium ingemuva kancane komncintiswano, kodwa i-2,3% kuphela yensimbi engenalutho, ngakho akuwona umphumela omubi. Ungakhohlwa ukuthi i-Cilium ne-Flannel kuphela enqume i-MTU ngendlela efanele ngokwabo, futhi lena imiphumela yabo ngaphandle kokucushwa okwengeziwe.

Kuthiwani ngesicelo sangempela? Njengoba ubona, ukusebenza jikelele kwe-HTTP kuphansi kancane kune-TCP. Ngisho noma usebenzisa i-HTTP nge-TCP, silungiselele iperf3 kubhentshimakhi ye-TCP ukugwema isiqalo esinensayo esingathinta ibhentshimakhi ye-HTTP. Wonke umuntu wenze umsebenzi omuhle lapha. I-Kube-router inenzuzo ecacile, kodwa i-WeaveNet ayizange isebenze kahle: cishe ama-20% amabi kunensimbi engenalutho. I-Cilium ne-WeaveNet enombhalo oyimfihlo ibukeka idabukile ngempela.

Nge-FTP, enye iphrothokholi esekwe ku-TCP, imiphumela iyahlukahluka. I-Flannel ne-Kube-router zenza umsebenzi, kodwa i-Calico, i-Canal ne-Cilium zisemuva kancane futhi zihamba kancane ngo-10% kunensimbi engenalutho. I-WeaveNet ingemuva ngo-17%, kodwa i-WeaveNet ebethelwe ingama-40% ngaphambi kwe-Cilium ebethelwe.

Nge-SCP singabona ngokushesha ukuthi kubiza malini ukubethela kwe-SSH. Cishe wonke ama-CNIs enza kahle, kodwa i-WeaveNet isalele emuva futhi. I-Cilium ne-WeaveNet ezinokubethela kulindeleke ukuthi zibe zimbi kakhulu ngenxa yokubethela okukabili (SSH + CNI).

Nali ithebula lesifinyezo elinemiphumela:

Ukusetshenziswa kwezinsiza

Manje ake siqhathanise ukuthi i-CNI idla kanjani izinsiza ngaphansi kwemithwalo esindayo (ngesikhathi sokudlulisa i-TCP, i-10 Gbps). Ekuhlolweni kokusebenza siqhathanisa i-CNI nensimbi engenalutho (umugqa oluhlaza). Ngokusetshenziswa kwezinsiza, masibonise i-Kubernetes emsulwa (umugqa onsomi) ngaphandle kwe-CNI futhi sibone ukuthi zingaki izinsiza ezengeziwe ezisetshenziswa yi-CNI.

Ake siqale ngenkumbulo. Nali isilinganiso senani le-RAM yamanodi (ngaphandle kwamabhafa nenqolobane) ku-MB phakathi nokudlulisa.

Ukusetshenziswa kwememori

I-Flannel ne-Kube-router ibonise imiphumela emihle kakhulu - 50 MB kuphela. I-Calico neCanal ngayinye inama-70. I-WeaveNet idla ngokusobala kakhulu kunezinye - 130 MB, futhi i-Cilium isebenzisa cishe okungama-400.
Manje ake sihlole ukusetshenziswa kwesikhathi se-CPU. Kuyaphawuleka: umdwebo awubonisi amaphesenti, kodwa i-ppm, okungukuthi, i-38 ppm "yensimbi engenalutho" ingu-3,8%. Nansi imiphumela:

Ukusetshenziswa kwe-CPU

I-Calico, Canal, Flannel ne-Kube-router zisebenza kahle kakhulu nge-CPU - kuphela ngo-2% ngaphezu kwe-Kubernetes ngaphandle kwe-CNI. I-WeaveNet isalele emuva kakhulu ngo-5% owengeziwe, ilandelwa yi-Cilium ngo-7%.

Nasi isifinyezo sokusetshenziswa kwensiza:

Imiphumela

Ithebula elinayo yonke imiphumela:

Imiphumela yebhentshimakhi evamile

isiphetho

Engxenyeni yokugcina ngizoveza umbono wami oqondile ngemiphumela. Khumbula ukuthi le benchmark ihlola kuphela ukuphuma koxhumano olulodwa kuqoqo elincane kakhulu (amanodi ama-3). Ayisebenzi kumaqoqo amakhulu (<50 nodes) noma ukuxhumana okuhambisanayo.

Ngincoma ukusebenzisa ama-CNI alandelayo kuye ngesimo:

• Ingabe unayo eqenjini lakho ama-node anezinsiza ezimbalwa (ama-GB amaningana we-RAM, ama-cores amaningana) futhi awudingi izici zokuphepha - khetha Flannel. Lena enye ye-CNI engabizi kakhulu. Futhi iyahambisana nezinhlobonhlobo zezakhiwo (i-amd64, ingalo, i-arm64, njll.). Ngaphezu kwalokho, lokhu kungenye yezibili (enye i-Cilium) i-CNI engakwazi ukunquma ngokuzenzakalelayo i-MTU, ngakho akudingeki ulungise noma yini. I-Kube-router nayo ifanelekile, kodwa ayiyona indinganiso futhi uzodinga ukumisa ngesandla i-MTU.
• Uma kudingeka bethela inethiwekhi ukuze uphephe, thatha I-WeaveNet. Ungakhohlwa ukucacisa usayizi we-MTU uma usebenzisa amafreyimu e-jumbo, futhi unike amandla ukubethela ngokucacisa igama-mfihlo ngokushintshashintsha kwendawo. Kodwa kungcono ukukhohlwa mayelana nokusebenza - lokho kuyizindleko zokubethela.
• Ukuze ukusetshenziswa okuvamile Ngiyeluleka UCalico. Le CNI isetshenziswa kabanzi kumathuluzi ahlukahlukene wokusabalalisa we-Kubernetes (Kops, Kubespray, Rancher, njll.). Njenge-WeaveNet, qiniseka ukuthi ulungiselela i-MTU ku-ConfigMap uma usebenzisa amafreyimu e-jumbo. Kuyithuluzi elinemisebenzi eminingi elisebenza kahle mayelana nokusetshenziswa kwezinsiza, ukusebenza nokuphepha.

Futhi ekugcineni, ngikweluleka ukuthi ulandele ukuthuthukiswa I-Cilium. Le CNI ineqembu elikhuthele kakhulu elisebenza kakhulu kumkhiqizo wabo (izici, ukonga izinsiza, ukusebenza, ukuphepha, ukuhlanganisa...) futhi banezinhlelo ezithakazelisa kakhulu.

Umdwebo obonakalayo wokukhethwa kwe-CNI

Source: www.habr.com