Kubernetes Network Plugin (CNI) Benchmark Mhinduro pamusoro pe10 Gbps Network (Yakagadziridzwa: Kubvumbi 2019)

Iyi ndiyo update yangu yapfuura benchmark, iyo ikozvino inomhanya paKubernetes 1.14 ine yazvino CNI vhezheni kubva muna Kubvumbi 2019.

Chokutanga pane zvose, ndinoda kutenda boka reCilium: vakomana vakandibatsira kutarisa nekugadzirisa zvinyorwa zvekuongorora metrics.

Chii chakachinja kubva munaNovember 2018

Hezvino izvo zvakashandurwa kubva ipapo (kana iwe uchifarira):

Flannel inoramba iri inokurumidza uye yakapusa CNI interface, asi zvakadaro haitsigire network marongero uye encryption.

Romana haisisiri kutsigirwa, saka takaibvisa kubva pabhenji.

WeaveNet ikozvino inotsigira network marongero eIngress uye Egress! Asi kubereka kwadzikira.

MuCalico, iwe uchiri kuda kugadzirisa nemaoko saizi yepamusoro yepakiti (MTU) yekuita zvakanaka. Calico inopa sarudzo mbiri dzekuisa CNI, saka unogona kuita pasina yakaparadzana ETCD repository:

• kuchengetedza mamiriro muKubernetes API sechitoro che data (saizi yeboka <50 nodes);
• kuchengetedza mamiriro muKubernetes API sechitoro chedhata neTypha proxy yekubvisa mutoro paK8S API (saizi yesumbu> 50 node).

Calico yakazivisa rutsigiro application level policy pamusoro peIstio kune application-level chengetedzo.

Cilium ikozvino inotsigira encryption! Cilium inopa encryption neIPSec tunnels uye inopa imwe nzira kune yakavharidzirwa WeaveNet network. Asi WeaveNet inokurumidza kupfuura Cilium ine encryption inogoneswa.

Cilium ikozvino yave nyore kuendesa kuvonga kune yakavakirwa-mukati ETCD opareta.

Chikwata cheCilium chakaedza kutema huremu kubva kuCNI yayo nekudzikisa ndangariro yekushandisa uye mutengo weCPU, asi vakwikwidzi vayo vachiri kureruka.

Benchmark context

Iyo bhenji inomhanyisa pane matatu asina-virtualized Supermicro maseva ane 10 Gb Supermicro switch. Masevha akabatana zvakananga kune switch kuburikidza neasina DAC SFP + tambo uye akagadziridzwa pane imwecheteyo VLAN ine jumbo mafuremu (MTU 9000).

Kubernetes 1.14.0 yakaiswa paUbuntu 18.04 LTS ine Docker 18.09.2 (iyo yakasarudzika Docker vhezheni mukuburitswa uku).

Kuti tivandudze kuberekana, takasarudza kugara tichigadzirisa tenzi pane yekutanga node, isa sevha chikamu chebhenji pane yechipiri sevha, uye chikamu chemutengi pane chechitatu. Kuti tiite izvi, isu tinoshandisa NodeSelector muKubernetes deployments.

Isu tichatsanangura mabhenji mhedzisiro pachiyero chinotevera:

Kusarudza CNI yekuenzanisa

Ichi chiratidzo cheCNI chete kubva pane rondedzero iri muchikamu nezve kugadzira imwe master cluster ne kubeadm Ona zviri pamutemo Kubernetes zvinyorwa. Pakati pe9 CNIs, tichatora 6 chete: isu tichabvisa izvo zvakaoma kuisa uye / kana kusashanda pasina kugadzirisa maererano nezvinyorwa (Romana, Contiv-VPP uye JuniperContrail/TungstenFabric).

Isu tichafananidza zvinotevera CNIs:

• Calico v3.6
• Canal v3.6 (chaizvoizvo Flannel ye networking + Calico se firewall)
• Cilium 1.4.2
• Flannel 0.11.0
• Kube-router 0.2.5
• WeaveNet 2.5.1

Kuiswa

Izvo zviri nyore iyo CNI kuisa, zviri nani maonero edu ekutanga. Ese maCNIs kubva pabhenji ari nyore kwazvo kuisa (nemirairo imwe kana miviri).

Sezvatakataura, maseva uye switch inogadziriswa nejumbo mafuremu anogoneswa (isu tinoisa iyo MTU ku9000). Isu taizofara kana CNI yakazvisarudzira iyo MTU zvichienderana nekugadziriswa kweadapter. Zvisinei, Cilium neFlannel chete ndivo vakakwanisa izvi. Mamwe ese maCNI ane zvikumbiro paGitHub yekuwedzera otomatiki MTU kuwanikwa, asi isu tichazvigadzirisa nemaoko nekushandura ConfigMap yeCalico, Canal uye Kube-router, kana kupfuura nharaunda inoshanduka yeWeaveNet.

Dambudziko nderipi neMTU isiriyo? Dhiagiramu iyi inoratidza mutsauko uripo pakati peWeaveNet ine default MTU uye jumbo mafuremu anogoneswa:

Ko MTU inokanganisa sei kubuda?

Isu taona kukosha kweMTU pakuita, ikozvino ngationei kuti maCNI edu anozviona sei:

CNI inoona otomatiki MTU

Girafu inoratidza kuti iwe unofanirwa kugadzirisa iyo MTU yeCalico, Canal, Kube-router uye WeaveNet kuti iite zvakakwana. Cilium neFlannel vakakwanisa kunyatsoona iyo MTU ivo pachavo pasina chero marongero.

Chengetedzo

Isu tichafananidza kuchengetedzeka kweCNI muzvikamu zviviri: kugona kuvharidzira data rakafambiswa uye kuita kweKubernetes network marongero (zvichienderana nemiedzo chaiyo, kwete zvinyorwa).

Maviri chete CNIs encrypt data: Cilium uye WeaveNet. Encryption WeaveNet inogoneswa nekuisa iyo encryption password seCNI nharaunda inoshanduka. IN zvinyorwa WeaveNet inoitsanangura nenzira yakaoma, asi zvese zvinoitwa zviri nyore. Encryption cilium yakagadziridzwa nemirairo, nekugadzira Kubernetes zvakavanzika, uye kuburikidza nekugadziriswa kwedaemonSet (yakanyanya kuomarara kupfuura muWeaveNet, asi Cilium ine nhanho-ne-nhanho. mirayiridzo).

Kana zviri zvekushandiswa kwenetwork policy, vakabudirira Calico, Canal, Cilium uye WeaveNet, iyo yaunogona kugadzirisa Ingress uye Egress mitemo. For Kube-router kune mitemo chete yeIngress, uye Flannel Iko hakuna network policy policy zvachose.

Heino mibairo yese:

Chengetedzo Yekuita Benchmark Mibairo

Kubudirira

Iyi bhenji inoratidza avhareji yekubuda pamusoro peanokwana matatu anomhanya bvunzo yega yega. Isu tinoedza mashandiro eTCP uye UDP (tichishandisa iperf3), zvikumbiro chaizvo seHTTP (ine Nginx uye curl) kana FTP (ine vsftpd uye curl) uye pakupedzisira kuita kwekushandisa uchishandisa SCP-based encryption (uchishandisa mutengi uye server OpenSSH).

Kune ese bvunzo, isu takaita isina simbi bhenji (yegirini mutsara) kuenzanisa kuita kweCNI nemaitiro emuno network. Pano tinoshandisa chiyero chimwe chete, asi muvara:

• Yero = yakanaka kwazvo
• Orange = yakanaka
• Bhuruu = saka-saizvozvo
• Tsvuku = zvakaipa

Hatizotore maCNI akarongedzerwa zvisizvo uye tichangoratidza mhinduro dzeCNIs neMTU chaiyo. (Cherechedza: Cilium haina kuverengera MTU nemazvo kana ukagonesa encryption, saka uchafanirwa kudzikisira nemaoko MTU kusvika 8900 mushanduro 1.4. Iyo inotevera vhezheni, 1.5, inozviita otomatiki.)

Heino mibairo:

TCP Performance

Ese maCNI akaita zvakanaka muTCP bhenji. CNI ine encryption inosarira kumashure nekuti encryption inodhura.

UDP performance

Pano, zvakare, ese maCNIs ari kuita zvakanaka. CNI ine encryption yakaratidza mhedzisiro yakafanana. Cilium ishoma kuseri kwemakwikwi, asi inongova 2,3% yesimbi isina chinhu, saka haisi mhedzisiro yakaipa. Usakanganwa kuti Cilium neFlannel chete ndivo vakasarudza MTU nemazvo ivo pachavo, uye izvi ndizvo mhedzisiro yavo pasina kumwe kugadziridzwa.

Zvakadini nekushandisa chaiko? Sezvauri kuona, kuita kwese kweHTTP kwakadzikira zvishoma pane kweTCP. Kunyangwe ukashandisa HTTP neTCP, takagadzira iperf3 muTCP benchmark kudzivirira kutangisa kunonokera kunozokanganisa HTTP benchmark. Munhu wese akaita basa rakanaka pano. Kube-router ine mukana wakajeka, asi WeaveNet haina kuita zvakanaka: inenge 20% yakaipisisa kupfuura simbi isina chinhu. Cilium uye WeaveNet ine encryption inotaridzika kusiririsa.

NeFTP, imwe TCP-based protocol, mhedzisiro inosiyana. Flannel neKube-router vanoita basa, asi Calico, Canal neCilium zviri kumashure zvishoma uye zvinononoka ne10% pane simbi isina chinhu. WeaveNet iri kumashure nekusvika gumi nenomwe muzana, asi yakavharidzirwa WeaveNet iri 17% pamberi peCilium yakavharidzirwa.

Ne SCP isu tinokwanisa kuona ipapo kuti yakawanda sei SSH encryption inotidhurira. Anenge ese maCNI ari kuita zvakanaka, asi WeaveNet iri kusarira kumashure zvakare. Cilium uye WeaveNet ine encryption inotarisirwa yakaipisisa nekuda kwekaviri encryption (SSH + CNI).

Heino tafura yepfupiso ine zvabuda:

Resource kushandiswa

Zvino ngatienzanisei kuti CNI inoshandisa sei zviwanikwa pasi pemitoro inorema (panguva yeTCP kutamiswa, 10 Gbps). Mukuedzwa kwekuita tinofananidza CNI nesimbi isina chinhu (green line). Nekushandisa zviwanikwa, ngatiratidze Kubernetes yakachena (yepepuru mutsara) isina CNI uye tione kuti mangani ekuwedzera zviwanikwa zvinodyiwa neCNI.

Ngatitange nendangariro. Heino avhareji kukosha kwemanodhi 'RAM (kusanganisa mabuffers uye cache) muMB panguva yekufambisa.

Memory kushandiswa

Flannel uye Kube-router yakaratidza mhedzisiro yakanaka - 50 MB chete. Calico neCanal imwe neimwe ine makumi manomwe. WeaveNet inodya zvakanyanya kupfuura mamwe - 70 MB, uye Cilium inoshandisa yakawanda se130.
Zvino ngatitarisei CPU nguva yekushandisa. Inokosha: dhayagiramu inoratidza kwete mapeji, asi ppm, kureva, 38 ppm ye "simbi isina chinhu" ndeye 3,8%. Heino mibairo:

CPU kushandiswa

Calico, Canal, Flannel uye Kube-router zvakanyanya CPU inoshanda - chete 2% kupfuura Kubernetes isina CNI. WeaveNet inosarira kumashure ne5% yakawedzerwa, ichiteverwa neCilium pa7%.

Heino chidimbu chekushandisa zviwanikwa:

Migumisiro

Tafura ine zvese zvabuda:

General benchmark results

mhedziso

Muchikamu chekupedzisira ndichataura maonero angu ega ega pane zvabuda. Rangarira kuti iyi bhenji inoyedza chete kubuda kwekubatanidza kumwe chete pane chidiki chidiki (3 nodes). Izvo hazvishande kune makuru masumbu (<50 nodes) kana parallel connections.

Ini ndinokurudzira kushandisa inotevera CNIs zvichienderana nemamiriro ezvinhu:

• Une muchikwata chako here node dzine zviwanikwa zvishoma (anoverengeka GB ye RAM, akati wandei cores) uye iwe haudi kuchengetedza maficha - sarudza Flannel. Iyi ndeimwe yeCNIs inodhura zvakanyanya. Uye inoenderana neakasiyana siyana ezvivakwa (amd64, ruoko, arm64, nezvimwewo). Mukuwedzera, iyi ndeimwe yezviviri (imwe yacho Cilium) CNI iyo inogona kungoona MTU, saka haufaniri kugadzirisa chero chinhu. Kube-router yakakodzerawo, asi haina kufanana uye iwe uchafanirwa kugadzirisa nemaoko MTU.
• Kana zvichidiwa encrypt network kuitira kuchengeteka, tora WeaveNet. Usakanganwe kudoma saizi yeMTU kana uri kushandisa jumbo mafuremu, uye gonesa encryption nekutsanangura password kuburikidza neyakasiyana siyana. Asi zviri nani kukanganwa nezvekuita - ndiyo mutengo we encryption.
• nokuti kushandiswa kwakajairika Ndinopa zano Calico. Iyi CNI inoshandiswa zvakanyanya mumhando dzakasiyana dzeKubernetes dzekutumira (Kops, Kubespray, Rancher, nezvimwewo). Sezvakaita WeaveNet, ita shuwa kugadzirisa iyo MTU muConfigMap kana uchishandisa jumbo mafuremu. Icho chishandiso chinoshanda-chakawanda chinoshanda maererano nekushandisa zviwanikwa, kuita uye kuchengetedza.

Uye pakupedzisira, ini ndinokupa zano kuti uteedzere budiriro cilium. Iyi CNI ine boka rinoshanda rinoshanda zvakanyanya pachigadzirwa chavo (maficha, kuchengetedza zviwanikwa, kuita, chengetedzo, kubatanidza ...) uye vane zvirongwa zvinonakidza kwazvo.

Dhiyagiramu yekuona yekusarudza kweCNI

Source: www.habr.com