Kubernetes Network Plugin (CNI) Benchmark Results over 10 Gbps Network (Zosinthidwa: April 2019)

Izi ndi zosintha zanga benchmark yam'mbuyomu, yomwe tsopano ikuyenda Kubernetes 1.14 ndi mtundu waposachedwa wa CNI kuyambira Epulo 2019.

Choyamba, ndikufuna kuthokoza gulu la Cilium: anyamatawo anandithandiza kuyang'ana ndi kukonza zolemba zowunikira ma metrics.

Zomwe zasintha kuyambira Novembara 2018

Nazi zomwe zasintha kuyambira pamenepo (ngati mukufuna):

Flannel imakhalabe mawonekedwe othamanga kwambiri komanso osavuta a CNI, komabe sichigwirizana ndi mfundo zama network ndi kubisa.

Romana sakuthandizidwanso, chifukwa chake tachotsa pa benchmark.

WeaveNet tsopano imathandizira mfundo za netiweki za Ingress ndi Egress! Koma zokolola zachepa.

Ku Calico, mukufunikabe kukonza pamanja kukula kwake kwa paketi (MTU) kuti mugwire bwino ntchito. Calico imapereka njira ziwiri zoyika CNI, kotero mutha kuchita popanda chosungira chapadera cha ETCD:

• kusunga boma mu Kubernetes API ngati sitolo ya data (cluster size <50 nodes);
• kusunga boma mu Kubernetes API ngati sitolo ya data ndi Typha proxy kuti athetse katundu pa K8S API (kukula kwamagulu> 50 nodes).

Calico adalengeza thandizo ndondomeko za msinkhu wa ntchito pamwamba pa Istio pachitetezo cha mulingo wogwiritsa ntchito.

Cilium tsopano imathandizira kubisa! Cilium imapereka ma encryption ndi ma IPSec tunnel ndipo imapereka njira ina yolumikizira netiweki ya WeaveNet. Koma WeaveNet ndiyothamanga kuposa Cilium yokhala ndi encryption yothandizidwa.

Cilium tsopano ndiyosavuta kugwiritsa ntchito chifukwa cha opangira ETCD.

Gulu la Cilium layesera kuchepetsa kulemera kwake kuchokera ku CNI yake pochepetsa kukumbukira kukumbukira ndi mtengo wa CPU, koma opikisana nawo akadali opepuka.

Benchmark nkhani

Benchmark imayendetsedwa pa maseva atatu osagwiritsa ntchito a Supermicro okhala ndi 10 Gb Supermicro switch. Ma seva amalumikizidwa mwachindunji ndi switch kudzera pazingwe za DAC SFP + ndipo amakonzedwa pa VLAN yomweyo yokhala ndi mafelemu a jumbo (MTU 9000).

Kubernetes 1.14.0 yoyikidwa pa Ubuntu 18.04 LTS yokhala ndi Docker 18.09.2 (yosasinthika Docker mtundu pakumasulidwa uku).

Kuti tipititse patsogolo kuberekana, tinaganiza zokonzekera mbuye pa node yoyamba, ikani seva gawo la benchmark pa seva yachiwiri, ndi gawo la kasitomala pa lachitatu. Kuti tichite izi, timagwiritsa ntchito NodeSelector mu Kubernetes deployments.

Tifotokoza zotsatira za benchmark pamlingo wotsatirawu:

Kusankha CNI kwa benchmark

Ichi ndi benchmark ya CNI yokha kuchokera pamndandanda wagawoli za kupanga gulu limodzi la master ndi kubeadm Onani zolemba zovomerezeka za Kubernetes. Pa 9 CNIs, tidzatenga 6 yokha: tidzapatula omwe ali ovuta kukhazikitsa ndi / kapena osagwira ntchito popanda kusintha malinga ndi zolemba (Romana, Contiv-VPP ndi JuniperContrail / TungstenFabric).

Tikufananiza ma CNI otsatirawa:

• Calico v3.6
• Canal v3.6 (makamaka Flannel ya network + Calico ngati chowotcha moto)
• Cilium 1.4.2
• Flannel 0.11.0
• Kube-rauta 0.2.5
• WeaveNet 2.5.1

kolowera

Kusavuta kwa CNI kuyika, m'pamenenso malingaliro athu oyamba adzakhala abwino. Ma CNI onse pa benchmark ndiosavuta kukhazikitsa (ndi lamulo limodzi kapena awiri).

Monga tanenera, ma seva ndi kusinthana zimakonzedwa ndi mafelemu a jumbo (tidayika MTU ku 9000). Tingakhale okondwa ngati CNI ingodzitsimikizira MTU potengera kasinthidwe ka ma adapter. Komabe, Cilium ndi Flannel okha ndi omwe adakwanitsa izi. Ma CNI ena onse ali ndi zopempha pa GitHub kuti awonjezere kupezeka kwa MTU, koma tidzakonza pamanja posintha ConfigMap ya Calico, Canal ndi Kube-router, kapena kudutsa kusintha kwa chilengedwe kwa WeaveNet.

Vuto ndi MTU yolakwika ndi chiyani? Chithunzichi chikuwonetsa kusiyana pakati pa WeaveNet yokhala ndi MTU yosasinthika ndi mafelemu a jumbo omwe adayatsidwa:

Kodi MTU imagwira ntchito bwanji?

Tawona kufunikira kwa MTU pakuchita bwino, tsopano tiyeni tiwone momwe ma CNIs athu amadziwira okha:

CNI imazindikira MTU yokha

Chithunzichi chikuwonetsa kuti muyenera kukonza MTU ya Calico, Canal, Kube-router ndi WeaveNet kuti mugwire bwino ntchito. Cilium ndi Flannel adatha kudziwa bwino MTU okha popanda zoikamo.

Chitetezo

Tidzafanizira chitetezo cha CNI m'magawo awiri: kuthekera kosunga deta yofalitsidwa ndikukhazikitsa mfundo za network ya Kubernetes (kutengera mayeso enieni, osati zolemba).

Ma data awiri okha a CNIs encrypt: Cilium ndi WeaveNet. Kubisa WeaveNet yambitsani pokhazikitsa mawu achinsinsi achinsinsi ngati CNI environment variable. MU zolemba WeaveNet imafotokoza m'njira yovuta, koma zonse zimachitika mosavuta. Kubisa cilium kukhazikitsidwa ndi malamulo, popanga zinsinsi za Kubernetes, ndi kusinthidwa kwa daemonSet (zovuta pang'ono kuposa WeaveNet, koma Cilium ili ndi sitepe ndi sitepe malangizo).

Ponena za kukhazikitsidwa kwa ndondomeko ya maukonde, apambana Calico, Canal, Cilium ndi WeaveNet, momwe mungakhazikitsire malamulo a Ingress ndi Egress. Za Kube-router pali malamulo a Ingress okha, ndi Flannel Palibe ndondomeko zapaintaneti konse.

Nazi zotsatira zonse:

Zotsatira za Benchmark ya Chitetezo

Kukonzekera

Benchmark iyi ikuwonetsa kuchuluka kwa zomwe zachitika pamayesero osachepera atatu pa mayeso aliwonse. Timayesa machitidwe a TCP ndi UDP (pogwiritsa ntchito iperf3), mapulogalamu enieni monga HTTP (ndi Nginx ndi curl) kapena FTP (yokhala ndi vsftpd ndi curl) ndipo potsiriza ntchito yogwiritsira ntchito pogwiritsa ntchito SCP-based encryption (pogwiritsa ntchito kasitomala ndi seva OpenSSH).

Pamayeso onse, tidapanga benchmark yachitsulo yopanda kanthu (mzere wobiriwira) kuti tifanizire magwiridwe antchito a CNI ndi magwiridwe antchito amtaneti. Apa timagwiritsa ntchito sikelo yofanana, koma yamtundu:

• Yellow = zabwino kwambiri
• Orange = zabwino
• Buluu = kotero-chakuti
• Red = zoipa

Sititenga ma CNI osinthidwa molakwika ndipo tidzangowonetsa zotsatira za CNIs ndi MTU yolondola. (Zindikirani: Cilium samawerengera MTU molondola ngati mutsegula kubisa, kotero muyenera kuchepetsa MTU pamanja mpaka 8900 mu mtundu 1.4. Mtundu wotsatira, 1.5, umachita izi zokha.)

Nazi zotsatira:

Kuchita kwa TCP

Ma CNI onse adachita bwino mu benchmark ya TCP. CNI yokhala ndi encryption imatsalira kwambiri chifukwa kubisa ndi okwera mtengo.

Kuchita bwino kwa UDP

Apanso, ma CNI onse akuchita bwino. CNI yokhala ndi encryption idawonetsa zotsatira zofanana. Cilium ndi pang'ono kumbuyo kwa mpikisano, koma ndi 2,3% yokha yazitsulo zopanda kanthu, kotero si zotsatira zoipa. Musaiwale kuti Cilium ndi Flannel okha adatsimikiza MTU molondola, ndipo izi ndi zotsatira zawo popanda kusinthidwa kwina kulikonse.

Nanga bwanji kugwiritsa ntchito kwenikweni? Monga mukuwonera, magwiridwe antchito onse a HTTP ndiotsika pang'ono kuposa a TCP. Ngakhale mutagwiritsa ntchito HTTP ndi TCP, tinakonza iperf3 mu benchmark ya TCP kuti tipewe kuyamba pang'onopang'ono komwe kungakhudze benchmark ya HTTP. Aliyense anachita ntchito yabwino pano. Kube-router ili ndi mwayi wowonekera, koma WeaveNet sinachite bwino: pafupifupi 20% yoyipa kuposa chitsulo chopanda kanthu. Cilium ndi WeaveNet okhala ndi encryption amawoneka achisoni kwambiri.

Ndi FTP, protocol ina yochokera ku TCP, zotsatira zimasiyana. Flannel ndi Kube-router zimagwira ntchitoyo, koma Calico, Canal ndi Cilium ndizotsalira pang'ono ndipo zimakhala pang'onopang'ono 10% kuposa zitsulo zopanda kanthu. WeaveNet ili kumbuyo ndi 17%, koma WeaveNet yosungidwa ndi 40% patsogolo pa Cilium yosungidwa.

Ndi SCP titha kuwona nthawi yomweyo kuti kubisa kwa SSH kumatitengera ndalama zingati. Pafupifupi ma CNI onse akuchita bwino, koma WeaveNet ikutsaliranso. Cilium ndi WeaveNet zokhala ndi encryption zikuyembekezeka kukhala zoyipa kwambiri chifukwa cha kubisa kawiri (SSH + CNI).

Nali tebulo lachidule lomwe lili ndi zotsatira zake:

Kugwiritsa ntchito zinthu

Tsopano tiyeni tifanizire momwe CNI imagwiritsira ntchito chuma pansi pa katundu wolemera (panthawi ya TCP, 10 Gbps). M'mayesero ogwira ntchito timafanizira CNI ndi chitsulo chopanda kanthu (mzere wobiriwira). Pakugwiritsa ntchito zida, tiyeni tiwonetse Kubernetes (mzere wofiirira) wopanda CNI ndikuwona kuchuluka kwazinthu zowonjezera zomwe CNI imawononga.

Tiyeni tiyambe ndi kukumbukira. Nayi mtengo wapakati wa node 'RAM (kupatula ma buffers ndi cache) mu MB pakusamutsa.

Kugwiritsa ntchito kukumbukira

Flannel ndi Kube-rauta adawonetsa zotsatira zabwino kwambiri - 50 MB yokha. Calico ndi Canal iliyonse ili ndi 70. WeaveNet imadya momveka bwino kuposa ena - 130 MB, ndipo Cilium imagwiritsa ntchito mpaka 400.
Tsopano tiyeni tiwone momwe CPU ikugwiritsira ntchito nthawi. Zochititsa chidwi: chithunzicho sichikuwonetsa maperesenti, koma ppm, ndiko kuti, 38 ppm ya "chitsulo chopanda kanthu" ndi 3,8%. Nazi zotsatira:

Kugwiritsa ntchito CPU

Calico, Canal, Flannel ndi Kube-router ndiwothandiza kwambiri CPU - 2% yokha kuposa Kubernetes popanda CNI. WeaveNet imatsalira kumbuyo ndi 5% yowonjezera, kutsatiridwa ndi Cilium pa 7%.

Nachi chidule cha kagwiritsidwe ntchito ka zinthu:

Zotsatira

Tebulo ndi zotsatira zonse:

Zotsatira za benchmark

Pomaliza

Mu gawo lotsiriza ine ndifotokoza maganizo anga subjective pa zotsatira. Kumbukirani kuti benchmark iyi imangoyesa kutulutsa kwa kulumikizana kumodzi pagulu laling'ono kwambiri (ma node atatu). Sichikugwira ntchito kumagulu akuluakulu (<3 nodes) kapena maulumikizano ofanana.

Ndikupangira kugwiritsa ntchito ma CNI otsatirawa kutengera zomwe zikuchitika:

• Kodi muli ndi gulu lanu mfundo ndi zopezera zochepa (ma GB angapo a RAM, ma cores angapo) ndipo simufunikira zida zachitetezo - sankhani Flannel. Ichi ndi chimodzi mwa CNIs otsika mtengo kwambiri. Ndipo imagwirizana ndi zomangamanga zosiyanasiyana (amd64, mkono, arm64, etc.). Kuphatikiza apo, iyi ndi imodzi mwa ziwiri (winayo ndi Cilium) CNI yomwe imatha kudziwa MTU yokha, kotero simuyenera kukonza chilichonse. Kube-rauta ndiyoyeneranso, koma sizofanana ndipo muyenera kukonza pamanja MTU.
• Ngati kuli kofunikira encrypt network kwa chitetezo, tenga WeaveNet. Musaiwale kufotokoza kukula kwa MTU ngati mukugwiritsa ntchito mafelemu a jumbo, ndikuthandizira kubisa potchula mawu achinsinsi kudzera pakusintha kwachilengedwe. Koma ndi bwino kuiwala za ntchito - ndiye mtengo wa kubisa.
• chifukwa kugwiritsa ntchito bwino Ndikulangiza Kalico. CNI iyi imagwiritsidwa ntchito kwambiri pazida zosiyanasiyana zotumizira Kubernetes (Kops, Kubespray, Rancher, etc.). Monga ndi WeaveNet, onetsetsani kuti mwakonza MTU mu ConfigMap ngati mukugwiritsa ntchito mafelemu a jumbo. Ndi chida chamitundu yambiri chomwe chimagwira ntchito bwino pakugwiritsa ntchito zida, magwiridwe antchito ndi chitetezo.

Ndipo potsiriza, ndikukulangizani kuti muzitsatira chitukuko cilium. CNI iyi ili ndi gulu logwira ntchito kwambiri lomwe limagwira ntchito kwambiri pazogulitsa zawo (zina, kusungirako zinthu, ntchito, chitetezo, kusonkhanitsa ...) ndipo ali ndi mapulani okondweretsa kwambiri.

Chithunzi chojambula chosankha CNI

Source: www.habr.com