TL; DR: Txhua CNIs ua haujlwm raws li lawv yuav tsum tau ua, tshwj tsis yog Kube-Router thiab Kube-OVN, Calico, tshwj tsis yog kev kuaj pom tsis siv neeg MTU, yog qhov zoo tshaj plaws.
Tsab xov xwm-hloov tshiab ntawm kuv daim tshev dhau los ( ΠΈ ), thaum lub sijhawm kuaj kuv tab tom siv Kubernetes 1.19 ntawm Ubuntu 18.04 nrog kho tshiab CNIs thaum Lub Yim Hli 2020.
Ua ntej peb dhia mus rau hauv metrics ...
Dab tsi tshiab txij lub Plaub Hlis 2019?
- Koj tuaj yeem sim ntawm koj tus kheej pawg: Koj tuaj yeem khiav cov kev xeem ntawm koj tus kheej pawg siv peb lub cuab yeej Kubernetes Network Benchmark:
- Cov tswv cuab tshiab tau tshwm sim
- los ntawm VMware Tanzu
- los ntawm alauda.io
- Cov xwm txheej tshiab: Cov kev kuaj tam sim no khiav "Pod-to-Pod" kev ntsuam xyuas kev ua haujlwm hauv network, thiab cov ntawv tshiab "Pod-to-Service" tau ntxiv uas khiav cov kev sim ze dua rau lub ntiaj teb tiag tiag. Hauv kev xyaum, koj lub Pod nrog API ua haujlwm nrog lub hauv paus ua ib qho kev pabcuam, thiab tsis yog los ntawm Pod ip chaw nyob (tau kawg peb xyuas ob qho tib si TCP thiab UDP rau ob qho xwm txheej).
- Kev siv peev txheej: txhua qhov kev sim tam sim no muaj nws tus kheej cov peev txheej sib piv
- Tshem Tawm Daim Ntawv Thov Kev Xeem: Peb tsis ua HTTP, FTP thiab SCP cov kev ntsuam xyuas ntxiv lawm raws li peb cov txiaj ntsig kev koom tes nrog cov zej zog thiab CNI cov neeg saib xyuas tau nthuav tawm qhov sib txawv ntawm cov txiaj ntsig iperf dhau TCP thiab curl cov txiaj ntsig vim qhov qeeb hauv CNI pib (thawj ob peb feeb ntawm Pod pib, uas tsis raug rau hauv cov xwm txheej tiag).
- Qhib qhov chaw: tag nrho cov ntawv xeem (scripts, yml nqis thiab thawj "raw" cov ntaub ntawv) muaj
Reference Test Protocol
Cov txheej txheem tau piav qhia meej Thov nco ntsoov tias tsab xov xwm no yog hais txog Ubuntu 18.04 nrog lub default kernel.
Xaiv CNI rau Kev Ntsuas
Qhov kev sim no yog tsom rau kev sib piv CNIs teeb tsa nrog ib daim ntawv yaml (yog li ntawd, tag nrho cov uas tau teeb tsa los ntawm cov ntawv sau, xws li VPP thiab lwm yam, tsis suav nrog).
Peb xaiv CNIs rau kev sib piv:
- Antrea v.0.9.1
- Calico v3.16
- Canal v3.16 (Flannel network + Calico Network Txoj Cai)
- Cilium 1.8.2
- Flannel 0.12.0
- Kube-router tshiab (2020-08-25)
- WeaveNet 2.7.0
Configuring MTU rau CNI
Ua ntej tshaj plaws, peb tshawb xyuas qhov cuam tshuam ntawm kev kuaj pom tsis siv neeg MTU ntawm TCP kev ua haujlwm:
Kev cuam tshuam ntawm MTU ntawm TCP Performance
Qhov sib txawv loj dua yog pom thaum siv UDP:
Kev cuam tshuam ntawm MTU ntawm UDP Performance
Muab qhov HUGE kev ua tau zoo tshwm sim hauv qhov kev sim, peb xav xa tsab ntawv ntawm kev cia siab rau txhua tus neeg saib xyuas CNI: thov ntxiv qhov tsis siv neeg MTU tshawb pom rau CNI. Koj yuav cawm tau kittens, unicorns thiab txawm tus cutest ib: tus me Devop.
Txawm li cas los xij, yog tias koj xav siv CNI yam tsis muaj kev txhawb nqa rau kev tshawb nrhiav MTU tsis siv neeg, koj tuaj yeem teeb tsa nws manually kom tau txais kev ua haujlwm. Thov nco ntsoov tias qhov no siv rau Calico, Canal thiab WeaveNet.
Kuv qhov kev thov me me rau CNIs nrog ...
CNI Test: Raw Data
Hauv seem no, peb yuav muab piv rau CNI nrog qhov tseeb MTU (txiav txim siab los yog teeb tsa manually). Lub hom phiaj tseem ceeb ntawm no yog qhia cov ntaub ntawv nyoos hauv cov duab.
Xim lus dab neeg:
- grey - qauv (piv txwv li liab qab hlau)
- ntsuab - bandwidth saum 9500 Mbps
- daj - bandwidth saum 9000 Mbps
- txiv kab ntxwv - bandwidth saum 8000 Mbps
- liab - bandwidth hauv qab 8000 Mbps
- xiav - nruab nrab (tsis cuam tshuam nrog bandwidth)
Tsis siv neeg thauj khoom
Ua ntej tshaj plaws, tshawb xyuas cov peev txheej siv thaum pawg "pw tsaug zog".
Tsis siv neeg thauj khoom
Pod-rau-Pod
Qhov xwm txheej no xav tias tus neeg siv khoom Pod txuas ncaj qha rau lub server Pod siv nws qhov chaw nyob IP.
Pod-to-Pod Scenario
TCP
Pod-to-Pod TCP cov txiaj ntsig thiab kev siv peev txheej:
UDP
Pod-to-Pod UDP cov txiaj ntsig thiab cov peev txheej siv tau:
Pod-rau-Service
Tshooj lus no muaj feem cuam tshuam rau kev siv tiag tiag, tus neeg siv khoom Pod txuas rau lub server Pod ntawm ClusterIP kev pabcuam.
Pod-to-Service Script
TCP
Pod-to-Service TCP cov txiaj ntsig thiab kev siv cov peev txheej sib xws:
UDP
Pod-to-Service UDP cov txiaj ntsig thiab siv cov peev txheej sib xws:
Network txoj cai txhawb nqa
Ntawm tag nrho cov saum toj no, tsuas yog ib qho uas tsis txhawb kev nom kev tswv yog Flannel. Tag nrho lwm tus ua raws li cov cai hauv network, suav nrog kev nkag mus thiab tawm mus. Txoj hauj lwm zoo!
CNI encryption
Ntawm cov tshuaj ntsuam CNIs muaj cov uas tuaj yeem encrypt network pauv ntawm Pods:
- Antrea siv IPsec
- Calico siv wireguard
- Cilium siv IPsec
- WeaveNet siv IPsec
Bandwidth
Txij li thaum muaj tsawg CNIs sab laug, cia peb muab tag nrho cov xwm txheej rau hauv ib daim duab:
Kev siv peev txheej
Hauv seem no, peb yuav ntsuas cov peev txheej siv thaum ua Pod-to-Pod kev sib txuas lus hauv TCP thiab UDP. Tsis muaj qhov taw tes hauv kev kos duab Pod-to-Service vim nws tsis muab cov ntaub ntawv ntxiv.
Muab tag nrho ua ke
Cia peb sim rov ua txhua daim duab, peb tau qhia me ntsis kev kawm ntawm no, hloov cov txiaj ntsig tiag tiag nrog cov lus "vwry nrawm", "tsawg", thiab lwm yam.
Qhov xaus thiab kuv cov lus xaus
Qhov no yog ib tug me ntsis subjective, txij li thaum kuv tab tom qhia kuv tus kheej txhais lus ntawm cov txiaj ntsig.
Kuv zoo siab tias CNIs tshiab tau tshwm sim, Antrea ua tau zoo, ntau lub zog tau ua tiav txawm tias nyob rau hauv cov qauv thaum ntxov: tsis siv neeg MTU nrhiav pom, encryption thiab yooj yim installation.
Yog tias peb piv kev ua tau zoo, txhua CNIs ua haujlwm zoo, tshwj tsis yog Kube-OVN thiab Kube-Router. Kube-Router kuj tseem tsis tuaj yeem ntes MTU, Kuv tsis pom txoj hauv kev los teeb tsa nws nyob qhov twg hauv cov ntaub ntawv ( qhov kev thov ntawm lub ncauj lus no tau qhib).
Hais txog kev siv cov peev txheej, Cilium tseem siv RAM ntau dua li lwm tus, tab sis cov chaw tsim khoom tau tsom mus rau cov pawg loj, uas yog qhov tseeb tsis zoo ib yam li kev sim ntawm peb-node pawg. Kube-OVN kuj tseem siv CPU thiab RAM ntau, tab sis nws yog CNI cov tub ntxhais hluas raws li Open vSwitch (xws li Antrea, nws ua tau zoo dua thiab siv tsawg dua).
Txhua tus tsuas yog Flannel muaj cov cai hauv network. Nws yog qhov zoo tshaj plaws uas nws yuav tsis txhawb nqa lawv, txij li lub hom phiaj yog yooj yim dua li steamed turnip: lub sib dua, zoo dua.
Tsis tas li ntawd, ntawm lwm yam, kev ua haujlwm encryption yog amazing. Calico yog ib qho ntawm CNIs qub tshaj plaws, tab sis kev encryption tsuas yog ntxiv ob peb lub lis piam dhau los. Lawv xaiv wireguard hloov IPsec, thiab tsuas yog muab tso, nws ua haujlwm zoo thiab zoo kawg nkaus, ua kom tag nrho lwm yam CNIs hauv qhov kev sim no. Tau kawg, kev siv cov peev txheej nce ntxiv vim yog encryption, tab sis cov txiaj ntsig tau ua tiav yog tsim nyog (Calico pom qhov kev txhim kho rau 9000 npaug hauv qhov kev xeem encryption piv rau Cilium, uas nyob qib thib ob). Tsis tas li ntawd, koj tuaj yeem ua kom muaj kev tiv thaiv kab mob txhua lub sijhawm tom qab koj xa Calico mus rau pawg, thiab koj tuaj yeem lov tes taw rau lub sijhawm luv luv lossis mus tas li yog tias koj xav tau. Nws yog incredibly yooj yim, tab sis! Peb nco ntsoov koj tias Calico tsis tam sim no nws pib-nrhiav MTU (qhov no yog npaj rau yav tom ntej versions), yog li nco ntsoov teeb tsa MTU yog tias koj lub network txhawb Jumbo Frames (MTU XNUMX).
Ntawm lwm yam, nco ntsoov tias Cilium tuaj yeem nkag mus rau hauv kev sib tw ntawm pawg nodes (thiab tsis yog ntawm Pods), uas tuaj yeem yog qhov tseem ceeb heev rau cov pej xeem pawg.
Raws li kev txiav txim siab, kuv xav kom siv cov xwm txheej hauv qab no:
- Xav tau CNI rau pawg me me lossis kuv tsis xav tau kev nyab xeeb: ua haujlwm nrog Flannel, lub lightest thiab ruaj khov tshaj CNI (Nws kuj yog ib tug ntawm cov laus tshaj plaws, raws li cov lus dab neeg nws tau tsim los ntawm Homo Kubernautus los yog Homo Contaitorus). Tej zaum koj kuj yuav txaus siab rau qhov project ingenious tshaj plaws ,kob!
- Xav tau CNI rau pawg tsis tu ncua: Calico - koj xaiv, tab sis tsis txhob hnov ββββqab teeb tsa MTU yog tias xav tau. Koj tuaj yeem yooj yim thiab ua si nrog cov cai hauv network, tig encryption rau thiab tawm, thiab lwm yam.
- Xav tau CNI rau (heev) loj scale pawg: Zoo, qhov kev sim tsis qhia tus cwj pwm ntawm pawg loj, Kuv yuav zoo siab los ua qhov kev sim, tab sis peb tsis muaj ntau pua servers nrog 10Gbps kev sib txuas. Yog li qhov kev xaiv zoo tshaj plaws yog khiav qhov kev hloov kho ntawm koj cov nodes, tsawg kawg nrog Calico thiab Cilium.
Tau qhov twg los: www.hab.com