🥇Amava ethu ekuphuhliseni umqhubi weCSI eKubernetes yeYandex.Cloud

Siyavuya ukwazisa ukuba iFlant yandisa igalelo layo kwi-Open Source izixhobo zeKubernetes ngokukhulula Uguqulelo lwealpha lomqhubi weCSI (Isikhongozeli sokuGcina isiNxibelelwano) seYandex.Cloud.

Kodwa ngaphambi kokuba siqhubele phambili kwiinkcukacha zokuphunyezwa, masiphendule umbuzo wokuba kutheni le nto iyimfuneko xa iYandex sele inenkonzo. Inkonzo eLawulwayo yeKubernetes.

Intshayelelo

Kutheni kunje?

Ngaphakathi kwinkampani yethu, kwasekuqaleni kokusebenzisa i-Kubernetes kwimveliso (okt iminyaka eliqela ngoku), besiphuhlisa isixhobo sethu (i-deckhouse), leyo, ngendlela, siceba ukwenza ukuba ifumaneke ngokukhawuleza njengeprojekthi yoMthombo oVulekileyo. . Ngoncedo lwayo, siqwalasela ngokulinganayo kwaye siqwalasele onke amaqela ethu, kwaye ngoku sele sele engaphezulu kwe-100 kuwo, kwiindidi ezininzi zoqwalaselo lwehardware nakuzo zonke iinkonzo zelifu ezikhoyo.

Amaqela asebenzisa i-deckhouse anazo zonke izinto eziyimfuneko ekusebenzeni: abalinganisi, ukubeka iliso ngeetshathi ezifanelekileyo, iimethrikhi kunye nezilumkiso, ukuqinisekiswa komsebenzisi ngokusebenzisa ababoneleli bangaphandle ukufikelela kuzo zonke iidashbhodi, njalo njalo. Akukho sizathu sokufakela i-cluster "pumped up" enjalo kwisisombululo esilawulwayo, kuba oku kudla ngokungenakwenzeka okanye kuya kukhokelela kwisidingo sokukhubaza isiqingatha samacandelo.

NB: La ngamava ethu, kwaye angqale. Asinakuze sicebise ukuba wonke umntu makasebenzise amaqela e-Kubernetes eyedwa endaweni yokusebenzisa izisombululo esele zenziwe. Ngendlela, asinawo amava okwenene ekusebenzeni i-Kubernetes evela kwi-Yandex kwaye asiyi kunika naluphi na uvavanyo lwale nkonzo kweli nqaku.

Yintoni kwaye isenzelwa bani?

Ke, sele sithethile malunga nendlela yangoku yokugcina eKubernetes: isebenza njani iCSI? и uluntu lweza njani kule ndlela.

Okwangoku, abaninzi ababoneleli ngenkonzo yelifu elikhulu baye baphuhlisa abaqhubi bokusebenzisa iidiski zabo zelifu njengeVolume eZiqhubekayo kwi-Kubernetes. Ukuba umboneleli akanalo umqhubi onjalo, kodwa yonke imisebenzi efunekayo inikezelwa nge-API, ngoko akukho nto ikuthintela ekuphumezeni umqhubi ngokwakho. Oku kwenzeka ngeYandex.Cloud.

Sathatha njengesiseko sophuhliso Umqhubi weCSI welifu leDijithaliOcean kunye neengcamango ezimbalwa ezivela abaqhubi be-GCP, ekubeni ukusebenzisana kunye ne-API yala mafu (i-Google kunye ne-Yandex) inenani lezinto ezifanayo. Ngokukodwa, i-API kunye I-GCP, kunye Yandex buyisela into Operation ukulandelela ubume bemisebenzi yexesha elide (umzekelo, ukwenza idiski entsha). Ukusebenzisana neYandex.Cloud API, sebenzisa Yandex.Cloud Go SDK.

Isiphumo somsebenzi owenziweyo ipapashwe kwi-GitHub kwaye inokuba luncedo kwabo bathi, ngesizathu esithile, basebenzise eyabo i-Kubernetes ufakelo kwiYandex.Cloud virtual machines (kodwa hayi iqela elilawulwayo elenziwe sele lilungile) kwaye bangathanda ukusebenzisa (odola) iidiski ngeCSI.

Ukuphunyezwa

Iimpawu eziphambili

Okwangoku umqhubi uxhasa le misebenzi ilandelayo:

Ukuodola iidiski kuyo yonke imimandla yeqela ngokwe-topology yee-nodes kwiqela;
Ukususa iidiski ezi-odolwe ngaphambili;
I-Offline yenza ubungakanani kwakhona kwiidiski (Yandex.Cloud musa ukuxhasa ukwandisa iidiski ezifakwe kumatshini wenyani). Ukufumana ulwazi malunga nokuba umqhubi kufuneka aguqulwe njani ukwenza uhlengahlengiso lube buhlungu kangangoko, jonga ngezantsi.

Kwixesha elizayo, siceba ukuphumeza inkxaso yokudala kunye nokucima i-disk snapshots.

Ubunzima obuphambili kunye nendlela yokuwoyisa

Ukunqongophala kokukwazi ukunyusa iidiski ngexesha langempela kwiYandex.Cloud API ngumda odibanisa ubungakanani bomsebenzi wePV (uMqulu oZingisayo): kulo mzekelo, kuyimfuneko ukuba i-pod yesicelo esebenzisa idisk imiswe, kwaye oku kunokubangela usetyenziso lwexesha lokuphumla.

Ngokutsho Iimpawu zeCSI, ukuba umlawuli weCSI unika ingxelo yokuba angenza ubungakanani kwakhona bediski "ngaphandle kweintanethi" (VolumeExpansion.OFFLINE), emva koko inkqubo yokunyusa idiski kufuneka ihambe ngolu hlobo:

Ukuba i-plugin inayo kuphela VolumeExpansion.OFFLINE Ukwandiswa kwesakhono kunye nomthamo ngoku kupapashwa okanye kufumaneke kwindawo yokuphumla ngoko ControllerExpandVolume KUFUNEKA ibizwe KUPHELA emva kwayo nayiphi na into:

I-plugin inomlawuli PUBLISH_UNPUBLISH_VOLUME ubunakho kunye ControllerUnpublishVolume Ibizwe ngempumelelo.

KWENYE INDAWO

I-plugin ayinasilawuli PUBLISH_UNPUBLISH_VOLUME amandla, iplagin ine node STAGE_UNSTAGE_VOLUME ubunakho, kunye NodeUnstageVolume igqityiwe ngempumelelo.

KWENYE INDAWO

I-plugin ayinasilawuli PUBLISH_UNPUBLISH_VOLUME ubunakho, okanye node STAGE_UNSTAGE_VOLUME ubunakho, kunye NodeUnpublishVolume igqitywe ngempumelelo.

Oku kuthetha ukuba kufuneka ukhuphe idiski kumatshini wenyani phambi kokuba uyandise.

Nangona kunjalo, ngelishwa umiliselo Iinkcukacha ze-CSI kusetyenziswa ii-sidecars azifikeleli ezi mfuno:

Kwisitya esisecaleni csi-attacher, ekufuneka ithwale uxanduva lobukho besithuba esifunekayo phakathi kwee-mounts, lo msebenzi awuphunyezwanga kwi-resize ye-offline. Ingxoxo malunga noku yaqaliswa apha.
Yintoni kanye kanye isikhongozeli semoto esecaleni kulo mxholo? I-plugin ye-CSI ngokwayo ayidibanisi ne-Kubernetes API, kodwa iphendula kuphela kwiifowuni ze-gRPC ezithunyelwe kuyo ngezikhongozeli ze-sidecar. Yakutshanje ziyaphuhliswa luluntu lwaseKubernetes.

Kwimeko yethu (iplagi yeCSI), ukusebenza kokunyusa idiski kujongeka ngolu hlobo:

Sifumana umnxeba we-gRPC ControllerExpandVolume;
Sizama ukwandisa idiski kwi-API, kodwa sifumana impazamo malunga nokungenzeki kokwenza umsebenzi kuba idiski inyusiwe;
Sigcina i-disk identifier kwimephu, equlethe iidiski apho kufuneka kwenziwe umsebenzi wokwandisa. Ngezantsi, ubufutshane, siya kuyibiza le mephu njenge volumeResizeRequired;
Susa ngesandla i-pod esebenzisa idiski. I-Kubernetes iyakuyiqala kwakhona. Ukuze idiski ingabi naxesha lokunyuka (ControllerPublishVolume) phambi kokugqiba umsebenzi wonyuso xa uzama ukunyuswa, sijonga ukuba idiski enikiweyo isangaphakathi volumeResizeRequired kwaye ubuyisele imposiso;
Umqhubi we-CSI uzama ukuphinda enze umsebenzi wokubuyisela ubungakanani. Ukuba umsebenzi ube yimpumelelo, ke susa idiski kwi volumeResizeRequired;
Ngokuba Isazisi seDiski silahlekile volumeResizeRequired, ControllerPublishVolume idlula ngempumelelo, idiski inyuswe, i-pod iqala.

Yonke into ibonakala ilula ngokwaneleyo, kodwa njengesiqhelo kukho imigibe. Uyandisa iidiski yangaphandle-resizer, apho kwenzeka impazamo ngexesha lokusebenza isebenzisa umgca ngokunyuka okubonakalayo kwixesha lokuvala ukuya kuthi ga kwimizuzwana eyi-1000:

func DefaultControllerRateLimiter() RateLimiter {
  return NewMaxOfRateLimiter(
  NewItemExponentialFailureRateLimiter(5*time.Millisecond, 1000*time.Second),
  // 10 qps, 100 bucket size.  This is only for retry speed and its only the overall factor (not per item)
  &BucketRateLimiter{Limiter: rate.NewLimiter(rate.Limit(10), 100)},
  )
}

Oku kunokubangela ukuba umsebenzi wokwandiswa kwediski ukwandiswe kangangemizuzu eyi-15+ kwaye, ngoko ke, i-pod ehambelanayo ayifumaneki.

Ekuphela kwenketho ethe yasivumela ngokulula nangokungenantlungu ukuba sinciphise ixesha elinokubakho kusetyenziso lwenguqulelo yethu ye-external-resizer kunye nomda wokuphuma kwexesha. kwimizuzwana emi-5:

workqueue.NewItemExponentialFailureRateLimiter(5*time.Millisecond, 5*time.Second)

Asizange sikubone kuyimfuneko ukuqalisa ingxoxo ngokukhawuleza kwaye sifake i-resizer yangaphandle, kuba ukulinganisa ubungakanani bediski ngaphandle kweintanethi kukuphosa umva okuya kuthi shwaka kungekudala kubo bonke ababoneleli belifu.

Ukuqala njani ukusebenzisa?

Umqhubi uxhaswa kwi-Kubernetes version 1.15 nangaphezulu. Ukuze umqhubi asebenze, ezi mfuno zilandelayo mazifezekiswe:

Iflegi --allow-privileged cwangcisa ixabiso true ye-API umncedisi kunye ne kubelet;
Ibandakanyiwe --feature-gates=VolumeSnapshotDataSource=true,KubeletPluginsWatcher=true,CSINodeInfo=true,CSIDriverRegistry=true ye-API umncedisi kunye ne kubelet;
Ukunyuka kwentaba (ukunyuka kwentaba) kufuneka yenziwe ukuba isebenze kwiqela. Xa usebenzisa i-Docker, i-daemon kufuneka iqwalaselwe ukuvumela ukunyuswa okwabelwana ngako.

Onke amanyathelo ayimfuneko ofakelo ngokwalo ichazwe kwi-FUNDA. Ukufakela kubandakanya ukudala izinto kwi-Kubernetes ukusuka kwi-manifest.

Ukuze umqhubi asebenze uya kufuna oku kulandelayo:

Chaza isichongi sikavimba weefayili kumboniso (folder-id) Yandex.Cloud (bona amaxwebhu);
Ukusebenzisana neYandex.Cloud API, umqhubi weCSI usebenzisa i-akhawunti yenkonzo. Kwi-manifest, iMfihlo kufuneka igqithiswe izitshixo ezigunyazisiweyo kwiakhawunti yenkonzo. Kumaxwebhu ichazwe, indlela yokwenza i-akhawunti yenkonzo kwaye ufumane izitshixo.

Konke kwinto enye - zama, kwaye siya kuvuya ukufumana ingxelo kunye imiba emitshaukuba ufumana naziphi na iingxaki!

Inkxaso eyongezelelweyo

Ngenxa yoko, singathanda ukuqaphela ukuba siphumeze lo mqhubi we-CSI kungekhona ngenxa yomnqweno omkhulu wokuzonwabisa ngokubhala izicelo kwi-Go, kodwa ngenxa yesidingo esiphuthumayo ngaphakathi kwenkampani. Akubonakali ngathi luncedo ukuba sigcine ukuphunyezwa kwethu, ngoko ke ukuba i-Yandex ibonisa umdla kwaye inquma ukuqhubeka nokuxhasa umqhubi, siya kuvuya ukudlulisela indawo yokugcina kubo.

Ukongeza, i-Yandex mhlawumbi inokuphunyezwa kwayo komqhubi we-CSI kwiqela layo elilawulwayo le-Kubernetes, elinokukhutshwa kwi-Open Source. Kwakhona sibona olu khetho lophuhliso luthandeka - uluntu luya kukwazi ukusebenzisa umqhubi oqinisekisiweyo ovela kumnikezeli wenkonzo, kwaye kungekhona kwinkampani yesithathu.

PS

Funda nakwibhlog yethu:

umthombo: www.habr.com

Amava ethu ekuphuhliseni umqhubi weCSI eKubernetes yeYandex.Cloud