Iingcebiso kunye namaqhinga okusebenza noCeph kwiiprojekthi ezixakekileyo

Ukusebenzisa i-Ceph njengokugcinwa kwenethiwekhi kwiiprojekthi ezinemithwalo eyahlukeneyo, sinokudibana nemisebenzi eyahlukeneyo enokuthi xa uyijonga nje ibonakale ilula okanye incinci. Umzekelo:

• ukufuduka kwedatha ukusuka kwiCeph endala ukuya kwentsha ngokusetyenziswa ngokuyinxenye kweeseva zangaphambili kwiqela elitsha;
• isisombululo kwingxaki yokwabiwa kwendawo kwidisk kwiCeph.

Ukujongana neengxaki ezinjalo, sijongene nesidingo sokususa ngokuchanekileyo i-OSD ngaphandle kokulahlekelwa idatha, ebaluleke kakhulu xa ujongene nobuninzi beedatha. Oku kuya kuxutyushwa kwinqaku.

Iindlela ezichazwe ngezantsi zifanelekile kuyo nayiphi na inguqulelo yeCeph. Ukongeza, into yokuba iCeph inokugcina inani elikhulu ledatha iya kuqwalaselwa: ukukhusela ukulahleka kwedatha kunye nezinye iingxaki, ezinye izenzo ziya "kwahlulwa" kwezinye ezininzi.

Intshayelelo malunga neOSD

Kuba ezimbini kwezi zintathu iiresiphi ezixoxiwe zinikezelwe kwiOSD (Into yokugcina iDaemon), ngaphambi kokuntywila kwinxalenye ebonakalayo - ngokufutshane malunga nokuba yintoni na eCeph kwaye kutheni ibaluleke kangaka.

Okokuqala, kufuneka kuthiwe lonke iqela le-Ceph liqukethe ii-OSD ezininzi. Okukhona kukho, kokukhona ubuninzi bedatha yasimahla kwiCeph. Kulula ukuyiqonda ukusuka apha umsebenzi ongundoqo we-OSD: Igcina idatha ye-Ceph yento kwiinkqubo zefayile yazo zonke ii-cluster nodes kwaye inikezela ukufikelela kwinethiwekhi kuyo (ukufunda, ukubhala, kunye nezinye izicelo).

Kwinqanaba elifanayo, iiparamitha zokuphindaphinda zisetwa ngokukopa izinto phakathi kwee-OSD ezahlukeneyo. Kwaye apha unokuhlangabezana neengxaki ezahlukeneyo, izisombululo eziza kuxutyushwa ngezantsi.

Inombolo yetyala 1. Susa ngokukhuselekileyo i-OSD kwiqela le-Ceph ngaphandle kokuphulukana nedatha

Isidingo sokususa i-OSD sinokubangelwa ngokususa umncedisi kwiqela - umzekelo, ukutshintshela omnye umncedisi - oko kwenzekayo kuthi, kubangele ukubhalwa kweli nqaku. Ke, eyona njongo iphambili yokukhohlisa kukukhupha zonke ii-OSD kunye neemons kwiseva enikezelweyo ukuze imiswe.

Ukuze kube lula kunye nokuphepha imeko apho senza impazamo ekuchazeni i-OSD efunekayo ngelixa siphumeza imiyalelo, siya kumisela ukuguquguquka okwahlukileyo, ixabiso eliya kuba linani le-OSD eliza kususwa. Masimfowunele ${ID} — apha nangezantsi, olo tshintsho luthatha indawo yenani leOSD esisebenza ngayo.

Makhe sijonge imeko ngaphambi kokuqala umsebenzi:

root@hv-1 ~ # ceph osd tree
ID CLASS WEIGHT  TYPE NAME      STATUS REWEIGHT PRI-AFF
-1       0.46857 root default
-3       0.15619      host hv-1
-5       0.15619      host hv-2
 1   ssd 0.15619      osd.1     up     1.00000  1.00000
-7       0.15619      host hv-3
 2   ssd 0.15619      osd.2     up     1.00000  1.00000

Ukuqalisa ukususa i-OSD, kuya kufuneka usebenze ngokutyibilikayo reweight kuyo ukuya ku-zero. Ngale ndlela sinciphisa inani ledatha kwi-OSD ngokuyilinganisa kwezinye ii-OSD. Ukwenza oku, sebenzisa le miyalelo ilandelayo:

ceph osd reweight osd.${ID} 0.98
ceph osd reweight osd.${ID} 0.88
ceph osd reweight osd.${ID} 0.78

... kwaye njalo kude kube ngu-zero.

Kufuneka ulungelelwaniso olugudileyoukuze ungalahlekelwa yidatha. Oku kuyinyani ngakumbi ukuba i-OSD iqulethe isixa esikhulu sedatha. Ukuqinisekisa ukuba emva kokuphumeza imiyalelo reweight yonke into ihambe kakuhle, ungayigqiba ceph -s okanye kwifestile yendlela eyahlukileyo yokusebenza ceph -w ukuze ubone utshintsho ngexesha lokwenyani.

Xa i-OSD "ikhutshwe", unokuqhubeka nomsebenzi oqhelekileyo wokuyisusa. Ukwenza oku, dlulisela i-OSD efunekayo kurhulumente down:

ceph osd down osd.${ID}

“Masitsale” i-OSD ngaphandle kweqela:

ceph osd out osd.${ID}

Masiyimise inkonzo ye-OSD kwaye sihlise isahlulelo sayo kwiFS:

systemctl stop ceph-osd@${ID}
umount /var/lib/ceph/osd/ceph-${ID}

Susa i-OSD kwi CRUSH imephu:

ceph osd crush remove osd.${ID}

Masicime umsebenzisi we-OSD:

ceph auth del osd.${ID}

Kwaye ekugqibeleni, masisuse i-OSD ngokwayo:

ceph osd rm osd.${ID}

Qaphela:: Ukuba usebenzisa uguqulelo lweCeph Luminous okanye ngaphezulu, la manyathelo okususa e-OSD angentla anokwehliswa abe yimiyalelo emibini:

ceph osd out osd.${ID}
ceph osd purge osd.${ID}

Ukuba, emva kokugqiba amanyathelo achazwe ngasentla, uqhuba umyalelo ceph osd tree, ngoko ke kufuneka kucace ukuba kumncedisi apho umsebenzi wenziwa khona azisekho ii-OSD apho le misebenzi ingentla yenziwe:

root@hv-1 ~ # ceph osd tree
ID CLASS WEIGHT  TYPE NAME     STATUS REWEIGHT PRI-AFF
-1       0.46857      root default
-3       0.15619      host hv-1
-5       0.15619      host hv-2
-7       0.15619      host hv-3
 2   ssd 0.15619      osd.2    up     1.00000  1.00000

Endleleni, qaphela ukuba imeko yeqela leCeph iya kuya HEALTH_WARN, kwaye siza kubona kwakhona ukuncipha kwenani le-OSD kunye nobungakanani bendawo yediski ekhoyo.

Oku kulandelayo kuya kuchaza amanyathelo aya kufuneka ukuba ufuna ukumisa ngokupheleleyo umncedisi kwaye, ngokufanelekileyo, uyisuse kwi-Ceph. Kule meko, kubalulekile ukukhumbula oko Ngaphambi kokuba uvale iseva, kufuneka ususe zonke ii-OSD kule seva.

Ukuba azisekho ii-OSD eziseleyo kule seva, emva kokuba uzisusile kufuneka ukhuphele ngaphandle umncedisi kwimephu yeOSD. hv-2ngokwenza lo myalelo ulandelayo:

ceph osd crush rm hv-2

Cima mon ukusuka kumncedisi hv-2ngokusebenzisa umyalelo ongezantsi komnye umncedisi (okt kulo mzekelo, kwi hv-1):

ceph-deploy mon destroy hv-2

Emva koku, unokumisa umncedisi kwaye uqalise iintshukumo ezilandelayo (ukuphinda uyisebenzise, ​​njl.).

Ityala le-2. Ukuhanjiswa kwendawo yedisk kwiqela esele lenziwe le-Ceph

Ndizakuqala ibali lesibini ngentshayelelo malunga nePG (Amaqela okubekwa). Indima ephambili ye-PG kwi-Ceph ikakhulu kukudibanisa izinto zeCeph kwaye uziphindaphinde kwakhona kwi-OSD. Ifomula onokuthi ngayo ubale isixa esifunekayo sePG icandelo elifanelekileyo Amaxwebhu eCeph. Lo mba ukwaxoxwa apho ngemizekelo ethile.

Ke: enye yeengxaki eziqhelekileyo xa usebenzisa i-Ceph linani elingalinganiyo le-OSD kunye ne-PG phakathi kwamachibi e-Ceph.

Okokuqala, ngenxa yoku, imeko inokuvela xa iiPGs ezininzi kakhulu zichazwe kwiphuli encinci, eyona nto isetyenziswa ngokungenangqiqo indawo yediski kwiqela. Okwesibini, ekusebenzeni kukho ingxaki enkulu ngakumbi: ukuphuphuma kwedatha kwenye ye-OSD. Oku kubandakanya utshintsho lokuqala lweqela ukuya kurhulumente HEALTH_WARN, ke ngoku HEALTH_ERR. Isizathu soku kukuba iCeph, xa ubala inani elikhoyo ledatha (ungayifumana ngayo MAX AVAIL kwimveliso yomyalelo ceph df kwiqula ngalinye ngokwahlukeneyo) isekelwe kwisixa sedatha ekhoyo kwiOSD. Ukuba akukho sithuba saneleyo kwi-OSD enye, akukho datha ingabhalwa de idatha isasazwe ngokufanelekileyo kuzo zonke ii-OSD.

Kufanelekile ukucacisa ukuba ezi ngxaki zigqitywe kakhulu kwinqanaba loqwalaselo lweqela leCeph. Esinye sezixhobo onokuzisebenzisa si Ceph PGCalc. Ngoncedo lwayo, inani elifunekayo lePG libalwe ngokucacileyo. Nangona kunjalo, unokubhenela kuyo kwimeko apho iqela leCeph sele sele ibumbeko ngokungalunganga. Kufanelekile ukucacisa apha ukuba njengenxalenye yomsebenzi wokulungisa uyakufuna kakhulu ukunciphisa inani le-PGs, kwaye eli nqaku alifumaneki kwiinguqulelo ezindala ze-Ceph (ivele kuphela kwinguqulo. INautilus).

Ngoko, makhe sicinge lo mfanekiso ulandelayo: iqela linesimo HEALTH_WARN ngenxa yokuba enye yeOSD iphelelwa yindawo. Oku kuya kuboniswa ngempazamo HEALTH_WARN: 1 near full osd. Ngezantsi i-algorithm yokuphuma kule meko.

Okokuqala, kufuneka usasaze idatha ekhoyo phakathi kwee-OSD eziseleyo. Senze utyando olufanayo kwimeko yokuqala, xa "sikhupha" i-node - ngokwahlukana okukuphela koku kuya kufuneka sinciphise kancinci. reweight. Umzekelo, ukuya kuthi ga kwi-0.95:

ceph osd reweight osd.${ID} 0.95

Oku kukhulula indawo yediski kwi-OSD kwaye ilungise impazamo kwimpilo ye-ceph. Nangona kunjalo, njengoko sele kukhankanyiwe, le ngxaki iyenzeka ikakhulu ngenxa yokucwangciswa okungalunganga kweCeph kumanqanaba okuqala: kubaluleke kakhulu ukwenza uhlengahlengiso ukuze kungabonakali kwixesha elizayo.

Kwimeko yethu ethile, zonke zehla zaba:

• ixabiso liphezulu kakhulu replication_count kwelinye lamadama,
• i-PG eninzi kakhulu kwelinye idama kwaye incinci kakhulu kwenye.

Masisebenzise ikhalityhuleyitha esele ikhankanyiwe. Ibonisa ngokucacileyo oko kufuneka kufakwe kwaye, ngokomgaqo, akukho nto inzima. Ukuseta iiparamitha eziyimfuneko, sifumana ezi ngcebiso zilandelayo:

Qaphela:: Ukuba useka iqela le-Ceph ukusuka ekuqaleni, omnye umsebenzi oluncedo wekhaltyhuleyitha sisizukulwana semiyalelo esiya kudala amachibi ukusuka ekuqaleni kunye neeparamitha ezichazwe kwitheyibhile.

Uluhlu lokugqibela lukunceda ukuba ukhangele - Ubalo lwePG olucetyisiweyo. Kwimeko yethu, eyesibini nayo iluncedo, apho iparameter yokuphindaphinda ibonisiwe, ekubeni sigqibe ekubeni sitshintshe ukuphindaphinda ukuphindaphinda.

Ngoko, okokuqala kufuneka utshintshe iiparamitha zokuphindaphinda - oku kufanelekile ukwenza kuqala, kuba ngokunciphisa umphindaphindi, siya kukhulula indawo yedisk. Njengoko umyalelo uphumeza, uya kuqaphela ukuba indawo ekhoyo kwidisk iya kwanda:

ceph osd pool $pool_name set $replication_size

Kwaye emva kokugqitywa kwayo, sitshintsha amaxabiso eparameter pg_num и pgp_num ngolu hlobo:

ceph osd pool set $pool_name pg_num $pg_number
ceph osd pool set $pool_name pgp_num $pg_number

ebalulekileyo: kufuneka sitshintshe inani le-PGs ngokulandelelanayo kwiphuli nganye kwaye singatshintshi amaxabiso kwamanye amachibi de izilumkiso ziphele. "Ukungafuneki kwedatha okuthotyiweyo" и "n-inani leepgs ezithotyiweyo".

Ungajonga kwakhona ukuba yonke into ihambe kakuhle usebenzisa iziphumo zomyalelo ceph health detail и ceph -s.

Inombolo yetyala yesi-3. Ukufuduka kumatshini obonakalayo ukusuka kwiLVM ukuya kwiCeph RBD

Kwimeko apho iprojekthi isebenzisa oomatshini bokwenene abafakwe kwiiseva eziqeshiweyo zetsimbi, umba wokugcinwa kwempazamo uhlala uvela. Kwakhona kunqweneleka kakhulu ukuba kukho indawo eyaneleyo kule ndawo yokugcina ... Enye imeko eqhelekileyo: kukho umatshini obonakalayo kunye nokugcinwa kwendawo kumncedisi kwaye kufuneka ukwandise idiski, kodwa akukho ndawo yokuya kuyo, kuba akukho ndawo. indawo yedisk yasimahla eshiywe kumncedisi.

Ingxaki ingasombululwa ngeendlela ezahlukeneyo - umzekelo, ngokufudukela kwenye iseva (ukuba kukho enye) okanye ukongeza iidiski ezintsha kumncedisi. Kodwa akusoloko kusenzeka ukwenza oku, ke ukufuduka kwi-LVM ukuya kwiCeph kunokuba sisisombululo esihle kule ngxaki. Ngokukhetha olu khetho, senza lula inkqubo eyongezelelweyo yokufuduka phakathi kweeseva, kuba akuyi kubakho mfuneko yokususa ugcino lwasekhaya ukusuka kwenye i-hypervisor ukuya kwenye. Ekuphela kokubambisa kukuba kuya kufuneka umise i-VM ngelixa umsebenzi usenziwa.

Le recipe ilandelayo ithathwe kuyo inqaku kule blog, imiyalelo ethe yavavanywa ngokusebenza. Kunjalo nje, indlela yokufuduka ngaphandle kobunzima nayo ichazwe apho, nangona kunjalo, kwimeko yethu yayingafuneki, ngoko asizange siyihlolisise. Ukuba oku kubalulekile kwiprojekthi yakho, siya kuvuya ukuva malunga neziphumo kwizimvo.

Masiqhubele phambili kwinxalenye ebonakalayo. Kumzekelo sisebenzisa i-virsh kwaye, ngokufanelekileyo, libvirt. Okokuqala, qiniseka ukuba ichibi le-Ceph apho idatha iya kufuduselwa khona iqhagamshelwe kwi-libvirt:

virsh pool-dumpxml $ceph_pool

Inkcazo ye-pool kufuneka iqulethe idatha yoqhagamshelwano kwi-Ceph enedatha yogunyaziso.

Inyathelo elilandelayo kukuba umfanekiso weLVM uguqulelwa kwiCeph RBD. Ixesha lokwenziwa lixhomekeke ikakhulu kubungakanani bomfanekiso:

qemu-img convert -p -O rbd /dev/main/$vm_image_name rbd:$ceph_pool/$vm_image_name

Emva kokuguqulwa, umfanekiso we-LVM uya kuhlala, oya kuba luncedo ukuba ukufudukela kwi-VM ukuya kwi-RBD kusilele kwaye kufuneka ubuyisele umva utshintsho. Kwakhona, ukuze ukwazi ukubuyisela umva ngokukhawuleza utshintsho, masenze i-backup yefayile yoqwalaselo lomatshini wenyani:

virsh dumpxml $vm_name > $vm_name.xml
cp $vm_name.xml $vm_name_backup.xml

... kwaye uhlele eyoqobo (vm_name.xml). Masifumane ibhloko enenkcazo yediski (iqala ngomgca <disk type='file' device='disk'> kwaye iphetha nge </disk>) kwaye uyinciphise ibe ngolu hlobo lulandelayo:

<disk type='network' device='disk'>
<driver name='qemu'/>
<auth username='libvirt'>
  <secret type='ceph' uuid='sec-ret-uu-id'/>
 </auth>
<source protocol='rbd' name='$ceph_pool/$vm_image_name>
  <host name='10.0.0.1' port='6789'/>
  <host name='10.0.0.2' port='6789'/>
</source>
<target dev='vda' bus='virtio'/> 
<alias name='virtio-disk0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</disk>

Makhe sijonge iinkcukacha ezithile:

1. Kwiprothokholi source idilesi kwindawo yokugcina kwi-Ceph RBD ibonisiwe (le dilesi ebonisa igama le-pool ye-Ceph kunye nomfanekiso we-RBD, owamiselwa kwinqanaba lokuqala).
2. Kwibloko secret uhlobo lubonisiwe ceph, kunye ne-UUID yemfihlo yokuqhagamshela kuyo. I-uuid yayo inokufunyanwa ngokusebenzisa umyalelo virsh secret-list.
3. Kwibloko host iidilesi eziya kubahloli beCeph zibonisiwe.

Emva kokuhlela ifayile yoqwalaselo kunye nokugqiba iLVM kwiRBD uguqulo, unokusebenzisa ifayile yoqwalaselo elungisiweyo kwaye uqale umatshini obonakalayo:

virsh define $vm_name.xml
virsh start $vm_name

Lixesha lokuba ujonge ukuba umatshini wenyani uqale ngokuchanekileyo: ungafumanisa, umzekelo, ngokuqhagamshela kuyo nge-SSH okanye nge-SSH. virsh.

Ukuba umatshini wenyani usebenza ngokuchanekileyo kwaye awufumananga naziphi na ezinye iingxaki, ngoko unokucima umfanekiso weLVM ongasasetyenziswayo:

lvremove main/$vm_image_name

isiphelo

Sidibene nazo zonke iimeko ezichazwe ekusebenzeni - sinethemba lokuba imiyalelo iya kunceda abanye abalawuli basombulule iingxaki ezifanayo. Ukuba unamagqabantshintshi okanye amanye amabali afanayo kumava akho usebenzisa iCeph, siya kuvuya ukubabona kwizimvo!

PS

Funda nakwibhlog yethu:

• «Izandla zethu ayizizo ezokukruquka: ukubuyisela iqela leRook kwii-K8s";
• «Ukuba Rook okanye hayi Rook - ngumbuzo lowo";
• «URook - "inkonzo yokuzenzela" indawo yokugcina idatha yeKubernetes";
• «Ukudala ukugcinwa okuqhubekayo kunye nokubonelela kwi-Kubernetes esekelwe kwi-Ceph».

umthombo: www.habr.com