Amathiphu namasu okusebenza no-Ceph kumaphrojekthi amatasa

Sisebenzisa i-Ceph njengesitoreji senethiwekhi kumaphrojekthi anemithwalo ehlukene, singase sihlangabezane nemisebenzi ehlukahlukene okungathi uma uqala ukuyibuka ingabonakali ilula noma iwubala. Ngokwesibonelo:

• ukuthuthwa kwedatha isuka ku-Ceph endala iye kwentsha ngokusetshenziswa ngokwengxenye kwamaseva wangaphambilini kuqoqo elisha;
• isixazululo senkinga yokwabiwa kwesikhala sediski ku-Ceph.

Ukubhekana nemisebenzi enjalo, sibhekene nesidingo sokususa i-OSD ngendlela efanele ngaphandle kokulahlekelwa idatha, okubaluleke kakhulu lapho usebenza nenani elikhulu ledatha. Lokhu kuzoxoxwa ngakho esihlokweni.

Izindlela ezichazwe ngezansi zifanelekile kunoma iyiphi inguqulo ye-Ceph. Ngaphezu kwalokho, iqiniso lokuthi i-Ceph ingagcina inani elikhulu ledatha izocatshangelwa: ukuvimbela ukulahleka kwedatha nezinye izinkinga, ezinye izenzo "zizohlukaniswa" zibe ezinye eziningana.

Isandulelo mayelana ne-OSD

Njengoba izindlela zokupheka ezimbili kwezintathu okuxoxwe ngazo zinikezelwe ku-OSD (I-Daemon Yesitoreji Sento), ngaphambi kokungena engxenyeni engokoqobo - kafushane mayelana nokuthi iyini kuCeph nokuthi kungani ibaluleke kangaka.

Okokuqala, kufanele kushiwo ukuthi lonke iqoqo le-Ceph liqukethe ama-OSD amaningi. Uma kukhona okuningi, iba nkulu ivolumu yedatha yamahhala ku-Ceph. Kulula ukukuqonda ukusuka lapha umsebenzi oyinhloko we-OSD: Igcina idatha yento ye-Ceph ezinhlelweni zefayela zawo wonke ama-cluster node futhi inikeza ukufinyelela kwenethiwekhi kuyo (ukufunda, ukubhala, nezinye izicelo).

Ezingeni elifanayo, imingcele yokuphindaphinda isethwa ngokukopisha izinto phakathi kwama-OSD ahlukene. Futhi lapha ungabhekana nezinkinga ezihlukahlukene, izixazululo okuzoxoxwa ngazo ngokuqhubekayo.

Inombolo yecala. Khipha ngokuphephile i-OSD kuqoqo le-Ceph ngaphandle kokulahlekelwa idatha

Isidingo sokususa i-OSD singase sibangelwe ukukhipha iseva kuqoqo - ngokwesibonelo, ukuyishintsha kufakwe enye iseva - okuyilokho okwenzeka kithi, okuholele ekulotshweni kwalesi sihloko. Ngakho-ke, umgomo omkhulu wokukhohlisa ukukhipha wonke ama-OSD nama-mons kuseva enikeziwe ukuze imiswe.

Ukuze kube lula futhi sigweme isimo lapho senza iphutha ekucaciseni i-OSD edingekayo ngenkathi senza imiyalo, sizosetha okuguquguqukayo okuhlukile, inani lakhona elizoba inombolo ye-OSD ezosuswa. Asimfonele ${ID} - lapha nangezansi, okuguquguqukayo okunjalo kuthatha indawo yenombolo ye-OSD esisebenza ngayo.

Ake sibheke isimo ngaphambi kokuqala umsebenzi:

root@hv-1 ~ # ceph osd tree
ID CLASS WEIGHT  TYPE NAME      STATUS REWEIGHT PRI-AFF
-1       0.46857 root default
-3       0.15619      host hv-1
-5       0.15619      host hv-2
 1   ssd 0.15619      osd.1     up     1.00000  1.00000
-7       0.15619      host hv-3
 2   ssd 0.15619      osd.2     up     1.00000  1.00000

Ukuze uqale ukususwa kwe-OSD, uzodinga ukwenza kahle reweight kuyo ku-zero. Ngale ndlela sehlisa inani ledatha ku-OSD ngokuyilinganisa kwamanye ama-OSD. Ukuze wenze lokhu, sebenzisa imiyalo elandelayo:

ceph osd reweight osd.${ID} 0.98
ceph osd reweight osd.${ID} 0.88
ceph osd reweight osd.${ID} 0.78

... nokunye kuze kube yiziro.

Kudingeka ukulinganisa okubusheleleziukuze ungalahlekelwa idatha. Lokhu kuyiqiniso ikakhulukazi uma i-OSD iqukethe inani elikhulu ledatha. Ukuqinisekisa ukuthi ngemva kokwenza imiyalo reweight konke kuhambe kahle, ungakuqedela ceph -s noma ku-terminal ehlukile yokusebenza kwewindi ceph -w ukuze ubone izinguquko ngesikhathi sangempela.

Uma i-OSD “ithululwa”, ungaqhubeka nomsebenzi ojwayelekile ukuze uyisuse. Ukwenza lokhu, dlulisela i-OSD oyifunayo kuhulumeni down:

ceph osd down osd.${ID}

“Masidonse” i-OSD eqoqweni:

ceph osd out osd.${ID}

Ake simise isevisi ye-OSD futhi sehlise ukuhlukaniswa kwayo ku-FS:

systemctl stop ceph-osd@${ID}
umount /var/lib/ceph/osd/ceph-${ID}

Susa i-OSD kusuka CRUSH imephu:

ceph osd crush remove osd.${ID}

Masisuse umsebenzisi we-OSD:

ceph auth del osd.${ID}

Futhi ekugcineni, ake sisuse i-OSD ngokwayo:

ceph osd rm osd.${ID}

Ukubhala: Uma usebenzisa inguqulo ye-Ceph Luminous noma ngaphezulu, izinyathelo zokususa ze-OSD ezingenhla zingancishiswa zibe imiyalo emibili:

ceph osd out osd.${ID}
ceph osd purge osd.${ID}

Uma, ngemva kokuqeda izinyathelo ezichazwe ngenhla, ugijima umyalo ceph osd tree, khona-ke kufanele kucace ukuthi kuseva lapho umsebenzi wenziwe khona awasekho ama-OSD okwenziwe ngawo lokhu okungenhla:

root@hv-1 ~ # ceph osd tree
ID CLASS WEIGHT  TYPE NAME     STATUS REWEIGHT PRI-AFF
-1       0.46857      root default
-3       0.15619      host hv-1
-5       0.15619      host hv-2
-7       0.15619      host hv-3
 2   ssd 0.15619      osd.2    up     1.00000  1.00000

Endleleni, qaphela ukuthi isimo seqoqo le-Ceph sizoya kulo HEALTH_WARN, futhi sizobona nokuncipha kwenani lama-OSD kanye nenani lesikhala sediski esitholakalayo.

Okulandelayo kuzochaza izinyathelo ezizodingeka uma ufuna ukumisa ngokuphelele iseva futhi, ngokufanelekile, uyisuse ku-Ceph. Kulokhu, kubalulekile ukukhumbula lokho Ngaphambi kokuvala iseva, kufanele ususe wonke ama-OSD kule seva.

Uma engasekho ama-OSD asele kule seva, ngemuva kokuwakhipha udinga ukukhipha iseva kumephu ye-OSD. hv-2ngokusebenzisa umyalo olandelayo:

ceph osd crush rm hv-2

Susa mon kusuka kuseva hv-2ngokusebenzisa umyalo ongezansi kwenye iseva (okungukuthi, kulokhu, kuvuliwe hv-1):

ceph-deploy mon destroy hv-2

Ngemva kwalokhu, ungakwazi ukumisa iseva futhi uqale izenzo ezilandelayo (ukuphinda uyisebenzise, ​​njll.).

Icala elingu-2. Ukusatshalaliswa kwesikhala sediski kuqoqo le-Ceph eselakhiwe kakade

Ngizoqala indaba yesibili ngesandulelo mayelana ne-PG (Amaqembu Okubeka). Indima enkulu ye-PG ku-Ceph ngokuyinhloko ukuhlanganisa izinto ze-Ceph nokuqhubeka nokuziphindaphinda ku-OSD. Ifomula ongabala ngayo inani elidingekayo le-PG ingaphakathi isigaba esifanele Ceph imibhalo. Lolu daba ludingidwa lapho ngezibonelo ezithile.

Ngakho-ke: enye yezinkinga ezivamile lapho usebenzisa i-Ceph inombolo engalingani ye-OSD ne-PG phakathi kwamachibi ku-Ceph.

Okokuqala, ngenxa yalokhu, kungase kuvele isimo lapho kucaciswa ama-PG amaningi kakhulu echibini elincane, okuwukusetshenziswa okungenangqondo kwesikhala sediski kuqoqo. Okwesibili, ekusebenzeni kunenkinga enkulu kakhulu: ukuchichima kwedatha kwenye ye-OSD. Lokhu kuhlanganisa ukushintshela kweqembu kuqala kuhulumeni HEALTH_WARN, bese HEALTH_ERR. Isizathu salokhu ukuthi i-Ceph, lapho ubala inani elitholakalayo ledatha (ungayithola ngokuthi MAX AVAIL ekuphumeni komyalo ceph df endaweni yokubhukuda ngayinye ngokwehlukana) kusekelwe enanini ledatha etholakalayo ku-OSD. Uma singekho isikhala esanele ku-OSD eyodwa, ayisekho idatha engabhalwa kuze kube yilapho idatha isatshalaliswa kahle kuwo wonke ama-OSD.

Kuyafaneleka ukucacisa ukuthi lezi zinkinga zinqunywa kakhulu esigabeni sokucushwa kweqoqo le-Ceph. Elinye lamathuluzi ongawasebenzisa yi Ceph PGCalc. Ngosizo lwayo, inani elidingekayo le-PG libalwa ngokucacile. Kodwa-ke, ungaphendukela kuyo futhi esimweni lapho iqoqo le-Ceph vele kulungiswe ngokungalungile. Kuyafaneleka ukucacisa lapha ukuthi njengengxenye yomsebenzi wokulungisa uzodinga kakhulu ukunciphisa inani lama-PG, futhi lesi sici asitholakali kuzinguqulo ezindala ze-Ceph (sivele kuphela enguqulweni Nautilus).

Ngakho-ke, ake sicabange ngesithombe esilandelayo: iqoqo linesimo HEALTH_WARN ngenxa ye-OSD eyodwa ephelelwa yisikhala. Lokhu kuzokhonjiswa ngephutha HEALTH_WARN: 1 near full osd. Ngezansi i-algorithm yokuphuma kulesi simo.

Okokuqala, udinga ukusabalalisa idatha etholakalayo phakathi kwama-OSD asele. Sesivele senze umsebenzi ofanayo esimweni sokuqala, lapho "sikhipha" i-node - umehluko kuphela manje esizodinga ukuwunciphisa kancane. reweight. Isibonelo, kufika ku-0.95:

ceph osd reweight osd.${ID} 0.95

Lokhu kukhulula isikhala sediski ku-OSD futhi kulungisa iphutha ku-ceph health. Kodwa-ke, njengoba sekushiwo kakade, le nkinga ngokuyinhloko ivela ngenxa yokucushwa okungalungile kwe-Ceph ezigabeni zokuqala: kubaluleke kakhulu ukwenza ukulungiswa kabusha ukuze kungabonakali esikhathini esizayo.

Esimweni sethu esithile, konke kwehlela ku:

• inani liphezulu kakhulu replication_count kwelinye lamachibi,
• i-PG eningi kakhulu kwelinye ichibi futhi incane kakhulu kwelinye.

Masisebenzise isibali esishiwo kakade. Ibonisa ngokucacile lokho okudingeka kufakwe futhi, ngokuyisisekelo, akukho lutho oluyinkimbinkimbi. Ngemva kokusetha imingcele edingekayo, sithola izincomo ezilandelayo:

Ukubhala: Uma uhlela iqoqo le-Ceph kusukela ekuqaleni, omunye umsebenzi owusizo wokubala ukukhiqiza imiyalo ezodala amachibi kusukela ekuqaleni namapharamitha ashiwo kuthebula.

Ikholomu yokugcina ikusiza ukuthi uzulazule - I-PG count ephakanyisiwe. Esimweni sethu, eyesibili nayo iyasiza, lapho kuboniswa khona ipharamitha yokuphindaphinda, njengoba sinqume ukushintsha isiphindaphinda sokuphindaphinda.

Ngakho-ke, okokuqala udinga ukushintsha imingcele yokuphindaphinda - lokhu kufanelekile ukwenza kuqala, ngoba ngokunciphisa isiphindaphinda, sizokhulula isikhala sediski. Njengoba umyalo ugijima, uzobona ukuthi isikhala sediski esitholakalayo sizokhula:

ceph osd pool $pool_name set $replication_size

Futhi ngemva kokuqedwa kwayo, sishintsha amanani wepharamitha pg_num и pgp_num kanje:

ceph osd pool set $pool_name pg_num $pg_number
ceph osd pool set $pool_name pgp_num $pg_number

ezibalulekile: kumele siguqule inombolo yama-PG ngokulandelana echibini ngalinye futhi singashintshi amanani kwamanye amachibi kuze kube yilapho izixwayiso zinyamalala "Ukwehliswa kwedatha yedatha" и "inombolo ye-pgs yehlisiwe".

Ungahlola futhi ukuthi konke kuhambe kahle yini usebenzisa okuphumayo komyalo ceph health detail и ceph -s.

Icala elingu-3. Ukuthutha umshini obonakalayo usuka ku-LVM uye ku-Ceph RBD

Esimeni lapho iphrojekthi isebenzisa imishini ebonakalayo efakwe kumaseva aqashiwe angenalutho, inkinga yokugcinwa kokumelana namaphutha ngokuvamile ivela. Kuyinto efiselekayo kakhulu ukuthi kunesikhala esanele kulesi sitoreji ... Esinye isimo esivamile: kukhona umshini obonakalayo onesitoreji sendawo kuseva futhi udinga ukwandisa idiski, kodwa akukho ndawo ongaya kuyo, ngoba akukho. isikhala sediski samahhala esisele kuseva.

Inkinga ingaxazululwa ngezindlela ezahlukene - isibonelo, ngokuthuthela kwenye iseva (uma ikhona) noma ukwengeza amadiski amasha kuseva. Kepha akwenzeki ngaso sonke isikhathi ukwenza lokhu, ngakho-ke ukufuduka usuka ku-LVM uye ku-Ceph kungaba yisixazululo esihle kakhulu kule nkinga. Ngokukhetha le nketho, senza kube lula inqubo eqhubekayo yokufuduka phakathi kwamaseva, njengoba ngeke kube nesidingo sokuhambisa isitoreji sasendaweni sisuka kwesinye i-hypervisor siye kwenye. Okuwukuphela kwento oyibambayo ukuthi kuzodingeka umise i-VM ngenkathi umsebenzi usaqhubeka.

Iresiphi elandelayo ithathwe kuyo isihloko esivela kule bhulogi, iziqondiso eziye zahlolwa ngokusebenza. Konje, indlela yokufuduka okungenazinkinga nayo ichazwe lapho, nokho, kithi yayingadingeki, ngakho asizange siyihlole. Uma lokhu kubalulekile kuphrojekthi yakho, sizojabula ukuzwa ngemiphumela kumazwana.

Asiqhubekele engxenyeni ebonakalayo. Esibonelweni sisebenzisa i-virsh futhi, ngokufanelekile, i-libvirt. Okokuqala, qiniseka ukuthi ichibi le-Ceph lapho idatha izothuthelwa khona lixhunywe ku-libvirt:

virsh pool-dumpxml $ceph_pool

Incazelo yephuli kumele iqukathe idatha yokuxhuma ku-Ceph enedatha yokugunyazwa.

Isinyathelo esilandelayo ukuthi isithombe se-LVM siguqulelwa ku-Ceph RBD. Isikhathi sokwenza sincike kakhulu kusayizi wesithombe:

qemu-img convert -p -O rbd /dev/main/$vm_image_name rbd:$ceph_pool/$vm_image_name

Ngemuva kokuguqulwa, isithombe se-LVM sizohlala, esizoba usizo uma ukuthuthela i-VM iye ku-RBD kwehluleka futhi kufanele ubuyisele emuva izinguquko. Futhi, ukuze sikwazi ukuhlehlisa ngokushesha izinguquko, masenze isipele sefayela lokucushwa komshini obonakalayo:

virsh dumpxml $vm_name > $vm_name.xml
cp $vm_name.xml $vm_name_backup.xml

... bese uhlela okwangempela (vm_name.xml). Ake sithole ibhulokhi enencazelo yediski (iqala ngomugqa <disk type='file' device='disk'> futhi iphetha ngokuthi </disk>) bese ulehlisa libe ngale ndlela elandelayo:

<disk type='network' device='disk'>
<driver name='qemu'/>
<auth username='libvirt'>
  <secret type='ceph' uuid='sec-ret-uu-id'/>
 </auth>
<source protocol='rbd' name='$ceph_pool/$vm_image_name>
  <host name='10.0.0.1' port='6789'/>
  <host name='10.0.0.2' port='6789'/>
</source>
<target dev='vda' bus='virtio'/> 
<alias name='virtio-disk0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</disk>

Ake sibheke eminye imininingwane:

1. Kuphrothokholi source ikheli eliya endaweni yokugcina ku-Ceph RBD libonisiwe (leli yikheli elibonisa igama le-Ceph pool nesithombe se-RBD, esanqunywa esigabeni sokuqala).
2. Kubhulokhi secret uhlobo lubonisiwe ceph, kanye ne-UUID yemfihlo yokuxhuma kuyo. I-uuid yayo ingatholakala kusetshenziswa umyalo virsh secret-list.
3. Kubhulokhi host amakheli kubaqaphi be-Ceph akhonjisiwe.

Ngemva kokuhlela ifayela lokumisa nokuqedela ukuguqulwa kwe-LVM kuya ku-RBD, ungasebenzisa ifayela lokucushwa eliguquliwe bese uqala umshini obonakalayo:

virsh define $vm_name.xml
virsh start $vm_name

Isikhathi sokuhlola ukuthi umshini we-virtual uqale kahle: ungathola, isibonelo, ngokuxhuma kuwo nge-SSH noma nge- virsh.

Uma umshini we-virtual usebenza kahle futhi ungakatholi ezinye izinkinga, ungasusa isithombe se-LVM esingasasetshenziswa:

lvremove main/$vm_image_name

isiphetho

Sihlangabezane nawo wonke amacala achaziwe ekusebenzeni - sithemba ukuthi imiyalelo izosiza abanye abalawuli ukuxazulula izinkinga ezifanayo. Uma unamazwana noma ezinye izindaba ezifanayo kokuhlangenwe nakho kwakho usebenzisa i-Ceph, sizojabula ukuzibona kumazwana!

PS

Funda futhi kubhulogi yethu:

• «Izandla zethu akuzona ezesithukuthezi: ukubuyisela iqoqo le-Rook kuma-K8s";
• «To Rook or not to Rook - lowo ngumbuzo";
• «I-Rook - indawo yokugcina idatha "yokuzisiza" ye-Kubernetes";
• «Ukudala isitoreji esiqhubekayo ngokuhlinzekwa ku-Kubernetes okusekelwe ku-Ceph".

Source: www.habr.com