Maupangiri ndi zidule zogwirira ntchito ndi Ceph muma projekiti otanganidwa

Pogwiritsa ntchito Ceph ngati malo osungira maukonde muma projekiti okhala ndi katundu wosiyanasiyana, titha kukumana ndi ntchito zosiyanasiyana zomwe poyang'ana koyamba sizimawoneka ngati zosavuta kapena zazing'ono. Mwachitsanzo:

• kusamuka kwa data kuchokera ku Ceph yakale kupita ku yatsopano ndikugwiritsa ntchito pang'ono ma seva am'mbuyomu mgulu latsopano;
• njira yothetsera vuto la kugawa malo a disk mu Ceph.

Kuthana ndi mavuto otere, tikukumana ndi kufunikira kochotsa molondola OSD popanda kutaya deta, zomwe ndizofunikira makamaka pochita ndi deta yambiri. Izi zidzakambidwa m’nkhaniyo.

Njira zomwe zafotokozedwa pansipa ndizogwirizana ndi mtundu uliwonse wa Ceph. Kuonjezera apo, mfundo yakuti Ceph ikhoza kusunga deta yambiri idzaganiziridwa: kuteteza kutayika kwa deta ndi mavuto ena, zochita zina "zidzagawanika" mu zina zingapo.

Mawu oyamba a OSD

Popeza maphikidwe awiri mwa atatu omwe akukambidwa adaperekedwa ku OSD (Chinthu Chosungira Daemon), musanalowe mu gawo lothandizira - mwachidule za zomwe zili mu Ceph komanso chifukwa chake ndizofunikira.

Choyamba, ziyenera kunenedwa kuti gulu lonse la Ceph lili ndi ma OSD ambiri. Kuchuluka komwe kulipo, kumapangitsanso kuchuluka kwa data yaulere mu Ceph. Ndi zophweka kumvetsa kuchokera apa chachikulu OSD ntchito: Imasunga zinthu za Ceph pamafayilo amagulu onse amgulu ndipo imapereka mwayi wopezeka pa intaneti (yowerenga, kulemba, ndi zopempha zina).

Pamlingo womwewo, magawo obwereza amayikidwa potengera zinthu pakati pa ma OSD osiyanasiyana. Ndipo apa mutha kukumana ndi mavuto osiyanasiyana, mayankho omwe tikambirana pansipa.

Mlandu nambala 1. Chotsani OSD mosamala ku gulu la Ceph osataya deta

Kufunika kochotsa OSD kungayambitsidwe ndi kuchotsa seva ku masango - mwachitsanzo, kuti m'malo mwake ndi seva ina - zomwe zidatichitikira, zomwe zimapangitsa kulembedwa kwa nkhaniyi. Chifukwa chake, cholinga chachikulu chakusintha ndikuchotsa ma OSD onse ndi ma mons pa seva yopatsidwa kuti aimitsidwe.

Kuti zitheke komanso kuti tipewe vuto lomwe, pochita malamulo, timalakwitsa kuwonetsa OSD yofunikira, tidzakhazikitsa mtundu wosiyana, womwe mtengo wake udzakhala nambala ya OSD yomwe iyenera kuchotsedwa. Tiyeni timuyimbire ${ID} - apa ndi pansipa, kusinthika kotereku kumalowa m'malo mwa nambala ya OSD yomwe tikugwira nayo ntchito.

Tiyeni tiwone momwe zilili tisanayambe ntchito:

root@hv-1 ~ # ceph osd tree
ID CLASS WEIGHT  TYPE NAME      STATUS REWEIGHT PRI-AFF
-1       0.46857 root default
-3       0.15619      host hv-1
-5       0.15619      host hv-2
 1   ssd 0.15619      osd.1     up     1.00000  1.00000
-7       0.15619      host hv-3
 2   ssd 0.15619      osd.2     up     1.00000  1.00000

Kuti muyambe kuchotsa OSD, muyenera kuchita bwino reweight ku zero. Mwanjira iyi timachepetsa kuchuluka kwa data mu OSD poyigwirizanitsa ndi ma OSD ena. Kuti muchite izi, yendetsani malamulo awa:

ceph osd reweight osd.${ID} 0.98
ceph osd reweight osd.${ID} 0.88
ceph osd reweight osd.${ID} 0.78

... ndi zina zotero mpaka zero.

Kuyang'ana kosalala kumafunikakuti musataye deta. Izi ndi zoona makamaka ngati OSD ili ndi deta yambiri. Kuonetsetsa kuti pambuyo pochita malamulo reweight zonse zidayenda bwino, mutha kumaliza ceph -s kapena pawindo lina la terminal ceph -w kuti muwone kusintha mu nthawi yeniyeni.

Pamene OSD "yopanda kanthu", mukhoza kupitiriza ndi ntchito yokhazikika kuti muchotse. Kuti muchite izi, tumizani OSD yomwe mukufuna ku boma down:

ceph osd down osd.${ID}

Tiyeni "tikoke" OSD kuchokera pagulu:

ceph osd out osd.${ID}

Tiyeni tiyime ntchito ya OSD ndikutsitsa magawo ake mu FS:

systemctl stop ceph-osd@${ID}
umount /var/lib/ceph/osd/ceph-${ID}

Chotsani OSD ku CRUSH mapu:

ceph osd crush remove osd.${ID}

Tiyeni tichotse wosuta OSD:

ceph auth del osd.${ID}

Ndipo potsiriza, tiyeni tichotse OSD yokha:

ceph osd rm osd.${ID}

ndemanga: Ngati mukugwiritsa ntchito Ceph Luminous Baibulo kapena apamwamba, ndiye pamwamba OSD njira kuchotsa akhoza kuchepetsedwa kukhala malamulo awiri:

ceph osd out osd.${ID}
ceph osd purge osd.${ID}

Ngati, mutamaliza njira zomwe tafotokozazi, mumayendetsa lamulo ceph osd tree, ndiye ziyenera kuwonekeratu kuti pa seva yomwe ntchitoyo idachitidwa palibenso ma OSD omwe zidachitika pamwambapa:

root@hv-1 ~ # ceph osd tree
ID CLASS WEIGHT  TYPE NAME     STATUS REWEIGHT PRI-AFF
-1       0.46857      root default
-3       0.15619      host hv-1
-5       0.15619      host hv-2
-7       0.15619      host hv-3
 2   ssd 0.15619      osd.2    up     1.00000  1.00000

Panjira, zindikirani kuti mkhalidwe wa gulu la Ceph upita HEALTH_WARN, ndipo tidzawonanso kuchepa kwa chiwerengero cha OSD ndi kuchuluka kwa malo a disk omwe alipo.

Zotsatirazi zifotokoza njira zomwe zidzafunikire ngati mukufuna kuyimitsa seva ndipo, motero, chotsani ku Ceph. Pankhaniyi, ndikofunikira kukumbukira izi Musanatseke seva, muyenera kuchotsa ma OSD onse pa seva iyi.

Ngati palibenso ma OSD otsala pa seva iyi, ndiye mutawachotsa muyenera kusiya seva pamapu a OSD. hv-2poyendetsa lamulo ili:

ceph osd crush rm hv-2

Chotsani mon kuchokera pa seva hv-2poyendetsa lamulo ili pansipa pa seva ina (mwachitsanzo, pankhaniyi, on hv-1):

ceph-deploy mon destroy hv-2

Pambuyo pake, mutha kuyimitsa seva ndikuyamba kuchitapo kanthu (kuyimitsanso, ndi zina).

Mlandu nambala 2. Kugawa kwa disk space mumagulu opangidwa kale a Ceph

Ndiyamba nkhani yachiwiri ndi mawu oyamba okhudza PG (Magulu Oyika). Udindo waukulu wa PG mu Ceph makamaka ndikuphatikiza zinthu za Ceph ndikuwonjezeranso mu OSD. Njira yomwe mungawerengere kuchuluka kofunikira kwa PG ili mkati gawo loyenera Ceph zolemba. Nkhaniyi ikukambidwanso pamenepo ndi zitsanzo zenizeni.

Chifukwa chake: amodzi mwamavuto omwe amafala mukamagwiritsa ntchito Ceph ndi kuchuluka kwa OSD ndi PG pakati pa maiwe ku Ceph.

Choyamba, chifukwa cha izi, vuto likhoza kuchitika pamene ma PG ambiri amatchulidwa mu dziwe laling'ono, lomwe ndilo kugwiritsa ntchito mopanda nzeru kwa disk space mumagulu. Kachiwiri, pochita pali vuto lalikulu kwambiri: kusefukira kwa data mu imodzi mwa OSD. Izi zikuphatikiza kusamutsidwa koyamba kwa gulu kupita ku boma HEALTH_WARN, Kenako HEALTH_ERR. Chifukwa chake ndikuti Ceph, powerengera kuchuluka kwa zomwe zilipo (mutha kuzipeza mwa MAX AVAIL mu lamulo linanena bungwe ceph df pa dziwe lirilonse padera) zimatengera kuchuluka kwa deta yomwe ilipo mu OSD. Ngati palibe malo okwanira mu OSD imodzi, palibenso deta yomwe ingalembedwe mpaka deta itagawidwa bwino pakati pa OSD onse.

Ndikoyenera kufotokoza kuti mavuto awa zimaganiziridwa kwambiri pagawo la Ceph cluster configuration. Chimodzi mwa zida zomwe mungagwiritse ntchito ndi Ceph PGCalc. Ndi chithandizo chake, kuchuluka kofunikira kwa PG kumawerengedwa momveka bwino. Komabe, mutha kuyigwiritsanso ntchito pomwe gulu la Ceph kale kusinthidwa molakwika. Ndikoyenera kufotokozera apa kuti ngati gawo la ntchito yokonza mudzafunika kuchepetsa kuchuluka kwa ma PG, ndipo izi sizipezeka m'matembenuzidwe akale a Ceph (zinangowonekera mu mtundu Nautilus).

Kotero, tiyeni tiyerekeze chithunzi chotsatirachi: gululi lili ndi udindo HEALTH_WARN chifukwa cha imodzi mwa OSD kutha kwa malo. Izi zidzawonetsedwa ndi cholakwika HEALTH_WARN: 1 near full osd. Pansipa pali ndondomeko yotulutsira izi.

Choyamba, muyenera kugawa zomwe zilipo pakati pa ma OSD otsala. Tidachita kale opareshoni yofananira koyamba, pomwe "tidakhetsa" node - ndi kusiyana kokha komwe tifunikira kuchepetsa pang'ono. reweight. Mwachitsanzo, mpaka 0.95:

ceph osd reweight osd.${ID} 0.95

Izi zimamasula malo a disk mu OSD ndikukonza zolakwika mu ceph health. Komabe, monga tanenera kale, vutoli limachitika makamaka chifukwa cha kusinthika kolakwika kwa Ceph pazigawo zoyamba: ndikofunikira kwambiri kukonzanso kuti zisawonekere m'tsogolomu.

M'malo athu, zonse zidatsikira ku:

• mtengo wokwera kwambiri replication_count m'modzi mwa dziwe,
• PG yochuluka mu dziwe limodzi ndi yocheperapo mu dziwe lina.

Tiyeni tigwiritse ntchito chowerengera chomwe chatchulidwa kale. Zimasonyeza momveka bwino zomwe ziyenera kulowetsedwa ndipo, kwenikweni, palibe zovuta. Kukhazikitsa magawo ofunikira, timapeza malingaliro awa:

ndemanga: Ngati mukukhazikitsa gulu la Ceph kuyambira pachiyambi, ntchito ina yothandiza ya chowerengera ndi kupanga malamulo omwe apanga maiwe kuyambira poyambira ndi magawo omwe afotokozedwa patebulo.

Gawo lomaliza limakuthandizani kuyenda - Kuwerengera kwa PG. Kwa ife, yachiwiri ndiyothandizanso, pomwe gawo lobwereza likuwonetsedwa, popeza tidaganiza zosintha chochulukitsa.

Chifukwa chake, choyamba muyenera kusintha magawo obwereza - izi ndizoyenera kuchita poyamba, popeza pochepetsa ochulukitsa, tidzamasula malo a disk. Pamene lamulo likuchita, mudzawona kuti malo a disk omwe alipo adzawonjezeka:

ceph osd pool $pool_name set $replication_size

Ndipo ikamaliza, timasintha magawo a parameter pg_num и pgp_num motere:

ceph osd pool set $pool_name pg_num $pg_number
ceph osd pool set $pool_name pgp_num $pg_number

chofunika: tiyenera kusintha kuchuluka kwa ma PG motsatizana mu dziwe lililonse ndipo osasintha zomwe zili m'madziwe ena mpaka machenjezo atha. "Degraded Data redundancy" и "n-nambala ya pgs yawonongeka".

Mutha kuwonanso kuti zonse zidayenda bwino pogwiritsa ntchito zotuluka zamalamulo ceph health detail и ceph -s.

Mlandu nambala 3. Kusamutsa makina enieni kuchokera ku LVM kupita ku Ceph RBD

Pamene polojekiti imagwiritsa ntchito makina enieni omwe amaikidwa pa ma seva opanda zitsulo zobwerekedwa, nkhani yosungiramo zolakwika nthawi zambiri imakhalapo. Ndizofunikanso kwambiri kuti pali malo okwanira mu yosungirako iyi ... Chinthu china chodziwika bwino: pali makina enieni omwe ali ndi malo osungirako pa seva ndipo muyenera kukulitsa disk, koma palibe komwe mungapite, chifukwa palibe. danga laulere la disk lomwe latsala pa seva.

Vuto likhoza kuthetsedwa m'njira zosiyanasiyana - mwachitsanzo, posamukira ku seva ina (ngati ilipo) kapena kuwonjezera ma disks atsopano ku seva. Koma sizotheka kuchita izi nthawi zonse, chifukwa chake kusamuka kuchokera ku LVM kupita ku Ceph kumatha kukhala njira yabwino yothetsera vutoli. Posankha njirayi, timachepetsanso njira yowonjezereka ya kusamuka pakati pa ma seva, popeza sipadzakhala chifukwa chosuntha zosungirako zapafupi kuchokera ku hypervisor kupita ku ina. Chokhacho chokha ndichakuti muyenera kuyimitsa VM pomwe ntchitoyo ikuchitika.

Chinsinsi chotsatirachi chatengedwa nkhani kuchokera pabulogu iyi, malangizo omwe ayesedwa ndikuchita. Ndisanayiwale, njira yakusamuka popanda zovuta ikufotokozedwanso pamenepo, komabe, kwa ife sizinali zofunikira, choncho sitinafufuze. Ngati izi ndizofunikira kwambiri pantchito yanu, tidzakhala okondwa kumva zotsatira zake mu ndemanga.

Tiyeni tipitirire ku gawo lothandiza. Mu chitsanzo timagwiritsa ntchito virsh ndipo, molingana, libvirt. Choyamba, onetsetsani kuti dziwe la Ceph lomwe deta idzasamutsidwira likugwirizana ndi libvirt:

virsh pool-dumpxml $ceph_pool

Kufotokozera kwa dziwe kuyenera kukhala ndi data yolumikizira ku Ceph yokhala ndi chidziwitso chovomerezeka.

Chotsatira ndichakuti chithunzi cha LVM chimasinthidwa kukhala Ceph RBD. Nthawi yokonza imadalira makamaka kukula kwa chithunzi:

qemu-img convert -p -O rbd /dev/main/$vm_image_name rbd:$ceph_pool/$vm_image_name

Pambuyo pa kutembenuka, chithunzi cha LVM chidzakhalabe, chomwe chingakhale chothandiza ngati kusamuka kwa VM kupita ku RBD sikulephera ndipo muyenera kubweza zosinthazo. Komanso, kuti muthe kubweza zosintha mwachangu, tiyeni tisungire zosunga zobwezeretsera makina osinthira:

virsh dumpxml $vm_name > $vm_name.xml
cp $vm_name.xml $vm_name_backup.xml

... ndikusintha choyambirira (vm_name.xml). Tiyeni tipeze chipika chokhala ndi kufotokozera kwa disk (kuyambira ndi mzere <disk type='file' device='disk'> ndipo akumaliza ndi </disk>) ndikuchichepetsa kukhala motere:

<disk type='network' device='disk'>
<driver name='qemu'/>
<auth username='libvirt'>
  <secret type='ceph' uuid='sec-ret-uu-id'/>
 </auth>
<source protocol='rbd' name='$ceph_pool/$vm_image_name>
  <host name='10.0.0.1' port='6789'/>
  <host name='10.0.0.2' port='6789'/>
</source>
<target dev='vda' bus='virtio'/> 
<alias name='virtio-disk0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</disk>

Tiyeni tiwone tsatanetsatane:

1. Ku protocol source adilesi yosungirako ku Ceph RBD ikuwonetsedwa (iyi ndi adilesi yomwe ikuwonetsa dzina la dziwe la Ceph ndi chithunzi cha RBD, chomwe chidatsimikiziridwa pagawo loyamba).
2. M'mbali secret mtundu wasonyezedwa ceph, komanso UUID yachinsinsi kuti mugwirizane nayo. Uuid wake ukhoza kupezeka pogwiritsa ntchito lamulo virsh secret-list.
3. M'mbali host maadiresi kwa Ceph monitors akuwonetsedwa.

Pambuyo pokonza fayilo yosinthira ndikumaliza kutembenuka kwa LVM kukhala RBD, mutha kugwiritsa ntchito fayilo yosinthidwa ndikuyambitsa makina enieni:

virsh define $vm_name.xml
virsh start $vm_name

Yakwana nthawi yoti muwone ngati makinawo adayambira molondola: mutha kudziwa, mwachitsanzo, polumikizana nawo kudzera pa SSH kapena kudzera virsh.

Ngati makina enieni akugwira ntchito bwino ndipo simunapeze zovuta zina, mutha kuchotsa chithunzi cha LVM chomwe sichikugwiritsidwanso ntchito:

lvremove main/$vm_image_name

Pomaliza

Tidakumana ndi milandu yonse yomwe tafotokozayi - tikukhulupirira kuti malangizowo athandiza oyang'anira ena kuthetsa mavuto omwewo. Ngati muli ndi ndemanga kapena nkhani zina zofanana ndi zomwe mwakumana nazo pogwiritsa ntchito Ceph, tidzakhala okondwa kuziwona mu ndemanga!

PS

Werenganinso pa blog yathu:

• «Manja athu si otopetsa: kubwezeretsa gulu la Rook mu K8s";
• «Rook kapena ayi Rook, limenelo ndi funso";
• «Rook - "self-service" yosungiramo deta ya Kubernetes";
• «Kupanga kusungirako kosalekeza ndikupereka ku Kubernetes kutengera Ceph".

Source: www.habr.com