Mazano & matipi ekushanda neCeph mumapurojekiti akabatikana

Tichishandisa Ceph sekuchengetedza network mumapurojekiti ane mitoro yakasiyana, tinogona kusangana nemabasa akasiyana siyana ayo pakutanga aasingaite seakapusa kana diki. Semuyenzaniso:

• kutama kwe data kubva kuCeph yekare kuenda kune imwe nekushandiswa zvishoma kwemaseva apfuura musumbu idzva;
• mhinduro kune dambudziko rekugoverwa kwedhisiki nzvimbo muCeph.

Kutarisana nezvinetso zvakadaro, takatarisana nechinodiwa chekubvisa zvakarurama OSD pasina kurasikirwa nedheta, iyo inonyanya kukosha pakubata nehuwandu hwemashoko. Izvi zvichakurukurwa munyaya.

Nzira dzakatsanangurwa pazasi dzakakosha kune chero vhezheni yeCeph. Uye zvakare, chokwadi chekuti Ceph inogona kuchengetedza yakawanda data ichaverengerwa: kudzivirira kurasikirwa kwedata uye mamwe matambudziko, zvimwe zviito "zvichapatsanurwa" mune mamwe akati wandei.

Nhanganyaya nezve OSD

Sezvo maviri ematatu ekubika akakurukurwa akatsaurirwa kuOSD (Chinhu Chekuchengetedza Daemon), usati wanyura muchikamu chinoshanda - muchidimbu nezvezviri muCeph uye nei zvakakosha.

Chekutanga pane zvese, zvinofanirwa kutaurwa kuti iyo yese Ceph cluster ine akawanda maOSD. Iyo yakawanda iripo, iyo yakakura iyo yemahara data vhoriyamu muCeph. Zviri nyore kunzwisisa kubva pano chikuru OSD basa: Inochengeta Ceph chinhu data pane faira masisitimu eese masumbu node uye inopa network kuwana kwairi (yekuverenga, kunyora, uye zvimwe zvikumbiro).

Padanho rimwe chete, kudzokorora paramita inoiswa nekukopa zvinhu pakati peOSD dzakasiyana. Uye pano unogona kusangana nematambudziko akasiyana-siyana, magadzirirwo ayo achakurukurwa pasi apa.

Nyaya nhamba 1. Bvisa zvakachengeteka OSD kubva kuCeph cluster pasina kurasikirwa nedata

Kudiwa kwekubvisa OSD kunogona kukonzerwa nekubvisa sevha kubva musumbu - semuenzaniso, kuitsiva neimwe sevha - izvo zvakaitika kwatiri, zvichipa kunyorwa kwechinyorwa ichi. Nekudaro, chinangwa chekupedzisira chekunyengera ndechekubvisa ese maOSD nemamons pane yakapihwa server kuti imiswe.

Kuti zvive nyore uye kudzivirira mamiriro ezvinhu apo, patinenge tichiita mirairo, tinoita chikanganiso mukuratidza OSD inodiwa, tichaisa yakasiyana yakasiyana, kukosha kwayo kuchava nhamba yeOSD ichabviswa. Ngatimufonerei ${ID} - pano ne pazasi, shanduko yakadaro inotsiva nhamba yeOSD yatiri kushanda nayo.

Ngatitarisei mamiriro acho tisati tatanga basa:

root@hv-1 ~ # ceph osd tree
ID CLASS WEIGHT  TYPE NAME      STATUS REWEIGHT PRI-AFF
-1       0.46857 root default
-3       0.15619      host hv-1
-5       0.15619      host hv-2
 1   ssd 0.15619      osd.1     up     1.00000  1.00000
-7       0.15619      host hv-3
 2   ssd 0.15619      osd.2     up     1.00000  1.00000

Kuti utange kubviswa kweOSD, iwe uchafanirwa kuita zvakanaka reweight pairi kusvika zero. Nenzira iyi tinodzikisa huwandu hwe data muOSD nekuienzanisa kune mamwe maOSD. Kuti uite izvi, shandisa mirairo inotevera:

ceph osd reweight osd.${ID} 0.98
ceph osd reweight osd.${ID} 0.88
ceph osd reweight osd.${ID} 0.78

... uye zvichingodaro kusvika zero.

Smooth balancing inodiwakuitira kuti usarase data. Izvi ndezvechokwadi kunyanya kana OSD iine huwandu hukuru hwe data. Kuita shuwa kuti mushure mekuita mirairo reweight zvese zvakafamba zvakanaka, unogona kuzvipedzisa ceph -s kana mune yakaparadzana terminal hwindo run ceph -w kuitira kuona shanduko munguva chaiyo.

Kana iyo OSD "yakadururwa", unogona kuenderera neyakajairwa oparesheni kuti uibvise. Kuti uite izvi, endesa iyo yaunoda OSD kune iyo nyika down:

ceph osd down osd.${ID}

Ngati "dhonza" iyo OSD kubva musumbu:

ceph osd out osd.${ID}

Ngatimisei sevhisi yeOSD uye tibvise chikamu chayo muFS:

systemctl stop ceph-osd@${ID}
umount /var/lib/ceph/osd/ceph-${ID}

Bvisa OSD kubva CRUSH mepu:

ceph osd crush remove osd.${ID}

Ngatibvisei OSD mushandisi:

ceph auth del osd.${ID}

Uye pakupedzisira, ngatibvise OSD pachayo:

ceph osd rm osd.${ID}

taura pfungwa: Kana uri kushandisa Ceph Luminous vhezheni kana kupfuura, ipapo matanho ari pamusoro ekubvisa OSD anogona kudzikiswa kuita mirairo miviri:

ceph osd out osd.${ID}
ceph osd purge osd.${ID}

Kana, mushure mekupedza matanho anotsanangurwa pamusoro, iwe unomhanya murairo ceph osd tree, zvino zvinofanirwa kuve pachena kuti pane sevha pakaitirwa basa hapasisina OSDs iyo yakaitwa pamusoro apa:

root@hv-1 ~ # ceph osd tree
ID CLASS WEIGHT  TYPE NAME     STATUS REWEIGHT PRI-AFF
-1       0.46857      root default
-3       0.15619      host hv-1
-5       0.15619      host hv-2
-7       0.15619      host hv-3
 2   ssd 0.15619      osd.2    up     1.00000  1.00000

Panzira, cherechedza kuti mamiriro eboka reCeph achaenda HEALTH_WARN, uye isu tichaonawo kuderera kwenhamba yeOSD uye uwandu huripo disk space.

Izvi zvinotevera zvinotsanangura matanho anozodiwa kana iwe uchida kumisa zvachose sevha uye, saizvozvo, bvisa kubva kuCeph. Muchiitiko ichi, zvakakosha kuyeuka izvozvo Usati wavhara sevha, unofanirwa kubvisa ese maOSD pane iyi server.

Kana kusisina OSD asara pane iyi server, saka mushure mekuabvisa iwe unofanirwa kusabvisa sevha kubva kuOSD mepu. hv-2nekumhanyisa murairo unotevera:

ceph osd crush rm hv-2

Delete mon kubva kune server hv-2nekumhanyisa rairo pazasi pane imwe server (kureva mune ino kesi, on hv-1):

ceph-deploy mon destroy hv-2

Mushure meizvi, unogona kumisa sevha uye wotanga zviito zvinotevera (kuitumirazve, nezvimwewo).

Nyaya yechipiri. Kugoverwa kwedhisiki nzvimbo mune yakatogadzirwa Ceph cluster

Ndichatanga nyaya yechipiri nenhanganyaya nezvePG (Placement Groups) Basa guru rePG muCeph ndere kuunganidza zvinhu zveCeph uye nekuwedzera kuzvidzokorora muOSD. Iyo formula iyo iwe yaunogona kuverenga iyo inodiwa huwandu hwePG iri mukati chikamu chakakodzera Ceph zvinyorwa. Nyaya iyi inokurukurwawo ipapo nemienzaniso chaiyo.

Saka: rimwe rematambudziko akajairika kana uchishandisa Ceph ndiyo isina kuenzana nhamba yeOSD uye PG pakati pemadziva muCeph.

Kutanga, nekuda kweizvi, mamiriro ezvinhu anogona kuitika kana maPG akawanda akatsanangurwa mune dziva duku, iyo inonyanya kushandiswa kusina musoro kwe diski nzvimbo musumbu. Chechipiri, mukuita pane dambudziko rakanyanya: data kufashukira mune imwe yeOSDs. Izvi zvinosanganisira cluster yekutanga shanduko kuenda kuhurumende HEALTH_WARN, ndokuzo HEALTH_ERR. Chikonzero cheizvi ndechekuti Ceph, kana uchiverenga huwandu huripo hwe data (unogona kuiwana na MAX AVAIL mune yekuraira kubuda ceph df padziva rega rega zvakasiyana) zvinoenderana nehuwandu hwe data iripo muOSD. Kana pasina nzvimbo yakakwana mune inokwana OSD imwe chete, hapana imwe data inogona kunyorwa kudzamara iyo data yagovaniswa zvakanaka pakati peOSD yese.

Zvakakodzera kujekesa kuti matambudziko aya zvinonyanya kusarudzwa paCeph cluster kumisikidza nhanho. Imwe yezvishandiso zvaunogona kushandisa ndeye Ceph PGCalc. Nekubatsira kwayo, chiyero chinodiwa chePG chinonyatso kuverengwa. Nekudaro, iwe unogona zvakare kutendeukira kwairi mumamiriro ezvinhu apo iyo Ceph cluster kare kugadzirwa zvisirizvo. Zvakakodzera kujekesa pano kuti sechikamu chekugadzirisa basa iwe ungangoda kudzikisa huwandu hwePGs, uye chimiro ichi hachiwanikwe mune yekare mavhezheni eCeph (yakangoonekwa muvhezheni. Nautilus).

Saka, ngatimbofungidzira mufananidzo unotevera: sumbu rine chimiro HEALTH_WARN nekuda kweimwe yeOSD iri kupera nzvimbo. Izvi zvicharatidzwa nekukanganisa HEALTH_WARN: 1 near full osd. Pazasi pane algorithm yekubuda mumamiriro aya.

Chekutanga pane zvese, iwe unofanirwa kugovera iyo iripo data pakati peasara OSDs. Isu takatoita oparesheni yakafanana mune yekutanga kesi, pataka "dhiza" node - nemusiyano chete watichazoda kuderedza zvishoma. reweight. Somuenzaniso, kusvika 0.95:

ceph osd reweight osd.${ID} 0.95

Izvi zvinosunungura dhisiki nzvimbo muOSD uye inogadzirisa chikanganiso muhutano hweceph. Zvisinei, sezvatotaurwa, dambudziko iri rinonyanya kuitika nekuda kwekugadzirisa kusina kunaka kweCeph pamatanho ekutanga: zvakakosha zvikuru kuita reconfiguration kuitira kuti irege kuonekwa mune ramangwana.

Muchiitiko chedu, zvese zvakadzika kune:

• kukosha kwakanyanya replication_count mune rimwe remadziva,
• yakawandisa PG mune rimwe dziva uye yakawandisa mune imwe.

Ngatishandisei Calculator yatotaurwa. Inoratidza zvakajeka izvo zvinoda kuiswa uye, mumutemo, hapana chakaoma. Mushure mekuisa iyo inodiwa paramita, tinowana anotevera kurudziro:

taura pfungwa: Kana iwe uri kumisikidza cluster yeCeph kubva kutanga, rimwe basa rinobatsira rekarukureta chizvarwa chemirairo chinozogadzira madziva kubva mukutanga nemaparamita akatsanangurwa patafura.

Koramu yekupedzisira inokubatsira kufamba - Yakakurudzirwa PG Count. Muchiitiko chedu, yechipiri inobatsirawo, uko kunoratidzwa parameter yekudzokorora, sezvo isu takasarudza kushandura replication multiplier.

Saka, chokutanga iwe unofanirwa kushandura maparameter ekudzokorora - izvi zvakakosha kuita kutanga, sezvo nekudzikisa kuwandisa, isu tichasunungura dhisiki nzvimbo. Sezvo murairo uchiita, iwe uchaona kuti inowanikwa disk nzvimbo ichawedzera:

ceph osd pool $pool_name set $replication_size

Uye mushure mekupedzwa kwayo, tinoshandura maitiro eparameter pg_num и pgp_num sezvinotevera:

ceph osd pool set $pool_name pg_num $pg_number
ceph osd pool set $pool_name pgp_num $pg_number

kosha: isu tinofanirwa kushandura huwandu hwePGs sequentially mudziva rega rega uye kwete kushandura hunhu mune mamwe madziva kusvika yambiro yanyangarika. "Degraded data redundancy" и "n-nhamba yepgs yakaderedzwa".

Iwe unogona zvakare kutarisa kuti zvese zvakafamba zvakanaka uchishandisa mirairo yekuraira ceph health detail и ceph -s.

Nyaya nhamba 3. Kutamisa muchina chaiwo kubva kuLVM kuenda kuCeph RBD

Mumamiriro ezvinhu apo purojekiti inoshandisa machina chaiwo akaiswa pamasevha akarenda asina-simbi, nyaya yekutadza-inoshivirira kuchengetedza inowanzomuka. Izvo zvinonyanya kudiwa kuti pane nzvimbo yakakwana munzvimbo ino yekuchengetedza ... Imwe mamiriro ezvinhu akajairika: kune dhiyabhorosi ine chengetedzo yemunharaunda pane sevha uye iwe unofanirwa kuwedzera dhisiki, asi hapana kwekuenda, nekuti hapana. free disk space yasara pane server.

Dambudziko rinogona kugadziriswa nenzira dzakasiyana - semuenzaniso, nekutamira kune imwe sevha (kana iripo) kana kuwedzera madhisiki matsva kune server. Asi hazvigoneke nguva dzose kuita izvi, saka kutama kubva kuLVM kuenda kuCeph kunogona kuve mhinduro yakanaka kudambudziko iri. Nekusarudza iyi sarudzo, isu zvakare tinorerutsa imwe nzira yekufambisa pakati pemaseva, sezvo pachange pasina chikonzero chekufambisa chengetedzo yenzvimbo kubva kune imwe hypervisor kuenda kune imwe. Chinongobata chete ndechekuti uchafanira kumisa VM basa richiitwa.

Recipe inotevera inotorwa kubva chinyorwa kubva kune ino blog, mirayiridzo iyo yakaedzwa mukuita. Zviya, nzira yekufamba-famba pasina kunetseka inotsanangurwawo ipapo, zvisinei, kwatiri zvakanga zvisingadiwi, saka hatina kuzviongorora. Kana izvi zvakakoshera purojekiti yako, isu tichafara kunzwa nezve mhedzisiro mumhinduro.

Ngatienderere mberi kune chikamu chinoshanda. Mumuenzaniso isu tinoshandisa virsh uye, maererano, libvirt. Kutanga, ita shuwa kuti dziva reCeph uko data richaendeswa rakabatana kune libvirt:

virsh pool-dumpxml $ceph_pool

Rondedzero yedziva inofanirwa kunge iine data yekubatanidza kuCeph ine mvumo yedata.

Nhanho inotevera ndeyekuti mufananidzo weLVM unoshandurwa kuita Ceph RBD. Nguva yekuuraya inoenderana zvakanyanya nehukuru hwemufananidzo:

qemu-img convert -p -O rbd /dev/main/$vm_image_name rbd:$ceph_pool/$vm_image_name

Mushure mekushandura, mufananidzo weLVM ucharamba uripo, unozobatsira kana kutama VM kuenda kuRBD kukatadza uye unofanirwa kudzoreredza shanduko. Zvakare, kuti tikwanise kukurumidza kudzosera kumashure shanduko, ngatiite backup yeiyo chaiyo muchina kumisikidza faira:

virsh dumpxml $vm_name > $vm_name.xml
cp $vm_name.xml $vm_name_backup.xml

... uye gadzirisa yekutanga (vm_name.xml) Ngatitsvagei block ine tsananguro ye diski (inotanga nemutsara <disk type='file' device='disk'> uye inopera ne </disk>) uye udzikise kune inotevera fomu:

<disk type='network' device='disk'>
<driver name='qemu'/>
<auth username='libvirt'>
  <secret type='ceph' uuid='sec-ret-uu-id'/>
 </auth>
<source protocol='rbd' name='$ceph_pool/$vm_image_name>
  <host name='10.0.0.1' port='6789'/>
  <host name='10.0.0.2' port='6789'/>
</source>
<target dev='vda' bus='virtio'/> 
<alias name='virtio-disk0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</disk>

Ngatitarisei mamwe mashoko:

1. Ku protocol source kero yekuchengetera muCeph RBD inoratidzwa (iyi ndiyo kero inoratidza zita reCeph dziva uye mufananidzo weRBD, wakatemerwa padanho rekutanga).
2. Muchivako secret mhando inoratidzwa ceph, pamwe neUUID yechakavanzika chekubatanidza nayo. Uuid yayo inogona kuwanikwa uchishandisa murairo virsh secret-list.
3. Muchivako host kero kuCeph monitors inoratidzwa.

Mushure mekugadzirisa iyo faira yekumisikidza uye kupedzisa iyo LVM kuenda kuRBD shanduko, unogona kuisa iyo yakagadziridzwa faira yekumisikidza uye wotanga iwo chaiwo muchina:

virsh define $vm_name.xml
virsh start $vm_name

Yave nguva yekutarisa kuti chaiyo muchina wakatanga nemazvo: unogona kuziva, semuenzaniso, nekubatanidza nayo kuburikidza neSSH kana kuburikidza virsh.

Kana iyo chaiyo muchina uri kushanda nemazvo uye usati wawana chero mamwe matambudziko, saka unogona kudzima iyo LVM mufananidzo usingachashandiswa:

lvremove main/$vm_image_name

mhedziso

Takasangana nezviitiko zvese zvakatsanangurwa mukuita - tinovimba kuti mirairo ichabatsira vamwe vatariri kugadzirisa matambudziko akafanana. Kana iwe uine makomendi kana dzimwe nyaya dzakafanana kubva pane chako chiitiko uchishandisa Ceph, isu tichafara kuvaona mumhinduro!

PS

Verenga zvakare pablog yedu:

• «Maoko edu haasi ekufinhwa: kudzoreredza iro Rook cluster muK8s";
• «Rook kana kwete Rook, ndiwo mubvunzo";
• «Rook - "self-service" data warehouse yeKubernetes";
• «Kugadzira inoenderera kuchengetedza nekupa muKubernetes yakavakirwa paCeph".

Source: www.habr.com