Lub tswv yim & tricks rau kev ua hauj lwm nrog Ceph nyob rau hauv tej yaam num tibneeg hu tauj coob

Siv Ceph ua lub network cia hauv cov haujlwm nrog cov khoom sib txawv, peb yuav ntsib ntau yam haujlwm uas thaum xub thawj siab ib muag tsis zoo li yooj yim lossis tsis tseem ceeb. Piv txwv li:

• kev tsiv teb tsaws ntawm cov ntaub ntawv los ntawm Ceph qub mus rau ib qho tshiab nrog ib nrab siv cov servers yav dhau los hauv pawg tshiab;
• kev daws teeb meem ntawm qhov chaw disk faib hauv Ceph.

Kev daws cov teeb meem zoo li no, peb tau ntsib nrog qhov yuav tsum tau ua kom raug tshem tawm OSD yam tsis tau poob cov ntaub ntawv, uas yog qhov tseem ceeb tshwj xeeb tshaj yog thaum cuam tshuam nrog ntau cov ntaub ntawv. Qhov no yuav tau tham hauv tsab xov xwm.

Cov txheej txheem tau piav qhia hauv qab no muaj feem cuam tshuam rau txhua yam ntawm Ceph. Tsis tas li ntawd, qhov tseeb tias Ceph tuaj yeem khaws cov ntaub ntawv ntau yuav raug coj mus rau hauv tus account: txhawm rau tiv thaiv cov ntaub ntawv poob thiab lwm yam teeb meem, qee qhov kev ua yuav raug "sib cais" rau ntau lwm tus.

Ua ntej txog OSD

Txij li ob ntawm peb cov zaub mov txawv tau tham txog yog mob siab rau OSD (Object Storage Daemon), ua ntej dhia mus rau hauv qhov ua tau zoo - luv luv txog qhov nws nyob hauv Ceph thiab vim li cas nws tseem ceeb heev.

Ua ntej tshaj plaws, nws yuav tsum tau hais tias tag nrho Ceph pawg muaj ntau OSDs. Qhov ntau muaj, ntau dua cov ntaub ntawv dawb ntim hauv Ceph. Nws yooj yim to taub los ntawm no lub luag haujlwm OSD: Nws khaws cov ntaub ntawv Ceph ntawm cov ntaub ntawv ntawm txhua pawg ntawm pawg thiab muab kev nkag mus rau hauv network (rau kev nyeem ntawv, sau ntawv, thiab lwm yam kev thov).

Nyob rau hauv tib theem, replication parameters yog teem los ntawm kev luam cov khoom ntawm txawv OSDs. Thiab ntawm no koj tuaj yeem ntsib ntau yam teeb meem, cov kev daws teeb meem uas yuav tau tham hauv qab no.

Case No.1. Muaj kev nyab xeeb tshem tawm OSD los ntawm Ceph pawg yam tsis tau poob cov ntaub ntawv

Qhov yuav tsum tau tshem tawm OSD tuaj yeem tshwm sim los ntawm kev tshem tawm cov neeg rau zaub mov ntawm pawg - piv txwv li, hloov nws nrog lwm tus neeg rau zaub mov - uas yog qhov tshwm sim rau peb, ua rau muaj kev sau ntawv ntawm kab lus no. Yog li, lub hom phiaj kawg ntawm kev tswj hwm yog rho tawm tag nrho OSDs thiab mons ntawm ib lub server kom nws tuaj yeem nres.

Txhawm rau kom yooj yim thiab zam qhov xwm txheej uas, thaum ua tiav cov lus txib, peb ua yuam kev hauv kev qhia txog OSD xav tau, peb yuav teeb tsa qhov sib txawv, tus nqi uas yuav yog tus lej ntawm OSD yuav raug tshem tawm. Wb hu nws ${ID} - ntawm no thiab hauv qab no, qhov hloov pauv no hloov pauv tus lej OSD uas peb ua haujlwm.

Cia peb saib cov xwm txheej ua ntej pib ua haujlwm:

root@hv-1 ~ # ceph osd tree
ID CLASS WEIGHT  TYPE NAME      STATUS REWEIGHT PRI-AFF
-1       0.46857 root default
-3       0.15619      host hv-1
-5       0.15619      host hv-2
 1   ssd 0.15619      osd.1     up     1.00000  1.00000
-7       0.15619      host hv-3
 2   ssd 0.15619      osd.2     up     1.00000  1.00000

Txhawm rau pib OSD tshem tawm, koj yuav tsum ua kom zoo reweight ntawm nws mus rau xoom. Txoj kev no peb txo cov ntaub ntawv hauv OSD los ntawm kev sib npaug rau lwm OSDs. Txhawm rau ua qhov no, khiav cov lus txib hauv qab no:

ceph osd reweight osd.${ID} 0.98
ceph osd reweight osd.${ID} 0.88
ceph osd reweight osd.${ID} 0.78

... thiab mus txog rau xoom.

Smooth sib npaug yuav tsum tauthiaj li tsis poob cov ntaub ntawv. Qhov no yog qhov tseeb tshwj xeeb yog OSD muaj cov ntaub ntawv loj. Txhawm rau kom paub tseeb tias tom qab ua tiav cov lus txib reweight txhua yam mus zoo, koj tuaj yeem ua tiav ceph -s los yog nyob rau hauv ib tug nyias muaj nyias qhov rais qhov rais khiav ceph -w txhawm rau soj ntsuam kev hloov pauv hauv lub sijhawm.

Thaum OSD yog "kho", koj tuaj yeem ua haujlwm nrog cov txheej txheem kom tshem tawm nws. Txhawm rau ua qhov no, hloov OSD xav tau mus rau lub xeev down:

ceph osd down osd.${ID}

Cia peb "rub" OSD tawm ntawm pawg:

ceph osd out osd.${ID}

Cia peb nres qhov kev pabcuam OSD thiab tshem tawm nws qhov kev faib tawm hauv FS:

systemctl stop ceph-osd@${ID}
umount /var/lib/ceph/osd/ceph-${ID}

Tshem tawm OSD los ntawm CRUSH daim ntawv qhia:

ceph osd crush remove osd.${ID}

Cia peb rho tawm cov neeg siv OSD:

ceph auth del osd.${ID}

Thiab thaum kawg, cia peb tshem tawm OSD nws tus kheej:

ceph osd rm osd.${ID}

Примечание: Yog tias koj tab tom siv Ceph Luminous version lossis siab dua, tom qab ntawd OSD tshem tawm cov kauj ruam saum toj no tuaj yeem raug txo rau ob qho lus txib:

ceph osd out osd.${ID}
ceph osd purge osd.${ID}

Yog tias, tom qab ua tiav cov kauj ruam tau piav qhia saum toj no, koj khiav cov lus txib ceph osd tree, ces nws yuav tsum paub meej tias ntawm lub server qhov chaw ua haujlwm tsis muaj OSDs uas cov haujlwm saum toj no tau ua:

root@hv-1 ~ # ceph osd tree
ID CLASS WEIGHT  TYPE NAME     STATUS REWEIGHT PRI-AFF
-1       0.46857      root default
-3       0.15619      host hv-1
-5       0.15619      host hv-2
-7       0.15619      host hv-3
 2   ssd 0.15619      osd.2    up     1.00000  1.00000

Raws li txoj kev, nco ntsoov tias lub xeev ntawm Ceph pawg yuav mus HEALTH_WARN, thiab peb tseem yuav pom qhov txo qis ntawm OSDs thiab tus nqi ntawm qhov chaw disk.

Cov hauv qab no yuav piav qhia txog cov kauj ruam uas yuav tsum tau ua yog tias koj xav kom nres tag nrho cov server thiab, raws li, tshem tawm ntawm Ceph. Hauv qhov no, nws yog ib qho tseem ceeb kom nco ntsoov tias Ua ntej kaw lub server, koj yuav tsum tshem tag nrho OSDs ntawm no server.

Yog tias tsis muaj OSDs ntxiv rau ntawm lub server no, tom qab tshem lawv koj yuav tsum tshem tawm cov neeg rau zaub mov los ntawm OSD daim ntawv qhia hv-2los ntawm kev khiav cov lus txib hauv qab no:

ceph osd crush rm hv-2

Rho tawm mon los ntawm server hv-2los ntawm kev khiav cov lus txib hauv qab no ntawm lwm tus neeg rau zaub mov (piv txwv li hauv qhov no, on hv-1):

ceph-deploy mon destroy hv-2

Tom qab qhov no, koj tuaj yeem nres lub server thiab pib ua haujlwm tom ntej (rov xa nws, thiab lwm yam).

Case No. 2. Kev faib tawm ntawm qhov chaw disk hauv ib pawg Ceph uas twb tau tsim lawm

Kuv mam li pib zaj dab neeg thib ob nrog cov lus qhia txog PG (Pawg chaw). Lub luag haujlwm tseem ceeb ntawm PG hauv Ceph yog feem ntau los sau cov khoom Ceph thiab txuas ntxiv rau lawv hauv OSD. Cov mis uas koj tuaj yeem xam qhov xav tau ntawm PG yog nyob rau hauv muaj feem cuam tshuam Ceph cov ntaub ntawv. Qhov teeb meem no kuj tau tham nrog cov piv txwv tshwj xeeb.

Yog li: ib qho teeb meem tshwm sim thaum siv Ceph yog tus lej tsis sib npaug ntawm OSD thiab PG ntawm cov pas dej hauv Ceph.

Ua ntej, vim qhov no, qhov xwm txheej yuav tshwm sim thaum muaj ntau PGs tau teev tseg hauv lub pas dej me me, uas yog qhov tseem ceeb ntawm kev siv tsis sib haum xeeb ntawm qhov chaw disk hauv pawg. Qhov thib ob, hauv kev xyaum muaj qhov teeb meem loj dua: cov ntaub ntawv dhau mus hauv ib qho ntawm OSDs. Qhov no ua rau pawg thawj zaug hloov mus rau lub xeev HEALTH_WARN, thiab tom qab ntawd HEALTH_ERR. Yog vim li cas rau qhov no yog tias Ceph, thaum xam cov ntaub ntawv muaj (koj tuaj yeem nrhiav tau los ntawm MAX AVAIL nyob rau hauv cov lus txib tso zis ceph df rau txhua lub pas dej sib cais) yog raws li tus nqi ntawm cov ntaub ntawv muaj nyob hauv OSD. Yog tias tsis muaj qhov chaw txaus hauv tsawg kawg ib OSD, tsis muaj cov ntaub ntawv ntxiv tuaj yeem sau tau kom txog thaum cov ntaub ntawv raug muab faib rau tag nrho OSDs.

Nws tsim nyog qhia meej tias cov teeb meem no tau txiav txim siab loj heev ntawm Ceph pawg teeb tsa theem. Ib qho ntawm cov cuab yeej koj siv tau yog Ceph PGCalc. Nrog nws cov kev pab, qhov xav tau ntawm PG yog xam kom meej. Txawm li cas los xij, koj tuaj yeem hloov mus rau nws hauv qhov xwm txheej uas Ceph pawg twb configured tsis raug. Nws tsim nyog qhia meej ntawm no tias yog ib feem ntawm kev txhim kho kev ua haujlwm koj feem ntau yuav xav tau txo tus naj npawb ntawm PGs, thiab cov yam ntxwv no tsis muaj nyob rau hauv cov laus versions ntawm Ceph (nws tsuas yog tshwm sim hauv version. Nautilus).

Yog li, cia peb xav txog cov duab hauv qab no: pawg muaj xwm txheej HEALTH_WARN vim yog ib qho ntawm OSD khiav tawm ntawm qhov chaw. Qhov no yuav raug qhia los ntawm qhov yuam kev HEALTH_WARN: 1 near full osd. Hauv qab no yog ib qho algorithm kom tawm ntawm qhov teeb meem no.

Ua ntej tshaj plaws, koj yuav tsum faib cov ntaub ntawv muaj nyob nruab nrab ntawm OSDs ntxiv. Peb twb tau ua haujlwm zoo sib xws hauv thawj rooj plaub, thaum peb "drained" ntawm node - nrog rau qhov sib txawv uas tam sim no peb yuav tsum tau txo qis me ntsis. reweight. Piv txwv li, mus txog 0.95:

ceph osd reweight osd.${ID} 0.95

Qhov no tso tawm qhov chaw disk hauv OSD thiab kho qhov yuam kev hauv ceph noj qab haus huv. Txawm li cas los xij, raws li twb tau hais lawm, qhov teeb meem no feem ntau tshwm sim vim qhov teeb meem tsis raug ntawm Ceph ntawm cov theem pib: nws yog ib qho tseem ceeb heev uas yuav tsum tau ua kom rov kho dua kom nws tsis tshwm sim yav tom ntej.

Hauv peb qhov xwm txheej tshwj xeeb, txhua yam tau nqis los rau:

• tus nqi siab heev replication_count hauv ib lub pas dej,
• PG ntau dhau hauv ib lub pas dej thiab tsawg dhau hauv lwm qhov.

Cia peb siv lub tshuab xam zauv uas twb tau hais lawm. Nws qhia meej tias qhov yuav tsum tau nkag mus thiab, hauv paus ntsiab lus, tsis muaj dab tsi nyuaj. Tom qab teeb tsa cov kev ntsuas tsim nyog, peb tau txais cov lus pom zoo hauv qab no:

Примечание: Yog tias koj tab tom teeb tsa Ceph pawg los ntawm kos, lwm txoj haujlwm tseem ceeb ntawm lub laij lej yog lub cim ntawm cov lus txib uas yuav tsim cov pas dej los ntawm kos nrog cov tsis tau teev tseg hauv cov lus.

Kab kawg pab koj taug kev - Pom zoo PG suav. Nyob rau hauv peb cov ntaub ntawv, qhov thib ob yog ib qho tseem ceeb, qhov twg replication parameter yog qhia, txij li thaum peb txiav txim siab los hloov lub replication multiplier.

Yog li, ua ntej koj yuav tsum tau hloov pauv qhov tsis sib xws - qhov no yog qhov tsim nyog ua ntej, txij li los ntawm kev txo tus lej sib npaug, peb yuav tso qhov chaw disk. Raws li cov lus txib ua tiav, koj yuav pom tias qhov chaw muaj disk yuav nce ntxiv:

ceph osd pool $pool_name set $replication_size

Thiab tom qab nws ua tiav, peb hloov cov nqi parameter pg_num и pgp_num raws li nram no:

ceph osd pool set $pool_name pg_num $pg_number
ceph osd pool set $pool_name pgp_num $pg_number

ib qho tseem ceeb: peb yuav tsum hloov tus naj npawb ntawm PGs ua ntu zus hauv txhua lub pas dej thiab tsis txhob hloov cov nqi hauv lwm lub pas dej kom txog thaum cov lus ceeb toom ploj "Degraded cov ntaub ntawv redundancy" и "n-ntaus pgs degraded".

Koj tuaj yeem tshawb xyuas tias txhua yam mus zoo siv cov lus hais tawm ceph health detail и ceph -s.

Case No.3. Migrating lub tshuab virtual los ntawm LVM rau Ceph RBD

Nyob rau hauv ib qho xwm txheej uas ib qhov project siv cov tshuab virtual ntsia rau ntawm cov neeg xauj tsev liab qab-hlau, qhov teeb meem ntawm kev ua txhaum kev cia siab feem ntau tshwm sim. Nws kuj yog qhov xav tau heev uas muaj qhov chaw txaus hauv qhov chaw cia no ... Lwm qhov xwm txheej: muaj lub tshuab virtual nrog lub chaw khaws cia hauv lub server thiab koj yuav tsum nthuav tawm lub disk, tab sis tsis muaj qhov twg mus, vim tias tsis muaj. qhov chaw dawb disk sab laug ntawm lub server.

Qhov teeb meem tuaj yeem daws tau ntau txoj hauv kev - piv txwv li, los ntawm kev tsiv mus rau lwm tus neeg rau zaub mov (yog tias muaj) lossis ntxiv cov disks tshiab rau lub server. Tab sis nws tsis yog ib txwm ua tau li no, yog li kev tsiv teb tsaws ntawm LVM mus rau Ceph tuaj yeem yog qhov kev daws teeb meem zoo rau qhov teeb meem no. Los ntawm kev xaiv qhov kev xaiv no, peb kuj ua kom yooj yim cov txheej txheem ntxiv ntawm kev tsiv teb tsaws chaw ntawm servers, vim tias yuav tsis tas yuav txav chaw cia hauv zos los ntawm ib qho hypervisor mus rau lwm qhov. Qhov tsuas yog ntes tau yog tias koj yuav tsum nres VM thaum lub sijhawm ua haujlwm.

Daim ntawv qhia nram qab no yog muab los ntawm tsab xov xwm los ntawm blog no, cov lus qhia uas tau sim hauv kev nqis tes ua. Los ntawm txoj kev, txoj kev ntawm hassle-free migration kuj piav nyob rau ntawdTxawm li cas los xij, nyob rau hauv peb rooj plaub nws tsuas yog tsis xav tau, yog li peb tsis tau kuaj nws. Yog tias qhov no tseem ceeb rau koj qhov project, peb yuav zoo siab tau hnov ​​​​txog cov txiaj ntsig hauv cov lus pom.

Cia peb mus rau qhov ua tau zoo. Hauv qhov piv txwv peb siv virsh thiab, raws li, libvirt. Ua ntej, xyuas kom meej tias Ceph pas dej ua ke uas cov ntaub ntawv yuav tsiv teb tsaws yog txuas nrog libvirt:

virsh pool-dumpxml $ceph_pool

Cov lus piav qhia ntawm pas dej yuav tsum muaj cov ntaub ntawv txuas rau Ceph nrog cov ntaub ntawv tso cai.

Cov kauj ruam tom ntej yog tias daim duab LVM hloov mus rau Ceph RBD. Lub sijhawm ua tiav yog nyob ntawm qhov loj ntawm daim duab:

qemu-img convert -p -O rbd /dev/main/$vm_image_name rbd:$ceph_pool/$vm_image_name

Tom qab hloov dua siab tshiab, ib daim duab LVM yuav nyob twj ywm, uas yuav muaj txiaj ntsig yog tias kev hloov pauv VM rau RBD tsis ua haujlwm thiab koj yuav tsum rov qab hloov pauv. Tsis tas li ntawd, kom muaj peev xwm thim rov qab hloov pauv sai sai, cia peb ua ib qho thaub qab ntawm cov ntaub ntawv teeb tsa lub tshuab virtual:

virsh dumpxml $vm_name > $vm_name.xml
cp $vm_name.xml $vm_name_backup.xml

... thiab kho qhov qub (vm_name.xml). Cia peb pom ib qho thaiv nrog cov lus piav qhia ntawm lub disk (pib nrog kab <disk type='file' device='disk'> thiab xaus nrog </disk>) thiab txo nws mus rau hauv daim ntawv hauv qab no:

<disk type='network' device='disk'>
<driver name='qemu'/>
<auth username='libvirt'>
  <secret type='ceph' uuid='sec-ret-uu-id'/>
 </auth>
<source protocol='rbd' name='$ceph_pool/$vm_image_name>
  <host name='10.0.0.1' port='6789'/>
  <host name='10.0.0.2' port='6789'/>
</source>
<target dev='vda' bus='virtio'/> 
<alias name='virtio-disk0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</disk>

Cia peb saib qee cov ntsiab lus:

1. Mus rau txoj cai source qhov chaw nyob rau qhov chaw cia hauv Ceph RBD yog qhia (qhov no yog qhov chaw nyob qhia lub npe ntawm lub pas dej Ceph thiab RBD duab, uas tau txiav txim siab thawj theem).
2. Hauv qhov thaiv secret hom yog qhia ceph, nrog rau UUID ntawm qhov zais cia txuas rau nws. Nws uuid tuaj yeem pom siv cov lus txib virsh secret-list.
3. Hauv qhov thaiv host chaw nyob rau Ceph saib yog qhia.

Tom qab kho cov ntaub ntawv teeb tsa thiab ua tiav LVM rau RBD hloov dua siab tshiab, koj tuaj yeem siv cov ntaub ntawv hloov kho thiab pib lub tshuab virtual:

virsh define $vm_name.xml
virsh start $vm_name

Nws yog lub sijhawm los xyuas tias lub tshuab virtual pib raug: koj tuaj yeem paub, piv txwv li, los ntawm kev txuas rau nws ntawm SSH lossis ntawm virsh.

Yog tias lub tshuab virtual ua haujlwm raug thiab koj tsis pom muaj lwm yam teeb meem, koj tuaj yeem rho tawm LVM duab uas tsis siv lawm:

lvremove main/$vm_image_name

xaus

Peb ntsib txhua qhov xwm txheej tau piav qhia hauv kev coj ua - peb cia siab tias cov lus qhia yuav pab lwm tus thawj coj daws teeb meem zoo sib xws. Yog tias koj muaj lus pom lossis lwm cov dab neeg zoo sib xws los ntawm koj qhov kev paub siv Ceph, peb yuav zoo siab pom lawv hauv cov lus!

PS

Nyeem kuj ntawm peb blog:

• «Peb txhais tes tsis yog rau kev dhuav: rov ua kom lub Rook pawg hauv K8s»;
• «Rook los tsis Rook, uas yog cov lus nug»;
• «Rook - "kev pabcuam tus kheej" cov ntaub ntawv khaws cia rau Kubernetes»;
• «Tsim kom tsis tu ncua cia nrog kev muab hauv Kubernetes raws li Ceph".

Tau qhov twg los: www.hab.com