Talooyin & tabaha lagula shaqeeyo Ceph mashaariicda mashquulka ah

Isticmaalka Ceph sida kaydinta shabakada mashaariicda leh culeysyo kala duwan, waxaa laga yaabaa inaan la kulanno hawlo kala duwan oo jaleecada hore aan u muuqan mid fudud ama aan waxtar lahayn. Tusaale ahaan:

• u guurista xogta Ceph hore una guurtay mid cusub iyadoo qayb ahaan adeegsigii hore ee kutlada cusub;
• xallinta dhibaatada qoondaynta booska diskka ee Ceph.

Wax ka qabashada dhibaatooyinkan oo kale, waxaan la kulannaa baahida loo qabo in si sax ah loo saaro OSD iyada oo aan lumin xogta, taas oo si gaar ah muhiim u ah marka la macaamilayo xog badan. Tan waxaa looga hadli doonaa maqaalka.

Hababka hoos lagu sharraxay waxay khuseeyaan nooc kasta oo Ceph ah. Intaa waxaa dheer, xaqiiqda ah in Ceph uu kaydin karo tiro badan oo xog ah ayaa lagu xisaabtami doonaa: si looga hortago luminta xogta iyo dhibaatooyinka kale, falalka qaarkood ayaa loo "kala qaybin doonaa" dhowr kale.

Hordhac ku saabsan OSD

Maadaama laba ka mid ah saddexda cunto ee laga wada hadlay ay u heellan yihiin OSD (Kaydinta Shayga Daemon), ka hor inta aanad u gelin qaybta la taaban karo - si kooban oo ku saabsan waxa ay ku jirto Ceph iyo sababta ay muhiim u tahay.

Marka hore, waa in la sheegaa in dhammaan kooxda Ceph ay ka kooban tahay OSD-yo badan. Inta badan ee ay jiraan, way sii weynaataa mugga xogta bilaashka ah ee Ceph. Way fududahay in halkan laga fahmo shaqada ugu weyn ee OSD: Waxay ku kaydisaa xogta shayga Ceph ee nidaamka faylalka dhammaan qanjidhada kooxda waxayna siisaa marin u helka shabakadeeda (wax akhrinta, qorista, iyo codsiyada kale).

Isla heerkaas, halbeegyada ku celcelinta waxaa lagu dejiyaa iyadoo la koobiyaynayo walxaha u dhexeeya OSD-yada kala duwan. Oo halkan waxaad la kulmi kartaa dhibaatooyin kala duwan, kuwaas oo xalalka kuwaas oo hoos looga hadli doonaa.

Kiis No. 1. Si badbaado leh uga saar OSD kooxda Ceph adoon lumin xogta

Baahida loo qabo in laga saaro OSD-ga waxa sabab u ah ka saarida server-ka kutlada - tusaale ahaan, in lagu beddelo server kale - taas oo nagu dhacday, taasoo keentay qoritaanka maqaalkan. Markaa, yoolka ugu dambeeya ee wax-is-daba-marinta ayaa ah in la soo saaro dhammaan OSD-yada iyo mons-ka server-ka la siiyay si loo joojiyo.

Si ay ugu habboonaato iyo si looga fogaado xaalad, marka fulinta amarrada, waxaan samayn doonaa qalad ah tilmaamaya OSD loo baahan yahay, waxaan dejin doonaa doorsoom gaar ah, kaas oo qiimihiisu noqon doonaa tirada OSD in la tirtiro. Aan soo wacno ${ID} - halkan iyo hoosta, doorsoomayaasha noocan oo kale ah ayaa beddelaya tirada OSD ee aan ku shaqeyneyno.

Aynu eegno xaalada ka hor inta aanad bilaabin shaqada:

root@hv-1 ~ # ceph osd tree
ID CLASS WEIGHT  TYPE NAME      STATUS REWEIGHT PRI-AFF
-1       0.46857 root default
-3       0.15619      host hv-1
-5       0.15619      host hv-2
 1   ssd 0.15619      osd.1     up     1.00000  1.00000
-7       0.15619      host hv-3
 2   ssd 0.15619      osd.2     up     1.00000  1.00000

Si aad u bilowdo saarista OSD, waxaad u baahan doontaa inaad si habsami leh u qabato reweight dusheeda ilaa eber. Sidan ayaanu ku dhimaynaa cadadka xogta ku jirta OSD annagoo isu dheellitirna OSD-yada kale. Si tan loo sameeyo, socodsii amarrada soo socda:

ceph osd reweight osd.${ID} 0.98
ceph osd reweight osd.${ID} 0.88
ceph osd reweight osd.${ID} 0.78

... iyo wixii la mid ah ilaa eber.

Isku dheellitirnaan khafiif ah ayaa loo baahan yahaysi aanay xogta u lumin. Tani waxay si gaar ah run u tahay haddii OSD ka kooban yahay xog badan. Si loo hubiyo in ka dib fulinta amarrada reweight wax walba si fiican ayey u dhaceen, waad dhameystiri kartaa ceph -s ama daaqad terminal gaar ah oo orod ceph -w si loo ilaaliyo isbedelada wakhtiga dhabta ah.

Marka OSD "la faaruqiyo", waxaad sii wadi kartaa hawlgalka caadiga ah si aad uga saarto. Si tan loo sameeyo, u wareeji OSD-ga la doonayo gobolka down:

ceph osd down osd.${ID}

Aan "ka saarno" OSD kooxda:

ceph osd out osd.${ID}

Aynu joojino adeegga OSD oo aan ka saarno qaybteeda FS:

systemctl stop ceph-osd@${ID}
umount /var/lib/ceph/osd/ceph-${ID}

Ka saar OSD Khariidadda CRUSH:

ceph osd crush remove osd.${ID}

Aynu tirtirno isticmaalaha OSD:

ceph auth del osd.${ID}

Ugu dambayntii, aynu iska saarno OSD lafteeda:

ceph osd rm osd.${ID}

tacliiqHaddii aad isticmaalayso nooca Ceph Luminous ama ka sareeya, markaa tillaabooyinka ka saarista OSD ee kor ku xusan waxaa lagu yarayn karaa laba amar:

ceph osd out osd.${ID}
ceph osd purge osd.${ID}

Haddii, ka dib markaad dhammaystirto tillaabooyinka kor lagu sharraxay, aad socodsiiso amarka ceph osd tree, markaas waa in ay caddaato in server-ka shaqada lagu qabtay aysan hadda jirin OSD-yada kuwaas oo lagu sameeyay hawlgallada kor ku xusan:

root@hv-1 ~ # ceph osd tree
ID CLASS WEIGHT  TYPE NAME     STATUS REWEIGHT PRI-AFF
-1       0.46857      root default
-3       0.15619      host hv-1
-5       0.15619      host hv-2
-7       0.15619      host hv-3
 2   ssd 0.15619      osd.2    up     1.00000  1.00000

Inta jidka lagu jiro, ogow in xaaladda kooxda Ceph ay tagi doonto HEALTH_WARN, waxaanan sidoo kale arki doonaa hoos u dhac ku yimaada tirada OSD-yada iyo cadadka booska saxanka ee la heli karo.

Kuwa soo socdaa waxay sharxi doonaan tillaabooyinka loo baahan doono haddii aad rabto inaad gabi ahaanba joojiso server-ka oo, si waafaqsan, ka saar Ceph. Xaaladdan oo kale, waa muhiim in la xasuusto taas Kahor intaadan xirin server-ka, waa inaad ka saartaa dhammaan OSD-yada on server this.

Haddii aysan jirin OSD dheeraad ah oo ku haray server-kan, ka dib markaad ka saarto waxaad u baahan tahay inaad ka saarto server-ka khariidada OSD hv-2adiga oo socodsiinaya amarka soo socda:

ceph osd crush rm hv-2

Tirtir mon ka server-ka hv-2adoo ku socodsiinaya amarka hoose ee server kale (ie kiiskan, on hv-1):

ceph-deploy mon destroy hv-2

Taas ka dib, waxaad joojin kartaa server-ka oo aad bilaabi kartaa ficilada xiga (dib u geynta, iwm.).

Kiis No. 2. Qaybinta booska saxanka ee kutlada Ceph horay loo abuuray

Waxaan sheekada labaad ku bilaabayaa horudhac ku saabsan PG (Kooxaha Meelaynta). Doorka ugu weyn ee PG ee Ceph waa in la isku daro walxaha Ceph oo lagu sii kordhiyo iyaga OSD. Qaabka aad ku xisaabin karto cadadka loo baahan yahay ee PG ayaa ku jira qaybta khuseeya Dukumentiyada Ceph Arintan ayaa sidoo kale halkaas looga hadlay tusaalayaal gaar ah.

Marka: mid ka mid ah dhibaatooyinka caadiga ah marka la isticmaalayo Ceph waa tirada aan dheelitirnayn ee OSD iyo PG inta u dhaxaysa barkadaha Ceph.

Marka hore, sababtan awgeed, xaalad ayaa laga yaabaa inay soo baxdo marka PG-yo badan lagu qeexo barkad yar, taas oo asal ahaan ah isticmaalka aan macquul ahayn ee booska diskka ee kutlada. Marka labaad, ficil ahaan waxaa jirta dhibaato ka sii daran: xogta oo buux dhaafisay mid ka mid ah OSD-yada. Tani waxa ay keeneysaa in kooxdu ay marka hore u gudubto gobolka HEALTH_WARNka dibna HEALTH_ERR. Sababta tani waa Ceph, marka la xisaabinayo tirada xogta la heli karo (waxaad ku ogaan kartaa by MAX AVAIL in wax soo saarka amarka ceph df barkad kasta si gaar ah) waxay ku salaysan tahay cadadka xogta la heli karo ee OSD. Haddii aysan jirin meel bannaan oo ku filan ugu yaraan hal OSD, xog dheeraad ah lama qori karo ilaa xogta si habboon loogu qaybiyo dhammaan OSD-yada.

Waxaa habboon in la caddeeyo dhibaatooyinkaas inta badan waxaa lagu go'aamiyaa heerka qaabeynta kooxda Ceph. Mid ka mid ah qalabka aad isticmaali karto waa Ceph PGCalc. Iyada oo caawinteeda, qaddarka loo baahan yahay ee PG ayaa si cad loo xisaabiyaa. Si kastaba ha ahaatee, waxaad sidoo kale u adeegsan kartaa xaalad ay ku jirto kooxda Ceph horay si khaldan loo habeeyey. Waxaa habboon in halkan lagu caddeeyo in qayb ka mid ah shaqada hagaajinta ay u badan tahay inaad u baahan tahay inaad yarayso tirada PG-yada, oo sifadan lagama heli karo noocyadii hore ee Ceph (waxay u muuqatay kaliya nooca Nautilus).

Haddaba, aan qiyaasno sawirka soo socda: kooxdu waxay leedahay xaalad HEALTH_WARN sababtoo ah mid ka mid ah OSD oo gabaabsi ah. Tan waxaa lagu tilmaami doonaa qalad HEALTH_WARN: 1 near full osd. Hoos waxaa ku yaal algorithm si looga baxo xaaladdan.

Marka hore, waxaad u baahan tahay inaad u qaybiso xogta la heli karo inta u dhaxaysa OSD-yada haray. Waxaan horeyba u sameynay qalliin la mid ah kiiskii ugu horreeyay, markii aan "miiray" noodhka - farqiga kaliya ee hadda waxaan u baahan doonaa inaan wax yar yareyno. reweight. Tusaale ahaan, ilaa 0.95:

ceph osd reweight osd.${ID} 0.95

Tani waxay xoraynaysaa booska diskka ee OSD waxayna hagaajinaysaa khaladka caafimaadka ceph. Si kastaba ha ahaatee, sidii horeba loo soo sheegay, dhibaatadani waxay inta badan ku dhacdaa sababtoo ah qaabeynta khaldan ee Ceph marxaladaha hore: aad bay muhiim u tahay in la sameeyo dib u habeyn si aysan u muuqan mustaqbalka.

Xaaladeena gaarka ah, dhammaan waxay ku soo degeen:

• qiimo aad u sarreeya replication_count mid ka mid ah barkadaha,
• PG aad u badan hal barkad iyo mid kale oo aad u yar.

Aynu isticmaalno xisaabiyaha hore loo sheegay. Waxay si cad u muujineysaa waxa loo baahan yahay in la galo, mabda'a ahaan, ma jiraan wax adag. Markaan dejinay cabbirrada lagama maarmaanka ah, waxaan helnaa talooyinka soo socda:

tacliiq: Haddii aad dejinayso koox Ceph ah oo xoq ah, shaqo kale oo faa'iido leh oo xisaabiyaha ayaa ah jiilka amarrada kuwaas oo ka abuuri doona barkado xoq ah oo leh cabbirada lagu qeexay shaxda.

Tiirka u dambeeya wuxuu kaa caawinayaa inaad dhex socoto - Tirinta PG ee la soo jeediyay. Xaaladeena, kan labaad ayaa sidoo kale faa'iido leh, halkaas oo cabbirka dib-u-celinta lagu tilmaamay, tan iyo markii aan go'aansanay inaan bedelno isku-dhufashada.

Sidaa darteed, marka hore waxaad u baahan tahay inaad bedesho cabbirada soo noqnoqda - tani waa mid mudan in marka hore la sameeyo, tan iyo marka la yareeyo xajmiga, waxaanu xoreyn doonaa booska diskka. Sida amarku u fuliyo, waxaad ogaan doontaa in booska diskka ee jira uu kordhin doono:

ceph osd pool $pool_name set $replication_size

Kadibna ka dib markii la dhammeeyo, waxaanu bedelnaa qiyamka qiyaasta pg_num и pgp_num sida soo socota:

ceph osd pool set $pool_name pg_num $pg_number
ceph osd pool set $pool_name pgp_num $pg_number

muhiim ah: Waa in aan u beddelnaa tirada PG-yada si isdaba joog ah barkad kasta oo aynaan beddelin qiyamka barkadaha kale ilaa digniintu ay meesha ka baxdo "ka-noqoshada xogta hoos u dhacday" и "n-lambarka bgs waa la dhimay".

Waxa kale oo aad hubin kartaa in wax walba si fiican u dhaceen adiga oo isticmaalaya soo-saarka amarka ceph health detail и ceph -s.

Kiis No. 3. U guurista mashiinka farsamada ee LVM una guurayo Ceph RBD

Marka ay jirto xaalad uu mashruucu isticmaalo mashiinada farsamada gacanta ee lagu rakibay adeegayaasha birta ah ee la kiraysto, arrinta kaydinta u dulqaadashada khaladaadka ayaa badanaa soo baxda. Waxa kale oo aad loo jecel yahay in ay jirto meel ku filan kaydintan ... Xaalad kale oo caadi ah: waxaa jira mashiinka farsamada leh ee kaydinta maxaliga ah ee server-ka waxaadna u baahan tahay inaad ballaariso diskka, laakiin ma jirto meel aad tagto, sababtoo ah ma jirto. meel diskooga oo bilaash ah ayaa kaga hadhay server-ka.

Dhibaatada waxaa lagu xalin karaa siyaabo kala duwan - tusaale ahaan, in loo guuro server kale (haddii uu jiro) ama lagu daro saxan cusub serverka. Laakiin had iyo jeer suurtagal maaha in tan la sameeyo, markaa ka guurista LVM ilaa Ceph waxay noqon kartaa xal aad u fiican dhibaatadan. Doorashadan ikhtiyaarkan, waxaan sidoo kale fududeyneynaa habka dheeraadka ah ee socdaalka ee u dhexeeya server-yada, maadaama aysan jiri doonin baahi loo qabo in kaydinta maxalliga ah laga wareejiyo hal hypervisor oo kale. Waxa kaliya ee la qabanayo ayaa ah inaad joojiso VM-ga inta shaqada la qabanayo.

Cuntada soo socota ayaa laga soo qaatay maqaal ka blog this, tilmaamaha kuwaas oo lagu tijaabiyey ficil ahaan. Dhanka kale, Habka socdaalka bilaa dhibka ah ayaa isna halkaas lagu tilmaamay, si kastaba ha ahaatee, kiiskayaga si fudud looma baahnayn, markaa maanu hubin. Haddii tani ay muhiim u tahay mashruucaaga, waxaan ku farxi doonaa inaan maqalno natiijada faallooyinka.

Aan u gudubno qaybta dhabta ah. Tusaalaha waxaan isticmaalnaa virsh iyo, si waafaqsan, libvirt. Marka hore, hubi in barkadda Ceph ee xogta loo haajiray ay ku xidhan tahay libvirt:

virsh pool-dumpxml $ceph_pool

Sharaxaadda barkadda waa in ay ka kooban tahay xogta ku xidhidhiyaha Ceph oo wata xogta oggolaanshaha.

Tallaabada xigta waa in sawirka LVM loo rogo Ceph RBD. Waqtiga fulinta wuxuu ku xiran yahay inta ugu badan sawirka:

qemu-img convert -p -O rbd /dev/main/$vm_image_name rbd:$ceph_pool/$vm_image_name

Beddelka ka dib, sawirka LVM waa uu jirayaa, kaas oo faa'iido u yeelan doona haddii u haajiridda VM ee RBD ay guuldareysato oo waa inaad dib u soo celisaa isbeddelada. Sidoo kale, si aad awood ugu yeelatid inaad si dhakhso leh dib ugu rogaan isbeddelada, aynu samayno nuqul ka mid ah faylka qaabeynta mashiinka farsamada:

virsh dumpxml $vm_name > $vm_name.xml
cp $vm_name.xml $vm_name_backup.xml

... oo wax ka beddel asalka (vm_name.xml). Aynu helno baloog leh sharraxaad saxan ah (wuxuu ka bilaabmaa khadka <disk type='file' device='disk'> kuna dhamaata </disk>) kuna soo koob qaabkan soo socda:

<disk type='network' device='disk'>
<driver name='qemu'/>
<auth username='libvirt'>
  <secret type='ceph' uuid='sec-ret-uu-id'/>
 </auth>
<source protocol='rbd' name='$ceph_pool/$vm_image_name>
  <host name='10.0.0.1' port='6789'/>
  <host name='10.0.0.2' port='6789'/>
</source>
<target dev='vda' bus='virtio'/> 
<alias name='virtio-disk0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</disk>

Aynu eegno qaar ka mid ah faahfaahinta:

1. Ku socota borotokoolka source ciwaanka kaydinta Ceph RBD waa la tilmaamay (kani waa ciwaanka tilmaamaya magaca barkada Ceph iyo sawirka RBD, kaas oo la go'aamiyay marxaladii kowaad).
2. In block secret nooca ayaa lagu tilmaamay ceph, iyo sidoo kale UUID ee sirta ah in lagu xiro. Uuidkeeda waxaa laga heli karaa iyadoo la adeegsanayo amarka virsh secret-list.
3. In block host Cinwaannada kormeerayaasha Ceph ayaa la tilmaamayaa.

Ka dib markaad tafatirto faylka qaabeynta oo aad dhammaystirto beddelka LVM ilaa RBD, waxaad codsan kartaa faylka qaabeynta wax laga beddelay oo waxaad bilaabi kartaa mashiinka farsamada:

virsh define $vm_name.xml
virsh start $vm_name

Waa waqtigii la hubin lahaa in mashiinka farsamada gacanta uu si sax ah u bilaabay: waxaad ku ogaan kartaa, tusaale ahaan, adigoo ku xiraya SSH ama iyada oo loo marayo virsh.

Haddii mashiinka farsamada uu si sax ah u shaqeeyo oo aadan helin wax dhibaato ah oo kale, markaa waxaad tirtiri kartaa sawirka LVM ee aan hadda la isticmaalin:

lvremove main/$vm_image_name

gunaanad

Waxaan la kulannay dhammaan kiisaska lagu sharraxay ficil ahaan - waxaan rajeyneynaa in tilmaamuhu ay ka caawin doonaan maamulayaasha kale inay xalliyaan dhibaatooyinka la midka ah. Haddii aad hayso faallooyin ama sheekooyin kale oo la mid ah khibradaada isticmaalka Ceph, waxaan ku farxi doonaa inaan ku aragno faallooyinka!

PS

Sidoo kale ka akhri boggayaga:

• «Gacmaheenu maaha caajis: soo celinta kooxda Rook ee K8s";
• «To Rook iyo in kale - taasi waa su'aasha";
• «Rook - bakhaarka xogta "is-adeegga" ee Kubernetes";
• «Abuuritaanka kaydinta joogtada ah ee bixinta Kubernetes ee ku salaysan Ceph".

Source: www.habr.com