Pakakhala zambiri kuposa momwe zingakwanire pa disk imodzi, ndi nthawi yoganizira za RAID. Ndili mwana, nthawi zambiri ndinkamva kuchokera kwa akulu anga kuti: "Tsiku lina RAID idzakhala chinthu chakale, kusungirako zinthu kudzadzaza dziko lapansi, ndipo simudziwa kuti CEPH ndi chiyani," kotero chinthu choyamba m'moyo wanga wodziimira. ndinali kupanga gulu langa. Cholinga cha kuyesaku chinali kudziwa momwe ceph imapangidwira ndikumvetsetsa kukula kwake. Ndikoyenera bwanji kukhazikitsidwa kwa ceph m'mabizinesi apakati komanso ang'onoang'ono? Pambuyo pazaka zingapo zogwira ntchito komanso kutayika kwa data zingapo kosasinthika, kumvetsetsa kwazovuta zidawuka kuti sikuti zonse ndizosavuta. Zodziwika bwino za CEPH zimabweretsa zolepheretsa kutengedwa kwake kofala, ndipo chifukwa cha izi, kuyesa kwafika pachimake. M'munsimu muli kufotokozera njira zonse zomwe zatengedwa, zotsatira zomwe zapezedwa ndi zomaliza. Ngati anthu odziwa bwino agawana zomwe akumana nazo ndikufotokozera mfundo zina, ndidzakhala woyamikira.
Zindikirani: Opereka ndemanga apeza zolakwika zazikulu m'malingaliro ena omwe amafunikira kuwunikiranso nkhani yonse.
CEPH Strategy
Gulu la CEPH limaphatikiza nambala yokhazikika K ya ma disks a kukula kosasinthasintha ndikusunga deta pa iwo, kubwereza chidutswa chilichonse (4 MB mwachisawawa) nambala yoperekedwa nthawi ya N.
Tiyeni tikambirane nkhani yosavuta yokhala ndi ma disks awiri ofanana. Kuchokera kwa iwo mutha kusonkhanitsa RAID 1 kapena gulu ndi N = 2 - zotsatira zake zidzakhala zofanana. Ngati pali ma disks atatu ndipo ali ndi kukula kosiyana, ndiye kuti n'zosavuta kusonkhanitsa gulu ndi N = 2: zina mwazo zidzakhala pa disks 1 ndi 2, zina zidzakhala pa disk 1 ndi 3, ndipo zina zidzakhala pa 2 ndi 3, pamene RAID sichidzatero (mukhoza kusonkhanitsa RAID yotere, koma kungakhale kupotoza). Ngati pali ma disks ochulukirapo, ndiye kuti n'zotheka kupanga RAID 5; CEPH ili ndi analogue - erasure_code, yomwe imatsutsana ndi malingaliro oyambirira a omanga, choncho sichiganiziridwa. RAID 5 imaganiza kuti pali ma drive angapo, onse omwe ali bwino. Ngati wina alephera, enawo ayenera kudikirira mpaka diski itasinthidwa ndikubwezeretsanso deta. CEPH, ndi N> = 3, imalimbikitsa kugwiritsa ntchito ma disks akale, makamaka, ngati musunga ma disks angapo abwino kusunga kopi imodzi ya deta, ndikusunga makope awiri kapena atatu otsala pa ma disks akale ambiri, ndiye chidziwitso. adzakhala otetezeka, popeza pakali pano ma disks atsopano ali ndi moyo - palibe mavuto, ndipo ngati mmodzi wa iwo athyoka, ndiye kuti kulephera panthawi imodzi ya disks atatu ndi moyo wautumiki wa zaka zoposa zisanu, makamaka kuchokera ku ma seva osiyanasiyana, ndizokayikitsa kwambiri. chochitika.
Pali chinyengo pa kugawa makope. Mwachikhazikitso, zimaganiziridwa kuti deta imagawidwa m'magulu ambiri (~ 100 pa disk) magulu ogawa a PG, omwe amapangidwanso pama disks ena. Tinene kuti K = 6, N = 2, ndiye ngati ma disks awiri alephera, deta imatsimikizika kuti itayika, chifukwa malinga ndi chiphunzitso chotheka, padzakhala PG imodzi yomwe idzakhala pa disks ziwirizi. Ndipo kutayika kwa gulu limodzi kumapangitsa kuti deta yonse mu dziwe isapezeke. Ngati ma disks agawika mawiri atatu ndipo deta imaloledwa kusungidwa pa disks mkati mwa gulu limodzi, ndiye kuti kugawa koteroko kumatsutsana ndi kulephera kwa disk imodzi, koma ngati ma disks awiri alephera, kuthekera kwa kutayika kwa deta sikungatheke. 100%, koma 3/15 okha, ndipo ngakhale kulephera atatu disks - 12/20 okha. Chifukwa chake, entropy pakugawa kwa data sikuthandizira kulekerera zolakwika. Komanso dziwani kuti pa seva yamafayilo, RAM yaulere imakulitsa kwambiri liwiro loyankha. Kukumbukira kochulukira mu node iliyonse, komanso kukumbukira kwambiri m'malo onse, kudzakhala mwachangu. Izi mosakayikira ndi mwayi wamagulu pa seva imodzi ndipo, makamaka, hardware NAS, kumene kukumbukira kochepa kwambiri kumapangidwira.
Izi zikutsatira kuti CEPH ndi njira yabwino yopangira njira yodalirika yosungiramo deta ya makumi khumi a TB ndi kuthekera kokulirapo ndi ndalama zochepa kuchokera ku zipangizo zakale (pano, ndithudi, ndalama zidzafunika, koma zazing'ono poyerekeza ndi machitidwe osungirako malonda).
Kukhazikitsa magulu
Pakuyesa, tiyeni titenge kompyuta yochotsedwa ya Intel DQ57TM + Intel core i3 540 + 16 GB ya RAM. Tidzakonza ma disks anayi a 2 TB kukhala chinthu chonga RAID10, titatha kuyesa bwino tidzawonjezera node yachiwiri ndi ma disks omwewo.
Kukhazikitsa Linux. Kugawa kumafuna luso lokonzekera ndi kukhala okhazikika. Debian ndi Suse amakwaniritsa zofunikira. Suse ili ndi choyikira chosinthika chomwe chimakupatsani mwayi woletsa phukusi lililonse; Tsoka ilo, sindinathe kudziwa kuti ndi ati omwe angatayidwe popanda kuwononga dongosolo. Ikani Debian pogwiritsa ntchito debootstrap buster. Njira ya min-base imayika makina osweka omwe alibe madalaivala. Kusiyana kwa kukula poyerekeza ndi mtundu wonse sikuli kwakukulu kotero kuti kuvutitsa. Popeza ntchito ikuchitika pa makina thupi, Ndikufuna kutenga zithunzi, monga makina pafupifupi. Izi zimaperekedwa ndi LVM kapena btrfs (kapena xfs, kapena zfs - kusiyana kwake sikwakukulu). Zithunzi za LVM si mfundo yamphamvu. Ikani btrfs. Ndipo bootloader ili mu MBR. Palibe chifukwa chophatikiza diski ya 50 MB yokhala ndi gawo la FAT pomwe mutha kukankhira mugawo la tebulo la 1 MB ndikugawa malo onse adongosolo. Anatenga 700 MB pa disk. Sindikukumbukira kuti kukhazikitsa kwa SUSE kuli ndi ndalama zingati, ndikuganiza kuti ndi za 1.1 kapena 1.4 GB.
Ikani CEPH. Timanyalanyaza mtundu wa 12 m'malo osungiramo debian ndikulumikizana mwachindunji kuchokera patsamba la 15.2.3. Timatsatira malangizo a gawo la "Ikani CEPH pamanja" ndi chenjezo lotsatirali:
- Musanalumikize chosungira, muyenera kukhazikitsa gnupg wget ca-satifiketi
- Pambuyo polumikiza chosungira, koma musanayike gululo, kuyika ma phukusi sikunasiyidwe: apt -y --no-install-recommends install ceph-common ceph-mon ceph-osd ceph-mds ceph-mgr
- Mukayika CEPH, pazifukwa zosadziwika, idzayesa kukhazikitsa lvm2. M'malo mwake, sizomvetsa chisoni, koma kukhazikitsa kumalephera, kotero CEPH siiyikanso.
Chigawo ichi chathandiza:
cat << EOF >> /var/lib/dpkg/status Package: lvm2 Status: install ok installed Priority: important Section: admin Installed-Size: 0 Maintainer: Debian Adduser Developers <[email protected]> Architecture: all Multi-Arch: foreign Version: 113.118 Description: No-install EOF
Chidule cha Cluster
ceph-osd - imayang'anira kusunga deta pa disk. Pa disk iliyonse, ntchito ya netiweki imayambitsidwa yomwe imavomereza ndikuchita zopempha kuti muwerenge kapena kulembera zinthu. Magawo awiri amapangidwa pa disk. Chimodzi mwa izo chili ndi chidziwitso cha cluster, disk nambala, ndi makiyi a gululo. Chidziwitso cha 1KB ichi chimapangidwa kamodzi powonjezera disk ndipo sichinawonekere kuti chikusintha. Gawo lachiwiri lilibe dongosolo la mafayilo ndipo limasunga data ya binary ya CEPH. Kuyika zokha m'matembenuzidwe am'mbuyomu kudapanga gawo la 100MB xfs kuti mudziwe zambiri zantchito. Ndinatembenuza disk kukhala MBR ndikugawa 16MB yokha - ntchitoyo sichidandaula. Ndikuganiza kuti ma xf atha kusinthidwa ndi ext popanda vuto lililonse. Gawoli limayikidwa mu / var/lib/β¦, pomwe ntchitoyo imawerenga zambiri za OSD komanso imapeza zolozera ku chipangizo chotchinga komwe data ya binary imasungidwa. Mwachidziwitso, mutha kuyika mafayilo othandizira nthawi yomweyo /var/lib/β¦, ndikugawa disk yonse kuti mupeze deta. Mukapanga OSD kudzera pa ceph-deploy, lamulo limapangidwa kuti likhazikitse magawo / var/lib/β¦, ndipo wogwiritsa ntchito ceph amapatsidwanso ufulu wowerengera chida chomwe mukufuna. Ngati muyika pamanja, muyenera kuchita izi nokha; zolembedwa sizikunena izi. Ndikofunikiranso kufotokozera osd memory target parameter kuti pakhale kukumbukira kwakuthupi kokwanira.
ceph-mds. Pa mlingo wotsika, CEPH ndi kusungirako zinthu. Kuthekera kosungirako block kumagwera mpaka kusunga chipika chilichonse cha 4MB ngati chinthu. Kusungirako mafayilo kumagwira ntchito mofananamo. Maiwe awiri amapangidwa: imodzi ya metadata, ina ya data. Iwo ali pamodzi wapamwamba dongosolo. Panthawiyi, zolemba zamtundu wina zimapangidwira, kotero ngati muchotsa mafayilo a fayilo, koma sungani maiwe onse awiri, simungathe kubwezeretsa. Pali njira yochotsera mafayilo ndi midadada, sindinayese. Utumiki wa ceph-mds uli ndi udindo wopezera mafayilo. Dongosolo lililonse la mafayilo limafunikira mtundu wina wautumiki. Pali njira ya "index", yomwe imakupatsani mwayi wopanga mawonekedwe amitundu ingapo yamafayilo amodzi - osayesedwa.
Ceph-mon - Ntchitoyi imasunga mapu a masango. Zimaphatikizanso zambiri zama OSD onse, algorithm yogawa ma PG mu OSDs ndipo, chofunikira kwambiri, chidziwitso chazinthu zonse (zambiri zamakinawa sizikudziwika kwa ine: pali chikwatu /var/lib/ceph/mon/β¦/ store.db, ili ndi fayilo yaikulu ndi 26MB, ndipo mumagulu a zinthu za 105K, zimakhala zopitirira pang'ono 256 byte pa chinthu chilichonse - ndikuganiza kuti polojekitiyi imasunga mndandanda wa zinthu zonse ndi PG yomwe iwo ali). Kuwonongeka kwa bukhuli kumabweretsa kutayika kwa deta yonse mumagulu. Chifukwa chake pamapeto pake adapeza kuti CRUSH ikuwonetsa momwe ma PG ali pa OSD, komanso momwe zinthu ziliri pa PGs - zimasungidwa mkati mwa nkhokwe, ngakhale opanga amapewa mawu awa. Chotsatira chake, choyamba, sitingathe kuyika dongosolo pa flash drive mu RO mode, popeza deta imalembedwa nthawi zonse, disk yowonjezera ikufunika kwa izi (zosaposa 1 GB), kachiwiri, ndikofunika kukhala ndi koperani mu nthawi yeniyeni maziko awa. Ngati pali oyang'anira angapo, ndiye kuti kulolerana kwa zolakwika kumatsimikiziridwa kokha, koma kwa ife pali polojekiti imodzi yokha, yopambana ziwiri. Pali njira yowerengera yobwezeretsanso chowunikira kutengera data ya OSD, ndidagwiritsa ntchito katatu pazifukwa zosiyanasiyana, ndipo katatu panalibe mauthenga olakwika, komanso palibe deta. Tsoka ilo, makinawa sagwira ntchito. Mwina timagwiritsa ntchito magawo ang'onoang'ono pa OSD ndikusonkhanitsa RAID kuti tisunge nkhokwe, zomwe zingakhale ndi zotsatira zoipa kwambiri pa ntchito, kapena timagawira zosachepera ziwiri zodalirika, makamaka USB, kuti tisatengere madoko.
rados-gw - imatumiza kusungirako zinthu kudzera pa protocol ya S3 ndi zina zofananira. Amapanga maiwe ambiri, sizikudziwika chifukwa chake. Sindinayese zambiri.
ceph-mgr - Mukakhazikitsa ntchitoyi, ma module angapo amayambitsidwa. Chimodzi mwa izo ndi autoscale yomwe singayimitsidwe. Imayesetsa kusunga kuchuluka koyenera kwa PG/OSD. Ngati mukufuna kulamulira chiΕ΅erengero pamanja, mukhoza kuletsa makulitsidwe pa dziwe lililonse, koma mu nkhani iyi gawo inagwa ndi magawano ndi 0, ndi masango udindo kukhala ERROR. Gawoli lidalembedwa mu Python, ndipo ngati mupereka ndemanga pamzere wofunikira momwemo, izi zimapangitsa kuti aletsedwe. Waulesi kukumbukira zambiri.
Mndandanda wazomwe zagwiritsidwa ntchito:
Zolemba:
Kuyika dongosolo kudzera pa debootstrap
blkdev=sdb1
mkfs.btrfs -f /dev/$blkdev
mount /dev/$blkdev /mnt
cd /mnt
for i in {@,@var,@home}; do btrfs subvolume create $i; done
mkdir snapshot @/{var,home}
for i in {var,home}; do mount -o bind @${i} @/$i; done
debootstrap buster @ http://deb.debian.org/debian; echo $?
for i in {dev,proc,sys}; do mount -o bind /$i @/$i; done
cp /etc/bash.bashrc @/etc/
chroot /mnt/@ /bin/bash
echo rbd1 > /etc/hostname
passwd
uuid=`blkid | grep $blkdev | cut -d """ -f 2`
cat << EOF > /etc/fstab
UUID=$uuid / btrfs noatime,nodiratime,subvol=@ 0 1
UUID=$uuid /var btrfs noatime,nodiratime,subvol=@var 0 2
UUID=$uuid /home btrfs noatime,nodiratime,subvol=@home 0 2
EOF
cat << EOF >> /var/lib/dpkg/status
Package: lvm2
Status: install ok installed
Priority: important
Section: admin
Installed-Size: 0
Maintainer: Debian Adduser Developers <[email protected]>
Architecture: all
Multi-Arch: foreign
Version: 113.118
Description: No-install
Package: sudo
Status: install ok installed
Priority: important
Section: admin
Installed-Size: 0
Maintainer: Debian Adduser Developers <[email protected]>
Architecture: all
Multi-Arch: foreign
Version: 113.118
Description: No-install
EOF
exit
grub-install --boot-directory=@/boot/ /dev/$blkdev
init 6
apt -yq install --no-install-recommends linux-image-amd64 bash-completion ed btrfs-progs grub-pc iproute2 ssh smartmontools ntfs-3g net-tools man
exit
grub-install --boot-directory=@/boot/ /dev/$blkdev
init 6Pangani gulu
apt -yq install --no-install-recommends gnupg wget ca-certificates
echo 'deb https://download.ceph.com/debian-octopus/ buster main' >> /etc/apt/sources.list
wget -q -O- 'https://download.ceph.com/keys/release.asc' | apt-key add -
apt update
apt -yq install --no-install-recommends ceph-common ceph-mon
echo 192.168.11.11 rbd1 >> /etc/hosts
uuid=`cat /proc/sys/kernel/random/uuid`
cat << EOF > /etc/ceph/ceph.conf
[global]
fsid = $uuid
auth cluster required = cephx
auth service required = cephx
auth client required = cephx
mon allow pool delete = true
mon host = 192.168.11.11
mon initial members = rbd1
mon max pg per osd = 385
osd crush update on start = false
#osd memory target = 2147483648
osd memory target = 1610612736
osd scrub chunk min = 1
osd scrub chunk max = 2
osd scrub sleep = .2
osd pool default pg autoscale mode = off
osd pool default size = 1
osd pool default min size = 1
osd pool default pg num = 1
osd pool default pgp num = 1
[mon]
mgr initial modules = dashboard
EOF
ceph-authtool --create-keyring ceph.mon.keyring --gen-key -n mon. --cap mon 'allow *'
ceph-authtool --create-keyring ceph.client.admin.keyring --gen-key -n client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *'
cp ceph.client.admin.keyring /etc/ceph/
ceph-authtool --create-keyring bootstrap-osd.ceph.keyring --gen-key -n client.bootstrap-osd --cap mon 'profile bootstrap-osd' --cap mgr 'allow r'
cp bootstrap-osd.ceph.keyring /var/lib/ceph/bootstrap-osd/ceph.keyring
ceph-authtool ceph.mon.keyring --import-keyring /etc/ceph/ceph.client.admin.keyring
ceph-authtool ceph.mon.keyring --import-keyring /var/lib/ceph/bootstrap-osd/ceph.keyring
monmaptool --create --add rbd1 192.168.11.11 --fsid $uuid monmap
rm -R /var/lib/ceph/mon/ceph-rbd1/*
ceph-mon --mkfs -i rbd1 --monmap monmap --keyring ceph.mon.keyring
chown ceph:ceph -R /var/lib/ceph
systemctl enable ceph-mon@rbd1
systemctl start ceph-mon@rbd1
ceph mon enable-msgr2
ceph status
# dashboard
apt -yq install --no-install-recommends ceph-mgr ceph-mgr-dashboard python3-distutils python3-yaml
mkdir /var/lib/ceph/mgr/ceph-rbd1
ceph auth get-or-create mgr.rbd1 mon 'allow profile mgr' osd 'allow *' mds 'allow *' > /var/lib/ceph/mgr/ceph-rbd1/keyring
systemctl enable ceph-mgr@rbd1
systemctl start ceph-mgr@rbd1
ceph config set mgr mgr/dashboard/ssl false
ceph config set mgr mgr/dashboard/server_port 7000
ceph dashboard ac-user-create root 1111115 administrator
systemctl stop ceph-mgr@rbd1
systemctl start ceph-mgr@rbd1Kuwonjezera OSD (gawo)
apt install ceph-osd
osdnum=`ceph osd create`
mkdir -p /var/lib/ceph/osd/ceph-$osdnum
mkfs -t xfs /dev/sda1
mount -t xfs /dev/sda1 /var/lib/ceph/osd/ceph-$osdnum
cd /var/lib/ceph/osd/ceph-$osdnum
ceph auth get-or-create osd.0 mon 'profile osd' mgr 'profile osd' osd 'allow *' > /var/lib/ceph/osd/ceph-$osdnum/keyring
ln -s /dev/disk/by-partuuid/d8cc3da6-02 block
ceph-osd -i $osdnum --mkfs
#chown ceph:ceph /dev/sd?2
chown ceph:ceph -R /var/lib/ceph
systemctl enable ceph-osd@$osdnum
systemctl start ceph-osd@$osdnumChidule
Ubwino waukulu wamalonda wa CEPH ndi CRUSH - ndondomeko yowerengera malo a deta. Owunika amagawira algorithm iyi kwa makasitomala, pambuyo pake makasitomala amapempha mwachindunji node yomwe akufuna ndi OSD yomwe akufuna. CRUSH imatsimikizira kuti palibe centralization. Ndi fayilo yaing'ono yomwe mungathe kusindikiza ndikupachika pakhoma. Zoyeserera zawonetsa kuti CRUSH si mapu otopetsa. Ngati muwononga ndikubwezeretsanso oyang'anira, kusunga OSD yonse ndi CRUSH, ndiye kuti izi sizokwanira kubwezeretsa masango. Kuchokera apa zimaganiziridwa kuti polojekiti iliyonse imasunga metadata ya gulu lonse. Kuchepa kwa metadata iyi sikuika malire pa kukula kwa masango, koma kumafuna kuonetsetsa chitetezo chawo, chomwe chimathetsa kusungirako kwa disk mwa kukhazikitsa dongosolo pa flash drive ndikupatula masango okhala ndi mfundo zosachepera zitatu. Mfundo zankhanza za oyambitsa zokhudzana ndi zomwe mwasankha. Kutali ndi minimalism. Zolembazo zili pamlingo wa "zikomo pazomwe tili nazo, koma ndizochepa kwambiri." Kutha kuyanjana ndi mautumiki pamlingo wochepa kumaperekedwa, koma zolembedwa zimakhudza mutuwu mozama kwambiri, kotero ndizotheka kuti ayi kuposa inde. Pali pafupifupi palibe mwayi achire deta kuchokera mwadzidzidzi.
Zosankha kuti muchiteponso: siyani CEPH ndikugwiritsa ntchito banal multi-disk btrfs (kapena xfs, zfs), fufuzani zatsopano za CEPH, zomwe zingakuthandizeni kuti muyigwiritse ntchito malinga ndi zomwe zanenedwa, yesani kulemba zosungira zanu ngati zapamwamba. maphunziro.
Source: www.habr.com