Thaum muaj cov ntaub ntawv ntau dua tuaj yeem haum rau ntawm ib lub disk, nws yog lub sijhawm los xav txog RAID. Raws li ib tug me nyuam, kuv tau hnov los ntawm kuv cov txwj laus: "Muaj ib hnub RAID yuav yog yav dhau los, cov khoom khaws cia yuav puv lub ntiaj teb, thiab koj tseem tsis paub tias CEPH yog dab tsi," yog li thawj yam hauv kuv lub neej ywj pheej yog los tsim kuv tus kheej pawg. Lub hom phiaj ntawm qhov kev sim yog kom paub txog cov qauv sab hauv ntawm ceph thiab nkag siab txog cov peev txheej ntawm nws daim ntawv thov. Yuav ua li cas ncaj ncees yog qhov kev siv ntawm ceph hauv cov lag luam nruab nrab thiab cov me me? Tom qab ob peb xyoos ntawm kev ua haujlwm thiab ob peb cov ntaub ntawv tsis tuaj yeem thim rov qab, kev nkag siab ntawm qhov tsis sib xws tau tshwm sim uas tsis yog txhua yam yooj yim heev. Qhov peculiarities ntawm CEPH ua rau muaj kev cuam tshuam rau nws txoj kev saws me nyuam, thiab vim yog lawv, kev sim tau mus txog qhov kawg. Hauv qab no yog cov lus piav qhia ntawm tag nrho cov kauj ruam ua, qhov tshwm sim tau thiab cov lus xaus tau kos. Yog tias cov neeg txawj ntse qhia lawv qhov kev paub thiab piav qhia qee cov ntsiab lus, kuv yuav ua tsaug.
Nco tseg: Cov neeg tawm tswv yim tau txheeb xyuas qhov yuam kev loj hauv qee qhov kev xav uas yuav tsum tau hloov kho tag nrho cov lus.
CEPH Strategy
CEPH pawg sib txuas ua ke tus lej K ntawm cov disks ntawm qhov loj me thiab khaws cov ntaub ntawv ntawm lawv, sib npaug txhua daim (4 MB los ntawm lub neej ntawd) tus lej N zaug.
Cia peb xav txog qhov yooj yim tshaj plaws nrog ob lub disks zoo ib yam. Los ntawm lawv koj tuaj yeem sib sau RAID 1 lossis pawg nrog N = 2 - qhov tshwm sim yuav zoo ib yam. Yog tias muaj peb lub disks thiab lawv muaj ntau qhov sib txawv, ces nws yooj yim rau kev sib sau ua ke nrog N = 2: qee cov ntaub ntawv yuav nyob ntawm disks 1 thiab 2, qee qhov yuav nyob ntawm disks 1 thiab 3, thiab qee qhov yuav yog. ntawm 2 thiab 3, thaum RAID yuav tsis (koj tuaj yeem sib sau ua ke xws li RAID, tab sis nws yuav yog qhov tsis zoo). Yog tias muaj ntau dua disks, ces nws muaj peev xwm tsim RAID 5; CEPH muaj ib qho analogue - erasure_code, uas cuam tshuam cov ntsiab lus thaum ntxov ntawm cov neeg tsim khoom, thiab yog li ntawd tsis suav nrog. RAID 5 xav tias muaj qee qhov me me ntawm cov tsav, tag nrho cov uas nyob rau hauv zoo. Yog tias ib qho ua tsis tiav, lwm tus yuav tsum tuav kom txog thaum lub disk hloov thiab cov ntaub ntawv rov qab los rau nws. CEPH, nrog N> = 3, txhawb kev siv cov qub disks, tshwj xeeb tshaj yog, yog tias koj khaws ob peb lub disks zoo khaws ib daim ntawv theej ntawm cov ntaub ntawv, thiab khaws ob lossis peb daim ntawv theej ntxiv rau ntau cov qub disks, ces cov ntaub ntawv. yuav muaj kev nyab xeeb, txij li tam sim no cov disks tshiab tseem muaj sia nyob - tsis muaj teeb meem, thiab yog tias ib qho ntawm lawv tawg, tom qab ntawd qhov tsis ua haujlwm ib txhij ntawm peb lub disk nrog lub neej kev pab cuam ntau dua tsib xyoos, nyiam dua los ntawm cov servers sib txawv, yog qhov tsis zoo. xwm txheej.
Nws muaj ib tug subtlety rau kev faib cov ntawv luam. Los ntawm lub neej ntawd, nws tau xav tias cov ntaub ntawv tau muab faib ua ntau dua (~ 100 ib lub disk) PG faib pawg, txhua tus uas tau muab luam tawm ntawm qee qhov disks. Wb hais tias K = 6, N = 2, ces yog tias ob lub disks tsis ua haujlwm, cov ntaub ntawv tau lees tias yuav ploj, vim tias raws li kev xav tau, yuav muaj tsawg kawg ib PG uas yuav nyob ntawm ob lub disks no. Thiab qhov poob ntawm ib pab pawg ua rau tag nrho cov ntaub ntawv hauv lub pas dej tsis muaj. Yog hais tias cov disks tau muab faib ua peb khub thiab cov ntaub ntawv raug tso cai khaws cia hauv disks hauv ib khub, ces qhov kev faib tawm no kuj tiv taus qhov tsis ua haujlwm ntawm ib qho disk, tab sis yog tias ob lub disks tsis ua haujlwm, qhov tshwm sim ntawm cov ntaub ntawv poob tsis yog. 100%, tab sis tsuas yog 3/15, thiab txawm nyob rau hauv cov ntaub ntawv ntawm tsis ua hauj lwm peb disks - tsuas yog 12/20. Yog li ntawd, entropy nyob rau hauv cov ntaub ntawv faib tsis ua rau muaj kev zam txim. Tsis tas li ntawd nco ntsoov tias rau cov neeg rau zaub mov cov ntaub ntawv, RAM dawb ua rau kom cov lus teb ceev. Lub cim xeeb ntau dua hauv txhua lub node, thiab ntau lub cim xeeb hauv txhua lub nodes, nws yuav nrawm dua. Qhov no yog undoubtedly ib tug zoo dua ntawm ib pawg nyob rau hauv ib tug tib neeg rau zaub mov thiab, txawm ntau li ntawd, ib tug hardware NAS, qhov twg ib tug me me npaum li cas ntawm lub cim xeeb nyob rau hauv.
Nws ua raws li CEPH yog ib txoj hauv kev zoo los tsim cov ntaub ntawv khaws cia ruaj khov rau kaum tawm TB nrog lub peev xwm los ntsuas nrog kev nqis peev tsawg los ntawm cov cuab yeej siv tsis tu ncua (ntawm no, tau kawg, cov nqi yuav xav tau, tab sis me me piv rau kev lag luam cia khoom).
Kev siv pawg
Rau qhov kev sim, cia peb siv lub khoos phis tawj uas tsis muaj Intel DQ57TM + Intel core i3 540 + 16 GB ntawm RAM. Peb yuav npaj plaub 2 TB disks rau hauv ib yam dab tsi zoo li RAID10, tom qab kev sim ua tiav peb yuav ntxiv ib qho thib ob thiab tib lub disks.
Txhim kho Linux. Kev faib tawm yuav tsum muaj peev xwm kho tau thiab ruaj khov. Debian thiab Suse ua tau raws li qhov yuav tsum tau ua. Suse muaj qhov hloov pauv hloov tau yooj yim dua uas tso cai rau koj los lov tes taw txhua pob; Hmoov tsis zoo, kuv tsis tuaj yeem txiav txim siab seb qhov twg tuaj yeem raug pov tseg yam tsis muaj kev puas tsuaj rau lub system. Nruab Debian siv debootstrap buster. Qhov kev xaiv min-base teeb tsa lub kaw lus tawg uas tsis muaj cov tsav tsheb. Qhov sib txawv ntawm qhov loj me piv rau tag nrho cov version tsis loj li thab. Txij li thaum ua haujlwm yog nqa tawm ntawm lub tshuab lub cev, kuv xav ua snapshots, zoo li ntawm cov tshuab virtual. Qhov kev xaiv no yog muab los ntawm LVM lossis btrfs (lossis xfs, lossis zfs - qhov txawv tsis loj). LVM snapshots tsis yog qhov tseem ceeb. Nruab btrfs. Thiab lub bootloader yog nyob rau hauv MBR. Tsis muaj qhov taw tes rau hauv cluttering 50 MB disk nrog FAT muab faib thaum koj tuaj yeem thawb nws mus rau hauv 1 MB muab faib lub rooj cheeb tsam thiab faib txhua qhov chaw rau lub kaw lus. Tau txais 700 MB ntawm disk. Kuv tsis nco qab ntau npaum li cas qhov yooj yim SUSE installation muaj, kuv xav tias nws yog 1.1 lossis 1.4 GB.
Nruab CEPH. Peb tsis quav ntsej version 12 nyob rau hauv lub debian repository thiab txuas ncaj qha los ntawm 15.2.3 site. Peb ua raws li cov lus qhia los ntawm ntu "Nruab CEPH manually" nrog cov lus ceeb toom hauv qab no:
- Ua ntej txuas lub repository, koj yuav tsum nruab gnupg wget ca-certificates
- Tom qab txuas lub repository, tab sis ua ntej txhim kho pawg, txhim kho cov pob khoom raug tshem tawm: apt -y --no-install-pom zoo rau nruab ceph-mon ceph-mon ceph-osd ceph-mds ceph-mgr
- Thaum txhim kho CEPH, tsis paub yog vim li cas, nws yuav sim rau nruab lvm2. Raws li txoj cai, nws tsis yog kev hlub, tab sis kev teeb tsa ua tsis tiav, yog li CEPH yuav tsis nruab ib qho.
Qhov no thaj pab tau:
cat << EOF >> /var/lib/dpkg/status Package: lvm2 Status: install ok installed Priority: important Section: admin Installed-Size: 0 Maintainer: Debian Adduser Developers <[email protected]> Architecture: all Multi-Arch: foreign Version: 113.118 Description: No-install EOF
Pawg saib xyuas
ceph-osd - yog lub luag haujlwm rau khaws cov ntaub ntawv ntawm disk. Rau txhua lub disk, ib qho kev pabcuam network tau pib uas lees txais thiab ua cov ntawv thov nyeem lossis sau rau cov khoom. Ob lub partitions yog tsim rau ntawm lub disk. Ib ntawm lawv muaj cov ntaub ntawv hais txog pawg, tus lej disk, thiab cov yuam sij rau pawg. Cov ntaub ntawv 1KB no yog tsim ib zaug thaum ntxiv cov disk thiab tsis tau pom tias hloov pauv. Qhov thib ob muab faib tsis muaj cov ntaub ntawv kaw lus thiab khaws CEPH binary cov ntaub ntawv. Kev teeb tsa tsis siv neeg hauv cov qauv dhau los tsim 100MB xfs muab faib rau cov ntaub ntawv pabcuam. Kuv hloov lub disk rau MBR thiab faib tsuas yog 16MB - qhov kev pabcuam tsis yws. Kuv xav tias xfs tuaj yeem hloov nrog ext yam tsis muaj teeb meem. Qhov kev faib tawm no tau nruab rau hauv /var/lib/…, qhov twg cov kev pabcuam nyeem cov ntaub ntawv hais txog OSD thiab tseem pom cov ntaub ntawv siv rau cov khoom siv thaiv qhov twg cov ntaub ntawv binary khaws cia. Raws li txoj cai, koj tuaj yeem tso cov ntaub ntawv pabcuam tam sim hauv /var/lib/…, thiab faib tag nrho disk rau cov ntaub ntawv. Thaum tsim OSD ntawm ceph-deploy, ib txoj cai raug tsim los txhim kho qhov muab faib hauv /var/lib/…, thiab tus neeg siv ceph kuj tau muab txoj cai los nyeem cov khoom thaiv uas xav tau. Yog tias koj nruab manually, koj yuav tsum ua qhov no koj tus kheej; cov ntaub ntawv tsis hais qhov no. Nws kuj tseem pom zoo kom qhia meej osd lub cim xeeb lub hom phiaj kom muaj lub cim xeeb txaus.
xab mds. Nyob rau theem qis, CEPH yog qhov khoom khaws cia. Lub peev xwm los thaiv kev khaws cia los khaws cia txhua 4MB thaiv ua ib yam khoom. Cov ntaub ntawv khaws cia ua haujlwm ntawm tib txoj cai. Ob lub pas dej tau tsim: ib qho rau metadata, lwm qhov rau cov ntaub ntawv. Lawv muab tso ua ke rau hauv cov ntaub ntawv system. Lub sijhawm no, qee cov ntaub ntawv raug tsim, yog li yog tias koj rho tawm cov ntaub ntawv kaw lus, tab sis khaws ob lub pas dej, koj yuav tsis tuaj yeem rov qab tau. Muaj cov txheej txheem rau rho tawm cov ntaub ntawv los ntawm cov blocks, Kuv tsis tau sim nws. Qhov kev pabcuam ceph-mds yog lub luag haujlwm rau kev nkag mus rau cov ntaub ntawv kaw lus. Txhua cov ntaub ntawv kaw lus xav tau ib qho piv txwv ntawm kev pabcuam. Muaj qhov "index" kev xaiv, uas tso cai rau koj los tsim qhov zoo ib yam ntawm ntau cov ntaub ntawv hauv ib qho - kuj tsis tau sim.
Ceph-mon - Qhov kev pabcuam no khaws ib daim ntawv qhia ntawm pawg. Nws suav nrog cov ntaub ntawv hais txog txhua OSDs, cov txheej txheem rau kev faib PGs hauv OSDs thiab, qhov tseem ceeb tshaj plaws, cov ntaub ntawv hais txog txhua yam khoom (cov ntsiab lus ntawm cov txheej txheem no tsis meej rau kuv: muaj cov npe /var/lib/ceph/mon/…/ store.db, nws muaj ib tug loj cov ntaub ntawv yog 26MB, thiab nyob rau hauv ib pawg ntawm 105K cov khoom, nws hloov tawm mus rau ib tug me ntsis tshaj 256 bytes ib yam khoom - Kuv xav tias lub monitor khaws cov npe ntawm tag nrho cov khoom thiab cov PGs nyob rau hauv uas. lawv nyob). Kev puas tsuaj rau daim ntawv teev npe no ua rau poob ntawm tag nrho cov ntaub ntawv hauv pawg. Yog li cov lus xaus tau kos tias CRUSH qhia tias PGs nyob hauv OSD li cas, thiab cov khoom nyob rau ntawm PGs li cas - lawv tau khaws cia hauv nruab nrab hauv cov ntaub ntawv, txawm tias ntau npaum li cas cov neeg tsim tawm zam lo lus no. Raws li qhov tshwm sim, thawj zaug, peb tsis tuaj yeem nruab lub kaw lus ntawm lub flash drive hauv RO hom, txij li cov ntaub ntawv tau raug kaw tas li, xav tau ib qho disk ntxiv rau cov no (kwv yees ntau dua 1 GB), qhov thib ob, nws yog ib qho tsim nyog yuav tsum muaj lub disk ntxiv. luam hauv lub sijhawm tiag tiag lub hauv paus no. Yog tias muaj ob peb lub monitor, ces qhov kev zam txim tau ua kom tiav, tab sis nyob rau hauv peb cov ntaub ntawv tsuas muaj ib tug saib, siab tshaj ob. Muaj cov txheej txheem theoretical rau kev rov qab saib xyuas raws li OSD cov ntaub ntawv, Kuv tau mus rau nws peb zaug rau ntau yam, thiab peb zaug tsis muaj lus yuam kev, nrog rau tsis muaj ntaub ntawv. Hmoov tsis zoo, qhov no mechanism tsis ua haujlwm. Txawm hais tias peb ua haujlwm me me ntawm OSD thiab sib sau ua ke RAID los khaws cov ntaub ntawv, uas yeej yuav muaj kev cuam tshuam tsis zoo rau kev ua haujlwm, lossis peb faib tsawg kawg yog ob qhov kev tshaj tawm lub cev, nyiam dua USB, kom tsis txhob muaj cov chaw nres nkoj.
rados-gw - xa cov khoom cia ntawm S3 raws tu qauv thiab zoo sib xws. Tsim ntau lub pas dej, tsis paub meej vim li cas. Kuv tsis tau sim ntau.
ceph-mgr - Thaum txhim kho qhov kev pabcuam no, ntau lub modules tau pib. Ib ntawm lawv yog autoscale uas tsis tuaj yeem ua tsis taus. Nws sib zog ua kom muaj qhov tseeb ntawm PG/OSD. Yog tias koj xav tswj qhov piv txwv ntawm tus kheej, koj tuaj yeem lov tes taw qhov ntsuas rau txhua lub pas dej, tab sis qhov no lub module tsoo nrog kev faib los ntawm 0, thiab cov xwm txheej hauv pawg ua yuam kev. Lub module tau sau hauv Python, thiab yog tias koj tawm tswv yim tawm cov kab tsim nyog hauv nws, qhov no ua rau nws tsis ua haujlwm. tub nkees heev nco ntsoov cov ntsiab lus.
Daim ntawv teev cov peev txheej siv:
Cov npe ntawm cov ntawv sau:
Txhim kho qhov system ntawm debootstrap
blkdev=sdb1
mkfs.btrfs -f /dev/$blkdev
mount /dev/$blkdev /mnt
cd /mnt
for i in {@,@var,@home}; do btrfs subvolume create $i; done
mkdir snapshot @/{var,home}
for i in {var,home}; do mount -o bind @${i} @/$i; done
debootstrap buster @ http://deb.debian.org/debian; echo $?
for i in {dev,proc,sys}; do mount -o bind /$i @/$i; done
cp /etc/bash.bashrc @/etc/
chroot /mnt/@ /bin/bash
echo rbd1 > /etc/hostname
passwd
uuid=`blkid | grep $blkdev | cut -d """ -f 2`
cat << EOF > /etc/fstab
UUID=$uuid / btrfs noatime,nodiratime,subvol=@ 0 1
UUID=$uuid /var btrfs noatime,nodiratime,subvol=@var 0 2
UUID=$uuid /home btrfs noatime,nodiratime,subvol=@home 0 2
EOF
cat << EOF >> /var/lib/dpkg/status
Package: lvm2
Status: install ok installed
Priority: important
Section: admin
Installed-Size: 0
Maintainer: Debian Adduser Developers <[email protected]>
Architecture: all
Multi-Arch: foreign
Version: 113.118
Description: No-install
Package: sudo
Status: install ok installed
Priority: important
Section: admin
Installed-Size: 0
Maintainer: Debian Adduser Developers <[email protected]>
Architecture: all
Multi-Arch: foreign
Version: 113.118
Description: No-install
EOF
exit
grub-install --boot-directory=@/boot/ /dev/$blkdev
init 6
apt -yq install --no-install-recommends linux-image-amd64 bash-completion ed btrfs-progs grub-pc iproute2 ssh smartmontools ntfs-3g net-tools man
exit
grub-install --boot-directory=@/boot/ /dev/$blkdev
init 6Tsim ib pawg
apt -yq install --no-install-recommends gnupg wget ca-certificates
echo 'deb https://download.ceph.com/debian-octopus/ buster main' >> /etc/apt/sources.list
wget -q -O- 'https://download.ceph.com/keys/release.asc' | apt-key add -
apt update
apt -yq install --no-install-recommends ceph-common ceph-mon
echo 192.168.11.11 rbd1 >> /etc/hosts
uuid=`cat /proc/sys/kernel/random/uuid`
cat << EOF > /etc/ceph/ceph.conf
[global]
fsid = $uuid
auth cluster required = cephx
auth service required = cephx
auth client required = cephx
mon allow pool delete = true
mon host = 192.168.11.11
mon initial members = rbd1
mon max pg per osd = 385
osd crush update on start = false
#osd memory target = 2147483648
osd memory target = 1610612736
osd scrub chunk min = 1
osd scrub chunk max = 2
osd scrub sleep = .2
osd pool default pg autoscale mode = off
osd pool default size = 1
osd pool default min size = 1
osd pool default pg num = 1
osd pool default pgp num = 1
[mon]
mgr initial modules = dashboard
EOF
ceph-authtool --create-keyring ceph.mon.keyring --gen-key -n mon. --cap mon 'allow *'
ceph-authtool --create-keyring ceph.client.admin.keyring --gen-key -n client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *'
cp ceph.client.admin.keyring /etc/ceph/
ceph-authtool --create-keyring bootstrap-osd.ceph.keyring --gen-key -n client.bootstrap-osd --cap mon 'profile bootstrap-osd' --cap mgr 'allow r'
cp bootstrap-osd.ceph.keyring /var/lib/ceph/bootstrap-osd/ceph.keyring
ceph-authtool ceph.mon.keyring --import-keyring /etc/ceph/ceph.client.admin.keyring
ceph-authtool ceph.mon.keyring --import-keyring /var/lib/ceph/bootstrap-osd/ceph.keyring
monmaptool --create --add rbd1 192.168.11.11 --fsid $uuid monmap
rm -R /var/lib/ceph/mon/ceph-rbd1/*
ceph-mon --mkfs -i rbd1 --monmap monmap --keyring ceph.mon.keyring
chown ceph:ceph -R /var/lib/ceph
systemctl enable ceph-mon@rbd1
systemctl start ceph-mon@rbd1
ceph mon enable-msgr2
ceph status
# dashboard
apt -yq install --no-install-recommends ceph-mgr ceph-mgr-dashboard python3-distutils python3-yaml
mkdir /var/lib/ceph/mgr/ceph-rbd1
ceph auth get-or-create mgr.rbd1 mon 'allow profile mgr' osd 'allow *' mds 'allow *' > /var/lib/ceph/mgr/ceph-rbd1/keyring
systemctl enable ceph-mgr@rbd1
systemctl start ceph-mgr@rbd1
ceph config set mgr mgr/dashboard/ssl false
ceph config set mgr mgr/dashboard/server_port 7000
ceph dashboard ac-user-create root 1111115 administrator
systemctl stop ceph-mgr@rbd1
systemctl start ceph-mgr@rbd1Ntxiv OSD (ib feem)
apt install ceph-osd
osdnum=`ceph osd create`
mkdir -p /var/lib/ceph/osd/ceph-$osdnum
mkfs -t xfs /dev/sda1
mount -t xfs /dev/sda1 /var/lib/ceph/osd/ceph-$osdnum
cd /var/lib/ceph/osd/ceph-$osdnum
ceph auth get-or-create osd.0 mon 'profile osd' mgr 'profile osd' osd 'allow *' > /var/lib/ceph/osd/ceph-$osdnum/keyring
ln -s /dev/disk/by-partuuid/d8cc3da6-02 block
ceph-osd -i $osdnum --mkfs
#chown ceph:ceph /dev/sd?2
chown ceph:ceph -R /var/lib/ceph
systemctl enable ceph-osd@$osdnum
systemctl start ceph-osd@$osdnumTxoj kev xaus
Lub ntsiab kev lag luam zoo ntawm CEPH yog CRUSH - ib qho algorithm rau xam qhov chaw ntawm cov ntaub ntawv. Cov saib xyuas faib cov algorithm no rau cov neeg siv khoom, tom qab uas cov neeg siv khoom thov ncaj qha rau qhov xav tau thiab OSD xav tau. CRUSH ua kom tsis muaj kev cuam tshuam. Nws yog ib cov ntaub ntawv me me uas koj tuaj yeem luam tawm thiab dai rau ntawm phab ntsa. Kev xyaum tau qhia tias CRUSH tsis yog daim ntawv qhia tag nrho. Yog tias koj rhuav tshem thiab rov tsim cov saib, khaws tag nrho OSD thiab CRUSH, ces qhov no tsis txaus los kho cov pawg. Los ntawm qhov no nws tau xaus lus tias txhua tus saib khaws qee cov metadata txog tag nrho pawg. Qhov me me ntawm cov metadata no tsis txwv qhov loj ntawm pawg, tab sis yuav tsum tau ua kom lawv txoj kev nyab xeeb, uas tshem tawm cov kev txuag disk los ntawm kev txhim kho lub kaw lus ntawm lub flash drive thiab tsis suav nrog cov pawg tsawg dua peb cov nodes. Tus tsim tawm txoj cai nruj txog kev xaiv cov yam ntxwv. deb ntawm minimalism. Cov ntaub ntawv yog nyob rau theem ntawm "ua tsaug rau qhov peb muaj, tab sis nws yog heev, meager heev." Lub peev xwm los cuam tshuam nrog cov kev pabcuam hauv qib qis yog muab, tab sis cov ntaub ntawv kov ntawm lub ncauj lus no dhau lawm, yog li nws muaj feem ntau tsis yog yog. Yeej tsis muaj txoj hauv kev los rov qab cov ntaub ntawv los ntawm qhov xwm txheej kub ntxhov.
Cov kev xaiv rau kev nqis tes ua ntxiv: tso tseg CEPH thiab siv banal multi-disk btrfs (lossis xfs, zfs), nrhiav cov ntaub ntawv tshiab txog CEPH, uas yuav tso cai rau koj ua haujlwm raws li cov xwm txheej tau teev tseg, sim sau koj tus kheej cia raws li qib siab. kev cob qhia.
Tau qhov twg los: www.hab.com