Kab lus no yuav xav txog cov software thaub qab uas, los ntawm kev tawg cov ntaub ntawv ntws mus rau hauv cov khoom sib cais (chunks), tsim cov chaw cia khoom.
Repository Cheebtsam tuaj yeem txuas ntxiv compressed thiab encrypted, thiab tseem ceeb tshaj - thaum rov ua cov txheej txheem rov qab - rov siv dua.
Ib daim ntawv luam theej rau hauv cov chaw khaws cia no yog lub npe hu ua cov khoom sib txuas ua ke, piv txwv li, raws li ntau yam hash functions.
Muaj ntau ntau cov kev daws teeb meem zoo sib xws, Kuv yuav tsom rau 3: zbackup, borgbackup thiab so.
Cov txiaj ntsig xav tau
Txij li thaum txhua tus neeg thov yuav tsum tau tsim lub chaw cia khoom hauv ib txoj kev lossis lwm qhov, ib qho tseem ceeb tshaj plaws yuav tsum kwv yees qhov loj ntawm qhov chaw cia khoom. Qhov zoo tshaj plaws, nws qhov loj me yuav tsum tsis pub ntau tshaj 13 GB raws li kev lees paub, lossis tsawg dua - raug rau kev ua kom zoo.
Nws kuj yog qhov tsim nyog los tsim cov ntawv luam theej ntawm cov ntaub ntawv ncaj qha, tsis tas siv cov ntaub ntawv xws li tar, nrog rau ua haujlwm nrog ssh / sftp yam tsis muaj cov cuab yeej ntxiv xws li rsync thiab sshfs.
Cwj pwm thaum tsim cov thaub qab:
- Qhov loj ntawm qhov chaw cia khoom yuav sib npaug rau qhov loj ntawm cov kev hloov pauv, lossis tsawg dua.
- Kev thauj khoom hnyav CPU yuav tsum tau siv thaum siv compression thiab / lossis encryption, thiab lub network siab heev thiab disk thauj khoom zoo li yog tias cov txheej txheem archiving thiab / lossis encryption tab tom khiav ntawm lub chaw khaws cia.
- Yog tias qhov chaw khaws cia puas lawm, qhov yuam kev qeeb yuav tshwm sim ob qho tib si thaum tsim cov thaub qab tshiab thiab thaum sim rov qab los. Nws yog ib qho tsim nyog los npaj cov kev ntsuas ntxiv los xyuas kom meej qhov kev ncaj ncees ntawm lub chaw cia khoom lossis siv cov cuab yeej built-in los xyuas nws qhov kev ncaj ncees.
Ua hauj lwm nrog tar yog coj raws li tus nqi siv, raws li tau pom nyob rau hauv ib qho ntawm cov kab lus dhau los.
Kev sim zbackup
Cov txheej txheem dav dav ntawm zbackup yog tias qhov kev zov me nyuam pom nyob rau hauv cov ntaub ntawv nkag mus hauv thaj chaw uas muaj tib cov ntaub ntawv, ces xaiv tau compresses thiab encrypts lawv, txuag txhua qhov chaw ib zaug xwb.
Deduplication siv 64-ntsis nplhaib hash muaj nuj nqi nrog lub qhov rais zawv zawg los kuaj xyuas byte-by-byte qhov sib tw tiv thaiv cov ntaub ntawv uas twb muaj lawm (zoo ib yam li rsync siv nws).
Multi-threaded lzma thiab lzo yog siv rau compression, thiab aes rau encryption. Qhov tseeb versions muaj peev xwm tshem tawm cov ntaub ntawv qub los ntawm qhov chaw khaws cia yav tom ntej.
Qhov kev zov me nyuam tau sau rau hauv C ++ nrog tsawg kawg nkaus dependencies. Tus sau tau tshwm sim los ntawm unix-txoj kev, yog li qhov kev pab cuam lees txais cov ntaub ntawv ntawm stdin thaum tsim cov thaub qab, tsim cov ntaub ntawv zoo sib xws ntawm stdout thaum rov kho dua. Yog li, zbackup tuaj yeem siv los ua "lub tsev thaiv" zoo heev thaum sau koj tus kheej cov kev daws teeb meem. Piv txwv li, tus sau ntawm tsab xov xwm tau siv qhov kev pab cuam no ua lub ntsiab thaub qab cov cuab yeej rau cov tshuab hauv tsev txij li kwv yees li 2014.
Cov ntaub ntawv kwj yuav yog ib txwm tar tshwj tsis yog hais tias lwm yam.
Cia peb saib seb cov txiaj ntsig yog dab tsi:
Kev ua haujlwm tau txheeb xyuas hauv 2 txoj kev xaiv:
- ib lub chaw cia khoom yog tsim thiab zbackup yog launched rau ntawm lub server nrog cov ntaub ntawv qhov chaw, ces cov ntsiab lus ntawm lub repository yog pauv mus rau lub backup cia server.
- ib lub chaw cia khoom yog tsim los ntawm cov chaw khaws ntaub ntawv khaws cia, zbackup yog tsim los ntawm ssh ntawm lub chaw khaws ntaub ntawv khaws cia, thiab cov ntaub ntawv raug xa mus rau nws ntawm cov yeeb nkab.
Cov txiaj ntsig ntawm thawj qhov kev xaiv tau ua raws li hauv qab no: 43m11s - thaum siv lub chaw khaws ntaub ntawv tsis muaj ntaub ntawv thiab lub lzma compressor, 19m13s - thaum hloov lub compressor nrog lzo.
Kev thauj khoom ntawm tus neeg rau zaub mov nrog cov ntaub ntawv qub yog raws li hauv qab no (ib qho piv txwv nrog lzma tau qhia; nrog lzo muaj kwv yees li tib daim duab, tab sis qhov sib faib ntawm rsync yog kwv yees li ib feem peb ntawm lub sijhawm):
Nws yog qhov tseeb tias cov txheej txheem thaub qab no tsuas yog tsim nyog rau cov kev hloov pauv tsawg thiab me me. Nws kuj tseem ceeb heev kom txwv zbackup rau 1 xov, txwv tsis pub yuav muaj CPU siab heev, vim Qhov kev zov me nyuam zoo heev ntawm kev ua haujlwm hauv ntau cov xov. Lub load ntawm lub disk yog me me, uas feem ntau yuav tsis pom nrog ib tug niaj hnub ssd-raws li disk subsystem. Koj tuaj yeem pom meej meej qhov pib ntawm cov txheej txheem ntawm synchronizing repository cov ntaub ntawv mus rau cov chaw taws teeb server; qhov ceev ntawm kev ua haujlwm yog piv rau rsync li niaj zaus thiab nyob ntawm qhov ua tau zoo ntawm disk subsystem ntawm backup cia server. Qhov tsis zoo ntawm txoj hauv kev no yog kev khaws cia ntawm lub chaw khaws cia hauv zos thiab, vim li ntawd, cov ntaub ntawv duplicate.
Ntau qhov nthuav thiab siv tau hauv kev xyaum yog qhov kev xaiv thib ob, khiav zbackup ncaj qha rau ntawm qhov chaw khaws cia ntawm cov neeg rau zaub mov.
Ua ntej, peb yuav sim ua haujlwm yam tsis siv encryption nrog lzma compressor:
Lub sijhawm ua haujlwm ntawm txhua qhov kev xeem khiav:
Tua tawm 1
Tua tawm 2
Tua tawm 3
39m45 ua
40m20 ua
40m3 ua
7m36 ua
8m3 ua
7m48 ua
15m35 ua
15m48 ua
15m38 ua
Yog tias koj qhib encryption siv aes, cov txiaj ntsig tau ze heev:
Lub sijhawm ua haujlwm ntawm tib cov ntaub ntawv, nrog encryption:
Tua tawm 1
Tua tawm 2
Tua tawm 3
43m40 ua
44m12 ua
44m3 ua
8m3 ua
8m15 ua
8m12 ua
15m0 ua
15m40 ua
15m25 ua
Yog hais tias encryption yog ua ke nrog compression siv lzo, nws zoo li no:
Cov sijhawm ua haujlwm:
Tua tawm 1
Tua tawm 2
Tua tawm 3
18m2 ua
18m15 ua
18m12 ua
5m13 ua
5m24 ua
5m20 ua
8m48 ua
9m3 ua
8m51 ua
Qhov loj ntawm qhov chaw khaws cia tau zoo ib yam ntawm 13GB. Qhov no txhais tau hais tias deduplication ua haujlwm raug. Tsis tas li ntawd, ntawm cov ntaub ntawv compressed lawm, siv lzo muab cov txiaj ntsig pom zoo; hais txog lub sijhawm ua haujlwm tag nrho, zbackup los ze rau duplicity / duplicati, tab sis lags qab cov raws li librsync los ntawm 2-5 zaug.
Qhov zoo yog qhov pom tseeb - txuag qhov chaw disk ntawm qhov chaw khaws cia ntawm lub server. Raws li rau repository xyuas cov cuab yeej, tus sau ntawm zbackup tsis muab rau lawv; nws yog pom zoo kom siv ib tug txhaum-tolerant disk array los yog huab muab.
Zuag qhia tag nrho, lub tswv yim zoo heev, txawm tias qhov tseeb tias qhov project tau sawv ntsug li ntawm 3 xyoos (qhov kev thov zaum kawg yog txog ib xyoos dhau los, tab sis tsis muaj lus teb).
Test borgbackup
Borgbackup yog rab rawg ntawm qab nthab, lwm qhov zoo ib yam li zbackup. Sau hauv python, nws muaj cov npe ntawm cov peev xwm zoo ib yam li zbackup, tab sis kuj tuaj yeem:
- Mount backups ntawm fuse
- Tshawb xyuas cov ntsiab lus hauv chaw khaws cia
- Ua haujlwm hauv hom neeg siv-server
- Siv ntau yam compressors rau cov ntaub ntawv, nrog rau kev txiav txim siab heuristic ntawm hom ntaub ntawv thaum compressing nws.
- 2 encryption xaiv, aes thiab blake
- Built-in cuab yeej rau
kev kuaj xyuas kev ua haujlwm
borgbackup benchmark crud ssh://backup_server/repo/path local_dir
Cov txiaj ntsig tau raws li hauv qab no:
CZ-BIG 96.51 MB/s (10 100.00 MB tag nrho-zero cov ntaub ntawv: 10.36s)
RZ-BIG 57.22 MB/s (10 100.00 MB tag nrho-zero cov ntaub ntawv: 17.48s)
UZ-BIG 253.63 MB/s (10 100.00 MB tag nrho-zero cov ntaub ntawv: 3.94s)
DZ-BIG 351.06 MB/s (10 100.00 MB tag nrho-zero cov ntaub ntawv: 2.85s)
CR-BIG 34.30 MB/s (10 100.00 MB random cov ntaub ntawv: 29.15s)
RR-BIG 60.69 MB/s (10 100.00 MB random cov ntaub ntawv: 16.48s)
UR-BIG 311.06 MB/s (10 100.00 MB random cov ntaub ntawv: 3.21s)
DR-BIG 72.63 MB/s (10 100.00 MB random cov ntaub ntawv: 13.77s)
CZ-MEDIUM 108.59 MB/s (1000 1.00 MB tag nrho-zero cov ntaub ntawv: 9.21s)
RZ-MEDIUM 76.16 MB/s (1000 1.00 MB tag nrho-zero cov ntaub ntawv: 13.13s)
UZ-MEDIUM 331.27 MB/s (1000 1.00 MB tag nrho-zero cov ntaub ntawv: 3.02s)
DZ-MEDIUM 387.36 MB/s (1000 1.00 MB tag nrho-zero cov ntaub ntawv: 2.58s)
CR-MEDIUM 37.80 MB/s (1000 1.00 MB random cov ntaub ntawv: 26.45s)
RR-MEDIUM 68.90 MB/s (1000 1.00 MB random cov ntaub ntawv: 14.51s)
UR-MEDIUM 347.24 MB/s (1000 1.00 MB random cov ntaub ntawv: 2.88s)
DR-MEDIUM 48.80 MB/s (1000 1.00 MB random cov ntaub ntawv: 20.49s)
CZ-SMALL 11.72 MB/s (10000 10.00 kB tag nrho-zero cov ntaub ntawv: 8.53s)
RZ-SMALL 32.57 MB/s (10000 10.00 kB tag nrho-zero cov ntaub ntawv: 3.07s)
UZ-SMALL 19.37 MB/s (10000 10.00 kB tag nrho-zero cov ntaub ntawv: 5.16s)
DZ-SMALL 33.71 MB/s (10000 10.00 kB tag nrho-zero cov ntaub ntawv: 2.97s)
CR-SMALL 6.85 MB/s (10000 10.00 kB random cov ntaub ntawv: 14.60s)
RR-SMALL 31.27 MB/s (10000 10.00 kB random cov ntaub ntawv: 3.20s)
UR-SMALL 12.28 MB/s (10000 10.00 kB random cov ntaub ntawv: 8.14s)
DR-SMALL 18.78 MB/s (10000 10.00 kB random cov ntaub ntawv: 5.32s)
Thaum kuaj, compression heuristics yuav siv los txiav txim siab hom ntaub ntawv (compression auto), thiab cov txiaj ntsig yuav ua raws li hauv qab no:
Ua ntej, cia peb xyuas seb nws ua haujlwm li cas yam tsis muaj encryption:
Cov sijhawm ua haujlwm:
Tua tawm 1
Tua tawm 2
Tua tawm 3
4m6 ua
4m10 ua
4m5 ua
56s
58s
54s
1m26 ua
1m34 ua
1m30 ua
Yog tias koj qhib kev tso cai tso cai rau chaw khaws ntaub ntawv (kev lees paub tseeb), cov txiaj ntsig yuav raug kaw:
Cov sijhawm ua haujlwm:
Tua tawm 1
Tua tawm 2
Tua tawm 3
4m11 ua
4m20 ua
4m12 ua
1m0 ua
1m3 ua
1m2 ua
1m30 ua
1m34 ua
1m31 ua
Thaum aes encryption tau qhib, cov txiaj ntsig tsis zoo zuj zus:
Tua tawm 1
Tua tawm 2
Tua tawm 3
4m55 ua
5m2 ua
4m58 ua
1m0 ua
1m2 ua
1m0 ua
1m49 ua
1m50 ua
1m50 ua
Thiab yog tias koj hloov aes rau blake, qhov xwm txheej yuav zoo dua:
Cov sijhawm ua haujlwm:
Tua tawm 1
Tua tawm 2
Tua tawm 3
4m33 ua
4m43 ua
4m40 ua
59s
1m0 ua
1m0 ua
1m38 ua
1m43 ua
1m40 ua
Raws li nyob rau hauv cov ntaub ntawv ntawm zbackup, lub repository loj yog 13GB thiab txawm me ntsis tsawg, uas yog feem ntau xav tau. Kuv txaus siab heev rau lub sijhawm khiav; nws yog piv rau cov kev daws teeb meem raws li librsync, muab lub peev xwm dav dua. Kuv kuj zoo siab nrog lub peev xwm los teeb tsa ntau yam tsis sib xws los ntawm ib puag ncig hloov pauv, uas ua rau muaj txiaj ntsig zoo heev thaum siv borgbackup hauv hom tsis siv neeg. Kuv kuj txaus siab rau lub load thaum lub sij hawm thaub qab: txiav txim los ntawm lub processor load, borgbackup ua hauj lwm nyob rau hauv 1 xov.
Tsis muaj qhov tsis zoo tshwj xeeb thaum siv nws.
kev kuaj mob
Txawm hais tias qhov tseeb tias kev so yog qhov kev daws teeb meem tshiab (thawj 2 tus neeg sib tw tau paub rov qab rau xyoo 2013 thiab laus dua), nws muaj cov yam ntxwv zoo heev. Sau rau hauv Go.
Thaum piv nrog zbackup, nws tseem muab:
- Tshawb xyuas qhov ncaj ncees ntawm qhov chaw cia khoom (nrog rau kev kuaj xyuas hauv qhov chaw).
- Ib daim ntawv teev npe loj ntawm cov txheej txheem txhawb nqa thiab cov chaw muab kev pabcuam rau khaws cia cov thaub qab, nrog rau kev txhawb nqa rau rclone - rsync rau huab daws.
- Sib piv 2 thaub qab nrog ib leeg.
- Mounting lub repository ntawm fuse.
Feem ntau, cov npe ntawm cov yam ntxwv yog ze rau borgbackup, hauv qee qhov chaw ntau dua, hauv lwm qhov tsawg dua. Ib qho ntawm cov yam ntxwv yog tias tsis muaj txoj hauv kev los lov tes taw encryption, thiab yog li ntawv thaub qab yuav ib txwm raug encrypted. Cia peb pom hauv kev xyaum dab tsi tuaj yeem raug nyem tawm ntawm qhov software no:
Cov txiaj ntsig tau raws li hauv qab no:
Cov sijhawm ua haujlwm:
Tua tawm 1
Tua tawm 2
Tua tawm 3
5m25 ua
5m50 ua
5m38 ua
35s
38s
36s
1m54 ua
2m2 ua
1m58 ua
Cov txiaj ntsig kev ua tau zoo kuj yog piv rau rsync-raws li kev daws teeb meem thiab, feem ntau, ze heev rau borgbackup, tab sis CPU load yog siab dua (ntau cov xov khiav) thiab sawtooth.
Feem ntau, qhov kev zov me nyuam raug txwv los ntawm kev ua haujlwm ntawm disk subsystem ntawm cov ntaub ntawv khaws cia server, raws li twb tau hais nrog rsync. Qhov chaw khaws cia loj yog 13GB, ib yam li zbackup lossis borgbackup, tsis muaj qhov tsis txaus ntseeg thaum siv cov tshuaj no.
Π Π΅Π·ΡΠ»ΡΡΠ°ΡΡ
Qhov tseeb, txhua tus neeg sib tw ua tiav cov txiaj ntsig zoo sib xws, tab sis ntawm tus nqi sib txawv. Borgbackup ua tau zoo tshaj plaws ntawm tag nrho cov, restic yog qeeb me ntsis, zbackup yog tej zaum tsis tsim nyog pib siv,
thiab yog tias nws twb tau siv lawm, sim hloov nws mus rau borgbackup lossis so.
tshawb pom
Qhov zoo tshaj plaws kev daws teeb meem zoo li yuav tsum tau so, vim tias ... nws yog nws uas muaj qhov sib piv zoo tshaj plaws ntawm kev muaj peev xwm rau kev khiav hauj lwm ceev, tab sis cia peb tsis maj mus rau cov lus xaus rau tam sim no.
Borgbackup yeej tsis phem dua, tab sis zbackup tej zaum yuav zoo dua hloov. Muaj tseeb, zbackup tseem tuaj yeem siv los xyuas kom meej 3-2-1 txoj cai ua haujlwm. Piv txwv li, ntxiv rau (lib)rsync-based backup chaw.
Tshaj tawm
Backup Part 5: Kuaj bacula thiab veeam thaub qab rau linux
Backup Part 6: Sib piv cov cuab yeej thaub qab
Backup Part 7: Cov Lus Nug
Tshaj tawm los ntawm: Pavel Demkovich
Tau qhov twg los: www.hab.com