Qhov no yog qhov redundancy zoo li

Redundancy codes * tau siv dav hauv cov tshuab computer kom muaj kev ntseeg siab ntawm cov ntaub ntawv khaws cia. Hauv Yandex lawv tau siv ntau yam haujlwm. Piv txwv li, siv redundancy codes es tsis txhob replication nyob rau hauv peb cov khoom nyob rau hauv cov khoom cia txuag tau ntau lab yam tsis muaj kev cia siab rau. Tab sis txawm hais tias lawv siv dav, cov lus piav qhia meej txog yuav ua li cas redundancy codes ua haujlwm tsawg heev. Cov uas xav nkag siab yog ntsib nrog kwv yees li hauv qab no (los ntawm Wikipedia):

Kuv lub npe yog Vadim, ntawm Yandex Kuv tab tom tsim cov khoom ntim hauv MDS. Hauv tsab xov xwm no, kuv yuav piav qhia hauv cov lus yooj yim theoretical foundations of redundancy codes (Reed-Solomon and LRC codes). Kuv mam li qhia koj seb nws ua haujlwm li cas, tsis muaj lej lej thiab cov ntsiab lus tsis tshua muaj. Thaum kawg kuv yuav muab piv txwv ntawm kev siv redundancy codes hauv Yandex.

Kuv yuav tsis xav txog ntau qhov kev qhia lej hauv kev nthuav dav, tab sis kuv yuav muab cov kev sib txuas rau cov neeg uas xav mus tob tob. Kuv tseem yuav nco ntsoov tias qee qhov kev txhais lej yuav tsis nruj, vim tias tsab xov xwm tsis yog rau cov kws ua lej, tab sis rau cov engineers uas xav nkag siab txog cov ntsiab lus ntawm qhov teeb meem.

* Hauv cov ntaub ntawv Askiv, cov lej redundancy feem ntau hu ua erasure codes.

1. Lub ntsiab lus ntawm redundancy codes

Lub ntsiab lus ntawm tag nrho cov lej redundancy yog qhov yooj yim heev: khaws cia (lossis xa tawm) cov ntaub ntawv kom nws tsis ploj thaum muaj teeb meem tshwm sim (disk tsis ua haujlwm, hloov cov ntaub ntawv yuam kev, thiab lwm yam).

Nyob rau hauv feem ntau * redundancy codes, cov ntaub ntawv muab faib ua n cov ntaub ntawv thaiv, uas m blocks ntawm redundancy codes raug suav, uas ua rau tag nrho cov n + m blocks. Redundancy cov lis dej num yog tsim nyob rau hauv xws li ib txoj kev uas n blocks ntawm cov ntaub ntawv yuav rov qab tau siv tsuas yog ib feem ntawm n + m blocks. Tom ntej no, peb yuav txiav txim siab tsuas yog thaiv cov lej redundancy, uas yog, cov ntaub ntawv muab faib ua cov blocks.

Yuav kom rov qab tau tag nrho n blocks ntawm cov ntaub ntawv, koj yuav tsum muaj tsawg kawg n ntawm n + m blocks, vim koj tsis tuaj yeem tau n blocks los ntawm tsuas yog n-1 thaiv (qhov no, koj yuav tsum tau siv 1 block "tawm ntawm nyias. air”). Puas yog n random blocks ntawm n + m blocks txaus los rov qab tau tag nrho cov ntaub ntawv? Qhov no yog nyob ntawm hom redundancy codes, piv txwv li, Reed-Solomon codes tso cai rau koj rov qab tau tag nrho cov ntaub ntawv siv arbitrary n blocks, tab sis LRC redundancy codes tsis tas li.

Cov ntaub ntawv cia

Nyob rau hauv cov ntaub ntawv khaws cia systems, raws li txoj cai, txhua tus ntawm cov ntaub ntawv thaiv thiab redundancy code blocks yog sau rau ib tug nyias muaj nyias disk. Tom qab ntawd, yog tias qhov kev txiav txim siab tsis ua tiav, cov ntaub ntawv qub tseem tuaj yeem rov qab los thiab nyeem. Cov ntaub ntawv tuaj yeem rov qab tau txawm tias ntau lub disks tsis ua haujlwm tib lub sijhawm.

Kev hloov ntaub ntawv

Redundancy codes tuaj yeem siv los xa cov ntaub ntawv hauv lub network tsis muaj kev ntseeg siab. Cov ntaub ntawv kis tau muab faib ua cov blocks, thiab cov lej rov qab raug suav rau lawv. Ob cov ntaub ntawv thaiv thiab redundancy code blocks raug xa mus rau hauv lub network. Yog tias qhov yuam kev tshwm sim hauv cov blocks arbitrary (txog rau qee tus lej ntawm cov blocks), cov ntaub ntawv tseem tuaj yeem xa mus rau hauv lub network yam tsis muaj qhov yuam kev. Piv txwv li, Reed-Solomon codes yog siv los xa cov ntaub ntawv hla kev sib txuas lus kho qhov muag thiab hauv kev sib txuas lus satellite.

* Tseem muaj cov lej redundancy uas cov ntaub ntawv tsis tau muab faib ua cov blocks, xws li Hamming cov lej thiab CRC cov lej, uas tau siv dav rau kev xa cov ntaub ntawv hauv Ethernet networks. Cov no yog cov lis dej num rau kev ua yuam kev-kho coding, lawv tau tsim los xyuas qhov yuam kev, thiab tsis kho lawv (lub Hamming code kuj tso cai rau kev kho ib feem ntawm qhov yuam kev).

2. Reed-Solomon codes

Reed-Solomon cov lis dej num yog ib qho ntawm feem ntau siv redundancy codes, invented rov qab nyob rau hauv 1960s thiab thawj zaug siv nyob rau hauv 1980s rau loj zus tau tej cov compact discs.

Muaj ob lo lus nug tseem ceeb rau kev nkag siab Reed-Solomon codes: 1) yuav ua li cas los tsim cov blocks ntawm redundancy codes; 2) yuav ua li cas rov qab tau cov ntaub ntawv siv redundancy code blocks. Cia peb nrhiav cov lus teb rau lawv.
Rau qhov yooj yim, peb yuav ntxiv xav tias n = 6 thiab m = 4. Lwm cov tswv yim yog xam los ntawm kev sib piv.

Yuav ua li cas los tsim redundancy code blocks

Txhua block ntawm redundancy codes raug suav ntawm nws tus kheej ntawm lwm tus. Tag nrho n cov ntaub ntawv thaiv yog siv los suav txhua qhov thaiv. Nyob rau hauv daim duab hauv qab no, X1-X6 yog cov ntaub ntawv thaiv, P1-P4 yog redundancy code blocks.

Tag nrho cov ntaub ntawv thaiv yuav tsum yog tib qhov loj, thiab xoom cov khoom siv tau rau kev sib tw. Cov txiaj ntsig redundancy code blocks yuav yog qhov loj tib yam li cov ntaub ntawv thaiv. Tag nrho cov ntaub ntawv thaiv tau muab faib ua cov lus (piv txwv li, 16 khoom). Wb hais tias peb faib cov ntaub ntawv blocks rau hauv k lo lus. Ces tag nrho cov blocks ntawm redundancy codes kuj yuav muab faib ua k lo lus.

Txhawm rau suav cov lus i-th ntawm txhua qhov kev rov ua dua tshiab, cov lus i-th ntawm tag nrho cov ntaub ntawv thaiv yuav raug siv. Lawv yuav raug xam raws li cov qauv hauv qab no:

Ntawm no tus nqi x yog cov lus ntawm cov ntaub ntawv thaiv, p yog cov lus ntawm redundancy code blocks, tag nrho alpha, beta, gamma thiab delta yog tshwj xeeb xaiv cov lej uas zoo ib yam rau tag nrho i. Nws yuav tsum tau hais tam sim ntawd tias tag nrho cov txiaj ntsig no tsis yog tus lej zoo tib yam, tab sis cov ntsiab lus ntawm Galois teb; kev ua haujlwm +, -, *, / tsis yog kev ua haujlwm paub peb txhua tus, tab sis cov haujlwm tshwj xeeb qhia txog cov ntsiab lus ntawm Galois. teb.

Vim li cas Galois teb xav tau?

Nws yuav zoo li tias txhua yam yooj yim: peb faib cov ntaub ntawv rau hauv blocks, cov blocks rau hauv cov lus, siv cov lus ntawm cov ntaub ntawv blocks peb suav cov lus ntawm redundancy code blocks - peb tau redundancy code blocks. Feem ntau qhov no yog nws ua haujlwm li cas, tab sis dab ntxwg nyoog yog nyob rau hauv cov ntsiab lus:

1. Raws li tau hais los saum no, lo lus loj yog tsau, hauv peb qhov piv txwv 16 khoom. Cov qauv saum toj no rau Reed-Solomon cov lej yog xws li thaum siv cov lej zoo tib yam, qhov tshwm sim ntawm kev suav p yuav tsis tuaj yeem sawv cev siv lo lus ntawm qhov loj me.
2. Thaum rov qab cov ntaub ntawv, cov qauv saum toj no yuav raug suav tias yog ib qho kev sib npaug uas yuav tsum tau daws kom thiaj li rov qab tau cov ntaub ntawv. Thaum lub sij hawm kev daws teeb meem, nws yuav tsum tau faib cov zauv los ntawm ib leeg, uas ua rau muaj tus lej tiag tiag uas tsis tuaj yeem sawv cev rau hauv lub cim xeeb hauv computer.

Cov teeb meem no tiv thaiv kev siv cov lej rau Reed-Solomon codes. Txoj kev daws teeb meem yog qhov qub, nws tuaj yeem piav qhia raws li hauv qab no: cia peb los nrog cov lej tshwj xeeb uas tuaj yeem sawv cev siv cov lus ntawm qhov xav tau ntev (piv txwv li, 16 khoom), thiab cov txiaj ntsig ntawm kev ua txhua yam haujlwm uas (ntxiv. , rho tawm, sib npaug, faib) kuj yuav nthuav tawm hauv lub computer nco siv cov lus ntawm qhov ntev yuav tsum tau ua.

Cov lej "tshwj xeeb" tau kawm los ntawm lej tau ntev lawm; lawv hu ua teb. Ib daim teb yog ib pawg ntawm cov ntsiab lus nrog kev ua haujlwm ntawm qhov sib ntxiv, rho tawm, sib faib thiab faib tau txhais rau lawv.

Galois * teb yog cov teb uas muaj qhov sib txawv ntawm txhua qhov haujlwm (+, -, *, /) rau ob lub ntsiab lus ntawm thaj teb. Galois teb tuaj yeem tsim tau rau cov lej uas muaj hwj chim ntawm 2: 2, 4, 8, 16, thiab lwm yam. (qhov tseeb yog lub zog ntawm txhua tus lej p, tab sis hauv kev xyaum peb tsuas yog xav txog lub zog ntawm 2). Piv txwv li, rau 16-ntsis cov lus, qhov no yog ib daim teb uas muaj 65 cov ntsiab lus, rau txhua khub uas koj tuaj yeem pom qhov tshwm sim ntawm kev ua haujlwm (+, -, *, /). Cov txiaj ntsig ntawm x, p, alpha, beta, gamma, delta los ntawm qhov sib npaug saum toj no yuav suav tias yog cov ntsiab lus ntawm Galois teb rau kev suav.

Yog li, peb muaj ib qho kev sib npaug uas peb tuaj yeem tsim cov blocks ntawm redundancy codes los ntawm kev sau ib qho kev pab cuam hauv computer. Siv tib qhov system ntawm kev sib npaug, koj tuaj yeem ua cov ntaub ntawv rov qab.

* Qhov no tsis yog ib qho kev txhais nruj, tab sis yog qhov piav qhia.

Yuav rov qab tau cov ntaub ntawv li cas

Kev kho dua tshiab yog xav tau thaum qee qhov n + m blocks ploj lawm. Cov no tuaj yeem yog ob qho tib si cov ntaub ntawv thaiv thiab redundancy code blocks. Qhov tsis muaj cov ntaub ntawv thaiv thiab / lossis rov ua dua cov lej thaiv yuav txhais tau tias qhov sib txuas x thiab / lossis p qhov sib txawv tsis paub nyob rau hauv qhov sib npaug saum toj no.

Qhov sib npaug rau Reed-Solomon cov lis dej num tuaj yeem pom tau tias yog ib qho kev sib npaug uas tag nrho cov alpha, beta, gamma, delta qhov tseem ceeb, tag nrho x thiab p sib raug rau cov blocks muaj yog paub txawv, thiab cov seem x thiab p. tsis paub.

Piv txwv li, cia cov ntaub ntawv thaiv 1, 2, 3 thiab redundancy code block 2 tsis muaj, ces rau pawg i-th ntawm cov lus yuav muaj cov kab ke hauv qab no ntawm kev sib npaug (tsis paub yog cim liab):

Peb muaj qhov system ntawm 4 qhov sib npaug nrog 4 yam tsis paub, uas txhais tau tias peb tuaj yeem daws nws thiab kho cov ntaub ntawv!

Los ntawm cov kab ke ntawm qhov sib npaug no ntau cov lus xaus ua raws li cov ntaub ntawv rov qab rau Reed-Solomon codes (n data blocks, m redundancy code blocks):

• Cov ntaub ntawv tuaj yeem rov qab tau yog tias ib qho m blocks lossis tsawg dua ploj. Yog tias m + 1 lossis ntau qhov blocks ploj lawm, cov ntaub ntawv tsis tuaj yeem rov qab los: nws tsis tuaj yeem daws qhov system ntawm m sib npaug nrog m + 1 tsis paub.
• Txhawm rau rov qab tau txawm tias ib qho thaiv cov ntaub ntawv, koj yuav tsum siv ib qho n ntawm cov seem seem, thiab koj tuaj yeem siv ib qho ntawm cov lej redundancy.

Koj yuav tsum paub dab tsi ntxiv

Hauv cov lus piav qhia saum toj no, kuv zam ntau qhov teeb meem tseem ceeb uas yuav tsum tau ua kom tob tob rau hauv lej los xav txog. Tshwj xeeb, kuv tsis tau hais dab tsi txog cov hauv qab no:

• Lub kaw lus ntawm kev sib npaug rau Reed-Solomon codes yuav tsum muaj ib qho kev daws teeb meem (tshwj xeeb) rau ib qho kev sib txuas ntawm cov tsis paub (tsis ntau tshaj m tsis paub). Raws li qhov yuav tsum tau muaj, cov txiaj ntsig ntawm alpha, beta, gamma thiab delta raug xaiv.
• Lub kaw lus ntawm kev sib npaug yuav tsum tuaj yeem ua tiav (nyob ntawm seb cov blocks twg tsis muaj) thiab daws tau.
• Peb yuav tsum tsim kom muaj Galois teb: rau ib lo lus loj, tuaj yeem pom qhov tshwm sim ntawm kev ua haujlwm (+, -, *, /) rau ob lub ntsiab lus.

Thaum kawg ntawm tsab xov xwm muaj cov lus hais txog cov ntaub ntawv ntawm cov teeb meem tseem ceeb no.

Kev xaiv ntawm n thiab m

Yuav ua li cas xaiv n thiab m hauv kev xyaum? Nyob rau hauv kev xyaum, nyob rau hauv cov ntaub ntawv cia systems, redundancy codes yog siv los cawm qhov chaw, yog li ntawd m yeej ib txwm xaiv tsawg tshaj n. Lawv cov txiaj ntsig tshwj xeeb yog nyob ntawm ntau yam, suav nrog:

• Kev ntseeg tau ntawm cov ntaub ntawv khaws cia. Qhov loj dua m, ntau dua cov disk tsis ua haujlwm uas tuaj yeem muaj sia nyob, uas yog, qhov kev ntseeg siab dua.
• Kev khaws cia tsis tu ncua. Qhov siab dua qhov sib piv m / n, qhov ntau dua qhov kev cia khoom rov ua dua yuav yog, thiab qhov kim dua qhov system yuav yog.
• Thov lub sijhawm ua haujlwm. Qhov loj dua tus lej n + m, lub sijhawm teb rau qhov kev thov yuav ntev dua. Txij li thaum nyeem cov ntaub ntawv (thaum rov qab los) yuav tsum tau nyeem n blocks khaws cia rau n sib txawv disks, lub sijhawm nyeem ntawv yuav txiav txim siab los ntawm qhov qeeb tshaj disk.

Tsis tas li ntawd, kev khaws cov ntaub ntawv hauv ntau lub DCs txwv tsis pub ntxiv rau kev xaiv ntawm n thiab m: yog tias 1 DC raug kaw, cov ntaub ntawv tseem yuav tsum muaj rau kev nyeem ntawv. Piv txwv li, thaum khaws cov ntaub ntawv hauv 3 DCs, yuav tsum ua raws li cov xwm txheej hauv qab no: m > = n / 2, txwv tsis pub yuav muaj qhov xwm txheej uas cov ntaub ntawv tsis muaj rau kev nyeem ntawv thaum 1 DC raug tua.

3. LRC - Local Reconstruction Codes

Yuav kom rov qab tau cov ntaub ntawv siv Reed-Solomon cov lej, koj yuav tsum siv cov ntaub ntawv tsis txaus ntseeg. Qhov no yog qhov tsis zoo heev rau kev faib cov ntaub ntawv khaws cia, vim tias txhawm rau kho cov ntaub ntawv ntawm ib qho tawg, koj yuav tsum tau nyeem cov ntaub ntawv los ntawm feem ntau ntawm lwm tus, tsim kom muaj qhov loj ntxiv ntawm cov disks thiab lub network.

Feem ntau yuam kev yog qhov tsis muaj peev xwm nkag tau ntawm ib qho thaiv cov ntaub ntawv vim qhov tsis ua haujlwm lossis dhau ntawm ib lub disk. Puas yog nws ua tau los txo qis cov load ntau dhau rau cov ntaub ntawv rov qab hauv qhov no (feem ntau)? Nws hloov tawm tias koj tuaj yeem: muaj LRC redundancy codes tshwj xeeb rau lub hom phiaj no.

LRC (Local Reconstruction Codes) yog cov lej redundancy tsim los ntawm Microsoft rau siv hauv Windows Azure Storage. Lub tswv yim ntawm LRC yog qhov yooj yim li sai tau: faib tag nrho cov ntaub ntawv blocks rau hauv ob (lossis ntau dua) pawg thiab nyeem ib feem ntawm cov lej rov ua dua rau txhua pab pawg sib cais. Tom qab ntawd qee qhov redundancy code blocks yuav raug suav siv tag nrho cov ntaub ntawv thaiv (hauv LRC lawv raug hu ua cov lej redundancy thoob ntiaj teb), thiab qee qhov - siv ib ntawm ob pawg ntawm cov ntaub ntawv thaiv (lawv hu ua cov lej redundancy hauv zos).

LRC yog txhais los ntawm peb tus lej: nrl, qhov twg n yog tus naj npawb ntawm cov ntaub ntawv thaiv, r yog tus lej ntawm lub ntiaj teb redundancy code blocks, l yog tus lej ntawm cov lej hauv zos redundancy code blocks. Txhawm rau nyeem cov ntaub ntawv thaum ib cov ntaub ntawv thaiv tsis muaj, koj yuav tsum nyeem tsuas yog n / l blocks - qhov no yog l zaus tsawg dua hauv Reed-Solomon codes.

Piv txwv li, xav txog LRC 6-2-2 tswvyim. X1–X6 — 6 cov ntaub ntawv thaiv, P1, P2 — 2 lub ntiaj teb redundancy blocks, P3, P4 — 2 lub zos redundancy blocks.

Redundancy code blocks P1, P2 raug suav siv tag nrho cov ntaub ntawv thaiv. Redundancy code thaiv P3 - siv cov ntaub ntawv thaiv X1-X3, redundancy code thaiv P4 - siv cov ntaub ntawv thaiv X4-X6.

Qhov seem yog ua tiav hauv LRC los ntawm kev sib piv nrog Reed-Solomon codes. Qhov sib npaug rau suav cov lus ntawm redundancy code blocks yuav yog:

Txhawm rau xaiv cov lej alpha, beta, gamma, delta, ntau cov xwm txheej yuav tsum tau ua kom tau raws li qhov yuav tau txais cov ntaub ntawv rov qab (uas yog, daws qhov kev sib npaug). Koj tuaj yeem nyeem ntxiv txog lawv hauv Tshooj.
Tsis tas li ntawd hauv kev xyaum, XOR kev ua haujlwm yog siv los xam cov lej rov qab hauv zos P3, P4.

Ib tug xov tooj ntawm cov lus xaus ua raws li qhov system ntawm kev sib npaug rau LRC:

• Txhawm rau rov qab tau 1 cov ntaub ntawv thaiv, nws txaus los nyeem n / l blocks (n / 2 hauv peb qhov piv txwv).
• Yog tias r + l blocks tsis muaj, thiab tag nrho cov blocks suav nrog hauv ib pab pawg, ces cov ntaub ntawv yuav rov qab tsis tau. Qhov no yog ib qho yooj yim los piav nrog ib qho piv txwv. Cia cov blocks X1–X3 thiab P3 tsis muaj: cov no yog r + l blocks los ntawm tib pab pawg, 4 hauv peb rooj plaub. Tom qab ntawd peb muaj qhov system ntawm 3 qhov sib npaug nrog 4 yam tsis paub uas daws tsis tau.
• Nyob rau hauv txhua rooj plaub ntawm kev tsis muaj r + l blocks (thaum tsawg kawg yog ib qho thaiv los ntawm txhua pab pawg), cov ntaub ntawv hauv LRC tuaj yeem rov qab los.

Yog li, LRC outperforms Reed-Solomon codes nyob rau hauv recovering cov ntaub ntawv tom qab ib tug yuam kev. Hauv Reed-Solomon cov lis dej num, kom rov qab tau txawm tias ib qho thaiv ntawm cov ntaub ntawv, koj yuav tsum siv n blocks, thiab hauv LRC, kom rov qab tau ib qho thaiv cov ntaub ntawv, nws txaus siv n / l blocks (n / 2 hauv peb qhov piv txwv). Ntawm qhov tod tes, LRC yog qhov qis dua rau Reed-Solomon cov lej raws li qhov siab tshaj plaws ntawm kev tso cai yuam kev. Hauv cov piv txwv saum toj no, Reed-Solomon cov lej tuaj yeem rov qab tau cov ntaub ntawv rau ib qho 4 yuam kev, thiab rau LRC muaj 2 kev sib txuas ntawm 4 yuam kev thaum cov ntaub ntawv tsis tuaj yeem rov qab tau.

Dab tsi yog qhov tseem ceeb tshaj yog nyob ntawm qhov xwm txheej tshwj xeeb, tab sis feem ntau qhov kev txuag nyiaj ntau dhau uas LRC muab ntau dua qhov kev cia siab me ntsis.

4. Lwm cov lej redundancy

Dhau li ntawm Reed-Solomon thiab LRC cov lej, muaj ntau lwm cov lej rov ua dua. Cov lej sib txawv siv cov lej sib txawv. Nov yog qee cov lej redundancy:

• Redundancy code siv tus neeg teb xov tooj XOR. Kev ua haujlwm XOR yog ua tiav ntawm n cov ntaub ntawv thaiv, thiab 1 thaiv cov lej redundancy tau txais, uas yog, n + 1 scheme (n data blocks, 1 redundancy code). Siv hauv RAID 5, qhov twg blocks ntawm cov ntaub ntawv thiab redundancy codes yog cyclically sau rau tag nrho cov disks ntawm lub array.
• Txawm- khib algorithm raws li kev ua haujlwm XOR. Tso cai rau koj los tsim 2 blocks ntawm redundancy codes, uas yog, lub tswv yim n + 2.
• STAR algorithm raws li kev ua haujlwm XOR. Tso cai rau koj los tsim 3 blocks ntawm redundancy codes, uas yog, lub tswv yim n + 3.
• Pyramide cov lej yog lwm cov lej rov ua dua los ntawm Microsoft.

5. Siv hauv Yandex

Ib tug xov tooj ntawm Yandex infrastructure tej yaam num siv redundancy lis dej num rau txhim khu kev qha cov ntaub ntawv cia. Nov yog qee qhov piv txwv:

• MDS sab hauv khoom cia, uas kuv tau sau txog thaum pib ntawm tsab xov xwm.
• YT - MapReduce system ntawm Yandex.
• YDB (Yandex DataBase) - newSQL faib database.

MDS siv LRC redundancy codes, 8-2-2 scheme. Cov ntaub ntawv nrog cov lej redundancy yog sau rau 12 qhov sib txawv disks hauv cov servers sib txawv hauv 3 qhov sib txawv DCs: 4 servers hauv txhua DC. Nyeem ntxiv txog qhov no hauv Tshooj.

YT siv ob qho tib si Reed-Solomon codes (Scheme 6-3), uas yog thawj zaug los siv, thiab LRC redundancy codes (Scheme 12-2-2), nrog LRC yog txoj kev nyiam khaws cia.

YDB siv cov lej sib txawv raws li qhov sib txawv (Daim duab 4-2). Hais txog redundancy codes hauv YDB twb hais rau Highload.

Kev siv cov txheej txheem sib txawv ntawm cov lej sib txawv yog vim muaj cov cai sib txawv rau cov tshuab. Piv txwv li, hauv MDS, cov ntaub ntawv khaws cia siv LRC muab tso rau hauv 3 DCs ib zaug. Nws yog ib qho tseem ceeb rau peb tias cov ntaub ntawv tseem muaj rau kev nyeem ntawv yog tias 1 ntawm ib qho DCs ua tsis tau, yog li cov blocks yuav tsum tau muab faib thoob plaws DCs yog li yog tias ib qho DC tsis muaj, tus naj npawb ntawm cov blocks tsis muaj ntau tshaj qhov tso cai. Hauv cov tswv yim 8-2-2, koj tuaj yeem tso 4 blocks hauv txhua DC, tom qab ntawd thaum twg DC raug kaw, 4 blocks yuav tsis muaj, thiab cov ntaub ntawv tuaj yeem nyeem tau. Txawm li cas los xij peb xaiv thaum muab tso rau hauv 3 DCs, nyob rau hauv txhua rooj plaub yuav tsum muaj (r + l) / n > = 0,5, uas yog, cia redundancy yuav tsawg kawg yog 50%.

Hauv YT qhov xwm txheej sib txawv: txhua YT pawg yog nyob hauv 1 DC (ib pawg sib txawv hauv DCs sib txawv), yog li tsis muaj qhov txwv tsis pub muaj. Lub tswv yim 12-2-2 muab 33% redundancy, uas yog, khaws cov ntaub ntawv yog pheej yig dua, thiab nws tseem tuaj yeem muaj sia nyob mus txog 4 qhov kev tawm tsam ib txhij, ib yam li MDS lub tswv yim.

Muaj ntau ntau yam ntxiv ntawm kev siv cov lej redundancy hauv cov ntaub ntawv khaws cia thiab kev ua haujlwm: nuances ntawm cov ntaub ntawv rov qab, qhov cuam tshuam ntawm kev rov qab los ntawm cov lus nug ua tiav lub sijhawm, cov yam ntxwv ntawm cov ntaub ntawv kaw, thiab lwm yam. Kuv yuav tham nyias txog cov no thiab lwm yam nta. ntawm kev siv cov lej redundancy hauv kev xyaum, yog lub ntsiab lus yuav nthuav.

6. Txuas

1. Ib cov kab lus hais txog Reed-Solomon codes thiab Galois teb: https://habr.com/ru/company/yadro/blog/336286/
   https://habr.com/ru/company/yadro/blog/341506/
   Lawv ua tib zoo saib cov lej hauv cov lus siv tau.
2. Tsab ntawv los ntawm Microsoft txog LRC: https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/LRC12-cheng20webpage.pdf
   Nqe 2 piav qhia luv luv txog txoj kev xav thiab tom qab ntawd tham txog kev paub nrog LRC hauv kev xyaum.
3. Qhov sib txawv-kawg scheme: https://people.eecs.berkeley.edu/~kubitron/courses/cs262a-F12/handouts/papers/p245-blaum.pdf
4. STAR scheme: https://www.usenix.org/legacy/event/fast05/tech/full_papers/huang/huang.pdf
5. Pyramid Codes: https://www.microsoft.com/en-us/research/publication/pyramid-codes-flexible-schemes-to-trade-space-for-access-efficiency-in-reliable-data-storage-systems/
6. Redundancy codes hauv MDS: https://habr.com/ru/company/yandex/blog/311806
7. Redundancy codes hauv YT: https://habr.com/ru/company/yandex/blog/311104/
8. Redundancy codes hauv YDB: https://www.youtube.com/watch?v=dCpfGJ35kK8

Tau qhov twg los: www.hab.com