Izvi ndizvo zvinoita redundancy
Redundancy codes * anoshandiswa zvakanyanya mumakomputa kuti awedzere kuvimbika kwekuchengetwa kwedata. MuYandex ivo vanoshandiswa mumapurojekiti akawanda. Semuenzaniso, kushandisa redundancy macode pachinzvimbo chekudzokorora mune yedu yemukati chinhu chengetedzo inochengetedza mamirioni pasina kupira kuvimbika. Asi zvisinei nekushandiswa kwavo kwakapararira, tsananguro dzakajeka dzekuti macode erudundancy anoshanda sei kushoma. Avo vanoda kunzwisisa vanotarisana nezvinenge zvinotevera (kubva ):
Zita rangu ndinonzi Vadim, paYandex ndiri kugadzira mukati mekuchengetedza chinhu MDS. Muchinyorwa chino, ini ndichatsanangura mumashoko akareruka nheyo dzedzidziso dzemakodhi ekudzosera (Reed-Solomon uye LRC makodhi). Ini ndichakuudza kuti zvinoshanda sei, pasina masvomhu akaoma uye zvisingawanzo mazwi. Pakupedzisira ini ndichapa mienzaniso yekushandisa redundancy codes muYandex.
Ini handingatarise huwandu hwemasvomhu zvakadzama, asi ini ndichapa zvinongedzo kune avo vanoda kunyura zvakadzika. Ini ndichacherechedzawo kuti dzimwe tsananguro dzemasvomhu dzinogona kunge dzisiri dzakaomarara, sezvo chinyorwa ichi chisina kuitirwa masvomhu, asi kune mainjiniya anoda kunzwisisa chinangwa chenyaya.
* Muzvinyorwa zvemutauro wechiRungu, macode erudundancy anowanzonzi erasure codes.
1. Mhedzisiro yemakodhi ekurasikirwa
Izvo zvakakosha zveese redundancy makodhi zviri nyore kwazvo: chengetedza (kana kuendesa) data kuitira kuti risarasike kana zvikanganiso zvikaitika (kutadza kwedhisiki, zvikanganiso zvekufambisa data, nezvimwewo).
Muzhinji * redundancy codes, iyo data yakakamurwa kuita n data blocks, iyo m blocks yeredundancy codes inoverengerwa, zvichikonzera huwandu hwen + m blocks. Redundancy makodhi anovakwa nenzira yekuti n zvidhinha zve data zvinogona kudzoserwa uchishandisa chikamu che n + m zvidhinha. Tevere, isu tichafunga chete block redundancy macode, kureva, ayo mune iyo data yakakamurwa kuita mabhuroko.
Kuti udzore ese n blocks yedata, iwe unofanirwa kuve uine angangoita n yen + m zvivharo, sezvo usingakwanise kuwana n zvidhinha nekungova ne-n-1 block (munyaya iyi, iwe unofanirwa kutora 1 block "kunze kwehutete. mweya"). Ko n + mabhuraki asina kujairika e n + m anovharira akakwana kuti adzore data rese? Izvi zvinoenderana nerudzi rwemakodhi ekudzokororwa, semuenzaniso, Reed-Solomon makodhi anobvumidza iwe kudzoreredza data rese uchishandisa arbitrary n blocks, asi LRC redundancy kodhi hazviite nguva dzose.
Dhata yekuchengetedza
Mune data kuchengetedza masisitimu, sekutonga, imwe neimwe yedata blocks uye redundancy code blocks inonyorerwa kune imwe dhisiki. Zvadaro, kana dhisiki yekupokana ikatadza, data rekutanga rinogona kudzoserwa uye kuverengwa. Dhata inogona kudzoserwa kunyangwe akati wandei dhisiki akatadza panguva imwe chete.
Kuendeswa kwedata
Redundancy macode anogona kushandiswa kufambisa data zvakavimbika pane isina kuvimbika network. Iyo data yakafambiswa yakakamurwa kuita mabhuraki, uye redundancy makodhi anoverengerwa ivo. Zvese zviri zviviri zvidhinha data uye redundancy kodhi mabhuroki anofambiswa pamusoro petiweki. Kana zvikanganiso zvikaitika mumabhuroko asina kurongeka (kusvika kune imwe nhamba yezvivharo), data inogona kuramba ichifambiswa pamusoro petiweki pasina kukanganisa. Reed-Solomon makodhi, semuenzaniso, anoshandiswa kufambisa data pamusoro pemitsara yekutaurirana uye mune satellite kutaurirana.
* Kune zvakare redundancy macode umo iyo data haina kukamurwa kuita mabhuroko, senge Hamming kodhi uye CRC makodhi, ayo anoshandiswa zvakanyanya kufambisa data muEthernet network. Aya makodhi ekukanganisa-kugadzirisa kodhi, akagadzirirwa kuona zvikanganiso, uye kwete kuagadzirisa (iyo Hamming kodhi inobvumirawo kururamisa zvishoma kwezvikanganiso).
2. Tsanga-Soromoni makodhi
Reed-Solomon makodhi ndeimwe yeanonyanya kushandiswa redundancy makodhi, akagadzirwa kumashure kuma1960s uye akatanga kushandiswa zvakanyanya muma1980s mukugadzirwa kwakawanda kwema compact disc.
Pane mibvunzo miviri yakakosha yekunzwisisa Reed-Solomon makodhi: 1) maitiro ekugadzira mabhuraki e redundancy makodhi; 2) maitiro ekudzoreredza data uchishandisa redundancy kodhi mabhuroko. Ngatiwane mhinduro kwavari.
Kuti zvive nyore, isu tichaenderera mberi nekufungidzira kuti n=6 uye m=4. Zvimwe zvirongwa zvinotariswa nekuenzanisa.
Nzira yekugadzira redundancy code blocks
Imwe neimwe block ye redundancy kodhi inoverengerwa yakazvimirira yevamwe. Yese n data blocks inoshandiswa kuverenga yega yega block. Mumufananidzo uri pazasi, X1-X6 zvidhinha data, P1-P4 ndeye redundancy kodhi mabhuroki.
Ese mabhuroki edata anofanirwa kunge akaenzana saizi, uye zero bits inogona kushandiswa kurongedza. Izvo zvinokonzeresa redundancy kodhi zvidhinha zvichave saizi yakafanana neiyo data blocks. Zvose zvidhinha zve data zvakakamurwa kuva mazwi (somuenzaniso, 16 bits). Ngatitii tinotsemura mabhuraki edata kuita k mazwi. Ipapo ese mabhuraki e redundancy kodhi anozopatsanurwa kuita k mazwi.
Kuverenga i-th izwi rega rega rega redundancy block, iwo i-th mazwi eese data blocks achashandiswa. Ivo vanozoverengerwa zvinoenderana neinotevera formula:
Heano maitiro x ndiwo mazwi e data blocks, p ndiwo mazwi eredundancy kodhi mabhuroki, ese alpha, beta, gamma uye delta nhamba dzakasarudzwa dzakafanana kune vese i. Izvo zvinofanirwa kutaurwa ipapo kuti ese aya maitiro haasi akajairwa nhamba, asi zvinhu zvemunda weGalois; mashandiro +, -, *, / haasi mashandiro atinoziva isu tese, asi mashandiro akakosha akaunzwa pazvinhu zveGalois. munda.
Nei minda yeGalois ichidikanwa?
Zvingaita sekuti zvinhu zvese zviri nyore: tinogovanisa data kuita mabhuroko, zvidhinha kuita mazwi, tichishandisa mazwi e data blocks tinoverenga mazwi eredundancy code blocks - tinowana redundancy code blocks. Kazhinji aya ndiwo maitiro azvinoita, asi dhiyabhorosi ari mune zvakadzama:
- Sezvataurwa pamusoro apa, izwi saizi rakagadziriswa, mumuenzaniso wedu 16 bits. Maformula ari pamusoro emakodhi eReed-Solomon ndeekuti kana uchishandisa akajairwa manhamba, mhedzisiro yekuverenga p inogona kusamiririka uchishandisa izwi rehukuru hunoshanda.
- Paunenge uchidzoreredza data, mafomula ari pamusoro anozoonekwa sehurongwa hwemaequation hunofanirwa kugadziriswa kuitira kudzoreredza iyo data. Munguva yekugadziriswa kwekugadzirisa, zvingave zvakakosha kuparadzanisa integers kune mumwe nemumwe, zvichiita kuti nhamba chaiyo isingagoni kunyatsomiririrwa mumusoro wekombuta.
Aya matambudziko anodzivirira kushandiswa kwenhamba dzeReed-Solomon macode. Mhinduro yedambudziko ndeyekutanga, inogona kutsanangurwa seizvi: ngatiuye nenhamba dzakakosha dzinogona kumiririrwa uchishandisa mazwi ehurefu hunodiwa (semuenzaniso, 16 bits), uye mhedzisiro yekuita mashandiro ese pairi (kuwedzera. , kubvisa, kuwanda, kupatsanura) zvichaiswawo mundangariro dzekombuta uchishandisa mazwi ehurefu hunodiwa.
Nhamba dzakadaro "dzakasiyana" dzakadzidzwa nemasvomhu kwenguva refu; dzinonzi ndima. Munda seti yezvinhu zvine mashandiro ekuwedzera, kubvisa, kuwanda uye kupatsanura zvakatsanangurwa kwavari.
Galois* minda iminda ine iyo yakasarudzika mhedzisiro yekushanda kwega kwega (+, -, *, /) kune chero maviri zvinhu zvemunda. Minda yeGalois inogona kuvakwa kune nhamba dziri masimba e2: 2, 4, 8, 16, nezvimwewo (chaizvoizvo masimba epi nhamba huru p, asi mukuita isu tinongofarira masimba e2). Semuenzaniso, kune 16-bit mazwi, iyi imunda ine 65 zvinhu, pairi yega yega yaunogona kuwana mhedzisiro yekushanda chero kupi (+, -, *, /). Hukoshi hwe x, p, alpha, beta, gamma, delta kubva kune equations pamusoro inozoonekwa sezvinhu zvemunda weGalois pakuverenga.
Nekudaro, isu tine hurongwa hweequation hwatinogona kuvaka nayo mabhuroko ekudzokorora macode nekunyora yakakodzera komputa chirongwa. Uchishandisa iyo yakafanana system ye equations, unogona kuita data kudzoreredza.
* Iyi haisi tsananguro yakasimba, asi rondedzero.
Maitiro ekudzoreredza data
Kudzoreredza kunodiwa kana mamwe ma n + m blocks asipo. Aya anogona kunge ari ese ari maviri madhiridhi data uye redundancy kodhi mabhuroko. Kusavapo kwe data blocks uye / kana redundancy code blocks zvinoreva kuti inoenderana x uye / kana p inosiyana hazvizivikanwe muequation iri pamusoro.
Iwo maequation eReed-Solomon macode anogona kutariswa sehurongwa hweequation umo ese alpha, beta, gamma, delta values ββari maconstants, ese x uye p anoenderana nemabhuraki aripo anozivikanwa akasiyana, uye asara x uye p. hazvizivikanwi.
Semuyenzaniso, rega data rivhare 1, 2, 3 uye redundancy kodhi block 2 risavepo, ipapo kune i-th boka remashoko pachave neinotevera system yeequation (zvisingazivikanwe zvakaiswa mutsvuku):
Tine hurongwa hwe4 equations ine 4 isingazivikanwe, zvinoreva kuti tinogona kuigadzirisa nekudzorera iyo data!
Kubva pane ino system ye equations mhedziso dzinoverengeka dzinotevera nezve kudzoreredza data kweReed-Solomon macode (n data blocks, m redundancy code blocks):
- Data inogona kudzoserwa kana chero m blocks kana mashoma akarasika. Kana m + 1 kana mamwe mabheji akarasika, iyo data haigone kudzorerwa: hazvibviri kugadzirisa hurongwa hwema equations ne m + 1 isingazivikanwe.
- Kuti udzore kunyange imwe dhata block, unofanirwa kushandisa chero n yeasara mabhuroko, uye unogona kushandisa chero yeadhindancy makodhi.
Chii chimwe chaunofanira kuziva
Mutsanangudzo iri pamusoro, ini ndinodzivirira akati wandei akakosha nyaya dzinoda kunyura zvakadzama mumasvomhu kuti ufunge. Kunyanya, ini handisi kutaura chero chinhu pamusoro pezvinotevera:
- Iyo system ye equations yeReed-Solomon macode inofanirwa kuve ne (yakasarudzika) mhinduro kune chero musanganiswa wezvisingazivikanwe (hapana kupfuura m isingazivikanwe). Zvichienderana nezvinodiwa izvi, kukosha kwealpha, beta, gamma uye delta kunosarudzwa.
- Sitimu ye equations inofanirwa kunge ichikwanisa kuvakwa otomatiki (zvichienderana nekuti zvivharo zvipi zvisiri kuwanikwa) uye kugadziriswa.
- Isu tinofanirwa kuvaka munda weGalois: kune saizi yeshoko rakapihwa, kwanisa kuwana mhedzisiro yekushanda chero kupi (+, -, *, /) kune chero zvinhu zviviri.
Pakupera kwechinyorwa pane zvinyorwa zvezvinyorwa pamusoro pezvinhu zvakakosha izvi.
Sarudzo ye n uye m
Nzira yekusarudza n uye m mukuita? Mukuita, mumasisitimu ekuchengetedza data, maredhindancy makodhi anoshandiswa kuchengetedza nzvimbo, saka m inogara yakasarudzwa isingasviki n. Hwawo hunokosha hunoenderana nehuwandu hwezvinhu, zvinosanganisira:
- Kuvimbika kwekuchengetedza data. Iyo yakakura m, iyo yakakura nhamba yekutadza kwe diski inogona kupona, ndiko kuti, iyo yakakwirira kuvimbika.
- Redundant storage. Iyo yakakwirira iyo m / n chiyero, iyo yakakwirira yekuchengetedza redundancy ichave, uye iyo inodhura iyo system ichave.
- Kumbira nguva yekugadzirisa. Iyo yakakura nhamba n + m, iyo nguva yekupindura kune zvikumbiro ichave yakareba. Sezvo kuverenga data (panguva yekudzoreredza) kunoda kuverenga n zvidhinha zvakachengetwa pa n dhisiki dzakasiyana, nguva yekuverenga ichatemwa neinononoka dhisiki.
Mukuwedzera, kuchengetedza data muDDC dzakawanda kunoisa zvirambidzo zvakawedzerwa pakusarudza n uye m: kana 1 DC yakadzimwa, iyo data inofanira kuramba iripo pakuverenga. Semuenzaniso, kana uchichengetedza data mu3 DCs, mamiriro anotevera anofanira kusangana: m > = n / 2, kana zvisina kudaro pangava nemamiriro ezvinhu apo data haiwaniki pakuverenga kana 1 DC yadzimwa.
3. LRC - Local Reconstruction Codes
Kuti udzore data uchishandisa Reed-Solomon makodhi, iwe unofanirwa kushandisa n arbitrary data blocks. Uku kushata kwakakosha kune akagoverwa ekuchengetedza data masisitimu, nekuti kudzoreredza data pane imwe yakaputsika dhisiki, iwe uchafanirwa kuverenga data kubva kune mazhinji evamwe, uchigadzira yakakura yakawedzera mutoro padhisiki uye network.
Zvikanganiso zvakajairika kusasvikika kwechivharo chimwe che data nekuda kwekutadza kana kuremerwa kweimwe dhisiki. Zvinogoneka here neimwe nzira kuderedza kuwanda kwemutoro wekudzoreredza data mune ino (yakajairika) kesi? Zvinoitika kuti iwe unogona: kune LRC redundancy makodhi zvakanangana neizvi.
LRC (Local Reconstruction Codes) macode ekudzokorodza akagadzirwa neMicrosoft kuti ashandiswe muWindows Azure Storage. Pfungwa yeLRC iri nyore sezvinobvira: patsanura ese mabhuraki edata mumapoka maviri (kana anopfuura) uye uverenge chikamu cheredundancy kodhi mabhuroki eboka rega rega zvakasiyana. Zvadaro mamwe mabhuroki e-redundancy code achaverengwa achishandisa zvose zvidhinha data (muLRC vanonzi pasi rose redundancy codes), uye vamwe - vachishandisa rimwe remapoka maviri ezvinyorwa zve data (zvinonzi makodhi emunharaunda redundancy).
LRC inotaridzwa nenhamba nhatu: nrl, apo n inhamba yedata blocks, r inhamba yepasi rose redundancy code blocks, l ndiyo nhamba yenzvimbo yemuno redundancy code blocks. Kuti uverenge data kana imwe data block isipo, iwe unofanirwa kuverenga chete n / l zvivharo - iyi inguva l zvishoma pane muReed-Solomon macode.
Semuenzaniso, funga nezveLRC 6-2-2 chirongwa. X1βX6 β 6 data blocks, P1, P2 β 2 global redundancy blocks, P3, P4 β 2 emuno redundancy blocks.
Redundancy kodhi inovhara P1, P2 inoverengerwa uchishandisa ese mabhuroki data. Redundancy code block P3 - kushandisa data blocks X1-X3, redundancy code block P4 - kushandisa data blocks X4-X6.
Zvimwe zvacho zvinoitwa muLRC nekuenzanisa neReed-Solomon makodhi. Iwo maequation ekuverenga mazwi eredundancy code blocks achave:
Kuti usarudze manhamba alpha, beta, gamma, delta, akati wandei mamiriro anofanirwa kusangana kuvimbisa mukana wekudzoreredza data (kureva, kugadzirisa equation system). Unogona kuverenga zvakawanda nezvavo mu .
Zvakare mukuita, iyo XOR mashandiro anoshandiswa kuverenga emuno redundancy makodhi P3, P4.
Mhedziso yakati wandei inotevera kubva kuhurongwa hweequations yeLRC:
- Kuti udzore chero 1 data block, zvakakwana kuverenga n/l zvidhinha (n/2 mumuenzaniso wedu).
- Kana r + l zvivharo zvisipo, uye zvivharo zvose zvakabatanidzwa muboka rimwe chete, saka data haigoni kudzorerwa. Izvi zviri nyore kutsanangura nemuenzaniso. Rega zvivhare X1βX3 uye P3 zvisavepo: aya ndiwo r + l zvivharo kubva muboka rimwe chete, 4 kwatiri. Ipapo isu tine hurongwa hwe 3 equations ine 4 isingazivikanwe isingagone kugadziriswa.
- Mune mamwe mamiriro ese ekushaikwa kwe r + l zvivharo (apo kana chivharo chimwe chete chinowanikwa kubva kuboka rega rega), iyo data muLRC inogona kudzoserwa.
Nekudaro, LRC inodarika Reed-Solomon makodhi mukudzoreredza data mushure mekukanganisa kumwe chete. MuReed-Solomon makodhi, kuti udzore kunyange chidimbu chimwe chete che data, unofanirwa kushandisa n zvidhinha, uye muLRC, kudzoreredza imwe block yedata, zvinokwana kushandisa n/l zvidhinha (n/2 mumuenzaniso wedu). Kune rimwe divi, LRC yakaderera kune Reed-Solomon makodhi maererano nehuwandu hwehuwandu hwezvikanganiso zvinotenderwa. Mumienzaniso iri pamusoro, Reed-Solomon makodhi anogona kudzoreredza data kune chero 4 zvikanganiso, uye kune LRC kune maviri masanganiswa e2 zvikanganiso kana data risingakwanisi kudzoserwa.
Izvo zvinonyanya kukosha zvinoenderana nemamiriro chaiwo, asi kazhinji mari yekuchengetera yakawandisa iyo LRC inopa inodarika iyo shoma shoma yakavimbika yekuchengetedza.
4. Mamwe macode ekurega basa
Kunze kweReed-Solomon neLRC makodhi, kune mamwe akawanda akawanda makodhi makodhi. Akasiyana macode edundancy anoshandisa akasiyana masvomhu. Heano mamwe mamwe macode edundancy:
- Redundancy kodhi uchishandisa XOR opareta. Kushanda kweXOR kunoitwa pa n data blocks, uye 1 block of redundancy codes inowanikwa, kureva, n + 1 chirongwa (n data blocks, 1 redundancy code). Inoshandiswa mu , uko mabhuraki e data uye redundancy makodhi anonyorwa cyclically kune ese madhisiki eiyo array.
- Even-odd algorithm yakavakirwa pakushanda kweXOR. Inokutendera iwe kuti uvake 2 zvidhinha zveredundancy makodhi, kureva, iyo n + 2 chirongwa.
- STAR algorithm yakavakirwa pane XOR mashandiro. Inokutendera kuti uvake matatu mabhuraki emakodhi ekudzokororwa, kureva, iyo n + 3 chirongwa.
- Pyramidi makodhi mamwe macode ekudhinda kubva kuMicrosoft.
5. Shandisa muYandex
Nhamba dzinoverengeka dzeYandex mapurojekiti ezvivakwa anoshandisa redundancy kodhi yekuchengetedza data yakavimbika. Heino mimwe mienzaniso:
- MDS mukati mechinhu chekuchengetedza, chandakanyora pamusoro pekutanga kwechinyorwa.
- - MepuReduce system yeYandex.
- (Yandex DataBase) - itsvaSQL yakagoverwa dhatabhesi.
MDS inoshandisa LRC redundancy kodhi, 8-2-2 chirongwa. Dhata ine redundancy codes inonyorerwa kune gumi nemaviri madhisiki akasiyana mumaseva akasiyana mu12 akasiyana maDC: maseva mana mune imwe neimwe DC. Verenga zvakawanda pamusoro peizvi mu .
YT inoshandisa ese Reed-Solomon macode (Scheme 6-3), ayo aive ekutanga kuita, uye LRC redundancy codes (Scheme 12-2-2), LRC iri iyo yakasarudzika nzira yekuchengetedza.
YDB inoshandisa even-odd based redundancy codes (Mufananidzo 4-2). Nezve redundancy makodhi muYDB kare .
Iko kushandiswa kweakasiyana redundancy kodhi zvirongwa kunokonzerwa nezvinodiwa zvakasiyana kune masisitimu. Semuenzaniso, muMDS, data yakachengetwa uchishandisa LRC inoiswa mu3 DCs kamwechete. Zvakakosha kwatiri kuti data inoramba iripo yekuverenga kana 1 yeimwe DCs ikakundikana, saka mabhuroki anofanira kugoverwa kumativi ose eDDCs kuitira kuti kana chero DC isingawaniki, nhamba yezvivharo zvisingasviki haisi kupfuura inobvumirwa. Muchirongwa che8-2-2, unogona kuisa mabhuroki mana muDC yega yega, zvino kana chero DC yadzimwa, mabhuroko mana anenge asipo, uye data inogona kuverengwa. Chero chirongwa chatinosarudza kana tichichiisa mu4 DCs, chero zvakadaro panofanira kuva (r + l) / n > = 4, kureva kuti, kuchengetedza redundancy kuchava zvishoma ne3%.
MuYT mamiriro acho akasiyana: sumbu rimwe nerimwe reYT rinowanikwa zvachose mu1 DC (masumbu akasiyana mumaDC akasiyana), saka hapana chirambidzo chakadaro. Iyo 12-2-2 chirongwa chinopa 33% redundancy, ndiko kuti, kuchengetedza data kwakachipa, uye inogona zvakare kurarama kusvika ku4 panguva imwe chete disk kubuda, senge MDS chirongwa.
Kune mamwe akawanda maficha ekushandiswa kweredundancy makodhi mukuchengetedza data uye masisitimu ekugadzirisa: nuances yekudzoreredza data, kukanganisa kwekudzoreredza pane query execution nguva, maficha ekurekodha data, nezvimwe. Ndiri kuzotaura zvakasiyana pamusoro peizvi nezvimwe zvinhu. yekushandiswa kweredundancy makodhi mukuita, kana iyo musoro ichave inonakidza.
6. Zvisungo
- Mutsara wezvinyorwa nezve Reed-Solomon makodhi uye Galois minda:
Vanotarisa zvakadzama masvomhu mumutauro unowanikwa. - Chinyorwa kubva kuMicrosoft nezve LRC:
Chikamu 2 chinotsanangura muchidimbu dzidziso uyezve chinokurukura zviitiko neLRC mukuita. - Even-odd scheme:
- STAR chirongwa:
- Piramidhi makodhi:
- Redundancy kodhi muMDS:
- Redundancy macode muYT:
- Redundancy makodhi muYDB:
Source: www.habr.com