Izi ndi zomwe redundancy ikuwoneka
Ma Code Redundancy * amagwiritsidwa ntchito kwambiri m'makompyuta kuti awonjezere kudalirika kwa kusunga deta. Mu Yandex amagwiritsidwa ntchito muzinthu zambiri. Mwachitsanzo, kugwiritsa ntchito ma code of redundancy m'malo mobwerezabwereza muzinthu zathu zamkati kumapulumutsa mamiliyoni ambiri popanda kusiya kudalirika. Koma ngakhale akugwiritsidwa ntchito kwambiri, mafotokozedwe omveka bwino a momwe ma code a redundancy amagwirira ntchito ndi osowa kwambiri. Iwo amene akufuna kumvetsetsa amakumana ndi pafupifupi zotsatirazi (kuchokera ):
Dzina langa ndi Vadim, ku Yandex ndikupanga MDS yosungirako zinthu zamkati. M'nkhaniyi, ndikufotokozera m'mawu osavuta maziko amalingaliro amtundu wa redundancy (Reed-Solomon ndi LRC codes). Ndikuuzani momwe zimagwirira ntchito, popanda masamu ovuta komanso mawu osowa. Pamapeto pake ndipereka zitsanzo zogwiritsa ntchito ma code a redundancy mu Yandex.
Sindingaganizire zambiri za masamu mwatsatanetsatane, koma ndipereka maulalo kwa iwo omwe akufuna kulowa pansi mozama. Ndiwonanso kuti matanthauzo ena a masamu sangakhale okhwima, popeza nkhaniyi sinalembedwe akatswiri a masamu, koma akatswiri omwe akufuna kumvetsetsa tanthauzo la nkhaniyi.
* M'mabuku a chinenero cha Chingerezi, zizindikiro za redundancy nthawi zambiri zimatchedwa erasure codes.
1. Zofunikira za ma code of redundancy
Chofunikira cha ma code a redundancy ndi osavuta kwambiri: sungani (kapena tumizani) deta kuti isataye zolakwika zikachitika (kulephera kwa disk, zolakwika zosamutsa deta, ndi zina).
M'makhodi ambiri a redundancy, deta imagawidwa mu n data blocks, zomwe m blocks of redundancy codes amawerengedwa, zomwe zimapangitsa kuti n + m midadada ikhale yonse. Ma code a Redundancy amapangidwa m'njira yoti n midadada ya data itha kubwezeretsedwanso pogwiritsa ntchito gawo la n + m midadada. Kenako, tingoganizira za ma code a block redundancy, ndiye kuti, omwe deta imagawidwa m'ma block.
Kuti mubwezeretse n midadada yonse ya data, muyenera kukhala ndi midadada ya n + m yosachepera, popeza simungapeze n midadada pokhala ndi n-1 block (panthawiyi, muyenera kutenga chipika chimodzi "chochepa kwambiri. mpweya"). Kodi midadada ya n + m mwachisawawa ndi yokwanira kuti ipezenso deta yonse? Izi zimatengera mtundu wa ma code of redundancy, mwachitsanzo, ma code a Reed-Solomon amakulolani kuti mubwezeretse deta yonse pogwiritsa ntchito ma block n blocks, koma ma code a LRC redundancy samachita nthawi zonse.
Kusunga deta
M'makina osungiramo deta, monga lamulo, deta iliyonse imalepheretsa ndi redundancy code blocks imalembedwa ku disk yosiyana. Ndiye, ngati diski yokhazikika ikulephera, deta yoyambirira ikhoza kubwezeretsedwanso ndikuwerengedwa. Deta ikhoza kubwezeretsedwanso ngakhale ma disks angapo alephera nthawi imodzi.
Kusamutsa deta
Ma Code of Redundancy atha kugwiritsidwa ntchito kutumiza deta modalirika pamaneti osadalirika. Deta yopatsirana imagawidwa kukhala midadada, ndipo ma code of redundancy amawerengedwa kwa iwo. Onse midadada deta ndi midadada redundancy code zimafalitsidwa pa netiweki. Ngati zolakwika zichitika pamakina osasinthika (mpaka kuchuluka kwa midadada), deta imatha kufalitsidwabe pamaneti popanda zolakwika. Ma code a Reed-Solomon, mwachitsanzo, amagwiritsidwa ntchito potumiza deta pamizere yolumikizirana ndi ma satellite.
* Palinso ma code of redundancy omwe deta siigawidwa m'ma block, monga ma Hamming codes ndi ma CRC codes, omwe amagwiritsidwa ntchito kwambiri potumiza deta mu maukonde a Ethernet. Awa ndi ma code owongolera zolakwika, adapangidwa kuti azindikire zolakwika, komanso kuti asawakonze (kodi ya Hamming imalolanso kuwongolera pang'ono zolakwika).
2. Reed-Solomon zizindikiro
Zizindikiro za Reed-Solomon ndi imodzi mwazinthu zomwe zimagwiritsidwa ntchito kwambiri pakubwezeretsanso ntchito, zomwe zidapangidwa kale m'ma 1960s ndipo zidayamba kugwiritsidwa ntchito kwambiri m'ma 1980 popanga ma compact disc.
Pali mafunso awiri ofunikira kuti mumvetsetse zizindikiro za Reed-Solomon: 1) momwe mungapangire midadada ya zizindikiro za redundancy; 2) momwe mungabwezeretsere deta pogwiritsa ntchito midadada ya redundancy code. Tiyeni tipeze mayankho kwa iwo.
Kuti zikhale zosavuta, tidzaganizanso kuti n=6 ndi m=4. Mapulani ena amaganiziridwa ndi fanizo.
Momwe mungapangire midadada ya redundancy code
Chida chilichonse cha zizindikiro za redundancy chimawerengedwa mopanda ena. Ma block onse a n amagwiritsidwa ntchito kuwerengera chipika chilichonse. Pachithunzi pansipa, X1-X6 ndi midadada ya data, P1-P4 ndi midadada ya redundancy code.
Ma block onse a data akuyenera kukhala ofanana kukula, ndipo ziro bits zitha kugwiritsidwa ntchito kuyanjanitsa. Zotsatira za redundancy code blocks zidzakhala zofanana ndi midadada ya data. Ma midadada onse amagawidwa m'mawu (mwachitsanzo, ma bits 16). Tiyerekeze kuti tagawa midadada ya data kukhala k mawu. Kenako midadada yonse ya ma code of redundancy nawonso agawidwa m'mawu k.
Kuti muwerenge mawu a i-th a chipika chilichonse cha redundancy, mawu a i-th a midadada yonse ya data adzagwiritsidwa ntchito. Adzawerengedwa motsatira ndondomeko iyi:
Apa mfundo za x ndi mawu a block block, p ndi mawu a redundancy code blocks, alpha, beta, gamma ndi delta ndi manambala osankhidwa mwapadera omwe ali ofanana ndi onse i. Ziyenera kunenedwa nthawi yomweyo kuti zikhalidwe zonsezi si ziwerengero wamba, koma zinthu za gawo la Galois; ntchito +, -, *, / sizodziwika kwa tonsefe, koma ntchito zapadera zomwe zimayambitsidwa pazinthu za Galois. munda.
Chifukwa chiyani minda ya Galois ikufunika?
Zingawonekere kuti chirichonse chiri chophweka: timagawanitsa deta mu midadada, midadada m'mawu, pogwiritsa ntchito mawu a midadada deta timawerengera mawu a redundancy code midadada - timapeza redundancy code midadada. Mwambiri umu ndi momwe zimagwirira ntchito, koma satana ali mwatsatanetsatane:
- Monga tafotokozera pamwambapa, kukula kwa mawu kumakhazikika, mu chitsanzo chathu 16 bits. Mapangidwe apamwamba a ma code a Reed-Solomon ndi oti mukamagwiritsa ntchito manambala wamba, zotsatira za kuwerengera p mwina sizingawonekere pogwiritsa ntchito liwu la kukula koyenera.
- Pamene akuchira deta, mafomula pamwamba adzaonedwa ngati dongosolo equations kuti ayenera kuthetsedwa kuti achire deta. Panthawi yothetsera vutoli, pangakhale kofunikira kugawanitsa ma integers wina ndi mzake, zomwe zimabweretsa nambala yeniyeni yomwe siingakhoze kuyimiridwa molondola mu kukumbukira makompyuta.
Mavutowa amalepheretsa kugwiritsa ntchito manambala a nambala ya Reed-Solomon. Njira yothetsera vutoli ndi yoyambirira, ikhoza kufotokozedwa motere: tiyeni tibwere ndi manambala apadera omwe angathe kuimiridwa pogwiritsa ntchito mawu a kutalika kofunikira (mwachitsanzo, ma bits 16), ndi zotsatira za ntchito zonse zomwe (kuwonjezera) , kuchotsa, kuchulukitsa, kugawa) zidzaperekedwanso mu kukumbukira kwa kompyuta pogwiritsa ntchito mawu autali wofunikira.
Manambala “apadera” oterowo akhala akuphunziridwa ndi masamu kwa nthaŵi yaitali; amatchedwa masukulu. Munda ndi gulu lazinthu zomwe zili ndi ntchito zowonjezera, kuchotsa, kuchulukitsa ndi kugawa zomwe zafotokozedwera.
Minda ya Galois * ndi minda yomwe imakhala ndi zotsatira zapadera pa ntchito iliyonse (+, -, *, /) pazigawo ziwiri zilizonse zamunda. Minda ya Galois ikhoza kupangidwira manambala omwe ali mphamvu za 2: 2, 4, 8, 16, etc. (kwenikweni mphamvu za nambala iliyonse p, koma pochita timangokonda mphamvu za 2). Mwachitsanzo, pa mawu a 16-bit, iyi ndi gawo lomwe lili ndi zinthu 65, pawiri iliyonse yomwe mungapeze zotsatira za ntchito iliyonse (+, -, *, /). Miyezo ya x, p, alpha, beta, gamma, delta kuchokera ku ma equation omwe ali pamwambapa amawerengedwa ngati zinthu za gawo la Galois powerengera.
Chifukwa chake, tili ndi dongosolo la equation lomwe tingamangire midadada ya redundancy code polemba pulogalamu yoyenera pakompyuta. Pogwiritsa ntchito dongosolo lomwelo la equations, mutha kuchita kuchira kwa data.
* Uku sikutanthauzira kokhazikika, koma kulongosola.
Momwe mungabwezeretsere deta
Kubwezeretsa kumafunika pamene zina za n + m zikusowa. Izi zitha kukhala midadada ya data komanso ma block code block. Kusakhalapo kwa midadada ya data ndi / kapena kubwezeredwa kwa code block kumatanthauza kuti zofananira za x ndi / kapena p sizidziwika mu equations pamwambapa.
Ma equation a ma code a Reed-Solomon amatha kuwonedwa ngati njira yofananira momwe ma alpha, beta, gamma, delta values ndi zokhazikika, zonse x ndi p zofananira ndi midadada zomwe zilipo zimadziwika, ndipo zotsalira za x ndi p. sadziwika.
Mwachitsanzo, lolani kuti deta itseke 1, 2, 3 ndi redundancy code block 2 isapezeke, ndiye kuti gulu la i-th la mawu padzakhala dongosolo ili la equation (zosadziwika zimalembedwa mofiira):
Tili ndi dongosolo la 4 equations ndi 4 osadziwika, zomwe zikutanthauza kuti tikhoza kuthetsa ndikubwezeretsa deta!
Kuchokera m'dongosolo lino la equation pali ziganizo zingapo zomwe zikutsatira za kubwezeretsa deta kwa Reed-Solomon codes (n data blocks, m redundancy code blocks):
- Deta ikhoza kubwezeretsedwanso ngati midadada iliyonse ya m kapena yochepa itayika. Ngati midadada ya m + 1 kapena yambiri yatayika, deta siyingabwezeretsedwe: sizingatheke kuthetsa dongosolo la ma equations ndi m + 1 osadziwika.
- Kuti mubwezeretsenso chipika chimodzi cha data, muyenera kugwiritsa ntchito n iliyonse mwa midadada yotsalayo, ndipo mutha kugwiritsa ntchito nambala iliyonse ya redundancy.
Chinanso chomwe muyenera kudziwa
M'mafotokozedwe omwe ali pamwambapa, ndimapewa zinthu zingapo zofunika zomwe zimafuna kuzama mu masamu kuti muganizire. Makamaka, sindikunena kalikonse pa izi:
- Dongosolo la equation la ma code a Reed-Solomon liyenera kukhala ndi yankho (lapadera) pazophatikiza zilizonse zosadziwika (osaposa m osadziwika). Kutengera izi, milingo ya alpha, beta, gamma ndi delta imasankhidwa.
- Dongosolo la ma equation liyenera kupangidwa lokha (kutengera ma block omwe sakupezeka) ndikuthetsedwa.
- Tiyenera kupanga gawo la Galois: pa kukula kwa mawu, mutha kupeza zotsatira za ntchito iliyonse (+, -, *, /) pazigawo ziwiri zilizonse.
Pamapeto pa nkhaniyi pali zolembedwa pa nkhani zofunika zimenezi.
Kusankha kwa n ndi m
Kodi kusankha n ndi m kuchita? Mwachizoloŵezi, m'makina osungiramo deta, zizindikiro za redundancy zimagwiritsidwa ntchito kusunga malo, kotero m nthawi zonse amasankhidwa osachepera n. Makhalidwe awo enieni amadalira zinthu zingapo, kuphatikizapo:
- Kudalirika kwa kusunga deta. Zokulirapo m, kuchuluka kwa zolephera za diski zomwe zitha kupulumuka, ndiko kuti, kudalirika kwakukulu.
- Zosungirako zosafunikira. Kukwera kwa chiŵerengero cha m / n, kuwonjezereka kwa kusungirako kudzakhala kochepa, ndipo dongosololi lidzakhala lokwera mtengo.
- Pemphani nthawi yokonza. Kuchuluka kwa chiwerengero n + m, ndipamenenso nthawi yoyankha zopempha idzatalikirapo. Popeza kuti kuwerenga deta (panthawi yochira) kumafuna kuwerenga n midadada yosungidwa pa n disks zosiyanasiyana, nthawi yowerengera idzatsimikiziridwa ndi disk yochepetsetsa kwambiri.
Kuonjezera apo, kusunga deta mu ma DC angapo kumapereka zoletsa zina pa chisankho cha n ndi m: ngati 1 DC yazimitsidwa, deta iyenera kukhalapo kuti iwerengedwe. Mwachitsanzo, posungira deta mu 3 DCs, zotsatirazi ziyenera kukumana: m> = n/2, mwinamwake pangakhale vuto limene deta silikupezeka kuti iwerengedwe pamene 1 DC yazimitsidwa.
3. LRC - Ma Code Omanganso Aderalo
Kuti mubwezeretse deta pogwiritsa ntchito zizindikiro za Reed-Solomon, muyenera kugwiritsa ntchito n midadada yosasinthika. Izi ndizovuta kwambiri kwa machitidwe osungiramo deta, chifukwa kubwezeretsa deta pa disk imodzi yosweka, muyenera kuwerenga deta kuchokera kwa ena ambiri, ndikupanga katundu wambiri wowonjezera pa disks ndi maukonde.
Zolakwa zofala kwambiri ndikusafikika kwa chipika chimodzi cha data chifukwa chakulephera kapena kuchulukira kwa disk imodzi. Kodi ndizotheka kuchepetsa kuchulukirachulukira kwa data munkhaniyi (yofala kwambiri)? Zikutheka kuti mutha: pali ma code a LRC redundancy makamaka pazifukwa izi.
LRC (Local Reconstruction Codes) ndi ma code of redundancy opangidwa ndi Microsoft kuti agwiritsidwe ntchito mu Windows Azure Storage. Lingaliro la LRC ndi losavuta momwe mungathere: gawani midadada yonse m'magulu awiri (kapena kupitilira apo) ndikuwerenga gawo lazoletsa zoletsa gulu lililonse padera. Ndiye midadada ina redundancy code midadada adzawerengedwa ntchito midadada deta onse (mu LRC amatchedwa zizindikiro zapadziko lonse redundancy), ndi ena - pogwiritsa ntchito limodzi mwa magulu awiri a midadada deta (amatchedwa m'deralo redundancy zizindikiro).
LRC imatanthauzidwa ndi manambala atatu: nrl, pamene n ndi chiwerengero cha midadada, r ndi chiwerengero cha midadada yapadziko lonse ya redundancy code blocks, l ndi chiwerengero cha midadada ya redundancy code blocks. Kuti muwerenge deta pamene chipika chimodzi sichikupezeka, muyenera kuwerenga n / l midadada - izi ndizochepa nthawi l kuposa ma code a Reed-Solomon.
Mwachitsanzo, taganizirani dongosolo la LRC 6-2-2. X1–X6 — 6 midadada, P1, P2 — 2 global redundancy midadada, P3, P4 — 2 midadada redundancy midadada.
Redundancy code blocks P1, P2 amawerengedwa pogwiritsa ntchito midadada yonse. Redundancy code block P3 - pogwiritsa ntchito midadada ya data X1-X3, redundancy code block P4 - pogwiritsa ntchito midadada ya data X4-X6.
Zina zonse zimachitika mu LRC mofananiza ndi ma code a Reed-Solomon. Ma equations powerengera mawu a redundancy code blocks adzakhala:
Kusankha manambala alpha, beta, gamma, delta, zinthu zingapo ziyenera kukumana kuti zitsimikizire kuthekera kwa kuchira kwa data (ndiko, kuthetsa dongosolo la equation). Mutha kuwerenga zambiri za iwo mu .
Komanso pochita, ntchito ya XOR imagwiritsidwa ntchito kuwerengera ma code a redundancy P3, P4.
Malingaliro angapo amatsatiridwa kuchokera mu dongosolo la equation la LRC:
- Kuti mubwezeretsenso chipika chilichonse cha data, ndikokwanira kuwerenga n/l midadada (n/1 mu chitsanzo chathu).
- Ngati r + l blocks palibe, ndipo midadada yonse ikuphatikizidwa mu gulu limodzi, ndiye kuti deta silingabwezeretsedwe. Izi ndizosavuta kufotokoza ndi chitsanzo. Lolani kuti X1-X3 ndi P3 asapezeke: awa ndi r + l midadada kuchokera ku gulu lomwelo, 4 kwa ife. Ndiye tili ndi dongosolo la ma equation 3 ndi 4 osadziwika omwe sangathe kuthetsedwa.
- Muzochitika zina zonse za kusapezeka kwa r + l midadada (pamene osachepera chipika chimodzi chikupezeka kuchokera ku gulu lirilonse), deta mu LRC ikhoza kubwezeretsedwa.
Chifukwa chake, LRC imachita bwino kuposa ma code a Reed-Solomon pakubwezeretsa deta pambuyo pa zolakwika imodzi. M'makhodi a Reed-Solomon, kuti mubwezeretsenso chipika chimodzi cha data, muyenera kugwiritsa ntchito n midadada, ndipo mu LRC, kuti mutengenso chipika chimodzi cha data, ndikokwanira kugwiritsa ntchito n/l midadada (n/2 mu chitsanzo chathu). Kumbali ina, LRC ndiyotsika ku Reed-Solomon codes malinga ndi kuchuluka kwa zolakwika zovomerezeka. M'zitsanzo pamwambapa, Reed-Solomon zizindikiro akhoza kuchira deta aliyense 4 zolakwa, ndi LRC pali 2 osakaniza 4 zolakwa pamene deta sangathe anachira.
Chofunika kwambiri chimadalira momwe zinthu zilili, koma nthawi zambiri ndalama zomwe LRC imapereka zimaposa zosungirako zosadalirika.
4. Zizindikiro zina za redundancy
Kupatula ma code a Reed-Solomon ndi LRC, palinso ma code ena ambiri osafunikira. Mitundu yosiyanasiyana ya redundancy imagwiritsa ntchito masamu osiyanasiyana. Nawa ma code ena osafunikira:
- Nambala ya Redundancy pogwiritsa ntchito XOR operator. Opaleshoni ya XOR imachitidwa pa n midadada ya data, ndipo 1 chipika cha zizindikiro za redundancy zimapezedwa, ndiko kuti, n + 1 ndondomeko (n data blocks, 1 redundancy code). Zogwiritsidwa ntchito mu , komwe midadada ya data ndi ma code of redundancy amalembedwa mozungulira ku ma disks onse a gululo.
- Ngakhale-odd algorithm yotengera ntchito ya XOR. Imakulolani kuti mupange midadada iwiri ya ma code of redundancy, ndiye kuti, n +2 scheme.
- STAR algorithm yotengera ntchito ya XOR. Imakulolani kuti mupange midadada 3 ya ma code of redundancy, ndiye kuti, n+3 scheme.
- Zizindikiro za piramidi ndi ma code ena owonjezeranso kuchokera ku Microsoft.
5. Gwiritsani ntchito mu Yandex
Ma projekiti angapo a Yandex amagwiritsira ntchito ma code of redundancy posungira zodalirika. Nazi zitsanzo:
- Kusungirako zinthu zamkati za MDS, zomwe ndidalemba kumayambiriro kwa nkhaniyi.
- - MapReduce system ya Yandex.
- (Yandex DataBase) - database yatsopano yaSQL.
MDS imagwiritsa ntchito manambala a LRC redundancy, 8-2-2 chiwembu. Deta yokhala ndi ma code of redundancy imalembedwa ku ma disks osiyanasiyana a 12 m'maseva osiyanasiyana mu ma DC a 3 osiyanasiyana: ma seva 4 mu DC iliyonse. Werengani zambiri za izi mu .
YT imagwiritsa ntchito ma code a Reed-Solomon (Scheme 6-3), omwe anali oyamba kukhazikitsa, ndi ma code LRC redundancy (Scheme 12-2-2), ndi LRC kukhala njira yosungira yomwe amakonda.
YDB imagwiritsa ntchito manambala osamvetseka (Chithunzi 4-2). Za ma code of redundancy mu YDB kale .
Kugwiritsiridwa ntchito kwa mitundu yosiyanasiyana ya ma rendancy code schemes chifukwa cha zofunikira zosiyanasiyana za machitidwe. Mwachitsanzo, mu MDS, deta yosungidwa pogwiritsa ntchito LRC imayikidwa mu 3 DCs nthawi imodzi. Ndikofunika kwa ife kuti deta ikhalepo kuti tiwerenge ngati 1 ya DCs ikulephera, choncho midadada iyenera kugawidwa kudutsa ma DCs kotero kuti ngati DC palibe, chiwerengero cha midadada yosafikirika sichiloledwa. Mu dongosolo la 8-2-2, mukhoza kuyika midadada 4 mu DC iliyonse, ndiye pamene DC iliyonse yazimitsidwa, midadada 4 sichidzapezeka, ndipo deta ikhoza kuwerengedwa. Chiwembu chilichonse chomwe timasankha pochiyika mu 3 DCs, mulimonsemo payenera kukhala (r + l) / n > = 0,5, ndiko kuti, kusungirako kusungirako kudzakhala osachepera 50%.
Mu YT zinthu ndi zosiyana: gulu lililonse la YT lili mu 1 DC (magulu osiyanasiyana mu ma DC osiyanasiyana), kotero palibe choletsa chotero. Chiwembu cha 12-2-2 chimapereka 33% redundancy, ndiko kuti, kusunga deta ndikotsika mtengo, ndipo kungathenso kupulumuka mpaka 4 kutayika kwa disk panthawi imodzi, monga ndondomeko ya MDS.
Pali zambiri mbali ya ntchito zizindikiro redundancy mu kusungirako deta ndi kachitidwe processing: nuances deta kuchira, zotsatira za kuchira pa funso kuphedwa nthawi, mbali za kujambula deta, etc. Ine ndikupita kulankhula mosiyana za izi ndi zina mbali. za kugwiritsa ntchito zizindikiro za redundancy pochita, ngati mutuwo udzakhala wosangalatsa.
6. Maulalo
- Mndandanda wazambiri zokhudzana ndi ma code a Reed-Solomon ndi magawo a Galois:
Amayang'ana mozama masamu m'chinenero chofikirika. - Nkhani yochokera ku Microsoft yokhudza LRC:
Ndime 2 ikufotokoza mwachidule za chiphunzitsochi kenako ndikukambirana zomwe zachitika ndi LRC pochita. - Ndondomeko yosiyana:
- STAR ndondomeko:
- Piramidi kodi:
- Zizindikiro za Redundancy mu MDS:
- Ma Code of Redundancy mu YT:
- Nambala za Redundancy mu YDB:
Source: www.habr.com