Masikati akanaka Zita rangu ndinonzi Danil Lipovoy, timu yedu kuSbertech yakatanga kushandisa HBase sekuchengetedza data rekushanda. Mukati mekuidzidza, ruzivo rwakaungana zvekuti ndaida kuronga nekutsanangura (tinovimba kuti zvichabatsira kune vakawanda). Zvose zviedzo pazasi zvakaitwa neHBase shanduro 1.2.0-cdh5.14.2 uye 2.0.0-cdh6.0.0-beta1.
- General architecture
- Kunyora data kuHBASE
- Kuverenga data kubva kuHBASE
- Data Caching
- Batch data kugadzirisa MultiGet/MultiPut
- Nzira yekutsemura matafura mumatunhu (kupatsanura)
- Kukanganisa kushivirira, compactification uye nzvimbo yedata
- Settings uye maitiro
- Stress Testing
- zvakawanikwa
1. General architecture
Iyo backup Master inoteerera kurova kwemoyo kweanoshanda pane ZooKeeper node uye, kana ikanyangarika, inotora mabasa eshe.
2. Nyora data kuHBASE
Kutanga, ngatitarisei iyo yakapusa kesi - kunyora kiyi-yakakosha chinhu patafura uchishandisa put(rowkey). Mutengi anofanira kutanga aziva kuti Root Region Server (RRS), inochengeta iyo hbase: meta tafura, iripo. Anogamuchira ruzivo urwu kubva kuZooKeeper. Mushure mezvo inowana RRS uye inoverenga iyo hbase: meta tafura, kubva kwainobvisa ruzivo rwekuti ndeupi RegionServer (RS) ine basa rekuchengetedza data kune yakapihwa rowkey patafura yekufarira. Kuti ishandiswe mune ramangwana, iyo meta tafura inochengetwa nemutengi uye nekudaro mafoni anozotevera anoenda nekukurumidza, akananga kuRS.
Tevere, RS, yagamuchira chikumbiro, chekutanga inoinyorera kuWritaAheadLog (WAL), iyo inofanirwa kupora kana yaparara. Wobva wachengeta iyo data kuMemStore. Iyi ibuffer mundangariro ine makiyi akarongwa enzvimbo yakapihwa. Tafura inogona kukamurwa kuita matunhu (partitions), imwe neimwe ine disjoint seti yemakiyi. Izvi zvinokutendera kuti uise matunhu pane akasiyana maseva kuti uwane kuita kwepamusoro. Zvisinei, pasinei nekujeka kwechirevo ichi, tichaona gare gare kuti izvi hazvishande mumatambudziko ose.
Mushure mekuisa yekupinda muMemStore, mhinduro inodzoserwa kumutengi kuti iyo yekupinda yakachengetedzwa zvakabudirira. Nekudaro, muchokwadi inochengetwa chete mubuffer uye inosvika kune dhisiki chete mushure mekunge imwe nguva yenguva yapfuura kana kana yakazadzwa nedata nyowani.
Paunenge uchiita basa re "Delete", data haina kubviswa mumuviri. Iwo anongomakwa sekunge akadzimwa, uye kuparadzwa pachayo kunoitika panguva yekudaidza iyo huru compact function, iyo inotsanangurwa zvakadzama mundima 7.
Mafaira ari muHFile fomati anounganidzwa muHDFS uye nguva nenguva iyo diki compact process inotangwa, iyo inongobatanidza mafaera madiki kuita makuru pasina kudzima chero chinhu. Nekufamba kwenguva, izvi zvinoshanduka kuita dambudziko rinongoonekwa kana uchiverenga data (tichadzoka kune izvi zvishoma gare gare).
Pamusoro peiyo yekurodha inotsanangurwa pamusoro apa, kune yakanyanya kushanda nzira, ingangove iyo yakasimba yedivi reiyi database - BulkLoad. Iri muchokwadi chekuti isu takazvimiririra tinoumba maHFiles uye toaisa padhisiki, izvo zvinotitendera kuti tikwire zvakakwana uye tiwane kumhanya kwakanyanya. Muchokwadi, iyo inogumira pano haisi HBase, asi kugona kweiyo Hardware. Pazasi pane mhedzisiro yebhutsu pacluster ine 16 RegionServers uye 16 NodeManager YARN (CPU Xeon E5-2680 v4 @ 2.40GHz * 64 threads), HBase version 1.2.0-cdh5.14.2.
Pano iwe unogona kuona kuti nekuwedzera nhamba yezvikamu (matunhu) mutafura, pamwe chete neSpark executors, tinowana kuwedzera kwekukurumidza kurodha. Zvakare, kumhanya kunoenderana nevhoriyamu yekurekodha. Mabhuroko makuru anopa kuwedzera kweMB/sec, zvidhinha zvidiki muhuwandu hweakaiswa marekodhi pachikamu chenguva, zvimwe zvinhu zvese zvakaenzana.
Iwe unogona zvakare kutanga kurodha mumatafura maviri panguva imwe uye kuwana kaviri kumhanya. Pazasi unogona kuona kuti kunyora 10 KB zvidhinha kumatafura maviri kamwechete kunoitika nekumhanya kwe600 MB/sec mune imwe neimwe (yakazara 1275 MB/sec), izvo zvinoenderana nekumhanya kwekunyora kune imwe tafura 623 MB/sec (ona Nhamba 11 pamusoro)
Asi yechipiri kumhanya ine marekodhi e50 KB inoratidza kuti kukurumidza kurodha kuri kukura zvishoma, izvo zvinoratidza kuti iri kusvika painogumira kukosha. Panguva imwecheteyo, iwe unofanirwa kuyeuka kuti hapana kana mutoro wakagadzirwa paHBASE pachayo, chinodiwa pairi kutanga kupa data kubva kuhbase: meta, uye mushure mekuisa HFiles, gadzirisa iyo BlockCache data uye chengetedza iyo MemStore buffer kune disk, kana isina chinhu.
3. Kuverenga data kubva kuHBASE
Kana isu tichifunga kuti mutengi atova neruzivo rwese kubva kuhbase: meta (ona pfungwa 2), ipapo chikumbiro chinoenda zvakananga kuRS uko kiyi inodiwa inochengetwa. Kutanga, kutsvaga kunoitwa muMemCache. Kunyangwe paine data ipapo kana kwete, kutsvaga kunoitwa muBlockCache buffer uye, kana zvichidikanwa, muHFiles. Kana data yakawanikwa mufaira, inoiswa muBlockCache uye inodzoserwa nekukurumidza pachikumbiro chinotevera. Kutsvaga muHFile kuri kukurumidza nekuda kwekushandiswa kweBloom sefa, i.e. waverenga data shoma, rinobva raona kana iyi faira ine kiyi inodiwa uye kana isiri, yozoenda kune inotevera.
Mushure mekugamuchira data kubva kune matatu masosi, RS inogadzira mhinduro. Kunyanya, inogona kuendesa akati wandei akawanikwa echinhu kamwechete kana mutengi akakumbira shanduro.
4. Data Caching
Iyo MemStore neBlockCache mabuffers anogara anosvika makumi masere muzana eiyo yakagoverwa pa-murwi RS ndangariro (yasara yakachengeterwa RS sevhisi mabasa). Kana iyo yakajairwa yekushandisa modhi yakaita sekuti maitiro anonyora uye nekukasira kuverenga iyo yakafanana data, saka zvine musoro kuderedza BlockCache uye kuwedzera MemStore, nekuti. Kana kunyora data kusingapinde mucache yekuverenga, BlockCache ichashandiswa zvishoma kazhinji. Iyo BlockCache buffer ine zvikamu zviviri: LruBlockCache (nguva dzose pa-murwi) uye BucketCache (kazhinji iri kure-murwi kana paSSD). BucketCache inofanira kushandiswa kana pane zvakawanda zvekuverenga zvikumbiro uye zvisingakodzeri muLruBlockCache, iyo inotungamirira kubasa rinoshanda reGarbage Collector. Panguva imwecheteyo, haufanirwe kutarisira kuwedzera kwakanyanya kwekuita kubva pakushandisa cache yekuverenga, asi isu tichadzokera kune izvi mundima 80.
Kune imwe BlockCache yeRS yese, uye kune imwe MemStore yetafura yega yega (imwe yega yega Column Mhuri).
sei muchirevo, kana uchinyora, data haipindi mu cache uye zvechokwadi, parameters yakadaro CACHE_DATA_ON_WRITE yetafura uye "Cache DATA paKunyora" yeRS inogadzirirwa nhema. Nekudaro, mukuita, kana tikanyorera data kuMemStore, tobva taisa dhisiki (saka tichiibvisa), wobva wadzima faira rabuda, uye nekuita chikumbiro tinozobudirira kugamuchira iyo data. Uyezve, kunyangwe iwe ukadzima zvachose BlockCache uye wozadza tafura nedata nyowani, wobva waisa zvakare MemStore kudhisiki, uibvise uye uikumbire kubva kune imwe chikamu, ivo vachiri kutorwa kubva kumwe kunhu. Saka HBase inochengetedza kwete data chete, asiwo zvisinganzwisisike zvakavanzika.
hbase(main):001:0> create 'ns:magic', 'cf'
Created table ns:magic
Took 1.1533 seconds
hbase(main):002:0> put 'ns:magic', 'key1', 'cf:c', 'try_to_delete_me'
Took 0.2610 seconds
hbase(main):003:0> flush 'ns:magic'
Took 0.6161 seconds
hdfs dfs -mv /data/hbase/data/ns/magic/* /tmp/trash
hbase(main):002:0> get 'ns:magic', 'key1'
cf:c timestamp=1534440690218, value=try_to_delete_me
Iyo "Cache DATA paKuverenga" parameter inoiswa kuti nhema. Kana uine chero mazano, unogamuchirwa kuti tikurukure mumashoko.
5. Batch data processing MultiGet/MultiPut
Kugadzirisa zvikumbiro zvimwechete (Tora / Isa / Delete) ibasa rinodhura, saka kana zvichibvira, unofanirwa kuzvisanganisa kuita Rondedzero kana Rondedzero, iyo inokutendera iwe kuti uwane yakakosha kuita kwekusimudzira. Izvi ndezvechokwadi kunyanya pakushanda kwekunyora, asi pakuverenga pane gomba rinotevera. Girafu iri pazasi inoratidza nguva yekuverenga 50 marekodhi kubva kuMemStore. Kuverenga kwakaitwa mune imwe shinda uye yakachinjika axis inoratidza nhamba yemakiyi muchikumbiro. Pano iwe unogona kuona kuti kana uchiwedzera kune makiyi ane chiuru muchikumbiro chimwe chete, nguva yekuuraya inodonha, i.e. kumhanya kunowedzera. Nekudaro, neMSLAB modhi inogoneswa nekusarudzika, mushure meichi chikumbaridzo kudonha kwakanyanya mukuita kunotanga, uye iyo yakakura huwandu hwe data murekodhi, iyo yakareba nguva yekushanda.
Miedzo yakaitwa pamushini chaiwo, 8 cores, shanduro HBase 2.0.0-cdh6.0.0-beta1.
Iyo MSLAB modhi yakagadzirirwa kudzikisa kupatsanurwa kwemurwi, izvo zvinoitika nekuda kwekusanganiswa kwedata idzva nerekare. Sekushanda, kana MSLAB ikagoneswa, iyo data inoiswa mumaseru madiki (chunks) uye inogadziriswa mumachunks. Nekuda kweizvozvo, kana vhoriyamu iri pakiti yedata yakakumbirwa ichipfuura saizi yakagoverwa, kuita kunodonha zvakanyanya. Nekune rimwe divi, kudzima modhi iyi zvakare hazvikurudzirwe, nekuti zvichizotungamira kumira nekuda kweGC panguva dzekugadzirisa data. Mhinduro yakanaka ndeyekuwedzera sero vhoriyamu munyaya yekunyora inoshanda kuburikidza nekuisa panguva imwe chete nekuverenga. Zvakakosha kucherechedza kuti dambudziko hariitike kana, mushure mekurekodha, iwe uchimhanyisa kuraira kweflush, iyo inogadzirisa iyo MemStore kudhisiki, kana kana iwe ukatakura uchishandisa BulkLoad. Tafura iri pazasi inoratidza kuti mibvunzo kubva kuMemStore yehukuru (uye huwandu hwakafanana) data inokonzeresa kudzikira. Nekudaro, nekuwedzera chunksize isu tinodzosera iyo nguva yekugadzirisa kune yakajairika.
Pamusoro pekuwedzera chunksize, kupatsanura data nedunhu kunobatsira, i.e. tafura kutsemuka. Izvi zvinoita kuti zvikumbiro zvishoma zviuye kudunhu rega rega uye kana zvikapinda musero, mhinduro inoramba yakanaka.
6. Hurongwa hwekutsemura matafura kuita matunhu (kupatsanura)
Sezvo HBase iri kiyi-yakakosha kuchengetedza uye kupatsanura kunoitwa nekiyi, zvakakosha zvakanyanya kugovera iyo data zvakaenzana kumatunhu ese. Semuenzaniso, kupatsanura tafura yakadaro muzvikamu zvitatu zvinozoita kuti data rigovane mumatunhu matatu:
Izvo zvinoitika kuti izvi zvinotungamira kukudzikira kwakadzika kana iyo data yakatakurwa gare gare ichiita senge, semuenzaniso, hurefu hwakareba, mazhinji acho achitanga nedhijiti imwechete, semuenzaniso:
1000001
1000002
...
1100003
Sezvo makiyi achichengetwa senge byte array, ese anotanga zvakafanana uye ari enzvimbo imwechete #1 inochengeta iyi makiyi. Pane nzira dzakawanda dzekuparadzanisa:
HexStringSplit - Inoshandura kiyi kuita hexadecimal encoded tambo mu "00000000" => "FFFFFFFF" uye padding kuruboshwe ne zero.
UniformSplit - Inoshandura kiyi kuita dhizaini ine hexadecimal encoding muhuwandu "00" => "FF" uye padding kurudyi ne zero.
Mukuwedzera, iwe unogona kutsanangura chero huwandu kana seti yemakiyi ekutsemura uye kugadzirisa otomatiki-kupatsanura. Nekudaro, imwe yedzakareruka uye inoshanda nzira ndeye UniformSplit uye kushandiswa kwehashi concatenation, semuenzaniso akakosha mabhaiti maviri kubva pakumhanyisa kiyi kuburikidza neCRC32(rowkey) basa uye rowkey pachayo:
hashi + rowkey
Ipapo data yese ichagoverwa zvakaenzana munzvimbo dzese. Kana uchiverenga, maviri ekutanga mabyte anongoraswa uye kiyi yepakutanga inoramba iripo. RS zvakare inodzora huwandu hwe data nemakiyi mudunhu uye, kana miganho ikapfuudzwa, inozvityora kuita zvikamu.
7. Kukanganisa kushivirira uye nzvimbo yedata
Sezvo dunhu rimwe chete rine mutoro weseti yemakiyi, mhinduro kumatambudziko ane chekuita nekuparara kweRS kana kubvisa basa ndeyekuchengetedza data rese rinodiwa muHDFS. Kana RS ikadonha, tenzi anoona izvi kuburikidza nekusavapo kwekurova kwemoyo paZooKeeper node. Zvadaro inopa nzvimbo yakashumirwa kune imwe RS uye sezvo maHFiles akachengetwa mune yakagoverwa faira system, muridzi mutsva anoaverenga uye anoenderera mberi nekushandira iyo data. Zvisinei, sezvo mamwe data angave ari muMemStore uye akanga asina nguva yekupinda muHFiles, WAL, iyo inochengetwawo muHDFS, inoshandiswa kudzorera nhoroondo yekushanda. Mushure mekunge shanduko dzashandiswa, RS inokwanisa kupindura kune zvikumbiro, asi kufamba kunotungamira kune chokwadi chekuti mamwe data uye maitiro ekuvashandira anoguma pane akasiyana node, i.e. nzvimbo iri kudzikira.
Mhinduro yedambudziko iri huru compaction - iyi nzira inofambisa mafaira kune iyo node iyo inotarisira kwavari (uko nzvimbo dzavo dziripo), semugumisiro wekuti panguva iyi inotakura mutoro pane network uye disks inowedzera zvakanyanya. Nekudaro, mune ramangwana, kuwana data kunoonekwa nekukurumidza. Uye zvakare, major_compaction inoita kusanganisa kwese maHFiles kuita faira rimwe mukati medunhu, uye zvakare inochenesa data zvichienderana netafura marongero. Semuenzaniso, unogona kutsanangura huwandu hweshanduro dzechinhu chinofanira kuchengetwa kana hupenyu hwese mushure mekunge chinhu chacho chabviswa.
Iyi nzira inogona kuve nemhedzisiro yakanaka pakushanda kweHBase. Mufananidzo uri pazasi unoratidza kuti mashandiro akadzikira sei nekuda kwekurekodha data. Pano iwe unogona kuona kuti 40 shinda dzakanyora sei kune imwe tafura uye makumi mana shinda panguva imwe chete kuverenga data. Mashinda ekunyora anoburitsa akawanda uye akawanda maHFiles, anoverengwa nedzimwe shinda. Nekuda kweizvozvo, yakawanda uye yakawanda data inoda kubviswa mundangariro uye pakupedzisira iyo GC inotanga kushanda, iyo inoremadza basa rese. Kutangwa kwekubatana kukuru kwakakonzera kucheneswa kwemarara akakonzeresa uye kudzoreredzwa kwegoho.
Muedzo wakaitwa pa3 DataNodes uye 4 RS (CPU Xeon E5-2680 v4 @ 2.40GHz * 64 threads). HBase shanduro 1.2.0-cdh5.14.2
Zvinokosha kucherechedza kuti compaction huru yakatangwa patafura ye "live", umo data rainyorwa nekuverengwa. Paive nekutaura pamhepo kuti izvi zvinogona kutungamira kumhinduro isiriyo kana uchiverenga data. Kuti utarise, maitiro akatangwa akaburitsa data nyowani uye akainyorera patafura. Mushure mezvo ndakabva ndaverenga uye ndakatarisa kana kukosha kwakabuda kwakabatana nezvakanyorwa. Ipo hurongwa uhu hwaiitwa, mubatanidzwa mukuru wakaitwa kanenge ka200 uye hapana kana kukundikana kumwe kwakanyorwa. Zvichida dambudziko rinoonekwa risingawanzo uye panguva yekuremerwa chete, saka zvakachengeteka kumisa kunyora uye kuverenga maitiro sezvakarongwa uye kuita kuchenesa kudzivirira kudhirowa kwakadaro kweGC.
Zvakare, kusungirirana kukuru hakukanganise mamiriro eMemStore; kuti uiswededze kudhisiki nekuibatanidza, unofanirwa kushandisa flush (connection.getAdmin().flush(TableName.valueOf(tblName))).
8. Zvirongwa uye kushanda
Sezvatotaurwa, HBase inoratidza kubudirira kwayo kukuru uko isingade kuita chero chinhu, kana ichiita BulkLoad. Nekudaro, izvi zvinoshanda kune mazhinji masisitimu uye vanhu. Nekudaro, chishandiso ichi chakanyanya kukodzera kuchengetedza data muhuwandu mumabhuroki makuru, nepo kana maitiro achida akawanda anokwikwidza kuverenga nekunyora zvikumbiro, iyo Get uye Put mirairo inotsanangurwa pamusoro inoshandiswa. Kuti uone iwo akakwana ma paramita, kutangwa kwakaitwa neakasiyana masanganiswa etafura paramita uye marongero:
- 10 tambo dzakatangwa panguva imwe chete 3 nguva dzakatevedzana (ngatidaidze iyi bhuroka reshinda).
- Nguva yekushanda yetambo dzese mubhuroko yakaverengerwa uye yaive mhedzisiro yekushanda kweblock.
- Tshini dzese dzakashanda netafura imwechete.
- Pamberi pekutanga kwega kwega kwechivharo cheshinda, kubatanidza kukuru kwakaitwa.
- Imwe neimwe block yakangoita imwe chete yeanotevera mashandiro:
βKuisa
βWana
-Tora+Isa
- Imwe neimwe block yakaita 50 iterations yekushanda kwayo.
- Saizi yebhuroka rekodhi i100 bytes, 1000 bytes kana 10000 bytes (random).
- Mabhuroko akatangwa aine nhamba dzakasiyana dzekiyi akakumbirwa (ringava kiyi imwe chete kana gumi).
- Mabhuroko aiitwa pasi pezvirongwa zvematafura akasiyana. Parameters yakashandurwa:
-BlockCache = yakabatidzwa kana kudzima
-BlockSize = 65 KB kana 16 KB
- Zvikamu = 1, 5 kana 30
- MSLAB = inogoneswa kana yakaremara
Saka block inotaridzika seizvi:
a. MSLAB modhi yakabatidzwa/yakavharwa.
b. Tafura yakagadzirwa iyo iyo inotevera paramita yakaiswa: BlockCache = chokwadi/hapana, BlockSize = 65/16 Kb, Partition = 1/5/30.
c. Compression yakaiswa kuGZ.
d. 10 tambo dzakatangwa panguva imwe chete ichiita 1/10 put/get/get+isa operations mutafura iyi ine marekodhi e100/1000/10000 bytes, ichiita zviuru makumi mashanu zvemibvunzo mumutsara (random keys).
e. Point d yakadzokororwa katatu.
f. Nguva yekushanda yeshinda dzese yakaverengerwa.
Zvose zvinogoneka zvakasanganiswa zvakaedzwa. Zvinofungidzirwa kuti kumhanya kunodonha sezvo rekodhi saizi inowedzera, kana kuti kudzima caching kuchakonzera kudzikira. Nekudaro, chinangwa chaive chekunzwisisa dhigirii uye kukosha kwepesvedzero yeparameter yega yega, saka iyo data yakaunganidzwa yakadyiswa mukupinza kwemutsara wekudzora basa, izvo zvinoita kuti zvikwanise kuongorora kukosha uchishandisa t-nhamba. Pazasi pane mhedzisiro yemabhuraki anoita maPut mashandiro. Seti yakazara yemasanganiswa 2 * 2 * 3 * 2 * 3 = 144 sarudzo + 72 tk. zvimwe zvakaitwa kaviri. Naizvozvo, pane 216 inomhanya yakazara:
Kuedzwa kwakaitwa pane mini-cluster ine 3 DataNodes uye 4 RS (CPU Xeon E5-2680 v4 @ 2.40GHz * 64 shinda). HBase shanduro 1.2.0-cdh5.14.2.
Iyo yakanyanya kupinza yekumhanyisa masekonzi 3.7 yakawanikwa neMSLAB modhi yakadzimwa, patafura ine chikamu chimwe, neBlockCache yakagoneswa, BlockSize = 16, marekodhi e100 bytes, zvidimbu gumi papakiti.
Iyo yakaderera yekuisa yekumhanyisa ye82.8 sec yakawanikwa ine MSLAB modhi yakagoneswa, patafura ine chikamu chimwe, neBlockCache yakagoneswa, BlockSize = 16, marekodhi e10000 bytes, 1 imwe neimwe.
Zvino ngatitarisei muenzaniso. Isu tinoona kunaka kwakanaka kweiyo modhi yakavakirwa paR2, asi zviri pachena kuti extrapolation inopesana pano. Iwo chaiwo maitiro eiyo system kana maparamita achinja haazove mutsara; iyi modhi inodiwa kwete yekufungidzira, asi kuti unzwisise zvakaitika mukati meiyo yakapihwa paramita. Semuenzaniso, pano tinoona kubva kuChiyero cheMudzidzi kuti BlockSize neBlockCache paramita hazvina basa nebasa rePut (izvo zvinowanzoitika):
Asi chokwadi chekuti kuwedzera kwenhamba yezvikamu zvinotungamira mukudzikira kwekuita kunenge kusingatarisirwi (tatoona mhedzisiro yakanaka yekuwedzera nhamba yezvikamu neBulkLoad), kunyangwe zvinonzwisisika. Chekutanga, pakugadzirisa, unofanirwa kugadzira zvikumbiro kumatunhu makumi matatu pachinzvimbo cheimwe, uye vhoriyamu yedata haina kudaro zvekuti izvi zvinopa purofiti. Chechipiri, nguva yekushanda yose inotarirwa neRS inononoka, uye sezvo nhamba yeDataNodes iri pasi pehuwandu hweRSs, mamwe matunhu ane zero nzvimbo. Zvakanaka, ngatitarisei kune mashanu epamusoro:
Zvino ngationgororei mibairo yekuita Tora zvidhinha:
Huwandu hwezvikamu zvakarasika kukosha, izvo zvinogona kutsanangurwa nenyaya yekuti data rakachengetwa zvakanaka uye cache yekuverenga ndiyo inonyanya kukosha (nhamba) parameter. Nomuzvarirwo, kuwedzera huwandu hwemeseji mukukumbira kunobatsirawo pakuita basa. Zvibodzwa zvepamusoro:
Zvakanaka, pakupedzisira, ngatitarisei modhi yebhuroka yakatanga kuita tora uye toisa:
Ese ma parameter akakosha pano. Uye mhedzisiro yevatungamiriri:
9. Kuedza kwemutoro
Zvakanaka, pakupedzisira tichatangisa mutoro wakawanda kana wakaderera, asi zvinogara zvichinakidza kana uine chimwe chekufananidza nacho. Pawebhusaiti yeDataStax, kiyi yekuvandudza yeCassandra, iripo NT yehuwandu hweNoSQL storages, kusanganisira HBase vhezheni 0.98.6-1. Kurodha kwakaitwa netambo makumi mana, saizi yedata 40 bytes, SSD disks. Mhedzisiro yekuyedza mashandiro eRead-Modify-Write yakaratidza zvinotevera.
Sezvandinonzwisisa, kuverenga kwakaitwa mumatombo ezvinyorwa zve100 uye nokuda kwe16 HBase nodes, muedzo weDataStax wakaratidza kushanda kwe10 zviuru zvekushanda kwechipiri.
Iro rombo rakanaka kuti cluster yedu inewo 16 node, asi haina "rombo rakanaka" kuti imwe neimwe ine 64 cores (threads), nepo muDataStax test pane 4 chete. Ukuwo, vane SSD drives, isu tine HDDs. kana kupfuura iyo itsva vhezheni yeHBase uye CPU kushandiswa panguva yekutakura haina kuwedzera zvakanyanya (zvinoonekwa ne5-10 muzana). Nekudaro, ngatiedzei kutanga kushandisa iyi gadziriso. Default tafura marongero, kuverenga kunoitwa mukiyi renji kubva pa0 kusvika 50 miriyoni zvisina tsarukano (kureva, zvitsva nguva dzese). Tafura yacho ine mamiriyoni makumi mashanu ezvinyorwa, akakamurwa kuita zvikamu makumi matanhatu nemana. Makiyi akakurumidza kushandisa crc50. Zvirongwa zvematafura ndezvekumisikidza, MSLAB inogoneswa. Kutangisa tambo makumi mana, shinda yega yega inoverenga seti yezana makiyi asina kujairika uye pakarepo anonyora akagadzirwa zana mabhayiti kumashure kune aya makiyi.
Mira: 16 DataNode uye 16 RS (CPU Xeon E5-2680 v4 @ 2.40GHz * 64 threads). HBase shanduro 1.2.0-cdh5.14.2.
Mhedzisiro yeavhareji iri pedyo ne40 zviuru zvekushanda pasekondi, iyo iri nani zvakanyanya kupfuura muyedzo yeDataStax. Zvisinei, nokuda kwezvinangwa zvekuedza, unogona kuchinja zvishoma mamiriro acho. Hazvigoneki kuti basa rose richaitwa chete patafura imwe chete, uye zvakare pamakiyi akasiyana. Ngatifungei kuti kune imwe "inopisa" seti yemakiyi inoburitsa iyo huru mutoro. Naizvozvo, ngatiedzei kugadzira mutoro une marekodhi makuru (10 KB), zvakare mumabheji e100, mumatafura mana akasiyana uye kudzikisira huwandu hwemakiyi akakumbirwa kusvika zviuru makumi mashanu. Girafu iri pazasi inoratidza kutangwa kweshinda makumi mana, shinda imwe neimwe inoverengwa. seti yemakiyi zana uye anobva anyora zvisina tsarukano 4 KB pamakiyi aya kumashure.
Mira: 16 DataNode uye 16 RS (CPU Xeon E5-2680 v4 @ 2.40GHz * 64 threads). HBase shanduro 1.2.0-cdh5.14.2.
Munguva yekuremerwa, kusungirirwa kukuru kwakatangwa kakawanda, sezvakaratidzwa pamusoro apa, pasina nzira iyi, kushanda kunoderera zvishoma nezvishoma, zvisinei, kuwedzera mutoro unomukawo panguva yekuurayiwa. Kushungurudzika kunokonzerwa nezvikonzero zvakasiyana-siyana. Dzimwe nguva tambo dzaipedza kushanda uye paive nekumbomira apo dzaitangwazve, dzimwe nguva zvechitatu-bato zvikumbiro zvakagadzira mutoro pasumbu.
Kuverenga uye nekukurumidza kunyora ndeimwe yeakanyanya kuoma basa mamiriro eHBase. Kana iwe ukangoita zvikumbiro zviduku, semuenzaniso 100 bytes, uchizvibatanidza mumapakiti ezvimedu zve 10-50 zviuru, unogona kuwana mazana ezviuru zvekushanda kwesekondi, uye mamiriro acho akafanana nekuverenga-chete zvikumbiro. Zvakakosha kuziva kuti zvigumisiro zviri nani zvakanyanya kudarika zvakawanikwa neDataStax, kunyanya nekuda kwezvikumbiro mumabhuroki e50 zviuru.
Mira: 16 DataNode uye 16 RS (CPU Xeon E5-2680 v4 @ 2.40GHz * 64 threads). HBase shanduro 1.2.0-cdh5.14.2.
10. Mhedziso
Iyi sisitimu yakanyatso gadziridzwa, asi pesvedzero yenhamba huru yemaparamita ichiri kuzivikanwa. Mamwe acho akaedzwa, asi haana kuverengerwa mune yakaguma test set. Semuyenzaniso, zviyedzo zvekutanga zvakaratidza kusakosha kweparameter yakadai seDATA_BLOCK_ENCODING, inokodha ruzivo uchishandisa kukosha kubva kumaseru akavakidzana, ayo anonzwisisika kune data rakagadzirwa zvisina tsarukano. Kana iwe ukashandisa nhamba huru yezvinhu zvakapetwa, kuwana kunogona kuve kwakakosha. Kazhinji, tinogona kutaura kuti HBase inopa fungidziro yedhatabhesi yakakomba uye yakanyatsofungwa, iyo inogona kunyatso bereka kana uchiita maoparesheni nemabhuru makuru edata. Kunyanya kana zvichikwanisika kuparadzanisa maitiro ekuverenga nekunyora nenguva.
Kana paine chimwe chinhu mumaonero ako chisina kuburitswa zvakakwana, ndakagadzirira kukuudza zvakadzama. Tinokukumbira kuti utaure zvakaitika kwauri kana kukurukura kana usingabvumirani nechimwe chinhu.
Source: www.habr.com