Chiphunzitso ndi machitidwe ogwiritsira ntchito HBase

Moni! Dzina langa ndine Danil Lipovoy, ndipo gulu lathu ku Sbertech layamba kugwiritsa ntchito HBase ngati malo osungiramo zinthu. Pamene tikuzifufuza, tapeza zina zofunika zomwe timafuna kuzikonza ndi kuzifotokoza (tikukhulupirira kuti izi zitha kuthandiza ambiri). Kuyesera konse pansipa kunachitika ndi mitundu ya HBase 1.2.0-cdh5.14.2 ndi 2.0.0-cdh6.0.0-beta1.

1. General zomangamanga
2. Kulembera deta ku HBASE
3. Kuwerenga deta kuchokera ku HBASE
4. Kusunga deta
5. MultiGet/MultiPut batch data processing
6. Kugawa Matebulo M'magawo (Spilting) Strategy
7. Kulekerera zolakwika, kuphatikizika ndi malo a data
8. Zokonda ndi machitidwe
9. Kuyesa Kupanikizika
10. anapezazo

1. Kamangidwe kake

Master Backup imamvera kugunda kwa mtima kwa ZooKeeper yogwira pa node ndikugwira ntchito za mbuyeyo ngati zitatha.

2. Kulembera deta ku HBASE

Choyamba, tiyeni tiganizire chinthu chosavuta: kulemba chinthu chofunika kwambiri patebulo pogwiritsa ntchito put(rowkey). Makasitomala amayenera kudziwa kaye komwe Root Region Server (RRS) ili, yomwe imasunga hbase:meta table. Imapeza izi kuchokera ku ZooKeeper. Imafika pa RRS ndikuwerenga hbase:meta table, pomwe imapezanso zambiri za RegionServer (RS) yomwe ili ndi udindo wosunga deta ya rowkey yomwe wapatsidwa patebulo lachidwi. Kuti mugwiritse ntchito m'tsogolo, ma metatable amasungidwa ndi kasitomala, kotero zolowera zotsatizana zimapita mwachangu ndikupita ku RS.

Atalandira pempho, RS poyamba amalembera kwa WriteAheadLog (WAL), yomwe ndi yofunikira kuti mubwezeretsedwe pakagwa ngozi. Imasunga zidziwitsozo ku MemStore, buffer yokumbukira yomwe ili ndi makiyi osankhidwa a dera lomwe laperekedwa. Gome litha kugawidwa m'magawo (magawo), chilichonse chimakhala ndi makiyi osagwirizana. Izi zimathandiza kuti pakhale ntchito zapamwamba poyika zigawo pa ma seva osiyanasiyana. Komabe, ngakhale mawu odziwikiratu awa, tiwona pambuyo pake kuti izi sizigwira ntchito muzochitika zonse.

Mbiri ikayikidwa ku MemStore, kasitomala amalandira yankho losonyeza kuti mbiriyo yasungidwa bwino. Komabe, mbiriyo imasungidwa mu buffer yokha ndipo idzangolembedwa ku diski pakapita nthawi inayake kapena buffer ikadzadza ndi data yatsopano.

Pamene "Chotsani" ntchito ikuchitika, deta si thupi zichotsedwa. Zimangodziwika kuti zachotsedwa, ndipo chiwonongeko chokha chimachitika pamene ntchito yaikulu yophatikizika imatchedwa, yomwe ikufotokozedwa mwatsatanetsatane mu Gawo 7.

Mafayilo amtundu wa HFile amasonkhanitsidwa mu HDFS, ndipo nthawi ndi nthawi, njira yaying'ono yophatikizika imayendetsedwa, yomwe imangophatikiza mafayilo ang'onoang'ono kukhala akuluakulu popanda kuchotsa chilichonse. Pakapita nthawi, izi zimakhala zovuta zomwe zimangodziwonetsera powerenga deta (tibwereranso ku izi pambuyo pake).

Kuphatikiza pa kutsitsa komwe kwafotokozedwa pamwambapa, pali njira yabwino kwambiri, yomwe mwina ndi gawo lamphamvu kwambiri pa database iyi: BulkLoad. Zimaphatikizapo kupanga ma HFiles pamanja ndikuwayika pa disk, kulola kuti scalability ikhale yabwino kwambiri ndikukwaniritsa kuthamanga kolemekezeka. Kwenikweni, malire apa si HBase, koma kuthekera kwa hardware. Pansipa pali zotsatsira pagulu lopangidwa ndi 16 RegionServers ndi 16 YARN NodeManagers (CPU Xeon E5-2680 v4 @ 2.40GHz * 64 ulusi), mtundu wa HBase 1.2.0-cdh5.14.2.

Apa tikuwona kuti kuwonjezera kuchuluka kwa magawo (magawo) patebulo, komanso Spark executors, kumawonjezera kuthamanga. Kuthamanga kumadaliranso kuchuluka kwa kulemba. Mipiringidzo ikuluikulu imapereka kuwonjezeka kwa MB / sec, pamene midadada yaying'ono imapereka kuwonjezeka kwa chiwerengero cha zolemba zomwe zayikidwa pa nthawi imodzi, zinthu zina zonse zimakhala zofanana.

Mukhozanso kutsegula magome awiri nthawi imodzi kuti pawiri liwiro. Pansipa, mutha kuwona kuti kulemba midadada ya 10 KB kumatebulo awiri nthawi imodzi kumayendera pafupifupi 600 MB/sec iliyonse (1275 MB/sec Total), zomwe ndi zofanana ndi kulembera patebulo limodzi pa 623 MB/sec (onani #11 pamwambapa).

Kuthamanga kwachiwiri ndi zolemba za 50 KB kumasonyeza kuti kuthamanga kwachangu kumangowonjezera pang'ono, kusonyeza kuti mfundo zazikulu zikuyandikira. Ndikofunika kukumbukira kuti HBASE palokha ilibe katundu pano. Zomwe zikuyenera kuchita ndikuyamba kutulutsa deta kuchokera ku hbase:meta, ndiye, mutatha kuthandizira ma HFiles, sungani deta ya BlockCache, ndikusunga buffer ya MemStore ku disk ngati ilibe kanthu.

3. Kuwerenga deta kuchokera ku HBASE

Poganiza kuti chidziwitso chonse chochokera ku hbase: meta chilipo kale kwa kasitomala (onani mfundo 2), pempho likupita mwachindunji ku RS kumene fungulo lofunikira limasungidwa. Kusakaku kumachitika koyamba ku MemCache. Mosasamala kanthu kuti deta ilipo, kufufuza kumachitidwanso mu BlockCache buffer ndipo, ngati kuli kofunikira, mu HFiles. Ngati deta ikupezeka mu fayilo, imayikidwa mu BlockCache ndipo idzabwezeredwa mwamsanga pa pempho lotsatira. Kusaka mu HFile ndikofulumira chifukwa chogwiritsa ntchito fyuluta ya Bloom; mutatha kuwerenga pang'ono deta, nthawi yomweyo imatsimikizira ngati fayiloyo ili ndi fungulo lofunikira, ndipo ngati sichoncho, limasunthira ku fayilo yotsatira.

Atalandira deta kuchokera kuzinthu zitatuzi, RS imapanga yankho. Mwachindunji, imatha kutumiza mitundu ingapo yachinthu nthawi imodzi ngati kasitomala apempha kuti asinthe.

4. Kusunga deta

Ma buffers a MemStore ndi BlockCache amakhala mpaka 80% ya kukumbukira kwapa mulu mu RS (zotsalazo zimasungidwa ntchito za RS). Ngati njira yogwiritsira ntchito ndiyomwe imalemba ndikuwerenga zomwezo nthawi yomweyo, ndizomveka kuchepetsa BlockCache ndikuwonjezera MemStore, popeza polemba deta, cache yowerengera siigunda, ndipo BlockCache idzagwiritsidwa ntchito kawirikawiri. Buffer ya BlockCache ili ndi magawo awiri: LruBlockCache (nthawi zonse pa-mulu) ndi BucketCache (nthawi zambiri imakhala yotsika kapena pa SSD). BucketCache iyenera kugwiritsidwa ntchito ngati pali zopempha zambiri zowerengedwa ndipo sizikugwirizana ndi LruBlockCache, zomwe zimatsogolera ku ntchito yosonkhanitsa zinyalala. Komabe, simuyenera kuyembekezera kukwera kwakukulu pogwiritsa ntchito cache yowerengera, koma tibwerera ku gawo 8.

Pali BlockCache imodzi ya RS yonse, ndipo tebulo lirilonse liri ndi MemStore yake (limodzi la Gulu lililonse la Banja).

Kodi anafotokoza Mwachidziwitso, zomwe zalembedwera ku cache sizimalembedwa, ndipo zowonadi, CACHE_DATA_ON_WRITE tebulo parameter ndi "Cache DATA on Write" RS parameter yasinthidwa kukhala zabodza. Komabe, m'malo mwake, ngati mulemba deta ku MemStore, kenako ndikuyiyika pa diski (potero kuichotsa), ndiyeno mufufuze fayilo yomwe mwapeza, mupezanso detayo mwakuchita pempho la GET. Komanso, ngakhale mutayimitsa BlockCache ndikudzaza tebulo ndi deta yatsopano, kenaka sungani MemStore ku disk, kuchotsani, ndikufunsanso pagawo lina, deta idzabwezeredwabe kwinakwake. Chifukwa chake, HBase imasunga osati zidziwitso zokha komanso zinsinsi zachinsinsi.

hbase(main):001:0> create 'ns:magic', 'cf'
Created table ns:magic
Took 1.1533 seconds
hbase(main):002:0> put 'ns:magic', 'key1', 'cf:c', 'try_to_delete_me'
Took 0.2610 seconds
hbase(main):003:0> flush 'ns:magic'
Took 0.6161 seconds
hdfs dfs -mv /data/hbase/data/ns/magic/* /tmp/trash
hbase(main):002:0> get 'ns:magic', 'key1'
 cf:c      timestamp=1534440690218, value=try_to_delete_me

Magawo a "Cache DATA on Read" asinthidwa kukhala zabodza. Ngati muli ndi malingaliro, chonde kambiranani nawo mu ndemanga.

5. Kusintha kwa data ya batch MultiGet/MultiPut

Kukonza zopempha zing'onozing'ono (Pezani / Ikani / Chotsani) ndizokwera mtengo kwambiri, kotero kuziphatikiza mu Mndandanda kapena Mndandanda ngati kuli kotheka kungathandize kwambiri ntchito. Izi ndizowona makamaka pazolemba, koma kuwerenga kumadzetsa msampha. Chithunzi chomwe chili pansipa chikuwonetsa nthawi yomwe imafunika kuti muwerenge zolemba za 50,000 kuchokera ku MemStore. Kuwerenga kunachitika mu ulusi umodzi, ndipo nsonga yopingasa imasonyeza chiwerengero cha makiyi mu pempho. Zikuwonekeratu kuti chiwerengero cha makiyi mu pempho limodzi chikuwonjezeka kufika pa chikwi, nthawi yakupha imachepa, kutanthauza kuti ntchito ikuwonjezeka. Komabe, ndi mawonekedwe a MSLAB omwe amathandizidwa mwachisawawa, ntchito imayamba kutsika kwambiri kupitirira malire awa, ndi kuchuluka kwa deta muzolemba, nthawi yayitali yopha.

Mayeserowa adachitidwa pamakina enieni, 8 cores, HBase version 2.0.0-cdh6.0.0-beta1.

MSLAB mode idapangidwa kuti ichepetse kugawika kwa milu chifukwa chosakanikirana ndi mibadwo yatsopano ndi yakale. Kuti athetse vutoli, MSLAB ikayatsidwa, deta imayikidwa m'magulu ang'onoang'ono ndikukonzedwa m'magawo. Zotsatira zake, kukula kwa paketi ya data yomwe yapemphedwa kupitilira kukula komwe kwaperekedwa, magwiridwe antchito amatsika kwambiri. Kuyimitsa njirayi, komabe, sikoyeneranso, chifukwa kudzatsogolera ku malo osungiramo GC panthawi yofikira kwambiri deta. Yankho labwino ndikuwonjezera kukula kwa chunk ngati zolemba zogwira ntchito zimayikidwa nthawi imodzi ndikuwerenga. Ndizofunikira kudziwa kuti vutoli silichitika ngati lamulo la flush lichitika pambuyo polemba, lomwe limatsitsa MemStore ku disk, kapena ngati kutsitsa kumachitidwa pogwiritsa ntchito BulkLoad. Gome ili pansipa likuwonetsa kuti kufunsira kwa data yayikulu (komanso yofananira) kuchokera ku MemStore kumabweretsa kuchepa. Komabe, kuchulukitsa chunksize kumabweretsanso nthawi yokonzekera bwino.

Kuphatikiza pa kuchuluka kwa chunksize, kugawa deta ndi dera, mwachitsanzo, kugawa tebulo, kumathandiza. Izi zikutanthauza kuti zopempha zocheperako zimatumizidwa kudera lililonse, ndipo ngati zikwanira mu selo, kuyankha kumakhalabe kwabwino.

6. Njira yogawa matebulo kukhala zigawo (kutaya)

Popeza HBase ndi sitolo yamtengo wapatali ndipo magawo amapangidwa ndi kiyi, ndikofunikira kugawa deta mofanana m'madera onse. Mwachitsanzo, kugawa tebulo lotere kukhala magawo atatu kungapangitse kuti deta igawidwe m'magawo atatu:

Izi nthawi zina zimatha kutsitsa pang'onopang'ono ngati zomwe zidzakwezedwa pambuyo pake, mwachitsanzo, zikhalidwe zazitali zomwe zimayamba ndi nambala yomweyo, monga:

1000001
1000002
...
1100003

Popeza makiyi amasungidwa ngati ma byte array, onse ayamba chimodzimodzi ndikukhala m'dera lomwelo #1, lomwe limasunga makiyi osiyanasiyana. Pali njira zingapo zogawanitsa:

HexStringSplit - Imatembenuza kiyi kukhala chingwe chokhala ndi encoding ya hexadecimal mu "00000000" => "FFFFFFFF" ndikuyika ziro kumanzere.

UniformSplit - Imatembenuza fungulo kukhala gulu la byte yokhala ndi ma encoding a hexadecimal mu "00" => "FF" ndikuyika ziro kumanja.

Kuphatikiza apo, mutha kufotokozera mtundu uliwonse kapena seti ya makiyi ogawa ndikusintha ma autosplitting. Komabe, imodzi mwa njira zosavuta komanso zothandiza kwambiri ndi UniformSplit komanso kugwiritsa ntchito hashi concatenation, mwachitsanzo, ma byte ofunikira kwambiri poyendetsa kiyi kudzera pa CRC32(rowkey) ntchito ndi rowkey yokha:

hash + rowkey

Kenako deta yonse idzagawidwa mofanana m'madera onse. Powerenga, ma byte awiri oyamba amangotayidwa, ndikusiya kiyi yoyambirira. RS imayang'aniranso kuchuluka kwa data ndi makiyi m'dera ndikuzigawa m'magawo ngati malire apitilira.

7. Kulekerera zolakwika ndi malo a data

Popeza kuti dera limodzi lokha liri ndi udindo pa makiyi aliwonse, njira yothetsera mavuto okhudzana ndi kuwonongeka kwa RS kapena kuchotsedwa ntchito ndikusunga zonse zofunika mu HDFS. RS ikagwa, mbuyeyo amazindikira izi chifukwa chosowa kugunda kwa mtima pa ZooKeeper node. Kenako imagawira dera lomwe likutumikira kwa RS ina. Popeza ma HFiles amasungidwa m'mafayilo ogawidwa, mbuye watsopanoyo amawawerenga ndikupitirizabe kutumiza deta. Komabe, popeza deta ina ikhoza kukhala mu MemStore ndipo sichinasamutsidwe ku HFiles, WAL, yomwe imasungidwa mu HDFS, imagwiritsidwa ntchito kubwezeretsa mbiri ya ntchito. Zosinthazo zitagwiritsidwa ntchito, RS imatha kuyankha zopempha, koma kusunthaku kumapangitsa kuti deta ina ndi njira zomwe zimawathandizira zikhale pamagulu osiyanasiyana, kuchepetsa malo.

Yankho lake ndikuphatikizana kwakukulu. Njirayi imasuntha mafayilo kumalo omwe amawatsogolera (kumene madera awo ali), zomwe zimapangitsa kuti pakhale kuwonjezeka kwakukulu kwa ma netiweki ndi ma disks panthawiyi. Komabe, kupeza deta mofulumira kwambiri pambuyo pake. Kuphatikiza apo, major_compaction amaphatikiza ma HFiles onse kukhala fayilo imodzi mkati mwachigawo ndikuyeretsa deta potengera makonzedwe a tebulo. Mwachitsanzo, mutha kufotokoza kuchuluka kwa mitundu yazinthu zomwe muyenera kusunga kapena moyo wonse womwe chinthucho chimachotsedwa.

Njirayi ikhoza kukhala ndi zotsatira zabwino pakuchita kwa HBase. Chithunzi chomwe chili pansipa chikuwonetsa momwe magwiridwe antchito adanyozeka chifukwa cha zolemba zolemetsa. Imawonetsa ulusi 40 ukulemba patebulo limodzi, ndi ulusi 40 nthawi imodzi ndikuwerenga deta. Ulusi wolembera umatulutsa ma HFiles ochulukirapo, omwe amawerengedwa ndi ulusi wina. Zotsatira zake, deta yowonjezereka iyenera kuchotsedwa pamtima, ndipo pamapeto pake, GC ikuyamba, kulepheretsa ntchito yonse. Kuthamanga kwakukulu kunachotsa zotsalira zomwe zatsala ndikubwezeretsa ntchito.

Mayesowa adayendetsedwa pa 3 DataNodes ndi 4 RSs (Xeon E5-2680 v4 CPU @ 2.40GHz * 64 ulusi). Mtundu wa HBase 1.2.0-cdh5.14.2

Ndizofunikira kudziwa kuti kuphatikiza kwakukulu kudayendetsedwa patebulo lamoyo, kulemba mwachangu ndikuwerenga deta. Panali zonena pa intaneti kuti izi zitha kubweretsa mayankho olakwika powerenga deta. Kuti atsimikizire izi, njira inayambika yomwe inapanga deta yatsopano ndikuyilemba patebulo. Nthawi yomweyo idawerenga ndikutsimikizira kuti mtengo wake umagwirizana ndi mtengo wolembedwa. Panthawi imeneyi, kugwirizanitsa kwakukulu kunayendetsedwa pafupifupi maulendo 200, ndipo palibe kulephera kamodzi komwe kunalembedwa. Ndizotheka kuti vutoli limapezeka kawirikawiri komanso panthawi ya katundu wambiri, choncho ndibwino kusiya nthawi zonse kulemba ndi kuwerenga ndi kuyeretsa kuti muteteze GC dips.

Komanso, compaction yayikulu sichikhudza dziko la MemStore; kuti muyigwetse ku disk ndi kuiphatikiza, muyenera kugwiritsa ntchito flush (connection.getAdmin().flush(TableName.valueOf(tblName))).

8. Zokonda ndi machitidwe

Monga tanena kale, HBase imakhala yopambana kwambiri ikapanda kuchita chilichonse - pochita BulkLoad. Izi zimagwira ntchito ku machitidwe ambiri ndi anthu, komabe. Komabe, chida ichi ndi choyenera kuyika deta yochuluka m'mabotolo akuluakulu, pamene ngati ndondomekoyo ikufuna kuwerengera ndi kulemba maulendo angapo nthawi imodzi, malamulo a Get ndi Put omwe afotokozedwa pamwambapa amagwiritsidwa ntchito. Kuti tidziwe magawo abwino, tidayesa mayeso ndi mitundu yosiyanasiyana yamitundu ndi zoikamo:

• Ulusi 10 unayambika nthawi imodzi katatu motsatizana (tiyeni titchule kuti ulusi).
• Nthawi yothamanga ya ulusi wonse mu block inali yowerengeka ndipo inali zotsatira zomaliza za ntchito ya block.
• Ulusi wonse unagwira ntchito ndi tebulo lomwelo.
• Asanayambe kuthamanga kulikonse kwa chipika cha ulusi, kugwirizanitsa kwakukulu kunkachitika.
• Chida chilichonse chimagwira ntchito imodzi yokha mwa izi:

- Ikani
—Pezani
- Pezani + Ikani

• Chida chilichonse chidachita kubwereza 50,000 kwa ntchito yake.
• Kukula kwa mbiri mu block ndi 100 byte, 1000 byte kapena 10000 byte (mwachisawawa).
• Ma blockwa adayambitsidwa ndi manambala osiyanasiyana a makiyi omwe adafunsidwa (kaya kiyi imodzi kapena 10).
• Ma midadada amayendetsedwa ndi makonda osiyanasiyana a tebulo. Ma parameter awa adasinthidwa:

- BlockCache = yathandizidwa kapena yolephereka
- BlockSize = 65 KB kapena 16 KB
- Gawo = 1, 5 kapena 30
- MSLAB = kuyatsa kapena kuzimitsa

Chifukwa chake block ikuwoneka motere:

a. MSLAB mode adayatsidwa/kuzimitsa.
b. Gome linapangidwira lomwe magawo otsatirawa adakhazikitsidwa: BlockCache = zoona / palibe, BlockSize = 65/16 Kb, Partitions = 1/5/30.
c. Kuponderezedwa kwa GZ kunakhazikitsidwa.
d. Ulusi 10 unayambika nthawi imodzi ndikuchita 1/10 put/get/get+ika ntchito patebuloli ndi zolemba za 100/1000/10000 byte, kumachita zopempha 50,000 motsatizana (makiyi osasintha).
e. Mfundo d inabwerezedwa katatu.
f. Nthawi yothamanga ya ulusi wonse inali yapakati.

Zosakaniza zonse zomwe zingatheke zinayesedwa. Zinali zodziwikiratu kuti kuwonjezera kukula kwa mbiriyo kungachepetse magwiridwe antchito, kapena kuti kuletsa caching kungayambitse kuchepa. Komabe, cholinga chake chinali kumvetsetsa mlingo ndi kufunikira kwa zotsatira za parameter iliyonse, kotero kuti deta yosonkhanitsidwa inadyetsedwa mu ntchito yowonongeka, yomwe imalola kuwunika kodalirika pogwiritsa ntchito t-statistics. Zotsatira za midadada yochita ntchito za Put zikuwonetsedwa pansipa. Zophatikizira zonse ndi 2 * 2 * 3 * 2 * 3 = 144 mitundu + 72 popeza ena adaphedwa kawiri. Chifukwa chake, okwana 216 amathamanga:

Kuyesa kunachitika pagulu laling'ono lokhala ndi ma DataNode atatu ndi ma RS anayi (Xeon E5-2680 v4 CPUs @ 2.40 GHz * 64 ulusi). Mtundu wa HBase 1.2.0-cdh5.14.2.

Kuthamanga kwapamwamba kwambiri kwa 3.7 sec kunapezedwa ndi MSLAB mode yolephereka, patebulo lomwe lili ndi gawo limodzi, BlockCache yathandizidwa, BlockSize = 16, yokhala ndi ma rekodi 100-byte mugulu la 10.
Kuthamanga kotsika kwambiri kwa 82.8 sec kunapezedwa ndi mawonekedwe a MSLAB, patebulo lomwe lili ndi gawo limodzi, BlockCache yathandizidwa, BlockSize = 16, yokhala ndi zolemba za 10000-byte, 1 iliyonse.

Tsopano tiyeni tione chitsanzo. Tikuwona zabwino zachitsanzo malinga ndi R2, koma zikuwonekeratu kuti extrapolation ndi contraindicated pano. Makhalidwe a dongosolo lenilenilo sadzakhala mzere pamene magawo akusintha; chitsanzo ichi ndi chofunika osati kulosera, koma kumvetsa zimene zinachitika mu magawo anapatsidwa. Mwachitsanzo, apa tikuwona kuchokera ku t-test ya Wophunzira kuti magawo a BlockSize ndi BlockCache alibe ntchito pa ntchito ya Put (yomwe nthawi zambiri imakhala yodziwikiratu):

Mfundo yakuti kuonjezera chiwerengero cha magawo kumabweretsa kuchepa kwa ntchito ndizosayembekezereka (tawona kale zotsatira zabwino zowonjezera chiwerengero cha magawo ndi BulkLoad), ngakhale ndizomveka. Choyamba, kukonza kumafuna kutulutsa mafunso ku zigawo 30 m'malo mwa chimodzi, ndipo kuchuluka kwa data sikokwanira kuti munthu apindule. Kachiwiri, nthawi yonse yophedwa imatsimikiziridwa ndi RS yochedwa kwambiri, ndipo popeza chiwerengero cha DataNodes ndi chochepa kuposa chiwerengero cha RSs, madera ena ali ndi malo a zero. Tiyeni tiwone zisanu zapamwamba:

Tsopano tiyeni tiwone zotsatira zakuchita Get blocks:

Kuchuluka kwa magawo sikunakhale kofunikira, mwina chifukwa deta imasungidwa bwino ndipo cache yowerengera ndiyofunikira kwambiri (mwachiwerengero). Mwachilengedwe, kuwonjezera kuchuluka kwa mauthenga pafunso lililonse kumapindulitsa kwambiri pakuchita. Zotsatira zabwino kwambiri:

Chabwino, potsiriza, tiyeni tiwone chitsanzo cha chipika chomwe chinayambika choyamba kupeza ndikuyika:

Ma parameters onse ndi ofunika apa. Ndipo zotsatira za atsogoleri:

9. Kuyesa kwa katundu

Pomaliza, tiyeni tigwire ntchito yabwino kapena yocheperako, koma nthawi zonse imakhala yosangalatsa mukakhala ndi chofananira. Webusaiti ya DataStax, wopanga wamkulu wa Cassandra, ali результаты NT pamakina angapo osungira a NoSQL, kuphatikiza mtundu wa HBase 0.98.6-1. Kutsitsa kunkachitika pogwiritsa ntchito ulusi wa 40, wokhala ndi kukula kwa data 100 byte, pogwiritsa ntchito ma SSD. Werengani-Sinthani-Lembani zotsatira za mayeso zawonetsa zotsatirazi.

Momwe ndikumvera, kuwerenga kunkachitika m'mabuku a zolemba za 100, ndi ma node a 16 HBase, kuyesa kwa DataStax kunasonyeza ntchito za 10 zikwi pa sekondi iliyonse.

Ndibwino kuti gulu lathu limakhalanso ndi mfundo za 16, koma osati mwamwayi kuti aliyense ali ndi 64 cores (ulusi), pamene mayesero a DataStax ali ndi 4. Komano, ali ndi ma drive a SSD, pamene tili ndi HDDs ndi mtundu watsopano wa HBase, kotero kugwiritsa ntchito CPU pansi pa katundu sikunayambe kuwonjezeka (mwachiwonekere ndi 5-10 peresenti). Komabe, tiyesa kukhazikitsa izi. Zokonda patebulo ndizosakhazikika: zowerengedwa zimachitidwa mwachisawawa pamakiyi kuchokera pa 0 mpaka 50 miliyoni (ie, zatsopano nthawi iliyonse). Gome ili ndi zolemba 50 miliyoni, zogawidwa m'magawo 64. Makiyi amathamangitsidwa pogwiritsa ntchito crc32. Zokonda patebulo ndizokhazikika, ndipo MSLAB imayatsidwa. Timatsegula ulusi 40, ulusi uliwonse ukuwerenga makiyi 100 osasintha ndipo nthawi yomweyo timalembanso ma byte 100 opangidwa ndi makiyi awa.

Imani: 16 DataNode ndi 16 RS (CPU Xeon E5-2680 v4 @ 2.40GHz * 64 ulusi). Mtundu wa HBase 1.2.0-cdh5.14.2.

Zotsatira zapakati zili pafupi ndi ntchito za 40,000 pamphindikati, zomwe ziri bwino kwambiri kusiyana ndi kuyesa kwa DataStax. Komabe, pazolinga zoyesera, mikhalidwe imatha kusinthidwa pang'ono. Ndizokayikitsa kuti ntchito zonse zizichitidwa ndi tebulo limodzi, komanso ndi makiyi apadera. Tiyerekeze kuti pali makiyi "otentha" omwe amapanga kuchuluka kwa katundu. Chifukwa chake, tiyesa kupanga katunduyo ndi zolemba zazikulu (10 KB), komanso m'magulu a 100, kukhala magome anayi osiyanasiyana, ndikuchepetsa makiyi omwe adafunsidwa kukhala 50,000. Chithunzi chomwe chili pansipa chikuwonetsa ulusi wa 40 ukuyenda, ulusi uliwonse ukuwerenga makiyi a 100 ndikulemba nthawi yomweyo 10 KB kubwerera ku makiyi amenewo.

Imani: 16 DataNode ndi 16 RS (CPU Xeon E5-2680 v4 @ 2.40GHz * 64 ulusi). Mtundu wa HBase 1.2.0-cdh5.14.2.

Panthawi yonyamula katundu, kuphatikizika kwakukulu kumayendetsedwa kangapo. Monga tawonera pamwambapa, popanda njirayi, magwiridwe antchito amatha kutsika pang'onopang'ono. Komabe, katundu wowonjezera adachitikanso panthawi ya kuphedwa. Kutsika kwapang'onopang'ono kudachitika chifukwa cha zinthu zosiyanasiyana. Nthawi zina ulusi udatha, kuchititsa kaye kaye pamene akuyambiranso, ndipo nthawi zina mapulogalamu a chipani chachitatu adapanga katundu pagulu.

Ntchito zowerengera ndi kulemba ndi imodzi mwazinthu zovuta kwambiri pa HBase. Pogwiritsa ntchito mafunso ang'onoang'ono, tinene kuti ma 100 mabayiti aliwonse, ophatikizidwa mumagulu 10-50, amatha kukwaniritsa ntchito masauzande pa sekondi iliyonse. Zomwezo zimagwiranso ntchito pamafunso owerengera okha. Ndizofunikira kudziwa kuti zotsatira zake ndizabwino kwambiri kuposa zomwe zidapezedwa ndi DataStax, makamaka chifukwa cha kukula kwafunso kwa 50-byte.

Imani: 16 DataNode ndi 16 RS (CPU Xeon E5-2680 v4 @ 2.40GHz * 64 ulusi). Mtundu wa HBase 1.2.0-cdh5.14.2.

10. Mapeto

Dongosololi ndi losavuta kusintha, koma zotsatira za magawo ambiri sizikudziwika. Ena a iwo adayesedwa koma sanaphatikizidwe mu mayeso omaliza. Mwachitsanzo, zoyeserera zoyambira zidawonetsa kuchepa pang'ono kwa gawo la DATA_BLOCK_ENCODING, lomwe limasindikiza zidziwitso pogwiritsa ntchito ma cell oyandikana nawo, zomwe zimamveka ndi data yopangidwa mwachisawawa. Pogwiritsa ntchito zinthu zambiri zobwerezabwereza, zopindula zingakhale zofunikira. Ponseponse, HBase ikuwoneka ngati nkhokwe yokhazikika komanso yopangidwa bwino yomwe ingakhale yothandiza kwambiri pogwira midadada yayikulu, makamaka ngati kuwerenga ndi kulemba kumatha kugawidwa.

Ngati mukuwona kuti pali chilichonse chomwe sindinafotokoze mokwanira, ndine wokondwa kufotokoza. Tikukupemphani kuti mufotokoze zomwe mwakumana nazo kapena kukambirana zomwe simukugwirizana nazo.

Source: www.habr.com