{"id":55302,"date":"2020-01-17T00:00:00","date_gmt":"2020-01-16T21:00:00","guid":{"rendered":"https:\/\/prohoster.info\/blog\/blog_prohoster\/teoriya-i-praktika-ispolzovaniya-hbase"},"modified":"2020-02-18T14:03:24","modified_gmt":"2020-02-18T11:03:24","slug":"teoriya-i-praktika-ispolzovaniya-hbase","status":"publish","type":"post","link":"https:\/\/prohoster.info\/sq\/blog\/administrirovanie\/teoriya-i-praktika-ispolzovaniya-hbase","title":{"rendered":"Teoria dhe praktika e p\u00ebrdorimit t\u00eb HBase","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

P\u00ebrsh\u00ebndetje! Un\u00eb quhem Danil Lipovoy, ekipi yn\u00eb n\u00eb Sbertech ka filluar t\u00eb p\u00ebrdor\u00eb HBase si nj\u00eb depo p\u00ebr t\u00eb dh\u00ebna operacionale. Gjat\u00eb studimit t\u00eb tij, \u00ebsht\u00eb grumbulluar p\u00ebrvoj\u00eb q\u00eb d\u00ebshironim ta sistematizojm\u00eb dhe ta p\u00ebrshkruajm\u00eb (shpresojm\u00eb se do t\u00eb jet\u00eb e dobishme p\u00ebr shum\u00eb). Eksperimentet e m\u00ebposhtme jan\u00eb kryer me versionet HBase 1.2.0-cdh5.14.2 dhe 2.0.0-cdh6.0.0-beta1. <\/p>\n

    \n
  1. Arkitektura e p\u00ebrgjithshme <\/li>\n
  2. Shkruarja e t\u00eb dh\u00ebnave n\u00eb HBASE<\/li>\n
  3. Leximi i t\u00eb dh\u00ebnave nga HBASE<\/li>\n
  4. Caching i t\u00eb dh\u00ebnave<\/li>\n
  5. Procesimi me grumbuj MultiGet\/MultiPut<\/li>\n
  6. Strategjia e ndarjes s\u00eb tabelave n\u00eb zona (splitting)<\/li>\n
  7. Q\u00ebndrueshm\u00ebria, kompresimi dhe lokaliteti i t\u00eb dh\u00ebnave<\/li>\n
  8. Cil\u00ebsimet dhe performanca<\/li>\n
  9. Testimi i ngarkes\u00ebs<\/li>\n
  10. P\u00ebrfundimet<\/li>\n<\/ol>\n

    <\/a><\/noindex><\/p>\n

    1. Arkitektura e p\u00ebrgjithshme<\/h2>\n

    \n\"Teoria
    \nMaster-i rezerv\u00eb d\u00ebgjon heartbeat-in e aktiv\u00ebve n\u00eb nodin ZooKeeper dhe n\u00eb rast se ai zhduket, merr funksionet e master-it p\u00ebr vete. <\/p>\n

    2. Shkruarja e t\u00eb dh\u00ebnave n\u00eb HBASE<\/h2>\n

    \nFillimisht, le t\u00eb shqyrtojm\u00eb rastin m\u00eb t\u00eb thjesht\u00eb \u2013 shkruarja e nj\u00eb objekti \u00e7el\u00ebs-vler\u00eb n\u00eb nj\u00eb tabel\u00eb p\u00ebrmes put(rowkey). Klienti fillimisht duhet t\u00eb zbuloj\u00eb ku ndodhet serveri rr\u00ebnj\u00ebsor (Root Region Server \u2013 RRS) q\u00eb ruan tabel\u00ebn hbase:meta. Informacionin e k\u00ebsaj e merr nga ZooKeeper. Pas k\u00ebsaj, ai i drejtohet RRS dhe lexon tabel\u00ebn hbase:meta, nga e cila nxjerr informacionin se cili RegionServer (RS) \u00ebsht\u00eb p\u00ebrgjegj\u00ebs p\u00ebr ruajtjen e t\u00eb dh\u00ebnave sipas \u00e7el\u00ebsit t\u00eb rreshtit rowkey n\u00eb tabel\u00ebn q\u00eb e intereson. P\u00ebr p\u00ebrdorim t\u00eb m\u00ebtejsh\u00ebm, tabela meta ruhen n\u00eb memorie nga klienti dhe k\u00ebshtu, k\u00ebrkesat e m\u00ebvonshme shkojn\u00eb m\u00eb shpejt, direkt te RS.<\/p>\n

    M\u00eb pas, RS, pasi merr k\u00ebrkes\u00ebn, s\u00eb pari e shkruan at\u00eb n\u00eb WriteAheadLog (WAL), q\u00eb \u00ebsht\u00eb e nevojshme p\u00ebr rikuperim n\u00eb rast r\u00ebnies. Pastaj ruan t\u00eb dh\u00ebnat n\u00eb MemStore. Kjo \u00ebsht\u00eb nj\u00eb tampon n\u00eb memorie, i cili mban nj\u00eb grup t\u00eb rregullt \u00e7elesh t\u00eb k\u00ebtij rajoni. Tabela mund t\u00eb ndahet n\u00eb rajone (particione), secili prej t\u00eb cil\u00ebve mban nj\u00eb grup t\u00eb pacaktuar \u00e7elesh. Kjo lejon q\u00eb, duke vendosur rajonet n\u00eb server\u00eb t\u00eb ndrysh\u00ebm, t\u00eb arrijm\u00eb nj\u00eb performanc\u00eb m\u00eb t\u00eb lart\u00eb. Megjithat\u00eb, pavar\u00ebsisht nga evidenca e k\u00ebsaj d\u00ebshmie, m\u00eb von\u00eb do t\u00eb shohim se kjo nuk funksionon n\u00eb t\u00eb gjitha rastet.<\/p>\n

    Pasi t\u00eb vendoset regjistrimi n\u00eb MemStore, klientit i kthehet nj\u00eb p\u00ebrgjigje q\u00eb regjistrimi \u00ebsht\u00eb ruajtur me sukses. Megjithat\u00eb, n\u00eb realitet ai ruhet vet\u00ebm n\u00eb tampon dhe do t\u00eb kaloj\u00eb n\u00eb disk vet\u00ebm pasi t\u00eb kaloj\u00eb nj\u00eb interval t\u00eb caktuar kohor ose n\u00eb rast t\u00eb mbushjes s\u00eb tij me t\u00eb dh\u00ebna t\u00eb reja. <\/p>\n

    \"Teoria
    \nGjat\u00eb ekzekutimit t\u00eb operacionit \"Fshi\" nuk ndodh nj\u00eb fshirje fizike e t\u00eb dh\u00ebnave. Ato thjesht sh\u00ebnohen si t\u00eb fshira, dhe shkat\u00ebrrimi efektiv ndodh n\u00eb momentin kur thirret funksioni major compact, p\u00ebr t\u00eb cilin \u00ebsht\u00eb shkruar m\u00eb shum\u00eb n\u00eb p.7.<\/p>\n

    Skedar\u00ebt n\u00eb formatin HFile grumbullohen n\u00eb HDFS dhe her\u00eb pas here startohet procesi minor compact, i cili thjesht bashkon skedar\u00ebt e vegj\u00ebl n\u00eb m\u00eb t\u00eb m\u00ebdhenj, pa hequr asgj\u00eb. Me kalimin e koh\u00ebs, kjo kthehet n\u00eb nj\u00eb problem q\u00eb shfaqet vet\u00ebm gjat\u00eb leximit t\u00eb t\u00eb dh\u00ebnave (n\u00eb k\u00ebt\u00eb do t\u00eb kthehemi m\u00eb von\u00eb). <\/p>\n

    P\u00ebrve\u00e7 procesit t\u00eb ngarkimit t\u00eb p\u00ebrshkruar m\u00eb sip\u00ebr, ekziston nj\u00eb procedur\u00eb shum\u00eb m\u00eb efikase, q\u00eb p\u00ebrb\u00ebn ndoshta pik\u00ebn m\u00eb t\u00eb fort\u00eb t\u00eb k\u00ebtij DB \u2013 BulkLoad. Ajo p\u00ebrfshin faktin q\u00eb ne formojm\u00eb vet\u00eb HFiles dhe i vendosim n\u00eb disk, e cila na lejon t\u00eb shkallojm\u00eb n\u00eb m\u00ebnyr\u00eb t\u00eb shk\u00eblqyer dhe t\u00eb arrijm\u00eb shpejt\u00ebsi t\u00eb k\u00ebnaqshme. N\u00eb thelb, kufizimi k\u00ebtu nuk \u00ebsht\u00eb HBase, por kapaciteti i harduerit. M\u00eb posht\u00eb jan\u00eb rezultatet e ngarkimit n\u00eb nj\u00eb klas\u00eb q\u00eb p\u00ebrb\u00ebhet nga 16 RegionServers dhe 16 NodeManager YARN (CPU Xeon E5-2680 v4 @ 2.40GHz * 64 d\u00ebrrasa), versioni HBase 1.2.0-cdh5.14.2. <\/p>\n

    \"Teoria
    \n
    \nK\u00ebtu shihet se duke rritur numrin e particioneve (rajoneve) n\u00eb tabel\u00eb, si dhe ekzekutor\u00ebve Spark, merrni nj\u00eb rritje t\u00eb shpejt\u00ebsis\u00eb s\u00eb ngarkimit. Gjithashtu, shpejt\u00ebsia varet nga sasia e shkrimit. Bloket e m\u00ebdha japin nj\u00eb rritje n\u00eb matjen MB\/s, nd\u00ebrsa ato t\u00eb vogla n\u00eb numrin e regjistrimeve t\u00eb futur n\u00eb nj\u00ebsi kohe, n\u00eb kushte t\u00eb tjera t\u00eb barabarta. <\/p>\n

    Gjithashtu, mund t\u00eb nisni ngarkimin n\u00eb dy tabela nj\u00ebkoh\u00ebsisht dhe t\u00eb merrni dyfishim t\u00eb shpejt\u00ebsis\u00eb. M\u00eb posht\u00eb shihet se shkrimi i blokut 10 KB menj\u00ebher\u00eb n\u00eb dy tabela po ndodh me nj\u00eb shpejt\u00ebsi prej rreth 600 MB\/s n\u00eb secil\u00ebn (n\u00eb total 1275 MB\/s), q\u00eb p\u00ebrputhet me shpejt\u00ebsin\u00eb e shkrimit n\u00eb nj\u00eb tabel\u00eb 623 MB\/s (shih. nr. 11 m\u00eb sip\u00ebr).<\/p>\n

    \"Teoria
    \nNd\u00ebrsa, ekzekutimi i dyt\u00eb me shkrime n\u00eb 50 KB tregon se shpejt\u00ebsia e ngarkimit po rritet vet\u00ebm pak, q\u00eb tregon se po afrohemi pran\u00eb vlerave maksimale. Gjithashtu, duhet t\u00eb kemi parasysh se HBASE nuk ka ngarkes\u00eb t\u00eb madhe k\u00ebtu, gjith\u00e7ka q\u00eb k\u00ebrkohet prej tij \u00ebsht\u00eb fillimisht t\u00eb jap\u00eb t\u00eb dh\u00ebnat nga hbase:meta, e m\u00eb pas, pas vendosjes s\u00eb HFiles, t\u00eb shlyej\u00eb t\u00eb dh\u00ebnat BlockCache dhe t\u00eb ruaj\u00eb tamponin MemStore n\u00eb disk, n\u00ebse ai nuk \u00ebsht\u00eb bosh.<\/p>\n

    3. Leximi i t\u00eb dh\u00ebnave nga HBASE<\/h2>\n

    \nN\u00ebse merret parasysh se t\u00eb gjitha informacionet nga hbase:meta tashm\u00eb i ka klienti (shiko p.2), at\u00ebher\u00eb k\u00ebrkesa shkon menj\u00ebher\u00eb n\u00eb at\u00eb RS ku ruhet \u00e7el\u00ebsi i nevojsh\u00ebm. Fillimisht, k\u00ebrkimi b\u00ebhet n\u00eb MemCache. Pavar\u00ebsisht n\u00ebse ka ose jo t\u00eb dh\u00ebna atje, k\u00ebrkimi kryhet gjithashtu n\u00eb tamponin BlockCache dhe, sipas nevoj\u00ebs, n\u00eb HFiles. N\u00ebse t\u00eb dh\u00ebnat gjenden n\u00eb skedarin, ato vendosen n\u00eb BlockCache dhe n\u00eb k\u00ebrkes\u00ebn e ardhshme do t\u00eb kthehen m\u00eb shpejt. K\u00ebrkimi n\u00eb HFile ndodh mjaft shpejt fal\u00eb p\u00ebrdorimit t\u00eb filtrit Bloom, dmth, pasi lexon nj\u00eb sasi t\u00eb vog\u00ebl t\u00eb dh\u00ebnash, ai menj\u00ebher\u00eb p\u00ebrcakton n\u00ebse ky skedar ka \u00e7el\u00ebsin e nevojsh\u00ebm dhe n\u00ebse jo, kalon te i ardhshmi.<\/p>\n

    \"Teoria
    \nDuke marr\u00eb t\u00eb dh\u00ebnat nga k\u00ebto tri burime, RS formon p\u00ebrgjigjen. N\u00eb ve\u00e7anti, ai mund t\u00eb d\u00ebrgoj\u00eb menj\u00ebher\u00eb disa versione t\u00eb gjetura t\u00eb objektit n\u00ebse klienti ka k\u00ebrkuar versionim.<\/p>\n

    4. Ruajtja e t\u00eb dh\u00ebnave n\u00eb cache<\/h2>\n

    \nTampon\u00ebt MemStore dhe BlockCache z\u00ebn\u00eb deri n\u00eb 80% t\u00eb memories on-heap t\u00eb RS (t\u00eb tjerat jan\u00eb rezervuar p\u00ebr detyrat sh\u00ebrbyese t\u00eb RS). N\u00ebse p\u00ebrdorimi tipik \u00ebsht\u00eb se proceset shkruajn\u00eb dhe menj\u00ebher\u00eb lexojn\u00eb k\u00ebto t\u00eb dh\u00ebna, ka kuptim t\u00eb zvog\u00eblohet BlockCache dhe t\u00eb rritet MemStore, pasi kur shkruhen t\u00eb dh\u00ebnat n\u00eb cache ato nuk hyn\u00eb n\u00eb memorjen p\u00ebr lexim, k\u00ebshtu q\u00eb p\u00ebrdorimi i BlockCache do t\u00eb ndodh\u00eb m\u00eb rrall\u00eb. Tamponi BlockCache p\u00ebrb\u00ebhet nga dy pjes\u00eb: LruBlockCache (p\u00ebrher\u00eb on-heap) dhe BucketCache (zakonisht off-heap ose n\u00eb SSD). BucketCache duhet p\u00ebrdorur kur ka shum\u00eb k\u00ebrkesa p\u00ebr lexim dhe ato nuk p\u00ebrshtaten n\u00eb LruBlockCache, q\u00eb \u00e7on n\u00eb pun\u00ebn aktive t\u00eb Garbage Collector. Megjithat\u00eb, nuk duhet pritur nj\u00eb rritje radikale t\u00eb performanc\u00ebs nga p\u00ebrdorimi i caches p\u00ebr lexim, p\u00ebr k\u00ebt\u00eb do t\u00eb rikthehemi n\u00eb p. 8.<\/p>\n

    \"Teoria
    \nBlockCache \u00ebsht\u00eb nj\u00eb p\u00ebr t\u00eb gjith\u00eb RS, nd\u00ebrsa MemStore \u00ebsht\u00eb specifik p\u00ebr secil\u00ebn tabel\u00eb (nj\u00eb p\u00ebr \u00e7do Column Family).<\/p>\n

    Si jan\u00eb p\u00ebrshkruar<\/a><\/noindex> N\u00eb teori, kur shkruhen t\u00eb dh\u00ebnat n\u00eb cache ato nuk hyn\u00eb dhe n\u00eb t\u00eb v\u00ebrtet\u00eb, parametrat CACHE_DATA_ON_WRITE p\u00ebr tabel\u00ebn dhe \"Cache DATA on Write\" p\u00ebr RS jan\u00eb vendosur n\u00eb false. Megjithat\u00eb, n\u00eb praktik\u00eb, n\u00ebse shkruhen t\u00eb dh\u00ebna n\u00eb MemStore, m\u00eb pas ato hidhen n\u00eb disk (duke pastruar k\u00ebshtu), duke fshir\u00eb skedarin e krijuar, at\u00ebher\u00eb duke kryer nj\u00eb k\u00ebrkes\u00eb get, ne do t\u00eb marrim me sukses t\u00eb dh\u00ebnat. P\u00ebr m\u00eb tep\u00ebr, madje edhe n\u00ebse plot\u00ebsisht \u00e7aktivizohet BlockCache dhe tabel\u00ebn e mbushim me t\u00eb dh\u00ebna t\u00eb reja, pastaj arrijm\u00eb t\u00eb hedhim MemStore n\u00eb disk, t'i fshijm\u00eb ato dhe t\u00eb k\u00ebrkojm\u00eb nga nj\u00eb seanc\u00eb tjet\u00ebr, ato p\u00ebrs\u00ebri do t\u00eb nxirren nga ndonj\u00eb vend. Pra, HBase ruan jo vet\u00ebm t\u00eb dh\u00ebna, por edhe mistere t\u00eb \u00e7uditshme.<\/p>\n

    hbase(main):001:0> create 'ns:magic', 'cf'\nCreated table ns:magic\nTook 1.1533 seconds\nhbase(main):002:0> put 'ns:magic', 'key1', 'cf:c', 'try_to_delete_me'\nTook 0.2610 seconds\nhbase(main):003:0> flush 'ns:magic'\nTook 0.6161 seconds\nhdfs dfs -mv \/data\/hbase\/data\/ns\/magic\/* \/tmp\/trash\nhbase(main):002:0> get 'ns:magic', 'key1'\n cf:c      timestamp=1534440690218, value=try_to_delete_me\n<\/code><\/pre>\n
    \nParametri \u00abCache DATA on Read\u00bb \u00ebsht\u00eb vendosur false. N\u00ebse keni ide, jeni t\u00eb mir\u00ebpritur t\u00eb diskutoni k\u00ebt\u00eb n\u00eb komentet.<\/p>\n
    5. Procesimi i t\u00eb dh\u00ebnave MultiGet\/MultiPut<\/h2>\n
    \nProcesi i k\u00ebrkesave t\u00eb vetme (Get\/Put\/Delete) \u00ebsht\u00eb nj\u00eb operacion mjaft i kushtuesh\u00ebm, prandaj \u00ebsht\u00eb e rekomandueshme t'i gruponi ata kur \u00ebsht\u00eb e mundur n\u00eb List ose List, q\u00eb lejon nj\u00eb rritje t\u00eb konsiderueshme n\u00eb performanc\u00eb. Kjo \u00ebsht\u00eb ve\u00e7an\u00ebrisht e r\u00ebnd\u00ebsishme p\u00ebr operacionin e shkruar, nd\u00ebrsa gjat\u00eb leximit ka nj\u00eb problem tjet\u00ebr. N\u00eb grafikun m\u00eb posht\u00eb \u00ebsht\u00eb treguar koha e leximit t\u00eb 50,000 regjistrave nga MemStore. Leximi \u00ebsht\u00eb kryer n\u00eb nj\u00eb rrjedh\u00eb dhe n\u00eb aksin horizontal \u00ebsht\u00eb numri i \u00e7el\u00ebseve n\u00eb k\u00ebrkes\u00eb. K\u00ebtu shihet se kur numri rritet deri n\u00eb nj\u00eb mij\u00eb \u00e7el\u00ebsa n\u00eb nj\u00eb k\u00ebrkes\u00eb, koha e ekzekutimit bie, dometh\u00ebn\u00eb shpejt\u00ebsia rritet. Megjithat\u00eb, n\u00eb modalitetin MSLAB t\u00eb aktivizuar n\u00eb m\u00ebnyr\u00eb p\u00ebrDefault, pas k\u00ebtij kufiri fillon nj\u00eb r\u00ebnie drastike e performanc\u00ebs, dhe sa m\u00eb shum\u00eb t\u00eb dh\u00ebna t\u00eb jen\u00eb n\u00eb regjistrim, aq m\u00eb e madhe \u00ebsht\u00eb koha e pun\u00ebs. <\/p>\n
    \"Teoria
    \n
    \nTestet jan\u00eb kryer n\u00eb nj\u00eb virtual, 8 b\u00ebrthama, versione HBase 2.0.0-cdh6.0.0-beta1.<\/p>\n
    Modaliteti MSLAB \u00ebsht\u00eb krijuar p\u00ebr t\u00eb reduktuar fragmentimin e heap-it, i cili ndodh p\u00ebr shkak t\u00eb p\u00ebrzierjes s\u00eb t\u00eb dh\u00ebnave t\u00eb brezit t\u00eb ri dhe t\u00eb vjet\u00ebr. Si zgjidhje e problematik\u00ebs, me aktivizimin e MSLAB, t\u00eb dh\u00ebnat vendosen n\u00eb qeliza relativisht t\u00eb vogla (chunk) dhe trajtohen n\u00eb grupe. Si rezultat, kur v\u00ebllimi n\u00eb paket\u00ebn e k\u00ebrkuar t\u00eb t\u00eb dh\u00ebnave tejkalon p\u00ebrmas\u00ebn e caktuar, performanca bie drastikisht. Nga ana tjet\u00ebr, \u00e7aktivizimi i k\u00ebtij modaliteti gjithashtu nuk \u00ebsht\u00eb i d\u00ebshiruesh\u00ebm, pasi do t\u00eb \u00e7oj\u00eb n\u00eb ndalime p\u00ebr shkak t\u00eb GC gjat\u00eb periudhave t\u00eb pun\u00ebs intensive me t\u00eb dh\u00ebna. Nj\u00eb zgjidhje e mir\u00eb \u00ebsht\u00eb rritja e p\u00ebrmas\u00ebs s\u00eb qeliz\u00ebs, n\u00eb rastin e shkrimit aktiv p\u00ebrmes put n\u00eb t\u00eb nj\u00ebjt\u00ebn koh\u00eb me leximin. Duhet theksuar se problemi nuk ndodh n\u00ebse pas regjistrimit ekzekutohet komanda flush q\u00eb shkarkon MemStore n\u00eb disk ose n\u00ebse b\u00ebhet ngarkimi p\u00ebrmes BulkLoad. N\u00eb tabel\u00ebn m\u00eb posht\u00eb tregohet se k\u00ebrkesat nga MemStore p\u00ebr t\u00eb dh\u00ebna m\u00eb t\u00eb m\u00ebdha (dhe me numra t\u00eb nj\u00ebjt\u00eb) \u00e7ojn\u00eb n\u00eb ngadal\u00ebsim. Megjithat\u00eb, duke rritur chunksize, e kthejm\u00eb koh\u00ebn e p\u00ebrpunimit n\u00eb normalitet.<\/p>\n
    \"Teoria
    \nP\u00ebrve\u00e7 rritjes s\u00eb chunksize, ndihmon ndarja e t\u00eb dh\u00ebnave sipas rajoneve, pra ndarja e tabelave. Kjo \u00e7on n\u00eb faktin se p\u00ebr \u00e7do rajon vijn\u00eb m\u00eb pak k\u00ebrkesa dhe n\u00ebse ato vendosen n\u00eb nj\u00eb cel duke mbetur nj\u00eb p\u00ebrgjigje e mir\u00eb.<\/p>\n
    6. Strategjia e ndarjes s\u00eb tabelave n\u00eb rajone (spiliting)<\/h2>\n
    \nDuke qen\u00eb se HBase \u00ebsht\u00eb nj\u00eb magazin\u00eb me \u00e7el\u00ebs dhe vler\u00eb dhe ndarja b\u00ebhet sipas \u00e7el\u00ebsit, \u00ebsht\u00eb shum\u00eb e r\u00ebnd\u00ebsishme t\u00eb ndahet e dh\u00ebnat nj\u00eblloj n\u00eb t\u00eb gjitha rajonet. P\u00ebr shembull, ndarja e k\u00ebsaj tabele n\u00eb tri pjes\u00eb do t\u00eb \u00e7onte n\u00eb ndarjen e t\u00eb dh\u00ebnave n\u00eb tre rajone:<\/p>\n
    \"Teoria
    \nNdonj\u00ebher\u00eb, kjo \u00e7on n\u00eb nj\u00eb ngadal\u00ebsim t\u00eb papritur, n\u00ebse t\u00eb dh\u00ebnat e ngarkuara m\u00eb von\u00eb do t\u00eb ken\u00eb form\u00ebn, p\u00ebr shembull, t\u00eb vlerave t\u00eb gjata, shum\u00eb prej t\u00eb cilave fillojn\u00eb me t\u00eb nj\u00ebjt\u00ebn shif\u00ebr, p.sh.:<\/p>\n
    1000001
    \n1000002
    \n\u2026
    \n1100003<\/p>\n
    Duke qen\u00eb se \u00e7el\u00ebsit ruhen si nj\u00eb array bytes, t\u00eb gjith\u00eb do t\u00eb fillojn\u00eb nj\u00eblloj dhe do t\u00eb p\u00ebrkasin nj\u00eb rajoni #1 q\u00eb mban k\u00ebt\u00eb gam\u00eb \u00e7el\u00ebsish. Ekzistojn\u00eb disa strategji ndarjeje:<\/p>\n
    HexStringSplit \u2013 Kthen \u00e7el\u00ebsin n\u00eb nj\u00eb varg me kodimin hexadecimal n\u00eb gam\u00ebn \"00000000\" => \"FFFFFFFF\" dhe e mbush at\u00eb me zero nga ana e majt\u00eb.<\/p>\n
    UniformSplit \u2013 Kthen \u00e7el\u00ebsin n\u00eb nj\u00eb array bytes me kodimin hexadecimal n\u00eb gam\u00ebn \"00\" => \"FF\" dhe e mbush at\u00eb me zero nga ana e djatht\u00eb.<\/p>\n
    P\u00ebr m\u00eb tep\u00ebr, \u00ebsht\u00eb e mundur t\u00eb specifikohet ndonj\u00eb gam\u00eb ose grup \u00e7el\u00ebsish p\u00ebr ndarje dhe t\u00eb konfigurohet autospiliting. Megjithat\u00eb, nj\u00eb nga qasjet m\u00eb t\u00eb thjeshta dhe m\u00eb efektive \u00ebsht\u00eb UniformSplit dhe p\u00ebrdorimi i konkatenimit t\u00eb hashit, p\u00ebr shembull, dy bajt\u00ebt m\u00eb t\u00eb lart\u00eb nga kalimi i \u00e7el\u00ebsit p\u00ebrmes funksionit CRC32(rowkey) dhe vet\u00eb rowkey:<\/p>\n
    hash + rowkey<\/p>\n
    K\u00ebshtu, t\u00eb gjitha t\u00eb dh\u00ebnat do t\u00eb shp\u00ebrndahen nj\u00eblloj n\u00eb rajone. Kur lexohet, dy bajt\u00ebt e par\u00eb thjesht hiqen dhe mbetet \u00e7el\u00ebsi origjinal. Gjithashtu, RS kontrollon sasin\u00eb e t\u00eb dh\u00ebnave dhe \u00e7el\u00ebsave n\u00eb rajon dhe kur tejkalohen limitet automatizmi e ndan at\u00eb n\u00eb pjes\u00eb. <\/p>\n
    7. Q\u00ebndrueshm\u00ebria dhe lokalizimi i t\u00eb dh\u00ebnave<\/h2>\n
    \nP\u00ebrderisa \u00e7do grup \u00e7el\u00ebsash \u00ebsht\u00eb p\u00ebrgjegj\u00ebs vet\u00ebm p\u00ebr nj\u00eb rajon, zgjidhja p\u00ebr problemet q\u00eb lidhen me r\u00ebniet e RS ose disfatimet \u00ebsht\u00eb ruajtja e t\u00eb dh\u00ebnave t\u00eb nevojshme n\u00eb HDFS. Kur ndodh r\u00ebnia e RS, master-i e zb discovers k\u00ebt\u00eb p\u00ebrmes munges\u00ebs s\u00eb heartbeat n\u00eb nodin ZooKeeper. At\u00ebher\u00eb ai em\u00ebron nj\u00eb rajon t\u00eb sh\u00ebrbyer tjet\u00ebr RS dhe, p\u00ebr shkak se HFiles ruhen n\u00eb sistemin e dosjeve t\u00eb shp\u00ebrndara, pronari i ri i lexon ato dhe vazhdon t\u00eb sh\u00ebrbej\u00eb t\u00eb dh\u00ebnat. Megjithat\u00eb, pasi nj\u00eb pjes\u00eb e t\u00eb dh\u00ebnave mund t\u00eb jet\u00eb n\u00eb MemStore dhe nuk ka arritur t\u00eb kaloj\u00eb n\u00eb HFiles, p\u00ebr t\u00eb rikuperuar historin\u00eb e operacioneve p\u00ebrdoret WAL, e cila gjithashtu ruhet n\u00eb HDFS. Pas aplikimit t\u00eb ndryshimeve, RS \u00ebsht\u00eb n\u00eb gjendje t\u00eb p\u00ebrgjigjet n\u00eb k\u00ebrkesa, megjithat\u00eb kalimi \u00e7on n\u00eb at\u00eb q\u00eb nj\u00eb pjes\u00eb e t\u00eb dh\u00ebnave dhe proceset q\u00eb i sh\u00ebrbejn\u00eb ato p\u00ebrfundimisht ndodhen n\u00eb nod\u00eb t\u00eb ndryshme, dmth. lokaliteti zvog\u00eblohet. <\/p>\n
    Zgjidhja p\u00ebr problemin \u00ebsht\u00eb major compaction \u2013 kjo procedur\u00eb transferon skedar\u00ebt n\u00eb ato nod\u00eb q\u00eb jan\u00eb p\u00ebrgjegj\u00ebse p\u00ebr to (aty ku jan\u00eb vendosur rajonet e tyre), si rezultat, gjat\u00eb k\u00ebsaj procedure ngarkesa n\u00eb rrjet dhe disqet rritet ndjesh\u00ebm. Megjithat\u00eb, n\u00eb t\u00eb ardhmen, qasja n\u00eb t\u00eb dh\u00ebna thelb\u00ebsisht shpejtohet. P\u00ebr m\u00eb tep\u00ebr, major_compaction kryen bashkimin e t\u00eb gjitha HFiles n\u00eb nj\u00eb skedar t\u00eb vet\u00ebm brenda rajonit, si dhe pastron t\u00eb dh\u00ebnat n\u00eb p\u00ebrputhje me cil\u00ebsimet e tavolines. P\u00ebr shembull, mund t\u00eb p\u00ebrcaktohet numri i versioneve t\u00eb objektit q\u00eb duhet t\u00eb ruhen ose koha e jet\u00ebs s\u00eb tij, pas skadimit t\u00eb s\u00eb cil\u00ebs objekti fshihet fizikisht.<\/p>\n
    Kjo procedur\u00eb mund t\u00eb ket\u00eb nj\u00eb ndikim t\u00eb r\u00ebnd\u00ebsish\u00ebm pozitiv n\u00eb funksionimin e HBase. N\u00eb figur\u00ebn m\u00eb posht\u00eb tregohet se si ka degraduar performanca si rezultat i shkruarjes aktive t\u00eb t\u00eb dh\u00ebnave. K\u00ebtu tregohet si n\u00eb nj\u00eb tavolin\u00eb 40 rrjedha shkruanin dhe 40 rrjedha lexonin t\u00eb dh\u00ebna n\u00eb t\u00eb nj\u00ebjt\u00ebn koh\u00eb. Rrjedhat e shkruara formojn\u00eb gjithnj\u00eb e m\u00eb shum\u00eb HFiles, t\u00eb cilat lexohen nga rrjedhat e tjera. Si rezultat, gjithnj\u00eb e m\u00eb shum\u00eb t\u00eb dh\u00ebna duhet t\u00eb fshihen nga memoria dhe n\u00eb fund fillon t\u00eb punoj\u00eb GC, i cili praktikisht paralizon t\u00eb gjith\u00eb pun\u00ebn. Aktivizimi i major compaction \u00e7oi n\u00eb pastrimin e grumbujve t\u00eb krijuar dhe rikthimin e performanc\u00ebs.<\/p>\n
    \"Teoria
    \nTesti u krye n\u00eb 3 DataNode dhe 4 RS (CPU Xeon E5-2680 v4 @ 2.40GHz * 64 rrjedha). Versioni i HBase \u00ebsht\u00eb 1.2.0-cdh5.14.2<\/p>\n
    \u00cbsht\u00eb e r\u00ebnd\u00ebsishme t\u00eb theksohet se lan\u00e7imi i major compaction u realizua n\u00eb nj\u00eb tabel\u00eb \"t\u00eb gjall\u00eb\", n\u00eb t\u00eb cil\u00ebn po shkruheshin dhe lexoheshin t\u00eb dh\u00ebna aktivisht. N\u00eb rrjet \u00ebsht\u00eb th\u00ebn\u00eb se kjo mund t\u00eb \u00e7oj\u00eb n\u00eb p\u00ebrgjigje t\u00eb pasakt\u00eb gjat\u00eb leximit t\u00eb t\u00eb dh\u00ebnave. P\u00ebr t\u00eb verifikuar, u lan\u00e7ua nj\u00eb proces q\u00eb gjeneronte t\u00eb dh\u00ebna t\u00eb reja dhe i shkruante ato n\u00eb tabel\u00eb. Pas k\u00ebsaj, menj\u00ebher\u00eb lexohej dhe verifikohej n\u00ebse vlera e marr\u00eb p\u00ebrputhej me at\u00eb q\u00eb ishte shkruar. Gjat\u00eb funksionimit t\u00eb k\u00ebtij procesi, major compaction u lan\u00e7ua rreth 200 her\u00eb dhe asnj\u00eb d\u00ebshtim nuk u regjistrua. Mund\u00ebsisht problemi shfaqet shpesh vet\u00ebm gjat\u00eb ngarkes\u00ebs s\u00eb lart\u00eb, prandaj \u00ebsht\u00eb m\u00eb e sigurt t\u00eb ndalen planifikisht proceset e shkrimit dhe leximit dhe t\u00eb b\u00ebhet pastrimi pa lejuar r\u00ebnie t\u00eb till\u00eb GC.<\/p>\n
    Gjithashtu, major compaction nuk ndikon n\u00eb gjendjen e MemStore, p\u00ebr t\u00eb shkarkuar at\u00eb n\u00eb disk dhe p\u00ebr kompaktiikim duhet t\u00eb p\u00ebrdoret flush (connection.getAdmin().flush(TableName.valueOf(tblName))).<\/p>\n
    8. Rregullat dhe performanca<\/h2>\n
    \nSi\u00e7 u tha m\u00eb par\u00eb, HBase arrin suksesin m\u00eb t\u00eb madh aty ku nuk i nevojitet t\u00eb b\u00ebj\u00eb asgj\u00eb, gjat\u00eb ekzekutimit t\u00eb BulkLoad. Megjithat\u00eb, kjo vlen p\u00ebr shumic\u00ebn e sistemeve dhe njer\u00ebzve. Sidoqoft\u00eb, ky instrument \u00ebsht\u00eb m\u00eb shum\u00eb i p\u00ebrshtatsh\u00ebm p\u00ebr vendosjen masive t\u00eb t\u00eb dh\u00ebnave n\u00eb blloqe t\u00eb m\u00ebdha, ndon\u00ebse n\u00ebse procesi k\u00ebrkon ekzekutimin e shum\u00eb k\u00ebrkesave konkurruese p\u00ebr lexim dhe shkrim, p\u00ebrdoren komandat e p\u00ebrshkruara m\u00eb sip\u00ebr Get dhe Put. P\u00ebr t\u00eb p\u00ebrcaktuar parametrat optimal\u00eb, jan\u00eb realizuar ekzekutime me kombinime t\u00eb ndryshme parametrash tabelash dhe konfigurimesh:<\/p>\n