{"id":55302,"date":"2020-01-17T00:00:00","date_gmt":"2020-01-16T21:00:00","guid":{"rendered":"https:\/\/prohoster.info\/blog\/blog_prohoster\/teoriya-i-praktika-ispolzovaniya-hbase"},"modified":"2020-02-18T14:03:24","modified_gmt":"2020-02-18T11:03:24","slug":"teoriya-i-praktika-ispolzovaniya-hbase","status":"publish","type":"post","link":"https:\/\/prohoster.info\/sq\/blog\/administrirovanie\/teoriya-i-praktika-ispolzovaniya-hbase","title":{"rendered":"Teoria dhe praktika e p\u00ebrdorimit t\u00eb HBase","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

P\u00ebrsh\u00ebndetje! Emri im \u00ebsht\u00eb Danil Lipovoy, ekipi yn\u00eb n\u00eb Sbertech ka filluar t\u00eb p\u00ebrdor\u00eb HBase si depozita p\u00ebr t\u00eb dh\u00ebnat operative. Gjat\u00eb studimit t\u00eb tij, \u00ebsht\u00eb grumbulluar p\u00ebrvoja, t\u00eb cil\u00ebn desh\u00ebm ta sistematizojm\u00eb dhe ta p\u00ebrshkruajm\u00eb (shpresojm\u00eb se do t\u00eb jet\u00eb e dobishme p\u00ebr shum\u00ebk\u00ebnd). T\u00eb gjitha eksperimentet e m\u00ebposhtme jan\u00eb kryer me versionet HBase 1.2.0-cdh5.14.2 dhe 2.0.0-cdh6.0.0-beta1. <\/p>\n

    \n
  1. Arkitektura e p\u00ebrgjithshme <\/li>\n
  2. Shkrimi i t\u00eb dh\u00ebnave n\u00eb HBASE<\/li>\n
  3. Leximi i t\u00eb dh\u00ebnave nga HBASE<\/li>\n
  4. Keshimi i t\u00eb dh\u00ebnave<\/li>\n
  5. Procesimi i paketave t\u00eb t\u00eb dh\u00ebnave MultiGet\/MultiPut<\/li>\n
  6. Strategjia e ndarjes s\u00eb tabelave n\u00eb rajone (shp\u00ebrndarja)<\/li>\n
  7. Koh\u00ebzgjatja, kompakti dhe lokaliteti i t\u00eb dh\u00ebnave<\/li>\n
  8. C\u00ebshtjet dhe performanca<\/li>\n
  9. Testimi i ngarkes\u00ebs<\/li>\n
  10. P\u00ebrfundimet<\/li>\n<\/ol>\n

    <\/a><\/noindex><\/p>\n

    1. Arkitektura e p\u00ebrgjithshme<\/h2>\n

    \n\"Teoria
    \nMaster-i rezerv\u00eb d\u00ebgjon heartbeat-in e aktiv n\u00eb nodin ZooKeeper dhe n\u00eb rast se zhduket merr funksionet e master-it mbi vete. <\/p>\n

    2. Shkrimi i t\u00eb dh\u00ebnave n\u00eb HBASE<\/h2>\n

    \nM\u00eb par\u00eb, le t\u00eb shqyrtojm\u00eb rastin m\u00eb t\u00eb thjesht\u00eb \u2013 shkrimin e nj\u00eb objekti \u00e7el\u00ebs-vler\u00eb n\u00eb nj\u00eb tabel\u00eb me an\u00eb t\u00eb put(rowkey). Klienti fillimisht duhet t\u00eb kuptoj\u00eb se ku ndodhet rajoni rr\u00ebnj\u00ebsor i serverit (Root Region Server \u2013 RRS), i cili ruan tabel\u00ebn hbase:meta. K\u00ebt\u00eb informacion e merr nga ZooKeeper. Pas k\u00ebsaj, ai drejtohet te RRS dhe lexon tabel\u00ebn hbase:meta, nga e cila nxjerr informacionin, se cili RegionServer (RS) \u00ebsht\u00eb p\u00ebrgjegj\u00ebs p\u00ebr ruajtjen e t\u00eb dh\u00ebnave p\u00ebr \u00e7el\u00ebsin e caktuar rowkey n\u00eb tabel\u00ebn q\u00eb e intereson. P\u00ebr q\u00ebllime p\u00ebrdorimi n\u00eb t\u00eb ardhmen, tabela meta keshon nga klienti dhe prandaj subvencionet e m\u00ebvonshme shkojn\u00eb m\u00eb shpejt, direkt te RS.<\/p>\n

    M\u00eb pas, RS, duke marr\u00eb k\u00ebrkes\u00ebn, fillimisht e shkruan at\u00eb n\u00eb WriteAheadLog (WAL), q\u00eb \u00ebsht\u00eb e nevojshme p\u00ebr rikuperimin n\u00eb rast r\u00ebnie. Pastaj ruan t\u00eb dh\u00ebnat n\u00eb MemStore. Ky \u00ebsht\u00eb nj\u00eb tampon n\u00eb kujtes\u00eb, i cili p\u00ebrmban nj\u00eb grup t\u00eb renditur t\u00eb \u00e7el\u00ebsave t\u00eb k\u00ebtij rajoni. Tabela mund t\u00eb ndahet n\u00eb rajone (partita), secili prej t\u00eb cil\u00ebve p\u00ebrmban nj\u00eb grup t\u00eb pandryshuesh\u00ebm \u00e7el\u00ebsh. Kjo lejon, duke vendosur rajonet n\u00eb server\u00eb t\u00eb ndrysh\u00ebm, t\u00eb arrihet nj\u00eb performanc\u00eb m\u00eb e lart\u00eb. Megjithat\u00eb, pavar\u00ebsisht qart\u00ebsis\u00eb s\u00eb k\u00ebtij pohimi, m\u00eb pas do t\u00eb shohim se kjo nuk funksionon n\u00eb t\u00eb gjitha rastet.<\/p>\n

    Pasi t\u00eb vendoset regjistrimi n\u00eb MemStore, klienti merr nj\u00eb p\u00ebrgjigje se regjistrimi \u00ebsht\u00eb ruajtur me sukses. Nd\u00ebrkoh\u00eb, n\u00eb t\u00eb v\u00ebrtet\u00eb ai ruhet vet\u00ebm n\u00eb tampon dhe do t\u00eb kaloj\u00eb n\u00eb disk vet\u00ebm pas kalimit t\u00eb nj\u00eb intervali t\u00eb caktuar kohe ose kur mbushet me t\u00eb dh\u00ebna t\u00eb reja. <\/p>\n

    \"Teoria
    \nKur operacioni \"Fshi\" ekzekutohet, nuk ndodh nj\u00eb eliminim fizik i t\u00eb dh\u00ebnave. Ato thjesht sh\u00ebnohen si t\u00eb fshira, nd\u00ebrsa shkat\u00ebrrimi ndodh n\u00eb momentin e thirrjes s\u00eb funksionit major compact, p\u00ebr t\u00eb cilin \u00ebsht\u00eb shkruar m\u00eb shum\u00eb n\u00eb p.7.<\/p>\n

    Skedar\u00ebt n\u00eb formatin HFile grumbullohen n\u00eb HDFS dhe nga koha n\u00eb koh\u00eb nis nj\u00eb proces minor compact, i cili thjesht ngjitesh skedar\u00ebt e vegj\u00ebl n\u00eb m\u00eb t\u00eb m\u00ebdhenj, pa fshir\u00eb asgj\u00eb. Me kalimin e koh\u00ebs, kjo b\u00ebhet nj\u00eb problem q\u00eb shfaqet vet\u00ebm gjat\u00eb leximit t\u00eb t\u00eb dh\u00ebnave (p\u00ebr k\u00ebt\u00eb do t\u00eb kthehemi m\u00eb von\u00eb). <\/p>\n

    P\u00ebrve\u00e7 procesit t\u00eb ngarkes\u00ebs t\u00eb p\u00ebrshkruar m\u00eb sip\u00ebr, ekziston nj\u00eb procedur\u00eb shum\u00eb m\u00eb efikase, e cila p\u00ebrmban ndoshta pik\u00ebn m\u00eb t\u00eb fort\u00eb t\u00eb k\u00ebsaj DB - BulkLoad. Ajo konsiston n\u00eb faktin se ne vet\u00eb formojm\u00eb HFiles dhe i vendosim n\u00eb disk, q\u00eb na lejon t\u00eb shkall\u00ebzojm\u00eb mjaft mir\u00eb dhe t\u00eb arrijm\u00eb shpejt\u00ebsi t\u00eb konsiderueshme. N\u00eb thelb, kufizimi k\u00ebtu nuk \u00ebsht\u00eb HBase, por mund\u00ebsit\u00eb e harduerit. M\u00eb posht\u00eb jan\u00eb rezultatet e ngarkes\u00ebs n\u00eb nj\u00eb klaster q\u00eb p\u00ebrb\u00ebhet nga 16 RegionServers dhe 16 NodeManager YARN (CPU Xeon E5-2680 v4 @ 2.40GHz * 64 procese), versioni HBase 1.2.0-cdh5.14.2. <\/p>\n

    \"Teoria
    \n
    \nK\u00ebtu shihet se duke rritur numrin e particioneve (rajoneve) n\u00eb tabel\u00eb, si dhe ekzekutor\u00ebt e Spark, arrijm\u00eb nj\u00eb rritje t\u00eb shpejt\u00ebsis\u00eb s\u00eb ngarkes\u00ebs. Gjithashtu, shpejt\u00ebsia varet nga volumi i shkruar. Blloqet e m\u00ebdha japin nj\u00eb rritje n\u00eb matjen MB\/sek, nd\u00ebrsa ato t\u00eb vogla n\u00eb numrin e sh\u00ebnimeve t\u00eb futur n\u00eb nj\u00eb nj\u00ebsi kohe, n\u00ebn kushte t\u00eb barabarta. <\/p>\n

    Gjithashtu, mund t\u00eb nisim ngarkimin n\u00eb dy tabela nj\u00ebkoh\u00ebsisht dhe t\u00eb marrim dyfishin e shpejt\u00ebsis\u00eb. M\u00eb posht\u00eb shihet se shkruarja e blloqeve 10 KB menj\u00ebher\u00eb n\u00eb dy tabela shkon me nj\u00eb shpejt\u00ebsi prej rreth 600 Mb\/sek n\u00eb secil\u00ebn (shuma 1275 Mb\/sek), q\u00eb p\u00ebrputhet me shpejt\u00ebsin\u00eb e shkruarjes n\u00eb nj\u00eb tabel\u00eb 623 MB\/sek (shih \u211611 m\u00eb sip\u00ebr)<\/p>\n

    \"Teoria
    \nNd\u00ebrsa, ekzekutimi i dyt\u00eb me sh\u00ebnime n\u00eb 50 KB tregon se shpejt\u00ebsia e ngarkes\u00ebs rritet tashm\u00eb n\u00eb m\u00ebnyr\u00eb t\u00eb vog\u00ebl, q\u00eb flet p\u00ebr afrim ndaj vlerave maksimale. N\u00eb t\u00eb nj\u00ebjt\u00ebn koh\u00eb, duhet t\u00eb merret parasysh se sam\u00eb HBASE k\u00ebtu nuk krijon ngarkes\u00eb t\u00eb r\u00ebnd\u00eb, gjith\u00e7ka q\u00eb k\u00ebrkohet prej tij \u00ebsht\u00eb s\u00eb pari t\u00eb dor\u00ebzoj\u00eb t\u00eb dh\u00ebnat nga hbase:meta, dhe pas vendosjes s\u00eb HFiles, t\u00eb shkarkoj\u00eb t\u00eb dh\u00ebnat BlockCache dhe t\u00eb ruaj\u00eb tamponin MemStore n\u00eb disk, n\u00ebse ai nuk \u00ebsht\u00eb bosh.<\/p>\n

    3. Leximi i t\u00eb dh\u00ebnave nga HBASE<\/h2>\n

    \nN\u00ebse merret parasysh se t\u00eb gjitha informacionet nga hbase:meta tashm\u00eb i ka klienti (shih. p.2), at\u00ebher\u00eb k\u00ebrkesa shkon menj\u00ebher\u00eb n\u00eb at\u00eb RS, ku ruhet \u00e7el\u00ebsi i nevojsh\u00ebm. Fillimisht, k\u00ebrkimi kryhet n\u00eb MemCache. Pavar\u00ebsisht n\u00ebse atje ka t\u00eb dh\u00ebna apo jo, k\u00ebrkimi kryhet gjithashtu n\u00eb tamponin BlockCache dhe, n\u00ebse \u00ebsht\u00eb e nevojshme, n\u00eb HFiles. N\u00ebse t\u00eb dh\u00ebnat jan\u00eb gjetur n\u00eb skedarin, ato vendosen n\u00eb BlockCache dhe p\u00ebr k\u00ebrkes\u00ebn e ardhshme do t\u00eb kthehen m\u00eb shpejt. K\u00ebrkimi n\u00eb HFile ndodh mjaft shpejt p\u00ebr shkak t\u00eb p\u00ebrdorimit t\u00eb filtrit Bloom, dmth duke lexuar nj\u00eb volum t\u00eb vog\u00ebl t\u00eb dh\u00ebnash, ai e p\u00ebrcakton menj\u00ebher\u00eb n\u00ebse ky skedar p\u00ebrmban \u00e7el\u00ebsin e nevojsh\u00ebm dhe, n\u00ebse nuk ka, kalon te i siguiente.<\/p>\n

    \"Teoria
    \nDuke marr\u00eb t\u00eb dh\u00ebnat nga k\u00ebto tre burime, RS formon p\u00ebrgjigjen. N\u00eb ve\u00e7anti, ai mund t\u00eb transmetoj\u00eb menj\u00ebher\u00eb disa versione t\u00eb gjetura t\u00eb objektit n\u00ebse klienti ka k\u00ebrkuar versionim.<\/p>\n

    4. Keshimi i t\u00eb dh\u00ebnave<\/h2>\n

    \nBuffers MemStore dhe BlockCache z\u00ebn\u00eb deri n\u00eb 80% t\u00eb memories s\u00eb rezervuar on-heap t\u00eb RS (pjesa tjet\u00ebr \u00ebsht\u00eb rezervuar p\u00ebr detyra sh\u00ebrbimi t\u00eb RS). N\u00ebse modelet tipike t\u00eb p\u00ebrdorimit jan\u00eb t\u00eb tilla q\u00eb proceset shkruajn\u00eb dhe menj\u00ebher\u00eb lexojn\u00eb t\u00eb nj\u00ebjtat t\u00eb dh\u00ebna, ka kuptim t\u00eb ulet BlockCache dhe t\u00eb rritet MemStore, pasi gjat\u00eb shkrimit t\u00eb dh\u00ebnat n\u00eb cache nuk hyjn\u00eb p\u00ebr lexim, k\u00ebshtu q\u00eb p\u00ebrdorimi i BlockCache do t\u00eb ndodh\u00eb m\u00eb rrall\u00eb. Buffer BlockCache p\u00ebrb\u00ebhet nga dy pjes\u00eb: LruBlockCache (p\u00ebrher\u00eb on-heap) dhe BucketCache (n\u00eb p\u00ebrgjith\u00ebsi off-heap ose n\u00eb SSD). BucketCache duhet t\u00eb p\u00ebrdoret kur ka shum\u00eb k\u00ebrkesa p\u00ebr lexim dhe ato nuk hyjn\u00eb n\u00eb LruBlockCache, gj\u00eb q\u00eb rezulton n\u00eb nj\u00eb pun\u00eb aktive t\u00eb Garbage Collector. Megjithat\u00eb, nuk duhet pritur nj\u00eb rritje dramatike t\u00eb performanc\u00ebs nga p\u00ebrdorimi i cache p\u00ebr lexim, por p\u00ebr k\u00ebt\u00eb do t\u00eb kthehemi p\u00ebrs\u00ebri n\u00eb p. 8.<\/p>\n

    \"Teoria
    \nBlockCache \u00ebsht\u00eb nj\u00eb p\u00ebr t\u00eb gjith\u00eb RS, nd\u00ebrsa MemStore \u00ebsht\u00eb specifike p\u00ebr secil\u00ebn tabel\u00eb (nj\u00eb p\u00ebr \u00e7do Column Family).<\/p>\n

    Si p\u00ebrshkruhet<\/a><\/noindex> N\u00eb teori, gjat\u00eb shkrimit, t\u00eb dh\u00ebnat nuk hyjn\u00eb n\u00eb cache, dhe me t\u00eb v\u00ebrtet\u00eb, parametrat e till\u00eb CACHE_DATA_ON_WRITE p\u00ebr tabel\u00ebn dhe \"Cache DATA on Write\" p\u00ebr RS jan\u00eb vendosur n\u00eb false. Megjithat\u00eb, n\u00eb praktik\u00eb, n\u00ebse shkruani t\u00eb dh\u00ebna n\u00eb MemStore, m\u00eb pas e shfryni at\u00eb n\u00eb disk (duke e pastruar k\u00ebshtu), pastaj fshini skedarin q\u00eb \u00ebsht\u00eb krijuar, duke realizuar nj\u00eb k\u00ebrkes\u00eb get, ne e marrim me sukses t\u00eb dh\u00ebnat. Madje, edhe n\u00ebse e fikni krejt BlockCache dhe mbushni tabel\u00ebn me t\u00eb dh\u00ebna t\u00eb reja, m\u00eb pas arritni t\u00eb shfryni MemStore n\u00eb disk, duke i fshir\u00eb ato dhe k\u00ebrkuar nga nj\u00eb sesion tjet\u00ebr, ato do t\u00eb nxirren s\u00ebrish nga diku. Pra, HBase ruan n\u00eb vetvete jo vet\u00ebm t\u00eb dh\u00ebna, por edhe misteret e \u00e7uditshme.<\/p>\n

    hbase(main):001:0> krijo 'ns:magic', 'cf'\nTabela ns:magic u krijua\nMori 1.1533 sekonda\nhbase(main):002:0> vendos 'ns:magic', 'key1', 'cf:c', 'provo_t\u00eb_fshihem'\nMori 0.2610 sekonda\nhbase(main):003:0> flush 'ns:magic'\nMori 0.6161 sekonda\nhdfs dfs -mv \/data\/hbase\/data\/ns\/magic\/* \/tmp\/trash\nhbase(main):002:0> merr 'ns:magic', 'key1'\n cf:c      timestamp=1534440690218, vlera=provo_t\u00eb_fshihem\n<\/code><\/pre>\n
    \nParametri \"Cache DATA on Read\" \u00ebsht\u00eb vendosur false. N\u00ebse keni ide, mir\u00ebpriteni t\u00eb diskutojm\u00eb k\u00ebt\u00eb n\u00eb komentet.<\/p>\n
    5. P\u00ebrpunimi i grupeve t\u00eb t\u00eb dh\u00ebnave MultiGet\/MultiPut<\/h2>\n
    \nP\u00ebrpunimi i k\u00ebrkesave t\u00eb vetme (Get\/Put\/Delete) \u00ebsht\u00eb nj\u00eb operacion mjaft i kushtuesh\u00ebm, prandaj \u00ebsht\u00eb e rekomandueshme t'i gruposh sa m\u00eb shum\u00eb t\u00eb jet\u00eb e mundur n\u00eb List ose List, duke lejuar k\u00ebshtu nj\u00eb rritje t\u00eb konsiderueshme t\u00eb performanc\u00ebs. Kjo ve\u00e7an\u00ebrisht vlen p\u00ebr operacionin e shkrimit, nd\u00ebrsa n\u00eb lexim ka nj\u00eb penges\u00eb tjet\u00ebr. N\u00eb grafikun m\u00eb posht\u00eb \u00ebsht\u00eb treguar koha e leximit t\u00eb 50,000 regjistrimeve nga MemStore. Leximi \u00ebsht\u00eb b\u00ebr\u00eb n\u00eb nj\u00eb t\u00eb vet\u00ebm dhe n\u00eb boshtin horizontal \u00ebsht\u00eb treguar numri i \u00e7el\u00ebsave n\u00eb k\u00ebrkes\u00eb. K\u00ebtu shihet se me rritjen deri n\u00eb nj\u00eb mij\u00eb \u00e7el\u00ebsa n\u00eb nj\u00eb k\u00ebrkes\u00eb, koha e ekzekutimit bie, pra shpejt\u00ebsia rritet. Megjithat\u00eb, me modalitetin MSLAB t\u00eb aktivizuar n\u00eb m\u00ebnyr\u00eb t\u00eb paracaktuar, pas k\u00ebtij prag, fillon nj\u00eb r\u00ebnien dramatike t\u00eb performanc\u00ebs, nd\u00ebrsa m\u00eb shum\u00eb \u00ebsht\u00eb volumi i t\u00eb dh\u00ebnave n\u00eb regjist\u00ebr, aq m\u00eb shum\u00eb \u00ebsht\u00eb koha e pun\u00ebs. <\/p>\n
    \"Teoria
    \n
    \nTestet jan\u00eb kryer n\u00eb nj\u00eb virtualka, 8 b\u00ebrthama, versioni HBase 2.0.0-cdh6.0.0-beta1.<\/p>\n
    Modaliteti MSLAB \u00ebsht\u00eb krijuar p\u00ebr t\u00eb zvog\u00ebluar fragmentimin e heap-it, q\u00eb ndodh p\u00ebr shkak t\u00eb p\u00ebrzierjes s\u00eb t\u00eb dh\u00ebnave t\u00eb brezit t\u00eb ri dhe atyre t\u00eb vjet\u00ebr. Si zgjidhje e problemit, kur aktivizohet MSLAB, t\u00eb dh\u00ebnat vendosen n\u00eb qeliza (chunk) relativisht t\u00eb vogla dhe p\u00ebrpunohen n\u00eb grupe. Si rezultat, kur volumi n\u00eb paket\u00ebn e t\u00eb dh\u00ebnave t\u00eb k\u00ebrkuara tejkalon madh\u00ebsin\u00eb e caktuar, performanca bie n\u00eb m\u00ebnyr\u00eb drastike. N\u00eb an\u00ebn tjet\u00ebr, \u00e7aktivizimi i k\u00ebtij modaliteti gjithashtu nuk \u00ebsht\u00eb i d\u00ebshiruesh\u00ebm, pasi do t\u00eb \u00e7oj\u00eb n\u00eb ndalime p\u00ebr shkak t\u00eb GC n\u00eb momentet e pun\u00ebs intensive me t\u00eb dh\u00ebna. Nj\u00eb zgjidhje e mir\u00eb \u00ebsht\u00eb rritja e madh\u00ebsive t\u00eb qelizave, n\u00eb rastin e shkrimit aktiv p\u00ebrmes put-it n\u00eb t\u00eb nj\u00ebjt\u00ebn koh\u00eb me leximin. Duhet t\u00eb theksohet se problemi nuk ndodh n\u00ebse pas shkrimit ekzekutohet komand\u00eb flush, e cila d\u00ebrgon MemStore n\u00eb disk ose n\u00ebse b\u00ebhet ngarkesa p\u00ebrmes BulkLoad. N\u00eb tabel\u00ebn m\u00eb posht\u00eb tregohet se k\u00ebrkesat nga MemStore p\u00ebr t\u00eb dh\u00ebna t\u00eb m\u00ebdha (dhe n\u00eb t\u00eb nj\u00ebjtin num\u00ebr) \u00e7ojn\u00eb n\u00eb ngadal\u00ebsim. Megjithat\u00eb, duke rritur chunksize, rikthejm\u00eb koh\u00ebn e p\u00ebrpunimit n\u00eb norm\u00eb.<\/p>\n
    \"Teoria
    \nP\u00ebrve\u00e7 rritjes s\u00eb chunksize, ndihmon ndarja e t\u00eb dh\u00ebnave sipas rajoneve, pra, ndarja e tabelave. Kjo \u00e7on n\u00eb at\u00eb q\u00eb n\u00eb \u00e7do rajon vijn\u00eb m\u00eb pak k\u00ebrkesa dhe n\u00ebse ato vendosen n\u00eb nj\u00eb qeliz\u00eb, p\u00ebrgjigja mbetet e mir\u00eb.<\/p>\n
    6. Strategjia e ndarjes s\u00eb tabelave n\u00eb rajone (splitting)<\/h2>\n
    \nDuke qen\u00eb se HBase \u00ebsht\u00eb nj\u00eb magazin\u00eb \u00e7el\u00ebs-vler\u00eb dhe ndarja b\u00ebhet sipas \u00e7el\u00ebsit, \u00ebsht\u00eb shum\u00eb e r\u00ebnd\u00ebsishme t\u00eb ndahet t\u00eb dh\u00ebnat n\u00eb m\u00ebnyr\u00eb t\u00eb barabart\u00eb n\u00eb t\u00eb gjitha rajonet. P\u00ebr shembull, ndarja e nj\u00eb tabele t\u00eb till\u00eb n\u00eb tre pjes\u00eb do t\u00eb \u00e7onte n\u00eb at\u00eb q\u00eb t\u00eb dh\u00ebnat do t\u00eb ishin t\u00eb ndara n\u00eb tri rajone:<\/p>\n
    \"Teoria
    \nNdodh q\u00eb kjo t\u00eb \u00e7oj\u00eb n\u00eb ngadal\u00ebsim t\u00eb menj\u00ebhersh\u00ebm, n\u00ebse t\u00eb dh\u00ebnat q\u00eb do t\u00eb ngarkohen m\u00eb pas kan\u00eb form\u00eb p\u00ebr shembull vlerash t\u00eb gjata, shumica e t\u00eb cilave fillojn\u00eb me t\u00eb nj\u00ebjt\u00ebn shif\u00ebr, p\u00ebr shembull:<\/p>\n
    1000001
    \n1000002
    \n\u2026
    \n1100003<\/p>\n
    Duke qen\u00eb se \u00e7el\u00ebsat ruhen si nj\u00eb array bajtesh, t\u00eb gjith\u00eb do t\u00eb fillojn\u00eb nj\u00ebsoj dhe do t'i p\u00ebrkasin nj\u00eb rajoni t\u00eb vet\u00ebm #1 q\u00eb ruan k\u00ebt\u00eb gam\u00eb \u00e7el\u00ebsash. Ka disa strategji ndarje:<\/p>\n
    HexStringSplit \u2013 Kthen \u00e7el\u00ebsin n\u00eb nj\u00eb varg me kodim gjasht\u00ebmb\u00ebdhjet\u00ebshe n\u00eb gam\u00ebn \u00ab00000000\u00bb => \u00abFFFFFFFF\u00bb dhe e mbush at\u00eb nga e majta me zero.<\/p>\n
    UniformSplit \u2013 Kthen \u00e7el\u00ebsin n\u00eb nj\u00eb array bajtesh me kodim gjasht\u00ebmb\u00ebdhjet\u00ebshe n\u00eb gam\u00ebn \u00ab00\u00bb => \u00abFF\u00bb dhe e mbush at\u00eb nga e djathta me zero.<\/p>\n
    P\u00ebr m\u00eb tep\u00ebr, mund t\u00eb specifikoni \u00e7do gam\u00eb ose set \u00e7el\u00ebsash p\u00ebr ndarje dhe t\u00eb konfiguroni ndarjen automatike. Megjithat\u00eb, nj\u00eb nga qasjet m\u00eb t\u00eb thjeshta dhe efikase \u00ebsht\u00eb UniformSplit dhe p\u00ebrdorimi i konkatenimit t\u00eb hash-it, p\u00ebr shembull, \u00e7ifti m\u00eb i lart\u00eb i bajt\u00ebve nga ekzekutimi i \u00e7el\u00ebsit p\u00ebrmes funksionit CRC32(rowkey) dhe vet\u00eb rowkey:<\/p>\n
    hash + rowkey<\/p>\n
    At\u00ebher\u00eb t\u00eb gjitha t\u00eb dh\u00ebnat do t\u00eb shp\u00ebrndahen n\u00eb m\u00ebnyr\u00eb t\u00eb barabart\u00eb n\u00eb rajone. Gjat\u00eb leximit, dy bajt\u00ebt e par\u00eb thjesht hiqen dhe mbetet \u00e7el\u00ebsi origjinal. Gjithashtu, RS kontrollon sasin\u00eb e t\u00eb dh\u00ebnave dhe \u00e7el\u00ebsave n\u00eb rajon dhe n\u00eb rast t\u00eb tejkalimit t\u00eb kufijve, automatikisht e ndan at\u00eb n\u00eb pjes\u00eb. <\/p>\n
    7. Q\u00ebndrueshm\u00ebria dhe lokaliteti i t\u00eb dh\u00ebnave<\/h2>\n
    \nDukeq\u00eb \u00e7do grup \u00e7el\u00ebsash menaxhohet nga nj\u00eb rajon i vet\u00ebm, zgjidhja e problemeve q\u00eb lidhen me r\u00ebniet e RS ose me daljen nga p\u00ebrdorimi \u00ebsht\u00eb ruajtja e t\u00eb dh\u00ebnave t\u00eb nevojshme n\u00eb HDFS. Kur ndodh nj\u00eb r\u00ebnie e RS, masteri e zbulon k\u00ebt\u00eb p\u00ebrmes munges\u00ebs s\u00eb heartbeat n\u00eb nyj\u00ebn ZooKeeper. At\u00ebher\u00eb ai i cakton nj\u00eb rajon tjet\u00ebr t\u00eb menaxhuar nj\u00eb RS t\u00eb ri dhe pasi q\u00eb HFiles ruhen n\u00eb sistemin e skedar\u00ebve t\u00eb shp\u00ebrndar\u00eb, pronari i ri i lexon dhe vazhdon t\u00eb sh\u00ebrbej\u00eb t\u00eb dh\u00ebnat. Megjithat\u00eb, pasi nj\u00eb pjes\u00eb e t\u00eb dh\u00ebnave mund t\u00eb jet\u00eb n\u00eb MemStore dhe nuk \u00ebsht\u00eb ruajtur ende n\u00eb HFiles, p\u00ebr rikthimin e historis\u00eb s\u00eb operacioneve p\u00ebrdoret WAL, i cili gjithashtu ruhet n\u00eb HDFS. Pas vendosjes s\u00eb ndryshimeve, RS \u00ebsht\u00eb n\u00eb gjendje t\u00eb p\u00ebrgjigjet ndaj k\u00ebrkesave, megjithat\u00eb kalimi \u00e7on n\u00eb faktin se nj\u00eb pjes\u00eb e t\u00eb dh\u00ebnave dhe proceset q\u00eb i sh\u00ebrbejn\u00eb atyre jan\u00eb n\u00eb nyja t\u00eb ndryshme, dmth, lokaliteti zvog\u00eblohet. <\/p>\n
    Zgjidhja p\u00ebr k\u00ebt\u00eb problem \u00ebsht\u00eb major compaction \u2013 kjo procedur\u00eb transferon skedar\u00ebt n\u00eb ato nyja q\u00eb jan\u00eb p\u00ebrgjegj\u00ebse p\u00ebr ta (aty ku ndodhen rajonet e tyre), si rezultat i t\u00eb cil\u00ebs gjat\u00eb k\u00ebsaj procedure ngarkesa n\u00eb rrjet dhe disqet rritet ndjesh\u00ebm. Megjithat\u00eb, m\u00eb von\u00eb, qasja n\u00eb t\u00eb dh\u00ebna p\u00ebrmir\u00ebsohet ndjesh\u00ebm. P\u00ebr m\u00eb tep\u00ebr, major_compaction kombinon t\u00eb gjitha HFiles n\u00eb nj\u00eb skedar t\u00eb vet\u00ebm n\u00eb kuad\u00ebr t\u00eb rajonit, si dhe pastron t\u00eb dh\u00ebnat n\u00eb var\u00ebsi t\u00eb cil\u00ebsimeve t\u00eb tabel\u00ebs. P\u00ebr shembull, mund t\u00eb caktohet numri i versioneve t\u00eb objektit q\u00eb duhet t\u00eb ruhen ose koha e jet\u00ebs s\u00eb tij, pas kalimit t\u00eb s\u00eb cil\u00ebs objekti fshihet fizikisht.<\/p>\n
    Kjo procedur\u00eb mund t\u00eb ket\u00eb nj\u00eb ndikim shum\u00eb pozitiv n\u00eb funksionimin e HBase. N\u00eb imazhin m\u00eb posht\u00eb shihet si degradoi performanca si rezultat i shkrimit aktiv t\u00eb t\u00eb dh\u00ebnave. K\u00ebtu shihet si 40 rrjedha shkruajn\u00eb n\u00eb nj\u00eb tabel\u00eb dhe 40 rrjedha lexojn\u00eb t\u00eb dh\u00ebna nj\u00ebkoh\u00ebsisht. Rrjedhat shkruese formojn\u00eb gjithnj\u00eb e m\u00eb shum\u00eb HFiles, t\u00eb cilat lexohen nga rrjedhat e tjera. Si rezultat, gjithnj\u00eb e m\u00eb shum\u00eb t\u00eb dh\u00ebna duhet t\u00eb fshihen nga memoria dhe n\u00eb fund fillon t\u00eb funksionoj\u00eb GC, q\u00eb pothuajse paralizon t\u00eb gjith\u00eb pun\u00ebn. Aktivizimi i major compaction \u00e7oi n\u00eb pastrimin e mbetjeve t\u00eb krijuara dhe rikthimin e performanc\u00ebs.<\/p>\n
    \"Teoria
    \nTesti u krye n\u00eb 3 DataNode dhe 4 RS (CPU Xeon E5-2680 v4 @ 2.40GHz * 64 rrjedha). Versioni HBase 1.2.0-cdh5.14.2<\/p>\n
    \u00cbsht\u00eb e r\u00ebnd\u00ebsishme t\u00eb theksohet se lan\u00e7imi i major compaction u krye n\u00eb nj\u00eb tav\u00ebll \"live\", n\u00eb t\u00eb cil\u00ebn ishin duke u shkruar dhe lexuar t\u00eb dh\u00ebna aktivisht. N\u00eb internet ka pasur deklarata q\u00eb sugjerojn\u00eb se kjo mund t\u00eb \u00e7oj\u00eb n\u00eb p\u00ebrgjigje t\u00eb gabuar gjat\u00eb leximit t\u00eb t\u00eb dh\u00ebnave. P\u00ebr t\u00eb verifikuar, u aktivizua nj\u00eb proces q\u00eb gjeneronte t\u00eb dh\u00ebna t\u00eb reja dhe i shkruante ato n\u00eb tav\u00ebll. Pas k\u00ebsaj, menj\u00ebher\u00eb lexonte dhe krahasonte n\u00ebse vlera e marr\u00eb p\u00ebrputhej me at\u00eb q\u00eb ishte shkruar. Gjat\u00eb pun\u00ebs s\u00eb k\u00ebtij procesi, major compaction u lan\u00e7ua rreth 200 her\u00eb dhe asnj\u00eb d\u00ebshtim nuk u regjistrua. Ndoshta problemi shfaqet s\u00eb shpejti dhe vet\u00ebm gjat\u00eb ngarkes\u00ebs s\u00eb lart\u00eb, prandaj \u00ebsht\u00eb m\u00eb e sigurt t\u00eb ndaloni procedurat e shkrimit dhe leximit n\u00eb m\u00ebnyr\u00eb planifikuese dhe t\u00eb kryeni pastrimin pa lejuar r\u00ebniet e tilla GC.<\/p>\n
    Po ashtu, major compaction nuk ndikon n\u00eb gjendjen e MemStore, p\u00ebr t\u00eb hedhur at\u00eb n\u00eb disk dhe p\u00ebr kompaktim duhet t\u00eb p\u00ebrdoret flush (connection.getAdmin().flush(TableName.valueOf(tblName))).<\/p>\n
    8. Cil\u00ebsimet dhe Performanca<\/h2>\n
    \nSi\u00e7 \u00ebsht\u00eb th\u00ebn\u00eb m\u00eb par\u00eb, HBase tregon suksesin m\u00eb t\u00eb madh atje ku nuk ka nevoj\u00eb t\u00eb b\u00ebj\u00eb asgj\u00eb, gjat\u00eb realizimit t\u00eb BulkLoad. Megjithat\u00eb, kjo i p\u00ebrket shumic\u00ebs s\u00eb sistemeve dhe njer\u00ebzve. Sidoqoft\u00eb, ky instrument \u00ebsht\u00eb m\u00eb i p\u00ebrshtatsh\u00ebm p\u00ebr vendosjen masive t\u00eb t\u00eb dh\u00ebnave n\u00eb blloqe t\u00eb m\u00ebdha, nd\u00ebrsa n\u00ebse procesi k\u00ebrkon realizimin e shum\u00eb k\u00ebrkesave konkurruese p\u00ebr lexim dhe shkrim, p\u00ebrdoren komandat e p\u00ebrshkruara m\u00eb sip\u00ebr Get dhe Put. P\u00ebr t\u00eb p\u00ebrcaktuar parametrat optimal\u00eb, u zhvilluan lan\u00e7ime me kombinime t\u00eb ndryshme t\u00eb parametrave t\u00eb tavllave dhe cil\u00ebsimeve:<\/p>\n