Waxqabadka sare waa mid ka mid ah shuruudaha muhiimka ah marka la shaqeynayo xogta weyn. Qaybta rarka xogta ee Sberbank, waxaan ku shubnaa ku dhawaad dhammaan wax kala iibsiga Cloud Xogta ku salaysan Hadoop sidaas darteedna waxaan la macaamilnaa qulqulka macluumaadka dhabta ah. Dabiici ahaan, waxaan had iyo jeer raadineynaa habab aan ku wanaajino waxqabadka, oo hadda waxaan rabnaa inaan kuu sheegno sida aan ugu suurtagashay in aan xirno RegionServer HBase iyo macmiilka HDFS, taas oo ay ugu mahadcelineyso taas oo aan awoodnay inaan si weyn u kordhino xawaaraha akhrinta.
Si kastaba ha noqotee, ka hor intaanad u gudbin nuxurka hagaajinta, waxaa habboon in laga hadlo xaddidaadaha, mabda'a ahaan, aan la hareer mari karin haddii aad ku fadhiisato HDD.
Waa maxay sababta HDD iyo akhrinta Random Access ee degdega ah aysan isqaban karin
Sidaad ogtahay, HBase, iyo xogo kale oo badan, waxay xogta ku kaydiyaan baloogyo dhawr iyo toban kilobytes ah oo cabbirkooda ah. Sida caadiga ah waxay ku saabsan tahay 64 KB. Hadda aan qiyaasno inaan u baahanahay inaan helno 100 bytes oo keliya waxaanan ka codsaneynaa HBase inay na siiso xogtan iyadoo la adeegsanayo fure gaar ah. Maadaama cabbirka xannibaadda ee HFiles uu yahay 64 KB, codsigu wuxuu noqon doonaa 640 jeer ka weyn (kaliya hal daqiiqo!) intii loo baahnaa.
Marka xigta, maadaama codsigu mari doono HDFS iyo habka kaydinta xogta badan ShortCircuitCache (taas oo u oggolaanaysa in si toos ah loo galo faylasha), tani waxay keenaysaa in horay loo akhriyo 1 MB ee diskka. Si kastaba ha ahaatee, tan waxaa lagu hagaajin karaa qiyaasta dfs.macmiilka.akhri.gaaban.buffer.cabbir marar badanna waxay macno samaynaysaa in la dhimo qiimahan, tusaale ahaan 126 KB.
Aynu nidhaahno waxaan samaynaa tan, laakiin marka lagu daro, marka aan bilowno akhrinta xogta iyada oo loo marayo Java api, sida hawlaha sida FileChannel.read oo weydiiso nidaamka hawlgalka inuu akhriyo qadarka xogta ee la cayimay, wuxuu akhriyaa "kaliya haddii ay dhacdo" 2 jeer ka badan. , i.e. 256 KB kiiskeena. Tani waa sababta oo ah Java ma laha hab fudud oo lagu dejiyo calanka FADV_RANDOM si looga hortago dhaqankan.
Natiijo ahaan, si loo helo 100 bytes, 2600 jeer ka badan ayaa lagu akhriyaa daboolka hoostiisa. Waxay u ekaan kartaa in xalku cad yahay, aynu ku soo koobno cabbirka baloogga oo aynu ka dhigno kiiloobyte, aynu dejino calankii aynu xusnay oo aynu dardar-gelin weyn helno. Laakiin dhibka jira ayaa ah in iyadoo la dhimayo cabbirka baloogga 2 jeer, waxaan sidoo kale yareynaa tirada bytes ee la akhriyo halbeeg kasta 2 jeer.
Qaar ka mid ah faa'iidooyin laga helo dejinta calanka FADV_RANDOM waa la heli karaa, laakiin kaliya leh xadhig badan oo badan oo leh cabbir xaddidan oo ah 128 KB, laakiin tani waa ugu badnaan boqolkiiba tobanaan boqolkiiba:
Tijaabooyin ayaa lagu sameeyay 100 fayl, midkiiba 1 GB cabbirkiisuna wuxuu ku yaal 10 HDD.
Aan xisaabin karno waxa aan awoodno, mabda 'ahaan, ku xisaabtan karno xawaarahan:
Aynu nidhaahno waxaan ka akhrinay 10 saxan oo xawaarihiisu yahay 280 MB/sec, i.e. 3 milyan jeer 100 bytes. Laakiin sida aan xasuusannahay, xogta aan u baahanahay ayaa 2600 jeer ka yar waxa la akhriyo. Markaa, waxaan u qaybineynaa 3 milyan 2600 oo aan helno 1100 diiwaan ilbiriqsikii.
Niyad-jab, miyaanay ahayn? Taasi waa dabeecadda Helitaanka Random Helitaanka xogta HDD - iyadoon loo eegin xajmiga xannibaadda. Tani waa xadka jireed ee helitaanka random-ka mana jirto xog ururin kara wax badan oo xaaladahan oo kale ah.
Sidee markaa xog ururintu u gaadhaan xawli sare oo aad u sarreeya? Si aan uga jawaabno su'aashan, aan eegno waxa dhacaya sawirka soo socda:
Halkan waxaan ku aragnaa in dhowrka daqiiqo ee ugu horreeya xawaaruhu uu dhab ahaantii ku saabsan yahay kun rikoodhada ilbiriqsi kasta. Si kastaba ha ahaatee, in ka badan, sababtoo ah xaqiiqda ah in wax badan oo ka badan inta la codsaday, xogtu waxay ku dhammaanaysaa buff / cache ee nidaamka hawlgalka (linux) oo xawaaruhu wuxuu kordhiyaa si ka wanaagsan 60 kun ilbiriqsi kasta.
Markaa, si dheeraad ah waxaanu wax ka qaban doonaa dardargelinta helitaanka xogta ku jirta kaydka OS-ka ama ku dhex yaal aaladaha kaydinta SSD/NVMe ee xawaaraha gelitaanka u dhigma.
Xaaladeena, waxaan ku sameyn doonaa imtixaano kursi ka kooban 4 server, mid kasta oo ka mid ah ayaa lagu soo dalacayaa sida soo socota.
CPU: Xeon E5-2680 v4 @ 2.40GHz 64 dunta.
Xusuusta: 730 GB.
nooca Java: 1.8.0_111
Oo halkan qodobka ugu muhiimsan waa qadarka xogta shaxda ee u baahan in la akhriyo. Xaqiiqdu waxay tahay in haddii aad akhrido xogta miis si buuxda loogu dhejiyay kaydka HBase, markaa xitaa ma iman doonto wax akhriska buff/cache nidaamka qalliinka. Sababtoo ah HBase asal ahaan waxay u qoondaysaa 40% xusuusta qaabdhismeedka loo yaqaan BlockCache. Asal ahaan kani waa ConcurrentHashMap, halkaasoo furuhu yahay magaca faylka + ka-goynta block-ka, qiimuhuna waa xogta dhabta ah ee kabitaankan.
Sidaa darteed, marka aad akhrinayso kaliya dhismahan, waxaan , sida hal milyan oo codsi ilbiriqsikii. Laakiin aan qiyaasno in aanan u qoondayn karin boqolaal gigabytes oo xusuusta ah oo keliya baahida kaydka xogta, sababtoo ah waxaa jira waxyaabo kale oo badan oo faa'iido leh oo ku shaqeeya server-yadaas.
Tusaale ahaan, kiiskeena, mugga BlockCache ee hal RS waa qiyaastii 12 GB. Waxaan ku degnay laba RS hal nood, i.e. 96 GB ayaa loo qoondeeyay BlockCache dhammaan noodyada. Waxaana jira marar badan oo xog ah, tusaale ahaan, ha ahaato 4 miis, midkiiba 130 gobol, kuwaas oo faylalku cabbirkoodu yahay 800 MB, oo ay ku cadaadinayaan FAST_DIFF, i.e. wadar ahaan 410 GB (tani waa xog saafi ah, i.e. iyada oo aan xisaabta lagu darin qodobka taranka).
Sidaa darteed, BlockCache waa qiyaastii 23% wadarta guud ee xogta tani waxay aad ugu dhowdahay xaaladaha dhabta ah ee waxa loo yaqaan BigData. Oo tani waa meesha madadaalada ay ka bilaabato - sababtoo ah sida cad, kaydinta yar ee garaaca, ayaa ka sii daraya waxqabadka. Ka dib oo dhan, haddii aad seegto, waxaad u baahan doontaa inaad qabato shaqo badan - i.e. hoos ugu dhaadhac hawlaha nidaamka wicitaanka. Si kastaba ha ahaatee, tan lama ilaalin karo, markaa aynu eegno dhinac gebi ahaanba ka duwan - maxaa ku dhacaya xogta gudaha kaydka?
Aan fududeyno xaalada oo aan ka qaadno in aan haysano kayd ku habboon 1 shay oo keliya. Halkan waxaa ah tusaale waxa dhici doona markaan isku dayno inaan ku shaqayno mugga xogta 3 jeer ka weyn kaydka, waa inaan:
1. Dhig block 1 khasnada
2. Ka saar xannibaadda 1 khasnadda
3. Dhig block 2 khasnada
4. Ka saar xannibaadda 2 khasnadda
5. Dhig block 3 khasnada
5 ficil oo la dhammaystiray! Si kastaba ha ahaatee, xaaladdan laguma magacaabi karo mid caadi ah; dhab ahaantii, waxaan ku qasbi karnaa HBase inay qabato farabadan oo ah shaqo aan faa'iido lahayn. Waxay si joogto ah u akhridaa xogta kaydka OS-ka, waxay gelisaa BlockCache, kaliya si ay u tuurto isla markiiba sababtoo ah qayb cusub oo xog ah ayaa timid. Animation-ka bilowga boostada wuxuu muujinayaa nuxurka dhibka - Ururiyaha qashinka ayaa kor u kacaya qiyaasta, jawiga ayaa kululeynaya, Greta yar oo fog iyo kulul Iswidhan ayaa xanaaqsan. Anaga dadka IT-ga ah runtii ma jecli marka caruurtu murugaysan yihiin, markaa waxaan bilaabeynaa inaan ka fikirno waxa aan ka qaban karno.
Maxaa dhacaya haddii aanad dhammaan baloogyada ku dhejin kaydka, laakiin kaliya boqolkiiba qaar ka mid ah, si aanu kaydku u buuxdhaafin? Aan ku bilowno innagoo si fudud ugu darsanayna dhowr sadar oo kood ah bilowga shaqada xogta gelinta BlockCache:
public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean inMemory) {
if (cacheDataBlockPercent != 100 && buf.getBlockType().isData()) {
if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent) {
return;
}
}
...
Dhibcaha halkan waa kuwan soo socda: offset waa booska block ee faylka iyo tirooyinka ugu dambeeya ayaa si aan kala sooc lahayn oo siman loo qaybiyaa 00 ilaa 99. Sidaa darteed, waxaan kaliya ka boodi doonaa kuwa ku dhaca xadka aan u baahanahay.
Tusaale ahaan, deji cacheDataBlockPercent = 20 oo arag waxa dhaca:
Natiijadu waa caddahay. Jaantusyada hoose, waxay caddaynaysaa sababta dardargelinta noocan oo kale ah u dhacday - waxaan badbaadinaa wax badan oo GC ah iyada oo aan samaynin shaqada Sisyphean ee ku dhejinta xogta kaydka kaliya si isla markaaba u tuurto qulqulka eeyaha Martian:
Isla mar ahaantaana, isticmaalka CPU wuu kordhaa, laakiin aad ayuu uga yar yahay wax soo saarka:
Waxa kale oo xusid mudan in baloogyada lagu kaydiyo BlockCache ay kala duwan yihiin. Inta badan, qiyaastii 95%, waa xog lafteeda. Inta soo hartayna waa xogta badan, sida filtarrada Bloom ama LEAF_INDEX iyo . Xogtani kuma filna, laakiin aad bay faa'iido u leedahay, sababtoo ah ka hor inta aanad si toos ah u helin xogta, HBase waxay u jeesataa meta si ay u fahmaan in ay lagama maarmaan tahay in la sii raadiyo halkan iyo, haddii ay sidaas tahay, meesha saxda ah ee danta ayaa ku taal.
Sidaa darteed, koodhka waxaan ku aragnaa xaalad hubin buf.getBlockType().isData() Mahadsanid meta-kan, waxaanu kaga tagi doonaa kaydka xaalad kasta.
Hadda aynu kordhino culayska oo aynu xoogaa adkayno sifada hal mar. Tijaabadii ugu horeysay waxaanu samaynay boqolkiiba goynta = 20 iyo BlockCache wax yar ayaa laga faa'iidaystay. Hadda aan dhigno 23% oo aan ku darno 100 dun 5tii daqiiqaba si aan u aragno meesha ay ku qanacsan tahay:
Halkan waxaan ku aragnaa in nooca asalka ah uu isla markiiba ku dhufto saqafka qiyaastii 100 kun oo codsi halkii labaad. Halka balastarku uu siinayo dardargelinta ilaa 300 kun. Isla mar ahaantaana, way caddahay in dardargelinta dheeraadka ah aysan hadda ahayn "free"; Isticmaalka CPU sidoo kale wuu sii kordhayaa.
Si kastaba ha ahaatee, tani ma aha xal aad u qurux badan, tan iyo markii aynaan hore u garanaynin waxa boqolkiiba blocks u baahan yahay in la kaydiyo, waxay ku xiran tahay profile load. Sidaa darteed, waxaa la hirgeliyay hab si toos ah loogu hagaajinayo cabbirkan iyadoo ku xiran tahay dhaqdhaqaaqa hawlaha akhriska.
Saddex doorasho ayaa lagu daray si tan loo xakameeyo:
hbase.lru.cache.culus.ka saarid.tirinta.xad - wuxuu dejiyaa inta jeer ee habka ka saarista xogta khasnadda ay tahay inuu socdo ka hor inta aanaan bilaabin isticmaalka hagaajinta (tusaale ka boodin blocks). Sida caadiga ah waxay la mid tahay MAX_INT = 2147483647 run ahaantiina waxay la macno tahay in sifadu aysan waligiis ku bilaabi doonin la shaqaynta qiimahan. Sababtoo ah habka ka saarista wuxuu bilaabmaa 5 - 10 ilbiriqsi kasta (waxay kuxirantahay culeyska) iyo 2147483647 * 10 / 60 / 60 / 24 / 365 = 680 sano. Si kastaba ha ahaatee, waxaan u dejin karnaa cabbirkan 0 oo ka dhigi karnaa sifada inay shaqeyso isla markiiba ka dib marka la bilaabo.
Si kastaba ha ahaatee, waxa kale oo jira culays ku jira cabbirkan. Haddii culeyskayagu yahay sida akhrinta muddada gaaban (ku dheh inta lagu jiro maalinta) iyo akhrinta muddada dheer (habeenkii) si joogto ah ayaa loo dhex wadaa, markaa waxaan hubin karnaa in muuqaalku uu shidan yahay kaliya marka hawlaha akhrinta dheer ay socdaan.
Tusaale ahaan, waxaan ognahay in wax-akhrinta wakhtiga-gaaban ay caadi ahaan socoto ilaa 1 daqiiqo. Looma baahna in la bilaabo tuurista baloogyada, kaydku ma heli doono wakhti uu ku noqdo mid duugoobay ka dibna waxaanu dejin karnaa cabbirkan la mid ah, tusaale ahaan, 10. Tani waxay horseedi doontaa xaqiiqda ah in hagaajintu ay bilaabi doonto shaqada kaliya marka ay dheer- Erayga akhriska firfircooni wuu bilaabmay, i.e. 100 ilbiriqsi gudahood. Sidaa darteed, haddii aan haysanno wax-akhris waqti-gaaban, markaa dhammaan baloogyada ayaa geli doona kaydka oo waa la heli doonaa (marka laga reebo kuwa laga saari doono algorithm-ka caadiga ah). Oo markaan samayno wax-akhrinta muddada-dheer, muuqaalku waa daaran yahay waxaanan yeelan doonnaa waxqabad aad u sarreeya.
hbase.lru.cache.culus.ka saarid.mb. cabbirka. xaddid - waxay dejisaa inta megabyte ee aan jeclaan lahayn inaan ku ridno khasnadda (iyo, dabcan, ka saarida) 10 ilbiriqsi gudahood. Tilmaamku wuxuu isku dayi doonaa inuu gaaro qiimahan oo uu ilaaliyo. Ujeedadu waa tan: haddii aan ku ridno gigabytes kaydka, markaa waa inaan ka saarnaa gigabytes, tani, sidaan kor ku soo aragnay, waa qaali. Si kastaba ha ahaatee, waa inaadan isku dayin inaad dejiso mid aad u yar, sababtoo ah tani waxay keenaysaa habka boodboodka xannibaadda inuu si degdeg ah u baxo. Adeegayaasha awoodda leh (qiyaastii 20-40 nooc oo ah xudunta jireed), way fiicantahay in la dejiyo qiyaastii 300-400 MB. Loogu talagalay fasalka dhexe (~ 10 cores) 200-300 MB. Nidaamyada daciifka ah (2-5 cores) 50-100 MB waxay noqon kartaa mid caadi ah (aan kuwan lagu tijaabin).
Aynu eegno sida ay tani u shaqeyso: aynu nidhaahno waxaan dejinay hbase.lru.cache.heavy.eviction.mb.size.limit = 500, waxaa jira nooc ka mid ah rar (akhris) ka dibna ~ 10 ilbiriqsi kasta waxaan xisaabineynaa inta bytes ee ahaa guriga laga saaray iyadoo la isticmaalayo qaacidada
Sare-sare = Wadarta Baytka La Xoreeyey (MB) * 100 / Xadka (MB) - 100;
Haddii dhab ahaantii 2000 MB laga saaray, markaa dusha sare waxay la mid tahay:
2000 * 100/500 - 100 = 300%
Algorithms-yadu waxay isku dayaan inay ilaaliyaan wax ka badan tobanaan boqolkiiba, markaa muuqaalku wuxuu yarayn doonaa boqolleyda baloogyada la kaydiyay, si loo hirgeliyo habka toosinta.
Si kastaba ha noqotee, haddii rarku hoos u dhaco, aynu nidhaahno 200 MB oo keliya ayaa laga saaray guriga oo kor u kaca ayaa noqda diidmo (waxa loo yaqaan 'overshooting):
200 * 100 / 500 - 100 = -60%
Taa lidkeeda, sifadu waxay kordhin doontaa boqolkiiba baloogyada kaydsan ilaa inta dusha sare ay noqoto mid togan.
Hoos waxaa ku yaal tusaale ah sida tani u muuqato xogta dhabta ah. Looma baahna in la isku dayo in la gaaro 0%, waa wax aan macquul aheyn. Aad bay u fiican tahay marka ay ku dhowdahay 30 - 100%, tani waxay ka caawisaa in laga fogaado ka bixitaanka degdega ah ee habka hagaajinta inta lagu jiro xilliga gaaban.
hbase.lru.cache.culus.ka saarid.overhead.coefficient - wuxuu dejiyaa sida ugu dhakhsaha badan ee aan u jeclaan lahayn inaan natiijada u helno. Haddii aan si hubaal ah u ogaano in wax-akhrisyadeenu ay inta badan dheer yihiin oo aan doonayn in aan sugno, waxaan kordhin karnaa saamigaan oo aan si dhaqso leh u helno waxqabad sare.
Tusaale ahaan, waxaan dejinay isku-dhafkan = 0.01. Tani waxay ka dhigan tahay in dusha sare (eeg xagga sare) lagu dhufan doono lambarkan natiijada ka soo baxdana waxaa la dhimi doonaa boqolkiiba boqolleyda kaydsan. Aynu ka soo qaadno in Overhead = 300% iyo coefficient = 0.01, markaas boqolleyda baloogyada la kaydiyay waxaa la dhimi doonaa 3%.
Caqli la mid ah "Backpressure" ayaa sidoo kale loo hirgeliyay qiimayaasha kor-dhaafka ah (overshooting). Maaddaama isbeddellada muddada-gaaban ee mugga akhrinta iyo ka saarista ay had iyo jeer suurtogal tahay, habkani wuxuu kuu ogolaanayaa inaad ka fogaato ka bixitaan degdeg ah oo ka yimaada habka hagaajinta. Cadaadiska dhabarku wuxuu leeyahay caqli-gal rogan: hadba sida ay u xoog badan tahay rasaasta xad-dhaafka ah, baloogyada badan ayaa la kaydiyaa.
Koodhka fulinta
LruBlockCache cache = this.cache.get();
if (cache == null) {
break;
}
freedSumMb += cache.evict()/1024/1024;
/*
* Sometimes we are reading more data than can fit into BlockCache
* and it is the cause a high rate of evictions.
* This in turn leads to heavy Garbage Collector works.
* So a lot of blocks put into BlockCache but never read,
* but spending a lot of CPU resources.
* Here we will analyze how many bytes were freed and decide
* decide whether the time has come to reduce amount of caching blocks.
* It help avoid put too many blocks into BlockCache
* when evict() works very active and save CPU for other jobs.
* More delails: https://issues.apache.org/jira/browse/HBASE-23887
*/
// First of all we have to control how much time
// has passed since previuos evict() was launched
// This is should be almost the same time (+/- 10s)
// because we get comparable volumes of freed bytes each time.
// 10s because this is default period to run evict() (see above this.wait)
long stopTime = System.currentTimeMillis();
if ((stopTime - startTime) > 1000 * 10 - 1) {
// Here we have to calc what situation we have got.
// We have the limit "hbase.lru.cache.heavy.eviction.bytes.size.limit"
// and can calculte overhead on it.
// We will use this information to decide,
// how to change percent of caching blocks.
freedDataOverheadPercent =
(int) (freedSumMb * 100 / cache.heavyEvictionMbSizeLimit) - 100;
if (freedSumMb > cache.heavyEvictionMbSizeLimit) {
// Now we are in the situation when we are above the limit
// But maybe we are going to ignore it because it will end quite soon
heavyEvictionCount++;
if (heavyEvictionCount > cache.heavyEvictionCountLimit) {
// It is going for a long time and we have to reduce of caching
// blocks now. So we calculate here how many blocks we want to skip.
// It depends on:
// 1. Overhead - if overhead is big we could more aggressive
// reducing amount of caching blocks.
// 2. How fast we want to get the result. If we know that our
// heavy reading for a long time, we don't want to wait and can
// increase the coefficient and get good performance quite soon.
// But if we don't sure we can do it slowly and it could prevent
// premature exit from this mode. So, when the coefficient is
// higher we can get better performance when heavy reading is stable.
// But when reading is changing we can adjust to it and set
// the coefficient to lower value.
int change =
(int) (freedDataOverheadPercent * cache.heavyEvictionOverheadCoefficient);
// But practice shows that 15% of reducing is quite enough.
// We are not greedy (it could lead to premature exit).
change = Math.min(15, change);
change = Math.max(0, change); // I think it will never happen but check for sure
// So this is the key point, here we are reducing % of caching blocks
cache.cacheDataBlockPercent -= change;
// If we go down too deep we have to stop here, 1% any way should be.
cache.cacheDataBlockPercent = Math.max(1, cache.cacheDataBlockPercent);
}
} else {
// Well, we have got overshooting.
// Mayby it is just short-term fluctuation and we can stay in this mode.
// It help avoid permature exit during short-term fluctuation.
// If overshooting less than 90%, we will try to increase the percent of
// caching blocks and hope it is enough.
if (freedSumMb >= cache.heavyEvictionMbSizeLimit * 0.1) {
// Simple logic: more overshooting - more caching blocks (backpressure)
int change = (int) (-freedDataOverheadPercent * 0.1 + 1);
cache.cacheDataBlockPercent += change;
// But it can't be more then 100%, so check it.
cache.cacheDataBlockPercent = Math.min(100, cache.cacheDataBlockPercent);
} else {
// Looks like heavy reading is over.
// Just exit form this mode.
heavyEvictionCount = 0;
cache.cacheDataBlockPercent = 100;
}
}
LOG.info("BlockCache evicted (MB): {}, overhead (%): {}, " +
"heavy eviction counter: {}, " +
"current caching DataBlock (%): {}",
freedSumMb, freedDataOverheadPercent,
heavyEvictionCount, cache.cacheDataBlockPercent);
freedSumMb = 0;
startTime = stopTime;
}
Bal hadda aan eegno waxaas oo dhan anagoo tusaale dhab ah u adeegsanayna. Waxaan haynaa qoraalka imtixaanka soo socda:
- Aynu bilowno samaynta Scan (25 threads, batch = 100)
- 5 daqiiqo ka dib, ku dar dhowr-helitaan (25 threads, batch = 100)
- 5 daqiiqo ka bacdi, dami helida badan (kaliya iskaanka ayaa hadhsan mar kale)
Waxaan sameynaa laba orod, marka hore hbase.lru.cache.heavy.eviction.count.limit = 10000 (taas oo dhab ahaantii curyaaminaysa astaanta), ka dibna xaddid = 0 (waxay awood u yeelataa).
Diiwaanada hoose waxaan ku aragnaa sida muuqaalku u daaran yahay oo dib u dejinayso Overshooting ilaa 14-71%. Waqti ka waqti culeysku wuu yaraadaa, kaas oo daara Backpressure iyo HBase wuxuu kaydiyaa baloogyo badan mar kale.
Log RegionServer
laga saaray (MB): 0, saamiga 0.0, dusha sare (%): -100, miiska saarista culus: 0, kaydinta DataBlock (%): 100
laga saaray (MB): 0, saamiga 0.0, dusha sare (%): -100, miiska saarista culus: 0, kaydinta DataBlock (%): 100
laga saaray (MB): 2170, saamiga 1.09, dusha sare (%): 985, miiska saarista culus: 1, kaydinta DataBlock (%): 91 <bilow
laga saaray (MB): 3763, saamiga 1.08, dusha sare (%): 1781, miiska saarista culus: 2, kaydinta DataBlock (%): 76
laga saaray (MB): 3306, saamiga 1.07, dusha sare (%): 1553, miiska saarista culus: 3, kaydinta DataBlock (%): 61
laga saaray (MB): 2508, saamiga 1.06, dusha sare (%): 1154, miiska saarista culus: 4, kaydinta DataBlock (%): 50
laga saaray (MB): 1824, saamiga 1.04, dusha sare (%): 812, miiska saarista culus: 5, kaydinta DataBlock (%): 42
laga saaray (MB): 1482, saamiga 1.03, dusha sare (%): 641, miiska saarista culus: 6, kaydinta DataBlock (%): 36
laga saaray (MB): 1140, saamiga 1.01, dusha sare (%): 470, miiska saarista culus: 7, kaydinta DataBlock (%): 32
laga saaray (MB): 913, saamiga 1.0, dusha sare (%): 356, miiska saarista culus: 8, kaydinta DataBlock (%): 29
laga saaray (MB): 912, saamiga 0.89, dusha sare (%): 356, miiska saarista culus: 9, kaydinta DataBlock (%): 26
laga saaray (MB): 684, saamiga 0.76, dusha sare (%): 242, miiska saarista culus: 10, kaydinta DataBlock (%): 24
laga saaray (MB): 684, saamiga 0.61, dusha sare (%): 242, miiska saarista culus: 11, kaydinta DataBlock (%): 22
laga saaray (MB): 456, saamiga 0.51, dusha sare (%): 128, miiska saarista culus: 12, kaydinta DataBlock (%): 21
laga saaray (MB): 456, saamiga 0.42, dusha sare (%): 128, miiska saarista culus: 13, kaydinta DataBlock (%): 20
laga saaray (MB): 456, saamiga 0.33, dusha sare (%): 128, miiska saarista culus: 14, kaydinta DataBlock (%): 19
laga saaray (MB): 342, saamiga 0.33, dusha sare (%): 71, miiska saarista culus: 15, kaydinta DataBlock (%): 19
laga saaray (MB): 342, saamiga 0.32, dusha sare (%): 71, miiska saarista culus: 16, kaydinta DataBlock (%): 19
laga saaray (MB): 342, saamiga 0.31, dusha sare (%): 71, miiska saarista culus: 17, kaydinta DataBlock (%): 19
laga saaray (MB): 228, saamiga 0.3, dusha sare (%): 14, miiska saarista culus: 18, kaydinta DataBlock (%): 19
laga saaray (MB): 228, saamiga 0.29, dusha sare (%): 14, miiska saarista culus: 19, kaydinta DataBlock (%): 19
laga saaray (MB): 228, saamiga 0.27, dusha sare (%): 14, miiska saarista culus: 20, kaydinta DataBlock (%): 19
laga saaray (MB): 228, saamiga 0.25, dusha sare (%): 14, miiska saarista culus: 21, kaydinta DataBlock (%): 19
laga saaray (MB): 228, saamiga 0.24, dusha sare (%): 14, miiska saarista culus: 22, kaydinta DataBlock (%): 19
laga saaray (MB): 228, saamiga 0.22, dusha sare (%): 14, miiska saarista culus: 23, kaydinta DataBlock (%): 19
laga saaray (MB): 228, saamiga 0.21, dusha sare (%): 14, miiska saarista culus: 24, kaydinta DataBlock (%): 19
laga saaray (MB): 228, saamiga 0.2, dusha sare (%): 14, miiska saarista culus: 25, kaydinta DataBlock (%): 19
laga saaray (MB): 228, saamiga 0.17, dusha sare (%): 14, miiska saarista culus: 26, kaydinta DataBlock (%): 19
laga saaray (MB): 456, saamiga 0.17, dusha sare (%): 128, counter saarid culus: 27, kaydinta DataBlock ee hadda (%): 18 < daray helay (laakiin miiska isku mid)
laga saaray (MB): 456, saamiga 0.15, dusha sare (%): 128, miiska saarista culus: 28, kaydinta DataBlock (%): 17
laga saaray (MB): 342, saamiga 0.13, dusha sare (%): 71, miiska saarista culus: 29, kaydinta DataBlock (%): 17
laga saaray (MB): 342, saamiga 0.11, dusha sare (%): 71, miiska saarista culus: 30, kaydinta DataBlock (%): 17
laga saaray (MB): 342, saamiga 0.09, dusha sare (%): 71, miiska saarista culus: 31, kaydinta DataBlock (%): 17
laga saaray (MB): 228, saamiga 0.08, dusha sare (%): 14, miiska saarista culus: 32, kaydinta DataBlock (%): 17
laga saaray (MB): 228, saamiga 0.07, dusha sare (%): 14, miiska saarista culus: 33, kaydinta DataBlock (%): 17
laga saaray (MB): 228, saamiga 0.06, dusha sare (%): 14, miiska saarista culus: 34, kaydinta DataBlock (%): 17
laga saaray (MB): 228, saamiga 0.05, dusha sare (%): 14, miiska saarista culus: 35, kaydinta DataBlock (%): 17
laga saaray (MB): 228, saamiga 0.05, dusha sare (%): 14, miiska saarista culus: 36, kaydinta DataBlock (%): 17
laga saaray (MB): 228, saamiga 0.04, dusha sare (%): 14, miiska saarista culus: 37, kaydinta DataBlock (%): 17
laga saaray (MB): 109, saamiga 0.04, dusha sare (%): -46, counter saarista culus: 37, kaydinta DataBlock (%): 22 <cadaadiska dambe
laga saaray (MB): 798, saamiga 0.24, dusha sare (%): 299, miiska saarista culus: 38, kaydinta DataBlock (%): 20
laga saaray (MB): 798, saamiga 0.29, dusha sare (%): 299, miiska saarista culus: 39, kaydinta DataBlock (%): 18
laga saaray (MB): 570, saamiga 0.27, dusha sare (%): 185, miiska saarista culus: 40, kaydinta DataBlock (%): 17
laga saaray (MB): 456, saamiga 0.22, dusha sare (%): 128, miiska saarista culus: 41, kaydinta DataBlock (%): 16
laga saaray (MB): 342, saamiga 0.16, dusha sare (%): 71, miiska saarista culus: 42, kaydinta DataBlock (%): 16
laga saaray (MB): 342, saamiga 0.11, dusha sare (%): 71, miiska saarista culus: 43, kaydinta DataBlock (%): 16
laga saaray (MB): 228, saamiga 0.09, dusha sare (%): 14, miiska saarista culus: 44, kaydinta DataBlock (%): 16
laga saaray (MB): 228, saamiga 0.07, dusha sare (%): 14, miiska saarista culus: 45, kaydinta DataBlock (%): 16
laga saaray (MB): 228, saamiga 0.05, dusha sare (%): 14, miiska saarista culus: 46, kaydinta DataBlock (%): 16
laga saaray (MB): 222, saamiga 0.04, dusha sare (%): 11, miiska saarista culus: 47, kaydinta DataBlock (%): 16
laga saaray (MB): 104, saamiga 0.03, dusha sare (%): -48, counter ka saarida culus: 47, kaydinta DataBlock (%): 21 < kala gooyn ayaa helaysa
laga saaray (MB): 684, saamiga 0.2, dusha sare (%): 242, miiska saarista culus: 48, kaydinta DataBlock (%): 19
laga saaray (MB): 570, saamiga 0.23, dusha sare (%): 185, miiska saarista culus: 49, kaydinta DataBlock (%): 18
laga saaray (MB): 342, saamiga 0.22, dusha sare (%): 71, miiska saarista culus: 50, kaydinta DataBlock (%): 18
laga saaray (MB): 228, saamiga 0.21, dusha sare (%): 14, miiska saarista culus: 51, kaydinta DataBlock (%): 18
laga saaray (MB): 228, saamiga 0.2, dusha sare (%): 14, miiska saarista culus: 52, kaydinta DataBlock (%): 18
laga saaray (MB): 228, saamiga 0.18, dusha sare (%): 14, miiska saarista culus: 53, kaydinta DataBlock (%): 18
laga saaray (MB): 228, saamiga 0.16, dusha sare (%): 14, miiska saarista culus: 54, kaydinta DataBlock (%): 18
laga saaray (MB): 228, saamiga 0.14, dusha sare (%): 14, miiska saarista culus: 55, kaydinta DataBlock (%): 18
laga saaray (MB): 112, saamiga 0.14, dusha sare (%): -44, counter saarista culus: 55, kaydinta DataBlock (%): 23 <cadaadiska dambe
laga saaray (MB): 456, saamiga 0.26, dusha sare (%): 128, miiska saarista culus: 56, kaydinta DataBlock (%): 22
laga saaray (MB): 342, saamiga 0.31, dusha sare (%): 71, miiska saarista culus: 57, kaydinta DataBlock (%): 22
laga saaray (MB): 342, saamiga 0.33, dusha sare (%): 71, miiska saarista culus: 58, kaydinta DataBlock (%): 22
laga saaray (MB): 342, saamiga 0.33, dusha sare (%): 71, miiska saarista culus: 59, kaydinta DataBlock (%): 22
laga saaray (MB): 342, saamiga 0.33, dusha sare (%): 71, miiska saarista culus: 60, kaydinta DataBlock (%): 22
laga saaray (MB): 342, saamiga 0.33, dusha sare (%): 71, miiska saarista culus: 61, kaydinta DataBlock (%): 22
laga saaray (MB): 342, saamiga 0.33, dusha sare (%): 71, miiska saarista culus: 62, kaydinta DataBlock (%): 22
laga saaray (MB): 342, saamiga 0.33, dusha sare (%): 71, miiska saarista culus: 63, kaydinta DataBlock (%): 22
laga saaray (MB): 342, saamiga 0.32, dusha sare (%): 71, miiska saarista culus: 64, kaydinta DataBlock (%): 22
laga saaray (MB): 342, saamiga 0.33, dusha sare (%): 71, miiska saarista culus: 65, kaydinta DataBlock (%): 22
laga saaray (MB): 342, saamiga 0.33, dusha sare (%): 71, miiska saarista culus: 66, kaydinta DataBlock (%): 22
laga saaray (MB): 342, saamiga 0.32, dusha sare (%): 71, miiska saarista culus: 67, kaydinta DataBlock (%): 22
laga saaray (MB): 342, saamiga 0.33, dusha sare (%): 71, miiska saarista culus: 68, kaydinta DataBlock (%): 22
laga saaray (MB): 342, saamiga 0.32, dusha sare (%): 71, miiska saarista culus: 69, kaydinta DataBlock (%): 22
laga saaray (MB): 342, saamiga 0.32, dusha sare (%): 71, miiska saarista culus: 70, kaydinta DataBlock (%): 22
laga saaray (MB): 342, saamiga 0.33, dusha sare (%): 71, miiska saarista culus: 71, kaydinta DataBlock (%): 22
laga saaray (MB): 342, saamiga 0.33, dusha sare (%): 71, miiska saarista culus: 72, kaydinta DataBlock (%): 22
laga saaray (MB): 342, saamiga 0.33, dusha sare (%): 71, miiska saarista culus: 73, kaydinta DataBlock (%): 22
laga saaray (MB): 342, saamiga 0.33, dusha sare (%): 71, miiska saarista culus: 74, kaydinta DataBlock (%): 22
laga saaray (MB): 342, saamiga 0.33, dusha sare (%): 71, miiska saarista culus: 75, kaydinta DataBlock (%): 22
laga saaray (MB): 342, saamiga 0.33, dusha sare (%): 71, miiska saarista culus: 76, kaydinta DataBlock (%): 22
laga saaray (MB): 21, saamiga 0.33, dusha sare (%): -90, miiska saarista culus: 76, kaydinta DataBlock (%): 32
laga saaray (MB): 0, saamiga 0.0, dusha sare (%): -100, miiska saarista culus: 0, kaydinta DataBlock (%): 100
laga saaray (MB): 0, saamiga 0.0, dusha sare (%): -100, miiska saarista culus: 0, kaydinta DataBlock (%): 100
Baadhitaannada ayaa loo baahday si ay u muujiyaan qaab isku mid ah qaabka garaafka xidhiidhka ka dhexeeya laba qaybood oo kayd ah - hal (halkaas oo baloogyo aan waligood hore loo codsan) iyo kuwo badan (xogta "la codsaday" ugu yaraan hal mar ayaa halkan lagu kaydiyaa):
Ugu dambeyntiina, waa maxay hawlgalka cabbiraadaha u eg qaabka garaafka. Marka la barbardhigo, khasnadda gebi ahaanba waa la damiyay bilowgii, ka dib HBase waxaa lagu bilaabay kaydinta iyo dib u dhigista bilawga shaqada hagaajinta 5 daqiiqo (30 wareegyo ka saarid).
Koodhka buuxa waxa laga heli karaa Codsiga Jiid github.
Si kastaba ha ahaatee, 300 kun oo akhrin ah halkii ilbiriqsi maaha dhammaan waxa lagu gaari karo qalabkan xaaladahan. Xaqiiqdu waxay tahay marka aad u baahan tahay inaad xogta ka hesho HDFS, habka ShortCircuitCache (oo hadda ka dib loo yaqaan SSC) ayaa la isticmaalaa, kaasoo kuu oggolaanaya inaad si toos ah u gasho xogta, adigoo iska ilaalinaya isdhexgalka shabakada.
Xog-ururinta ayaa muujisay in inkasta oo habkani uu bixiyo faa'iido weyn, sidoo kale mararka qaarkood waxay noqotaa dhalo, sababtoo ah ku dhawaad dhammaan hawlgallada culus waxay ka dhacaan gudaha qufulka, taas oo keenta in la xiro inta badan.
Markaan taas ogaannay, waxaan ogaannay in dhibaatada looga gudbi karo iyadoo la abuurayo kooxo SSC ah oo madax-bannaan:
private final ShortCircuitCache[] shortCircuitCache;
...
shortCircuitCache = new ShortCircuitCache[this.clientShortCircuitNum];
for (int i = 0; i < this.clientShortCircuitNum; i++)
this.shortCircuitCache[i] = new ShortCircuitCache(…);
Ka dibna la shaqee iyaga, marka laga reebo isgoysyada sidoo kale tirada ugu dambeysa ee la dhimay:
public ShortCircuitCache getShortCircuitCache(long idx) {
return shortCircuitCache[(int) (idx % clientShortCircuitNum)];
}
Hadda waxaad bilaabi kartaa tijaabada Si tan loo sameeyo, waxaan akhrin doonaa faylasha HDFS oo leh codsi fudud oo badan oo xadhig ah. Deji xuduudaha:
conf.set("dfs.client.read.shortcircuit", "true");
conf.set("dfs.client.read.shortcircuit.buffer.size", "65536"); // по дефолту = 1 МБ и это сильно замедляет чтение, поэтому лучше привести в соответствие к реальным нуждам
conf.set("dfs.client.short.circuit.num", num); // от 1 до 10
Oo kaliya akhri faylasha:
FSDataInputStream in = fileSystem.open(path);
for (int i = 0; i < count; i++) {
position += 65536;
if (position > 900000000)
position = 0L;
int res = in.read(position, byteBuffer, 0, 65536);
}
Koodhkan waxaa lagu fuliyaa qaybo kala duwan waxaanan kordhin doonaa tirada isku mar la akhriyo faylasha (laga bilaabo 10 ilaa 200 - dhidibka jiifka) iyo tirada khasnadaha (laga bilaabo 1 ilaa 10 - garaafyada). dhidibka toosan wuxuu muujinayaa dardargelinta ka dhalanaysa kororka SSC marka loo eego kiiska marka uu jiro hal kayd oo keliya.
Sida loo akhriyo garaafyada: Waqtiga fulinta ee 100 kun waxa lagu akhriyaa 64 KB blocks oo leh hal kayd waxay u baahan tahay 78 ilbiriqsi. Halka 5 cache ay ku qaadanayso 16 sekan. Kuwaas. waxaa jira dardargelin ~5 jeer. Sida ka muuqata garaafka, saamayntu maaha mid laga dareemo tiro yar oo isbarbar socda, waxa ay bilaabataa in ay ciyaarto door muuqda marka la akhriyo in ka badan 50 dun, waxa kale oo la ogaan karaa in la kordhiyo tirada SSC oo ah 6. iyo korka waxay ku siinaysaa koror waxqabad oo aad u yar.
Xusuusin 1: maadaama natiijooyinka imtixaanku ay yihiin kuwo isbedel badan (hoos eeg), 3 orod ayaa la sameeyay oo qiyamka natiijada ayaa la isku celceliyay.
Xusuusin 2: Faa'iidada waxqabadka ee habaynta helitaanka randomku waa isku mid, inkasta oo gelitaanka laftiisu uu yara gaabis ah.
Si kastaba ha ahaatee, waa lagama maarmaan in la caddeeyo, si ka duwan kiiskii HBase, dardargelintani had iyo jeer maaha bilaash. Halkan waxaan ku "fureynaa" awooda CPU si uu u qabto shaqo badan, halkii uu ka laadlaadsan lahaa qufullada.
Halkan waxaad ku arki kartaa in, guud ahaan, kororka tirada kaydinta ay siinayso ku dhawaad korodhka isticmaalka CPU. Si kastaba ha ahaatee, waxaa jira wax yar oo dheeraad ah oo la isku daray.
Tusaale ahaan, aan si qoto dheer u eegno goobta SSC = 3. Kor u kaca waxqabadka ee xadka waa qiyaastii 3.3 jeer. Hoos waxaa ku yaal natiijooyinka dhammaan saddexda orod ee kala duwan.
Halka isticmaalka CPU uu kordho ilaa 2.8 jeer. Farqiga u dhexeeya maaha mid aad u weyn, laakiin Greta yar ayaa durba faraxsan waxaana laga yaabaa inay waqti u hesho inay dhigto dugsiga oo ay casharro ku qaadato.
Sidaa darteed, tani waxay saameyn togan ku yeelan doontaa qalab kasta oo isticmaala marinka badan ee HDFS (tusaale Spark, iwm.), waase haddii koodhka codsigu uu khafiif yahay (tusaale, furku wuxuu ku yaal dhinaca macmiilka HDFS) oo uu jiro awood CPU oo bilaash ah . Si aad u hubiso, aynu tijaabino saamaynta isku darka isticmaalka BlockCache optimization iyo habaynta SSC ee akhriska HBase ay yeelan doonto.
Waxaa la arki karaa in xaaladahan oo kale saameyntu aysan u weynayn sida imtixaannada la safeeyey (akhrinta iyada oo aan wax habayn), laakiin waa suurtogal in la tuujiyo 80K dheeraad ah halkan. Si wada jir ah, labada kor-u-qaadiddu waxay bixiyaan xawaare ilaa 4x ah.
PR ayaa sidoo kale loo sameeyay hagaajintan , kaas oo la isku daray oo shaqadani waxay diyaar noqon doontaa sii daynta mustaqbalka.
Ugu dambayntii, waxa ay ahayd mid xiiso leh in la is barbar dhigo waxqabadka akhriska ee xog-ururin ballaadhan oo la mid ah, Cassandra iyo HBase.
Si tan loo sameeyo, waxaan bilownay tusaalooyin utility tijaabinta culeyska YCSB ee ka yimid laba martigal (800 dunta guud ahaan). Dhinaca server-ka - 4 xaaladood oo ah RegionServer iyo Cassandra ee 4 martigeliyaha (ma aha kuwa ay macaamiishu ku socdaan, si looga fogaado saameyntooda). Wax-akhrisyadu waxay ka yimaadeen jaantusyada cabbirka:
HBase - 300 GB HDFS (100 GB xog saafi ah)
Cassandra - 250 GB (cutubka soo noqnoqda = 3)
Kuwaas. mugga wuxuu ahaa qiyaastii isku mid (HBase in yar oo dheeraad ah).
Halbeegyada HBase:
dfs.client.short.circuit.num = 5 (HDFS hagaajinta macmiilka)
hbase.lru.cache.culus.eviction.count.limit = 30 Tani waxay ka dhigan tahay in balastarku uu bilaabi doono inuu shaqeeyo 30 ka saarista ka dib (~ 5 daqiiqo)
hbase.lru.cache.culus.ka saarid.mb.size.limit = 300 - mugga bartilmaameedka kaydinta iyo ka saarida
Logyada YCSB waa la miisaamay waxaana lagu soo ururiyay garaafyada Excel:
Sida aad arki karto, hagaajintaani waxay suurtogal ka dhigaysaa in la isbarbardhigo waxqabadka xogta macluumaadkan ee shuruudahan oo lagu gaaro 450 kun oo akhrin halkii labaad.
Waxaan rajaynaynaa in macluumaadkani u noqon karo mid waxtar u leh qof inta lagu jiro halganka xiisaha leh ee wax soo saarka.
Source: www.habr.com