Kuchita bwino ndi chimodzi mwazofunikira kwambiri mukamagwira ntchito ndi data yayikulu. Mu dipatimenti yotsitsa deta ku Sberbank, timapopera pafupifupi zochitika zonse mumtambo wathu wa data wa Hadoop ndipo chifukwa chake timakhala ndi chidziwitso chochuluka kwambiri. Mwachilengedwe, nthawi zonse timakhala tikuyang'ana njira zosinthira magwiridwe antchito, ndipo tsopano tikufuna kukuuzani momwe tidakwanitsira kukhazikitsa RegionServer HBase ndi kasitomala wa HDFS, chifukwa chomwe tidatha kukulitsa kuthamanga kwa ntchito zowerengera.
Komabe, musanapitirire pazomwe zasintha, ndi bwino kunena za zoletsa zomwe, makamaka, sizingalephereke ngati mutakhala pa HDD.
Chifukwa chiyani HDD ndi kuwerengera mwachangu kwa Random Access sizigwirizana
Monga mukudziwa, HBase, ndi nkhokwe zina zambiri, sungani deta mu midadada ya ma kilobytes angapo kukula kwake. Mwachikhazikitso ndi pafupifupi 64 KB. Tsopano tiyeni tiyerekeze kuti tikufunika kupeza ma byte 100 okha ndipo tikupempha HBase kutipatsa izi pogwiritsa ntchito kiyi inayake. Popeza chipika kukula HFiles ndi 64 KB, pempho adzakhala 640 nthawi zazikulu (mphindi chabe!) kuposa zofunika.
Chotsatira, popeza pempholi lidzadutsa mu HDFS ndi makina ake osungira metadata ShortCircuitCache (zomwe zimalola mwayi wofikira ku mafayilo), izi zimapangitsa kuti muwerenge kale 1 MB kuchokera pa disk. Komabe, izi zitha kusinthidwa ndi parameter dfs.client.read.shortcircuit.buffer.size ndipo nthawi zambiri zimakhala zomveka kuchepetsa mtengo uwu, mwachitsanzo mpaka 126 KB.
Tinene kuti timachita izi, koma kuwonjezera apo, tikayamba kuwerenga deta kudzera mu java api, monga ntchito monga FileChannel.read ndikupempha opareshoni kuti awerenge kuchuluka kwa deta, amawerenga "ngati" 2 nthawi zambiri. ,ndi. 256 KB kwa ife. Izi ndichifukwa choti java ilibe njira yosavuta yokhazikitsira mbendera ya FADV_RANDOM kuti mupewe izi.
Zotsatira zake, kuti tipeze ma byte athu 100, nthawi 2600 zambiri zimawerengedwa pansi pa hood. Zingawonekere kuti yankho ndi lodziwikiratu, tiyeni tichepetse kukula kwa chipika kukhala kilobyte, tiyike mbendera yomwe tatchulayi ndikupeza chiwongolero chachikulu chowunikira. Koma vuto ndiloti pochepetsa kukula kwa chipika ndi ka 2, timachepetsanso kuchuluka kwa ma byte omwe amawerengedwa pa nthawi imodzi ndi kawiri.
Kupindula kwina pakuyika mbendera ya FADV_RADOM kutha kupezeka, koma kokha ndi ulusi wambiri komanso kukula kwake kwa 128 KB, koma uku ndikokwera kwa makumi angapo peresenti:
Kuyesedwa kunachitika pamafayilo a 100, kukula kwa 1 GB iliyonse ndipo kuli pa 10 HDDs.
Tiyeni tiwerenge zomwe tingathe, makamaka, kudalira pa liwiro ili:
Tiyerekeze kuti timawerenga kuchokera ku disks 10 pa liwiro la 280 MB / sec, i.e. 3 miliyoni nthawi 100 mabayiti. Koma monga tikukumbukira, zomwe timafunikira ndizochepera nthawi 2600 kuposa zomwe zimawerengedwa. Chifukwa chake, timagawaniza 3 miliyoni ndi 2600 ndikupeza 1100 mbiri pa sekondi iliyonse.
Zokhumudwitsa, sichoncho? Ndicho chilengedwe Kupezeka Mwachisawawa kupeza deta pa HDD - mosasamala kanthu za kukula kwa chipika. Uwu ndiye malire opezeka mwachisawawa ndipo palibe nkhokwe yomwe ingathe kufinya zambiri pamikhalidwe yotere.
Nanga ma database amafika bwanji kuthamanga kwambiri? Kuti tiyankhe funsoli, tiyeni tione zimene zikuchitika pachithunzi chotsatirachi:
Apa tikuwona kuti kwa mphindi zingapo zoyamba liwiro lilidi pafupifupi ma rekodi chikwi pamphindikati. Komabe, kupitilira apo, chifukwa chakuti zambiri zimawerengedwa kuposa zomwe zidafunsidwa, deta imathera mu buff / cache ya opareting'i sisitimu (linux) ndipo liwiro limakwera mpaka 60 zikwizikwi pamphindikati.
Chifukwa chake, kupitilira apo tithana ndi kuthamangitsa mwayi wongopeza zomwe zili mu cache ya OS kapena zomwe zili mu zida zosungira za SSD/NVMe za liwiro lofananira.
Kwa ife, tidzayesa pa benchi ya ma seva 4, omwe amalipidwa motere:
CPU: Xeon E5-2680 v4 @ 2.40GHz 64 ulusi.
Kukumbukira: 730 GB.
mtundu wa java: 1.8.0_111
Ndipo apa mfundo yaikulu ndi kuchuluka kwa deta mu matebulo omwe ayenera kuwerengedwa. Chowonadi ndi chakuti ngati muwerenga deta kuchokera patebulo yomwe yayikidwa kwathunthu mu cache ya HBase, ndiye kuti sichingafike powerenga kuchokera ku buff / cache ya opareshoni. Chifukwa HBase mwachisawawa imagawira 40% ya kukumbukira kumapangidwe otchedwa BlockCache. Kwenikweni iyi ndi ConcurrentHashMap, pomwe fungulo ndi dzina lafayilo + kuchotsera kwa chipika, ndipo mtengo wake ndi deta yeniyeni pazida izi.
Choncho, powerenga kokha kuchokera ku dongosololi, ife , ngati zopempha miliyoni pa sekondi iliyonse. Koma tiyerekeze kuti sitingathe kugawa mazana a ma gigabytes a kukumbukira pazosowa za database, chifukwa pali zinthu zina zambiri zothandiza zomwe zikuyenda pa maseva awa.
Mwachitsanzo, kwa ife, voliyumu ya BlockCache pa RS imodzi ili pafupi 12 GB. Tinatera ma RS awiri pa mfundo imodzi, i.e. 96 GB yaperekedwa ku BlockCache pama node onse. Ndipo pali zambiri zochulukirapo, mwachitsanzo, zikhale 4 matebulo, zigawo 130 chilichonse, momwe mafayilo ndi 800 MB kukula, oponderezedwa ndi FAST_DIFF, i.e. chiwerengero cha 410 GB (iyi ndi deta yoyera, i.e. popanda kuganizira zobwerezabwereza).
Choncho, BlockCache ndi pafupifupi 23% yokha ya chiwerengero cha deta ndipo izi ziri pafupi kwambiri ndi zochitika zenizeni zomwe zimatchedwa BigData. Ndipo apa ndipamene zosangalatsa zimayambira - chifukwa mwachiwonekere, kuchepa kwa cache kumapangitsa kuti ntchito ikhale yoipitsitsa. Pambuyo pake, ngati muphonya, mudzayenera kugwira ntchito zambiri - i.e. kupita pansi kuyitana ntchito zadongosolo. Komabe, izi sizingapewedwe, kotero tiyeni tiwone mbali yosiyana kwambiri - chimachitika ndi chiyani ndi data mkati mwa cache?
Tiyeni tifewetse zinthuzo ndikuganiza kuti tili ndi posungira yomwe imangokwanira chinthu chimodzi. Nachi chitsanzo cha zomwe zidzachitike tikayesa kugwira ntchito ndi voliyumu ya data kuwirikiza katatu kuposa posungira, tidzayenera:
1. Ikani chipika 1 mu cache
2. Chotsani chipika 1 kuchokera ku cache
3. Ikani chipika 2 mu cache
4. Chotsani chipika 2 kuchokera ku cache
5. Ikani chipika 3 mu cache
Zochita 5 zatha! Komabe, izi sizingatchulidwe kuti ndizabwinobwino, kwenikweni, tikukakamiza HBase kuti achite ntchito zambiri zopanda pake. Imawerenga nthawi zonse deta kuchokera ku cache ya OS, kuiyika ku BlockCache, kungoyitaya nthawi yomweyo chifukwa gawo latsopano la deta lafika. Makanema omwe ali koyambirira kwa positi akuwonetsa chomwe chimayambitsa vutoli - Wotolera Zinyalala akuyenda pang'onopang'ono, mlengalenga ukutentha, Greta wakutali komanso wotentha waku Sweden akukwiya. Ndipo ife anthu a IT sitikonda kwenikweni pamene ana ali achisoni, kotero timayamba kuganizira zomwe tingachite nazo.
Nanga bwanji ngati simuyika midadada yonse mu cache, koma peresenti yokha ya izo, kuti posungira zisasefukire? Tiyeni tiyambe ndikungowonjezera mizere ingapo ya code kumayambiriro kwa ntchito yoyika deta mu BlockCache:
public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean inMemory) {
if (cacheDataBlockPercent != 100 && buf.getBlockType().isData()) {
if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent) {
return;
}
}
...
Mfundo apa ndi iyi: offset ndi malo a chipika mu fayilo ndipo manambala ake otsiriza amagawidwa mwachisawawa komanso mofanana kuchokera ku 00 mpaka 99. Choncho, tidzangodumpha zomwe zimagwera mumtundu womwe tikufuna.
Mwachitsanzo, ikani cacheDataBlockPercent = 20 ndikuwona zomwe zimachitika:
Zotsatira zake ndi zoonekeratu. M'ma graph omwe ali pansipa, zikuwonekeratu chifukwa chake kufulumira koteroko kunachitika - timasunga zinthu zambiri za GC popanda kuchita ntchito ya Sisyphean yoyika deta mu cache kuti nthawi yomweyo igwetse pansi kukhetsa kwa agalu a Martian:
Nthawi yomweyo, kugwiritsa ntchito kwa CPU kumawonjezeka, koma ndizochepa kwambiri kuposa zokolola:
Ndizoyeneranso kudziwa kuti midadada yosungidwa mu BlockCache ndi yosiyana. Ambiri, pafupifupi 95%, ndi deta yokha. Ndipo zina zonse ndi metadata, monga zosefera za Bloom kapena LEAF_INDEX ndi . Deta iyi sikokwanira, koma ndiyothandiza kwambiri, chifukwa musanayambe kupeza deta mwachindunji, HBase amatembenukira ku meta kuti amvetse ngati kuli kofunikira kufufuza apa ndipo, ngati ndi choncho, komwe kuli chipika cha chidwi.
Chifukwa chake, mu code timawona cheke buf.getBlockType().isData() ndipo chifukwa cha meta iyi, tidzayisiya mu cache mulimonse.
Tsopano tiyeni tiwonjeze katunduyo ndikumangitsa pang'ono mawonekedwewo kamodzi. Pachiyeso choyamba tinapanga cutoff percentage = 20 ndipo BlockCache idagwiritsidwa ntchito pang'ono. Tsopano tiyeni tiyike ku 23% ndikuwonjezera ulusi 100 mphindi 5 zilizonse kuti tiwone kuti machulukitsidwe amapezeka pati:
Apa tikuwona kuti mtundu wapachiyambi pafupifupi nthawi yomweyo umagunda padenga pa zopempha 100 zikwi pamphindi. Pomwe chigambachi chimapereka mathamangitsidwe mpaka 300 zikwi. Nthawi yomweyo, zikuwonekeratu kuti kuthamangitsa kwina sikulinso "kwaulere"; Kugwiritsa ntchito CPU kukuchulukiranso.
Komabe, iyi si njira yabwino kwambiri, chifukwa sitikudziwa pasadakhale kuchuluka kwa midadada yomwe iyenera kusungidwa, zimatengera mbiri ya katundu. Chifukwa chake, njira idakhazikitsidwa kuti ingosintha izi potengera ntchito yowerengera.
Zosankha zitatu zawonjezedwa kuwongolera izi:
hbase.lru.cache.heavy.eviction.count.limit - imakhazikitsa kangati kangati kachitidwe kochotsa deta mu cache iyenera kuchitika tisanayambe kugwiritsa ntchito kukhathamiritsa (ie kudumpha midadada). Mwachikhazikitso ndizofanana ndi MAX_INT = 2147483647 ndipo kwenikweni zikutanthauza kuti mawonekedwewo sadzayamba kugwira ntchito ndi mtengowu. Chifukwa ndondomeko kuthamangitsidwa akuyamba aliyense 5 - 10 masekondi (zimadalira katundu) ndi 2147483647 * 10/60/60/24/365 = 680 zaka. Komabe, titha kukhazikitsa parameter iyi ku 0 ndikupanga mawonekedwewo kuti agwire ntchito mukangoyambitsa.
Komabe, palinso malipiro mu parameter iyi. Ngati katundu wathu ndi woti kuwerengera kwakanthawi kochepa (kunena masana) ndi kuwerenga kwanthawi yayitali (usiku) kumalumikizidwa mosalekeza, ndiye kuti titha kuwonetsetsa kuti mawonekedwewo amayatsidwa pokhapokha ntchito yowerenga nthawi yayitali ikuchitika.
Mwachitsanzo, tikudziwa kuti kuwerengera kwakanthawi kochepa kumatha pafupifupi mphindi imodzi. Palibe chifukwa choyambira kutulutsa midadada, posungira sadzakhala ndi nthawi yoti ikhale yachikale ndipo titha kuyika chizindikiro ichi chofanana ndi, mwachitsanzo, 1. Izi zipangitsa kuti kukhathamiritsa kumayamba kugwira ntchito pokhapokha ngati nthawi yayitali- nthawi yowerenga mwachangu yayamba, i.e. mu masekondi 10. Chifukwa chake, ngati tili ndi kuwerenga kwakanthawi kochepa, ndiye kuti midadada yonse idzalowa mu cache ndipo ipezeka (kupatula omwe adzathamangitsidwe ndi algorithm yokhazikika). Ndipo tikamawerenga kwanthawi yayitali, mawonekedwewo amayatsidwa ndipo timakhala ndi magwiridwe antchito apamwamba kwambiri.
hbase.lru.cache.heavy.eviction.mb.size.limit - imakhazikitsa ma megabytes angati omwe tingafune kuyika mu cache (ndipo, kutulutsa) mumasekondi 10. Gawoli liyesa kufikira mtengowu ndikuusamalira. Mfundo yake ndi iyi: ngati tikukankhira gigabytes mu cache, ndiye kuti tithamangitse ma gigabytes, ndipo izi, monga tawonera pamwambapa, ndizokwera mtengo kwambiri. Komabe, musayese kuyiyika yaying'ono kwambiri, chifukwa izi zipangitsa kuti block skip mode ituluke msanga. Kwa ma seva amphamvu (pafupifupi 20-40 ma cores), ndikwabwino kukhazikitsa pafupifupi 300-400 MB. Kwa kalasi yapakati (~ 10 cores) 200-300 MB. Kwa machitidwe ofooka (2-5 cores) 50-100 MB akhoza kukhala abwinobwino (osayesedwa pa izi).
Tiyeni tiwone momwe izi zimagwirira ntchito: tinene kuti takhazikitsa hbase.lru.cache.heavy.eviction.mb.size.limit = 500, pali mtundu wina wa katundu (kuwerenga) ndiyeno ~ masekondi 10 aliwonse timawerengera kuchuluka kwa ma byte omwe anali. kuthamangitsidwa pogwiritsa ntchito formula:
Pamwamba = Chiwerengero cha Ma Byte Omasulidwa (MB) * 100 / Malire (MB) - 100;
Ngati kwenikweni 2000 MB adathamangitsidwa, ndiye kuti Overhead ndi yofanana ndi:
2000 * 100 / 500 - 100 = 300%
Ma algorithms amayesa kusunga zosaposa makumi angapo peresenti, kotero mawonekedwewo amachepetsa kuchuluka kwa midadada yosungidwa, potero akugwiritsa ntchito makina osinthira okha.
Komabe, ngati katunduyo atsika, tinene kuti 200 MB yokha ndiyomwe imachotsedwa ndipo Kupitilira muyeso kumakhala koyipa (komwe kumatchedwa overshooting):
200 * 100 / 500 - 100 = -60%
M'malo mwake, chiwonetserochi chidzawonjezera kuchuluka kwa midadada yosungidwa mpaka Kumutu kukhale kolimbikitsa.
Pansipa pali chitsanzo cha momwe izi zimawonekera pa data yeniyeni. Palibe chifukwa choyesera kufikira 0%, sizingatheke. Ndibwino kwambiri pamene ili pafupi ndi 30 - 100%, izi zimathandiza kupewa kutuluka msanga kuchokera kumachitidwe okhathamiritsa panthawi ya maopaleshoni akanthawi kochepa.
hbase.lru.cache.heavy.eviction.overhead.coefficient - imayika momwe timafunira kuti tipeze zotsatira. Ngati tikudziwa motsimikiza kuti zowerenga zathu nthawi zambiri zimakhala zazitali ndipo sitikufuna kudikirira, titha kuwonjezera chiŵerengerochi ndikuchita bwino kwambiri.
Mwachitsanzo, timayika coefficient = 0.01. Izi zikutanthauza kuti Kupitilira (onani pamwambapa) kuchulukitsidwa ndi nambalayi ndi zotsatira zake ndipo kuchuluka kwa midadada yosungidwa kudzachepetsedwa. Tiyerekeze kuti Overhead = 300% ndi coefficient = 0.01, ndiye kuti kuchuluka kwa midadada yosungidwa kudzachepetsedwa ndi 3%.
Lingaliro lofananira la "Backpressure" limakhazikitsidwanso pamakhalidwe olakwika a Overhead (overshooting). Popeza kusinthasintha kwakanthawi kochepa kwa kuchuluka kwa zowerengera ndi kuthamangitsidwa kumakhala kotheka nthawi zonse, makinawa amakulolani kuti mupewe kutuluka msanga kuchokera pakukhathamiritsa. Kubwerera mmbuyo kumakhala ndi malingaliro opotoka: kukulitsa kukulirakulira, mipiringidzo yambiri imasungidwa.
Kodi kukhazikitsa
LruBlockCache cache = this.cache.get();
if (cache == null) {
break;
}
freedSumMb += cache.evict()/1024/1024;
/*
* Sometimes we are reading more data than can fit into BlockCache
* and it is the cause a high rate of evictions.
* This in turn leads to heavy Garbage Collector works.
* So a lot of blocks put into BlockCache but never read,
* but spending a lot of CPU resources.
* Here we will analyze how many bytes were freed and decide
* decide whether the time has come to reduce amount of caching blocks.
* It help avoid put too many blocks into BlockCache
* when evict() works very active and save CPU for other jobs.
* More delails: https://issues.apache.org/jira/browse/HBASE-23887
*/
// First of all we have to control how much time
// has passed since previuos evict() was launched
// This is should be almost the same time (+/- 10s)
// because we get comparable volumes of freed bytes each time.
// 10s because this is default period to run evict() (see above this.wait)
long stopTime = System.currentTimeMillis();
if ((stopTime - startTime) > 1000 * 10 - 1) {
// Here we have to calc what situation we have got.
// We have the limit "hbase.lru.cache.heavy.eviction.bytes.size.limit"
// and can calculte overhead on it.
// We will use this information to decide,
// how to change percent of caching blocks.
freedDataOverheadPercent =
(int) (freedSumMb * 100 / cache.heavyEvictionMbSizeLimit) - 100;
if (freedSumMb > cache.heavyEvictionMbSizeLimit) {
// Now we are in the situation when we are above the limit
// But maybe we are going to ignore it because it will end quite soon
heavyEvictionCount++;
if (heavyEvictionCount > cache.heavyEvictionCountLimit) {
// It is going for a long time and we have to reduce of caching
// blocks now. So we calculate here how many blocks we want to skip.
// It depends on:
// 1. Overhead - if overhead is big we could more aggressive
// reducing amount of caching blocks.
// 2. How fast we want to get the result. If we know that our
// heavy reading for a long time, we don't want to wait and can
// increase the coefficient and get good performance quite soon.
// But if we don't sure we can do it slowly and it could prevent
// premature exit from this mode. So, when the coefficient is
// higher we can get better performance when heavy reading is stable.
// But when reading is changing we can adjust to it and set
// the coefficient to lower value.
int change =
(int) (freedDataOverheadPercent * cache.heavyEvictionOverheadCoefficient);
// But practice shows that 15% of reducing is quite enough.
// We are not greedy (it could lead to premature exit).
change = Math.min(15, change);
change = Math.max(0, change); // I think it will never happen but check for sure
// So this is the key point, here we are reducing % of caching blocks
cache.cacheDataBlockPercent -= change;
// If we go down too deep we have to stop here, 1% any way should be.
cache.cacheDataBlockPercent = Math.max(1, cache.cacheDataBlockPercent);
}
} else {
// Well, we have got overshooting.
// Mayby it is just short-term fluctuation and we can stay in this mode.
// It help avoid permature exit during short-term fluctuation.
// If overshooting less than 90%, we will try to increase the percent of
// caching blocks and hope it is enough.
if (freedSumMb >= cache.heavyEvictionMbSizeLimit * 0.1) {
// Simple logic: more overshooting - more caching blocks (backpressure)
int change = (int) (-freedDataOverheadPercent * 0.1 + 1);
cache.cacheDataBlockPercent += change;
// But it can't be more then 100%, so check it.
cache.cacheDataBlockPercent = Math.min(100, cache.cacheDataBlockPercent);
} else {
// Looks like heavy reading is over.
// Just exit form this mode.
heavyEvictionCount = 0;
cache.cacheDataBlockPercent = 100;
}
}
LOG.info("BlockCache evicted (MB): {}, overhead (%): {}, " +
"heavy eviction counter: {}, " +
"current caching DataBlock (%): {}",
freedSumMb, freedDataOverheadPercent,
heavyEvictionCount, cache.cacheDataBlockPercent);
freedSumMb = 0;
startTime = stopTime;
}
Tiyeni tsopano tione zonsezi pogwiritsa ntchito chitsanzo chenicheni. Tili ndi mayeso otsatirawa:
- Tiyeni tiyambe kupanga Scan (25 ulusi, batch = 100)
- Pambuyo pa mphindi zisanu, onjezani zopeza zambiri (zingwe 5, batch = 25)
- Pambuyo pa mphindi 5, zimitsani ma multi-gets (kujambula kokha kumatsaliranso)
Timachita maulendo awiri, choyamba hbase.lru.cache.heavy.eviction.count.limit = 10000 (yomwe imalepheretsa mawonekedwewo), ndikuyika malire = 0 (imathandizira).
M'zipika pansipa tikuwona momwe mawonekedwewo amayatsidwa ndikukhazikitsanso Overshooting ku 14-71%. Nthawi ndi nthawi katundu amachepetsa, zomwe zimayatsa Backpressure ndi HBase caches zambiri midadada kachiwiri.
Log RegionServer
kuthamangitsidwa (MB): 0, chiŵerengero cha 0.0, pamwamba (%): -100, kauntala yothamangitsa kwambiri: 0, DataBlock yamakono (%): 100
kuthamangitsidwa (MB): 0, chiŵerengero cha 0.0, pamwamba (%): -100, kauntala yothamangitsa kwambiri: 0, DataBlock yamakono (%): 100
kuthamangitsidwa (MB): 2170, chiŵerengero cha 1.09, pamwamba (%): 985, katundu wotulutsa katundu: 1, caching panopa DataBlock (%): 91 <kuyamba
othamangitsidwa (MB): 3763, chiŵerengero cha 1.08, pamwamba (%): 1781, katundu wotulutsa katundu: 2, caching panopa DataBlock (%): 76
othamangitsidwa (MB): 3306, chiŵerengero cha 1.07, pamwamba (%): 1553, katundu wotulutsa katundu: 3, caching panopa DataBlock (%): 61
othamangitsidwa (MB): 2508, chiŵerengero cha 1.06, pamwamba (%): 1154, katundu wotulutsa katundu: 4, caching panopa DataBlock (%): 50
othamangitsidwa (MB): 1824, chiŵerengero cha 1.04, pamwamba (%): 812, katundu wotulutsa katundu: 5, caching panopa DataBlock (%): 42
othamangitsidwa (MB): 1482, chiŵerengero cha 1.03, pamwamba (%): 641, katundu wotulutsa katundu: 6, caching panopa DataBlock (%): 36
othamangitsidwa (MB): 1140, chiŵerengero cha 1.01, pamwamba (%): 470, katundu wotulutsa katundu: 7, caching panopa DataBlock (%): 32
othamangitsidwa (MB): 913, chiŵerengero cha 1.0, pamwamba (%): 356, katundu wotulutsa katundu: 8, caching panopa DataBlock (%): 29
othamangitsidwa (MB): 912, chiŵerengero cha 0.89, pamwamba (%): 356, katundu wotulutsa katundu: 9, caching panopa DataBlock (%): 26
othamangitsidwa (MB): 684, chiŵerengero cha 0.76, pamwamba (%): 242, katundu wotulutsa katundu: 10, caching panopa DataBlock (%): 24
othamangitsidwa (MB): 684, chiŵerengero cha 0.61, pamwamba (%): 242, katundu wotulutsa katundu: 11, caching panopa DataBlock (%): 22
othamangitsidwa (MB): 456, chiŵerengero cha 0.51, pamwamba (%): 128, katundu wotulutsa katundu: 12, caching panopa DataBlock (%): 21
othamangitsidwa (MB): 456, chiŵerengero cha 0.42, pamwamba (%): 128, katundu wotulutsa katundu: 13, caching panopa DataBlock (%): 20
othamangitsidwa (MB): 456, chiŵerengero cha 0.33, pamwamba (%): 128, katundu wotulutsa katundu: 14, caching panopa DataBlock (%): 19
othamangitsidwa (MB): 342, chiŵerengero cha 0.33, pamwamba (%): 71, katundu wotulutsa katundu: 15, caching panopa DataBlock (%): 19
othamangitsidwa (MB): 342, chiŵerengero cha 0.32, pamwamba (%): 71, katundu wotulutsa katundu: 16, caching panopa DataBlock (%): 19
othamangitsidwa (MB): 342, chiŵerengero cha 0.31, pamwamba (%): 71, katundu wotulutsa katundu: 17, caching panopa DataBlock (%): 19
othamangitsidwa (MB): 228, chiŵerengero cha 0.3, pamwamba (%): 14, katundu wotulutsa katundu: 18, caching panopa DataBlock (%): 19
othamangitsidwa (MB): 228, chiŵerengero cha 0.29, pamwamba (%): 14, katundu wotulutsa katundu: 19, caching panopa DataBlock (%): 19
othamangitsidwa (MB): 228, chiŵerengero cha 0.27, pamwamba (%): 14, katundu wotulutsa katundu: 20, caching panopa DataBlock (%): 19
othamangitsidwa (MB): 228, chiŵerengero cha 0.25, pamwamba (%): 14, katundu wotulutsa katundu: 21, caching panopa DataBlock (%): 19
othamangitsidwa (MB): 228, chiŵerengero cha 0.24, pamwamba (%): 14, katundu wotulutsa katundu: 22, caching panopa DataBlock (%): 19
othamangitsidwa (MB): 228, chiŵerengero cha 0.22, pamwamba (%): 14, katundu wotulutsa katundu: 23, caching panopa DataBlock (%): 19
othamangitsidwa (MB): 228, chiŵerengero cha 0.21, pamwamba (%): 14, katundu wotulutsa katundu: 24, caching panopa DataBlock (%): 19
othamangitsidwa (MB): 228, chiŵerengero cha 0.2, pamwamba (%): 14, katundu wotulutsa katundu: 25, caching panopa DataBlock (%): 19
othamangitsidwa (MB): 228, chiŵerengero cha 0.17, pamwamba (%): 14, katundu wotulutsa katundu: 26, caching panopa DataBlock (%): 19
othamangitsidwa (MB): 456, chiŵerengero cha 0.17, pamwamba (%): 128, katundu wothamangitsidwa: 27, caching panopa DataBlock (%): 18 <yowonjezera amapeza (koma tebulo lomwelo)
othamangitsidwa (MB): 456, chiŵerengero cha 0.15, pamwamba (%): 128, katundu wotulutsa katundu: 28, caching panopa DataBlock (%): 17
othamangitsidwa (MB): 342, chiŵerengero cha 0.13, pamwamba (%): 71, katundu wotulutsa katundu: 29, caching panopa DataBlock (%): 17
othamangitsidwa (MB): 342, chiŵerengero cha 0.11, pamwamba (%): 71, katundu wotulutsa katundu: 30, caching panopa DataBlock (%): 17
othamangitsidwa (MB): 342, chiŵerengero cha 0.09, pamwamba (%): 71, katundu wotulutsa katundu: 31, caching panopa DataBlock (%): 17
othamangitsidwa (MB): 228, chiŵerengero cha 0.08, pamwamba (%): 14, katundu wotulutsa katundu: 32, caching panopa DataBlock (%): 17
othamangitsidwa (MB): 228, chiŵerengero cha 0.07, pamwamba (%): 14, katundu wotulutsa katundu: 33, caching panopa DataBlock (%): 17
othamangitsidwa (MB): 228, chiŵerengero cha 0.06, pamwamba (%): 14, katundu wotulutsa katundu: 34, caching panopa DataBlock (%): 17
othamangitsidwa (MB): 228, chiŵerengero cha 0.05, pamwamba (%): 14, katundu wotulutsa katundu: 35, caching panopa DataBlock (%): 17
othamangitsidwa (MB): 228, chiŵerengero cha 0.05, pamwamba (%): 14, katundu wotulutsa katundu: 36, caching panopa DataBlock (%): 17
othamangitsidwa (MB): 228, chiŵerengero cha 0.04, pamwamba (%): 14, katundu wotulutsa katundu: 37, caching panopa DataBlock (%): 17
kuthamangitsidwa (MB): 109, chiŵerengero cha 0.04, pamwamba (%): -46, katundu wothamangitsidwa: 37, caching panopa DataBlock (%): 22 < kupanikizika kumbuyo
othamangitsidwa (MB): 798, chiŵerengero cha 0.24, pamwamba (%): 299, katundu wotulutsa katundu: 38, caching panopa DataBlock (%): 20
othamangitsidwa (MB): 798, chiŵerengero cha 0.29, pamwamba (%): 299, katundu wotulutsa katundu: 39, caching panopa DataBlock (%): 18
othamangitsidwa (MB): 570, chiŵerengero cha 0.27, pamwamba (%): 185, katundu wotulutsa katundu: 40, caching panopa DataBlock (%): 17
othamangitsidwa (MB): 456, chiŵerengero cha 0.22, pamwamba (%): 128, katundu wotulutsa katundu: 41, caching panopa DataBlock (%): 16
othamangitsidwa (MB): 342, chiŵerengero cha 0.16, pamwamba (%): 71, katundu wotulutsa katundu: 42, caching panopa DataBlock (%): 16
othamangitsidwa (MB): 342, chiŵerengero cha 0.11, pamwamba (%): 71, katundu wotulutsa katundu: 43, caching panopa DataBlock (%): 16
othamangitsidwa (MB): 228, chiŵerengero cha 0.09, pamwamba (%): 14, katundu wotulutsa katundu: 44, caching panopa DataBlock (%): 16
othamangitsidwa (MB): 228, chiŵerengero cha 0.07, pamwamba (%): 14, katundu wotulutsa katundu: 45, caching panopa DataBlock (%): 16
othamangitsidwa (MB): 228, chiŵerengero cha 0.05, pamwamba (%): 14, katundu wotulutsa katundu: 46, caching panopa DataBlock (%): 16
othamangitsidwa (MB): 222, chiŵerengero cha 0.04, pamwamba (%): 11, katundu wotulutsa katundu: 47, caching panopa DataBlock (%): 16
kuthamangitsidwa (MB): 104, chiŵerengero cha 0.03, pamwamba (%): -48, katundu wothamangitsidwa: 47, caching panopa DataBlock (%): 21 < kusokoneza amapeza
othamangitsidwa (MB): 684, chiŵerengero cha 0.2, pamwamba (%): 242, katundu wotulutsa katundu: 48, caching panopa DataBlock (%): 19
othamangitsidwa (MB): 570, chiŵerengero cha 0.23, pamwamba (%): 185, katundu wotulutsa katundu: 49, caching panopa DataBlock (%): 18
othamangitsidwa (MB): 342, chiŵerengero cha 0.22, pamwamba (%): 71, katundu wotulutsa katundu: 50, caching panopa DataBlock (%): 18
othamangitsidwa (MB): 228, chiŵerengero cha 0.21, pamwamba (%): 14, katundu wotulutsa katundu: 51, caching panopa DataBlock (%): 18
othamangitsidwa (MB): 228, chiŵerengero cha 0.2, pamwamba (%): 14, katundu wotulutsa katundu: 52, caching panopa DataBlock (%): 18
othamangitsidwa (MB): 228, chiŵerengero cha 0.18, pamwamba (%): 14, katundu wotulutsa katundu: 53, caching panopa DataBlock (%): 18
othamangitsidwa (MB): 228, chiŵerengero cha 0.16, pamwamba (%): 14, katundu wotulutsa katundu: 54, caching panopa DataBlock (%): 18
othamangitsidwa (MB): 228, chiŵerengero cha 0.14, pamwamba (%): 14, katundu wotulutsa katundu: 55, caching panopa DataBlock (%): 18
kuthamangitsidwa (MB): 112, chiŵerengero cha 0.14, pamwamba (%): -44, katundu wothamangitsidwa: 55, caching panopa DataBlock (%): 23 < kupanikizika kumbuyo
othamangitsidwa (MB): 456, chiŵerengero cha 0.26, pamwamba (%): 128, katundu wotulutsa katundu: 56, caching panopa DataBlock (%): 22
othamangitsidwa (MB): 342, chiŵerengero cha 0.31, pamwamba (%): 71, katundu wotulutsa katundu: 57, caching panopa DataBlock (%): 22
othamangitsidwa (MB): 342, chiŵerengero cha 0.33, pamwamba (%): 71, katundu wotulutsa katundu: 58, caching panopa DataBlock (%): 22
othamangitsidwa (MB): 342, chiŵerengero cha 0.33, pamwamba (%): 71, katundu wotulutsa katundu: 59, caching panopa DataBlock (%): 22
othamangitsidwa (MB): 342, chiŵerengero cha 0.33, pamwamba (%): 71, katundu wotulutsa katundu: 60, caching panopa DataBlock (%): 22
othamangitsidwa (MB): 342, chiŵerengero cha 0.33, pamwamba (%): 71, katundu wotulutsa katundu: 61, caching panopa DataBlock (%): 22
othamangitsidwa (MB): 342, chiŵerengero cha 0.33, pamwamba (%): 71, katundu wotulutsa katundu: 62, caching panopa DataBlock (%): 22
othamangitsidwa (MB): 342, chiŵerengero cha 0.33, pamwamba (%): 71, katundu wotulutsa katundu: 63, caching panopa DataBlock (%): 22
othamangitsidwa (MB): 342, chiŵerengero cha 0.32, pamwamba (%): 71, katundu wotulutsa katundu: 64, caching panopa DataBlock (%): 22
othamangitsidwa (MB): 342, chiŵerengero cha 0.33, pamwamba (%): 71, katundu wotulutsa katundu: 65, caching panopa DataBlock (%): 22
othamangitsidwa (MB): 342, chiŵerengero cha 0.33, pamwamba (%): 71, katundu wotulutsa katundu: 66, caching panopa DataBlock (%): 22
othamangitsidwa (MB): 342, chiŵerengero cha 0.32, pamwamba (%): 71, katundu wotulutsa katundu: 67, caching panopa DataBlock (%): 22
othamangitsidwa (MB): 342, chiŵerengero cha 0.33, pamwamba (%): 71, katundu wotulutsa katundu: 68, caching panopa DataBlock (%): 22
othamangitsidwa (MB): 342, chiŵerengero cha 0.32, pamwamba (%): 71, katundu wotulutsa katundu: 69, caching panopa DataBlock (%): 22
othamangitsidwa (MB): 342, chiŵerengero cha 0.32, pamwamba (%): 71, katundu wotulutsa katundu: 70, caching panopa DataBlock (%): 22
othamangitsidwa (MB): 342, chiŵerengero cha 0.33, pamwamba (%): 71, katundu wotulutsa katundu: 71, caching panopa DataBlock (%): 22
othamangitsidwa (MB): 342, chiŵerengero cha 0.33, pamwamba (%): 71, katundu wotulutsa katundu: 72, caching panopa DataBlock (%): 22
othamangitsidwa (MB): 342, chiŵerengero cha 0.33, pamwamba (%): 71, katundu wotulutsa katundu: 73, caching panopa DataBlock (%): 22
othamangitsidwa (MB): 342, chiŵerengero cha 0.33, pamwamba (%): 71, katundu wotulutsa katundu: 74, caching panopa DataBlock (%): 22
othamangitsidwa (MB): 342, chiŵerengero cha 0.33, pamwamba (%): 71, katundu wotulutsa katundu: 75, caching panopa DataBlock (%): 22
othamangitsidwa (MB): 342, chiŵerengero cha 0.33, pamwamba (%): 71, katundu wotulutsa katundu: 76, caching panopa DataBlock (%): 22
kuthamangitsidwa (MB): 21, chiŵerengero cha 0.33, pamwamba (%): -90, kauntala yothamangitsa kwambiri: 76, DataBlock yamakono (%): 32
kuthamangitsidwa (MB): 0, chiŵerengero cha 0.0, pamwamba (%): -100, kauntala yothamangitsa kwambiri: 0, DataBlock yamakono (%): 100
kuthamangitsidwa (MB): 0, chiŵerengero cha 0.0, pamwamba (%): -100, kauntala yothamangitsa kwambiri: 0, DataBlock yamakono (%): 100
Makani amafunikira kuti awonetse njira yomweyi mu mawonekedwe a graph ya ubale pakati pa magawo awiri a cache - amodzi (pomwe midadada yomwe sinapemphedwepo kale) ndi yambiri (deta "yofunsidwa" kamodzi imasungidwa apa):
Ndipo potsiriza, kodi ntchito ya magawo amawoneka bwanji mu mawonekedwe a graph. Poyerekeza, posungira anazimitsidwa kotheratu pachiyambi, ndiye HBase anapezerapo ndi caching ndi kuchedwetsa chiyambi cha kukhathamiritsa ntchito ndi mphindi 5 (30 kuthamangitsidwa mkombero).
Khodi yonse ikupezeka mu Pull Request pa github.
Komabe, 300 zikwi zowerengera pamphindi sizinthu zonse zomwe zingatheke pa hardware iyi pansi pazimenezi. Chowonadi ndi chakuti pamene mukufunikira kupeza deta kudzera pa HDFS, njira ya ShortCircuitCache (yotchedwa SSC) imagwiritsidwa ntchito, yomwe imakupatsani mwayi wopeza deta mwachindunji, kupewa kuyanjana kwa intaneti.
Kufotokozera kunasonyeza kuti ngakhale kuti makinawa amapereka phindu lalikulu, nthawi zina amakhala ndi vuto, chifukwa pafupifupi ntchito zonse zolemetsa zimachitika mkati mwa loko, zomwe zimayambitsa kutsekereza nthawi zambiri.
Titazindikira izi, tidazindikira kuti vutoli litha kupindika popanga ma SSC angapo odziyimira pawokha:
private final ShortCircuitCache[] shortCircuitCache;
...
shortCircuitCache = new ShortCircuitCache[this.clientShortCircuitNum];
for (int i = 0; i < this.clientShortCircuitNum; i++)
this.shortCircuitCache[i] = new ShortCircuitCache(…);
Kenako gwirani nawo ntchito, osaphatikizanso zopingasa pagawo lomaliza:
public ShortCircuitCache getShortCircuitCache(long idx) {
return shortCircuitCache[(int) (idx % clientShortCircuitNum)];
}
Tsopano mukhoza kuyamba kuyesa. Kuti tichite izi, tiwerenga mafayilo kuchokera ku HDFS ndi pulogalamu yosavuta yamitundu yambiri. Khazikitsani magawo:
conf.set("dfs.client.read.shortcircuit", "true");
conf.set("dfs.client.read.shortcircuit.buffer.size", "65536"); // по дефолту = 1 МБ и это сильно замедляет чтение, поэтому лучше привести в соответствие к реальным нуждам
conf.set("dfs.client.short.circuit.num", num); // от 1 до 10
Ndipo ingowerengani mafayilo:
FSDataInputStream in = fileSystem.open(path);
for (int i = 0; i < count; i++) {
position += 65536;
if (position > 900000000)
position = 0L;
int res = in.read(position, byteBuffer, 0, 65536);
}
Khodi iyi imachitidwa mu ulusi wosiyana ndipo tidzawonjezera kuchuluka kwa mafayilo omwe amawerengedwa nthawi imodzi (kuyambira 10 mpaka 200 - opingasa opingasa) ndi kuchuluka kwa ma cache (kuyambira 1 mpaka 10 - zithunzi). Mzere woyima ukuwonetsa kuthamangitsa komwe kumabwera chifukwa cha kuchuluka kwa SSC pokhudzana ndi mlandu pakakhala cache imodzi yokha.
Momwe mungawerengere graph: Nthawi yopha anthu 100 imawerengedwa mu midadada ya 64 KB yokhala ndi cache imodzi imafuna masekondi 78. Pomwe ndi 5 cache zimatenga masekondi 16. Iwo. pali mathamangitsidwe ~ nthawi 5. Monga momwe tikuonera pa graph, zotsatira zake siziwoneka bwino kwa chiwerengero chochepa chowerengera chofanana; imayamba kugwira ntchito yodziwika pamene pali ulusi wochuluka wa 50. Zikuwonekeranso kuti kuwonjezeka kwa chiwerengero cha SSCs kuchokera ku 6 ndipo pamwambapa amapereka chiwonjezeko chochepa kwambiri cha magwiridwe antchito.
Zindikirani 1: popeza zotsatira zake zimakhala zosasinthika (onani pansipa), maulendo atatu adachitika ndipo zotsatira zake zidawerengedwa.
Zindikirani 2: Kupindula kwa magwiridwe antchito pokonza mwayi wopezeka mwachisawawa ndi womwewo, ngakhale kuti mwayiwo umakhala wocheperako pang'ono.
Komabe, m'pofunika kufotokoza kuti, mosiyana ndi HBase, mathamangitsidwe izi si nthawi zonse kwaulere. Apa "tikutsegula" mphamvu ya CPU kuti igwire ntchito zambiri, m'malo mopachikidwa pamaloko.
Apa mutha kuwona kuti, makamaka, kuchuluka kwa ma cache kumapereka chiwonjezeko chofananira pakugwiritsa ntchito CPU. Komabe, pali zophatikizira zopambana pang'ono.
Mwachitsanzo, tiyeni tiyang'ane mwatsatanetsatane zoikamo SSC = 3. Kuwonjezeka kwa ntchito pamtunduwu kuli pafupifupi nthawi 3.3. M'munsimu muli zotsatira za maulendo atatu osiyana.
Pomwe kugwiritsa ntchito kwa CPU kumawonjezeka pafupifupi nthawi 2.8. Kusiyanitsa sikuli kwakukulu, koma Greta wamng'ono ali kale wokondwa ndipo akhoza kukhala ndi nthawi yopita kusukulu ndi kutenga maphunziro.
Chifukwa chake, izi zitha kukhala ndi zotsatira zabwino pachida chilichonse chomwe chimagwiritsa ntchito mwayi wochulukirapo ku HDFS (mwachitsanzo Spark, ndi zina), bola ngati nambala yofunsira ndiyopepuka (ie pulagi ili kumbali ya kasitomala wa HDFS) ndipo pali mphamvu ya CPU yaulere. . Kuti tiwone, tiyeni tiyese momwe kukhathamiritsa kwa BlockCache ndi SSC powerenga kuchokera ku HBase kudzakhala nako.
Zitha kuwoneka kuti m'mikhalidwe yotere zotsatira zake sizowoneka bwino ngati mayeso oyengedwa (kuwerenga popanda kukonzedwa), koma ndizotheka kufinya 80K yowonjezera apa. Pamodzi, kukhathamiritsa konseku kumapereka kuthamanga kwa 4x.
PR idapangidwanso kuti izi zitheke , zomwe zaphatikizidwa ndipo izi zitha kupezeka muzotulutsa zamtsogolo.
Ndipo pomaliza, zinali zosangalatsa kuyerekeza kuwerengera kowerengera kofananako kofananako kofananako, Cassandra ndi HBase.
Kuti tichite izi, tidayambitsa zida zoyezera katundu wa YCSB kuchokera ku makamu awiri (800 ulusi wonse). Pa mbali ya seva - zochitika za 4 za RegionServer ndi Cassandra pa makamu a 4 (osati omwe makasitomala akuthamanga, kuti apewe chikoka chawo). Zowerengera zidachokera kumatebulo akulu:
HBase - 300 GB pa HDFS (100 GB data pure)
Cassandra - 250 GB (replication factor = 3)
Iwo. voliyumuyo inali pafupifupi yofanana (mu HBase pang'ono).
Zigawo za HBase:
dfs.client.short.circuit.num = 5 (Kukhathamiritsa kwa kasitomala wa HDFS)
hbase.lru.cache.heavy.eviction.count.limit = 30 - izi zikutanthauza kuti chigambacho chidzayamba kugwira ntchito pambuyo pa kuthamangitsidwa kwa 30 (~ 5 mphindi)
hbase.lru.cache.heavy.eviction.mb.size.limit = 300 - chandamale kuchuluka kwa caching ndi kuthamangitsidwa
Zolemba za YCSB zidagawidwa ndikuphatikizidwa kukhala ma graph a Excel:
Monga mukuwonera, kukhathamiritsa uku kumapangitsa kuti zitheke kufananiza magwiridwe antchito awa pansi pazifukwa izi ndikukwaniritsa kuwerengera 450 zikwi pamphindikati.
Tikukhulupirira kuti chidziwitsochi chingakhale chothandiza kwa wina panthawi yankhondo yosangalatsa yopanga zokolola.
Source: www.habr.com