Ts'ebetso e phahameng ke e 'ngoe ea litlhoko tsa bohlokoa ha u sebetsa le data e kholo. Lefapheng la ho kenya data ho Sberbank, re pompa hoo e batlang e le lits'ebetso tsohle ho Hadoop-based Data Cloud 'me ka hona re sebetsana le phallo e kholo ea tlhahisoleseding. Ka tlhaho, re lula re batla litsela tsa ho ntlafatsa ts'ebetso, 'me joale re batla ho u joetsa hore na re khonne joang ho patch RegionServer HBase le moreki oa HDFS, ka lebaka leo re khonneng ho eketsa lebelo la ts'ebetso ea ho bala haholo.
Leha ho le joalo, pele u fetela molemong oa ntlafatso, ho bohlokoa ho bua ka lithibelo tseo, ha e le hantle, li ke keng tsa qojoa haeba u lutse holim'a HDD.
Hobaneng ha HDD le ho bala ka potlako Random Access li sa tsamaisane
Joalo ka ha u tseba, HBase, le li-database tse ling tse ngata, boloka data ka li-block tsa mashome a 'maloa a kilobyte ka boholo. Ka ho sa feleng e ka ba 64 KB. Joale ha re nahane hore re hloka ho fumana li-byte tse 100 feela mme re kopa HBase ho re fa data ena ho sebelisa senotlolo se itseng. Kaha boholo ba boloko ho HFiles ke 64 KB, kopo e tla ba makhetlo a 640 a maholo (motsotso feela!) ho feta kamoo ho hlokahalang.
E latelang, kaha kopo e tla feta HDFS le mochine oa eona oa ho boloka metadata ShortCircuitCache (e lumellang ho fihlella ka ho toba lifaeleng), sena se lebisa ho bala ho se ho ntse ho le 1 MB ho tswa ho disk. Leha ho le joalo, sena se ka fetoloa le parameter dfs.client.read.shortcircuit.buffer.size 'me maemong a mangata hoa utloahala ho fokotsa boleng bona, mohlala ho 126 KB.
A re re re etsa sena, empa ho phaella moo, ha re qala ho bala data ka java api, joalo ka mesebetsi e kang FileChannel.read 'me u kope mokhoa oa ho sebetsa ho bala palo e boletsoeng ea data, e baleha "feela haeba" makhetlo a 2 ho feta. , ke. 256 KB molemong oa rona. Lebaka ke hore java ha e na mokhoa o bonolo oa ho beha folakha ea FADV_RANDOM ho thibela boitšoaro bona.
Ka lebaka leo, ho fumana li-byte tsa rona tse 100, makhetlo a 2600 ho feta a baloa tlas'a hood. Ho ka bonahala eka tharollo e hlakile, a re fokotseng boholo ba block ho kilobyte, re behe folakha e boletsoeng 'me re fumane leseli le leholo ka potlako. Empa bothata ke hore ka ho fokotsa boholo ba li-block ka makhetlo a 2, re boetse re fokotsa palo ea li-byte tse baloang yuniti ea nako ka makhetlo a 2.
Phaello e 'ngoe ka ho beha folakha ea FADV_RANDOM e ka fumanoa, empa feela ka mefuta e mengata ea likhoele le boholo ba block ea 128 KB, empa sena ke boholo ba liperesente tse mashome a mabeli:
Liteko li entsoe ka lifaele tse 100, e 'ngoe le e' ngoe e le 1 GB ka boholo 'me e fumaneha ho li-HDD tse 10.
Ha re bale hore na re ka itšetleha ka eng ka lebelo lena:
A re re re bala ho tloha ho 10 disks ka lebelo la 280 MB / sec, i.e. 3 milione makhetlo a 100 byte. Empa ha re ntse re hopola, data eo re e hlokang e ka tlase ho makhetlo a 2600 ho feta se baloang. Kahoo, re arola limilione tse 3 ka 2600 mme re fumana Lirekoto tse 1100 motsotsoana.
Ho nyahamisa, na ha ho joalo? Ke tlhaho eo Phihlello e sa Lebelloang phihlello ea data ho HDD - ho sa tsotelehe boholo ba block. Ona ke moeli oa 'mele oa phihlello e sa reroang mme ha ho database e ka pepesehang ho feta tlasa maemo a joalo.
Joale li-database li fihlela lebelo le phahameng hakae? Ho araba potso ena, a re shebeng se etsahalang setšoantšong se latelang:
Mona re bona hore metsotso e seng mekae ea pele lebelo le hlile le ka ba lirekoto tse sekete ka motsotsoana. Leha ho le joalo, ho ea pele, ka lebaka la hore ho baloa ho hongata ho feta kamoo ho neng ho kōptjoa kateng, data e qetella e le buff / cache ea tsamaiso ea ts'ebetso (linux) 'me lebelo le eketseha ho ea ho 60 likete tse hlomphehang ka motsotsoana.
Kahoo, ho ea pele re tla sebetsana le ho potlakisa phihlello ea data e ka har'a cache ea OS kapa e fumanehang lisebelisoa tsa polokelo tsa SSD/NVMe tsa lebelo le lekanang la phihlello.
Tabeng ea rona, re tla etsa liteko bencheng ea li-server tse 4, e 'ngoe le e' ngoe ea tsona e lefisoa ka tsela e latelang:
CPU: Xeon E5-2680 v4 @ 2.40GHz 64 likhoele.
Memori: 730 GB.
mofuta oa java: 1.8.0_111
'Me mona ntlha ea bohlokoa ke palo ea data litafoleng tse lokelang ho baloa. 'Nete ke hore haeba u bala lintlha tse tsoang tafoleng e behiloeng ka ho feletseng ka har'a cache ea HBase, joale e ke ke ea tla ho bala ho tsoa ho buff / cache ea tsamaiso ea tsamaiso. Hobane HBase ka ho sa feleng e fana ka 40% ea memori ho sebopeho se bitsoang BlockCache. Ha e le hantle ena ke ConcurrentHashMap, moo senotlolo e leng lebitso la faele + offset ea block, 'me boleng ke data ea sebele nakong ena.
Kahoo, ha re bala feela sebopeho sena, re , joalo ka likopo tse milione ka motsotsoana. Empa ha re nahane hore re ke ke ra arola li-gigabytes tse makholo bakeng sa litlhoko tsa database, hobane ho na le lintho tse ling tse ngata tse sebetsang ho li-server tsena.
Ka mohlala, molemong oa rona, molumo oa BlockCache ho RS e le 'ngoe e ka bang 12 GB. Re ile ra fihla li-RS tse peli sebakeng se le seng, ke hore. 96 GB e abetsoe BlockCache libakeng tsohle. 'Me ho na le lintlha tse ngata ka makhetlo a mangata, mohlala, e ke e be litafole tse 4, libaka tse 130 ka 'ngoe, moo lifaele li leng 800 MB ka boholo, li hatelloa ke FAST_DIFF, ke hore. kakaretso ea 410 GB (ena ke data e hloekileng, ke hore ntle le ho ela hloko ntlha ea ho pheta-pheta).
Ka hona, BlockCache ke hoo e ka bang 23% feela ea palo ea kakaretso ea data mme sena se haufi haholo le maemo a sebele a se bitsoang BigData. 'Me mona ke moo monate o qalang - hobane ho hlakile hore ho fokotseha ha cache, ho mpefala le ho feta. Ha e le hantle, haeba u hloloheloa, u tla tlameha ho etsa mosebetsi o mongata - i.e. theohela ho lits'ebetso tsa sistimi ea ho letsetsa. Leha ho le joalo, sena se ke ke sa qojoa, kahoo a re shebeng karolo e fapaneng ka ho feletseng - ho etsahala'ng ka data ka har'a cache?
Ha re nolofatse boemo mme re nke hore re na le cache e lekanang ntho e le 'ngoe feela. Mona ke mohlala oa se tla etsahala ha re leka ho sebetsa ka bophahamo ba data ka makhetlo a 1 ho feta cache, re tla tlameha ho:
1. Beha boloko ba 1 ka har'a cache
2. Tlosa boloko ba 1 ho cache
3. Beha boloko ba 2 ka har'a cache
4. Tlosa boloko ba 2 ho cache
5. Beha boloko ba 3 ka har'a cache
Liketso tse 5 li phethiloe! Leha ho le joalo, boemo bona bo ke ke ba bitsoa bo tloaelehileng; ha e le hantle, re qobella HBase ho etsa mosebetsi o mongata o se nang thuso. E lula e bala data ho tswa ho cache ea OS, e e beha ho BlockCache, feela ho e lahlela hang-hang hobane karolo e ncha ea data e fihlile. Papali e qalong ea poso e bonts'a moelelo oa bothata - 'Mokelli oa Lithōle o ntse a theoha, sepakapaka se ntse se futhumala, Greta e nyenyane ea Sweden e hōle le e chesang e ntse e ferekana. 'Me rona batho ba IT ha re rate ha bana ba hloname, kahoo re qala ho nahana ka seo re ka se etsang ka hona.
Ho thoe'ng haeba u sa behe li-blocks tsohle ka har'a cache, empa ke karolo e itseng feela ea tsona, e le hore cache e se ke ea phalla? Ha re qaleng ka ho kenyelletsa mela e 'maloa ea khoutu qalong ea ts'ebetso ea ho kenya data ho BlockCache:
public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean inMemory) {
if (cacheDataBlockPercent != 100 && buf.getBlockType().isData()) {
if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent) {
return;
}
}
...
Taba mona ke e latelang: offset ke boemo ba boloko ka faeleng mme linomoro tsa eona tsa ho qetela li ajoa ka mokhoa o sa reroang le ka mokhoa o ts'oanang ho tloha ho 00 ho isa ho 99. Ka hona, re tla tlola feela tse oelang mofuteng oo re o hlokang.
Ka mohlala, beha cacheDataBlockPercent = 20 'me u bone se etsahalang:
Phello e hlakile. Lirapeng tse ka tlase, hoa hlaka hore na ke hobane'ng ha ho potlaka ho joalo ho etsahetse - re boloka lisebelisoa tse ngata tsa GC ntle le ho etsa mosebetsi oa Sisyphean oa ho beha data ka har'a cache feela ho e lahlela hang-hang metsi a lintja tsa Martian:
Ka nako e ts'oanang, ts'ebeliso ea CPU ea eketseha, empa e tlase haholo ho feta tlhahiso:
Hape ke habohlokoa ho hlokomela hore li-blocks tse bolokiloeng ho BlockCache li fapane. Boholo, hoo e ka bang 95%, ke data ka boeona. 'Me tse ling kaofela ke metadata, joalo ka li-filters tsa Bloom kapa LEAF_INDEX le . Lintlha tsena ha lia lekana, empa li bohlokoa haholo, hobane pele o fumana data ka kotloloho, HBase e fetohela meta ho utloisisa hore na hoa hlokahala ho batla mona ho ea pele, 'me haeba ho joalo, moo sebaka sa thahasello se leng teng.
Ka hona, ka khoutu re bona boemo ba ho hlahloba buf.getBlockType().isData() 'me ka lebaka la meta ena, re tla e tlohela ka har'a cache ho sa tsotellehe hore na boemo ke bofe.
Joale ha re eketse mojaro 'me re tiise tšobotsi hanyenyane ka nako e le' ngoe. Tekong ea pele re entse peresente ea cutoff = 20 mme BlockCache e ne e sa sebelisoe hanyane. Joale ha re e behe ho 23% 'me re kenye likhoele tse 100 metsotso e meng le e meng e 5 ho bona hore na ho tlala ho etsahala neng:
Mona re bona hore mofuta oa pele o batla o otla siling hang-hang ka likopo tse ka bang likete tse 100 motsotsoana. Athe patch e fana ka ho potlakisa ho fihla ho 300 tse likete. Ka nako e ts'oanang, ho hlakile hore ho potlakisa ho eketsehileng ha ho sa le "mahala"; Tšebeliso ea CPU le eona e ntse e eketseha.
Leha ho le joalo, ena ha se tharollo e ntle haholo, kaha ha re tsebe esale pele hore na ke liperesente life tsa li-block tse lokelang ho bolokoa, ho ipapisitse le profil ea mojaro. Ka hona, ho ile ha sebelisoa mochine ho fetola parameter ena ka bo eona ho latela mosebetsi oa ho bala.
Likhetho tse tharo li kentsoe ho laola sena:
hbase.lru.cache.heavy.eviction.count.limit - e beha hore na ts'ebetso ea ho leleka data ho cache e lokela ho sebetsa ka makhetlo a makae pele re qala ho sebelisa optimization (ke hore, ho tlola li-blocks). Ka kamehla e lekana le MAX_INT = 2147483647 'me ha e le hantle e bolela hore tšobotsi e ke ke ea qala ho sebetsa ka boleng bona. Hobane ts'ebetso ea ho lelekoa e qala metsotsoana e meng le e meng e 5 - 10 (e itšetlehile ka mojaro) le 2147483647 * 10 / 60 / 60 / 24 / 365 = 680 lilemo. Leha ho le joalo, re ka beha paramente ena ho 0 mme ra etsa hore karolo e sebetse hang kamora ho qala.
Leha ho le joalo, ho boetse ho na le phallo ea moputso ho parameter ena. Haeba mojaro oa rona o le joalo hoo ho bala ha nako e khuts'oane (e re motšehare) le ho bala nako e telele (bosiu) ho lula ho kopane, joale re ka etsa bonnete ba hore sebopeho se buloa feela ha ts'ebetso ea ho bala nako e telele e ntse e tsoela pele.
Mohlala, rea tseba hore 'malo oa nakoana hangata o nka motsotso o le mong. Ha ho na tlhoko ea ho qala ho lahla li-blocks, cache e ke ke ea ba le nako ea ho felloa ke nako ebe joale re ka beha parameter ena e lekanang le, mohlala, 1. Sena se tla lebisa tabeng ea hore optimization e tla qala ho sebetsa feela ha nako e telele- nako ea ho bala ka mafolofolo e qalile, ke. ka metsotsoana e 10. Kahoo, haeba re e-na le nako e khutšoanyane ea ho bala, joale li-blocks tsohle li tla kena ka har'a cache 'me li tla ba teng (ntle le tse tla lelekoa ke algorithm e tloaelehileng). 'Me ha re bala nako e telele, karolo ea buloa' me re tla ba le ts'ebetso e phahameng haholo.
hbase.lru.cache.heavy.eviction.mb.size.limit - e beha hore na re ka rata ho beha li-megabyte tse kae ka har'a cache ('me, ehlile, re e leleke) ka metsotsoana e 10. Karolo e tla leka ho fihlela boleng bona le ho bo boloka. Taba ke ena: haeba re sutumelletsa li-gigabyte ka har'a cache, joale re tla tlameha ho leleka li-gigabyte, 'me sena, joalokaha re bone ka holimo, se theko e boima haholo. Leha ho le joalo, ha ua lokela ho leka ho e beha e nyane haholo, kaha sena se tla etsa hore block skip mode e tsoe pele ho nako. Bakeng sa li-server tse matla (li-cores tse ka bang 20-40), ho loketse ho beha hoo e ka bang 300-400 MB. Bakeng sa sehlopha se bohareng (~ 10 cores) 200-300 MB. Bakeng sa mekhoa e fokolang (2-5 cores) 50-100 MB e ka 'na ea e-ba e tloaelehileng (e sa lekoa ho tsena).
Ha re shebeng hore na sena se sebetsa joang: ha re re re beha hbase.lru.cache.heavy.eviction.mb.size.limit = 500, ho na le mofuta o itseng oa mojaro (ho bala) ebe metsotsoana e meng le e meng ea ~ 10 re bala hore na ke li-byte tse kae. ba lelekoa ka ho sebelisa foromo:
Overhead = Kakaretso ea Li-Byte (MB) * 100 / Limit (MB) - 100;
Ha e le hantle 2000 MB e ile ea lelekoa, joale Overhead e lekana le:
2000 * 100 / 500 - 100 = 300%
Li-algorithms li leka ho boloka liperesente tse seng kae ho feta liperesente tse mashome, kahoo karolo e tla fokotsa liperesente tsa li-blocks tse bolokiloeng, ka ho etsa joalo e kenye ts'ebetso ea mochini oa auto-tuning.
Leha ho le joalo, haeba mojaro o theoha, ha re re ke 200 MB feela e lelekoang 'me Overhead e fetoha e mpe (seo ho thoeng ke overshooting):
200 * 100 / 500 - 100 = -60%
Ho fapana le hoo, karolo e tla eketsa peresente ea li-blocks tse bolokiloeng ho fihlela Overhead e fetoha e ntle.
Ka tlase ke mohlala oa kamoo sena se shebahalang kateng ho data ea sebele. Ha ho hlokahale hore u leke ho fihla ho 0%, ha ho khonehe. E ntle haholo ha e ka ba 30 - 100%, sena se thusa ho qoba ho tsoa pele ho nako mokhoeng oa ho ntlafatsa nakong ea ts'ebetso ea nako e khuts'oane.
hbase.lru.cache.heavy.eviction.overhead.coefficient - e beha hore na re ka rata ho fumana sephetho kapele hakae. Haeba re tseba hantle hore lintho tseo re li balang hangata li telele 'me ha re batle ho ema, re ka eketsa karolelano ena mme ra fumana ts'ebetso e phahameng kapele.
Ka mohlala, re beha coefficient ena = 0.01. Sena se bolela hore Overhead (sheba ka holimo) e tla atisa ka palo ena ka sephetho sa sephetho le peresente ea li-block blocks e tla fokotsoa. Ha re nke hore Overhead = 300% le coefficient = 0.01, joale peresente ea li-block blocks e tla fokotseha ka 3%.
Mokhoa o tšoanang oa "Backpressure" o boetse oa kengoa ts'ebetsong bakeng sa litekanyetso tse mpe tsa Overhead (overshooting). Kaha ho na le ho feto-fetoha ha nako e khuts'oane ha palo ea ho baloa le ho lelekoa ka tlung ho lula ho khoneha, mochini ona o u lumella ho qoba ho tsoa pele ho nako molemong oa ho ntlafatsa. Khatello ea morao-rao e na le mohopolo o sothehileng: ha ho feta ho feta, ho na le li-blocks tse ngata tse bolokiloeng.
Khoutu ea ts'ebetsong
LruBlockCache cache = this.cache.get();
if (cache == null) {
break;
}
freedSumMb += cache.evict()/1024/1024;
/*
* Sometimes we are reading more data than can fit into BlockCache
* and it is the cause a high rate of evictions.
* This in turn leads to heavy Garbage Collector works.
* So a lot of blocks put into BlockCache but never read,
* but spending a lot of CPU resources.
* Here we will analyze how many bytes were freed and decide
* decide whether the time has come to reduce amount of caching blocks.
* It help avoid put too many blocks into BlockCache
* when evict() works very active and save CPU for other jobs.
* More delails: https://issues.apache.org/jira/browse/HBASE-23887
*/
// First of all we have to control how much time
// has passed since previuos evict() was launched
// This is should be almost the same time (+/- 10s)
// because we get comparable volumes of freed bytes each time.
// 10s because this is default period to run evict() (see above this.wait)
long stopTime = System.currentTimeMillis();
if ((stopTime - startTime) > 1000 * 10 - 1) {
// Here we have to calc what situation we have got.
// We have the limit "hbase.lru.cache.heavy.eviction.bytes.size.limit"
// and can calculte overhead on it.
// We will use this information to decide,
// how to change percent of caching blocks.
freedDataOverheadPercent =
(int) (freedSumMb * 100 / cache.heavyEvictionMbSizeLimit) - 100;
if (freedSumMb > cache.heavyEvictionMbSizeLimit) {
// Now we are in the situation when we are above the limit
// But maybe we are going to ignore it because it will end quite soon
heavyEvictionCount++;
if (heavyEvictionCount > cache.heavyEvictionCountLimit) {
// It is going for a long time and we have to reduce of caching
// blocks now. So we calculate here how many blocks we want to skip.
// It depends on:
// 1. Overhead - if overhead is big we could more aggressive
// reducing amount of caching blocks.
// 2. How fast we want to get the result. If we know that our
// heavy reading for a long time, we don't want to wait and can
// increase the coefficient and get good performance quite soon.
// But if we don't sure we can do it slowly and it could prevent
// premature exit from this mode. So, when the coefficient is
// higher we can get better performance when heavy reading is stable.
// But when reading is changing we can adjust to it and set
// the coefficient to lower value.
int change =
(int) (freedDataOverheadPercent * cache.heavyEvictionOverheadCoefficient);
// But practice shows that 15% of reducing is quite enough.
// We are not greedy (it could lead to premature exit).
change = Math.min(15, change);
change = Math.max(0, change); // I think it will never happen but check for sure
// So this is the key point, here we are reducing % of caching blocks
cache.cacheDataBlockPercent -= change;
// If we go down too deep we have to stop here, 1% any way should be.
cache.cacheDataBlockPercent = Math.max(1, cache.cacheDataBlockPercent);
}
} else {
// Well, we have got overshooting.
// Mayby it is just short-term fluctuation and we can stay in this mode.
// It help avoid permature exit during short-term fluctuation.
// If overshooting less than 90%, we will try to increase the percent of
// caching blocks and hope it is enough.
if (freedSumMb >= cache.heavyEvictionMbSizeLimit * 0.1) {
// Simple logic: more overshooting - more caching blocks (backpressure)
int change = (int) (-freedDataOverheadPercent * 0.1 + 1);
cache.cacheDataBlockPercent += change;
// But it can't be more then 100%, so check it.
cache.cacheDataBlockPercent = Math.min(100, cache.cacheDataBlockPercent);
} else {
// Looks like heavy reading is over.
// Just exit form this mode.
heavyEvictionCount = 0;
cache.cacheDataBlockPercent = 100;
}
}
LOG.info("BlockCache evicted (MB): {}, overhead (%): {}, " +
"heavy eviction counter: {}, " +
"current caching DataBlock (%): {}",
freedSumMb, freedDataOverheadPercent,
heavyEvictionCount, cache.cacheDataBlockPercent);
freedSumMb = 0;
startTime = stopTime;
}
Joale ha re shebeng tsena tsohle re sebelisa mohlala oa sebele. Re na le mongolo o latelang oa tlhahlobo:
- Ha re qale ho etsa Scan (likhoele tse 25, batch = 100)
- Kamora metsotso e 5, eketsa mefuta e mengata (likhoele tse 25, batch = 100)
- Kamora metsotso e 5, tima li-multi-gets (ho sala feela skena hape)
Re etsa li-run tse peli, pele hbase.lru.cache.heavy.eviction.count.limit = 10000 (e hlileng e thibelang tšobotsi), ebe o beha moeli = 0 (e e nolofalletsa).
Ka har'a li-log tse ka tlase re bona kamoo karolo e buloang kateng le ho tsosolosa Overshooting ho 14-71%. Nako le nako mojaro oa fokotseha, o bulelang Backpressure le HBase e boloka li-blocks tse ling hape.
Log RegionServer
lelekoa (MB): 0, karo-karolelano ea 0.0, ka holimo (%): -100, k'haonte e boima ea ho leleka: 0, dataBlock ea hona joale ea caching (%): 100
lelekoa (MB): 0, karo-karolelano ea 0.0, ka holimo (%): -100, k'haonte e boima ea ho leleka: 0, dataBlock ea hona joale ea caching (%): 100
ho lelekoa (MB): 2170, karo-karolelano ea 1.09, ka holimo (%): 985, k'hamera e boima ea ho leleka: 1, dataBlock ea hona joale ea caching (%): 91 < qala
ho lelekoa (MB): 3763, karo-karolelano ea 1.08, ka holimo (%): 1781, k'hamera e boima ea ho leleka: 2, caching ea hona joale DataBlock (%): 76
ho lelekoa (MB): 3306, karo-karolelano ea 1.07, ka holimo (%): 1553, k'hamera e boima ea ho leleka: 3, caching ea hona joale DataBlock (%): 61
ho lelekoa (MB): 2508, karo-karolelano ea 1.06, ka holimo (%): 1154, k'hamera e boima ea ho leleka: 4, caching ea hona joale DataBlock (%): 50
ho lelekoa (MB): 1824, karo-karolelano ea 1.04, ka holimo (%): 812, k'hamera e boima ea ho leleka: 5, caching ea hona joale DataBlock (%): 42
ho lelekoa (MB): 1482, karo-karolelano ea 1.03, ka holimo (%): 641, k'hamera e boima ea ho leleka: 6, caching ea hona joale DataBlock (%): 36
ho lelekoa (MB): 1140, karo-karolelano ea 1.01, ka holimo (%): 470, k'hamera e boima ea ho leleka: 7, caching ea hona joale DataBlock (%): 32
ho lelekoa (MB): 913, karo-karolelano ea 1.0, ka holimo (%): 356, k'hamera e boima ea ho leleka: 8, caching ea hona joale DataBlock (%): 29
ho lelekoa (MB): 912, karo-karolelano ea 0.89, ka holimo (%): 356, k'hamera e boima ea ho leleka: 9, caching ea hona joale DataBlock (%): 26
ho lelekoa (MB): 684, karo-karolelano ea 0.76, ka holimo (%): 242, k'hamera e boima ea ho leleka: 10, caching ea hona joale DataBlock (%): 24
ho lelekoa (MB): 684, karo-karolelano ea 0.61, ka holimo (%): 242, k'hamera e boima ea ho leleka: 11, caching ea hona joale DataBlock (%): 22
ho lelekoa (MB): 456, karo-karolelano ea 0.51, ka holimo (%): 128, k'hamera e boima ea ho leleka: 12, caching ea hona joale DataBlock (%): 21
ho lelekoa (MB): 456, karo-karolelano ea 0.42, ka holimo (%): 128, k'hamera e boima ea ho leleka: 13, caching ea hona joale DataBlock (%): 20
ho lelekoa (MB): 456, karo-karolelano ea 0.33, ka holimo (%): 128, k'hamera e boima ea ho leleka: 14, caching ea hona joale DataBlock (%): 19
ho lelekoa (MB): 342, karo-karolelano ea 0.33, ka holimo (%): 71, k'hamera e boima ea ho leleka: 15, caching ea hona joale DataBlock (%): 19
ho lelekoa (MB): 342, karo-karolelano ea 0.32, ka holimo (%): 71, k'hamera e boima ea ho leleka: 16, caching ea hona joale DataBlock (%): 19
ho lelekoa (MB): 342, karo-karolelano ea 0.31, ka holimo (%): 71, k'hamera e boima ea ho leleka: 17, caching ea hona joale DataBlock (%): 19
ho lelekoa (MB): 228, karo-karolelano ea 0.3, ka holimo (%): 14, k'hamera e boima ea ho leleka: 18, caching ea hona joale DataBlock (%): 19
ho lelekoa (MB): 228, karo-karolelano ea 0.29, ka holimo (%): 14, k'hamera e boima ea ho leleka: 19, caching ea hona joale DataBlock (%): 19
ho lelekoa (MB): 228, karo-karolelano ea 0.27, ka holimo (%): 14, k'hamera e boima ea ho leleka: 20, caching ea hona joale DataBlock (%): 19
ho lelekoa (MB): 228, karo-karolelano ea 0.25, ka holimo (%): 14, k'hamera e boima ea ho leleka: 21, caching ea hona joale DataBlock (%): 19
ho lelekoa (MB): 228, karo-karolelano ea 0.24, ka holimo (%): 14, k'hamera e boima ea ho leleka: 22, caching ea hona joale DataBlock (%): 19
ho lelekoa (MB): 228, karo-karolelano ea 0.22, ka holimo (%): 14, k'hamera e boima ea ho leleka: 23, caching ea hona joale DataBlock (%): 19
ho lelekoa (MB): 228, karo-karolelano ea 0.21, ka holimo (%): 14, k'hamera e boima ea ho leleka: 24, caching ea hona joale DataBlock (%): 19
ho lelekoa (MB): 228, karo-karolelano ea 0.2, ka holimo (%): 14, k'hamera e boima ea ho leleka: 25, caching ea hona joale DataBlock (%): 19
ho lelekoa (MB): 228, karo-karolelano ea 0.17, ka holimo (%): 14, k'hamera e boima ea ho leleka: 26, caching ea hona joale DataBlock (%): 19
ho lelekoa (MB): 456, karo-karolelano ea 0.17, ka holimo (%): 128, k'hamera e boima ea ho lelekoa: 27, caching ea hona joale DataBlock (%): 18 < e kentse e fumana (empa tafole e tšoanang)
ho lelekoa (MB): 456, karo-karolelano ea 0.15, ka holimo (%): 128, k'hamera e boima ea ho leleka: 28, caching ea hona joale DataBlock (%): 17
ho lelekoa (MB): 342, karo-karolelano ea 0.13, ka holimo (%): 71, k'hamera e boima ea ho leleka: 29, caching ea hona joale DataBlock (%): 17
ho lelekoa (MB): 342, karo-karolelano ea 0.11, ka holimo (%): 71, k'hamera e boima ea ho leleka: 30, caching ea hona joale DataBlock (%): 17
ho lelekoa (MB): 342, karo-karolelano ea 0.09, ka holimo (%): 71, k'hamera e boima ea ho leleka: 31, caching ea hona joale DataBlock (%): 17
ho lelekoa (MB): 228, karo-karolelano ea 0.08, ka holimo (%): 14, k'hamera e boima ea ho leleka: 32, caching ea hona joale DataBlock (%): 17
ho lelekoa (MB): 228, karo-karolelano ea 0.07, ka holimo (%): 14, k'hamera e boima ea ho leleka: 33, caching ea hona joale DataBlock (%): 17
ho lelekoa (MB): 228, karo-karolelano ea 0.06, ka holimo (%): 14, k'hamera e boima ea ho leleka: 34, caching ea hona joale DataBlock (%): 17
ho lelekoa (MB): 228, karo-karolelano ea 0.05, ka holimo (%): 14, k'hamera e boima ea ho leleka: 35, caching ea hona joale DataBlock (%): 17
ho lelekoa (MB): 228, karo-karolelano ea 0.05, ka holimo (%): 14, k'hamera e boima ea ho leleka: 36, caching ea hona joale DataBlock (%): 17
ho lelekoa (MB): 228, karo-karolelano ea 0.04, ka holimo (%): 14, k'hamera e boima ea ho leleka: 37, caching ea hona joale DataBlock (%): 17
ho lelekoa (MB): 109, karo-karolelano ea 0.04, holimo (%): -46, k'hamera e boima ea ho lelekoa: 37, caching ea hona joale DataBlock (%): 22 < khatello ea morao
ho lelekoa (MB): 798, karo-karolelano ea 0.24, ka holimo (%): 299, k'hamera e boima ea ho leleka: 38, caching ea hona joale DataBlock (%): 20
ho lelekoa (MB): 798, karo-karolelano ea 0.29, ka holimo (%): 299, k'hamera e boima ea ho leleka: 39, caching ea hona joale DataBlock (%): 18
ho lelekoa (MB): 570, karo-karolelano ea 0.27, ka holimo (%): 185, k'hamera e boima ea ho leleka: 40, caching ea hona joale DataBlock (%): 17
ho lelekoa (MB): 456, karo-karolelano ea 0.22, ka holimo (%): 128, k'hamera e boima ea ho leleka: 41, caching ea hona joale DataBlock (%): 16
ho lelekoa (MB): 342, karo-karolelano ea 0.16, ka holimo (%): 71, k'hamera e boima ea ho leleka: 42, caching ea hona joale DataBlock (%): 16
ho lelekoa (MB): 342, karo-karolelano ea 0.11, ka holimo (%): 71, k'hamera e boima ea ho leleka: 43, caching ea hona joale DataBlock (%): 16
ho lelekoa (MB): 228, karo-karolelano ea 0.09, ka holimo (%): 14, k'hamera e boima ea ho leleka: 44, caching ea hona joale DataBlock (%): 16
ho lelekoa (MB): 228, karo-karolelano ea 0.07, ka holimo (%): 14, k'hamera e boima ea ho leleka: 45, caching ea hona joale DataBlock (%): 16
ho lelekoa (MB): 228, karo-karolelano ea 0.05, ka holimo (%): 14, k'hamera e boima ea ho leleka: 46, caching ea hona joale DataBlock (%): 16
ho lelekoa (MB): 222, karo-karolelano ea 0.04, ka holimo (%): 11, k'hamera e boima ea ho leleka: 47, caching ea hona joale DataBlock (%): 16
ho lelekoa (MB): 104, karo-karolelano ea 0.03, ka holimo (%): -48, k'hamera e boima ea ho lelekoa: 47, caching ea hona joale DataBlock (%): 21 < interrupt gets
ho lelekoa (MB): 684, karo-karolelano ea 0.2, ka holimo (%): 242, k'hamera e boima ea ho leleka: 48, caching ea hona joale DataBlock (%): 19
ho lelekoa (MB): 570, karo-karolelano ea 0.23, ka holimo (%): 185, k'hamera e boima ea ho leleka: 49, caching ea hona joale DataBlock (%): 18
ho lelekoa (MB): 342, karo-karolelano ea 0.22, ka holimo (%): 71, k'hamera e boima ea ho leleka: 50, caching ea hona joale DataBlock (%): 18
ho lelekoa (MB): 228, karo-karolelano ea 0.21, ka holimo (%): 14, k'hamera e boima ea ho leleka: 51, caching ea hona joale DataBlock (%): 18
ho lelekoa (MB): 228, karo-karolelano ea 0.2, ka holimo (%): 14, k'hamera e boima ea ho leleka: 52, caching ea hona joale DataBlock (%): 18
ho lelekoa (MB): 228, karo-karolelano ea 0.18, ka holimo (%): 14, k'hamera e boima ea ho leleka: 53, caching ea hona joale DataBlock (%): 18
ho lelekoa (MB): 228, karo-karolelano ea 0.16, ka holimo (%): 14, k'hamera e boima ea ho leleka: 54, caching ea hona joale DataBlock (%): 18
ho lelekoa (MB): 228, karo-karolelano ea 0.14, ka holimo (%): 14, k'hamera e boima ea ho leleka: 55, caching ea hona joale DataBlock (%): 18
ho lelekoa (MB): 112, karo-karolelano ea 0.14, holimo (%): -44, k'hamera e boima ea ho lelekoa: 55, caching ea hona joale DataBlock (%): 23 < khatello ea morao
ho lelekoa (MB): 456, karo-karolelano ea 0.26, ka holimo (%): 128, k'hamera e boima ea ho leleka: 56, caching ea hona joale DataBlock (%): 22
ho lelekoa (MB): 342, karo-karolelano ea 0.31, ka holimo (%): 71, k'hamera e boima ea ho leleka: 57, caching ea hona joale DataBlock (%): 22
ho lelekoa (MB): 342, karo-karolelano ea 0.33, ka holimo (%): 71, k'hamera e boima ea ho leleka: 58, caching ea hona joale DataBlock (%): 22
ho lelekoa (MB): 342, karo-karolelano ea 0.33, ka holimo (%): 71, k'hamera e boima ea ho leleka: 59, caching ea hona joale DataBlock (%): 22
ho lelekoa (MB): 342, karo-karolelano ea 0.33, ka holimo (%): 71, k'hamera e boima ea ho leleka: 60, caching ea hona joale DataBlock (%): 22
ho lelekoa (MB): 342, karo-karolelano ea 0.33, ka holimo (%): 71, k'hamera e boima ea ho leleka: 61, caching ea hona joale DataBlock (%): 22
ho lelekoa (MB): 342, karo-karolelano ea 0.33, ka holimo (%): 71, k'hamera e boima ea ho leleka: 62, caching ea hona joale DataBlock (%): 22
ho lelekoa (MB): 342, karo-karolelano ea 0.33, ka holimo (%): 71, k'hamera e boima ea ho leleka: 63, caching ea hona joale DataBlock (%): 22
ho lelekoa (MB): 342, karo-karolelano ea 0.32, ka holimo (%): 71, k'hamera e boima ea ho leleka: 64, caching ea hona joale DataBlock (%): 22
ho lelekoa (MB): 342, karo-karolelano ea 0.33, ka holimo (%): 71, k'hamera e boima ea ho leleka: 65, caching ea hona joale DataBlock (%): 22
ho lelekoa (MB): 342, karo-karolelano ea 0.33, ka holimo (%): 71, k'hamera e boima ea ho leleka: 66, caching ea hona joale DataBlock (%): 22
ho lelekoa (MB): 342, karo-karolelano ea 0.32, ka holimo (%): 71, k'hamera e boima ea ho leleka: 67, caching ea hona joale DataBlock (%): 22
ho lelekoa (MB): 342, karo-karolelano ea 0.33, ka holimo (%): 71, k'hamera e boima ea ho leleka: 68, caching ea hona joale DataBlock (%): 22
ho lelekoa (MB): 342, karo-karolelano ea 0.32, ka holimo (%): 71, k'hamera e boima ea ho leleka: 69, caching ea hona joale DataBlock (%): 22
ho lelekoa (MB): 342, karo-karolelano ea 0.32, ka holimo (%): 71, k'hamera e boima ea ho leleka: 70, caching ea hona joale DataBlock (%): 22
ho lelekoa (MB): 342, karo-karolelano ea 0.33, ka holimo (%): 71, k'hamera e boima ea ho leleka: 71, caching ea hona joale DataBlock (%): 22
ho lelekoa (MB): 342, karo-karolelano ea 0.33, ka holimo (%): 71, k'hamera e boima ea ho leleka: 72, caching ea hona joale DataBlock (%): 22
ho lelekoa (MB): 342, karo-karolelano ea 0.33, ka holimo (%): 71, k'hamera e boima ea ho leleka: 73, caching ea hona joale DataBlock (%): 22
ho lelekoa (MB): 342, karo-karolelano ea 0.33, ka holimo (%): 71, k'hamera e boima ea ho leleka: 74, caching ea hona joale DataBlock (%): 22
ho lelekoa (MB): 342, karo-karolelano ea 0.33, ka holimo (%): 71, k'hamera e boima ea ho leleka: 75, caching ea hona joale DataBlock (%): 22
ho lelekoa (MB): 342, karo-karolelano ea 0.33, ka holimo (%): 71, k'hamera e boima ea ho leleka: 76, caching ea hona joale DataBlock (%): 22
lelekoa (MB): 21, karo-karolelano ea 0.33, ka holimo (%): -90, k'haonte e boima ea ho leleka: 76, dataBlock ea hona joale ea caching (%): 32
lelekoa (MB): 0, karo-karolelano ea 0.0, ka holimo (%): -100, k'haonte e boima ea ho leleka: 0, dataBlock ea hona joale ea caching (%): 100
lelekoa (MB): 0, karo-karolelano ea 0.0, ka holimo (%): -100, k'haonte e boima ea ho leleka: 0, dataBlock ea hona joale ea caching (%): 100
Lits'oants'o li ne li hlokahala ho bonts'a ts'ebetso e ts'oanang ka sebopeho sa graph ea kamano lipakeng tsa likarolo tse peli tsa cache - e le 'ngoe (moo li-blocks tse e-so koptjoe pele) le tse ngata (data "e kopiloeng" bonyane hang e bolokiloe mona):
'Me qetellong, ts'ebetso ea li-parameter e shebahala joang ka mokhoa oa graph. Ha ho bapisoa, cache e ile ea tingoa ka ho feletseng qalong, 'me HBase e ile ea qalisoa ka caching le ho liehisa ho qala ha mosebetsi oa ho ntlafatsa ka metsotso e 5 (lipotoloho tse 30 tsa ho leleka).
Khoutu e felletseng e ka fumaneha ho Pull Request ho github.
Leha ho le joalo, ho bala likete tse 300 ka motsotsoana hase sohle se ka finyelloang ho hardware ena tlas'a maemo ana. 'Nete ke hore ha o hloka ho fumana data ka HDFS, ho sebelisoa mokhoa oa ShortCircuitCache (eo ka mor'a moo o bitsoang SSC), o u lumellang hore u fihlele data ka ho toba, u qoba ho sebelisana le marang-rang.
Profiling e bontšitse hore le hoja mochine ona o fana ka phaello e kholo, hape ka nako e 'ngoe e fetoha botlolo, hobane hoo e ka bang mesebetsi eohle e boima e etsahala ka hare ho senotlolo, e leng se lebisang ho thibela boholo ba nako.
Ha re se re hlokometse sena, re hlokometse hore bothata bo ka qojoa ka ho theha letoto la li-SSC tse ikemetseng:
private final ShortCircuitCache[] shortCircuitCache;
...
shortCircuitCache = new ShortCircuitCache[this.clientShortCircuitNum];
for (int i = 0; i < this.clientShortCircuitNum; i++)
this.shortCircuitCache[i] = new ShortCircuitCache(…);
Ebe u sebetsa le bona, u sa kenyelle mateano a litsela le nomorong ea ho qetela ea offset:
public ShortCircuitCache getShortCircuitCache(long idx) {
return shortCircuitCache[(int) (idx % clientShortCircuitNum)];
}
Joale u ka qala ho etsa liteko. Ho etsa sena, re tla bala lifaele tsa HDFS ka sesebelisoa se bonolo sa likhoele tse ngata. Beha li-parameter:
conf.set("dfs.client.read.shortcircuit", "true");
conf.set("dfs.client.read.shortcircuit.buffer.size", "65536"); // по дефолту = 1 МБ и это сильно замедляет чтение, поэтому лучше привести в соответствие к реальным нуждам
conf.set("dfs.client.short.circuit.num", num); // от 1 до 10
Ebe u bala lifaele feela:
FSDataInputStream in = fileSystem.open(path);
for (int i = 0; i < count; i++) {
position += 65536;
if (position > 900000000)
position = 0L;
int res = in.read(position, byteBuffer, 0, 65536);
}
Khoutu ena e etsoa ka likhoele tse arohaneng mme re tla eketsa palo ea lifaele tse baloang ka nako e le ngoe (ho tloha ho 10 ho isa ho 200 - axis e otlolohileng) le palo ea li-cache (ho tloha ho 1 ho isa ho 10 - Grafiken). Axis e otlolohileng e bonts'a ho potlakisa ho hlahisoang ke keketseho ea SSC e amanang le nyeoe ha ho na le cache e le 'ngoe feela.
Mokhoa oa ho bala graph: Nako ea ts'ebetso ea likete tse 100 e baloa ka li-block tsa 64 KB tse nang le cache e le 'ngoe e hloka metsotsoana e 78. Athe ka li-cache tse 5 ho nka metsotsoana e 16. Tseo. ho na le lebelo la ~ makhetlo a 5. Joalokaha ho ka bonoa ho graph, phello ha e hlokomelehe haholo bakeng sa palo e nyenyane ea lipalo tse tšoanang; e qala ho bapala karolo e hlokomelehang ha ho na le likhoele tse fetang 50. Hape hoa hlokomeleha hore ho eketsa palo ea SSC ho tloha 6. 'me ka holimo e fana ka keketseho e nyane haholo ea ts'ebetso.
Tlhokomeliso ea 1: kaha liphetho tsa liteko ha li fetohe (sheba ka tlase), ho ile ha etsoa li-run tse 3 mme litekanyetso tse hlahisitsoeng li ile tsa lekanngoa.
Tlhokomeliso ea 2: Phaello ea ts'ebetso ho tsoa ho tlhophiso ea phihlello e sa reroang ea ts'oana, leha phihlello ka boeona e lieha hanyane.
Leha ho le joalo, hoa hlokahala ho hlakisa hore, ho fapana le boemo ba HBase, ho potlakisa hona ha se kamehla ho leng mahala. Mona re "notlolla" bokhoni ba CPU ba ho etsa mosebetsi ho feta, ho e-na le ho fanyeha lilokong.
Mona o ka bona hore, ka kakaretso, keketseho ea palo ea li-cache e fana ka keketseho e batlang e lekana ts'ebelisong ea CPU. Leha ho le joalo, ho na le lihlopha tse ling tse hapileng hanyane.
Ka mohlala, a re hlahlobisiseng boemo ba SSC = 3. Keketseho ea ts'ebetso ho mefuta e ka bang makhetlo a 3.3. Ka tlase ke liphetho ho tsoa ho mathang a mararo a arohaneng.
Ha ts'ebeliso ea CPU e eketseha ka makhetlo a ka bang 2.8. Phapang ha e kholo haholo, empa Greta e monyenyane o se a ntse a thabile mme a ka ba le nako ea ho ea sekolong le ho ithuta.
Kahoo, sena se tla ba le phello e ntle bakeng sa sesebelisoa leha e le sefe se sebelisang phihlello ea HDFS ka bongata (mohlala Spark, joalo-joalo), ha feela khoutu ea kopo e le bobebe (ke hore plug e ka lehlakoreng la bareki ba HDFS) mme ho na le matla a mahala a CPU. . Ho hlahloba, a re lekeng hore na ts'ebeliso e kopaneng ea BlockCache le tokiso ea SSC bakeng sa ho bala ho tsoa HBase e tla ba le phello efe.
Hoa hlokomeleha hore tlas'a maemo a joalo phello ha e kholo joaloka litekong tse hloekisitsoeng (ho bala ntle le ts'ebetso leha e le efe), empa hoa khoneha ho hlahisa 80K e eketsehileng mona. Ka kopanelo, lintlafatso ka bobeli li fana ka lebelo la 4x.
Ho boetse ho entsoe PR bakeng sa ntlafatso ena , e kopantsoeng 'me ts'ebetso ena e tla fumaneha litokollong tse tlang.
'Me qetellong, ho ne ho thahasellisa ho bapisa ts'ebetso ea ho bala ea database e tšoanang ea kholomo e pharaletseng, Cassandra le HBase.
Ho etsa sena, re hlahisitse mehlala ea ts'ebeliso e tloaelehileng ea tlhahlobo ea mojaro oa YCSB ho tsoa ho mabotho a mabeli (likhoele tse 800 ka kakaretso). Ka lehlakoreng la seva - liketsahalo tse 4 tsa RegionServer le Cassandra ho mabotho a 4 (eseng ao bareki ba mathang teng, ho qoba tšusumetso ea bona). Lipalo li tsoa litafoleng tsa boholo:
HBase - 300 GB ho HDFS (data e hloekileng ea 100 GB)
Cassandra - 250 GB (replication factor = 3)
Tseo. molumo o ne o batla o lekana (ho HBase hanyane ho feta).
Litekanyetso tsa HBase:
dfs.client.short.circuit.num = 5 (Katoloso ea bareki ba HDFS)
hbase.lru.cache.heavy.eviction.count.limit = 30 - sena se bolela hore patch e tla qala ho sebetsa ka mor'a ho lelekoa ha 30 (~ metsotso e 5)
hbase.lru.cache.heavy.eviction.mb.size.limit = 300 - sepheo sa bophahamo ba ho boloka le ho lelekoa
Li-log tsa YCSB li ile tsa aroloa 'me tsa hlophisoa ka li-graph tsa Excel:
Joalokaha u ka bona, lintlafatso tsena li etsa hore ho khonehe ho bapisa ts'ebetso ea li-database tsena tlas'a maemo ana le ho finyella lipalo tse likete tse 450 ka motsotsoana.
Re tšepa hore boitsebiso bona bo ka ba molemo ho motho e mong nakong ea ntoa e thabisang ea tlhahiso.
Source: www.habr.com