O le maualuga o le faʻatinoga o se tasi lea o manaʻoga autu pe a galue ma faʻamatalaga tetele. I totonu o le matagaluega e utaina faʻamaumauga i Sberbank, matou te pamuina toetoe lava o fefaʻatauaiga uma i totonu o la matou Hadoop-based Data Cloud ma o lea e feagai ai ma le tele o faʻamatalaga o faʻamatalaga. E masani lava, o loʻo matou sailia pea ni auala e faʻaleleia ai le faʻatinoga, ma o lea matou te fia taʻu atu ia te oe le auala na matou faʻaogaina ai le RegionServer HBase ma le HDFS client, faʻafetai lea na mafai ai ona matou faʻateleina le saoasaoa o faʻagaioiga faitau.
Ae ui i lea, aʻo leʻi agai i luma i le ute o le faʻaleleia, e taua le talanoa e uiga i tapulaʻa, i le mataupu faavae, e le mafai ona faʻafefe pe afai e te nofo i luga o se HDD.
Aisea e le fetaui ai le HDD ma le vave Random Access faitau
E pei ona e iloa, HBase, ma le tele o isi faʻamaumauga, teuina faʻamaumauga i poloka o le tele o sefulu kilobytes le lapopoa. I le fa'aletonu e tusa ma le 64 KB. Sei o tatou mafaufau e manaʻomia le naʻo le 100 bytes ma matou ole atu ia HBase e tuʻuina mai ia i matou nei faʻamatalaga e faʻaaoga ai se ki. Talu ai o le poloka poloka i HFiles e 64 KB, o le talosaga o le a 640 taimi sili atu (naʻo le minute!) nai lo le manaʻomia.
O le isi, talu ai o le talosaga o le a alu i le HDFS ma lona metadata caching mechanism ShortCircuitCache (lea e mafai ai ona tuʻu saʻo i faila), e taʻitaʻia ai le faitau 1 MB mai le disk. Ae ui i lea, e mafai ona fetuunai lenei mea i le parakalafa dfs.client.read.shortcircuit.buffer.size ma i le tele o tulaga e talafeagai le faʻaitiitia o lenei tau, mo se faʻataʻitaʻiga i le 126 KB.
Seʻi tatou fai atu tatou te faia lenei mea, ae i le faʻaopoopoga, pe a amata ona tatou faitau faʻamatalaga e ala i le java api, e pei o galuega e pei o FileChannel.read ma fai atu i le faiga faʻaogaina e faitau le aofaʻi o faʻamaumauga, e faitau "naʻo le tulaga" 2 taimi sili atu. , i.e. 256 KB i la matou tulaga. E mafua ona o le java e leai se auala faigofie e seti ai le FADV_RANDOM fu'a e puipuia ai lenei amio.
O se taunuuga, ina ia maua a tatou 100 bytes, 2600 taimi sili atu e faitau i lalo o le pulou. E foliga mai e manino le fofo, seʻi o tatou faʻaititia le poloka poloka i le kilobyte, seti le fuʻa taʻua ma maua ai le faʻamalamalamaga tele. Ae o le faʻalavelave o le faʻaititia o le poloka poloka i le 2 taimi, matou te faʻaititia foi le numera o bytes faitau i le iunite o le taimi e 2 taimi.
O nisi mea e maua mai le setiina o le FADV_RANDOM fu'a e mafai ona maua, ae na'o le maualuga tele-filo ma le poloka poloka o le 128 KB, ae o le maualuga lea o le lua sefulu pasene:
Na faia suʻega i luga o faila 100, taʻitasi 1 GB i le lapopoa ma faʻamau i luga ole 10 HDDs.
Sei o tatou fuafua mea tatou te mafaia, i le mataupu faavae, faalagolago i ai i lenei saoasaoa:
Faapea tatou te faitau mai le 10 tisiketi i le saoasaoa o le 280 MB/sec, i.e. 3 miliona taimi 100 paita. Ae pei ona tatou manatua, o faʻamaumauga tatou te manaʻomia e 2600 taimi e itiiti ifo nai lo le mea o loʻo faitauina. O lea, matou te vaevaeina le 3 miliona i le 2600 ma maua 1100 faamaumauga i le sekone.
Fa'anoanoa, a ea? O le natura lena Avanoa Faʻatasi avanoa i faʻamatalaga i luga o le HDD - e tusa lava po o le a le tele o poloka. Ole tapula'a fa'aletino lea ole avanoa fa'afuase'i ma e leai se fa'amaumauga e mafai ona fa'apipi'i atili i lalo o ia tulaga.
E fa'apefea la ona ausia e fa'amaumauga fa'amaumauga ni saoasaoa maualuga atu? Ina ia tali lenei fesili, seʻi o tatou tilotilo i le mea o loʻo tupu i le ata o loʻo i lalo:
O iinei tatou te iloa ai mo nai minute muamua o le saoasaoa e tusa ma le afe faamaumauga i le sekone. Ae ui i lea, e le gata i lea, ona o le mea moni e sili atu le faitau nai lo le mea na talosagaina, o faʻamaumauga e muta i le buff / cache o le faiga faʻaogaina (linux) ma o le saoasaoa e faʻateleina i le sili atu 60 afe i le sekone
O le mea lea, o le a tatou feagai atili ma le faʻavaveina o avanoa i faʻamatalaga o loʻo i totonu o le OS cache pe o loʻo i totonu o SSD / NVMe masini faʻapipiʻi e tutusa le saoasaoa avanoa.
I la matou tulaga, o le a matou faia suʻega i luga o le nofoa o le 4 servers, o loʻo molia taʻitasi e pei ona taua i lalo:
PPU: Xeon E5-2680 v4 @ 2.40GHz 64 filo.
Manatua: 730 GB.
java version: 1.8.0_111
Ma o iinei o le autu autu o le aofaʻi o faʻamaumauga i laulau e manaʻomia ona faitau. O le mea moni afai e te faitauina faʻamatalaga mai se laulau o loʻo tuʻuina atoa i totonu o le HBase cache, o le a le oʻo lava i le faitau mai le faʻaogaina o le buff / cache. Ona o le HBase e le mafai ona tuʻuina atu le 40% o le manatua i se fausaga e taʻua o BlockCache. O le mea moni o le ConcurrentHashMap lea, o le ki o le igoa faila + offset o le poloka, ma o le tau o le faʻamatalaga moni i lenei offset.
O lea la, pe a faitau na o lenei fausaga, tatou , pei o le miliona talosaga i le sekone. Ae se'i tatou manatu e le mafai ona tatou fa'asoaina le faitau selau o gigabytes o manatua mo na'o mana'oga fa'amaumauga, aua e tele isi mea aoga o lo'o fa'agaoioi i luga o nei 'au'aunaga.
Mo se faʻataʻitaʻiga, i la matou tulaga, o le tele o BlockCache i le tasi RS e tusa ma le 12 GB. Na matou tulaueleele lua RS i le tasi node, i.e. 96 GB o loʻo tuʻuina atu mo BlockCache i nodes uma. Ma e tele taimi e sili atu faʻamatalaga, mo se faʻataʻitaʻiga, tuʻu 4 laulau, 130 itulagi taʻitasi, lea o faila e 800 MB le tele, faʻapipiʻi e FAST_DIFF, i.e. le aofaʻi o 410 GB (o faʻamaumauga mama lea, e aunoa ma le amanaia o le faʻasologa o mea).
O le mea lea, BlockCache e naʻo le 23% o le aofaʻi o faʻamaumauga ma e sili atu ona latalata i tulaga moni o le mea e taʻua o BigData. Ma o le mea lea e amata ai le fiafia - aua e mautinoa lava, o le itiiti ifo o le cache e lavea, o le leaga o le faʻatinoga. A uma, afai e te misia, e tatau ona e faia le tele o galuega - i.e. alu i lalo i le vala'au i galuega fa'aletino. Ae ui i lea, e le mafai ona aloese mai lenei mea, o lea seʻi o tatou tilotilo i se itu e matua ese lava - o le a le mea e tupu i faʻamatalaga i totonu o le cache?
Se'i fa'afaigofie le tulaga ma fa'apea o lo'o iai sau fa'aoga e na'o le 1 mea e fetaui. O se faʻataʻitaʻiga lea o le mea o le a tupu pe a tatou taumafai e galulue ma se faʻamaumauga faʻamaumauga 3 taimi sili atu nai lo le cache, e tatau ona tatou:
1. Tu'u le poloka 1 i totonu o le pusa
2. Aveese poloka 1 mai le fa'aoga
3. Tu'u le poloka 2 i totonu o le pusa
4. Aveese poloka 2 mai le fa'aoga
5. Tu'u le poloka 3 i totonu o le pusa
5 gaioiga ua maeʻa! Ae ui i lea, o lenei tulaga e le mafai ona taʻua masani; o le mea moni, o loʻo matou faʻamalosia le HBase e fai le tele o galuega le aoga. E faitau i taimi uma faʻamatalaga mai le OS cache, tuʻu i BlockCache, naʻo le lafoina i fafo toetoe lava a vave ona oʻo mai se vaega fou o faʻamatalaga. O le animation i le amataga o le pou o loʻo faʻaalia ai le aʻafiaga o le faʻafitauli - o le Collector o lapisi ua alu ese, ua vevela le siosiomaga, o Greta laitiiti i Suetena mamao ma vevela ua amata ona le fiafia. Ma o matou tagata IT e matua le fiafia i ai pe a faʻanoanoa tamaiti, o lea matou te amata mafaufau ai i mea e mafai ona matou faia i ai.
Ae fa'apefea pe afai e te le tu'uina uma poloka i totonu o le cache, ae na'o se pasene patino o latou, ina ia le sosolo ai le cache? Sei o tatou amata i le na o le faʻaopoopoina o ni nai laina faʻailoga i le amataga o le galuega mo le tuʻuina o faʻamatalaga i BlockCache:
public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean inMemory) {
if (cacheDataBlockPercent != 100 && buf.getBlockType().isData()) {
if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent) {
return;
}
}
...
O le manatu iinei o le mea lea: offset o le tulaga lea o le poloka i le faila ma o ona numera mulimuli e faʻafuaseʻi ma tufatufaina tutusa mai le 00 i le 99. O le mea lea, o le a matou faʻafefe naʻo i latou e pa'ū i totonu o le laina matou te manaʻomia.
Mo se faʻataʻitaʻiga, seti cacheDataBlockPercent = 20 ma vaʻai pe o le a le mea e tupu:
Ua manino le i'uga. I le kalafi o loʻo i lalo, ua manino ai le mafuaʻaga na tupu ai sea faʻavavevave - matou te faʻasaoina le tele o punaoa GC e aunoa ma le faia o le Sisyphean galuega o le tuʻuina o faʻamaumauga i totonu o le faʻamaufaʻailoga na o le vave lafoina i lalo o le alavai o taifau Martian:
I le taimi lava e tasi, faʻateleina le faʻaogaina o le CPU, ae sili atu le itiiti ifo nai lo le gaosiga:
E taua foi le matauina o poloka o loʻo teuina i BlockCache e ese. Ole tele, e tusa ma le 95%, o faʻamaumauga lava ia. A'o isi o metadata, e pei o Bloom filters po'o LEAF_INDEX ma . O nei faʻamatalaga e le lava, ae e aoga tele, aua aʻo leʻi mauaina saʻo faʻamatalaga, e liliu HBase i le meta e malamalama ai pe tatau ona suʻesuʻe atili iinei ma, afai o lea, o fea tonu o loʻo i ai le poloka o tului.
O le mea lea, i totonu o le code tatou te vaʻai i se tulaga siaki buf.getBlockType().isData() ma faʻafetai i lenei meta, matou te tuʻuina i totonu o le cache i soʻo se tulaga.
Se'i o tatou fa'ateleina le uta ma fa'amalosi teisi le ata i le alu e tasi. I le suʻega muamua na matou faia le pasene tipi = 20 ma BlockCache sa fai si faʻaaogaina. Sei o tatou seti i le 23% ma faʻaopoopo 100 filo i 5 minute uma e vaʻai ai pe o le a le mea e tupu ai le saturation:
O iinei tatou te vaʻai ai o le uluai kopi e toetoe lava a vave ona oso i luga o le taualuga i le tusa ma le 100 afe talosaga i le sekone. A o le patch e maua ai le saoasaoa e oʻo atu i le 300 afe. I le taimi lava e tasi, e manino lava o le faʻavavevave e le o toe "saoloto"; O loʻo faʻateleina foi le faʻaogaina o le CPU.
Ae ui i lea, e le o se fofo sili ona matagofie, talu ai tatou te le iloa muamua pe o le a le pasene o poloka e manaʻomia ona teu, e faʻalagolago i le faʻailoga o le uta. O le mea lea, na faʻatinoina ai se masini e faʻapipiʻi otometi ai lenei parakalafa e faʻatatau i le gaioiga o gaioiga faitau.
E tolu filifiliga ua faʻaopoopoina e pulea ai lenei mea:
hbase.lru.cache.heavy.eviction.count.limit - fa'atulaga le tele o taimi e tatau ona fa'agasolo ai le fa'agasologa o fa'amatalaga mai le cache a'o le'i amata fa'aogaina le fa'ata'ita'iga (e pei o poloka poloka). I le faaletonu e tutusa ma MAX_INT = 2147483647 ma o le mea moni o lona uiga o le a le amata galue le ata ma lenei tau. Aua o le faʻagasologa o le faʻateʻaina e amata i le 5 - 10 sekone (e faʻalagolago i le uta) ma le 2147483647 * 10 / 60 / 60 / 24 / 365 = 680 tausaga. Ae ui i lea, e mafai ona tatou setiina lenei parakalafa i le 0 ma faʻaogaina le ata i le taimi lava e maeʻa ai le faʻalauiloaina.
Ae ui i lea, o loʻo i ai foi se uta i lenei parakalafa. Afai o la tatou uta e faʻapea o le faitau pupuu (fai i le ao) ma le umi umi (i le po) o loʻo vaʻaia i taimi uma, ona mafai lea ona tatou mautinoa e faʻaalia le faʻaaliga pe a faʻagasolo galuega faitau umi.
Mo se faʻataʻitaʻiga, matou te iloa o faitauga pupuu e masani ona umi pe a ma le 1 minute. E leai se mea e manaʻomia e amata ai le togiina o poloka, o le cache o le a le maua se taimi e faʻaumatia ai ona mafai lea ona tatou setiina lenei parakalafa e tutusa ma, mo se faʻataʻitaʻiga, 10. O lenei mea o le a taʻitaʻia ai le mea moni o le a amata ona galue le optimization pe a umi- ua amata upu toaga faitau, i.e. i le 100 sekone. O le mea lea, afai ei ai sau faitauga puupuu, ona alu uma lea o poloka i totonu o le faʻaoga ma o le a avanoa (sei vagana ai i latou o le a tuliesea e le algorithm masani). Ma a matou faia ni faitauga umi, e faʻaalia le faʻaaliga ma o le a sili atu le maualuga o a matou faʻatinoga.
hbase.lru.cache.heavy.eviction.mb.size.limit - seti pe fia megabytes matou te mananaʻo e tuʻu i totonu o le cache (ma, ioe, tuliesea) i le 10 sekone. O le a taumafai le vaega e ausia lenei tau ma tausia. O le manatu lenei: afai tatou te tuleia gigabytes i totonu o le cache, ona tatau lea ona tatou tuliesea gigabytes, ma o lenei, e pei ona tatou vaʻaia i luga, e taugata tele. Ae ui i lea, e le tatau ona e taumafai e faʻapipiʻi laʻititi, aua o le a mafua ai ona alu ese vave le faiga poloka poloka. Mo 'au'aunaga mamana (e tusa ma le 20-40 cores fa'aletino), e sili ona lelei le setiina o le 300-400 MB. Mo le vasega ogatotonu (~ 10 cores) 200-300 MB. Mo faiga vaivai (2-5 cores) 50-100 MB atonu e masani (e le tofotofoina i nei mea).
Se'i o tatou va'ai pe fa'afefea ona fa'aogaina lenei mea: fa'apea tatou seti hbase.lru.cache.heavy.eviction.mb.size.limit = 500, o lo'o i ai se ituaiga o uta (faitau) ona o'o lea i le ~ 10 sekone tatou te fa'atatauina pe fia paita na i ai. tuliesea e faʻaaoga ai le fua faʻatatau:
Ulua'i = Aofa'i Paita Sa'oloto (MB) * 100 / Tapulaa (MB) - 100;
Afai o le mea moni 2000 MB na tuliesea, o lona uiga o le Overhead e tutusa ma:
2000 * 100 / 500 - 100 = 300%
O algorithms e taumafai e tausia le sili atu nai lo le sefulu pasene, o lea o le a faʻaitiitia ai le pasene o poloka poloka, ma faʻaaogaina ai se masini faʻaogaina.
Ae peitaʻi, afai e paʻu le uta, faʻapea naʻo le 200 MB e faʻateʻaina ma o le Overhead e le lelei (le mea e taʻua o le overshooting):
200 * 100 / 500 - 100 = -60%
I se isi itu, o le faʻaaliga o le a faʻateleina le pasene o poloka poloka seʻia oʻo ina lelei le Overhead.
O loʻo i lalo se faʻataʻitaʻiga o le auala e foliga mai ai i faʻamatalaga moni. E leai se manaʻoga e taumafai e ausia le 0%, e le mafai. E lelei tele pe a tusa ma le 30 - 100%, e fesoasoani lea e aloese ai mai le vave alu ese mai le tulaga sili ona lelei i taimi pupuu.
hbase.lru.cache.heavy.eviction.overhead.coefficient — faatulaga le vave tatou te mananao e maua le taunuuga. Afai tatou te iloa ma le mautinoa o a tatou faitauga e tele lava e umi ma e le manaʻo e faʻatali, e mafai ona tatou faʻateleina le fua faatatau ma maua vave le faʻatinoga.
Mo se faʻataʻitaʻiga, matou te setiina lenei coefficient = 0.01. O lona uiga o le Overhead (silasila i luga) o le a faʻateleina i lenei numera e ala i le taunuuga ma o le pasene o poloka poloka o le a faʻaititia. Seʻi tatou faʻapea o le Overhead = 300% ma le coefficient = 0.01, ona faʻaititia lea o le pasene o poloka poloka i le 3%.
O se manatu tutusa "Backpressure" o loʻo faʻatinoina foʻi mo tau le lelei Overhead (overshooting). Talu ai o suiga pupuu i le tele o faitau ma tuliesea e masani lava ona mafai, o lenei masini e mafai ai ona e aloese mai le vave alu ese mai le tulaga sili ona lelei. O le backpressure e i ai se manatu faʻafefe: o le malosi o le soʻo, o le tele foi lea o poloka e teu.
Faiga fa'atinoga
LruBlockCache cache = this.cache.get();
if (cache == null) {
break;
}
freedSumMb += cache.evict()/1024/1024;
/*
* Sometimes we are reading more data than can fit into BlockCache
* and it is the cause a high rate of evictions.
* This in turn leads to heavy Garbage Collector works.
* So a lot of blocks put into BlockCache but never read,
* but spending a lot of CPU resources.
* Here we will analyze how many bytes were freed and decide
* decide whether the time has come to reduce amount of caching blocks.
* It help avoid put too many blocks into BlockCache
* when evict() works very active and save CPU for other jobs.
* More delails: https://issues.apache.org/jira/browse/HBASE-23887
*/
// First of all we have to control how much time
// has passed since previuos evict() was launched
// This is should be almost the same time (+/- 10s)
// because we get comparable volumes of freed bytes each time.
// 10s because this is default period to run evict() (see above this.wait)
long stopTime = System.currentTimeMillis();
if ((stopTime - startTime) > 1000 * 10 - 1) {
// Here we have to calc what situation we have got.
// We have the limit "hbase.lru.cache.heavy.eviction.bytes.size.limit"
// and can calculte overhead on it.
// We will use this information to decide,
// how to change percent of caching blocks.
freedDataOverheadPercent =
(int) (freedSumMb * 100 / cache.heavyEvictionMbSizeLimit) - 100;
if (freedSumMb > cache.heavyEvictionMbSizeLimit) {
// Now we are in the situation when we are above the limit
// But maybe we are going to ignore it because it will end quite soon
heavyEvictionCount++;
if (heavyEvictionCount > cache.heavyEvictionCountLimit) {
// It is going for a long time and we have to reduce of caching
// blocks now. So we calculate here how many blocks we want to skip.
// It depends on:
// 1. Overhead - if overhead is big we could more aggressive
// reducing amount of caching blocks.
// 2. How fast we want to get the result. If we know that our
// heavy reading for a long time, we don't want to wait and can
// increase the coefficient and get good performance quite soon.
// But if we don't sure we can do it slowly and it could prevent
// premature exit from this mode. So, when the coefficient is
// higher we can get better performance when heavy reading is stable.
// But when reading is changing we can adjust to it and set
// the coefficient to lower value.
int change =
(int) (freedDataOverheadPercent * cache.heavyEvictionOverheadCoefficient);
// But practice shows that 15% of reducing is quite enough.
// We are not greedy (it could lead to premature exit).
change = Math.min(15, change);
change = Math.max(0, change); // I think it will never happen but check for sure
// So this is the key point, here we are reducing % of caching blocks
cache.cacheDataBlockPercent -= change;
// If we go down too deep we have to stop here, 1% any way should be.
cache.cacheDataBlockPercent = Math.max(1, cache.cacheDataBlockPercent);
}
} else {
// Well, we have got overshooting.
// Mayby it is just short-term fluctuation and we can stay in this mode.
// It help avoid permature exit during short-term fluctuation.
// If overshooting less than 90%, we will try to increase the percent of
// caching blocks and hope it is enough.
if (freedSumMb >= cache.heavyEvictionMbSizeLimit * 0.1) {
// Simple logic: more overshooting - more caching blocks (backpressure)
int change = (int) (-freedDataOverheadPercent * 0.1 + 1);
cache.cacheDataBlockPercent += change;
// But it can't be more then 100%, so check it.
cache.cacheDataBlockPercent = Math.min(100, cache.cacheDataBlockPercent);
} else {
// Looks like heavy reading is over.
// Just exit form this mode.
heavyEvictionCount = 0;
cache.cacheDataBlockPercent = 100;
}
}
LOG.info("BlockCache evicted (MB): {}, overhead (%): {}, " +
"heavy eviction counter: {}, " +
"current caching DataBlock (%): {}",
freedSumMb, freedDataOverheadPercent,
heavyEvictionCount, cache.cacheDataBlockPercent);
freedSumMb = 0;
startTime = stopTime;
}
Sei o tatou tilotilo nei i nei mea uma e faʻaaoga ai se faʻataʻitaʻiga moni. O lo'o i ai le matou su'ega fa'ata'ita'i:
- Se'i amata ona fai Fa'ailoga (25 filo, vaega = 100)
- A maeʻa le 5 minute, faʻaopoopo le tele-gets (25 filo, batch = 100)
- A mae'a le 5 minute, tape le tele-gets (na'o le su'esu'e e toe fa'aauau)
Matou te faia ni taʻaloga se lua, muamua hbase.lru.cache.heavy.eviction.count.limit = 10000 (lea e faʻamalo moni ai le ata), ona seti lea o le tapulaʻa = 0 (e mafai ai).
I totonu o ogalaau o loʻo i lalo tatou te vaʻai pe faʻafefea ona faʻaalia le ata ma toe faʻafoʻi le Overshooting i le 14-71%. Mai lea taimi i lea taimi e faʻaitiitia ai le uta, lea e faʻaalia ai le Backpressure ma le HBase toe faʻapipiʻi nisi poloka.
Log RegionServer
tuliesea (MB): 0, fua faatatau 0.0, i luga o le ulu (%): -100, fa'asaga mamafa: 0, fa'amaumauga o lo'o i ai nei Fa'amaumauga (%): 100
tuliesea (MB): 0, fua faatatau 0.0, i luga o le ulu (%): -100, fa'asaga mamafa: 0, fa'amaumauga o lo'o i ai nei Fa'amaumauga (%): 100
tuliesea (MB): 2170, fua faatatau 1.09, luga aʻe (%): 985, faʻataʻitaʻiga mamafa: 1, faʻamaumauga o loʻo i ai nei DataBlock (%): 91 <amata
tuliesea (MB): 3763, fua faatatau 1.08, luga aʻe (%): 1781, faʻataʻitaʻiga faʻateʻa mamafa: 2, faʻamaumauga o loʻo i ai nei DataBlock (%): 76
tuliesea (MB): 3306, fua faatatau 1.07, luga aʻe (%): 1553, faʻataʻitaʻiga faʻateʻa mamafa: 3, faʻamaumauga o loʻo i ai nei DataBlock (%): 61
tuliesea (MB): 2508, fua faatatau 1.06, luga aʻe (%): 1154, faʻataʻitaʻiga faʻateʻa mamafa: 4, faʻamaumauga o loʻo i ai nei DataBlock (%): 50
tuliesea (MB): 1824, fua faatatau 1.04, luga aʻe (%): 812, faʻataʻitaʻiga faʻateʻa mamafa: 5, faʻamaumauga o loʻo i ai nei DataBlock (%): 42
tuliesea (MB): 1482, fua faatatau 1.03, luga aʻe (%): 641, faʻataʻitaʻiga faʻateʻa mamafa: 6, faʻamaumauga o loʻo i ai nei DataBlock (%): 36
tuliesea (MB): 1140, fua faatatau 1.01, luga aʻe (%): 470, faʻataʻitaʻiga faʻateʻa mamafa: 7, faʻamaumauga o loʻo i ai nei DataBlock (%): 32
tuliesea (MB): 913, fua faatatau 1.0, luga aʻe (%): 356, faʻataʻitaʻiga faʻateʻa mamafa: 8, faʻamaumauga o loʻo i ai nei DataBlock (%): 29
tuliesea (MB): 912, fua faatatau 0.89, luga aʻe (%): 356, faʻataʻitaʻiga faʻateʻa mamafa: 9, faʻamaumauga o loʻo i ai nei DataBlock (%): 26
tuliesea (MB): 684, fua faatatau 0.76, luga aʻe (%): 242, faʻataʻitaʻiga faʻateʻa mamafa: 10, faʻamaumauga o loʻo i ai nei DataBlock (%): 24
tuliesea (MB): 684, fua faatatau 0.61, luga aʻe (%): 242, faʻataʻitaʻiga faʻateʻa mamafa: 11, faʻamaumauga o loʻo i ai nei DataBlock (%): 22
tuliesea (MB): 456, fua faatatau 0.51, luga aʻe (%): 128, faʻataʻitaʻiga faʻateʻa mamafa: 12, faʻamaumauga o loʻo i ai nei DataBlock (%): 21
tuliesea (MB): 456, fua faatatau 0.42, luga aʻe (%): 128, faʻataʻitaʻiga faʻateʻa mamafa: 13, faʻamaumauga o loʻo i ai nei DataBlock (%): 20
tuliesea (MB): 456, fua faatatau 0.33, luga aʻe (%): 128, faʻataʻitaʻiga faʻateʻa mamafa: 14, faʻamaumauga o loʻo i ai nei DataBlock (%): 19
tuliesea (MB): 342, fua faatatau 0.33, luga aʻe (%): 71, faʻataʻitaʻiga faʻateʻa mamafa: 15, faʻamaumauga o loʻo i ai nei DataBlock (%): 19
tuliesea (MB): 342, fua faatatau 0.32, luga aʻe (%): 71, faʻataʻitaʻiga faʻateʻa mamafa: 16, faʻamaumauga o loʻo i ai nei DataBlock (%): 19
tuliesea (MB): 342, fua faatatau 0.31, luga aʻe (%): 71, faʻataʻitaʻiga faʻateʻa mamafa: 17, faʻamaumauga o loʻo i ai nei DataBlock (%): 19
tuliesea (MB): 228, fua faatatau 0.3, luga aʻe (%): 14, faʻataʻitaʻiga faʻateʻa mamafa: 18, faʻamaumauga o loʻo i ai nei DataBlock (%): 19
tuliesea (MB): 228, fua faatatau 0.29, luga aʻe (%): 14, faʻataʻitaʻiga faʻateʻa mamafa: 19, faʻamaumauga o loʻo i ai nei DataBlock (%): 19
tuliesea (MB): 228, fua faatatau 0.27, luga aʻe (%): 14, faʻataʻitaʻiga faʻateʻa mamafa: 20, faʻamaumauga o loʻo i ai nei DataBlock (%): 19
tuliesea (MB): 228, fua faatatau 0.25, luga aʻe (%): 14, faʻataʻitaʻiga faʻateʻa mamafa: 21, faʻamaumauga o loʻo i ai nei DataBlock (%): 19
tuliesea (MB): 228, fua faatatau 0.24, luga aʻe (%): 14, faʻataʻitaʻiga faʻateʻa mamafa: 22, faʻamaumauga o loʻo i ai nei DataBlock (%): 19
tuliesea (MB): 228, fua faatatau 0.22, luga aʻe (%): 14, faʻataʻitaʻiga faʻateʻa mamafa: 23, faʻamaumauga o loʻo i ai nei DataBlock (%): 19
tuliesea (MB): 228, fua faatatau 0.21, luga aʻe (%): 14, faʻataʻitaʻiga faʻateʻa mamafa: 24, faʻamaumauga o loʻo i ai nei DataBlock (%): 19
tuliesea (MB): 228, fua faatatau 0.2, luga aʻe (%): 14, faʻataʻitaʻiga faʻateʻa mamafa: 25, faʻamaumauga o loʻo i ai nei DataBlock (%): 19
tuliesea (MB): 228, fua faatatau 0.17, luga aʻe (%): 14, faʻataʻitaʻiga faʻateʻa mamafa: 26, faʻamaumauga o loʻo i ai nei DataBlock (%): 19
tuliesea (MB): 456, fua faatatau 0.17, i luga o le ulu (%): 128, mamafa eviction counter: 27, le taimi nei caching DataBlock (%): 18 <faaopoopo maua (ae laulau tutusa)
tuliesea (MB): 456, fua faatatau 0.15, luga aʻe (%): 128, faʻataʻitaʻiga faʻateʻa mamafa: 28, faʻamaumauga o loʻo i ai nei DataBlock (%): 17
tuliesea (MB): 342, fua faatatau 0.13, luga aʻe (%): 71, faʻataʻitaʻiga faʻateʻa mamafa: 29, faʻamaumauga o loʻo i ai nei DataBlock (%): 17
tuliesea (MB): 342, fua faatatau 0.11, luga aʻe (%): 71, faʻataʻitaʻiga faʻateʻa mamafa: 30, faʻamaumauga o loʻo i ai nei DataBlock (%): 17
tuliesea (MB): 342, fua faatatau 0.09, luga aʻe (%): 71, faʻataʻitaʻiga faʻateʻa mamafa: 31, faʻamaumauga o loʻo i ai nei DataBlock (%): 17
tuliesea (MB): 228, fua faatatau 0.08, luga aʻe (%): 14, faʻataʻitaʻiga faʻateʻa mamafa: 32, faʻamaumauga o loʻo i ai nei DataBlock (%): 17
tuliesea (MB): 228, fua faatatau 0.07, luga aʻe (%): 14, faʻataʻitaʻiga faʻateʻa mamafa: 33, faʻamaumauga o loʻo i ai nei DataBlock (%): 17
tuliesea (MB): 228, fua faatatau 0.06, luga aʻe (%): 14, faʻataʻitaʻiga faʻateʻa mamafa: 34, faʻamaumauga o loʻo i ai nei DataBlock (%): 17
tuliesea (MB): 228, fua faatatau 0.05, luga aʻe (%): 14, faʻataʻitaʻiga faʻateʻa mamafa: 35, faʻamaumauga o loʻo i ai nei DataBlock (%): 17
tuliesea (MB): 228, fua faatatau 0.05, luga aʻe (%): 14, faʻataʻitaʻiga faʻateʻa mamafa: 36, faʻamaumauga o loʻo i ai nei DataBlock (%): 17
tuliesea (MB): 228, fua faatatau 0.04, luga aʻe (%): 14, faʻataʻitaʻiga faʻateʻa mamafa: 37, faʻamaumauga o loʻo i ai nei DataBlock (%): 17
tuliesea (MB): 109, fua faatatau 0.04, i luga o le ulu (%): -46, mamafa fa'atauva'a: 37, fa'amaumauga o lo'o i ai nei DataBlock (%): 22 <fa'asolo i tua
tuliesea (MB): 798, fua faatatau 0.24, luga aʻe (%): 299, faʻataʻitaʻiga faʻateʻa mamafa: 38, faʻamaumauga o loʻo i ai nei DataBlock (%): 20
tuliesea (MB): 798, fua faatatau 0.29, luga aʻe (%): 299, faʻataʻitaʻiga faʻateʻa mamafa: 39, faʻamaumauga o loʻo i ai nei DataBlock (%): 18
tuliesea (MB): 570, fua faatatau 0.27, luga aʻe (%): 185, faʻataʻitaʻiga faʻateʻa mamafa: 40, faʻamaumauga o loʻo i ai nei DataBlock (%): 17
tuliesea (MB): 456, fua faatatau 0.22, luga aʻe (%): 128, faʻataʻitaʻiga faʻateʻa mamafa: 41, faʻamaumauga o loʻo i ai nei DataBlock (%): 16
tuliesea (MB): 342, fua faatatau 0.16, luga aʻe (%): 71, faʻataʻitaʻiga faʻateʻa mamafa: 42, faʻamaumauga o loʻo i ai nei DataBlock (%): 16
tuliesea (MB): 342, fua faatatau 0.11, luga aʻe (%): 71, faʻataʻitaʻiga faʻateʻa mamafa: 43, faʻamaumauga o loʻo i ai nei DataBlock (%): 16
tuliesea (MB): 228, fua faatatau 0.09, luga aʻe (%): 14, faʻataʻitaʻiga faʻateʻa mamafa: 44, faʻamaumauga o loʻo i ai nei DataBlock (%): 16
tuliesea (MB): 228, fua faatatau 0.07, luga aʻe (%): 14, faʻataʻitaʻiga faʻateʻa mamafa: 45, faʻamaumauga o loʻo i ai nei DataBlock (%): 16
tuliesea (MB): 228, fua faatatau 0.05, luga aʻe (%): 14, faʻataʻitaʻiga faʻateʻa mamafa: 46, faʻamaumauga o loʻo i ai nei DataBlock (%): 16
tuliesea (MB): 222, fua faatatau 0.04, luga aʻe (%): 11, faʻataʻitaʻiga faʻateʻa mamafa: 47, faʻamaumauga o loʻo i ai nei DataBlock (%): 16
tuliesea (MB): 104, fua faatatau 0.03, i luga o le ulu (%): -48, fa'asaga mamafa: 47, fa'amaumauga o lo'o iai Fa'amaumauga (%): 21 < fa'alavelave maua.
tuliesea (MB): 684, fua faatatau 0.2, luga aʻe (%): 242, faʻataʻitaʻiga faʻateʻa mamafa: 48, faʻamaumauga o loʻo i ai nei DataBlock (%): 19
tuliesea (MB): 570, fua faatatau 0.23, luga aʻe (%): 185, faʻataʻitaʻiga faʻateʻa mamafa: 49, faʻamaumauga o loʻo i ai nei DataBlock (%): 18
tuliesea (MB): 342, fua faatatau 0.22, luga aʻe (%): 71, faʻataʻitaʻiga faʻateʻa mamafa: 50, faʻamaumauga o loʻo i ai nei DataBlock (%): 18
tuliesea (MB): 228, fua faatatau 0.21, luga aʻe (%): 14, faʻataʻitaʻiga faʻateʻa mamafa: 51, faʻamaumauga o loʻo i ai nei DataBlock (%): 18
tuliesea (MB): 228, fua faatatau 0.2, luga aʻe (%): 14, faʻataʻitaʻiga faʻateʻa mamafa: 52, faʻamaumauga o loʻo i ai nei DataBlock (%): 18
tuliesea (MB): 228, fua faatatau 0.18, luga aʻe (%): 14, faʻataʻitaʻiga faʻateʻa mamafa: 53, faʻamaumauga o loʻo i ai nei DataBlock (%): 18
tuliesea (MB): 228, fua faatatau 0.16, luga aʻe (%): 14, faʻataʻitaʻiga faʻateʻa mamafa: 54, faʻamaumauga o loʻo i ai nei DataBlock (%): 18
tuliesea (MB): 228, fua faatatau 0.14, luga aʻe (%): 14, faʻataʻitaʻiga faʻateʻa mamafa: 55, faʻamaumauga o loʻo i ai nei DataBlock (%): 18
tuliesea (MB): 112, fua faatatau 0.14, i luga o le ulu (%): -44, mamafa fa'atauva'a: 55, fa'amaumauga o lo'o i ai nei DataBlock (%): 23 <fa'asolo i tua
tuliesea (MB): 456, fua faatatau 0.26, luga aʻe (%): 128, faʻataʻitaʻiga faʻateʻa mamafa: 56, faʻamaumauga o loʻo i ai nei DataBlock (%): 22
tuliesea (MB): 342, fua faatatau 0.31, luga aʻe (%): 71, faʻataʻitaʻiga faʻateʻa mamafa: 57, faʻamaumauga o loʻo i ai nei DataBlock (%): 22
tuliesea (MB): 342, fua faatatau 0.33, luga aʻe (%): 71, faʻataʻitaʻiga faʻateʻa mamafa: 58, faʻamaumauga o loʻo i ai nei DataBlock (%): 22
tuliesea (MB): 342, fua faatatau 0.33, luga aʻe (%): 71, faʻataʻitaʻiga faʻateʻa mamafa: 59, faʻamaumauga o loʻo i ai nei DataBlock (%): 22
tuliesea (MB): 342, fua faatatau 0.33, luga aʻe (%): 71, faʻataʻitaʻiga faʻateʻa mamafa: 60, faʻamaumauga o loʻo i ai nei DataBlock (%): 22
tuliesea (MB): 342, fua faatatau 0.33, luga aʻe (%): 71, faʻataʻitaʻiga faʻateʻa mamafa: 61, faʻamaumauga o loʻo i ai nei DataBlock (%): 22
tuliesea (MB): 342, fua faatatau 0.33, luga aʻe (%): 71, faʻataʻitaʻiga faʻateʻa mamafa: 62, faʻamaumauga o loʻo i ai nei DataBlock (%): 22
tuliesea (MB): 342, fua faatatau 0.33, luga aʻe (%): 71, faʻataʻitaʻiga faʻateʻa mamafa: 63, faʻamaumauga o loʻo i ai nei DataBlock (%): 22
tuliesea (MB): 342, fua faatatau 0.32, luga aʻe (%): 71, faʻataʻitaʻiga faʻateʻa mamafa: 64, faʻamaumauga o loʻo i ai nei DataBlock (%): 22
tuliesea (MB): 342, fua faatatau 0.33, luga aʻe (%): 71, faʻataʻitaʻiga faʻateʻa mamafa: 65, faʻamaumauga o loʻo i ai nei DataBlock (%): 22
tuliesea (MB): 342, fua faatatau 0.33, luga aʻe (%): 71, faʻataʻitaʻiga faʻateʻa mamafa: 66, faʻamaumauga o loʻo i ai nei DataBlock (%): 22
tuliesea (MB): 342, fua faatatau 0.32, luga aʻe (%): 71, faʻataʻitaʻiga faʻateʻa mamafa: 67, faʻamaumauga o loʻo i ai nei DataBlock (%): 22
tuliesea (MB): 342, fua faatatau 0.33, luga aʻe (%): 71, faʻataʻitaʻiga faʻateʻa mamafa: 68, faʻamaumauga o loʻo i ai nei DataBlock (%): 22
tuliesea (MB): 342, fua faatatau 0.32, luga aʻe (%): 71, faʻataʻitaʻiga faʻateʻa mamafa: 69, faʻamaumauga o loʻo i ai nei DataBlock (%): 22
tuliesea (MB): 342, fua faatatau 0.32, luga aʻe (%): 71, faʻataʻitaʻiga faʻateʻa mamafa: 70, faʻamaumauga o loʻo i ai nei DataBlock (%): 22
tuliesea (MB): 342, fua faatatau 0.33, luga aʻe (%): 71, faʻataʻitaʻiga faʻateʻa mamafa: 71, faʻamaumauga o loʻo i ai nei DataBlock (%): 22
tuliesea (MB): 342, fua faatatau 0.33, luga aʻe (%): 71, faʻataʻitaʻiga faʻateʻa mamafa: 72, faʻamaumauga o loʻo i ai nei DataBlock (%): 22
tuliesea (MB): 342, fua faatatau 0.33, luga aʻe (%): 71, faʻataʻitaʻiga faʻateʻa mamafa: 73, faʻamaumauga o loʻo i ai nei DataBlock (%): 22
tuliesea (MB): 342, fua faatatau 0.33, luga aʻe (%): 71, faʻataʻitaʻiga faʻateʻa mamafa: 74, faʻamaumauga o loʻo i ai nei DataBlock (%): 22
tuliesea (MB): 342, fua faatatau 0.33, luga aʻe (%): 71, faʻataʻitaʻiga faʻateʻa mamafa: 75, faʻamaumauga o loʻo i ai nei DataBlock (%): 22
tuliesea (MB): 342, fua faatatau 0.33, luga aʻe (%): 71, faʻataʻitaʻiga faʻateʻa mamafa: 76, faʻamaumauga o loʻo i ai nei DataBlock (%): 22
tuliesea (MB): 21, fua faatatau 0.33, i luga o le ulu (%): -90, fa'asaga mamafa: 76, fa'amaumauga o lo'o i ai nei Fa'amaumauga (%): 32
tuliesea (MB): 0, fua faatatau 0.0, i luga o le ulu (%): -100, fa'asaga mamafa: 0, fa'amaumauga o lo'o i ai nei Fa'amaumauga (%): 100
tuliesea (MB): 0, fua faatatau 0.0, i luga o le ulu (%): -100, fa'asaga mamafa: 0, fa'amaumauga o lo'o i ai nei Fa'amaumauga (%): 100
O suʻesuʻega na manaʻomia e faʻaalia ai le faiga lava lea e tasi i foliga o se kalafi o le va o vaega e lua o le cache - tasi (lea e leʻi talosagaina muamua poloka) ma le tele (faʻamatalaga "manaʻomia" a itiiti ifo ma le tasi e teuina iinei):
Ma le mea mulimuli, o le a le foliga o le faʻaogaina o faʻamaufaʻailoga i foliga o se kalafi. Mo le faʻatusatusaga, na tape atoa le cache i le amataga, ona faʻalauiloaina lea o le HBase ma le faʻaogaina ma faʻatuai le amataga o le galuega faʻapitoa e 5 minute (30 taamilosaga faʻateʻa).
E mafai ona maua le code atoa ile Pull Request luga github.
Ae ui i lea, 300 afe faitau i le sekone e le o mea uma e mafai ona ausia i luga o lenei meafaigaluega i lalo o nei tulaga. O le mea moni o le taimi e te manaʻomia ai le mauaina o faʻamatalaga e ala i le HDFS, o le ShortCircuitCache (o loʻo taʻua mulimuli ane o le SSC) e faʻaaogaina, lea e mafai ai ona e maua saʻo faʻamatalaga, aloese mai fegalegaleaiga fesoʻotaʻiga.
O faʻamatalaga na faʻaalia ai e ui lava o lenei masini e maua ai se tupe tele, ae i se taimi e avea ma fagu, aua e toetoe lava o gaioiga mamafa uma e tupu i totonu o se loka, lea e mafua ai le poloka o le tele o taimi.
I le iloaina o lenei mea, na matou iloa ai o le faʻafitauli e mafai ona faʻafefe e ala i le fatuina o se seti o SSC tutoʻatasi:
private final ShortCircuitCache[] shortCircuitCache;
...
shortCircuitCache = new ShortCircuitCache[this.clientShortCircuitNum];
for (int i = 0; i < this.clientShortCircuitNum; i++)
this.shortCircuitCache[i] = new ShortCircuitCache(…);
Ona galulue ai lea ma i latou, e le aofia ai foʻi fesoʻotaʻiga i le numera offset mulimuli:
public ShortCircuitCache getShortCircuitCache(long idx) {
return shortCircuitCache[(int) (idx % clientShortCircuitNum)];
}
Ua mafai nei ona e amata su'ega. Ina ia faia lenei mea, matou te faitau faila mai le HDFS faʻatasi ai ma se faʻaoga faigofie tele-filo. Seti fa'amaufa'ailoga:
conf.set("dfs.client.read.shortcircuit", "true");
conf.set("dfs.client.read.shortcircuit.buffer.size", "65536"); // по дефолту = 1 МБ и это сильно замедляет чтение, поэтому лучше привести в соответствие к реальным нуждам
conf.set("dfs.client.short.circuit.num", num); // от 1 до 10
Ma na ona faitau faila:
FSDataInputStream in = fileSystem.open(path);
for (int i = 0; i < count; i++) {
position += 65536;
if (position > 900000000)
position = 0L;
int res = in.read(position, byteBuffer, 0, 65536);
}
O lenei tulafono o loʻo faʻatinoina i filo eseese ma o le a matou faʻateleina le numera o faila faitau i le taimi e tasi (mai le 10 i le 200 - laina faʻasolosolo) ma le numera o faʻamau (mai le 1 i le 10 - ata). O le axis vertical o loʻo faʻaalia ai le saoasaoa e mafua mai i le faʻateleina o le SSC e faʻatatau i le mataupu pe a naʻo le tasi le cache.
Faʻafefea ona faitau le kalafi: O le faʻatinoga o le taimi mo le 100 afe faitau i le 64 KB poloka ma le tasi cache e manaʻomia le 78 sekone. Ae o le 5 caches e manaʻomia le 16 sekone. O na. o loʻo i ai se faʻavavevave o ~ 5 taimi. E pei ona mafai ona vaʻaia mai le kalafi, o le aʻafiaga e leʻo iloa tele mo se numera laʻititi o faitauga tutusa, e amata ona faʻaalia se sao pe a sili atu nai lo le 50 filo faitau. E maitauina foi le faʻateleina o le numera o SSCs mai le 6 ma o lo'o i luga e maua ai se si'itaga la'ititi fa'atino.
Fa'aaliga 1: talu ai o fa'ai'uga o su'ega e fai si fe'avea'i (silasila i lalo), 3 ta'aloga na fa'atinoina ma fa'afuainumera tau maua.
Fa'aaliga 2: O le fa'atinoga e maua mai le fa'atulagaina o avanoa fa'afuase'i e tutusa, e ui o le avanoa lava ia e fa'agesegese teisi.
Ae ui i lea, e tatau ona faʻamalamalamaina, e le pei o le HBase, o lenei faʻavavevave e le o taimi uma e saoloto. O iinei tatou te "tatala" ai le malosi ole PPU e sili atu galuega, nai lo le tautau i loka.
O iinei e mafai ona e matauina, i se tulaga lautele, o le faateleina o le numera o caches e maua ai se faʻaopoopoga faʻatusatusa i le faʻaogaina o le CPU. Ae ui i lea, o loʻo i ai teisi sili atu tuʻufaʻatasiga manumalo.
Mo se faʻataʻitaʻiga, seʻi o tatou vaʻavaʻai totoʻa i le seti SSC = 3. O le faʻatupulaia o le faʻatinoga i luga ole laiga e tusa ma le 3.3 taimi. O lo'o i lalo fa'ai'uga mai ta'aloga eseese uma e tolu.
Aʻo faʻateleina le faʻaaogaina o le CPU e tusa ma le 2.8 taimi. E le tele tele le eseesega, ae ua fiafia Greta laitiiti ma atonu e maua se taimi e alu ai i le aoga ma fai lesona.
O le mea lea, o le ai ai se aafiaga lelei mo soʻo se meafaigaluega e faʻaogaina le tele o avanoa i le HDFS (mo se faʻataʻitaʻiga Spark, ma isi), pe a fai e mama le numera o talosaga (o lona uiga o le polo o loʻo i luga o le HDFS tagata faʻatau) ma o loʻo i ai le mana CPU saoloto. . Ina ia siaki, seʻi o tatou suʻeina pe o le a le aafiaga o le tuʻufaʻatasia o le faʻaogaina o le BlockCache optimization ma le SSC faʻalogo mo le faitau mai le HBase o le ai ai.
E mafai ona iloa i lalo o ia tulaga o le aʻafiaga e le sili atu e pei o suʻega faʻamamaina (faitau e aunoa ma se gaioiga), ae e mafai lava ona faʻapipiʻi se 80K faaopoopo iinei. Faʻatasi, o faʻataʻitaʻiga uma e tuʻuina atu i le 4x saoasaoa.
Sa faia fo'i se PR mo lea fa'atonuga , lea ua tu'ufa'atasia ma o lenei fa'atinoga o le a maua i fa'asalalauga i le lumana'i.
Ma le mea mulimuli, e manaia le faʻatusatusaina o le faitau faʻatinoga o se faʻamaumauga lautele lautele-koluma, Cassandra ma HBase.
Ina ia faia lenei mea, na matou faʻalauiloaina faʻataʻitaʻiga o le YCSB uta faʻataʻitaʻiga aoga mai 'au e lua (800 filo i le aofaʻi). I luga o le 'auʻaunaga - 4 faʻataʻitaʻiga o RegionServer ma Cassandra i luga o 4 'au (e le o mea o loʻo taʻavale ai tagata faʻatau, e aloese mai a latou faatosinaga). Faitauga na maua mai laulau tetele:
HBase - 300 GB ile HDFS (100 GB faʻamaumauga mama)
Cassandra - 250 GB (tulaga fa'atusa = 3)
O na. o le voluma e tutusa lava (i le HBase teisi atu).
HBase tapula'a:
dfs.client.short.circuit.num = 5 (Fa'ata'ita'iga a tagata fa'atau HDFS)
hbase.lru.cache.heavy.eviction.count.limit = 30 - o lona uiga o le patch o le a amata ona galue pe a maeʻa le 30 tuliesea (~ 5 minute)
hbase.lru.cache.heavy.eviction.mb.size.limit = 300 - fa'atatau le tele o le fa'aoga ma le fa'ate'aina
O ogalaau a le YCSB sa fa'avasega ma tu'ufa'atasia i kalafi Excel:
E pei ona e vaʻaia, o nei faʻataʻitaʻiga e mafai ai ona faʻatusatusa le faʻatinoga o nei faʻamaumauga i lalo o nei tulaga ma ausia le 450 afe faitau i le sekone.
Matou te faʻamoemoe e mafai ona aoga lenei faʻamatalaga i se tasi i le taimi o tauiviga fiafia mo le gaosiga.
puna: www.habr.com