Paunenge uchishanda nehuwandu hwe data, dambudziko rekushayikwa kwedhisiki nzvimbo inogona dzimwe nguva kubuda. Imwe nzira yekugadzirisa dambudziko iri ndeyekumanikidza, nekuda kwekuti, pamidziyo imwe chete, iwe unogona kukwanisa kuwedzera mavhoriyamu ekuchengetedza. Muchikamu chino, tichatarisa kuti kudzvanywa kwedata kunoshanda sei muApache Ignite. Ichi chinyorwa chinotsanangura chete dhisiki yekumanikidza nzira dzinoitwa mukati mechigadzirwa. Dzimwe nzira dzekumanikidza data (pamusoro penetiweki, mundangariro), ingave yaitwa kana kwete, icharamba iri kunze kwechikamu.
Saka, nekushingirira modhi yakagoneswa, semhedzisiro yekuchinja mune data mumacache, Ignite inotanga kunyora ku diski:
- Zviri mukati me cache
- Nyora Pamberi Rogi (pano zviri nyore WAL)
Pave paine nzira yekumanikidza WAL kwenguva yakati rebei, inonzi WAL compaction. Iyo ichangobva kuburitswa Apache Ignite 2.8 yakaunza mamwe maviri mashandiro ayo anobvumidza iwe kudzvanya data pa diski: dhisiki peji compression yekumanikidza zviri mukati mecache uye WAL peji snapshot compression yekumanikidza mamwe maWAL. Mamwe ruzivo nezve ese matatu ezvigadziriso pazasi.
Diski peji compression
Sei basa iri
Kutanga, ngatitorei pfupi pfupi kutarisa kuti Ignite inochengeta sei data. Peji ndangariro inoshandiswa kuchengetedza. Saizi yepeji inoiswa pakutanga kwenode uye haigone kuchinjika pane dzinotevera nhanho; zvakare, saizi yepeji inofanira kunge iri simba remaviri uye akawanda eiyo faira system block saizi. Mapeji anoiswa muRAM kubva kudhisiki sezvinodiwa; saizi yedata padhisiki inogona kudarika huwandu hwe RAM yakagoverwa. Kana pasina nzvimbo yakakwana mu RAM yekuisa peji kubva kudhisiki, yekare, mapeji asingachashandiswi achadzingwa kubva ku RAM.
Iyo data inochengetwa pa diski mune inotevera fomu: faira rakasiyana rinogadzirwa kune yega yega yeboka cache; mune iyi faira, mapeji anoonekwa rimwe mushure meimwe mukukwira index order. Iyo yakazara peji identifier ine cache group identifier, partition nhamba, uye peji index mufaira. Nekudaro, tichishandisa iyo yakazara peji identifier, isu tinokwanisa kusarudza zvakasiyana faira uye offset mufaira repeji rega rega. Unogona kuverenga zvakawanda nezve paging memory mune iyo Apache Ignite Wiki chinyorwa: .
Iyo disk peji compression mechanism, sezvaungafungidzira kubva pazita, inoshanda padanho repeji. Kana iyi michina ikagoneswa, data mu RAM inogadziriswa sezvairi, pasina kudzvanywa, asi kana mapeji achengetwa kubva ku RAM kuenda ku diski, anomanikidzwa.
Asi kudzvanya peji yega yega haisi mhinduro kudambudziko; iwe unofanirwa kudzikisa saizi yemafaira edatha. Kana saizi yepeji isingachagadziriswi, hatichakwanisa kunyora mapeji kufaira rimwe nerimwe, sezvo izvi zvichigona kugadzira akati wandei matambudziko:
- Tichishandisa iyo index yepeji, hatizokwanisi kuverenga iyo yekubvisa iyo inowanikwa mufaira.
- Hazvisi pachena kuti chii chekuita nemapeji asiri pamagumo efaira uye kuchinja ukuru hwavo. Kana saizi yepeji ichiderera, nzvimbo yayakasunungura inonyangarika. Kana saizi yepeji ichiwedzera, unofanirwa kutsvaga nzvimbo nyowani mufaira yacho.
- Kana peji ikafamba nehuwandu hwemabhaiti asiri akawanda eiyo faira system block saizi, ipapo kuiverenga kana kuinyora kunoda kubata imwezve faira system block, izvo zvinogona kutungamirira mukuderera kwekuita.
Kuti udzivise kugadzirisa matambudziko aya pamwero wayo wega, disk peji compression muApache Ignite inoshandisa faira system system inonzi sparse mafaera. A sparse faira nderimwe umo mamwe matunhu akazadzwa zero anogona kunyorwa se "makomba". Muchiitiko ichi, hapana zvivharo zvefaira system zvichagoverwa kuchengetedza maburi aya, zvichikonzera kuchengetedza pane disk nzvimbo.
Zvine musoro kuti kusunungura faira system block, saizi yegomba inofanira kunge yakakura kupfuura kana kuenzana neiyo faira system block, iyo inoisa imwezve miganho pahukuru hwepeji uye Apache Ignite: kuti compression iite chero mhedzisiro, saizi yepeji inofanira kunge yakakura zvakanyanya kupfuura saizi yefaira system block. Kana saizi yepeji yakaenzana nehukuru hwebhuroka, saka isu hatife takakwanisa kusunungura bhuroka rimwe chete, sezvo kuti usunungure bhuroka rimwe chete, peji rakamanikidzwa rinofanira kutora 0 bytes. Kana saizi yepeji yakaenzana nehukuru hwe2 kana 4 mabhuroko, isu tinenge tatokwanisa kusunungura kamwechete bhuroko kana peji yedu yakamanikidzwa kusvika pa50% kana 75% zvichiteerana.
Saka, tsananguro yekupedzisira yemashandisirwo anoita michina: Pakunyora peji kune diski, kuyedza kunoitwa kumanikidza peji. Kana saizi yepeji yakamanikidzwa ichibvumira imwe kana akawanda faira system mabhuroki kusunungurwa, ipapo peji inonyorwa mufomu yakamanikidzwa, uye "gomba" rinoitwa panzvimbo yezvivharo zvakasunungurwa (kufona system kunoitwa. fallocate() nemureza wechibhakera). Kana saizi yepeji yakamanikidzwa isingabvumiri kuti mabhuraki asunungurwe, peji inochengetwa sezvairi, isina kudzvanywa. Ese mapeji ekubvisa anoverengerwa nenzira imwechete pasina kudzvanya, nekuwanza indekisi yepeji nehukuru hwepeji. Hapana kutamiswa kwemapeji kunodiwa uri wega. Mapeji ekubvisa, senge pasina kudzvanywa, anowira pamiganhu yefaira system blocks.
Mukuita kwazvino, Ignite inogona kungoshanda nemafaira mashoma pasi peLinux OS; saizvozvo, dhisiki peji yekumanikidza inogona chete kugoneswa kana uchishandisa Ignite pane ino inoshanda sisitimu.
Compression algorithms inogona kushandiswa kudhisiki peji peji: ZSTD, LZ4, Snappy. Mukuwedzera, kune maitiro ekushanda (SKIP_GARBAGE), umo nzvimbo isina kushandiswa chete peji inokandwa kunze pasina kushandisa kudzvinyirira pane data yakasara, iyo inoderedza mutoro paCPU kana ichienzaniswa nealgorithms yakanyorwa kare.
Performance Impact
Nehurombo, ini handina kuitisa zviyero zvekuita chaiko pamastands chaiwo, sezvo isu tisingaronge kushandisa iyi michina mukugadzira, asi isu tinogona kufungidzira kwaticharasikirwa uye kwatichahwina.
Kuti tiite izvi, tinofanira kuyeuka kuti mapeji anoverengwa nekunyorwa sei kana awanikwa:
- Paunenge uchiita basa rekuverenga, rinotanga kutsvakwa muRAM; kana kutsvaga kukasabudirira, peji inoiswa muRAM kubva kudhisiki neshinda imwechete inoita kuverenga.
- Kana basa rekunyora richiitwa, peji iri mu RAM inotaridzwa setsvina, asi peji racho harina kuchengetwa mumuviri ku diski nekukasira neshinda iri kuita kunyora. Ese mapeji akasviba anochengetwa kudhisiki gare gare mukutarisa kwekutarisa mune akasiyana shinda.
Saka zvinokanganisa kuverenga mabasa ndeizvi:
- Positive (disk IO), nekuda kwekuderera kwenhamba yekuverenga mafaira system blocks.
- Negative (CPU), nekuda kwekuwedzera mutoro unodiwa neanoshanda sisitimu yekushanda nemafaira mashoma. Izvo zvakare zvinogoneka kuti ekuwedzera IO mashandiro anozoonekwa zvakajeka pano kuchengetedza yakaomesesa sparse faira chimiro (nehurombo, ini handizive nezvese ruzivo rwekuti sparse mafaera anoshanda sei).
- Negative (CPU), nekuda kwekudiwa kwekudzikisa mapeji.
- Iko hakuna kukanganisa pakunyora mashandiro.
- Impact pane yekutarisa maitiro (zvese zviri pano zvakafanana nekuverenga mashandiro):
- Positive (disk IO), nekuda kwekuderera kwenhamba yakanyorwa faira system blocks.
- Negative (CPU, pamwe disk IO), nekuda kwekushanda nemafaira mashoma.
- Negative (CPU), nekuda kwekudiwa kwekumanikidza peji.
Idivi ripi rechikero richapinza chikero? Izvi zvese zvakanyanya zvinoenderana nenharaunda, asi ini ndinoda kutenda kuti dhisiki peji compression ingangotungamira mukuderedzwa kwekuita pane mazhinji masisitimu. Uyezve, bvunzo kune mamwe maDBMS anoshandisa nzira yakafanana nemafaira mashoma anoratidza kudonha kwekuita kana kudzvanywa kuchigoneswa.
Nzira yekugonesa uye kugadzirisa
Sezvambotaurwa pamusoro, iyo shoma vhezheni yeApache Ignite inotsigira disk peji compression ndeye 2.8 uye chete Linux inoshanda sisitimu inotsigirwa. Gonesa uye gadzirisa sezvinotevera:
- Panofanira kuva nemoto-kumanikidza module mukirasi-nzira. Nekumisikidza, inowanikwa muApache Ignite kugovera mune libs / sarudzo dhairekitori uye haina kubatanidzwa mukirasi-nzira. Iwe unogona kungofambisa iyo dhairekitori kumusoro kune imwe nhanho kune libs uye kana iwe uchimhanyisa kuburikidza neignite.sh inozogoneswa otomatiki.
- Kushingirira kunofanirwa kugoneswa (Inogoneswa kuburikidza
DataRegionConfiguration.setPersistenceEnabled(true)). - Saizi yepeji inofanirwa kuve yakakura kupfuura iyo faira system block saizi (unogona kuiisa uchishandisa
DataStorageConfiguration.setPageSize()). - Kune yega yega cache ine data inoda kudzvanywa, iwe unofanirwa kugadzirisa iyo compression nzira uye (optional) iyo yekumanikidza level (maitiro.
CacheConfiguration.setDiskPageCompression() , CacheConfiguration.setDiskPageCompressionLevel()).
WAL compaction
Sei basa iri
Chii chinonzi WAL uye nei ichidikanwa? Muchidimbu: iyi ilogi ine zvese zviitiko zvinozopedzisira zvachinja peji rekuchengetedza. Zvinodiwa zvakanyanya kuti ugone kupora kana ukadonha. Chero kushanda, kusati kwapa kutonga kumushandisi, kunofanirwa kutanga kurekodha chiitiko muWAL, kuitira kuti kana ikatadza, inogona kuridzwa zvakare mugiyo uye kudzoreredza mashandiro ayo mushandisi akagamuchira mhinduro yakabudirira, kunyangwe izvi zviitwa. yakanga isina nguva yekuratidzwa mune peji rekuchengetedza pa diski (yatova pamusoro Yakatsanangurwa kuti kunyora chaiko kuchitoro chepeji kunoitwa nenzira inonzi "checkpointing" nekumwe kunonoka neshinda dzakasiyana).
Mapinda muWAL akakamurwa kuita zvine musoro uye zvemuviri. Boolean ndiwo makiyi uye vanozvikoshesa ivo pachavo. Muviri - inoratidza shanduko kumapeji muchitoro chepeji. Nepo marekodhi ane musoro achigona kubatsira kune dzimwe nyaya, marekodhi emuviri anodiwa chete pakupora kana paita tsaona uye marekodhi anongodiwa chete kubva pakabudirira chekupedzisira. Pano isu hatisi kuzopinda mune zvakadzama uye kutsanangura kuti sei ichishanda nenzira iyi, asi avo vanofarira vanogona kutarisa kune yakambotaurwa chinyorwa paApache Ignite Wiki: .
Kune kazhinji kanoverengeka zvinyorwa zvemuviri pane zvine musoro rekodhi. Ndokunge, semuenzaniso, kumwe kuisa kushanda mu cache kunobata akati wandei mapeji mundangariro peji (peji ine data pachayo, mapeji ane indexes, mapeji ane emahara-zvinyorwa). Mune mamwe maedzo ekugadzira, ndakaona kuti zvinyorwa zvenyama zvakagara kusvika makumi mapfumbamwe muzana eWAL faira. Nekudaro, ivo vanodiwa kwenguva pfupi kwazvo (nekusagadzikana, iyo nguva pakati pekutarisa maminetsi matatu). Zvingave zvine musoro kubvisa iyi data mushure mekurasikirwa nekukosha kwayo. Izvi ndizvo chaizvo zvinoitwa neWAL compaction mechanism: inobvisa marekodhi emuviri uye inodzvanya marekodhi ane musoro asara uchishandisa zip, nepo saizi yefaira yakaderedzwa zvakanyanya (dzimwe nguva nemakumi enguva).
Panyama, WAL ine zvikamu zvakati wandei (gumi nekusingaperi) zvehukuru hwakatarwa (10MB nekusarudzika), izvo zvinonyorwa pamusoro nenzira yedenderedzwa. Kana chikamu chazvino chazadzwa, chikamu chinotevera chinopihwa sechizvino, uye chikamu chakazadzwa chinoteedzerwa kudura netambo yakasiyana. WAL compaction inotoshanda nezvikamu zvekuchengetedza. Zvakare, setambo yakaparadzana, inotarisisa kuitwa kwecheki uye inotanga kudzvanya muzvikamu zvekuchengetedza izvo zvinyorwa zvenyama zvisingachadiwi.
Performance Impact
Sezvo WAL compaction ichimhanya setambo yakaparadzana, hapafanirwe kuve nemhedzisiro yakananga pama operation ari kuitwa. Asi ichiri kuisa yakawedzera kumashure mutoro paCPU (compression) uye dhisiki (kuverenga yega yega WAL chikamu kubva mudura uye kunyora zvikamu zvakamanikidzwa), saka kana sisitimu iri kushanda pakukwanisa kwayo, inotungamira mukuderedzwa kwekuita.
Nzira yekugonesa uye kugadzirisa
Iwe unogona kugonesa WAL compaction uchishandisa iyo pfuma WalCompactionEnabled Π² DataStorageConfiguration (DataStorageConfiguration.setWalCompactionEnabled(true)) Zvakare, uchishandisa iyo DataStorageConfiguration.setWalCompactionLevel() nzira, unogona kuseta iyo yekumanikidza level kana iwe usingagutsikane neiyo default kukosha (BEST_SPEED).
WAL peji mufananidzo wekumanikidza
Sei basa iri
Takatoona kuti muWAL zvinyorwa zvakakamurwa kuva zvine musoro uye zvenyama. Pakuchinja kwega kwega kune peji rega rega, rekodhi yeWAL yemuviri inogadzirwa mundangariro yepeji. Zvinyorwa zvenyama, zvakare, zvakakamurwa kuita 2 subtypes: peji snapshot rekodhi uye delta rekodhi. Nguva imwe neimwe yatinoshandura chimwe chinhu pane peji uye tichichiendesa kubva kunzvimbo yakachena kuenda kune yakasviba, kopi yakakwana yeiyi peji inochengetwa muWAL (peji snapshot rekodhi). Kunyangwe tikashandura biti imwe chete muWAL, rekodhi ichave yakakura zvishoma pane saizi yepeji. Kana tikashandura chimwe chinhu pane peji yakatosvibiswa, ipapo delta rekodhi inoumbwa muWAL, iyo inongoratidzira kuchinja chete kana ichienzaniswa neyakapfuura mamiriro epeji, asi kwete peji rose. Sezvo kugadzirisazve mamiriro emapeji kubva kune tsvina kusvika kukuchenesa kunoitwa panguva yekutarisa, pakarepo mushure mekutanga kwekutarisa, anenge ese marekodhi emuviri achange aine snapshots yemapeji (sezvo mapeji ese pakarepo mushure mekutanga kwekutarisa akachena) , zvino sezvatinoswedera kune inotevera cheki, chikamu che delta rekodhi chinotanga kukura nekugadzirisa zvakare pakutanga kwecheki inotevera. Zviyero mune mamwe maedzo ekugadzira zvakaratidza kuti mugove wepeji snapshots muhuwandu hwehuwandu hwezvinyorwa zvemuviri unosvika 90%.
Pfungwa yeWAL peji snapshot compression ndeyekumanikidza mapeji snapshots uchishandisa yakagadzirira-yakagadzirwa peji compression chishandiso (ona disk peji compression). Panguva imwecheteyo, muWAL, zvinyorwa zvinochengetwa zvakatevedzana muappend-chete modhi uye hapana chikonzero chekusunga marekodhi kumiganhu yefaira system blocks, saka pano, kusiyana nedhisiki peji yekumanikidza nzira, isu hatidi sparse mafaira pa. zvese; nekudaro, iyi michina haishande kwete chete pane OS Linux. Pamusoro pezvo, hazvichina basa kwatiri kuti takakwanisa sei kudzvanya peji. Kunyangwe tikasunungura 1 byte, iyi yatove mhedzisiro uye tinogona kuchengetedza data yakamanikidzwa muWAL, kusiyana nedhisiki peji yekumanikidza, kwatinochengetedza peji yakamanikidzwa chete kana isu takasunungura zvinopfuura 1 faira system block.
Mapeji ane data yakanyanya kudzvanywa, mugove wavo muhuwandu hweWAL huwandu hwakanyanya, saka pasina kushandura WAL faira fomati tinogona kuwana kuderedzwa kwakanyanya muhukuru hwayo. Kudzvinyirira, kusanganisira zvinyorwa zvine musoro, zvingada kuchinja kwemaitiro uye kurasikirwa kwekuenderana, semuenzaniso, kune vatengi vekunze vangave vachifarira zvinyorwa zvine musoro, asi hazvizoiti kuti kuderedzwa kukuru kwefaira refaira.
Sezvakaita disk peji compression, WAL peji snapshot compression inogona kushandisa ZSTD, LZ4, Snappy compression algorithms, pamwe neSKIP_GARBAGE modhi.
Performance Impact
Hazvina kuoma kuona kuti kugonesa zvakananga WAL peji snapshot compression inongobata tambo dzinonyora data kune peji ndangariro, ndiko kuti, idzo tambo dzinoshandura data mumacache. Kuverenga zvinyorwa zvemuviri kubva kuWAL kunoitika kamwe chete, panguva iyo node inosimudzwa mushure mekudonha (uye chete kana ikawira panguva yekutarisa).
Izvi zvinokanganisa tambo dzinoshandura data nenzira inotevera: tinowana kukanganisa (CPU) nekuda kwekudiwa kwekumanikidza peji nguva imwe neimwe tisati tanyorera ku diski, uye mhedzisiro yakanaka (disk IO) nekuda kwekuderedzwa kwehuwandu hwe. data rakanyorwa. Saizvozvo, zvinhu zvese zviri nyore pano: kana hurongwa hwekuita hwakaganhurirwa neCPU, tinowana kuderera kudiki, kana ikaganhurwa ne diski I / O, tinowana kuwedzera.
Zvisina kunanga, kuderedza saizi yeWAL kunokanganisawo (zvakanaka) hova dzinorasa zvikamu zveWAL mudura uye WAL compaction hova.
Miedzo chaiyo yekushanda munharaunda yedu uchishandisa data yekugadzira yakaratidza kuwedzera kuduku (kuburikidza kwakawedzera ne10% -15%, latency yakaderera ne10% -15%).
Nzira yekugonesa uye kugadzirisa
Minimum Apache Ignite vhezheni: 2.8. Gonesa uye gadzirisa sezvinotevera:
- Panofanira kuva nemoto-kumanikidza module mukirasi-nzira. Nekumisikidza, inowanikwa muApache Ignite kugovera mune libs / sarudzo dhairekitori uye haina kubatanidzwa mukirasi-nzira. Iwe unogona kungofambisa iyo dhairekitori kumusoro kune imwe nhanho kune libs uye kana iwe uchimhanyisa kuburikidza neignite.sh inozogoneswa otomatiki.
- Kushingirira kunofanirwa kugoneswa (Inogoneswa kuburikidza
DataRegionConfiguration.setPersistenceEnabled(true)). - Iyo compression mode inofanirwa kusetwa uchishandisa nzira
DataStorageConfiguration.setWalPageCompression(), compression yakadzimwa nekusarudzika (DISABLED modhi). - Optionally, iwe unogona kuseta iyo yekumanikidza level uchishandisa nzira
DataStorageConfiguration.setWalPageCompression(), ona iyo javadoc yenzira yemaitiro anoshanda kune yega yega modhi.
mhedziso
Iyo inotariswa data yekumanikidza nzira muApache Ignite inogona kushandiswa yakazvimiririra kune imwe neimwe, asi chero musanganiswa wavo unogamuchirwawo. Kunzwisisa mashandisiro avanoita kuchakubvumidza kuti uone kuti vakakodzera sei mabasa ako munzvimbo yaunogara uye zvauchafanira kuzvipira kana uchivashandisa. Disk peji compression yakagadzirirwa kudzvanya iyo huru yekuchengetedza uye inogona kupa yepakati compression reshiyo. WAL peji muchidimbu kudzvanya ichapa avhareji dhigirii yekumanikidza yeWAL mafaera, uye ingango wedzera mashandiro. WAL compaction haizove nemigumisiro yakanaka pakushanda, asi ichaderedza hukuru hweWAL mafaira zvakanyanya sezvinobvira nekubvisa zvinyorwa zvepanyama.
Source: www.habr.com