{"id":95783,"date":"2020-10-03T13:42:02","date_gmt":"2020-10-03T11:42:02","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/nastrojka-yadra-linux-dlya-glusterfs"},"modified":"2020-10-03T13:42:02","modified_gmt":"2020-10-03T11:42:02","slug":"nastrojka-yadra-linux-dlya-glusterfs","status":"publish","type":"post","link":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/nastrojka-yadra-linux-dlya-glusterfs","title":{"rendered":"Nastaven\u00ed j\u00e1dra Linuxu pro GlusterFS","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

P\u0159eklad \u010dl\u00e1nku byl vyhotoven p\u0159ed za\u010d\u00e1tkem kurzu \u00abLinux Administrator. Professional\u00bb<\/a><\/noindex>.<\/i><\/b><\/p>\n

\"Nastaven\u00ed<\/p>\n

Ob\u010das se vyno\u0159uj\u00ed dotazy ohledn\u011b doporu\u010den\u00ed Gluster k nastaven\u00ed j\u00e1dra a zda je to v\u016fbec nutn\u00e9.<\/p>\n

Takov\u00e1 pot\u0159eba vznik\u00e1 z\u0159\u00eddka. Ve v\u011bt\u0161in\u011b z\u00e1t\u011b\u017e\u00ed j\u00e1dro funguje velmi dob\u0159e. I kdy\u017e je tu i druh\u00e1 strana. Historicky j\u00e1dro Linuxu ochotn\u011b spot\u0159ebov\u00e1v\u00e1 hodn\u011b pam\u011bti, pokud mu to dovol\u00edte, v\u010detn\u011b vyu\u017e\u00edv\u00e1n\u00ed jako hlavn\u00edho zp\u016fsobu zv\u00fd\u0161en\u00ed v\u00fdkonu.<\/a><\/noindex><\/p>\n

V\u011bt\u0161inou to funguje skv\u011ble, ale p\u0159i vysok\u00e9 z\u00e1t\u011b\u017ei to m\u016f\u017ee zp\u016fsobit probl\u00e9my. <\/p>\n

M\u00e1me bohat\u00e9 zku\u0161enosti se syst\u00e9my s vysokou spot\u0159ebou pam\u011bti, jako jsou CAD, EDA a podobn\u011b, kter\u00e9 za\u010daly za vysok\u00e9 z\u00e1t\u011b\u017ee zadrh\u00e1vat. A ob\u010das jsme se setk\u00e1vali s probl\u00e9my v Glusteru. Pe\u010dliv\u00fdm sledov\u00e1n\u00edm spot\u0159eby pam\u011bti a \u010dekac\u00ed doby disk\u016f jsme za\u017eili jejich p\u0159et\u00ed\u017een\u00ed, obrovsk\u00e9 iowait, chyby j\u00e1dra (kernel oops), zamrznut\u00ed atd. <\/p>\n

Tento \u010dl\u00e1nek je v\u00fdsledkem mnoha experiment\u016f s konfigurac\u00ed parametr\u016f proveden\u00fdch v r\u016fzn\u00fdch situac\u00edch. D\u00edky t\u011bmto parametr\u016fm se nejen zlep\u0161ila celkov\u00e1 odezva, ale tak\u00e9 se v\u00fdrazn\u011b stabilizoval provoz clustera.<\/p>\n

Kdy\u017e jde o nastaven\u00ed pam\u011bti, je t\u0159eba se nejprve zam\u011b\u0159it na subsyst\u00e9m virtu\u00e1ln\u00ed pam\u011bti (VM, virtual memory), kter\u00fd m\u00e1 spoustu mo\u017enost\u00ed, je\u017e v\u00e1s mohou zm\u00e1st.<\/p>\n

vm.swappiness<\/h4>\n

\nParameter vm.swappiness<\/code> ur\u010duje, jak moc j\u00e1dro pou\u017e\u00edv\u00e1 swapov\u00e1n\u00ed ve srovn\u00e1n\u00ed s opera\u010dn\u00ed pam\u011bt\u00ed. V zdrojov\u00e9m k\u00f3du je tak\u00e9 definov\u00e1no jako \u201etendency to steal mapped memory\u201c (sklon k kr\u00e1de\u017ei mapovan\u00e9 pam\u011bti). Vysok\u00e1 hodnota swappiness znamen\u00e1, \u017ee j\u00e1dro bude ochotn\u011bj\u0161\u00ed vyhazovat mapovan\u00e9 str\u00e1nky. N\u00edzk\u00e1 hodnota swappiness znamen\u00e1 opak: j\u00e1dro bude m\u00e9n\u011b vyhazovat str\u00e1nky z pam\u011bti. Jin\u00fdmi slovy, \u010d\u00edm vy\u0161\u0161\u00ed je hodnota vm.swappiness<\/code>, t\u00edm v\u00edce syst\u00e9m vyu\u017e\u00edv\u00e1 swap.<\/p>\n

Velk\u00e9 vyu\u017e\u00edv\u00e1n\u00ed swapov\u00e1n\u00ed nen\u00ed \u017e\u00e1douc\u00ed, proto\u017ee do opera\u010dn\u00ed pam\u011bti se na\u010d\u00edtaj\u00ed a vyhazuj\u00ed obrovsk\u00e9 bloky dat. Mnoz\u00ed tvrd\u00ed, \u017ee hodnota swapiness by m\u011bla b\u00fdt vysok\u00e1, ale z m\u00e9 zku\u0161enosti vede k zv\u00fd\u0161en\u00ed v\u00fdkonu nastaven\u00ed na \u201e0\u201c.<\/p>\n

Dal\u0161\u00ed informace lze p\u0159e\u010d\u00edst zde \u2014 lwn.net\/Articles\/100978<\/a><\/noindex> <\/p>\n

Av\u0161ak tato nastaven\u00ed by m\u011bla b\u00fdt pou\u017e\u00edv\u00e1na opatrn\u011b a pouze po testov\u00e1n\u00ed konkr\u00e9tn\u00ed aplikace. Pro vysoce zat\u00ed\u017een\u00e9 streamovac\u00ed aplikace by m\u011bla b\u00fdt tato hodnota nastavena na '0'. P\u0159i zm\u011bn\u011b na '0' se zlep\u0161\u00ed odezva syst\u00e9mu.<\/p>\n

vm.vfs_cache_pressure<\/h4>\n

\nTato parametrov\u00e1 hodnota \u0159\u00edd\u00ed pam\u011b\u0165, kterou j\u00e1dro vyu\u017e\u00edv\u00e1 pro ke\u0161ov\u00e1n\u00ed objekt\u016f adres\u00e1\u0159\u016f a indexov\u00fdch deskriptor\u016f (dentry a inode).<\/p>\n

P\u0159i v\u00fdchoz\u00ed hodnot\u011b 100 se j\u00e1dro pokus\u00ed uvolnit ke\u0161 dentry a inode 'spravedliv\u011b' ve vztahu k pagecache a swapcache. Sn\u00ed\u017een\u00ed vfs_cache_pressure zp\u016fsob\u00ed, \u017ee j\u00e1dro si uchov\u00e1 ke\u0161e dentry a inode. Kdy\u017e je hodnota nastavena na '0', j\u00e1dro nikdy nevyma\u017ee ke\u0161 dentry a inode z d\u016fvodu nedostatku pam\u011bti (memory pressure), co\u017e m\u016f\u017ee snadno v\u00e9st k chyb\u011b out-of-memory. Zv\u00fd\u0161en\u00ed vfs_cache_pressure nad 100 d\u00e1v\u00e1 j\u00e1dru prioritu k uvoln\u011bn\u00ed dentry a inode.<\/p>\n

P\u0159i pou\u017eit\u00ed GlusterFS mnoho u\u017eivatel\u016f s velk\u00fdmi objemy dat a mnoha mal\u00fdmi soubory m\u016f\u017ee snadno vyu\u017e\u00edt na serveru zna\u010dn\u00e9 mno\u017estv\u00ed opera\u010dn\u00ed pam\u011bti kv\u016fli ke\u0161ov\u00e1n\u00ed inode\/dentry, co\u017e m\u016f\u017ee v\u00e9st k poklesu v\u00fdkonu, proto\u017ee j\u00e1dro mus\u00ed zpracov\u00e1vat datov\u00e9 struktury v syst\u00e9mu s 40 GB pam\u011bti. Nastaven\u00ed t\u00e9to parameterov\u00e9 hodnoty nad 100 pomohlo mnoha u\u017eivatel\u016fm dos\u00e1hnout spravedliv\u011bj\u0161\u00edho ke\u0161ov\u00e1n\u00ed a zlep\u0161it odezvu j\u00e1dra.<\/p>\n

vm.dirty_background_ratio a vm.dirty_ratio<\/h4>\n

\nPrvn\u00ed parametr (vm.dirty_background_ratio<\/code>) ur\u010duje procento pam\u011bti se \u0161pinav\u00fdmi str\u00e1nkami, p\u0159i jeho\u017e dosa\u017een\u00ed je t\u0159eba zah\u00e1jit na pozad\u00ed vypr\u00e1zdn\u011bn\u00ed \u0161pinav\u00fdch str\u00e1nek na disk. Dokud tato procentn\u00ed hodnota nen\u00ed dosa\u017eena, str\u00e1nky se na disk nevypr\u00e1zdn\u00ed. A kdy\u017e za\u010d\u00edn\u00e1 vypr\u00e1zdn\u011bn\u00ed, prov\u00e1d\u00ed se na pozad\u00ed, ani\u017e by to p\u0159eru\u0161ovalo b\u011b\u017e\u00edc\u00ed procesy. <\/p>\n

Druh\u00fd parametr (vm.dirty_ratio<\/code>) ur\u010duje procento pam\u011bti, kter\u00e9 mohou zab\u00edrat<\/p>\n

Sn\u00ed\u017een\u00ed hodnot t\u011bchto parametr\u016f vede k \u010dast\u011bj\u0161\u00edmu z\u00e1pisu dat na disk a k jejich men\u0161\u00edmu uchov\u00e1v\u00e1n\u00ed v RAM. To m\u016f\u017ee pomoci syst\u00e9m\u016fm s velk\u00fdm mno\u017estv\u00edm pam\u011bti, v nich\u017e je b\u011b\u017en\u00e9 zapisovat na disk cache str\u00e1nek o velikosti 45\u201390 GB, co\u017e zp\u016fsobuje obrovsk\u00e9 doby \u010dek\u00e1n\u00ed pro frontendov\u00e9 aplikace a sni\u017euje celkovou odezvu a interaktivitu.<\/p>\n

\u201e1\u201c > \/proc\/sys\/vm\/pagecache<\/h4>\n

\nStr\u00e1nkov\u00e1 cache (page cache) je cache, ve kter\u00e9 jsou ulo\u017eena data soubor\u016f a spustiteln\u00fdch program\u016f, tedy str\u00e1nky s aktu\u00e1ln\u00edm obsahem soubor\u016f nebo blokov\u00fdch za\u0159\u00edzen\u00ed. Tato cache se pou\u017e\u00edv\u00e1 k redukci po\u010dtu \u010dten\u00ed z disku. Hodnota \u201e1\u201c znamen\u00e1, \u017ee pro cache se pou\u017e\u00edv\u00e1 1% RAM a \u017ee operac\u00ed \u010dten\u00ed z disku bude v\u00edce ne\u017e z RAM. Nen\u00ed nutn\u00e9 m\u011bnit tento parametr, ale pokud jste paranoidn\u00ed ohledn\u011b kontroly nad str\u00e1nkovou cache, m\u016f\u017eete jej pou\u017e\u00edt.<\/p>\n

\u201edeadline\u201c > \/sys\/block\/sdc\/queue\/scheduler<\/h4>\n

\nPl\u00e1nova\u010d vstupu a v\u00fdstupu (I\/O scheduler) je komponenta j\u00e1dra Linuxu, kter\u00e1 zpracov\u00e1v\u00e1 fronty \u010dten\u00ed a z\u00e1pisu. Teoreticky je pro chytr\u00fd RAID \u0159adi\u010d lep\u0161\u00ed pou\u017e\u00edt \u201enoop\u201c, proto\u017ee Linux nic nev\u00ed o fyzick\u00e9 geometrii disku, a proto je efektivn\u011bj\u0161\u00ed nechat \u0159adi\u010d, kter\u00fd dob\u0159e zn\u00e1 geometrii disku, zpracovat po\u017eadavek co nejrychleji. Ale zd\u00e1 se, \u017ee \u201edeadline\u201c zvy\u0161uje v\u00fdkon. V\u00edce o pl\u00e1nova\u010d\u00edch si m\u016f\u017eete p\u0159e\u010d\u00edst v dokumentaci ke zdrojov\u00e9mu k\u00f3du j\u00e1dra Linuxu: linux\/Documentation\/block\/*osched.txt<\/code>. A tak\u00e9 jsem pozoroval n\u00e1r\u016fst propustnosti \u010dten\u00ed b\u011bhem sm\u00ed\u0161en\u00fdch operac\u00ed (mnoho z\u00e1pisov\u00fdch operac\u00ed).<\/p>\n

\u201e256\u201c > \/sys\/block\/sdc\/queue\/nr_requests<\/h4>\n

\nPo\u010det I\/O po\u017eadavk\u016f v bufferu p\u0159ed t\u00edm, ne\u017e jsou p\u0159ed\u00e1ny pl\u00e1nova\u010di. Velikost vnit\u0159n\u00ed fronty n\u011bkter\u00fdch ovlada\u010d\u016f (queue_depth) je v\u011bt\u0161\u00ed ne\u017e nr_requests pl\u00e1nova\u010de I\/O, tak\u017ee pl\u00e1nova\u010d I\/O m\u00e1 m\u00e1lo \u0161anc\u00ed spr\u00e1vn\u011b priorizovat a prov\u00e9st slou\u010den\u00ed po\u017eadavk\u016f. Pro pl\u00e1nova\u010de deadline a CFQ je lep\u0161\u00ed, kdy\u017e je nr_requests dvakr\u00e1t v\u011bt\u0161\u00ed ne\u017e vnit\u0159n\u00ed fronta ovlada\u010de. Slou\u010den\u00ed a p\u0159e\u0159azen\u00ed po\u017eadavk\u016f pom\u00e1h\u00e1 pl\u00e1nova\u010di b\u00fdt citliv\u011bj\u0161\u00ed p\u0159i vysok\u00e9m zat\u00ed\u017een\u00ed.<\/p>\n

echo \"16\" > \/proc\/sys\/vm\/page-cluster<\/h4>\n

\nParametr page-cluster \u0159\u00edd\u00ed po\u010det str\u00e1nek, kter\u00e9 se zapisuj\u00ed do swapu najednou. V uveden\u00e9m p\u0159\u00edkladu je hodnota nastavena na \"16\" podle velikosti stripe (stripe size) RAIDu, kter\u00e1 je 64 KB. To ned\u00e1v\u00e1 smysl p\u0159i swappiness = 0, ale pokud jste nastavili swappiness na 10 nebo 20, pak pou\u017eit\u00ed t\u00e9to hodnoty v\u00e1m pom\u016f\u017ee, kdy\u017e velikost stripe RAIDu \u010din\u00ed 64 KB.<\/p>\n

blockdev \u2014setra 4096 \/dev\/<<\/code>devname><\/code> (-sdb, hdc nebo dev_mapper)<\/h4>\n

\nV\u00fdchoz\u00ed nastaven\u00ed blokov\u00fdch za\u0159\u00edzen\u00ed pro mnoho RAID ovlada\u010d\u016f \u010dasto vedou k hrozn\u00e9 v\u00fdkonnosti. P\u0159id\u00e1n\u00ed v\u00fd\u0161e uveden\u00e9 mo\u017enosti nastav\u00ed p\u0159edb\u011b\u017en\u00e9 \u010dten\u00ed na 4096 * 512-bajtov\u00fdch sektor\u016f. Nejm\u00e9n\u011b pro streamov\u00e9 operace se zvy\u0161uje rychlost, napl\u0148uj\u00edc\u00ed vestav\u011bnou pam\u011b\u0165 disku d\u00edky p\u0159edb\u011b\u017en\u00e9mu \u010dten\u00ed b\u011bhem obdob\u00ed, kter\u00e9 j\u00e1dro pou\u017e\u00edv\u00e1 pro p\u0159\u00edpravu I\/O. Do vyrovn\u00e1vac\u00ed pam\u011bti mohou b\u00fdt um\u00edst\u011bna data, kter\u00e1 budou po\u017eadov\u00e1na p\u0159i dal\u0161\u00edm \u010dten\u00ed. P\u0159\u00edli\u0161 velk\u00e9 p\u0159edb\u011b\u017en\u00e9 \u010dten\u00ed m\u016f\u017ee zni\u010dit n\u00e1hodn\u00e9 I\/O pro velk\u00e9 soubory, pokud vyu\u017e\u00edv\u00e1 potenci\u00e1ln\u011b u\u017eite\u010dn\u00fd \u010das disku nebo na\u010d\u00edt\u00e1 data mimo vyrovn\u00e1vac\u00ed pam\u011b\u0165.<\/p>\n

N\u00ed\u017ee jsou uvedena dal\u0161\u00ed doporu\u010den\u00ed na \u00farovni souborov\u00e9ho syst\u00e9mu. Ale dosud nebyla testov\u00e1na. Ujist\u011bte se, \u017ee v\u00e1\u0161 souborov\u00fd syst\u00e9m zn\u00e1 velikost stripe a po\u010det disk\u016f v poli. Nap\u0159\u00edklad, \u017ee se jedn\u00e1 o pole RAID5 se stripe o velikosti 64K sest\u00e1vaj\u00edc\u00ed ze \u0161esti disk\u016f (ve skute\u010dnosti p\u011bti, proto\u017ee jeden disk se pou\u017e\u00edv\u00e1 pro paritu). Tato doporu\u010den\u00ed jsou zalo\u017eena na teoretick\u00fdch p\u0159edpokladech a shrom\u00e1\u017ed\u011bna z r\u016fzn\u00fdch blog\u016f\/\u010dl\u00e1nk\u016f expert\u016f na RAID.<\/p>\n

-> ext4 fs, 5 disk\u016f, 64K stripe, jednotky ve 4K bloc\u00edch\nmkfs -text4 -E stride=$((64\/4))\n-> xfs, 5 disk\u016f, 64K stripe, jednotky v 512-bajtov\u00fdch sektorech\nmkfs -txfs -d sunit=$((64*2)) -d swidth=$((5*64*2))<\/code><\/pre>\n
\nPro velk\u00e9 soubory lze zv\u00e1\u017eit zv\u00fd\u0161en\u00ed v\u00fd\u0161e uveden\u00fdch velikost\u00ed stripe.<\/p>\n
POZOR!<\/b> V\u0161e, co bylo v\u00fd\u0161e pops\u00e1no, je pro n\u011bkter\u00e9 typy aplikac\u00ed velmi subjektivn\u00ed. Tento \u010dl\u00e1nek nezaru\u010duje \u017e\u00e1dn\u00e1 zlep\u0161en\u00ed bez p\u0159edchoz\u00edho testov\u00e1n\u00ed p\u0159\u00edslu\u0161n\u00fdch aplikac\u00ed ze strany u\u017eivatele. M\u011bl by b\u00fdt aplikov\u00e1n pouze v p\u0159\u00edpad\u011b zlep\u0161en\u00ed celkov\u00e9 odezvy syst\u00e9mu nebo pokud \u0159e\u0161\u00ed sou\u010dasn\u00e9 probl\u00e9my.<\/p>\n
Dal\u0161\u00ed materi\u00e1ly:<\/h4>\n
<\/p>\n