{"id":36999,"date":"2019-10-31T22:15:13","date_gmt":"2019-10-31T19:15:13","guid":{"rendered":"https:\/\/prohoster.info\/blog\/haki-pri-rabote-s-bolshim-chislom-melkih-fajlov\/"},"modified":"2019-10-31T22:15:13","modified_gmt":"2019-10-31T19:15:13","slug":"haki-pri-rabote-s-bolshim-chislom-melkih-fajlov","status":"publish","type":"post","link":"https:\/\/prohoster.info\/sq\/blog\/administrirovanie\/haki-pri-rabote-s-bolshim-chislom-melkih-fajlov","title":{"rendered":"Hacks p\u00ebr t\u00eb punuar me nj\u00eb num\u00ebr t\u00eb madh skedar\u00ebsh t\u00eb vegj\u00ebl","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

Ideja e artikullit lindi spontanisht nga diskutimi n\u00eb komentet e artikullit \u00abDisa fjal\u00eb rreth inode\u00bb<\/a><\/noindex>.<\/p>\n

\"Hacks
\n
\nE v\u00ebrteta \u00ebsht\u00eb se specifikimi i brendsh\u00ebm i funksionimit t\u00eb sh\u00ebrbimeve tona \u00ebsht\u00eb ruajtja e nj\u00eb numri t\u00eb madh skedar\u00ebsh t\u00eb vegj\u00ebl. Aktualisht kemi rreth qindra terabajt t\u00eb till\u00eb t\u00eb dh\u00ebnash. Dhe kemi hasur n\u00eb disa pengesa t\u00eb qarta dhe jo shum\u00eb t\u00eb qarta dhe i kemi kaluar ato me sukses.<\/p>\n

Prandaj po ndaj p\u00ebrvoj\u00ebn ton\u00eb, ndoshta do t'i sh\u00ebrbej\u00eb dikujt.
\n<\/a><\/noindex><\/p>\n

Problemi i par\u00eb: \u00abNuk ka hap\u00ebsir\u00eb t\u00eb mbetur n\u00eb pajisje\u00bb<\/h2>\n

\nSi\u00e7 u p\u00ebrmend n\u00eb artikullin e m\u00ebsip\u00ebrm, problemi \u00ebsht\u00eb se ka blloqe t\u00eb lira n\u00eb sistemin e dosjeve, por inode-\u00ebt jan\u00eb ezauruar.<\/p>\n

Numrin e inode-ve t\u00eb p\u00ebrdorura dhe t\u00eb lira mund ta kontrolloni me komand\u00ebn df -ih<\/code>:<\/p>\n

\"Hacks
\n
\nNuk do ta p\u00ebrs\u00ebris artikullin, n\u00ebse e shohim shkurt, n\u00eb disk ka blloqe t\u00eb drejta p\u00ebr t\u00eb dh\u00ebnat, si dhe blloqe p\u00ebr metainformacionin, t\u00eb njohura si inode (indeks i nyj\u00ebs). Numri i tyre p\u00ebrcaktohet gjat\u00eb inicializimit t\u00eb sistemit t\u00eb dosjeve (flasim p\u00ebr ext2 dhe pasardh\u00ebsit e saj) dhe m\u00eb tej nuk ndryshohet. Balancimi i blloqeve t\u00eb t\u00eb dh\u00ebnave dhe inode-ve llogaritet nga t\u00eb dh\u00ebnat mesatare, nd\u00ebrsa n\u00eb rastin ton\u00eb, ku ka shum\u00eb skedar\u00eb t\u00eb vegj\u00ebl, balancimi duhet t\u00eb zhvendoset n\u00eb an\u00ebn e numrit t\u00eb inode-ve \u2014 duhet t\u00eb jen\u00eb m\u00eb shum\u00eb.<\/p>\n

N\u00eb Linux gjithashtu jan\u00eb parashikuar variante me nj\u00eb balancim t\u00eb ndrysh\u00ebm, dhe t\u00eb gjitha k\u00ebto konfiguracione t\u00eb paracaktuara ndodhen n\u00eb skedarin \/etc\/mke2fs.conf<\/code>.
\nPrandaj, gjat\u00eb inicializimit t\u00eb par\u00eb t\u00eb sistemit t\u00eb dosjeve p\u00ebrmes mke2fs, mund t\u00eb specifikoni profilin e nevojsh\u00ebm.<\/p>\n

Ja disa shembuj nga skedari:<\/p>\n

    small = {\n        blocksize = 1024\n        inode_size = 128\n        inode_ratio = 4096\n    }\n\n    big = {\n        inode_ratio = 32768\n    }\n\n    largefile = {\n        inode_ratio = 1048576\n        blocksize = -1\n    }\n<\/code><\/pre>\n
\nMund t\u00eb zgjidhni variantin e duhur t\u00eb p\u00ebrdorimit me opsionin «-T» gjat\u00eb thirrjes s\u00eb mke2fs. Gjithashtu, mund t\u00eb caktoni manualisht parametrat e nevojsh\u00ebm n\u00ebse nuk ka nj\u00eb zgjidhje t\u00eb gatshme.<\/p>\n
M\u00eb shum\u00eb detaje jan\u00eb p\u00ebrshkruar n\u00eb manualet p\u00ebr mke2fs.conf<\/code> dhe mke2fs<\/code>.<\/p>\n
Nj\u00eb karakteristik\u00eb e pap\u00ebrmendur n\u00eb artikullin e m\u00ebsip\u00ebrm \u2014 mund t\u00eb specifikoni madh\u00ebsin\u00eb e bllokut t\u00eb t\u00eb dh\u00ebnave. Ose, p\u00ebr skedar\u00eb t\u00eb m\u00ebdhenj ka kuptim q\u00eb blloku t\u00eb jet\u00eb m\u00eb i madh, p\u00ebr t\u00eb vegjlit \u2014 m\u00eb i vog\u00ebl. <\/p>\n
Megjithat\u00eb, \u00ebsht\u00eb e r\u00ebnd\u00ebsishme t\u00eb merret parasysh nj\u00eb karakteristik\u00eb interesante, si\u00e7 \u00ebsht\u00eb arhitektura e procesorit.
\nN\u00eb nj\u00eb moment un\u00eb mendoja se m\u00eb duhej nj\u00eb madh\u00ebsi m\u00eb e madhe blloku p\u00ebr skedar\u00ebt e m\u00ebdhenj t\u00eb fotografive. Kjo ndodhi n\u00eb kushte sht\u00ebpiake, n\u00eb nj\u00eb ruajt\u00ebs skedar\u00ebsh WD mbi arkitektur\u00ebn ARM. Un\u00eb, pa menduar shum\u00eb, vendosa madh\u00ebsin\u00eb e bllokut 8k ose 16k n\u00eb vend t\u00eb standardit 4k, pas nj\u00eb matjeje t\u00eb kursimit. Dhe gjith\u00e7ka ishte shk\u00eblqyer deri n\u00eb momentin kur ruajt\u00ebsi d\u00ebshtoi, nd\u00ebrkoh\u00eb q\u00eb disku ishte i gjall\u00eb. Pasi e vendosa diskun n\u00eb nj\u00eb kompjuter t\u00eb zakonsh\u00ebm me nj\u00eb procesor Intel t\u00eb zakonsh\u00ebm, mora nj\u00eb surpriz\u00eb: madh\u00ebsi blloku e pap\u00ebrshtatshme. E shkuara. T\u00eb dh\u00ebnat jan\u00eb aty, gjith\u00e7ka \u00ebsht\u00eb mir\u00eb, por nuk \u00ebsht\u00eb e mundur t\u00eb lexohen. Procesor\u00ebt i386 dhe t\u00eb ngjashmit nuk din\u00eb t\u00eb punojn\u00eb me madh\u00ebsi blloku q\u00eb nuk p\u00ebrputhen me madh\u00ebsin\u00eb e faqeve t\u00eb memories, e cila \u00ebsht\u00eb sakt\u00ebsisht 4k. N\u00eb p\u00ebrgjith\u00ebsi, kjo p\u00ebrfundoi me p\u00ebrdorimin e utiliteteve nga hap\u00ebsira e p\u00ebrdoruesit, gjith\u00e7ka ishte e ngadalshme dhe e trishtueshme, por t\u00eb dh\u00ebnat u shp\u00ebtuan. P\u00ebr ata q\u00eb jan\u00eb t\u00eb interesuar \u2014 k\u00ebrkoni me emrin e utilitetit. fuseext2<\/code>. M\u00ebsimi: ose t\u00eb mendosh p\u00ebr t\u00eb gjitha rastet paraprakisht, ose t\u00eb mos b\u00ebsh si nj\u00eb superheroi dhe t\u00eb p\u00ebrdor\u00ebsh cil\u00ebsime standarde p\u00ebr amvisat.<\/p>\n
UPD. Sipas v\u00ebrejtjes s\u00eb p\u00ebrdoruesit berez<\/a><\/noindex> duke sqaruar se p\u00ebr i386 madh\u00ebsia e bllokut nuk duhet t\u00eb tejkaloj\u00eb 4k, por nuk \u00ebsht\u00eb e nevojshme t\u00eb jet\u00eb sakt\u00ebsisht 4k, pra jan\u00eb t\u00eb pranueshme 1k dhe 2k.<\/p>\n
Pra, si i zgjidh\u00ebm problemet ne.<\/p>\n
S\u00eb pari, ne u p\u00ebrball\u00ebm me problemin kur disku me shum\u00eb terabajt ishte i mbushur me t\u00eb dh\u00ebna dhe ne nuk mund t\u00eb riparonim konfigurimin e sistemit t\u00eb skedar\u00ebve.<\/p>\n
S\u00eb dyti, zgjidhja k\u00ebrkohej menj\u00ebher\u00eb.<\/p>\n
N\u00eb fund arrit\u00ebm n\u00eb p\u00ebrfundimin se duhej t\u00eb ndryshonim balanc\u00ebn, duke reduktuar numrin e skedar\u00ebve.
\nP\u00ebr t\u00eb reduktuar numrin e skedar\u00ebve, u vendos t\u00eb vendosim skedar\u00ebt n\u00eb nj\u00eb arkiv t\u00eb p\u00ebrbashk\u00ebt. Duke marr\u00eb parasysh specifik\u00ebn ton\u00eb, ne vendos\u00ebm n\u00eb nj\u00eb arkiv t\u00eb gjith\u00eb skedar\u00ebt p\u00ebr nj\u00eb periudh\u00eb t\u00eb caktuar kohore dhe e b\u00ebm\u00eb arkivimin me nj\u00eb task cron \u00e7do nat\u00eb.<\/p>\n
Zgjodh\u00ebm nj\u00eb arkiv zip. N\u00eb komentet e artikullit t\u00eb m\u00ebparsh\u00ebm u propozuan tar, por me t\u00eb ka nj\u00eb kompleksitet: nuk ka indeks, dhe skedar\u00ebt n\u00eb t\u00eb jan\u00eb t\u00eb radhitur n\u00eb m\u00ebnyr\u00eb kaotike (nuk \u00ebsht\u00eb rast\u00ebsi q\u00eb \"tar\" \u00ebsht\u00eb shkurtim p\u00ebr \"Tape Archive\", trash\u00ebgimi e paisjeve me kaset\u00eb), dmth. n\u00ebse duam t\u00eb lexojm\u00eb nj\u00eb skedar n\u00eb fund t\u00eb arkivit \u2014 duhet t\u00eb lexojm\u00eb t\u00eb gjith\u00eb arkivin, pasi nuk ka offset p\u00ebr secilin skedar n\u00eb raport me fillimin e arkivit. Pra, kjo \u00ebsht\u00eb nj\u00eb operacion i gjat\u00eb. N\u00eb zip, gj\u00ebrat jan\u00eb shum\u00eb m\u00eb mir\u00eb: ai ka at\u00eb indeksin dhe offset skedar\u00ebve brenda arkivit, dhe koha e qasjes n\u00eb \u00e7do skedar nuk varet nga pozita e tij. Gjithashtu, n\u00eb rastin ton\u00eb mund t\u00eb ishte caktuar opsioni i kompresimit \"0\", pasi t\u00eb gjith\u00eb skedar\u00ebt ishin tashm\u00eb kompresuar n\u00eb gzip.<\/p>\n
Klient\u00ebt marrin skedar\u00ebt p\u00ebrmes nginx, dhe sipas API t\u00eb vjet\u00ebr, thjesht jepet emri i skedarit, p\u00ebr shembull k\u00ebshtu:<\/p>\n
http:\/\/www.server.com\/hydra\/20170416\/0453\/3bd24ae7-1df4-4d76-9d28-5b7fcb7fd8e5\n<\/code><\/pre>\n
\nP\u00ebr t\u00eb shpaketuar skedar\u00ebt n\u00eb fluks, gjet\u00ebm dhe lidh\u00ebm modulin nginx-unzip-module (https:\/\/github.com\/youzee\/nginx-unzip-module<\/a><\/noindex>) dhe konfiguruam dy upstream.<\/p>\n
Si rezultat, u krijua nj\u00eb konfigurim i till\u00eb:<\/p>\n
\"Hacks
\n
\nDy host-e n\u00eb konfigurim dukeshin k\u00ebshtu:<\/p>\n
server {\n  listen *:8081;\n\n  location \/ {\n    root \/home\/filestorage;\n  }\n}<\/code><\/pre>\n
<\/p>\n
server {\n  listen *:8082;\n\n  location ~ ^\/hydra\/(d+)\/(\n  root \/home\/filestorage;\n    file_in_unzip_archivefile \"\/home\/filestorage\/hydra\/$1\/$2.zip\";\n    file_in_unzip_extract \"$2\/$3\";\n    file_in_unzip;\n  }\n}\n<\/code><\/pre>\n
\nDhe konfigurimi i upstream-eve n\u00eb nginx e lart\u00eb:<\/p>\n
upstream storage {\n  server server.com:8081;\n  server server.com:8082;\n}\n<\/code><\/pre>\n
\nSi funksionon:<\/p>\n