{"id":55408,"date":"2020-01-20T00:00:00","date_gmt":"2020-01-19T21:00:00","guid":{"rendered":"https:\/\/prohoster.info\/blog\/blog_prohoster\/effektivnoe-hranenie-soten-millionov-malenkih-fajlov-self-hosted-reshenie"},"modified":"2020-02-18T14:03:31","modified_gmt":"2020-02-18T11:03:31","slug":"effektivnoe-hranenie-soten-millionov-malenkih-fajlov-self-hosted-reshenie","status":"publish","type":"post","link":"https:\/\/prohoster.info\/sq\/blog\/administrirovanie\/effektivnoe-hranenie-soten-millionov-malenkih-fajlov-self-hosted-reshenie","title":{"rendered":"Ruajtja efikase e qindra milion\u00eb skedar\u00ebve t\u00eb vegj\u00ebl. Zgjidhja e vet\u00eb-hostuar","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

\"Ruajtja
\n
\nE dashur komunitet, ky artikull do t\u00eb fokusohet n\u00eb ruajtjen dhe shp\u00ebrndarjen efektive t\u00eb qindra milion\u00eb skedar\u00ebve t\u00eb vegj\u00ebl. N\u00eb k\u00ebt\u00eb faz\u00eb, po propozohet nj\u00eb zgjidhje p\u00ebrfundimtare p\u00ebr sistemet e skedar\u00ebve q\u00eb jan\u00eb t\u00eb pajtueshme me POSIX, me mb\u00ebshtetje t\u00eb plot\u00eb p\u00ebr bllokimet, p\u00ebrfshir\u00eb ato klasterike, dhe duket se madje edhe pa ndihm\u00ebs p\u00ebr zgjidhje.<\/p>\n

Prandaj, p\u00ebr k\u00ebt\u00eb q\u00ebllim, kam shkruar serverin tim t\u00eb specializuar.
\nN\u00eb procesin e zbatimit t\u00eb k\u00ebsaj detyre arrita t\u00eb zgjidh problemin kryesor, duke arrijtur gjithashtu nj\u00eb kursim t\u00eb hap\u00ebsir\u00ebs n\u00eb disk dhe memories operative, t\u00eb cilat sistemet tona klasterike t\u00eb skedar\u00ebve i konsumonin pa m\u00ebshir\u00eb. N\u00eb fund t\u00eb fundit, kaq shum\u00eb skedar\u00eb \u00ebsht\u00eb e d\u00ebmshme p\u00ebr \u00e7do sistem klaster t\u00eb skedar\u00ebve. <\/a><\/noindex><\/p>\n

Ideja \u00ebsht\u00eb k\u00ebshtu:<\/p>\n

Me fjal\u00eb t\u00eb thjeshta, skedar\u00ebt e vegj\u00ebl ngarkohen p\u00ebrmes serverit, ata ruhen drejtp\u00ebrdrejt n\u00eb arkiv dhe gjithashtu lexohen prej tij, nd\u00ebrsa skedar\u00ebt e m\u00ebdhenj vendosen af\u00ebr. Schemat: 1 dosje = 1 arkiv, k\u00ebshtu kemi disa milion\u00eb arkiva me skedar\u00eb t\u00eb vegj\u00ebl, e jo disa qindra milion\u00eb skedar\u00eb. Dhe gjith\u00e7ka \u00ebsht\u00eb realizuar n\u00eb m\u00ebnyr\u00eb t\u00eb plot\u00eb, pa ndonj\u00eb skript dhe pa shp\u00ebrndarjen e skedar\u00ebve n\u00eb arkiva tar\/zip.<\/p>\n

Do t\u00eb p\u00ebrpiqem t\u00eb shpjegoj shkurtimisht, paraprakisht k\u00ebrkoj falje n\u00ebse posta do t\u00eb jet\u00eb e gjat\u00eb.<\/p>\n

E gjith\u00eb filloi kur nuk mund t\u00eb gjeja nj\u00eb server t\u00eb p\u00ebrshtatsh\u00ebm n\u00eb bot\u00eb q\u00eb mund t\u00eb ruante t\u00eb dh\u00ebnat, t\u00eb marra p\u00ebrmes protokollit HTTP, drejtp\u00ebrdrejt n\u00eb arkiva, n\u00eb m\u00ebnyr\u00eb q\u00eb t\u00eb mos kishte disavantazhe tipike t\u00eb arkivave t\u00eb zakonsh\u00ebm dhe ruajtjeve objektive. Arsyeja e k\u00ebrkimeve ishte nj\u00eb klaster Origin i rritur n\u00eb p\u00ebrmasa t\u00eb m\u00ebdha me 10 server\u00eb, n\u00eb t\u00eb cilin kishim grumbulluar tashm\u00eb 250,000,000 skedar\u00eb t\u00eb vegj\u00ebl dhe tendenca e rritjes nuk kishte nd\u00ebrmend t\u00eb ndalonte.<\/p>\n

P\u00ebr ata q\u00eb nuk e duan t\u00eb lexojn\u00eb artikuj dhe preferojn\u00eb dokumentacionin e vog\u00ebl m\u00eb t\u00eb thjesht\u00eb:<\/b><\/p>\n

k\u00ebtu<\/a><\/noindex> dhe k\u00ebtu<\/a><\/noindex>.<\/p>\n

Dhe docker gjithashtu, tani ka nj\u00eb opsion vet\u00ebm me nginx brenda p\u00ebr \u00e7do rast:<\/p>\n

docker run -d --restart=always -e host=localhost -e root=\\\/var\\\/storage \\n-v \\\/var\\\/storage:\\\/var\\\/storage --name wzd -p 80:80 eltaline\\\/wzd<\/code><\/pre>\n
\nPastaj:<\/p>\n
N\u00ebse ka shum\u00eb skedar\u00eb, nevojiten burime t\u00eb konsiderueshme, dhe, m\u00eb e keqja, nj\u00eb pjes\u00eb e tyre humbet n\u00eb m\u00ebnyr\u00eb t\u00eb panevojshme. P\u00ebr shembull, kur p\u00ebrdoret nj\u00eb sistem skedar\u00ebsh n\u00eb klaster (n\u00eb k\u00ebt\u00eb rast - MooseFS), skedari, pavar\u00ebsisht nga madh\u00ebsia reale, gjithmon\u00eb z\u00eb t\u00eb pakt\u00ebn 64 KB. K\u00ebshtu q\u00eb p\u00ebr skedar\u00ebt me madh\u00ebsi 3, 10 ose 30 KB nevojiten 64 KB n\u00eb disk. N\u00ebse ka nj\u00eb \u00e7erek miliard skedar\u00eb, humbim nga 2 deri n\u00eb 10 terabajt. Nuk \u00ebsht\u00eb e mundur t\u00eb krijosh skedar\u00eb t\u00eb rinj deri n\u00eb pafund\u00ebsi, sepse n\u00eb MooseFS ka nj\u00eb kufizim: jo m\u00eb shum\u00eb se 1 miliard me nj\u00eb replik\u00eb p\u00ebr \u00e7do skedar.<\/p>\n
Me rritjen e numrit t\u00eb skedar\u00ebve, nevojitet shum\u00eb memorie p\u00ebr metadat\u00eb. Gjithashtu, dump-et e shpeshta t\u00eb m\u00ebdha t\u00eb metadatave kontribuojn\u00eb n\u00eb konsumimin e SSD-ve.<\/p>\n
Serveri wZD. Rregullojm\u00eb rendin n\u00eb disqet.<\/b><\/p>\n
Serveri \u00ebsht\u00eb shkruar n\u00eb gjuh\u00ebn Go. Para s\u00eb gjithash, m\u00eb duhej t\u00eb reduktonja numrin e skedar\u00ebve. Si ta b\u00ebj k\u00ebt\u00eb? P\u00ebrmes arkivimit, por n\u00eb k\u00ebt\u00eb rast pa kompresim, sepse skedar\u00ebt e mi jan\u00eb imazhe t\u00eb ngjeshur. N\u00eb ndihm\u00eb erdhi BoltDB, t\u00eb cil\u00ebn gjithashtu duhej ta hiqja nga mang\u00ebsit\u00eb e saj, si\u00e7 \u00ebsht\u00eb reflektuar n\u00eb dokumentacion.<\/p>\n
P\u00ebrfundimisht, n\u00eb vend t\u00eb \u00e7erek miliard skedar\u00ebve, n\u00eb rastin tim mbet\u00ebn vet\u00ebm 10 milion arkivash Bolt. Po t\u00eb kishim mund\u00ebsin\u00eb t\u00eb ndryshonim struktur\u00ebn aktuale t\u00eb mbushjes s\u00eb skedar\u00ebve n\u00eb direktori, ndoshta do t\u00eb ishte e mundur ta ulja at\u00eb edhe deri n\u00eb 1 milion skedar\u00eb. <\/p>\n
T\u00eb gjith\u00eb skedar\u00ebt e vegj\u00ebl paketohet n\u00eb arkivat Bolt, duke marr\u00eb automatikisht emrat e direktorive n\u00eb t\u00eb cilat ndodhen, nd\u00ebrsa t\u00eb gjith\u00eb skedar\u00ebt e m\u00ebdhenj mbeten af\u00ebr arkivave; nuk ka kuptim t'i paketojm\u00eb, kjo \u00ebsht\u00eb e konfigurable. T\u00eb vegjlit - arkivohen, t\u00eb m\u00ebdhenjt - mbeten pa ndryshim. Serveri punon n\u00eb m\u00ebnyr\u00eb t\u00eb transparent\u00eb si me ato ashtu edhe me k\u00ebta.<\/p>\n
Arkitektura dhe ve\u00e7orit\u00eb e serverit wZD.<\/b><\/p>\n
\"Ruajtja
\n
\nServeri funksionon n\u00ebn sistemet operative Linux, BSD, Solaris dhe OSX. E kam testuar vet\u00ebm p\u00ebr arkitektur\u00ebn AMD64 n\u00ebn Linux, por duhet t\u00eb jet\u00eb e p\u00ebrshtatshme edhe p\u00ebr ARM64, PPC64, MIPS64.<\/p>\n
Ve\u00e7orit\u00eb kryesore:<\/b><\/p>\n