{"id":55408,"date":"2020-01-20T00:00:00","date_gmt":"2020-01-19T21:00:00","guid":{"rendered":"https:\/\/prohoster.info\/blog\/blog_prohoster\/effektivnoe-hranenie-soten-millionov-malenkih-fajlov-self-hosted-reshenie"},"modified":"2020-02-18T14:03:31","modified_gmt":"2020-02-18T11:03:31","slug":"effektivnoe-hranenie-soten-millionov-malenkih-fajlov-self-hosted-reshenie","status":"publish","type":"post","link":"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/effektivnoe-hranenie-soten-millionov-malenkih-fajlov-self-hosted-reshenie","title":{"rendered":"Efekt\u00edvne ukladanie stoviek mili\u00f3nov mal\u00fdch s\u00faborov. Self-Hosted rie\u0161enie","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

\"Efekt\u00edvne
\n
\nV\u00e1\u017een\u00e1 komunita, tento \u010dl\u00e1nok bude venovan\u00fd efekt\u00edvnemu skladovaniu a spr\u00e1ve stoviek mili\u00f3nov mal\u00fdch s\u00faborov. V tejto f\u00e1ze sa navrhuje kone\u010dn\u00e9 rie\u0161enie pre s\u00faborov\u00e9 syst\u00e9my kompatibiln\u00e9 s POSIX s plnou podporou zamykania, vr\u00e1tane klastra, a zd\u00e1 sa, \u017ee u\u017e aj bez obmedzen\u00ed.<\/p>\n

Preto som pre tento \u00fa\u010del nap\u00edsal vlastn\u00fd \u0161pecializovan\u00fd server.
\nPo\u010das realiz\u00e1cie tejto \u00falohy sa podarilo vyrie\u0161i\u0165 hlavn\u00fa problematiku, z\u00e1rove\u0148 dosiahnu\u0165 \u00fasporu diskov\u00e9ho priestoru a opera\u010dnej pam\u00e4te, ktor\u00fa nezadr\u017eate\u013ene spotrebov\u00e1val n\u00e1\u0161 klastrov\u00fd s\u00faborov\u00fd syst\u00e9m. Vlastne tak\u00e9 mno\u017estvo s\u00faborov je \u0161kodliv\u00e9 pre ak\u00fdko\u013evek klastrov\u00fd s\u00faborov\u00fd syst\u00e9m. <\/a><\/noindex><\/p>\n

My\u0161lienka je tak\u00e1:<\/p>\n

Jednoduch\u00fdmi slovami, cez server sa nahr\u00e1vaj\u00fa mal\u00e9 s\u00fabory, ktor\u00e9 sa ukladaj\u00fa priamo do arch\u00edvu a rovnako sa z neho na\u010d\u00edtavaj\u00fa, zatia\u013e \u010do ve\u013ek\u00e9 s\u00fabory sa ukladaj\u00fa ved\u013ea. Sch\u00e9ma: 1 prie\u010dinok = 1 arch\u00edv, tak\u017ee m\u00e1me nieko\u013eko mili\u00f3nov arch\u00edvov s mal\u00fdmi s\u00fabormi, nie nieko\u013eko stoviek mili\u00f3nov s\u00faborov. A to v\u0161etko je plne implementovan\u00e9, bez ak\u00fdchko\u013evek skriptov a rozkladania s\u00faborov do tar\/zip arch\u00edvov.<\/p>\n

Pok\u00fasim sa vysvetli\u0165 stru\u010dnej\u0161ie, vopred sa ospravedl\u0148ujem, ak bude pr\u00edspevok rozsiahly.<\/p>\n

V\u0161etko sa za\u010dalo t\u00fdm, \u017ee som nedok\u00e1zal n\u00e1js\u0165 vhodn\u00fd server na svete, ktor\u00fd by mohol uklada\u0165 d\u00e1ta prijat\u00e9 cez protokol HTTP priamo do arch\u00edvov, aby sa predi\u0161lo nedostatkom typick\u00fdm pre be\u017en\u00e9 arch\u00edvy a objektov\u00e9 \u00falo\u017eisk\u00e1. D\u00f4vodom mojich h\u013eadania bol roz\u0161\u00edren\u00fd klaster Origin s desiatimi servermi, v ktorom sa u\u017e nahromadilo 250 000 000 mal\u00fdch s\u00faborov a tendencia rastu sa nemala zastavi\u0165.<\/p>\n

T\u00fdm, ktor\u00ed nemaj\u00fa radi \u010d\u00edtanie \u010dl\u00e1nkov a trocha dokument\u00e1cie je jednoduch\u0161ia:<\/b><\/p>\n

sem<\/a><\/noindex> a sem<\/a><\/noindex>.<\/p>\n

A docker z\u00e1rove\u0148, teraz je k dispoz\u00edcii iba s nginxom vo vn\u00fatri pre ka\u017ed\u00fd pr\u00edpad:<\/p>\n

docker run -d --restart=always -e host=localhost -e root=\\\/var\\\/storage \n-v \\\/var\\\/storage:\\\/var\\\/storage --name wzd -p 80:80 eltaline\\\/wzd<\/code><\/pre>\n
\n\u010ealej:<\/p>\n
Ak je s\u00faborov ve\u013emi ve\u013ea, potrebn\u00e9 s\u00fa zna\u010dn\u00e9 zdroje; \u010do je v\u0161ak najhor\u0161ie, \u010das\u0165 ich sa str\u00e1ca nadarmo. Napr\u00edklad pri pou\u017eit\u00ed klastrov\u00e9ho s\u00faborov\u00e9ho syst\u00e9mu (v tomto pr\u00edpade \u2013 MooseFS) s\u00fabor, bez oh\u013eadu na jeho skuto\u010dn\u00fa ve\u013ekos\u0165, v\u017edy zaber\u00e1 minim\u00e1lne 64 KB. To znamen\u00e1, \u017ee pre s\u00fabory ve\u013ekosti 3, 10 alebo 30 KB potrebujeme na disku 64 KB. Ak je s\u00faborov \u0161tvr\u0165 miliarda, prich\u00e1dzame o 2 a\u017e 10 terabajtov. S vytv\u00e1ran\u00edm nov\u00fdch s\u00faborov do nekone\u010dna to nebude mo\u017en\u00e9, preto\u017ee v tej istej MooseFS existuje obmedzenie: nie viac ako 1 miliarda s jednou replikou ka\u017ed\u00e9ho s\u00faboru.<\/p>\n
S rast\u00facim po\u010dtom s\u00faborov je potrebn\u00e9 ve\u013ea opera\u010dnej pam\u00e4te pre metad\u00e1ta. Rovnako \u010dast\u00e9 ve\u013ek\u00e9 dumpy metad\u00e1t prispievaj\u00fa k opotrebovaniu SSD \u00falo\u017e\u00edsk.<\/p>\n
Server wZD. Urob\u00edme poriadok na diskoch.<\/b><\/p>\n
Server je naprogramovan\u00fd v jazyku Go. Predov\u0161etk\u00fdm som potreboval zredukova\u0165 po\u010det s\u00faborov. Ako to spravi\u0165? Arch\u00edvovan\u00edm, ale v tomto pr\u00edpade bez kompresie, preto\u017ee m\u00e1m s\u00fabory, ktor\u00e9 s\u00fa vylo\u017eene zhu\u0161\u0165ovan\u00e9 obr\u00e1zky. Na pomoc mi pri\u0161la BoltDB, ktor\u00fa som musel e\u0161te zbavi\u0165 nedostatkov, \u010do je odr\u00e1\u017ean\u00e9 v dokument\u00e1cii.<\/p>\n
Tak namiesto \u0161tvr\u0165 miliardy s\u00faborov zostalo len 10 mili\u00f3nov Bolt arch\u00edvov. Keby som mal mo\u017enos\u0165 zmeni\u0165 aktu\u00e1lnu \u0161trukt\u00faru nap\u013a\u0148ania adres\u00e1rov s\u00fabormi, bolo by mo\u017eno mo\u017en\u00e9 zredukova\u0165 to na pribli\u017ene 1 mili\u00f3n s\u00faborov. <\/p>\n
V\u0161etky mal\u00e9 s\u00fabory s\u00fa zabalen\u00e9 do Bolt arch\u00edvov, ktor\u00e9 automaticky dost\u00e1vaj\u00fa n\u00e1zvy adres\u00e1rov, v ktor\u00fdch sa nach\u00e1dzaj\u00fa, a v\u0161etky ve\u013ek\u00e9 s\u00fabory ost\u00e1vaj\u00fa ved\u013ea arch\u00edvov, nem\u00e1 zmysel ich balenie, je to nastavite\u013en\u00e9. Mal\u00e9 \u2013 arch\u00edvujeme, ve\u013ek\u00e9 \u2013 ponech\u00e1vame bezo zmien. Server funguje transparentne s oboma typmi.<\/p>\n
Architekt\u00fara a vlastnosti servera wZD.<\/b><\/p>\n
\"Efekt\u00edvne
\n
\nServer funguje pod opera\u010dn\u00fdmi syst\u00e9mami Linux, BSD, Solaris a OSX. Testoval som ho len pre architekt\u00faru AMD64 pod Linuxom, ale mal by by\u0165 vhodn\u00fd aj pre ARM64, PPC64, MIPS64.<\/p>\n
Hlavn\u00e9 funkcie:<\/b><\/p>\n