{"id":36999,"date":"2019-10-31T22:15:13","date_gmt":"2019-10-31T19:15:13","guid":{"rendered":"https:\/\/prohoster.info\/blog\/haki-pri-rabote-s-bolshim-chislom-melkih-fajlov\/"},"modified":"2019-10-31T22:15:13","modified_gmt":"2019-10-31T19:15:13","slug":"haki-pri-rabote-s-bolshim-chislom-melkih-fajlov","status":"publish","type":"post","link":"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/haki-pri-rabote-s-bolshim-chislom-melkih-fajlov","title":{"rendered":"Triky pri pr\u00e1ci s ve\u013ek\u00fdm po\u010dtom mal\u00fdch s\u00faborov","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p>N\u00e1pad na tento \u010dl\u00e1nok sa zrodil spont\u00e1nne z diskusie v koment\u00e1roch k \u010dl\u00e1nku <noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/post\/462849\/\">\"Nie\u010do o inode\"<\/a><\/noindex>.<\/p>\n<p><img decoding=\"async\" alt=\"Triky pri pr\u00e1ci s ve\u013ek\u00fdm po\u010dtom mal\u00fdch s\u00faborov\" src=\"\/wp-content\/uploads\/2019\/08\/e488f5985cfb0b3274f57f6baeeedf01.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nIde o to, \u017ee vn\u00fatornou \u0161pecifikou fungovania na\u0161ich slu\u017eieb je uchov\u00e1vanie obrovsk\u00e9ho mno\u017estva drobn\u00fdch s\u00faborov. V s\u00fa\u010dasnosti m\u00e1me zhruba stovky terabajtov t\u00fdchto \u00fadajov. A narazili sme na niektor\u00e9 zrejm\u00e9 aj menej zrejm\u00e9 prek\u00e1\u017eky, ktor\u00fdmi sme \u00faspe\u0161ne pre\u0161li.<\/p>\n<p>Preto sa del\u00edm o na\u0161u sk\u00fasenos\u0165, mo\u017eno to niekomu pom\u00f4\u017ee.<br \/>\n<noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><\/p>\n<h2>Probl\u00e9m prv\u00fd: \"Na zariaden\u00ed nie je miesto\"<\/h2>\n<p>\nAko bolo spomenut\u00e9 v predch\u00e1dzaj\u00facom \u010dl\u00e1nku, probl\u00e9m spo\u010d\u00edva v tom, \u017ee vofilesysteme s\u00fa vo\u013en\u00e9 bloky, ale inody sa minuli.<\/p>\n<p>Skontrolova\u0165 po\u010det pou\u017eit\u00fdch a vo\u013en\u00fdch inode je mo\u017en\u00e9 pr\u00edkazom <code>df -ih<\/code>:<\/p>\n<p><img decoding=\"async\" alt=\"Triky pri pr\u00e1ci s ve\u013ek\u00fdm po\u010dtom mal\u00fdch s\u00faborov\" src=\"\/wp-content\/uploads\/2019\/08\/ceb86a22562aa08c8ffbbde2c2618545.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nNebudem rekapitulova\u0165 \u010dl\u00e1nok, ak to zhrniem, na disku s\u00fa bloky priamo pre d\u00e1ta a bloky pre meta-inform\u00e1cie, ktor\u00e9 s\u00fa inody (indexov\u00e9 uzly). Ich po\u010det sa ur\u010duje pri inicializ\u00e1cii s\u00faborov\u00e9ho syst\u00e9mu (re\u010d je o ext2 a jeho n\u00e1stupcoch) a \u010falej sa nemen\u00ed. Vyv\u00e1\u017eenie blokov d\u00e1t a inode je vypo\u010d\u00edtan\u00e9 z priemern\u00fdch \u00fadajov, v na\u0161om pr\u00edpade, ke\u010f je ve\u013ea drobn\u00fdch s\u00faborov, by sa mala vyv\u00e1\u017eenie posun\u00fa\u0165 na stranu inodes \u2014 malo by ich by\u0165 viac.<\/p>\n<p>V Linuxe u\u017e s\u00fa predp\u00edsan\u00e9 mo\u017enosti s r\u00f4znymi vyv\u00e1\u017een\u00edm a v\u0161etky tieto predpo\u010d\u00edtan\u00e9 konfigur\u00e1cie sa nach\u00e1dzaj\u00fa v s\u00fabore <code>\/etc\/mke2fs.conf<\/code>.<br \/>\nPreto pri po\u010diato\u010dnej inicializ\u00e1cii s\u00faborov\u00e9ho syst\u00e9mu cez mke2fs m\u00f4\u017eete zada\u0165 potrebn\u00fd profil.<\/p>\n<p>Tu je nieko\u013eko pr\u00edkladov zo s\u00faboru:<\/p>\n<pre><code class=\"json\">    small = {\n        blocksize = 1024\n        inode_size = 128\n        inode_ratio = 4096\n    }\n\n    big = {\n        inode_ratio = 32768\n    }\n\n    largefile = {\n        inode_ratio = 1048576\n        blocksize = -1\n    }\n<\/code><\/pre>\n<p>\nSpr\u00e1vnu mo\u017enos\u0165 pou\u017eitia je mo\u017en\u00e9 vybra\u0165 pomocou vo\u013eby &#171;-T&#187; pri volan\u00ed mke2fs. M\u00f4\u017eete tie\u017e ru\u010dne zada\u0165 potrebn\u00e9 parametre, ak neexistuje hotov\u00e9 rie\u0161enie.<\/p>\n<p>Viac podrobnost\u00ed je uveden\u00fdch v manu\u00e1loch pre <code>mke2fs.conf<\/code> a <code>mke2fs<\/code>.<\/p>\n<p>V \u010dl\u00e1nku vy\u0161\u0161ie sa nespomenula \u0161peci\u00e1lna vlastnos\u0165 \u2014 je mo\u017en\u00e9 nastavi\u0165 ve\u013ekos\u0165 bloku d\u00e1t. Je jasn\u00e9, \u017ee pre ve\u013ek\u00e9 s\u00fabory m\u00e1 zmysel v\u00e4\u010d\u0161ia ve\u013ekos\u0165 bloku, pre mal\u00e9 \u2014 men\u0161ia. <\/p>\n<p>Av\u0161ak je potrebn\u00e9 bra\u0165 do \u00favahy tak\u00fa zauj\u00edmav\u00fa vlastnos\u0165, ako je architekt\u00fara procesora.<br \/>\nRazm\u00fd\u0161\u013eal som, \u017ee pre ve\u013ek\u00e9 s\u00fabory s fotografiami potrebujem v\u00e4\u010d\u0161iu ve\u013ekos\u0165 bloku. Bolo to v dom\u00e1cich podmienkach, na dom\u00e1com WD s\u00faborovom syst\u00e9me na architekt\u00fare ARM. Bez v\u00e1hania som nastavil ve\u013ekos\u0165 bloku bu\u010f 8k, alebo 16k namiesto \u0161tandardn\u00fdch 4k, pri\u010dom som najprv zmeral \u00fasporu. A v\u0161etko bolo skvel\u00e9 a\u017e do okamihu, ke\u010f sa samotn\u00fd syst\u00e9m prestal da\u0165 pou\u017ei\u0165, pri\u010dom disk bol funk\u010dn\u00fd. Ke\u010f som disk vlo\u017eil do be\u017en\u00e9ho po\u010d\u00edta\u010da s oby\u010dajn\u00fdm Intel procesorom, dostal som prekvapenie: nepodporovan\u00e1 ve\u013ekos\u0165 bloku. Na to sme narazili. D\u00e1ta s\u00fa, v\u0161etko je v poriadku, ale nie je mo\u017en\u00e9 ich pre\u010d\u00edta\u0165. Procesory i386 a podobn\u00e9 nevedia pracova\u0165 s ve\u013ekos\u0165ami bloku, ktor\u00e9 nie s\u00fa v s\u00falade s ve\u013ekos\u0165ou str\u00e1nok pam\u00e4te, a t\u00e1 je presne 4k. Tak\u017ee sme skon\u010dili pou\u017e\u00edvan\u00edm util\u00edt z u\u017e\u00edvate\u013esk\u00e9ho priestoru, v\u0161etko bolo pomal\u00e9 a smutn\u00e9, ale d\u00e1ta sme zachr\u00e1nili. Kto m\u00e1 z\u00e1ujem, vyh\u013eadajte n\u00e1zov utility. <code>fuseext2<\/code>. Mor\u00e1lka: bu\u010f premyslie\u0165 v\u0161etky mo\u017enosti vopred, alebo si nehra\u0165 na superhrdinu a pou\u017e\u00edva\u0165 \u0161tandardn\u00e9 nastavenia pre dom\u00e1cnosti.<\/p>\n<p>UPD. Pod\u013ea pozorovania pou\u017e\u00edvate\u013ea <noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/users\/berez\/\" class=\"user_link\">berez<\/a><\/noindex> udr\u017eiavam, \u017ee pre i386 ve\u013ekos\u0165 bloku nesmie presiahnu\u0165 4k, ale nemus\u00ed by\u0165 presne 4k, t.j. s\u00fa povolen\u00e9 aj 1k a 2k.<\/p>\n<p>Tak\u017ee, ako sme rie\u0161ili probl\u00e9my.<\/p>\n<p>Po prv\u00e9, s probl\u00e9mom sme sa stretli, ke\u010f bola viac ako terabajtov\u00e1 disk naplnen\u00fd d\u00e1tami a nemohli sme zmeni\u0165 konfigur\u00e1ciu s\u00faborov\u00e9ho syst\u00e9mu.<\/p>\n<p>Po druh\u00e9, rie\u0161enie bolo potrebn\u00e9 okam\u017eite.<\/p>\n<p>Nakoniec sme dospeli k z\u00e1veru, \u017ee mus\u00edme zmeni\u0165 rovnov\u00e1hu, zn\u00ed\u017ei\u0165 po\u010det s\u00faborov.<br \/>\nAby sme zn\u00ed\u017eili po\u010det s\u00faborov, rozhodli sme sa ulo\u017ei\u0165 s\u00fabory do jedn\u00e9ho spolo\u010dn\u00e9ho arch\u00edvu. Vzh\u013eadom na na\u0161u \u0161pecifiku sme do jedn\u00e9ho arch\u00edvu ukl\u00e1dali v\u0161etky s\u00fabory za ur\u010dit\u00fd \u010dasov\u00fd \u00fasek a archiva\u010dn\u00e9 \u00falohy sme vykon\u00e1vali pomocou cron ka\u017edej noci.<\/p>\n<p>Vybrali sme zip-arch\u00edv. V koment\u00e1roch k predch\u00e1dzaj\u00facemu \u010dl\u00e1nku bol navrhnut\u00fd tar, ale s t\u00fdm je jedna komplik\u00e1cia: nem\u00e1 obsah a s\u00fabory v \u0148om s\u00fa usporiadan\u00e9 sekven\u010dne (nie nadarmo sa 'tar' skr\u00e1til z 'Tape Archive', dedi\u010dstvo p\u00e1skov\u00fdch pam\u00e4t\u00ed), t.j. ak potrebujete pre\u010d\u00edta\u0165 s\u00fabor na konci arch\u00edvu, mus\u00edte pre\u010d\u00edta\u0165 cel\u00fd arch\u00edv, preto\u017ee neobsahuje posuny pre ka\u017ed\u00fd s\u00fabor vzh\u013eadom na za\u010diatok arch\u00edvu. A preto je to pomal\u00e1 oper\u00e1cia. V zip je to ove\u013ea lep\u0161ie: m\u00e1 predpokladan\u00fd obsah a posuny s\u00faborov v arch\u00edve, a \u010das pr\u00edstupu k ka\u017ed\u00e9mu s\u00faboru nez\u00e1vis\u00ed od jeho umiestnenia. No a v na\u0161om pr\u00edpade sme mohli stanovi\u0165 kompresn\u00fa mo\u017enos\u0165 '0', preto\u017ee v\u0161etky s\u00fabory boli u\u017e vopred skomprimovan\u00e9 v gzip.<\/p>\n<p>Klienti s\u0165ahuj\u00fa s\u00fabory cez nginx, a pod\u013ea star\u00e9ho API, sa uv\u00e1dza jednoducho n\u00e1zov s\u00faboru, napr\u00edklad takto:<\/p>\n<pre><code class=\"plaintext\">http:\/\/www.server.com\/hydra\/20170416\/0453\/3bd24ae7-1df4-4d76-9d28-5b7fcb7fd8e5\n<\/code><\/pre>\n<p>\nAby sme mohli na let priamo rozbali\u0165 s\u00fabory, na\u0161li sme a pripojili modul nginx-unzip-module (<noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/youzee\/nginx-unzip-module\">https:\/\/github.com\/youzee\/nginx-unzip-module<\/a><\/noindex>) a nastavili sme dva upstreamy.<\/p>\n<p>V\u00fdsledkom bola tak\u00e1to konfigur\u00e1cia:<\/p>\n<p><img decoding=\"async\" alt=\"Triky pri pr\u00e1ci s ve\u013ek\u00fdm po\u010dtom mal\u00fdch s\u00faborov\" src=\"\/wp-content\/uploads\/2019\/08\/56f5210669fecfe194ac5907d563aa1d.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nDva hostitelia mali nastavenia takto:<\/p>\n<pre><code class=\"json\">server {\n  listen *:8081;\n\n  location \/ {\n    root      \/home\/filestorage;\n  }\n}<\/code><\/pre>\n<p><\/p>\n<pre><code class=\"json\">server {\n  listen *:8082;\n\n  location ~ ^\/hydra\/(d+)\/(\n    root      \/home\/filestorage;\n    file_in_unzip_archivefile \"\/home\/filestorage\/hydra\/$1\/$2.zip\";\n    file_in_unzip_extract \"$2\/$3\";\n    file_in_unzip;\n  }\n}\n<\/code><\/pre>\n<p>\nA konfigur\u00e1cia upstreamov na vy\u0161\u0161om nginx:<\/p>\n<pre><code class=\"json\">upstream storage {\n  server server.com:8081;\n  server server.com:8082;\n}\n<\/code><\/pre>\n<p>\nAko to funguje:<\/p>\n<ul>\n<li>Klient ide na front nginx<\/li>\n<li>Front nginx sa sna\u017e\u00ed poskytn\u00fa\u0165 s\u00fabor z prv\u00e9ho upstreamu, t.j. priamo zo s\u00faborov\u00e9ho syst\u00e9mu<\/li>\n<li>Ak s\u00fabor neexistuje \u2014 sna\u017e\u00ed sa ho poskytn\u00fa\u0165 z druh\u00e9ho upstreamu, ktor\u00fd sa pok\u00fa\u0161a n\u00e1js\u0165 s\u00fabor v arch\u00edve<\/li>\n<\/ul>\n<p><\/p>\n<h2>Druh\u00e1 probl\u00e9m: op\u00e4\u0165 \"No space left on device\"<\/h2>\n<p>\nTo je druh\u00fd probl\u00e9m, s ktor\u00fdm sme sa stretli, ke\u010f je v adres\u00e1ri ve\u013ea s\u00faborov.<br \/>\nSna\u017e\u00edme sa vytvori\u0165 s\u00fabor, syst\u00e9m sa s\u0165a\u017euje, \u017ee nie je miesto. Zmen\u00edme n\u00e1zov s\u00faboru a znovu sa pok\u00fasime ho vytvori\u0165.<\/p>\n<p>Podar\u00ed sa.<\/p>\n<p>Vyzer\u00e1 to pribli\u017ene takto:<\/p>\n<p><img decoding=\"async\" alt=\"Triky pri pr\u00e1ci s ve\u013ek\u00fdm po\u010dtom mal\u00fdch s\u00faborov\" src=\"\/wp-content\/uploads\/2019\/08\/f365358b59bf81d450a236dfb58e4874.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nKontrola inodes ni\u010d nedala \u2014 vo\u013en\u00fdch je ich dos\u0165. <br \/>\nKontrola miesta \u2014 to ist\u00e9.<br \/>\nPrem\u00fd\u0161\u013eali sme, \u010di nie je v adres\u00e1ri pr\u00edli\u0161 ve\u013ea s\u00faborov, a na to je obmedzenie, ale op\u00e4\u0165 nie: Maxim\u00e1lny po\u010det s\u00faborov na adres\u00e1r: ~1,3 \u00d7 10^20<\/p>\n<p>A predsa s\u00fabor mo\u017eno vytvori\u0165, ak zmen\u00edte n\u00e1zov.<br \/>\nZ\u00e1ver \u2014 probl\u00e9m je v n\u00e1zve s\u00faboru.<\/p>\n<p>\u010eal\u0161ie vyh\u013ead\u00e1vanie uk\u00e1zalo, \u017ee probl\u00e9m je v hashovacom algoritme pri kon\u0161trukcii indexu adres\u00e1ra, pri ve\u013ekom po\u010dte s\u00faborov doch\u00e1dza ku kol\u00edzi\u00e1m so v\u0161etk\u00fdmi vypl\u00fdvaj\u00facimi n\u00e1sledkami. Podrobnej\u0161ie si m\u00f4\u017eete pre\u010d\u00edta\u0165 tu: <noindex><a rel=\"nofollow\" href=\"https:\/\/ext4.wiki.kernel.org\/index.php\/Ext4_Disk_Layout#Hash_Tree_Directories\">https:\/\/ext4.wiki.kernel.org\/index.php\/Ext4_Disk_Layout#Hash_Tree_Directories<\/a><\/noindex><\/p>\n<p>T\u00fato mo\u017enos\u0165 m\u00f4\u017eete vypn\u00fa\u0165, ale... vyh\u013ead\u00e1vanie s\u00faboru pod\u013ea n\u00e1zvu m\u00f4\u017ee by\u0165 nepredv\u00eddate\u013ene dlh\u00e9 pri prechode v\u0161etk\u00fdmi s\u00fabormi. <\/p>\n<pre><code class=\"bash\"> tune2fs -O \"^dir_index\" \/dev\/sdb3\n<\/code><\/pre>\n<p>\nVo v\u0161eobecnosti to m\u00f4\u017ee fungova\u0165 ako do\u010dasn\u00e9 rie\u0161enie.<\/p>\n<p>Mor\u00e1lka: ve\u013ea s\u00faborov v adres\u00e1ri je spravidla zl\u00e9. Takto to nerobte.<\/p>\n<p>V tak\u00fdchto pr\u00edpadoch sa zvy\u010dajne vytv\u00e1raj\u00fa vnoren\u00e9 adres\u00e1re, pod\u013ea prv\u00fdch p\u00edsmen n\u00e1zvu s\u00faboru alebo in\u00fdch parametrov, napr\u00edklad pod\u013ea d\u00e1tumov, vo v\u00e4\u010d\u0161ine pr\u00edpadov to pom\u00e1ha.<br \/>\nAle celkov\u00fd po\u010det mal\u00fdch s\u00faborov je st\u00e1le zl\u00fd, aj ke\u010f s\u00fa rozdelen\u00e9 do adres\u00e1rov \u2014 potom pozrite prv\u00fd probl\u00e9m.<\/p>\n<h2>Tret\u00ed probl\u00e9m: ako zobrazi\u0165 zoznam s\u00faborov, ak ich je ve\u013ea<\/h2>\n<p>\nV na\u0161ej situ\u00e1cii, ke\u010f m\u00e1me ve\u013ea s\u00faborov, sa tak \u010di onak stret\u00e1vame s probl\u00e9mom, ako si pozrie\u0165 obsah adres\u00e1ra.<\/p>\n<p>\u0160tandardn\u00e9 rie\u0161enie je pr\u00edkaz <code>ls<\/code>.<br \/>\nDobre, pozrime sa, \u010do dostaneme na 4772098 s\u00faboroch:<\/p>\n<pre><code class=\"bash\">\n$ time ls \/home\/app\/express.repository\/offercache\/ &gt;\/dev\/null\n\nreal\t0m30.203s\nuser\t0m28.327s\nsys\t0m1.876s\n<\/code><\/pre>\n<p>\n30 sek\u00fand... to je dos\u0165. Hlavn\u00fd \u010das ide na spracovanie s\u00faborov v u\u017e\u00edvate\u013eskom priestore, nie na pr\u00e1cu jadra.<\/p>\n<p>Ale existuje rie\u0161enie:<\/p>\n<pre><code class=\"bash\">\n$ time find \/home\/app\/express.repository\/offercache\/ &gt;\/dev\/null\n\nreal\t0m3.714s\nuser\t0m1.998s\nsys\t0m1.717s\n<\/code><\/pre>\n<p>\n3 sekundy. Desakr\u00e1t r\u00fdchlej\u0161ie.<br \/>\nHur\u00e1!<\/p>\n<p><b>UPD.<\/b><\/p>\n<p>E\u0161te r\u00fdchlej\u0161ie rie\u0161enie od u\u017e\u00edvate\u013ea <noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/users\/berez\/\" class=\"user_link\">berez<\/a><\/noindex> \u2014 vypnutie triedenia pre <code>ls<\/code><\/p>\n<pre><code class=\"bash\">\ntime ls -U \/home\/app\/express.repository\/offercache\/ &gt;\/dev\/null\nreal\t0m2.985s\nuser\t0m1.377s\nsys\t0m1.608s\n<\/code><\/pre>\n<h2>\u0160tvrt\u00fd probl\u00e9m: vysok\u00fd LA pri pr\u00e1ci so s\u00fabormi<\/h2>\n<p>\nPravidelne nast\u00e1va situ\u00e1cia, ke\u010f je potrebn\u00e9 skop\u00edrova\u0165 mno\u017estvo s\u00faborov z jedn\u00e9ho stroja na druh\u00fd. Pri tom \u010dasto rap\u00eddne st\u00fapa LA, preto\u017ee v\u0161etko nar\u00e1\u017ea na v\u00fdkon samotn\u00fdch diskov.<\/p>\n<p>Najrozumnej\u0161ie, \u010do chcete \u2014 je pou\u017ei\u0165 SSD. Ove\u013ea lep\u0161ie. Ot\u00e1zka je len v cene mnohaterabytov\u00fdch SSD.<\/p>\n<p>Ale ak s\u00fa disky be\u017en\u00e9, s\u00fabory treba skop\u00edrova\u0165, a to je okrem toho aj produk\u010dn\u00fd syst\u00e9m, kde pre\u0165a\u017eovanie vedie k nespokojnosti z\u00e1kazn\u00edkov? Existuj\u00fa aspo\u0148 dva u\u017eito\u010dn\u00e9 n\u00e1stroje: <code>nice<\/code> a <code>ionice<\/code>.<\/p>\n<p><code>nice<\/code> \u2014 zni\u017euje prioritu procesu, tak\u017ee pl\u00e1nova\u010d pride\u013euje viac \u010dasov\u00fdch kvant in\u00fdm, prioritnej\u0161\u00edm procesom.<br \/>\nV na\u0161ej praxi pomohlo nastavi\u0165 nice na maximum (19 \u2014 to je minim\u00e1lna priorita, -20 (m\u00ednus 20) \u2014 maxim\u00e1lna).<\/p>\n<p><code>ionice<\/code> \u2014 teda koreguje prioritu vstupu\/v\u00fdstupu (I\/O scheduling).<\/p>\n<p>Ak m\u00e1te RAID a je potrebn\u00e9 ho n\u00e1hle synchronizova\u0165 (po zlyhan\u00ed re\u0161tartu alebo potrebn\u00e9 obnovenie RAID po v\u00fdmene disku), v ur\u010dit\u00fdch situ\u00e1ci\u00e1ch m\u00e1 zmysel zn\u00ed\u017ei\u0165 r\u00fdchlos\u0165 synchroniz\u00e1cie, aby ostatn\u00e9 procesy mohli fungova\u0165 aspo\u0148 akosi adekv\u00e1tne. Na to pom\u00f4\u017ee nasleduj\u00faci pr\u00edkaz:<\/p>\n<pre><code class=\"bash\">\necho 1000 &gt; \/proc\/sys\/dev\/raid\/speed_limit_max\n<\/code><\/pre>\n<p><\/p>\n<h2>Probl\u00e9m piaty: Ako synchronizova\u0165 s\u00fabory v re\u00e1lnom \u010dase<\/h2>\n<p>\nM\u00e1me v\u0161etky tie obrovsk\u00e9 mno\u017estv\u00e1 s\u00faborov, ktor\u00e9 je potrebn\u00e9 z\u00e1lohova\u0165 na druh\u00fd server, aby sa predi\u0161lo... S\u00fabory sa neust\u00e1le zapisuj\u00fa, a preto, aby sme mali minim\u00e1lne straty, mus\u00edme ich kop\u00edrova\u0165 maxim\u00e1lne r\u00fdchlo.<\/p>\n<p>\u0160tandardn\u00e9 rie\u0161enie: Rsync cez SSH.<\/p>\n<p>Je to dobr\u00e1 vo\u013eba, pokia\u013e to nemus\u00edte robi\u0165 ka\u017ed\u00fdch nieko\u013eko sek\u00fand. A s\u00faborov je mnoho. Aj ke\u010f ich nekop\u00edrujeme \u2014 aj tak je potrebn\u00e9 zisti\u0165, \u010do sa zmenilo, a porovnanie nieko\u013ek\u00fdch mili\u00f3nov s\u00faborov zaberie \u010das a za\u0165a\u017eenie diskov.<\/p>\n<p>T.j. potrebujeme vedie\u0165 okam\u017eite, \u010do je potrebn\u00e9 kop\u00edrova\u0165, bez toho, aby sme ka\u017ed\u00fa chv\u00ed\u013eu sp\u00fa\u0161\u0165ali porovnanie.<\/p>\n<p>Rie\u0161enie \u2014 <code>lsyncd<\/code>. <code>Lsyncd<\/code> \u2014 <noindex><a rel=\"nofollow\" href=\"https:\/\/axkibe.github.io\/lsyncd\/\">Demon pre \u017eiv\u00e9 synchronizovanie (Mirror)<\/a><\/noindex>. Funguje tie\u017e cez rsync, ale navy\u0161e monitoruje s\u00faborov\u00fd syst\u00e9m na zmeny pomocou inotify a fsevents a sp\u00fa\u0161\u0165a kop\u00edrovanie iba pre tie s\u00fabory, ktor\u00e9 sa objavili alebo zmenili.<\/p>\n<h2>Probl\u00e9m \u0161iesty: ako pochopi\u0165, kto za\u0165a\u017euje disky<\/h2>\n<p>\nTo asi vie ka\u017ed\u00fd, ale predsa len pre celkov\u00fd obraz: na sledovanie diskov\u00e9ho subsyst\u00e9mu existuje pr\u00edkaz <code>iotop<\/code> \u2014 podobne ako <code>top<\/code>, ale zobrazuje procesy, ktor\u00e9 najakt\u00edvnej\u0161ie pou\u017e\u00edvaj\u00fa disky.<\/p>\n<p><img decoding=\"async\" alt=\"Triky pri pr\u00e1ci s ve\u013ek\u00fdm po\u010dtom mal\u00fdch s\u00faborov\" src=\"\/wp-content\/uploads\/2019\/08\/56612c618469ef340a31ae446d65ed4e.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nMimochodom, star\u00fd dobr\u00fd top tie\u017e umo\u017e\u0148uje zisti\u0165, \u010di s\u00fa nejak\u00e9 probl\u00e9my s diskami. Na to s\u00fa dva najvhodnej\u0161ie parametre: <b>Load Average<\/b> a <b>IOwait<\/b>.<\/p>\n<p><img decoding=\"async\" alt=\"Triky pri pr\u00e1ci s ve\u013ek\u00fdm po\u010dtom mal\u00fdch s\u00faborov\" src=\"\/wp-content\/uploads\/2019\/08\/01a0565947de65d13b0045f7180caf5d.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nPrv\u00fd zobrazuje, ko\u013eko procesov \u010dak\u00e1 na obsluhu, zvy\u010dajne viac ako 2 \u2014 u\u017e nie\u010do nie je v poriadku. Pri akt\u00edvnom kop\u00edrovan\u00ed na z\u00e1lo\u017en\u00e9 servery povolujeme a\u017e 6-8, po tom sa situ\u00e1cia pova\u017euje za mimoriadnu.<\/p>\n<p>Druh\u00fd \u2014 ako ve\u013emi je procesor za\u0165a\u017een\u00fd diskov\u00fdmi oper\u00e1ciami. IOwait &gt;10% \u2014 je to d\u00f4vod na obavy, hoci na na\u0161ich serveroch so \u0161pecifick\u00fdm profilom za\u0165a\u017eenia b\u00fdva stabilne 40-50%, a to je naozaj norma.<\/p>\n<p>T\u00fdmto kon\u010d\u00edm, hoci je ur\u010dite mnoho momentov, s ktor\u00fdmi sme sa nestretli, s rados\u0165ou budem \u010daka\u0165 na koment\u00e1re a opisy zauj\u00edmav\u00fdch skuto\u010dn\u00fdch pr\u00edpadov.<br \/>\n<br \/>Zdroj: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/srg\/blog\/462967\/\">habr.com<\/a><\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u0418\u0434\u0435\u044f \u0441\u0442\u0430\u0442\u044c\u0438 \u0440\u043e\u0434\u0438\u043b\u0430\u0441\u044c \u0441\u043f\u043e\u043d\u0442\u0430\u043d\u043d\u043e \u0438\u0437 \u0434\u0438\u0441\u043a\u0443\u0441\u0441\u0438\u0438 \u0432 \u043a\u043e\u043c\u043c\u0435\u043d\u0442\u0430\u0440\u0438\u044f\u0445 \u043a \u0441\u0442\u0430\u0442\u044c\u0435 \u00ab\u041a\u043e\u0435-\u0447\u0442\u043e \u043e\u0431 inode\u00bb. \u0414\u0435\u043b\u043e \u0432 \u0442\u043e\u043c, \u0447\u0442\u043e \u0432\u043d\u0443\u0442\u0440\u0435\u043d\u043d\u0435\u0439 \u0441\u043f\u0435\u0446\u0438\u0444\u0438\u043a\u043e\u0439 \u0440\u0430\u0431\u043e\u0442\u044b \u043d\u0430\u0448\u0438\u0445 \u0441\u0435\u0440\u0432\u0438\u0441\u043e\u0432 \u044f\u0432\u043b\u044f\u0435\u0442\u0441\u044f \u0445\u0440\u0430\u043d\u0435\u043d\u0438\u0435 \u043e\u0433\u0440\u043e\u043c\u0430\u0434\u043d\u043e\u0433\u043e \u0447\u0438\u0441\u043b\u0430 \u043c\u0435\u043b\u043a\u0438\u0445 \u0444\u0430\u0439\u043b\u043e\u0432. \u041d\u0430 \u0434\u0430\u043d\u043d\u044b\u0439 \u043c\u043e\u043c\u0435\u043d\u0442 \u0443 \u043d\u0430\u0441 \u043f\u043e\u0440\u044f\u0434\u043a\u0430 \u0441\u043e\u0442\u0435\u043d \u0442\u0435\u0440\u0430\u0431\u0430\u0439\u0442 \u0442\u0430\u043a\u0438\u0445 \u0434\u0430\u043d\u043d\u044b\u0445. \u0418 \u043c\u044b \u043d\u0430\u0442\u043e\u043b\u043a\u043d\u0443\u043b\u0438\u0441\u044c \u043d\u0430 \u043d\u0435\u043a\u043e\u0442\u043e\u0440\u044b\u0435 \u043e\u0447\u0435\u0432\u0438\u0434\u043d\u044b\u0435 \u0438 \u043d\u0435 \u043e\u0447\u0435\u043d\u044c \u0433\u0440\u0430\u0431\u0435\u043b\u044c\u043a\u0438 \u0438 \u0443\u0441\u043f\u0435\u0448\u043d\u043e \u043f\u043e \u043d\u0438\u043c \u043f\u0440\u043e\u0448\u043b\u0438\u0441\u044c. \u041f\u043e\u044d\u0442\u043e\u043c\u0443 \u0434\u0435\u043b\u044e\u0441\u044c [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":27728,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-36999","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.4 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u0418\u0434\u0435\u044f \u0441\u0442\u0430\u0442\u044c\u0438 \u0440\u043e\u0434\u0438\u043b\u0430\u0441\u044c \u0441\u043f\u043e\u043d\u0442\u0430\u043d\u043d\u043e \u0438\u0437 \u0434\u0438\u0441\u043a\u0443\u0441\u0441\u0438\u0438 \u0432 \u043a\u043e\u043c\u043c\u0435\u043d\u0442\u0430\u0440\u0438\u044f\u0445 \u043a \u0441\u0442\u0430\u0442\u044c\u0435 \u00ab\u041a\u043e\u0435-\u0447\u0442\u043e \u043e\u0431 inode\u00bb.\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/haki-pri-rabote-s-bolshim-chislom-melkih-fajlov\" \/>\n\t\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.4\" \/>\n\t\t<meta property=\"og:locale\" content=\"sk_SK\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47\u0425\u0430\u043a\u0438 \u043f\u0440\u0438 \u0440\u0430\u0431\u043e\u0442\u0435 \u0441 \u0431\u043e\u043b\u044c\u0448\u0438\u043c \u0447\u0438\u0441\u043b\u043e\u043c \u043c\u0435\u043b\u043a\u0438\u0445 \u0444\u0430\u0439\u043b\u043e\u0432 | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u0418\u0434\u0435\u044f \u0441\u0442\u0430\u0442\u044c\u0438 \u0440\u043e\u0434\u0438\u043b\u0430\u0441\u044c \u0441\u043f\u043e\u043d\u0442\u0430\u043d\u043d\u043e \u0438\u0437 \u0434\u0438\u0441\u043a\u0443\u0441\u0441\u0438\u0438 \u0432 \u043a\u043e\u043c\u043c\u0435\u043d\u0442\u0430\u0440\u0438\u044f\u0445 \u043a \u0441\u0442\u0430\u0442\u044c\u0435 \u00ab\u041a\u043e\u0435-\u0447\u0442\u043e \u043e\u0431 inode\u00bb.\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/haki-pri-rabote-s-bolshim-chislom-melkih-fajlov\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2019-10-31T19:15:13+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2019-10-31T19:15:13+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47Tipy na pr\u00e1cu s ve\u013ek\u00fdm po\u010dtom mal\u00fdch s\u00faborov | ProHoster","description":"N\u00e1pad na tento \u010dl\u00e1nok vznikol spont\u00e1nne z diskusie v koment\u00e1roch k \u010dl\u00e1nku \u201eNie\u010do o inode\u201c.","canonical_url":"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/haki-pri-rabote-s-bolshim-chislom-melkih-fajlov","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"sk_SK","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u0425\u0430\u043a\u0438 \u043f\u0440\u0438 \u0440\u0430\u0431\u043e\u0442\u0435 \u0441 \u0431\u043e\u043b\u044c\u0448\u0438\u043c \u0447\u0438\u0441\u043b\u043e\u043c \u043c\u0435\u043b\u043a\u0438\u0445 \u0444\u0430\u0439\u043b\u043e\u0432 | ProHoster","og:description":"\u0418\u0434\u0435\u044f \u0441\u0442\u0430\u0442\u044c\u0438 \u0440\u043e\u0434\u0438\u043b\u0430\u0441\u044c \u0441\u043f\u043e\u043d\u0442\u0430\u043d\u043d\u043e \u0438\u0437 \u0434\u0438\u0441\u043a\u0443\u0441\u0441\u0438\u0438 \u0432 \u043a\u043e\u043c\u043c\u0435\u043d\u0442\u0430\u0440\u0438\u044f\u0445 \u043a \u0441\u0442\u0430\u0442\u044c\u0435 \u00ab\u041a\u043e\u0435-\u0447\u0442\u043e \u043e\u0431 inode\u00bb.","og:url":"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/haki-pri-rabote-s-bolshim-chislom-melkih-fajlov","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2019-10-31T19:15:13+00:00","article:modified_time":"2019-10-31T19:15:13+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"36999","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":"2026-01-22 05:41:19","breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-03-01 01:35:26","updated":"2026-01-22 05:41:19","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/posts\/36999","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/comments?post=36999"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/posts\/36999\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/media\/27728"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/media?parent=36999"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/categories?post=36999"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/tags?post=36999"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}