Lingaliro la nkhaniyi linabadwa mwachisawawa kuchokera pazokambirana mu ndemanga mpaka ku nkhaniyi .
Chowonadi ndi chakuti tsatanetsatane wamkati wa mautumiki athu ndikusungirako chiwerengero chachikulu cha mafayilo ang'onoang'ono. Pakalipano tili ndi pafupifupi mazana a ma terabytes a data yotere. Ndipo tidapeza zowoneka bwino komanso zosawoneka bwino ndikuziyendetsa bwino.
Choncho, ndikugawana zomwe takumana nazo, mwinamwake zingakhale zothandiza kwa wina.
Vuto loyamba: "Palibe malo otsala pachida"
Monga tafotokozera m'nkhani yomwe yatchulidwa pamwambapa, vuto ndiloti pali midadada yaulere pamafayilo, koma inode yatha.
Mutha kuyang'ana kuchuluka kwa ma innode ogwiritsidwa ntchito komanso aulere ndi lamulo df -ih:
Sindinenanso nkhaniyo; Mwachidule, diski ili ndi midadada yonse ya data yokha komanso midadada ya chidziwitso cha meta, chomwe chimadziwikanso kuti inodes (index node). Nambala yawo imayikidwa pomwe mafayilo amakhazikitsidwa (tikulankhula za ext2 ndi olowa m'malo mwake) ndipo sasinthanso. Kuchuluka kwa midadada ya deta ndi ma innode kumawerengedwa kuchokera ku chiwerengero cha ziwerengero, koma kwa ife, pamene pali mafayilo ang'onoang'ono ambiri, ndalamazo ziyenera kusunthira ku chiwerengero cha ma inode - payenera kukhala zambiri.
В Linux Tapereka kale njira zosiyanasiyana, ndipo makonzedwe onsewa omwe awerengedwa kale ali mufayilo /etc/mke2fs.conf.
Chifukwa chake, pakuyambitsa koyambirira kwamafayilo kudzera pa mke2fs, mutha kufotokoza zomwe mukufuna.
Nazi zitsanzo kuchokera pafayilo:
small = {
blocksize = 1024
inode_size = 128
inode_ratio = 4096
}
big = {
inode_ratio = 32768
}
largefile = {
inode_ratio = 1048576
blocksize = -1
}
Mutha kusankha njira yomwe mukufuna kugwiritsa ntchito "-T" poyimba mke2fs. Mukhozanso kukhazikitsa pamanja magawo ofunikira ngati palibe yankho lokonzekera.
Zambiri zafotokozedwa m'mabuku a mke2fs.conf и mke2fs.
Chinthu chomwe sichinakhudzidwe m'nkhani yomwe tatchulayi ndikuti mutha kukhazikitsa kukula kwa chipika cha data. Mwachiwonekere, kwa mafayilo akuluakulu ndizomveka kukhala ndi kukula kwakukulu kwa chipika, kwa mafayilo ang'onoang'ono ndizomveka kukhala ndi kakang'ono.
Komabe, ndikofunikira kulingalira chinthu chosangalatsa ngati kamangidwe ka purosesa.
Nthawi ina ndimaganiza kuti pamafayilo akulu azithunzi ndimafunikira kukula kwa chipika. Izo zinachitika kunyumba, pa nyumba wapamwamba sitolo dzina lake WD pa ARM zomangamanga. Mosakayikira, ndinayika kukula kwa chipika kukhala 8k kapena 16k m'malo mwa 4k, nditayesa kale kusunga. Ndipo zonse zinali bwino mpaka kusungirako komweko kunalephera, ndipo disk idakali yamoyo. Nditayika diskiyo pakompyuta yokhazikika yokhala ndi purosesa ya Intel wamba, ndidadabwa: kukula kwa block yosagwiritsidwa ntchito. Tafika. Deta ilipo, zonse zili bwino, koma sizingatheke kuwerenga. i386 ndi mapurosesa ofanana sangathe kugwira ntchito ndi makulidwe a block omwe sagwirizana ndi kukula kwa tsamba lokumbukira, lomwe lili ndendende 4k. Kawirikawiri, nkhaniyi inatha ndi kugwiritsa ntchito zipangizo kuchokera kumalo ogwiritsira ntchito, chirichonse chinali chochepa komanso chachisoni, koma deta inasungidwa. Ngati wina ali ndi chidwi, google dzina lazothandizira fuseext2. Makhalidwe: mwina ganizirani milandu yonse pasadakhale, kapena musadzipangire kukhala ngwazi ndipo gwiritsani ntchito makonda a amayi apakhomo.
UPD. Malinga ndi ndemanga ya ogwiritsa ntchito Ndikufuna kufotokozera kuti kwa i386 kukula kwa chipika sikuyenera kupitirira 4k, koma sikuyenera kukhala ndendende 4k, i.e. 1k ndi 2k ndizovomerezeka.
Ndiye tinathetsa bwanji mavutowo?
Choyamba, tinakumana ndi vuto pamene disk ya multi-terabyte inali yodzaza ndi deta, ndipo sitinathe kusintha mawonekedwe a fayilo.
Kachiwiri, yankho linkafunika mwachangu.
Chotsatira chake, tinafika pamapeto kuti tifunika kusintha malire mwa kuchepetsa chiwerengero cha mafayilo.
Kuti muchepetse kuchuluka kwa mafayilo, adaganiza zoyika mafayilowo muakale imodzi wamba. Poganizira zomwe tafotokozazi, timayika mafayilo onse pakanthawi kochepa munkhokwe imodzi, ndipo timasunga zakale pogwiritsa ntchito cron usiku uliwonse.
Zosungidwa za zip zosankhidwa. M'mawu a m'nkhani yapitayi, phula idanenedwa, koma pali vuto limodzi: ilibe mndandanda wazomwe zili mkati, ndipo mafayilo omwe ali mmenemo amasungidwa mumtsinje (sikuti "phula" ndi chidule cha mawu. kwa "Tape Archive", cholowa cha ma drive a tepi), mwachitsanzo. ngati mukufuna kuwerenga fayilo kumapeto kwa zosungirako, muyenera kuwerenga zolemba zonse, popeza ilibe zochotsera pa fayilo iliyonse yokhudzana ndi chiyambi cha zosungirako. Choncho ndi yaitali ntchito. Chilichonse chili bwino mu zip: ili ndi mndandanda womwewo wa zomwe zili mkati ndi mafayilo omwe ali mkati mwazosungirako, ndipo nthawi yofikira fayilo iliyonse sikudalira malo ake. Chabwino, kwa ife, zinali zotheka kukhazikitsa njira yopondereza kukhala "0", chifukwa mafayilo onse anali atapanikizidwa kale mu gzip pasadakhale.
Makasitomala amatenga mafayilo kudzera pa nginx, ndipo molingana ndi API yakale, dzina lafayilo limatchulidwa, mwachitsanzo monga chonchi:
http://www.server.com/hydra/20170416/0453/3bd24ae7-1df4-4d76-9d28-5b7fcb7fd8e5
Kuti titulutse mafayilo pa ntchentche, tidapeza ndikulumikiza nginx-unzip-module module () ndikukhazikitsa mitsinje iwiri yokwera.
Zotsatira zake zidakhala motere:
Mitundu yambiri yamitundu yosiyanasiyana imawonekera motere:
server {
listen *:8081;
location / {
root /home/filestorage;
}
}server {
listen *:8082;
location ~ ^/hydra/(d+)/(d+)/(.*)$ {
root /home/filestorage;
file_in_unzip_archivefile "/home/filestorage/hydra/$1/$2.zip";
file_in_unzip_extract "$2/$3";
file_in_unzip;
}
}
Ndipo kasinthidwe kamtunda pamtsinje wa nginx:
upstream storage {
server server.com:8081;
server server.com:8082;
}
Zimagwira ntchito bwanji:
- Wothandizira amapita kutsogolo kwa nginx
- Front nginx ikuyesera kutumiza fayilo kuchokera kumtunda woyamba, i.e. mwachindunji kuchokera ku fayilo ya fayilo
- Ngati palibe fayilo, imayesa kutumiza kuchokera kumtunda wachiwiri, womwe umayesa kupeza fayilo mkati mwazosungirako.
Vuto lachiwiri: kachiwiri "Palibe malo otsalira pa chipangizo"
Ili ndi vuto lachiwiri lomwe tidakumana nalo pakakhala mafayilo ambiri m'ndandanda.
Tikuyesera kupanga fayilo, dongosolo likudandaula kuti palibe malo. Timasintha dzina la fayilo ndikuyesera kulipanganso.
Zikukhalira.
Zikuwoneka motere:
Kuwona ma innode sikunapereke kalikonse - pali ambiri aulere.
Kuyang'ana malo ndi chimodzimodzi.
Tinkaganiza kuti pakhoza kukhala mafayilo ochulukirapo m'ndandanda, ndipo pali malire pa izi, koma ayi: Chiwerengero chachikulu cha mafayilo pa bukhuli: ~ 1.3 × 10 ^ 20
Inde, ndipo mutha kupanga fayilo ngati mutasintha dzina.
Mapeto - vuto liri mu dzina la fayilo.
Kufufuza kwina kunawonetsa kuti vuto liri mu hashing algorithm popanga index index; ndi kuchuluka kwa mafayilo, kugunda kumawonedwa ndi zotsatira zake zonse. Mutha kuwerenga zambiri apa:
Mutha kuletsa izi, koma ... kufunafuna fayilo ndi dzina kumatha kukhala motalika mosadziwika bwino mukusaka mafayilo onse.
tune2fs -O "^dir_index" /dev/sdb3
Nthawi zambiri, iyi ndi njira yosakhalitsa yomwe ingagwire ntchito.
Makhalidwe: mafayilo ambiri mu bukhuli nthawi zambiri amakhala oipa. Palibe chifukwa chochitira izi.
Nthawi zambiri pazifukwa zotere, zolembera zomwe zimayikidwa zimapangidwa ndi zilembo zoyambirira za dzina la fayilo kapena magawo ena, mwachitsanzo, ndi masiku, nthawi zambiri izi zimasunga tsiku.
Koma chiwerengero cha mafayilo ang'onoang'ono akadali oipa, ngakhale atagawidwa m'mabuku - ndiye onani vuto loyamba.
Vuto lachitatu: momwe mungawonere mndandanda wa mafayilo ngati alipo ambiri
M'mikhalidwe yathu, tikakhala ndi mafayilo ambiri, mwanjira ina tidakumana ndi vuto la momwe tingawonere zomwe zili m'ndandanda.
Njira yothetsera - lamulo ls.
Chabwino, tiyeni tiwone zomwe zimachitika pamafayilo a 4772098:
$ time ls /home/app/express.repository/offercache/ >/dev/null
real 0m30.203s
user 0m28.327s
sys 0m1.876s
Masekondi 30... adzakhala ochuluka kwambiri. Komanso, nthawi zambiri amagwiritsidwa ntchito pokonza mafayilo pamalo ogwiritsira ntchito, osati pakugwira ntchito kwa kernel.
Koma pali yankho:
$ time find /home/app/express.repository/offercache/ >/dev/null
real 0m3.714s
user 0m1.998s
sys 0m1.717s
3 masekondi. 10 nthawi mwachangu.
Hooray!
UPD.
Yankho lachangu kwambiri kuchokera kwa wogwiritsa ntchito - zimitsani kusanja ls
time ls -U /home/app/express.repository/offercache/ >/dev/null
real 0m2.985s
user 0m1.377s
sys 0m1.608s
Vuto lachinayi: lalikulu LA mukamagwira ntchito ndi mafayilo
Nthawi ndi nthawi zinthu zimachitika pamene muyenera kukopera gulu la mafayilo kuchokera pamakina amodzi kupita ku ena. Nthawi yomweyo, LA nthawi zambiri imakula mopanda nzeru, chifukwa chilichonse chimadalira momwe ma disks amagwirira ntchito.
Chinthu chanzeru kwambiri chomwe mukufuna kuchita ndikugwiritsa ntchito SSD. Zozizira kwambiri. Funso lokhalo ndi mtengo wa ma SSD a multi-terabyte.
Koma ngati ma disks ndi wamba, muyenera kukopera mafayilo, ndipo iyi ndi njira yopangira, yomwe imadzaza ndi mawu osakhutira kuchokera kwa makasitomala? Pali zida ziwiri zothandiza: nice и ionice.
nice - imachepetsa kufunikira kwa ndondomekoyi, motero wokonza ndondomekoyo amagawira magawo a nthawi yambiri kuzinthu zina zofunika kwambiri.
M'zochita zathu, zidathandizira kuyika bwino kwambiri (19 ndiye chofunikira kwambiri, -20 (kuchotsa 20) ndiye pazipita).
ionice - imasintha makonzedwe a I/O moyenerera
Ngati mukugwiritsa ntchito RAID ndipo imayenera kulumikizidwa mwadzidzidzi (pambuyo kuyambiranso kosachita bwino kapena muyenera kubwezeretsanso gulu la RAID mutasintha disk), ndiye kuti nthawi zina ndizomveka kuchepetsa liwiro lolumikizana kuti njira zina zizigwira ntchito. kapena zochepa mokwanira. Lamulo lotsatirali lithandiza izi:
echo 1000 > /proc/sys/dev/raid/speed_limit_max
Vuto lachisanu: Momwe mungalumikizire mafayilo munthawi yeniyeni
Tili ndi chiwerengero chofanana cha mafayilo omwe amayenera kuthandizidwa ku seva yachiwiri kuti asapewe ... Mafayilo amalembedwa nthawi zonse, kotero kuti mukhale ndi zotayika zochepa, muyenera kuzikopera mwamsanga.
Yankho lokhazikika: Rsync pa SSH.
Iyi ndi njira yabwino pokhapokha ngati muyenera kuchita masekondi angapo aliwonse. Ndipo pali mafayilo ambiri. Ngakhale simukuwakopera, muyenerabe kumvetsetsa zomwe zasintha, ndipo kufananiza mafayilo mamiliyoni angapo kumatenga nthawi ndikuyika pa disk.
Iwo. tiyenera kudziwa nthawi yomweyo zomwe ziyenera kukopera, osayesa kufananitsa nthawi zonse.
Kupulumutsa - lsyncd. Lsyncd - . Imagwiranso ntchito kudzera pa rsync, koma imayang'aniranso mawonekedwe a fayilo kuti asinthe pogwiritsa ntchito inotify ndi fsevents ndikungoyamba kukopera mafayilo omwe awonekera kapena asinthidwa.
Vuto lachisanu ndi chimodzi: momwe mungamvetsetse yemwe akukweza ma disks
Aliyense amadziwa izi, komabe, kuti amalize chithunzichi: pali lamulo loyang'anira gawo la disk iotop - monga top, koma ikuwonetsa njira zomwe zimagwiritsa ntchito ma disks mwachangu.
Mwa njira, nsonga yabwino yakale imakulolani kuti mumvetsetse ngati pali vuto ndi ma disks kapena ayi. Pali njira ziwiri zoyenera kuchita izi: Katundu Wapakati и IOwait.
Yoyamba ikuwonetsa kuchuluka kwa njira zomwe zili pamzere wogwirira ntchito, nthawi zambiri kuposa 2 - china chake chalakwika kale. Tikamakopera mwachangu ku maseva osunga zobwezeretsera, timalola mpaka 6-8, pambuyo pake zinthu zimawonedwa ngati zachilendo.
Chachiwiri ndi momwe purosesa imagwirira ntchito ndi disk. IOwait> 10% ndi chifukwa chodetsa nkhawa, ngakhale pa ma seva athu omwe ali ndi mbiri yolemetsa nthawi zonse amakhala 40-50%, ndipo izi ndizokhazikika.
Ndimaliza apa, ngakhale pali mfundo zambiri zomwe sitinakumane nazo, ndikhala wokondwa kuyembekezera ndemanga ndi mafotokozedwe amilandu yosangalatsa.
Source: www.habr.com
