Okondedwa Community, Nkhaniyi ifotokoza za kusunga bwino ndi kupeza mazana a mamiliyoni a mafayilo ang'onoang'ono. Pakadali pano, yankho lomaliza likuperekedwa kwa mafayilo ogwirizana ndi POSIX omwe ali ndi chithandizo chokwanira cha maloko, kuphatikiza maloko amagulu, ndipo akuwoneka opanda ndodo.
Chifukwa chake ndidalemba seva yanga yangayo pazifukwa izi.
Pogwira ntchitoyi, tidatha kuthetsa vuto lalikulu, ndipo nthawi yomweyo timapeza ndalama mu disk space ndi RAM, zomwe makina athu amafayilo amawononga mopanda chifundo. Kwenikweni, kuchuluka kwa mafayilo ndikowopsa pamafayilo aliwonse ophatikizika.
Lingaliro ndi ili:
M'mawu osavuta, mafayilo ang'onoang'ono amatsitsidwa kudzera pa seva, amasungidwa mwachindunji muzosungirako, ndikuwerenganso kuchokera pamenepo, ndipo mafayilo akuluakulu amaikidwa mbali ndi mbali. Chiwembu: 1 chikwatu = 1 archive, palimodzi tili ndi mamiliyoni angapo archives omwe ali ndi mafayilo ang'onoang'ono, osati mafayilo mamiliyoni angapo. Ndipo zonsezi zimakwaniritsidwa mokwanira, popanda zolemba kapena kuyika mafayilo muzosungira za tar/zip.
Ndiyesetsa kuzifupikitsa, ndikupepesa pasadakhale ngati positiyo ndi yayitali.
Zonse zinayamba ndi mfundo yakuti sindinapeze seva yoyenera padziko lapansi yomwe ingasunge deta yomwe inalandilidwa kudzera mu protocol ya HTTP mwachindunji muzosungirako, popanda zovuta zomwe zimapezeka muzosungira zakale ndi kusungirako zinthu. Ndipo chifukwa chofufuzira chinali gulu la Origin la ma seva a 10 omwe adakula mpaka kukula kwakukulu, momwe mafayilo ang'onoang'ono a 250,000,000 anali atasonkhanitsa kale, ndipo kukula kwake sikunayime.
Kwa iwo omwe sakonda kuwerenga zolemba, zolemba zochepa ndizosavuta:
ΠΈ .
Ndipo docker nthawi yomweyo, pali njira yokhayo ndi nginx mkati ngati:
docker run -d --restart=always -e host=localhost -e root=/var/storage
-v /var/storage:/var/storage --name wzd -p 80:80 eltaline/wzdYotsatira:
Ngati pali mafayilo ambiri, zofunikira zofunikira zimafunikira, ndipo choyipa kwambiri ndikuti ena amawonongeka. Mwachitsanzo, pogwiritsa ntchito clustered file system (pankhaniyi, MooseFS), fayilo, mosasamala kanthu za kukula kwake, nthawi zonse imatenga pafupifupi 64 KB. Ndiye kuti, pamafayilo a 3, 10 kapena 30 KB kukula, 64 KB amafunikira pa disk. Ngati pali mafayilo kotala biliyoni, timataya 2 mpaka 10 terabytes. Sizingatheke kupanga mafayilo atsopano kwamuyaya, popeza MooseFS ili ndi malire: osapitirira 1 biliyoni ndi chithunzi chimodzi cha fayilo iliyonse.
Pamene chiwerengero cha mafayilo chikuwonjezeka, RAM yochuluka imafunika kuti mudziwe zambiri. Kutaya kwa metadata kwakukulu pafupipafupi kumathandizanso kuti ma drive a SSD awonongeke.
wZD seva. Timayika zinthu mwadongosolo pa disks.
Seva yalembedwa mu Go. Choyamba, ndinafunika kuchepetsa chiwerengero cha mafayilo. Kodi kuchita izo? Chifukwa cha kusungidwa, koma mu nkhani iyi popanda psinjika, popeza owona wanga ndi chabe wothinikizidwa zithunzi. BoltDB idabwera kudzapulumutsa, yomwe idayenerabe kuchotsedwa ku zofooka zake, izi zikuwonetsedwa muzolemba.
Ponseponse, m'malo mwa kotala la mafayilo biliyoni, mwa ine panali zolemba za 10 miliyoni za Bolt zomwe zatsala. Ndikadakhala ndi mwayi wosintha mawonekedwe a fayilo yachikwatu, ndizotheka kuchepetsa mpaka pafupifupi mafayilo 1 miliyoni.
Mafayilo ang'onoang'ono onse amanyamulidwa muzosunga zakale za Bolt, zomwe zimangolandira mayina a zolemba zomwe zilipo, ndipo mafayilo onse akulu amakhalabe pafupi ndi zosungira; palibe chifukwa chowanyamula, izi ndizosintha. Zing'onozing'ono zimasungidwa, zazikulu zimasiyidwa zosasinthika. Seva imagwira ntchito mowonekera ndi onse awiri.
Zomangamanga ndi mawonekedwe a seva ya wZD.
Seva imagwira ntchito pansi pa Linux, BSD, Solaris ndi OSX. Ndinangoyesa zomangamanga za AMD64 pansi pa Linux, koma ziyenera kugwira ntchito kwa ARM64, PPC64, MIPS64.
Zofunikira zazikulu:
- Kuwerenga zambiri;
- Multiserver, kupereka kulolerana kwa zolakwika ndi kusanja katundu;
- Kuwonekera kwakukulu kwa wogwiritsa ntchito kapena wopanga;
- Njira zothandizira HTTP: GET, HEAD, PUT ndi DELETE;
- Kuwongolera machitidwe owerenga ndi kulemba kudzera pamutu wamakasitomala;
- Thandizo kwa osinthika pafupifupi makamu;
- Thandizani kukhulupirika kwa data ya CRC polemba / kuwerenga;
- Semi-dynamic buffers kuti mugwiritse ntchito kukumbukira pang'ono komanso kukonza bwino kwa maukonde;
- Kusasinthika kwa data;
- Kuphatikiza apo, wZA yokhala ndi ulusi wambiri imaperekedwa kuti isamuke mafayilo popanda kuyimitsa ntchitoyo.
Zochitika Zenizeni:
Ndakhala ndikupanga ndikuyesa seva ndi zosunga zobwezeretsera pa data yamoyo kwa nthawi yayitali, tsopano ikugwira ntchito bwino pagulu lomwe limaphatikizapo mafayilo ang'onoang'ono a 250,000,000 (zithunzi) omwe ali m'mawu 15,000,000 pama drive osiyanasiyana a SATA. Gulu la ma seva 10 ndi seva ya Origin yomwe imayikidwa kuseri kwa netiweki ya CDN. Kuti muyitumikire, ma seva a 2 Nginx + 2 wZD ma seva amagwiritsidwa ntchito.
Kwa iwo omwe asankha kugwiritsa ntchito seva iyi, kungakhale kwanzeru kukonzekera dongosolo lachikwatu, ngati kuli kotheka, musanagwiritse ntchito. Ndiroleni ndisungitse nthawi yomweyo kuti seva sinapangidwe kuti iwononge chilichonse mu 1 Bolt archive.
Kuyesa magwiridwe antchito:
Zochepa kukula kwa fayilo ya zip, ntchito za GET ndi PUT zachangu zimachitikira pamenepo. Tiyeni tifanizire nthawi yonse yolemba kasitomala wa HTTP ndi mafayilo wamba ndi zolemba zakale za Bolt, komanso kuwerenga. Gwirani ntchito ndi mafayilo amitundu 32 KB, 256 KB, 1024 KB, 4096 KB ndi 32768 KB ifananizidwa.
Pogwira ntchito ndi Bolt archives, kukhulupirika kwa deta ya fayilo iliyonse kumafufuzidwa (CRC ikugwiritsidwa ntchito), isanayambe kujambula komanso itatha kujambula, kuwerenga ndi kubwerezabwereza kumachitika, izi mwachibadwa zimabweretsa kuchedwa, koma chinthu chachikulu ndi chitetezo cha deta.
Ndidayesa magwiridwe antchito pama drive a SSD, popeza mayeso pama drive a SATA sawonetsa kusiyana koonekeratu.
Zithunzi zotengera zotsatira zoyeserera:
Monga mukuwonera, pamafayilo ang'onoang'ono kusiyana kwa nthawi zowerengera ndi kulemba pakati pa mafayilo osungidwa ndi omwe sanasungidwe ndikochepa.
Timapeza chithunzi chosiyana kwambiri poyesa kuwerenga ndi kulemba mafayilo a 32 MB kukula kwake:
Kusiyana kwa nthawi pakati pa mafayilo owerenga kuli mkati mwa 5-25 ms. Ndi kujambula, zinthu zikuipiraipira, kusiyana kuli pafupifupi 150 ms. Koma pakadali pano palibe chifukwa chokweza mafayilo akulu; palibe chifukwa chochitira izi; amatha kukhala padera ndi zakale.
* Mwaukadaulo, mutha kugwiritsa ntchito seva iyi pazinthu zomwe zimafuna NoSQL.
Njira zoyambira zogwirira ntchito ndi seva ya wZD:
Kutsegula fayilo yokhazikika:
curl -X PUT --data-binary @test.jpg http://localhost/test/test.jpgKukweza fayilo ku Bolt archive (ngati seva parameter fmaxsize, yomwe imatsimikizira kukula kwa fayilo yomwe ingaphatikizidwe muzosungirako, sikudutsa; ngati ipyola, fayilo idzakwezedwa monga mwachizolowezi pafupi ndi zosungirako):
curl -X PUT -H "Archive: 1" --data-binary @test.jpg http://localhost/test/test.jpgKutsitsa fayilo (ngati pali mafayilo omwe ali ndi mayina omwewo pa disk ndi pankhokwe, ndiye mukatsitsa, choyambirira chimaperekedwa mwachisawawa pafayilo yosasungidwa):
curl -o test.jpg http://localhost/test/test.jpgKutsitsa fayilo kuchokera ku Bolt archive (yokakamizidwa):
curl -o test.jpg -H "FromArchive: 1" http://localhost/test/test.jpgMafotokozedwe a njira zina ali muzolemba.
Seva pakadali pano imangogwiritsa ntchito protocol ya HTTP; sikugwirabe ntchito ndi HTTPS pano. Njira ya POST siyothandizidwanso (siyinaganizidwe ngati ikufunika kapena ayi).
Aliyense amene amakumba mu code source adzapeza butterscotch pamenepo, si onse omwe amawakonda, koma sindinamangirire code yayikulu ku ntchito zapaintaneti, kupatula chowongolera chosokoneza, kotero mtsogolomo nditha kulembanso mwachangu pafupifupi chilichonse. injini.
Zochita:
- Kupanga makina anu obwereza ndi ogawa + geo kuti athe kugwiritsidwa ntchito m'makina akuluakulu opanda ma fayilo a cluster (Chilichonse cha akulu)
- Kuthekera kobwezeretsanso metadata ngati itayika kwathunthu (ngati mukugwiritsa ntchito wogawa)
- Native protocol yotha kugwiritsa ntchito ma network osalekeza ndi madalaivala azilankhulo zosiyanasiyana zamapulogalamu
- Mwayi wapamwamba wogwiritsa ntchito gawo la NoSQL
- Kuphatikizika kwamitundu yosiyanasiyana (gzip, zstd, snappy) kwamafayilo kapena zofunikira mkati mwazosunga zakale za Bolt komanso mafayilo okhazikika
- Kubisa kwamitundu yosiyanasiyana kwamafayilo kapena mamtengo mkati mwazosungidwa za Bolt komanso mafayilo okhazikika
- Kuchedwetsa kutembenuka kwamavidiyo a mbali ya seva, kuphatikiza pa GPU
Ndili ndi chirichonse, ndikuyembekeza kuti seva iyi idzakhala yothandiza kwa wina, chilolezo cha BSD-3, umwini wapawiri, popeza ngati panalibe kampani yomwe ndimagwira ntchito, seva sikanalembedwa. Ndine wokonza ndekha. Ndingakhale wothokoza chifukwa cha zovuta zilizonse ndi zopempha zomwe mungapeze.
Source: www.habr.com