Nyob Zoo Zej Zog, Kab lus no yuav tsom mus rau kev khaws cia thiab khaws ntau pua lab ntawm cov ntaub ntawv me me. Nyob rau theem no, qhov kev daws teeb meem kawg yog npaj rau POSIX-tshaj cov ntaub ntawv tshuab nrog kev txhawb nqa tag nrho rau cov xauv, suav nrog cov xauv xauv, thiab zoo li txawm tias tsis muaj cuab yeej cuab tam.
Yog li kuv sau kuv tus kheej kev cai server rau lub hom phiaj no.
Nyob rau hauv cov chav kawm ntawm kev siv txoj hauj lwm no, peb tau tswj los daws qhov teeb meem tseem ceeb, thiab tib lub sij hawm ua tiav kev txuag ntawm qhov chaw disk thiab RAM, uas peb cov ntaub ntawv pawg tau siv tsis muaj kev cia siab. Tiag tiag, xws li ib tug xov tooj ntawm cov ntaub ntawv yog teeb meem rau ib tug clustered ntaub ntawv system.
Lub tswv yim yog qhov no:
Hauv cov lus yooj yooj yim, cov ntaub ntawv me me raug xa tawm los ntawm cov neeg rau zaub mov, lawv tau txais kev cawmdim ncaj qha rau hauv cov ntaub ntawv, thiab tseem nyeem los ntawm nws, thiab cov ntaub ntawv loj tau muab tso rau ib sab. Tswvyim: 1 folder = 1 archive, tag nrho peb muaj ob peb lab archives nrog me me ntaub ntawv, thiab tsis ntau pua lab cov ntaub ntawv. Thiab tag nrho cov no yog siv tag nrho, tsis muaj ntawv sau lossis muab cov ntaub ntawv rau hauv tar / zip archives.
Kuv yuav sim ua kom luv luv, Kuv thov zam txim ua ntej yog tias tus ncej ntev.
Nws tag nrho pib nrog qhov tseeb tias kuv nrhiav tsis tau ib tus neeg rau zaub mov tsim nyog hauv ntiaj teb uas tuaj yeem txuag cov ntaub ntawv tau txais los ntawm HTTP raws tu qauv ncaj qha rau hauv cov ntaub ntawv, tsis muaj qhov tsis zoo nyob hauv cov ntaub ntawv qub thiab cov khoom khaws cia. Thiab yog vim li cas rau kev tshawb fawb yog lub hauv paus pawg ntawm 10 servers uas tau loj hlob mus rau ib qho loj, uas 250,000,000 cov ntaub ntawv me twb tau sau tseg, thiab qhov kev loj hlob tsis tau nres.
Rau cov neeg uas tsis nyiam nyeem cov ntawv, cov ntaub ntawv me me yooj yim dua:
ΠΈ .
Thiab docker tib lub sijhawm, tam sim no muaj kev xaiv tsuas yog nrog nginx sab hauv xwb:
docker run -d --restart=always -e host=localhost -e root=/var/storage
-v /var/storage:/var/storage --name wzd -p 80:80 eltaline/wzdTom ntej:
Yog tias muaj ntau cov ntaub ntawv, cov peev txheej tseem ceeb yog xav tau, thiab qhov phem tshaj plaws yog qee qhov ntawm lawv tau nkim. Piv txwv li, thaum siv cov ntaub ntawv sib koom ua ke (qhov no, MooseFS), cov ntaub ntawv, tsis hais nws qhov loj npaum li cas, ib txwm siv tsawg kawg 64 KB. Ntawd yog, rau cov ntaub ntawv ntawm 3, 10 lossis 30 KB hauv qhov loj, 64 KB yog xav tau ntawm disk. Yog tias muaj ib feem peb ntawm ib lab cov ntaub ntawv, peb poob ntawm 2 mus rau 10 terabytes. Nws yuav tsis tuaj yeem tsim cov ntaub ntawv tshiab tas li, txij li MooseFS muaj qhov txwv: tsis pub ntau tshaj 1 billion nrog ib qho kev hloov pauv ntawm txhua cov ntaub ntawv.
Raws li tus naj npawb ntawm cov ntaub ntawv nce, ntau RAM xav tau rau metadata. Nquag loj metadata dumps kuj ua rau hnav thiab tsim kua muag ntawm SSD drives.
wZD server. Peb muab cov khoom hauv kev txiav txim rau ntawm cov disks.
Lub server tau sau rau hauv Go. Ua ntej ntawm tag nrho cov, kuv yuav tsum tau txo tus naj npawb ntawm cov ntaub ntawv. Yuav ua li cas? Vim yog archiving, tab sis nyob rau hauv cov ntaub ntawv no tsis muaj compression, vim kuv cov ntaub ntawv tsuas yog compressed duab. BoltDB tuaj rau kev cawm, uas tseem yuav tsum tau tshem tawm ntawm nws qhov tsis txaus, qhov no tau cuam tshuam hauv cov ntaub ntawv.
Nyob rau hauv tag nrho, es tsis txhob ntawm ib lub hlis twg ntawm ib tug billion ntaub ntawv, nyob rau hauv kuv cov ntaub ntawv muaj tsuas yog 10 lab Bolt archives sab laug. Yog tias kuv muaj sijhawm los hloov cov ntaub ntawv teev npe tam sim no, nws tuaj yeem txo nws mus rau kwv yees li 1 lab cov ntaub ntawv.
Tag nrho cov ntaub ntawv me tau ntim rau hauv Bolt archives, uas tau txais cov npe ntawm cov npe uas lawv nyob, thiab tag nrho cov ntaub ntawv loj nyob ib sab ntawm cov ntaub ntawv; tsis muaj qhov taw tes rau hauv ntim lawv, qhov no yog customizable. Cov me me tau khaws cia, cov loj loj tseem tsis hloov pauv. Lub server ua haujlwm pob tshab nrog ob qho tib si.
Architecture thiab nta ntawm wZD server.
Lub server ua haujlwm hauv Linux, BSD, Solaris thiab OSX operating systems. Kuv tsuas yog sim rau AMD64 architecture hauv Linux, tab sis nws yuav tsum ua haujlwm rau ARM64, PPC64, MIPS64.
Cov yam ntxwv tseem ceeb:
- Multithreading;
- Multiserver, muab kev zam txim thiab kev sib npaug ntawm kev thauj khoom;
- Qhov siab tshaj plaws pob tshab rau tus neeg siv lossis tus tsim tawm;
- Txhawb HTTP txoj kev: Tau, HEAD, PUT thiab DELETE;
- Tswj kev nyeem ntawv thiab sau tus cwj pwm ntawm tus neeg siv khoom headers;
- Kev them nyiaj yug rau hloov pauv virtual hosts;
- Txhawb CRC cov ntaub ntawv ncaj ncees thaum sau / nyeem ntawv;
- Semi-dynamic buffers rau kev siv lub cim xeeb tsawg thiab kev pom kev ua haujlwm zoo hauv network;
- Deferred cov ntaub ntawv compaction;
- Tsis tas li ntawd, muaj ntau txoj xov archiver wZA muaj rau kev tsiv teb tsaws cov ntaub ntawv yam tsis tau tso tseg cov kev pabcuam.
Kev paub tiag:
Kuv tau tsim thiab sim cov neeg rau zaub mov thiab archiver ntawm cov ntaub ntawv nyob tau ntev heev, tam sim no nws tau ua tiav kev ua haujlwm ntawm pawg uas suav nrog 250,000,000 cov ntaub ntawv me (duab) nyob hauv 15,000,000 phau ntawv qhia ntawm SATA drives cais. Ib pawg ntawm 10 servers yog lub hauv paus server ntsia tom qab CDN network. Txhawm rau pab nws, 2 Nginx servers + 2 wZD servers tau siv.
Rau cov neeg uas txiav txim siab siv cov neeg rau zaub mov no, nws yuav yog qhov zoo los npaj cov qauv ntawm cov npe, yog tias tsim nyog, ua ntej siv. Cia kuv ua ib qho kev tshwj tseg tam sim ntawd tias tus neeg rau zaub mov tsis yog npaj los ua txhua yam rau hauv 1 Bolt archive.
Kev ntsuas kev ua tau zoo:
Qhov me me ntawm cov ntaub ntawv zipped, qhov ceev tau txais thiab PUT ua haujlwm ntawm nws. Cia peb piv tag nrho lub sijhawm rau HTTP tus neeg siv sau ntawv rau cov ntaub ntawv tsis tu ncua thiab Bolt archives, nrog rau kev nyeem ntawv. Ua haujlwm nrog cov ntaub ntawv ntawm qhov ntau thiab tsawg 32 KB, 256 KB, 1024 KB, 4096 KB thiab 32768 KB yog piv.
Thaum ua haujlwm nrog Bolt archives, cov ntaub ntawv kev ncaj ncees ntawm txhua cov ntaub ntawv raug kuaj xyuas (CRC yog siv), ua ntej sau thiab tom qab sau cia, kev nyeem ntawv thiab kev suav rov qab tshwm sim, qhov no ib txwm qhia txog kev qeeb, tab sis qhov tseem ceeb yog cov ntaub ntawv kev ruaj ntseg.
Kuv tau ua qhov kev ntsuam xyuas kev ua tau zoo ntawm SSD drives, vim tias kev sim ntawm SATA drives tsis pom qhov txawv txav.
Graphs raws li kev xeem tau:
Raws li koj tuaj yeem pom, rau cov ntaub ntawv me me qhov sib txawv ntawm kev nyeem thiab sau lub sijhawm ntawm cov ntaub ntawv khaws tseg thiab cov ntaub ntawv tsis yog cov ntaub ntawv me me.
Peb tau txais ib daim duab sib txawv kiag li thaum kuaj nyeem ntawv thiab sau cov ntaub ntawv ntawm 32 MB loj:
Lub sij hawm sib txawv ntawm kev nyeem cov ntaub ntawv yog nyob rau hauv 5-25 ms. Nrog rau kev kaw cia, tej yam tsis zoo, qhov sib txawv yog li 150 ms. Tab sis nyob rau hauv cov ntaub ntawv no tsis tas yuav upload cov ntaub ntawv loj; nws tsuas yog tsis muaj qhov taw tes ua li ntawd; lawv tuaj yeem nyob sib nrug ntawm cov ntaub ntawv.
*Technically, koj tuaj yeem siv cov neeg rau zaub mov no rau cov haujlwm uas xav tau NoSQL.
Cov txheej txheem yooj yim ntawm kev ua haujlwm nrog wZD server:
Loading ib cov ntaub ntawv tsis tu ncua:
curl -X PUT --data-binary @test.jpg http://localhost/test/test.jpgUploading ib cov ntaub ntawv mus rau Bolt archive (yog hais tias tus neeg rau zaub mov parameter fmaxsize, uas txiav txim siab qhov siab tshaj plaws cov ntaub ntawv loj uas muaj peev xwm muab tso rau hauv lub archive, yog tsis tshaj; yog hais tias nws yog tshaj, cov ntaub ntawv yuav tsum uploaded li niaj zaus ib sab ntawm lub archive):
curl -X PUT -H "Archive: 1" --data-binary @test.jpg http://localhost/test/test.jpgDownloading ib cov ntaub ntawv (yog tias muaj cov ntaub ntawv nrog tib lub npe ntawm lub disk thiab hauv archive, ces thaum rub tawm, qhov tseem ceeb yog muab los ntawm lub neej ntawd rau cov ntaub ntawv tsis muaj ntaub ntawv):
curl -o test.jpg http://localhost/test/test.jpgRub tawm cov ntaub ntawv los ntawm Bolt archive (yuav yuam):
curl -o test.jpg -H "FromArchive: 1" http://localhost/test/test.jpgCov lus piav qhia ntawm lwm txoj hauv kev yog nyob rau hauv cov ntaub ntawv.
Lub server tam sim no tsuas yog txhawb HTTP raws tu qauv; nws tsis ua haujlwm nrog HTTPS tseem. Txoj kev POST tseem tsis tau txais kev txhawb nqa (nws tseem tsis tau txiav txim siab seb nws xav tau lossis tsis xav tau).
Leej twg khawb rau hauv qhov chaws yuav pom butterscotch nyob ntawd, tsis yog txhua tus nyiam nws, tab sis kuv tsis tau khi lub ntsiab cai rau kev ua haujlwm ntawm lub vev xaib, tshwj tsis yog rau tus neeg saib xyuas cuam tshuam, yog li yav tom ntej kuv tuaj yeem sau rov qab sai sai rau yuav luag txhua yam. cav.
ToDo:
- Kev tsim kho ntawm koj tus kheej replicator thiab distributor + geo rau kev siv tau hauv cov tshuab loj yam tsis muaj pawg ntaub ntawv (Txhua yam rau cov neeg laus)
- Muaj peev xwm ua tiav rov qab rov qab ntawm cov metadata yog tias nws ploj lawm (yog tias siv tus neeg faib khoom)
- Native raws tu qauv rau lub peev xwm los siv txuas ntxiv network txuas thiab tsav tsheb rau ntau hom lus programming
- Advanced possibilities rau siv NoSQL tivthaiv
- Compressions ntawm ntau hom (gzip, zstd, snappy) rau cov ntaub ntawv lossis qhov tseem ceeb hauv Bolt archives thiab rau cov ntaub ntawv tsis tu ncua
- Encryption ntawm ntau hom rau cov ntaub ntawv lossis cov txiaj ntsig hauv Bolt archives thiab rau cov ntaub ntawv tsis tu ncua
- Kev ncua sijhawm rau kev hloov pauv ntawm server-sab video, suav nrog ntawm GPU
Kuv muaj txhua yam, Kuv vam tias cov neeg rau zaub mov no yuav muaj txiaj ntsig zoo rau ib tus neeg, BSD-3 daim ntawv tso cai, muab ob npaug rau, vim tias tsis muaj lub tuam txhab uas kuv ua haujlwm, server yuav tsis tau sau. Kuv yog tus tsim tawm xwb. Kuv yuav ua tsaug rau txhua yam kab thiab cov lus thov uas koj pom.
Tau qhov twg los: www.hab.com