kev taw qhia
Kuv tau muab tsab ntawv ceeb toom no ua lus Askiv ntawm GopherCon Russia 2019 lub rooj sib tham hauv Moscow thiab hauv Lavxias ntawm kev sib ntsib hauv Nizhny Novgorod. Peb tab tom tham txog qhov ntsuas qhov ntsuas qhov ntsuas - tsawg dua li B-ntoo, tab sis tsis muaj qhov nthuav tsawg. Sib koom hais lus ntawm lub rooj sib tham ua lus Askiv thiab cov ntawv sau ua lus Lavxias.
Peb yuav saib yuav ua li cas ib tug bitmap Performance index ua hauj lwm, thaum nws yog zoo dua, thaum nws yog phem tshaj li lwm yam indexes, thiab nyob rau hauv dab tsi tshwm sim nws yog ho ceev tshaj lawv; Cia peb pom cov DBMSs uas nrov tshaj plaws uas twb muaj qhov ntsuas qhov ntsuas me me; Cia peb sim sau peb tus kheej hauv Go. Thiab "rau cov khoom qab zib" peb yuav siv cov tsev qiv ntawv npaj txhij los tsim peb tus kheej super-ceev tshwj xeeb database.
Kuv vam tias kuv cov haujlwm yuav muaj txiaj ntsig thiab nthuav rau koj. Mus!
Taw qhia
Nyob zoo sawv daws! Nws yog rau thaum yav tsaus ntuj thiab peb txhua tus nkees heev. Lub sijhawm zoo los tham txog kev tho txawv database index theory, txoj cai? Tsis txhob txhawj, Kuv yuav muaj ob peb kab ntawm qhov chaws ntawm no thiab muaj. π
Txhua qhov kev tso dag tso tseg, daim ntawv tshaj tawm yog chock-tag nrho cov ntaub ntawv, thiab peb tsis muaj sijhawm ntau. Yog li cia peb pib.
Hnub no kuv yuav tham txog cov hauv qab no:
- dab tsi yog indexes;
- bitmap index yog dab tsi;
- qhov twg nws siv thiab qhov twg tsis siv thiab vim li cas;
- yooj yim siv nyob rau hauv Go thiab me ntsis tawm tsam nrog lub compiler;
- yooj yim me ntsis, tab sis ua haujlwm ntau dua hauv Go assembler;
- "teeb ββmeem" ntawm bitmap indexes;
- cov kev siv uas twb muaj lawm.
Yog li dab tsi yog indexes?
Qhov Performance index yog cov ntaub ntawv sib cais uas peb tswj hwm thiab hloov kho ntxiv rau cov ntaub ntawv tseem ceeb. Nws yog siv los ua kom ceev qhov kev tshawb nrhiav. Yog tsis muaj indexes, searching yuav tsum tau mus los ntawm cov ntaub ntawv tag nrho (ib txoj kev hu ua tag nrho scan), thiab cov txheej txheem no muaj linear algorithmic complexity. Tab sis cov databases feem ntau muaj cov ntaub ntawv loj heev thiab linear complexity qeeb dhau. Qhov zoo tshaj plaws, peb yuav tau txais ib lub logarithmic lossis qhov tsis tu ncua.
Qhov no yog ib lub ntsiab lus nyuaj heev, tau sau nrog subtleties thiab kev lag luam tawm, tab sis tom qab saib ntau xyoo ntawm kev txhim kho database thiab kev tshawb fawb, kuv txaus siab hais tias tsuas muaj ob peb txoj hauv kev los tsim cov database indexes.
Thawj txoj hauv kev yog hierarchically txo qhov chaw tshawb nrhiav, faib qhov chaw tshawb nrhiav rau hauv me me.
Peb feem ntau ua qhov no siv ntau hom ntoo. Ib qho piv txwv yuav yog lub thawv loj ntawm cov ntaub ntawv hauv koj lub txee dai khaub ncaws uas muaj cov thawv me me ntawm cov ntaub ntawv muab faib ua ntau yam. Yog tias koj xav tau cov ntaub ntawv, koj yuav zaum nrhiav lawv hauv lub thawv uas hais tias "Cov Khoom" es tsis yog ib qho uas hais tias "Cookies," puas yog?
Qhov thib ob mus kom ze yog tam sim ntawd xaiv cov khoom xav tau lossis pab pawg ntawm cov ntsiab lus. Peb ua qhov no hauv hash maps lossis rov qab indexes. Siv daim ntawv qhia hash zoo ib yam li cov piv txwv yav dhau los, tab sis tsis yog lub thawv ntawm lub thawv, koj muaj ib pawg me me ntawm cov khoom kawg hauv koj lub txee dai khaub ncaws.
Qhov thib peb mus kom ze yog tshem tawm qhov xav tau kev tshawb nrhiav. Peb ua qhov no siv Bloom lim lossis cuckoo lim. Thawj tus muab lus teb tam sim ntawd, txuag koj los ntawm kev tshawb nrhiav.
Qhov kawg mus kom ze yog siv tag nrho cov hwj chim uas niaj hnub kho vajtse muab rau peb. Qhov no yog raws nraim li peb ua hauv bitmap indexes. Yog lawm, thaum siv lawv qee zaum peb yuav tsum mus dhau tag nrho qhov ntsuas, tab sis peb ua nws zoo heev.
Raws li kuv tau hais, lub ntsiab lus ntawm database indexes yog qhov loj thiab tag nrho ntawm kev cuam tshuam. Qhov no txhais tau hais tias qee zaum peb tuaj yeem siv ntau txoj hauv kev tib lub sijhawm: yog tias peb xav tau kev tshawb nrhiav nrawm dua, lossis yog tias peb xav tau los npog txhua hom kev tshawb nrhiav.
Hnub no kuv yuav tham txog qhov kev paub tsawg tshaj plaws ntawm cov no - bitmap indexes.
Kuv yog leej twg los tham txog lub ncauj lus no?
Kuv ua haujlwm ua ib pab neeg ua haujlwm ntawm Badoo (tej zaum koj paub ntau dua nrog peb lwm yam khoom, Bumble). Peb twb muaj ntau dua 400 lab cov neeg siv thoob ntiaj teb thiab ntau yam nta uas xaiv qhov zoo tshaj plaws match rau lawv. Peb ua qhov no siv cov kev pabcuam kev cai, suav nrog bitmap indexes.
Yog li dab tsi yog qhov bitmap index?
Bitmap indexes, raws li lub npe qhia, siv bitmap lossis bitsets los siv qhov kev tshawb fawb index. Los ntawm tus noog lub qhov muag pom, qhov ntsuas no muaj ib lossis ntau dua li cov duab kos uas sawv cev rau txhua qhov chaw (xws li tib neeg) thiab lawv cov khoom lossis tsis muaj (hnub nyoog, qhov muag xim, thiab lwm yam), thiab ib qho algorithm siv kev ua haujlwm me ntsis (AND, LOSSIS, TSIS TAU. ) los teb cov lus nug.
Peb tau hais tias qhov ntsuas qhov ntsuas tau zoo tshaj plaws thiab ua tau zoo heev rau cov xwm txheej uas muaj cov kev tshawb fawb sib txuas cov lus nug hla ntau cov kab lus qis qis (xav tias "qhov muag xim" lossis "kev ua txij ua nkawm" piv rau qee yam xws li "qhov deb ntawm lub nroog"). Tab sis kuv mam li qhia tom qab tias lawv ua haujlwm zoo rau cov kab siab cardinality ib yam nkaus.
Cia peb saib qhov piv txwv yooj yim ntawm qhov ntsuas qhov ntsuas qhov ntsuas.
Xav txog tias peb muaj cov npe ntawm Moscow cov khw noj mov nrog binary khoom zoo li no:
- nyob ze metro;
- muaj chaw nres tsheb ntiag tug;
- muaj ib tug veranda (muaj terrace);
- koj tuaj yeem khaws ib lub rooj ( lees txais kev tshwj tseg);
- haum rau cov neeg tsis noj nqaij (vegan phooj ywg);
- kim (keem).
Cia peb muab txhua lub tsev noj mov ib ntu ntu pib ntawm 0 thiab faib lub cim xeeb rau 6 bitmaps (ib qho rau txhua tus yam ntxwv). Peb mam li populate cov bitmaps nyob ntawm seb lub tsev noj mov muaj cov cuab yeej no los tsis. Yog tias lub tsev noj mov 4 muaj veranda, ces me ntsis No. 4 hauv "muaj veranda" bitmap yuav raug teem rau 1 (yog tias tsis muaj veranda, ces mus rau 0).
Tam sim no peb muaj qhov yooj yim bitmap Performance index ua tau, thiab peb tuaj yeem siv nws los teb cov lus nug xws li:
- "Qhia kuv cov khw noj zaub mov zoo";
- "Qhia kuv cov khw noj mov pheej yig nrog veranda qhov twg koj tuaj yeem khaws lub rooj."
Yuav ua li cas? Cia peb saib. Thawj qhov kev thov yog qhov yooj yim heev. Txhua yam peb yuav tsum tau ua yog coj cov "neeg tsis noj nqaij tus phooj ywg" bitmap thiab tig nws mus rau hauv cov npe ntawm cov khw noj mov uas cov khoom raug nthuav tawm.
Qhov kev thov thib ob yog qhov nyuaj me ntsis. Peb yuav tsum siv NOT bitmap ntawm "kim" bitmap kom tau txais ib daim ntawv teev cov khw noj mov pheej yig, tom qab ntawd THIAB nws nrog "Kuv tuaj yeem book ib lub rooj" bitmap thiab THIAB qhov tshwm sim nrog "muaj veranda" bitmap. Cov txiaj ntsig bitmap yuav muaj cov npe ntawm cov chaw tsim khoom uas ua tau raws li tag nrho peb cov txheej txheem. Hauv qhov piv txwv no, qhov no tsuas yog lub tsev noj mov Yunost xwb.
Muaj ntau txoj kev xav koom nrog, tab sis tsis txhob txhawj, peb yuav pom cov cai sai sai.
Bitmap indexes siv qhov twg?
Yog tias koj Google bitmap indexes, 90% ntawm cov lus teb yuav cuam tshuam nrog Oracle DB hauv ib txoj kev lossis lwm qhov. Tab sis lwm yam DBMSs tej zaum kuj txhawb qhov txias, txoj cai? Tsis yog tiag.
Cia peb mus dhau cov npe ntawm cov neeg raug liam loj.
MySQL tseem tsis tau txhawb nqa qhov ntsuas qhov ntsuas me me, tab sis muaj Kev Pom Zoo qhia ntxiv qhov kev xaiv no ().
PostgreSQL tsis txhawb nqa qhov ntsuas qhov ntsuas me me, tab sis siv cov duab yooj yim thiab cov haujlwm me me los muab cov txiaj ntsig tshawb fawb hla ntau lwm qhov ntsuas.
Tarantool muaj qhov ntsuas ntsuas me ntsis thiab txhawb kev tshawb nrhiav yooj yim rau lawv.
Redis muaj cov bitfields yooj yim) tsis muaj peev xwm nrhiav tau lawv.
MongoDB tseem tsis tau txhawb nqa qhov ntsuas ntsuas qhov ntsuas, tab sis kuj tseem muaj Kev Pom Zoo qhia tias qhov kev xaiv no ntxiv
Elasticsearch siv bitmap sab hauv).
- Tab sis ib tug neeg nyob ze tshiab tau tshwm sim hauv peb lub tsev: Pilosa. Nov yog cov ntaub ntawv tshiab uas tsis muaj feem cuam tshuam nrog sau hauv Go. Nws tsuas muaj bitmap indexes thiab ua txhua yam ntawm lawv. Peb mam li tham txog nws me ntsis tom qab.
Kev siv hauv Go
Tab sis vim li cas bitmap indexes tsis tshua siv? Ua ntej teb cov lus nug no, kuv xav qhia koj yuav ua li cas siv qhov yooj yim bitmap Performance index hauv Go.
Bitmaps yog qhov tseem ceeb tsuas yog cov ntaub ntawv. Hauv Go, cia peb siv byte hlais rau qhov no.
Peb muaj ib qho bitmap rau ib lub tsev noj mov yam ntxwv, thiab txhua qhov me me hauv bitmap qhia seb lub tsev noj mov tshwj xeeb puas muaj cov khoom no lossis tsis.
Peb yuav xav tau ob txoj haujlwm pabcuam. Ib qho yuav raug siv los sau peb cov duab kos nrog cov ntaub ntawv random. Random, tab sis nrog qee qhov tshwm sim uas lub tsev noj mov muaj txhua yam khoom. Piv txwv li, kuv ntseeg hais tias muaj tsawg heev lub tsev noj mov nyob rau hauv Moscow qhov twg koj yuav tsis tau teem ib lub rooj, thiab nws zoo nkaus li kuv hais tias txog 20% ββntawm cov chaw tsim nyog rau cov neeg tsis noj nqaij.
Qhov thib ob muaj nuj nqi yuav hloov lub bitmap mus rau hauv ib daim ntawv teev cov khw nojmov.
Txhawm rau teb cov lus nug "Qhia kuv cov khw noj mov pheej yig uas muaj lub patio thiab tuaj yeem ua kev tshwj tseg," peb xav tau ob txoj haujlwm me me: TSIS TAU thiab THIAB.
Peb tuaj yeem ua kom yooj yim peb cov cai me ntsis los ntawm kev siv cov neeg siv ntau dua THIAB TSIS TAU.
Peb muaj lub luag haujlwm rau txhua qhov haujlwm no. Ob leeg mus dhau lub hlais, nqa cov ntsiab lus sib thooj los ntawm txhua tus, muab lawv nrog kev ua haujlwm me ntsis thiab muab tso rau hauv qhov tshwm sim.
Thiab tam sim no peb tuaj yeem siv peb cov bitmaps thiab cov haujlwm los teb cov lus nug tshawb.
Kev ua tau zoo tsis yog qhov siab, txawm hais tias cov haujlwm yooj yim heev thiab peb tau txais nyiaj ntau los ntawm kev tsis rov qab los ntawm qhov tshwm sim tshiab txhua zaus lub luag haujlwm hu ua.
Tom qab ua me ntsis ntawm profiling nrog pprof, kuv pom tias Go compiler ploj lawm ib qho yooj yim heev tab sis tseem ceeb heev optimization: muaj nuj nqi inlining.
Qhov tseeb yog tias Go compiler ntshai heev ntawm cov loops uas mus hla cov hlais, thiab categorically tsis kam rau inline functions uas muaj xws li loops.
Tab sis kuv tsis ntshai thiab kuv tuaj yeem dag lub compiler los ntawm kev siv goto es tsis txhob lub voj, zoo li nyob rau hauv cov hnub qub zoo.
Thiab, raws li koj tuaj yeem pom, tam sim no lub compiler yuav zoo siab rau hauv peb txoj haujlwm! Yog li ntawd, peb tswj kom txuag tau li 2 microseconds. Tsis phem!
Qhov thib ob bottleneck yog ib qho yooj yim kom pom yog tias koj saib ze ntawm cov khoom sib dhos. Lub compiler tau ntxiv ib daim ntawv hla tebchaws nyob rau hauv peb lub voj kub kub. Qhov tseeb yog tias Go yog cov lus muaj kev nyab xeeb, lub compiler ntshai tias kuv cov lus sib cav (peb daim) yog qhov sib txawv. Tom qab tag nrho, ces yuav muaj ib tug theoretical muaj peev xwm ntawm qhov tshwm sim ntawm ib tug thiaj li hu ua buffer overflow.
Cia peb rov hais dua lub compiler los ntawm kev qhia nws tias tag nrho cov hlais yog tib qhov loj. Peb tuaj yeem ua qhov no los ntawm kev ntxiv ib daim tshev yooj yim thaum pib ntawm peb txoj haujlwm.
Pom qhov no, tus sau tau zoo siab hla daim tshev, thiab peb xaus rau kev txuag lwm 500 nanoseconds.
Loj loj
Okay, peb tau tswj kom nyem qee qhov kev ua tau zoo ntawm peb qhov kev siv yooj yim, tab sis qhov tshwm sim no yog qhov phem tshaj qhov ua tau nrog cov khoom siv tam sim no.
Txhua yam peb ua yog kev ua haujlwm me ntsis, thiab peb cov processors ua tau zoo heev. Tab sis, hmoov tsis, peb "noj" peb lub processor nrog me me ntawm kev ua haujlwm. Peb cov haujlwm ua haujlwm ua haujlwm raws li byte-by-byte. Peb tuaj yeem yooj yim tweak peb cov lej ua haujlwm nrog 8-byte chunks siv UInt64 hlais.
Raws li koj tuaj yeem pom, qhov kev hloov me me no tau ua rau peb qhov kev pab cuam yim zaug los ntawm kev nce cov batch loj los ntawm yim zaug. Qhov nce tuaj yeem hais tias yog linear.
Implementation nyob rau hauv assembler
Tab sis qhov no tsis yog qhov kawg. Peb cov processors tuaj yeem ua haujlwm nrog chunks ntawm 16, 32 thiab txawm 64 bytes. Cov kev ua haujlwm "dav" no yog hu ua ib qho kev qhia ntau cov ntaub ntawv (SIMD; ib qho kev qhia, ntau cov ntaub ntawv), thiab cov txheej txheem ntawm kev hloov cov lej kom nws siv cov haujlwm no hu ua vectorization.
Hmoov tsis zoo, Go compiler nyob deb ntawm qhov zoo ntawm vectorization. Tam sim no, tib txoj hauv kev los vectorize Go code yog coj thiab tso cov haujlwm no manually siv Go assembler.
Mus assembler yog ib tug tsiaj nyaum coj txawv txawv. Tej zaum koj yuav paub tias cov lus sib dhos yog ib yam uas hnyav heev rau cov qauv ntawm lub khoos phis tawj koj sau rau, tab sis qhov ntawd tsis yog hauv Go. Go assembler yog zoo li IRL (intermediate sawv cev lus) los yog lus nruab nrab: nws yog suab platform ywj siab. Rob Pike tau ua haujlwm zoo heev ntawm cov ncauj lus no ntau xyoo dhau los ntawm GopherCon hauv Denver.
Tsis tas li ntawd, Go siv ib qho txawv txawv Plan 9 hom, uas txawv ntawm feem ntau lees txais AT&T thiab Intel hom.
Nws muaj kev nyab xeeb hais tias kev sau Go assembler los ntawm tes tsis yog qhov kev lom zem tshaj plaws.
Tab sis, hmoov zoo, twb muaj ob lub cuab yeej qib siab uas pab peb sau Go assembler: PeachPy thiab avo. Ob qhov kev siv hluav taws xob tsim Go assembler los ntawm qib siab sau code sau hauv Python thiab Go, raws li.
Cov khoom siv hluav taws xob no ua kom yooj yim rau tej yam xws li sau npe faib, sau cov voj voog, thiab feem ntau ua kom yooj yim cov txheej txheem nkag mus rau hauv lub ntiaj teb ntawm kev sib sau ua ke hauv Go.
Peb yuav siv avo, yog li peb cov kev pab cuam yuav luag txhua yam Go program.
Nov yog qhov piv txwv yooj yim tshaj plaws ntawm avo program zoo li. Peb muaj lub ntsiab () ua haujlwm, uas txhais tau hais tias nyob rau hauv nws tus kheej lub Add() muaj nuj nqi, lub ntsiab lus yog ntxiv ob tug lej. Muaj cov pab cuam ua haujlwm ntawm no kom tau txais cov kev txwv los ntawm lub npe thiab tau txais ib qho ntawm cov ntawv sau npe pub dawb thiab tsim nyog. Txhua qhov kev ua haujlwm processor muaj qhov sib txuam ntawm avo, raws li pom hauv ADDQ. Thaum kawg, peb pom tus pab cuam ua haujlwm rau khaws cov txiaj ntsig tshwm sim.
Los ntawm kev hu mus tsim, peb yuav ua tiav qhov kev pab cuam ntawm avo thiab vim li ntawd, ob cov ntaub ntawv yuav raug tsim tawm:
- add.s nrog cov txiaj ntsig code hauv Go assembler;
- stub.go nrog kev ua haujlwm headers los txuas ob lub ntiaj teb: Mus thiab assembler.
Tam sim no peb tau pom dab tsi avo ua thiab yuav ua li cas, cia saib peb cov haujlwm. Kuv tau siv ob qho tib si scalar thiab vector (SIMD) versions ntawm cov haujlwm.
Cia peb saib cov scalar versions ua ntej.
Ib yam li hauv qhov piv txwv dhau los, peb tab tom thov rau kev sau npe dawb thiab siv tau, peb tsis tas yuav suav cov offsets thiab qhov ntau thiab tsawg rau cov lus sib cav. avo ua txhua yam no rau peb.
Peb tau siv cov ntawv sau thiab goto (lossis dhia) txhawm rau txhim kho kev ua haujlwm thiab ua kom yuam kev Go compiler, tab sis tam sim no peb tab tom ua txij thaum pib. Lub ntsiab lus yog tias cycles yog lub tswvyim siab dua. Hauv kev sib dhos, peb tsuas muaj cov ntawv sau thiab dhia.
Cov cai ntxiv yuav tsum tau paub thiab nkag siab. Peb emulate ib lub voj nrog cov ntawv cim thiab dhia, nqa ib daim me me ntawm cov ntaub ntawv los ntawm peb ob lub hlais, muab lawv ua haujlwm me ntsis (TSIS TSIS yog qhov no) thiab tom qab ntawd muab cov txiaj ntsig tso rau hauv cov txiaj ntsig tau. Tag nrho.
Qhov no yog qhov kawg assembler code zoo li. Peb tsis tas yuav suav cov offsets thiab qhov ntau thiab tsawg (highlighted ntsuab) los yog khaws cov ntawv sau npe siv (highlighted red).
Yog tias peb sib piv qhov kev ua tau zoo ntawm kev siv cov lus sib dhos nrog kev ua tau zoo ntawm kev ua haujlwm zoo tshaj plaws hauv Go, peb yuav pom tias nws yog tib yam. Thiab qhov no xav tau. Tom qab tag nrho, peb tsis tau ua dab tsi tshwj xeeb - peb tsuas yog rov tsim dua qhov Go compiler yuav ua.
Hmoov tsis zoo, peb tsis tuaj yeem yuam kom lub compiler inline peb lub luag haujlwm sau ua lus sib dhos. Lub Go compiler tam sim no tsis muaj qhov zoo li no, txawm hais tias tau thov kom ntxiv nws rau qee lub sijhawm.
Qhov no yog vim li cas nws tsis tuaj yeem tau txais txiaj ntsig los ntawm cov haujlwm me me hauv cov lus sib dhos. Peb yuav tsum sau cov haujlwm loj, lossis siv cov lej / cov khoom tshiab, lossis hla cov lus sib dhos.
Tam sim no cia peb saib cov vector versions ntawm peb cov haujlwm.
Rau qhov piv txwv no, kuv txiav txim siab siv AVX2, yog li peb yuav siv cov haujlwm uas ua haujlwm ntawm 32-byte chunks. Cov qauv ntawm cov cai yog heev zoo ib yam li scalar version: loading tsis, thov rau ib tug dawb sib koom sau npe, thiab lwm yam.
Ib qho kev tsim kho tshiab yog tias kev ua haujlwm dav dav siv cov npe dav dav tshwj xeeb. Nyob rau hauv rooj plaub ntawm 32-byte chunks, cov no yog cov npe ua ntej nrog Y. Qhov no yog vim li cas koj pom YMM() muaj nuj nqi hauv cov cai. Yog tias kuv tau siv AVX-512 nrog 64-ntsis chunks, cov lus ua ntej yuav yog Z.
Qhov kev hloov tshiab thib ob yog tias kuv tau txiav txim siab siv qhov kev ua kom zoo dua hu ua voj unrolling, uas txhais tau tias ua yim lub voj voog ua haujlwm manually ua ntej dhia mus rau qhov pib ntawm lub voj. Qhov kev ua kom zoo no txo ββqis cov ceg ntoo hauv cov lej, thiab tsuas yog txwv los ntawm cov npe pub dawb muaj.
Zoo, ua li cas txog kev ua haujlwm? Nws zoo nkauj! Peb ua tiav qhov nrawm ntawm xya zaus piv rau qhov zoo tshaj plaws Go daws. Txaus ntshai, txoj cai?
Tab sis txawm tias qhov kev siv no tuaj yeem ua kom nrawm los ntawm kev siv AVX-512, prefetching lossis JIT (tsuas yog-hauv-lub sijhawm compiler) rau cov lus nug teem sijhawm. Tab sis qhov no yeej yog lub ntsiab lus rau ib daim ntawv qhia cais.
Teeb meem nrog bitmap indexes
Tam sim no peb twb tau saib qhov kev siv yooj yim ntawm bitmap Performance index hauv Go thiab ua haujlwm ntau dua hauv cov lus sib dhos, peb thaum kawg tham txog vim li cas cov bitmap indexes tsis tshua siv.
Cov ntawv qub hais txog peb qhov teeb meem nrog qhov ntsuas qhov ntsuas me me, tab sis cov ntawv tshiab thiab kuv sib cav tias lawv tsis cuam tshuam. Peb yuav tsis nkag siab tob rau hauv txhua yam ntawm cov teeb meem no, tab sis yuav saib lawv superficially.
Qhov teeb meem ntawm siab cardinality
Yog li, peb tau hais tias bitmap indexes tsuas yog haum rau cov teb uas tsis tshua muaj cardinality, uas yog, cov uas muaj ob peb yam tseem ceeb (piv txwv li, poj niam los txiv neej los yog qhov muag xim), thiab yog vim li cas yog qhov ib txwm sawv cev ntawm xws li teb (ib tug. ntsis ib tus nqi) nyob rau hauv cov ntaub ntawv ntawm siab cardinality, nws yuav siv sij hawm ntau qhov chaw thiab, ntxiv rau, cov bitmap indexes yuav tsis zoo (tsis tshua muaj) sau.
Qee zaum peb yuav siv tus sawv cev sib txawv, xws li tus qauv peb siv los sawv cev tus lej. Tab sis nws yog lub advent ntawm compression algorithms uas hloov txhua yam. Tau ntau xyoo dhau los, cov kws tshawb fawb thiab cov kws tshawb fawb tau tuaj nrog ntau tus compression algorithms rau bitmap. Lawv lub ntsiab kom zoo dua yog tias tsis tas yuav decompress bitmaps los ua haujlwm me ntsis - peb tuaj yeem ua haujlwm me ntsis ncaj qha ntawm compressed bitmap.
Tsis ntev los no, hybrid mus kom ze tau pib tshwm, xws li roaring bitmap. Lawv ib txhij siv peb qhov sib txawv sawv cev rau bitmaps - bitmaps lawv tus kheej, arrays thiab thiaj li hu ua me ntsis khiav - thiab sib npaug ntawm lawv kom ua tau zoo tshaj plaws thiab txo qis kev nco.
Koj tuaj yeem pom roaring bitmap hauv cov ntawv thov nrov tshaj plaws. Muaj twb muaj ntau qhov kev siv rau ntau hom lus programming, suav nrog ntau tshaj peb qhov kev siv rau Go.
Lwm txoj hauv kev uas tuaj yeem pab peb daws qhov siab cardinality yog hu ua binning. Xav txog tias koj muaj ib daim teb uas sawv cev rau tus neeg qhov siab. Qhov siab yog tus lej taw tes ntab, tab sis peb tib neeg tsis xav txog qhov ntawd. Rau peb tsis muaj qhov sib txawv ntawm qhov siab 185,2 cm thiab 185,3 cm.
Nws hloov tawm tias peb tuaj yeem pab pawg zoo sib xws rau hauv pawg hauv 1 cm.
Thiab yog tias peb tseem paub tias tsawg tus neeg luv dua 50 cm thiab siab dua 250 cm, ces peb tuaj yeem hloov qhov chaw uas tsis muaj qhov kawg rau hauv daim teb nrog cardinality txog 200 qhov tseem ceeb.
Tau kawg, yog tias tsim nyog, peb tuaj yeem ua qhov lim dej ntxiv tom qab ntawd.
Teeb meem Bandwidth siab
Qhov teeb meem tom ntej nrog bitmap indexes yog qhov hloov kho lawv tuaj yeem kim heev.
Databases yuav tsum tuaj yeem hloov kho cov ntaub ntawv thaum muaj peev xwm ntau pua lwm cov lus nug tab tom tshawb nrhiav cov ntaub ntawv. Peb xav tau cov xauv kom tsis txhob muaj teeb meem nrog kev nkag mus rau cov ntaub ntawv los yog lwm yam teeb meem sib koom. Thiab qhov twg muaj ib lub xauv loj, muaj ib qho teeb meem - kev sib cav sib cav, thaum lub xauv no dhau los ua ib lub raj mis.
Qhov teeb meem no tuaj yeem daws tau lossis dhau los ntawm kev siv sharding lossis siv cov ntawv ntsuas ntsuas.
Sharding yog ib qho yooj yim thiab paub zoo. Koj muaj peev xwm shard ib bitmap Performance index raws li koj xav tau lwm yam ntaub ntawv. Hloov chaw ntawm ib lub xauv loj, koj yuav tau txais ib pawg ntawm cov xauv me me thiab yog li tshem tawm kev sib cav sib ceg.
Qhov thib ob txoj kev los daws qhov teeb meem yog siv versioned indexes. Koj tuaj yeem muaj ib daim qauv ntawm qhov ntsuas uas koj siv rau kev tshawb nrhiav lossis nyeem ntawv, thiab ib qho uas koj siv los sau lossis hloov kho. Thiab ib zaug hauv ib lub sijhawm (piv txwv li, ib zaug txhua 100 ms lossis 500 ms) koj duplicate lawv thiab sib pauv lawv. Tau kawg, txoj hauv kev no tsuas yog siv tau nyob rau hauv rooj plaub uas koj daim ntawv thov tuaj yeem tswj xyuas qhov kev tshawb fawb me ntsis.
Ob txoj hauv kev no tuaj yeem siv ib txhij: koj tuaj yeem muaj qhov sib txawv ntawm qhov ntsuas ntsuas.
Ntau cov lus nug nyuaj
Qhov teeb meem kawg nrog bitmap indexes yog tias peb tau hais tias lawv tsis haum rau ntau hom lus nug, xws li cov lus nug span.
Tseeb tiag, yog tias koj xav txog nws, kev ua haujlwm me ntsis xws li THIAB, LOSSIS, thiab lwm yam tsis haum rau cov lus nug a la "Qhia kuv cov tsev so nrog chav nqi ntawm 200 txog 300 las ib hmo."
Ib qho kev daws teeb meem tsis zoo thiab tsis zoo yuav yog coj cov txiaj ntsig rau txhua tus nqi duas thiab muab lawv nrog kev ua haujlwm me ntsis LOSSIS.
Ib qho kev daws teeb meem zoo me ntsis yuav yog siv kev ua pab pawg. Piv txwv li, hauv pab pawg ntawm 50 las. Qhov no yuav ua kom peb cov txheej txheem los ntawm 50 npaug.
Tab sis qhov teeb meem kuj tau yooj yim daws tau los ntawm kev siv qhov kev pom tsim tshwj xeeb rau hom kev thov no. Hauv cov ntaub ntawv tshawb fawb nws yog hu ua range-encoded bitmaps.
Hauv qhov kev sawv cev no, peb tsis yog teem ib me ntsis rau qee tus nqi (piv txwv li, 200), tab sis teeb tsa tus nqi no thiab txhua yam siab dua. 200 thiab siab dua. Tib yam rau 300: 300 thiab siab dua. Thiab li ntawd.
Siv qhov sawv cev no, peb tuaj yeem teb cov lus nug zoo li no los ntawm kev hla qhov ntsuas ob zaug xwb. Ua ntej, peb yuav tau txais ib daim ntawv teev cov tsev so uas chav nqi tsawg dua los yog $ 300, thiab tom qab ntawd peb yuav tshem tawm ntawm cov chaw uas chav tsev nqi tsawg dua lossis $ 199. Npaj txhij.
Koj yuav xav tsis thoob, tab sis txawm tias geoqueries tuaj yeem siv bitmap indexes. Qhov ua kom yuam kev yog siv tus sawv cev geometric uas nyob ib puag ncig koj qhov kev sib koom tes nrog daim duab geometric. Piv txwv li, S2 los ntawm Google. Daim duab yuav tsum muaj peev xwm sawv cev rau hauv daim ntawv ntawm peb lossis ntau qhov sib tshuam kab uas tuaj yeem suav tau. Txoj kev no peb tuaj yeem tig peb cov geoquery mus rau ntau cov lus nug "raws qhov sib txawv" (raws li cov lej no).
Npaj daws teeb meem
Kuv vam tias kuv txaus siab rau koj me ntsis thiab tam sim no koj muaj lwm yam cuab yeej tseem ceeb hauv koj lub arsenal. Yog tias koj xav tau ua ib yam dab tsi zoo li no, koj yuav paub tias yuav saib qhov twg.
Txawm li cas los xij, tsis yog txhua tus muaj lub sijhawm, kev ua siab ntev, lossis cov peev txheej los tsim cov kev ntsuas bitmap los ntawm kos. Tshwj xeeb tshaj yog cov tshaj tawm, siv SIMD, piv txwv li.
Hmoov zoo, muaj ob peb txoj kev npaj ua tiav los pab koj.
Roaring bitmap
Ua ntej, muaj tib lub suab nrov bitmaps tsev qiv ntawv uas kuv twb tham txog. Nws muaj tag nrho cov ntim tsim nyog thiab kev ua haujlwm me ntsis uas koj yuav tsum tau ua kom tiav qhov ntsuas qhov ntsuas bitmap.
Hmoov tsis zoo, tam sim no, tsis muaj qhov kev siv Go siv SIMD, uas txhais tau hais tias kev siv Go tsis ua haujlwm tsawg dua li C siv, piv txwv.
Pilosa
Lwm yam khoom uas tuaj yeem pab koj yog Pilosa DBMS, uas, qhov tseeb, tsuas yog muaj qhov ntsuas qhov ntsuas me ntsis. Qhov no yog ib qho kev daws teeb meem tshiab, tab sis nws yeej lub siab lub siab ntawm kev ceev ceev.
Pilosa siv roaring bitmaps sab hauv thiab muab koj lub peev xwm los siv lawv, yooj yim thiab piav qhia txhua yam uas kuv tau tham txog saum toj no: pab pawg, ntau-encoded bitmaps, lub tswv yim ntawm thaj teb, thiab lwm yam.
Cia wb mus saib ib qho piv txwv ntawm kev siv Pilosa los teb cov lus nug uas koj twb paub lawm.
Qhov piv txwv zoo ib yam li qhov koj pom ua ntej. Peb tsim ib tus neeg siv khoom rau Pilosa server, tsim qhov ntsuas thiab cov teb tsim nyog, tom qab ntawd sau peb cov ntaub ntawv nrog cov ntaub ntawv random nrog qhov tshwm sim thiab, thaum kawg, ua tiav cov lus nug paub.
Tom qab ntawd, peb siv TSIS TXAUS SIAB rau ntawm "khoom kim" teb, tom qab ntawd hla qhov tshwm sim (los yog THIAB nws) nrog rau "terrace" teb thiab nrog "kev tshwj tseg" teb. Thiab thaum kawg, peb tau txais qhov kawg tshwm sim.
Kuv yeej cia siab tias yav tom ntej no hom tshiab ntawm qhov ntsuas no tseem yuav tshwm sim hauv DBMSs zoo li MySQL thiab PostgreSQL - bitmap indexes.
xaus
Yog koj tseem tsis tau tsaug zog, ua tsaug. Kuv yuav tsum tau luv luv kov ntawm ntau lub ntsiab lus vim lub sijhawm txwv, tab sis kuv vam tias qhov kev hais lus muaj txiaj ntsig thiab tej zaum kuj txhawb nqa.
Bitmap indexes yog qhov zoo kom paub txog, txawm tias koj tsis xav tau tam sim no. Cia lawv ua lwm yam cuab yeej hauv koj lub cuab yeej.
Peb tau saib ntau yam kev ua tau zoo rau Go thiab tej yam uas Go compiler ua tsis tau zoo heev. Tab sis qhov no yog qhov muaj txiaj ntsig zoo rau txhua tus Go programmer paub.
Qhov ntawd yog txhua yam kuv xav qhia rau koj. Ua tsaug!
Tau qhov twg los: www.hab.com