HighLoad++ Moscow 2018, Congress Hall. Peb 9, 15:00
Abstracts thiab kev nthuav qhia:
Yuri Nasretdinov (VKontakte): daim ntawv tshaj tawm yuav tham txog cov kev paub ntawm kev siv ClickHouse hauv peb lub tuam txhab - vim li cas peb xav tau nws, peb khaws cov ntaub ntawv ntau npaum li cas, peb sau nws li cas, thiab lwm yam.
Ополнительные материалы:
Yuri Nasretdinov: - Nyob zoo txhua tus! Kuv lub npe yog Yuri Nasretdinov, raws li kuv twb tau qhia. Kuv ua haujlwm hauv VKontakte. Kuv yuav tham txog yuav ua li cas peb ntxig cov ntaub ntawv rau hauv ClickHouse los ntawm peb cov neeg rau zaub mov fleet (ntau txhiab).
Cov log yog dab tsi thiab vim li cas thiaj sau tau?
Peb yuav qhia koj li cas: peb tau ua dab tsi, vim li cas peb xav tau "ClickHouse", raws li, vim li cas peb xaiv nws, qhov kev ua tau zoo li cas koj tuaj yeem kwv yees tau yam tsis muaj teeb meem tshwj xeeb. Kuv yuav qhia koj ntxiv txog cov rooj tsis sib xws, txog cov teeb meem uas peb muaj nrog lawv thiab txog peb cov kev daws teeb meem uas peb tsim los ntawm qhov qhib - KittenHouse thiab Lighthouse.
Vim li cas peb thiaj li yuav tsum tau ua txhua yam (txhua yam yog ib txwm zoo ntawm VKontakte, txoj cai?). Peb xav sau cov cav debug (thiab muaj ntau pua terabytes ntawm cov ntaub ntawv nyob ntawd), tej zaum nws yuav yooj yim dua los xam cov txheeb cais; thiab peb muaj lub nkoj ntawm kaum tawm txhiab tus servers los ntawm txhua qhov no yuav tsum tau ua.
Vim li cas peb thiaj txiav txim siab? Tej zaum peb muaj cov kev daws teeb meem rau khaws cov cav. Ntawm no - muaj xws li pej xeem "Backend VK". Kuv xav kom subscribe rau nws.
Cov log yog dab tsi? Qhov no yog lub cav uas xa rov qab cov arrays. Cov cav hauv VK yog qhov lwm tus hu ua microservices. Thiab ntawm no yog daim nplaum nplaum (nyiam heev). Ua li cas? Zoo, mloog ntxiv!
Dab tsi tuaj yeem siv los khaws cov cav? Nws yog tsis yooj yim sua tsis hais Hadup. Tom qab ntawd, piv txwv li, Rsyslog ( khaws cov cav no hauv cov ntaub ntawv ). LSD. Leej twg paub LSD yog dab tsi? Tsis yog, tsis yog LSD no. Khaws cov ntaub ntawv, feem, thiab. Zoo, ClickHouse yog qhov kev xaiv coj txawv txawv.
Clickhouse thiab cov neeg sib tw: cov cai thiab cov hauv kev
Peb xav tau dab tsi? Peb xav kom ntseeg tau tias peb tsis tas yuav txhawj xeeb ntau dhau ntawm kev ua haujlwm, kom nws ua haujlwm tawm ntawm lub thawv, nyiam dua nrog kev teeb tsa tsawg. Peb xav sau ntau, thiab sau sai. Thiab peb xav khaws cia rau txhua yam ntawm lub hlis, xyoo, uas yog, ntev. Tej zaum peb yuav xav nkag siab qee qhov teeb meem uas lawv tuaj rau peb nrog thiab hais tias, "Ib yam dab tsi tsis ua haujlwm ntawm no," thiab qhov ntawd yog 3 lub hlis dhau los), thiab peb xav pom dab tsi tshwm sim 3 lub hlis dhau los " Cov ntaub ntawv compression - nws yog qhov tseeb vim li cas nws yuav yog qhov ntxiv - vim nws txo qhov chaw nws siv.
Thiab peb muaj qhov xav tau zoo li no: qee zaum peb sau cov zis ntawm qee cov lus txib (piv txwv li, cav), nws tuaj yeem ua tau ntau dua 4 kilobytes yooj yim heev. Thiab yog tias qhov no ua haujlwm dhau UDP, ces nws tsis tas yuav siv ... nws yuav tsis muaj "nyiaj siv ua haujlwm" rau kev sib txuas, thiab rau ntau tus servers qhov no yuav yog qhov ntxiv.
Cia peb pom dab tsi qhib qhov chaw muab rau peb. Ua ntej, peb muaj lub cav cav - qhov no yog peb lub cav; Hauv txoj ntsiab cai, nws muaj peev xwm ua tau txhua yam, nws tuaj yeem sau cov kab ntev. Zoo, nws tsis pom tseeb compress cov ntaub ntawv - peb tuaj yeem nthuav cov kab loj ntawm peb tus kheej yog tias peb xav tau ... peb, ntawm chav kawm, tsis xav (yog tias ua tau). Qhov teeb meem tsuas yog tias nws tsuas tuaj yeem muab yam uas haum rau hauv nws lub cim xeeb; Txhawm rau nyeem cov seem, koj yuav tsum tau txais binlog ntawm lub cav no thiab, raws li, nws yuav siv sij hawm ntev heev.
Puas muaj lwm txoj kev xaiv? Piv txwv li, "Hadup". Yooj yim khiav lag luam... Leej twg xav tias Hadup yooj yim teeb tsa? Tau kawg, tsis muaj teeb meem nrog kev kaw. Thaum nyeem ntawv, qee zaum muaj lus nug. Hauv txoj cai, kuv xav hais tias tej zaum tsis yog, tshwj xeeb tshaj yog rau cov cav. Kev khaws cia ntev - tau kawg, yog, cov ntaub ntawv compression - yog, cov hlua ntev - nws yog qhov tseeb tias koj tuaj yeem sau. Tab sis kaw los ntawm ntau tus servers ... Koj tseem yuav tsum ua qee yam ntawm koj tus kheej!
Rsyslog. Qhov tseeb, peb siv nws los ua ib qho kev xaiv thaub qab kom peb nyeem tau yam tsis tau muab pov tseg hauv binlog, tab sis nws tsis tuaj yeem sau cov kab ntev; hauv txoj cai, nws tsis tuaj yeem sau ntau dua 4 kilobytes. Koj yuav tsum ua cov ntaub ntawv compression tib yam koj tus kheej. Kev nyeem ntawv yuav los ntawm cov ntaub ntawv.
Tom qab ntawd muaj kev txhim kho "badushka" ntawm LSD. Qhov tseem ceeb tib yam li "Rsyslog": nws txhawb nqa cov hlua ntev, tab sis nws tsis tuaj yeem ua haujlwm ntawm UDP thiab, qhov tseeb, vim qhov no, hmoov tsis, ntau ntau yam yuav tsum tau rov sau dua muaj. LSD yuav tsum tau kho dua tshiab kom tuaj yeem sau los ntawm kaum tawm txhiab tus servers.
Thiab ntawm no! Ib qho kev xaiv lom zem yog ElasticSearch. Yuav hais li cas? Nws ua tau zoo nrog kev nyeem ntawv, uas yog, nws nyeem sai, tab sis tsis zoo nrog kev sau ntawv. Ua ntej, yog tias nws compresses cov ntaub ntawv, nws tsis muaj zog heev. Feem ntau yuav, kev tshawb nrhiav tag nrho yuav tsum muaj cov ntaub ntawv loj dua li qhov qub ntim. Nws yog qhov nyuaj rau kev khiav lag luam thiab teeb meem feem ntau tshwm sim nrog nws. Thiab, dua, kaw hauv Elastic - peb yuav tsum ua txhua yam peb tus kheej.
Ntawm no ClickHouse yog qhov kev xaiv zoo tshaj plaws, tau kawg. Qhov tsuas yog qhov uas sau los ntawm kaum tawm txhiab tus servers yog qhov teeb meem. Tab sis tsawg kawg yog ib qho teeb meem, peb tuaj yeem sim daws nws li cas. Thiab qhov seem ntawm tsab ntawv ceeb toom yog hais txog qhov teeb meem no. Dab tsi ntawm kev ua tau zoo koj tuaj yeem xav tau los ntawm ClickHouse?
Peb yuav ntxig nws li cas? MergeTree
Leej twg ntawm koj tsis tau hnov lossis paub txog "ClickHouse"? Kuv yuav tsum qhia koj, puas yog? ceev heev. Qhov kev tso rau ntawd - 1-2 gigabits ib ob, tawg mus txog 10 gigabits ib ob tuaj yeem tiv taus qhov kev teeb tsa no - muaj ob lub 6-core Xeons (uas yog, tsis yog lub zog tshaj), 256 gigabytes ntawm RAM, 20 terabytes hauv RAID (tsis muaj leej twg configured, default settings). Alexey Milovidov, tus tsim tawm ClickHouse, tej zaum zaum ntawd quaj vim peb tsis tau teeb tsa ib yam dab tsi (txhua yam ua haujlwm zoo li ntawd rau peb). Raws li, scanning ceev ntawm, hais, txog 6 billion kab ib ob tuaj yeem tau txais yog tias cov ntaub ntawv zoo compressed. Yog tias koj nyiam % ntawm cov ntawv nyeem - 100 lab kab ib ob, uas yog, nws zoo li nrawm heev.
Peb yuav ntxig nws li cas? Zoo, koj paub tias VK siv PHP. Peb yuav ntxig los ntawm txhua tus neeg ua haujlwm PHP ntawm HTTP rau hauv "ClickHouse", rau hauv MergeTree rooj rau txhua cov ntaub ntawv. Leej twg pom muaj teeb meem nrog lub tswv yim no? Rau qee yam, tsis yog txhua tus tau tsa tes. Cia kuv qhia koj.
Ua ntej, muaj ntau cov servers - raws li, yuav muaj kev sib txuas ntau (tsis zoo). Tom qab ntawd nws yog qhov zoo dua los ntxig cov ntaub ntawv rau hauv MergeTree tsis pub ntau tshaj ib zaug ib ob. Thiab leej twg paub yog vim li cas? Okay, ok. Kuv mam qhia koj me ntsis txog qhov no. Lwm cov lus nug nthuav yog tias peb tsis ua kev tshuaj ntsuam xyuas, peb tsis tas yuav ntxiv cov ntaub ntawv, peb tsis xav tau cov servers nruab nrab, peb xav ntxig ncaj qha rau hauv "ClickHouse" (zoo dua - qhov ncaj qha, qhov zoo dua).
Yog li ntawd, yuav ua li cas nkag mus rau hauv MergeTree? Vim li cas nws zoo dua rau ntxig rau hauv nws tsis ntau dua li ib zaug ib ob lossis tsawg dua? Qhov tseeb yog tias "ClickHouse" yog columnar database thiab txheeb cov ntaub ntawv nyob rau hauv qhov kev txiav txim nce mus txog qhov tseem ceeb, thiab thaum koj ua ib qho ntxig, ntau cov ntaub ntawv raug tsim tsawg kawg yog sib npaug ntawm cov kab ntawv uas cov ntaub ntawv raug txheeb xyuas. nyob rau hauv ascending kev txiav txim ntawm thawj tus yuam sij (ib tug cais directory yog tsim, ib txheej ntawm cov ntaub ntawv ntawm disk rau txhua qhov ntxig). Tom qab ntawd qhov kev ntxig ntxiv tom ntej tuaj, thiab hauv keeb kwm yav dhau lawv tau muab tso ua ke rau hauv cov "partitions" loj dua. Txij li cov ntaub ntawv raug txheeb xyuas, nws muaj peev xwm "merge" ob yam ntaub ntawv txheeb xyuas yam tsis tau siv ntau lub cim xeeb.
Tab sis, raws li koj tuaj yeem twv, yog tias koj sau 10 cov ntaub ntawv rau txhua qhov ntxig, ces ClickHouse (lossis koj lub server) yuav xaus sai sai, yog li nws pom zoo kom ntxig rau hauv cov khoom loj. Raws li, peb yeej tsis tau pib thawj lub tswv yim rau hauv kev tsim khoom. Peb tam sim ntawd launched ib, uas ntawm no No. 2 muaj:
Ntawm no xav txog tias muaj txog ib txhiab servers uas peb tau pib, tsuas muaj PHP xwb. Thiab ntawm txhua tus neeg rau zaub mov muaj peb tus neeg sawv cev hauv zos, uas peb hu ua "Kittenhouse", uas tuav ib qho kev sib txuas nrog "ClickHouse" thiab ntxig cov ntaub ntawv txhua ob peb feeb. Ntxig cov ntaub ntawv tsis nyob rau hauv MergeTree, tab sis mus rau hauv ib lub rooj tsis, uas ua hauj lwm meej kom tsis txhob tso ncaj qha rau hauv MergeTree tam sim ntawd.
Ua hauj lwm nrog buffer tables
Nws yog dab tsi? Cov ntxhuav tsis yog ib daim ntawm lub cim xeeb uas yog sharded (uas yog, nws tuaj yeem muab tso rau hauv nws nquag). Lawv muaj ob peb daim, thiab txhua daim ua hauj lwm raws li ib tug ywj siab tsis, thiab lawv yog flushed ntawm nws tus kheej (yog hais tias koj muaj ntau daim nyob rau hauv tsis, ces yuav muaj ntau inserts ib ob). Nws muaj peev xwm nyeem tau los ntawm cov ntxhuav no - tom qab ntawd koj nyeem lub koomhaum ntawm cov ntsiab lus ntawm qhov tsis sib xws thiab cov lus niam txiv, tab sis lub sijhawm no sau tau raug thaiv, yog li nws yog qhov zoo dua tsis txhob nyeem los ntawm qhov ntawd. Thiab cov rooj tsis pom zoo QPS, uas yog, txog li 3 txhiab QPS koj yuav tsis muaj teeb meem txhua lub sijhawm thaum ntxig. Nws yog tseeb hais tias yog hais tias tus neeg rau zaub mov poob hwj chim, ces cov ntaub ntawv yuav ploj, vim hais tias nws tsuas yog khaws cia rau hauv lub cim xeeb.
Nyob rau tib lub sijhawm, cov tswv yim nrog qhov tsis sib haum ua rau ALTER, vim tias koj thawj zaug yuav tsum tso lub rooj qub qub nrog cov txheej txheem qub (cov ntaub ntawv yuav tsis ploj mus qhov twg, vim tias nws yuav ntws tawm ua ntej lub rooj yuav raug tshem tawm). Tom qab ntawd koj "hloov" lub rooj koj xav tau thiab tsim lub rooj tsis muaj dua. Raws li, thaum tsis muaj lub rooj tsis muaj, koj cov ntaub ntawv yuav tsis ntws mus qhov twg, tab sis koj tuaj yeem muaj nws ntawm disk yam tsawg kawg hauv zos.
Kittenhouse yog dab tsi thiab nws ua haujlwm li cas?
KittenHouse yog dab tsi? Qhov no yog ib tug proxy. Xav seb yam lus twg? Kuv sau cov ntsiab lus hype tshaj plaws hauv kuv daim ntawv tshaj tawm - "Clickhouse", Mus, tej zaum kuv yuav nco txog lwm yam. Yog, qhov no tau sau rau hauv Go, vim kuv tsis paub yuav sau li cas hauv C, Kuv tsis xav.
Raws li, nws tswj kev sib txuas nrog txhua tus neeg rau zaub mov thiab tuaj yeem sau rau hauv nco. Piv txwv li, yog tias peb sau cov ntawv yuam kev rau Clickhouse, ces yog Clickhouse tsis muaj sijhawm los ntxig cov ntaub ntawv (tom qab tag nrho, yog tias sau ntau dhau), ces peb tsis o lub cim xeeb - peb tsuas muab pov tseg. Vim tias yog peb sau ntau gigabits ib ob ntawm qhov yuam kev, ces peb tuaj yeem muab pov tseg. Kittenhouse tuaj yeem ua qhov no. Ntxiv rau, nws tuaj yeem ua tiav kev xa khoom zoo, uas yog, sau rau disk ntawm lub tshuab hauv zos thiab ib zaug txhua zaus (nyob ntawd, ib zaug ob peb vib nas this) nws sim xa cov ntaub ntawv los ntawm cov ntaub ntawv no. Thiab thaum xub thawj peb tau siv tus nqi tsis tu ncua - tsis yog qee hom binary, ib hom ntawv nyeem (raws li hauv SQL li niaj zaus).
Tab sis tom qab ntawd qhov no tshwm sim. Peb tau siv cov khoom xa tuaj, sau cov cav, tom qab ntawd txiav txim siab (nws yog ib qho kev sim ntsuas pawg) ... Nws tau muab tso rau ob peb teev thiab coj rov qab, thiab kev tso tawm tau pib los ntawm ib txhiab servers - nws tig tawm tias Clickhouse tseem muaj lub "Xov ntawm kev sib txuas" - raws li, nyob rau hauv ib txhiab kev sib txuas, ib qho active insertion ua rau lub load nruab nrab ntawm lub server ntawm ib thiab ib nrab txhiab. Kuj ceeb tias, tus neeg rau zaub mov tau txais kev thov, tab sis cov ntaub ntawv tseem raug tso tom qab qee lub sijhawm; tab sis nws nyuaj heev rau server los pab nws ...
Ntxiv nginx
Xws li ib qho kev daws teeb meem rau Thread per kev sib txuas qauv yog nginx. Peb tau nruab nginx nyob rau hauv pem hauv ntej ntawm Clickhouse, tib lub sij hawm teeb tsa qhov sib npaug rau ob qhov kev hloov pauv (peb qhov nrawm nrawm tau nce 2 zaug, txawm hais tias nws tsis yog qhov tseeb tias qhov no yuav tsum yog qhov teeb meem) thiab txwv cov kev sib txuas rau Clickhouse, mus rau upstream thiab, raws li, ntau dua, tshaj li ntawm 50 kev sib txuas, nws zoo li tsis muaj qhov cuam tshuam rau kev ntxig.
Tom qab ntawd peb pom tau hais tias qhov txheej txheem no feem ntau muaj qhov tsis zoo, vim tias peb tsuas muaj ib qho nginx ntawm no. Raws li, yog tias qhov no nginx crashes, txawm tias muaj cov replicas, peb poob cov ntaub ntawv los yog, tsawg kawg, tsis txhob sau nyob qhov twg. Tias yog vim li cas peb tau ua peb tus kheej load ntsuas. Peb kuj pom tau hais tias "Clickhouse" tseem haum rau cov cav, thiab "dab" kuj pib sau nws cov cav hauv "Clickhouse" - yooj yim heev, ua siab ncaj. Peb tseem siv nws rau lwm tus "dab".
Tom qab ntawd peb nrhiav tau qhov teeb meem nthuav dav no: yog tias koj siv txoj hauv kev tsis yog txheej txheem ntawm kev ntxig rau hauv SQL hom, ces tag nrho SQL parser raws li AST raug yuam, uas yog qeeb heev. Raws li, peb tau ntxiv cov kev teeb tsa los xyuas kom meej tias qhov no tsis tshwm sim. Peb tau thauj khoom ntsuas, kuaj xyuas kev noj qab haus huv, yog li yog tias ib tus neeg tuag, peb tseem tso cov ntaub ntawv. Peb tam sim no muaj ntau ntau lub rooj uas peb yuav tsum muaj sib txawv Clickhouse pawg. Thiab peb kuj pib xav txog lwm yam kev siv - piv txwv li, peb xav sau cov cav los ntawm nginx modules, tab sis lawv tsis paub yuav ua li cas sib txuas lus siv peb RPC. Zoo, kuv xav qhia lawv yuav ua li cas xa tsawg kawg li cas - piv txwv li, kom tau txais cov xwm txheej ntawm localhost ntawm UDP thiab xa mus rau Clickhouse.
Ib kauj ruam deb ntawm kev daws teeb meem
Lub tswv yim kawg pib zoo li qhov no (qhov thib plaub ntawm cov tswv yim no): ntawm txhua tus neeg rau zaub mov nyob rau hauv pem hauv ntej ntawm Clickhouse muaj nginx (ntawm tib lub server) thiab nws tsuas yog proxies thov rau localhost nrog kev txwv ntawm tus lej ntawm kev sib txuas ntawm 50. daim. Thiab cov txheej txheem no twb tau ua haujlwm heev, txhua yam zoo nkauj nrog nws.
Peb nyob zoo li no txog ib hlis. Txhua leej txhua tus zoo siab, lawv ntxiv cov ntxhuav, lawv ntxiv, lawv ntxiv... Feem ntau, nws tau tig tawm tias txoj kev uas peb ntxiv cov rooj tsis zoo tsis zoo heev (cia peb muab tso rau ntawd). Peb tau ua 16 daim hauv txhua lub rooj thiab lub sijhawm flash ntawm ob peb vib nas this; peb muaj 20 lub rooj thiab txhua lub rooj tau txais 8 qhov ntxig rau ib ob - thiab lub sijhawm no "Clickhouse" pib ... cov ntaub ntawv pib qeeb. Lawv tsis txawm mus los ntawm ... Nginx los ntawm lub neej ntawd muaj qhov txaus nyiam tshaj plaws yog tias kev sib txuas tau xaus rau ntawm cov dej ntws, ces nws tsuas yog rov qab "502" rau txhua qhov kev thov tshiab.
Thiab ntawm no peb muaj (Kuv nyuam qhuav saib cov cav hauv Clickhouse nws tus kheej) txog ib nrab ntawm qhov kev thov ua tsis tiav. Raws li, kev siv disk yog siab, muaj ntau qhov sib koom ua ke. Zoo, kuv ua dab tsi? Lawm, kuv tsis thab los xav seb vim li cas qhov kev sib txuas thiab cov dej ntws tiav lawm.
Hloov nginx nrog lub npe rov qab
Kuv txiav txim siab tias peb yuav tsum tswj hwm qhov no peb tus kheej, peb tsis tas yuav tso nws mus rau nginx - nginx tsis paub tias lub rooj twg muaj nyob hauv Clickhouse, thiab kuv tau hloov nginx nrog lub npe rov qab, uas kuv kuj tau sau kuv tus kheej.
Nws ua dab tsi? Nws ua haujlwm raws li lub tsev qiv ntawv ceev http "goshnoy", uas yog, nrawm, yuav luag nrawm li nginx. Thov txim, Igor, yog tias koj nyob ntawm no (ceeb toom: Igor Sysoev yog tus kws tshaj lij Lavxias uas tsim lub web server nginx). Nws tuaj yeem nkag siab tias hom lus nug no yog - INSERT lossis SELECT - raws li, nws tuav cov kev sib txuas sib txawv rau ntau hom lus nug.
Raws li, txawm tias peb tsis muaj sijhawm los ua kom tiav qhov kev thov nkag, qhov "xaiv" yuav dhau mus, thiab rov ua dua. Thiab nws muab cov ntaub ntawv tso rau hauv cov rooj tsis sib xws - nrog rau qhov tsis muaj me me: yog tias muaj qhov yuam kev, syntax tsis raug, thiab lwm yam - kom lawv tsis cuam tshuam rau cov ntaub ntawv ntxiv, vim tias thaum peb tso rau hauv cov rooj tsis sib xws, peb muaj me me "bachi", thiab tag nrho cov syntax yuam kev tsuas yog cuam tshuam rau daim me me no; thiab ntawm no lawv twb yuav cuam tshuam ib tug loj tsis. Me me yog 1 megabyte, uas yog, tsis yog li ntawd me me.
Ntxig lub synchronization thiab qhov tseem ceeb hloov nginx, qhov tseem ceeb tib yam uas nginx tau ua ua ntej - koj tsis tas yuav hloov lub zos "Kittenhouse" rau qhov no. Thiab txij li thaum nws siv fasthttp, nws yog ceev heev - koj tuaj yeem ua ntau tshaj 100 txhiab thov ib ob rau ib qho kev ntxig los ntawm lub npe rov qab. Raws li txoj cai, koj tuaj yeem ntxig ib kab ntawm ib lub sijhawm rau hauv lub kittenhouse thim rov qab npe, tab sis tau kawg peb tsis ua li ntawd.
Cov tswv yim pib zoo li no: "Kittenhouse", qhov rov qab tso npe pab pawg ntau thov rau hauv cov rooj thiab, dhau los, cov rooj tsis sib txuas rau hauv cov ntsiab lus tseem ceeb.
Killer yog ib qho kev daws teeb meem ib ntus, Kitten nyob ruaj khov
Qhov no yog ib qho teeb meem nthuav... Koj puas tau siv fasthttp? Leej twg siv fasthttp nrog POST thov? Tej zaum, qhov no tiag tiag yuav tsum tsis tau ua tiav, vim tias nws buffers qhov kev thov lub cev los ntawm lub neej ntawd, thiab peb qhov loj me tsis tau teeb tsa rau 16 megabytes. Qhov kev tso tawm tau nres ntawm qee qhov taw qhia, thiab 16-megabyte chunks pib tuaj txog ntawm txhua kaum tawm txhiab tus servers, thiab lawv tag nrho cov buffered hauv nco ua ntej xa mus rau Clickhouse. Raws li, lub cim xeeb khiav tawm, Out-Of-Memory Killer tuaj thiab tua lub npe rov qab (lossis "Clickhouse", uas tuaj yeem ua "noj" ntau dua li rov qab tso npe). Lub voj voog rov ua nws tus kheej. Tsis yog ib qho teeb meem zoo siab heev. Txawm hais tias peb dawm qhov no tsuas yog tom qab ob peb lub hlis ntawm kev ua haujlwm.
Kuv tau ua dab tsi? Ib zaug ntxiv, kuv tsis nyiam nkag siab qhov tseeb tiag. Kuv xav tias nws yog ib qho yooj yim heev uas koj yuav tsum tsis txhob nyob rau hauv lub cim xeeb. Kuv tsis tuaj yeem kho saihttp, txawm tias kuv sim. Tab sis kuv nrhiav tau ib txoj hauv kev los ua kom nws tsis tas yuav tsum tau kho dab tsi, thiab kuv tuaj nrog kuv tus kheej txoj hauv kev HTTP - Kuv hu nws KITTEN. Zoo, nws yog qhov laj thawj - "VK", "Kitten" ... Dab tsi ntxiv? ..
Yog tias qhov kev thov tuaj rau tus neeg rau zaub mov nrog rau txoj kev Kitten, ces tus neeg rau zaub mov yuav tsum teb "meow" - logically. Yog tias nws teb rau qhov no, nws raug txiav txim siab tias nws nkag siab cov txheej txheem no, thiab tom qab ntawd kuv cuam tshuam qhov kev sib txuas (fasthttp muaj cov qauv zoo li no), thiab kev sib txuas mus rau hom "raws". Vim li cas kuv thiaj xav tau nws? Kuv xav tswj kev nyeem ntawv los ntawm TCP kev sib txuas li cas. TCP muaj cov cuab yeej zoo: yog tias tsis muaj leej twg nyeem los ntawm lwm sab, ces cov ntawv sau pib tos, thiab kev nco tsis tshwj xeeb yog siv rau qhov no.
Thiab yog li kuv tau nyeem los ntawm kwv yees li 50 tus neeg siv khoom ib zaug (los ntawm tsib caug vim tias tsib caug yuav tsum tau txaus, txawm tias tus nqi los ntawm lwm tus DC) ... Kev noj haus tau txo qis nrog qhov no tsawg kawg 20 zaug, tab sis kuv, kom ncaj ncees , Kuv ntsuas tsis tau raws nraim li lub sij hawm, vim nws twb pointless (nws twb mus txog theem ntawm kev ua yuam kev). Cov txheej txheem yog binary, uas yog, nws muaj lub npe lub rooj thiab cov ntaub ntawv; tsis muaj http headers, yog li kuv tsis tau siv lub vev xaib (Kuv tsis tas yuav sib txuas lus nrog browsers - Kuv tau ua cov txheej txheem uas haum rau peb cov kev xav tau). Thiab txhua yam ua tau zoo nrog nws.
Lub rooj tsis tu siab
Tsis ntev los no peb tau hla lwm qhov nthuav dav ntawm cov rooj tsis sib xws. Thiab qhov teeb meem no twb mob ntau dua li lwm tus. Cia peb xav txog qhov xwm txheej no: koj twb nquag siv Clickhouse, koj muaj kaum ob ntawm Clickhouse servers, thiab koj muaj qee qhov kev thov uas siv sijhawm ntev heev los nyeem (siv, ntau dua 60 vib nas this); thiab koj los ua Alter tam sim no... Lub sijhawm no, "xaiv" uas pib ua ntej "Alter" yuav tsis suav nrog hauv cov lus no, "Alter" yuav tsis pib - tej zaum qee qhov nta ntawm "Clickhouse" ua haujlwm li cas hauv qhov chaw no. Tej zaum qhov no tuaj yeem kho tau? Los yog nws ua tsis tau?
Feem ntau, nws yog qhov tseeb tias qhov no tsis yog qhov teeb meem loj, tab sis nrog cov rooj tsis sib haum nws ua rau mob ntau dua. Vim hais tias, yog tias, cia peb hais tias, koj lub sijhawm "Alter" (thiab nws yuav siv sij hawm tawm ntawm lwm tus tswv tsev - tsis yog ntawm koj li, tab sis ntawm tus qauv, piv txwv), ces ... lossis qee tus tswv tsev) tas sijhawm, tom qab ntawd qhov "Alter" yuam kev tau tshwm sim) - koj tseem yuav tsum tau xyuas kom meej tias cov ntaub ntawv txuas ntxiv yuav raug sau: koj tsim cov rooj tsis pub rov qab (raws li tib lub tswv yim raws li cov lus niam txiv), ces "Hloov" dhau mus, xaus tom qab tag nrho, thiab qhov tsis muaj lub rooj pib sib txawv hauv schema los ntawm niam txiv. Nyob ntawm seb qhov "Alter" yog dab tsi, qhov ntxig yuav tsis mus rau lub rooj tsis sib xws - qhov no tu siab heev.
Kuj tseem muaj qhov kos npe zoo li no (tej zaum ib tus neeg pom nws) - nws hu ua query_thread_log hauv cov ntawv tshiab ntawm Clickhouse. Los ntawm lub neej ntawd, hauv qee qhov version muaj ib qho. Ntawm no peb tau sau 840 lab cov ntaub ntawv hauv ob peb lub hlis (100 gigabytes). Qhov no yog vim qhov tseeb tias "inserts" tau sau rau ntawd (tej zaum tam sim no, los ntawm txoj kev, lawv tsis tau sau). Raws li kuv tau hais rau koj, peb cov "inserts" me me - peb muaj ntau "ntxig" rau hauv cov rooj tsis muaj. Nws yog qhov tseeb tias qhov no yog neeg xiam - Kuv tsuas yog qhia koj qhov kuv pom ntawm peb lub server. Vim li cas? Nov yog lwm qhov kev sib cav tawm tsam siv cov rooj tsis sib haum! Spotty tu siab heev.
Leej twg paub tias tus txiv neej no lub npe yog Spotty? Cov neeg ua haujlwm VK tsa tes. OK.
Hais txog cov phiaj xwm rau "KitttenHouse"
Cov phiaj xwm feem ntau tsis sib koom, txoj cai? Mam li nco dheev koj yuav tsis ua kom tiav lawv thiab yuav tsis zoo heev nyob rau hauv lwm tus neeg lub qhov muag. Tab sis kuv mam li pheej hmoo! Peb xav ua cov hauv qab no: cov rooj tsis sib xws, zoo li kuv, tseem yog tus ntoo khaub lig thiab peb yuav tsum tsis txhob cuam tshuam rau peb tus kheej. Tab sis peb tseem tsis xav kom nws tsis nyob hauv disk, yog li peb yuav tsis cuam tshuam qhov ntxig rau hauv lub cim xeeb.
Raws li, thaum "insert" yog ua, nws yuav tsis synchronous - nws yuav twb ua hauj lwm raws li ib tug tsis muaj rooj, yuav ntxig rau hauv lub rooj rau niam txiv (zoo, ib hnub tom qab) thiab tshaj tawm los ntawm ib tug cais channel uas inserts tau dhau mus thiab qhov twg tsis muaj.
Vim li cas kuv thiaj tsis tuaj yeem tso lub synchronous ntxig? Nws yooj yim dua. Qhov tseeb yog tias koj ntxig los ntawm 10 txhiab tus tswv, ces txhua yam zoo - koj yuav tau txais me ntsis los ntawm txhua tus tswv tsev, koj ntxig rau ib zaug ib zaug, txhua yam zoo. Tab sis kuv xav kom cov tswv yim no ua haujlwm, piv txwv li, los ntawm ob lub tshuab, kom koj tuaj yeem rub tawm ntawm kev kub ceev - tej zaum tsis tau txais qhov siab tshaj plaws ntawm Clickhouse, tab sis sau tsawg kawg yog 100 megabytes ib ob ntawm ib lub tshuab los ntawm lub npe rov qab - Qhov no lub tswv yim yuav tsum scale rau ob qho tib si loj thiab me me, yog li peb tsis tuaj yeem tos ib pliag rau txhua qhov ntxig, yog li nws yuav tsum yog asynchronous. Thiab nyob rau hauv tib txoj kev, asynchronous kev pom zoo yuav tsum tuaj tom qab qhov kev tso tawm tiav lawm. Peb yuav paub seb nws dhau los tsis dhau.
Qhov tseem ceeb tshaj plaws yog tias nyob rau hauv cov tswv yim no peb paub tseeb tias qhov kev ntxig coj qhov chaw los yog tsis. Xav txog qhov xwm txheej no: koj muaj lub rooj tsis khoom, koj tau sau ib yam dab tsi rau hauv nws, thiab tom qab ntawd, cia peb hais, lub rooj nkag mus rau hauv hom nyeem nkaus xwb thiab sim yaug qhov tsis. Cov ntaub ntawv yuav mus qhov twg? Lawv yuav nyob twj ywm hauv qhov tsis muaj. Tab sis peb tsis tuaj yeem paub meej txog qhov no - yuav ua li cas yog tias muaj qee qhov yuam kev, vim tias cov ntaub ntawv yuav tsis nyob hauv qhov tsis... (Addresses Alexey Milovidov, Yandex, ClickHouse tus tsim tawm) Los yog nws puas yuav nyob? Ib txwm? Alexey ntxias peb tias txhua yam yuav zoo. Peb tsis muaj laj thawj tsis ntseeg nws. Tab sis txhua yam zoo ib yam: yog tias peb tsis siv cov rooj tsis sib xws, ces yuav tsis muaj teeb meem nrog lawv. Tsim ob zaug ntau lub rooj kuj tsis yooj yim, txawm hais tias hauv txoj cai tsis muaj teeb meem loj. Qhov no yog txoj kev npaj.
Wb tham txog kev nyeem ntawv
Tam sim no cia peb tham txog kev nyeem ntawv. Peb kuj tau sau peb tus kheej cov cuab yeej ntawm no. Nws yuav zoo li, yog vim li cas sau koj tus kheej ntsuas ntawm no?.. Thiab leej twg siv Tabix? Cas ob peb tug tsa tes... Thiab leej twg txaus siab rau qhov kev ua tau zoo ntawm Tabix? Zoo, peb tsis txaus siab rau nws, thiab nws tsis yooj yim heev rau kev saib cov ntaub ntawv. Nws yog qhov zoo rau kev txheeb xyuas, tab sis tsuas yog rau kev saib nws tsis meej meej tsis ua kom zoo. Yog li kuv sau kuv tus kheej, kuv tus kheej interface.
Nws yog qhov yooj yim heev - nws tsuas tuaj yeem nyeem cov ntaub ntawv. Nws tsis paub yuav ua li cas qhia cov duab, nws tsis paub yuav ua li cas. Tab sis nws tuaj yeem qhia qhov peb xav tau: piv txwv li, muaj pes tsawg kab nyob hauv lub rooj, qhov chaw nws siv ntau npaum li cas (tsis tau tawg rau hauv kab), uas yog, qhov yooj yim interface yog qhov peb xav tau.
Thiab nws zoo li zoo ib yam li Sequel Pro, tab sis tsuas yog ua hauv Twitter's Bootstrap, thiab thib ob version. Koj nug: "Yuri, vim li cas ntawm qhov thib ob version?" Xyoo twg? 2018? Feem ntau, kuv tau ua qhov no ntev dhau los rau "Muscle" (MySQL) thiab nyuam qhuav hloov ob peb kab hauv cov lus nug nyob ntawd, thiab nws pib ua haujlwm rau "Clickhouse", uas tshwj xeeb ua tsaug! Vim hais tias cov parser zoo ib yam li "cov leeg" ib qho, thiab cov lus nug zoo sib xws - yooj yim heev, tshwj xeeb tshaj yog thaum xub thawj.
Zoo, nws tuaj yeem lim cov ntxhuav, tuaj yeem qhia cov qauv thiab cov ntsiab lus ntawm lub rooj, tso cai rau koj xaiv, lim los ntawm txhua kab, qhia cov lus nug uas tau tshwm sim, cov kab cuam tshuam (ntau npaum li cas), uas yog, yam yooj yim rau saib cov ntaub ntawv. ceev heev.
Kuj tseem muaj tus editor. Kuv ua siab ncaj sim nyiag tag nrho cov editor los ntawm Tabix, tab sis kuv ua tsis tau. Tab sis qee qhov nws ua haujlwm. Hauv txoj cai, yog tag nrho.
"Clickhouse" yog haum rau dens
Kuv xav qhia rau koj tias Clickhouse, txawm tias tag nrho cov teeb meem tau piav qhia, yog qhov zoo heev rau cov cav. Qhov tseem ceeb tshaj plaws, nws daws peb qhov teeb meem - nws ceev heev thiab tso cai rau koj lim cov cav los ntawm txhua kab. Raws li txoj cai, cov rooj tsis tau ua tsis tau zoo, tab sis feem ntau tsis muaj leej twg paub vim li cas ... Tej zaum tam sim no koj paub zoo dua qhov twg koj yuav muaj teeb meem.
TCP? Feem ntau, hauv VK nws yog kev cai siv UDP. Thiab thaum kuv siv TCP... Tau kawg, tsis muaj leej twg hais rau kuv: "Yuri, koj hais dab tsi! Koj ua tsis tau, koj xav tau UDP. " Nws muab tawm tias TCP tsis txaus ntshai. Qhov tsuas yog, yog tias koj muaj kaum tawm txhiab tus active compounds uas koj sau, koj yuav tsum tau npaj nws me ntsis kom zoo zoo; tab sis nws yog ua tau, thiab yooj yim heev.
Kuv tau cog lus tias yuav tshaj tawm "Kittenhouse" thiab "Lighthouse" ntawm HighLoad Siberia yog tias txhua tus tuaj yeem tso npe rau peb cov pej xeem "VK backend" ... Thiab koj paub, tsis yog txhua tus tau sau npe ... Tau kawg, Kuv yuav tsis thov kom koj sau npe rau peb. pej xeem. Tseem muaj koj ntau dhau lawm, tej zaum ib tug neeg txawm yuav chim, tab sis tseem, thov pab nias (thiab kuv yuav tsum ua qhov muag zoo li cov miv). Qhov ntawd yog . Ua tsaug ntau! Github yog peb li . Nrog Clickhouse koj cov plaub hau yuav mos thiab silky.
Thawj: - Cov phooj ywg, tam sim no rau cov lus nug. Txoj cai tom qab peb nthuav tawm daim ntawv pov thawj txaus siab thiab koj daim ntawv tshaj tawm ntawm VHS.
Yuri Nasretdinov (tom qab no hu ua YN): – Koj puas tuaj yeem sau kuv daim ntawv qhia txog VHS yog tias nws nyuam qhuav tas lawm?
Thawj: - Koj, ib yam nkaus, tsis tuaj yeem txiav txim siab tag nrho yuav ua li cas "Clickhouse" yuav ua haujlwm lossis tsis ua! Cov phooj ywg, 5 feeb rau cov lus nug!
Lus nug
Cov lus nug los ntawm cov neeg tuaj saib (tom qab no hu ua Q): - Nyob zoo tav su. Ua tsaug ntau rau daim ntawv qhia. Kuv muaj ob lo lus nug. Kuv mam li pib nrog qee yam tsis txaus ntseeg: tus lej ntawm cov ntawv t hauv lub npe "Kittenhouse" hauv daim duab (3, 4, 7 ...) puas cuam tshuam rau kev txaus siab ntawm miv?
YN: - Ntau npaum li cas?
Z: -Lab t. Muaj peb t's, qhov chaw nyob ib ncig ntawm peb t's.
YN: - Kuv tsis kho nws? Zoo, tau kawg nws ua! Cov no yog cov khoom sib txawv - Kuv tsuas yog dag koj txhua lub sijhawm. Okay, kuv kidding - tsis muaj teeb meem. Aw, tam sim no! Tsis yog, nws yog tib yam, Kuv ua typo.
Z: - Ua tsaug. Qhov thib ob lo lus nug tseem ceeb heev. Raws li kuv nkag siab, hauv Clickhouse, cov ntxhuav tsis nyob hauv lub cim xeeb, tsis buffered rau disk thiab, raws li, tsis tas li.
YN: - Yog.
Z: - Thiab nyob rau tib lub sijhawm, koj cov neeg siv khoom buffers rau disk, uas txhais tau hais tias qee qhov kev lav phib xaub ntawm tib lub cav. Tab sis qhov no tsis txhais tau tias tau lees paub ntawm Clickhouse. Piav seb qhov lav tau ua li cas, vim li cas?.. Ntawm no yog qhov txheej txheem no kom ntxaws ntxiv
YN: - Yog lawm, txoj kev xav tsis muaj qhov tsis sib haum xeeb ntawm no, vim tias thaum Clickhouse ntog, koj tuaj yeem ntes nws hauv ntau lab txoj kev sib txawv. Yog tias Clickhouse tsoo (yog tias nws xaus tsis raug), koj tuaj yeem hais lus ntxhib, rov qab me ntsis ntawm koj lub cav uas koj tau sau thiab pib txij lub sijhawm thaum txhua yam zoo. Cia peb hais tias koj rov qab ib feeb, uas yog, nws tau txiav txim siab tias koj tau yaug txhua yam hauv ib feeb.
Z: - Qhov ntawd yog, "Kittenhouse" tuav lub qhov rais ntev dua thiab, thaum lub caij nplooj zeeg, puas tuaj yeem paub txog nws thiab rov thim rov qab?
YN: - Tab sis qhov no yog nyob rau hauv txoj kev xav. Hauv kev xyaum, peb tsis ua qhov no, thiab kev xa khoom txhim khu kev qha yog los ntawm xoom mus rau lub sijhawm infinity. Tab sis nyob rau nruab nrab ib. Peb txaus siab tias yog Clickhouse poob rau qee qhov laj thawj lossis cov servers "reboot," ces peb poob me ntsis. Hauv lwm qhov xwm txheej, tsis muaj dab tsi yuav tshwm sim.
Z: - Nyob zoo. Txij thaum pib nws zoo li kuv tias koj yuav siv UDP los ntawm qhov pib ntawm tsab ntawv ceeb toom. Koj muaj http, tag nrho cov ntawd... Thiab feem ntau ntawm cov teeb meem uas koj tau piav, raws li kuv nkag siab nws, tau tshwm sim los ntawm qhov kev daws teeb meem no ...
YN: - Peb siv TCP dab tsi?
Z: - Qhov tseem ceeb yog.
YN: - Tsis muaj.
Z: - Nws yog nrog fasthttp tias koj muaj teeb meem, nrog kev sib txuas koj muaj teeb meem. Yog tias koj nyuam qhuav siv UDP koj yuav tau cawm koj tus kheej qee lub sijhawm. Zoo, yuav muaj teeb meem nrog cov lus ntev lossis lwm yam ...
YN: - Nrog dab tsi?
Z: - Nrog cov lus ntev, vim nws yuav tsis haum rau hauv MTU, lwm yam ... Zoo, tej zaum yuav muaj teeb meem ntawm lawv tus kheej. Cov lus nug yog: vim li cas ho tsis UDP?
YN: - Kuv ntseeg tias cov kws sau ntawv uas tsim TCP / IP yog qhov ntse dua kuv thiab paub zoo dua kuv yuav ua li cas serialize pob ntawv (kom lawv mus), tib lub sijhawm kho lub qhov rai xa, tsis dhau lub network, muab cov lus qhia txog dab tsi tsis tau nyeem, tsis suav rau sab nraud... Tag nrho cov teeb meem no, hauv kuv lub tswv yim, yuav muaj nyob hauv UDP, tsuas yog kuv yuav tsum tau sau cov lej ntau dua li kuv tau sau los txhawm rau txhawm rau ua raws li kuv tus kheej thiab feem ntau yuav. tsis zoo. Kuv yeej tsis nyiam sau ntawv hauv C, cia nyob ib leeg...
Z: - Tsuas yog yooj yim! Xa ok thiab tsis txhob tos dab tsi - nws yog asynchronous kiag li. Kev ceeb toom rov qab los tias txhua yam zoo - qhov ntawd txhais tau tias nws tuaj txog; Yog nws tsis tuaj, nws txhais tau tias nws tsis zoo.
YN: - Kuv xav tau ob qho tib si - Kuv yuav tsum tuaj yeem xa ob qho tib si nrog kev lees paub ntawm kev xa khoom thiab tsis muaj kev lees paub ntawm kev xa khoom. Nov yog ob qhov xwm txheej sib txawv. Kuv yuav tsum tsis txhob poob qee lub cav lossis tsis poob lawv vim li cas.
Z: - Kuv yuav tsis nkim sijhawm. Qhov no yuav tsum tau tham ntxiv. Ua tsaug.
Thawj: - Leej twg muaj lus nug - txhais tes rau saum ntuj!
Z: - Nyob zoo, Kuv yog Sasha. Ib qho chaw hauv nruab nrab ntawm daim ntawv tshaj tawm, qhov kev xav tau tshwm sim tias, ntxiv rau TCP, nws muaj peev xwm siv cov tshuaj npaj tau - qee yam ntawm Kafka.
YN: - Zoo ... Kuv tau hais rau koj tias kuv tsis xav siv cov servers nruab nrab, vim tias ... hauv Kafka, nws hloov tawm tias peb muaj kaum txhiab tus tswv; qhov tseeb, peb muaj ntau dua - kaum tawm txhiab tus tswv. Nws kuj tuaj yeem mob siab ua nrog Kafka yam tsis muaj npe. Tsis tas li ntawd, qhov tseem ceeb tshaj plaws, nws tseem muab "latency", nws muab cov tswv tsev ntxiv uas koj yuav tsum muaj. Tab sis kuv tsis xav muaj lawv - Kuv xav ...
Z: "Tab sis thaum kawg nws tau ua li ntawd lawm."
YN: - Tsis yog, tsis muaj tus tswv! Qhov no txhua yam ua haujlwm ntawm Clickhouse hosts.
Z: - Zoo, thiab "Kittenhouse", uas yog qhov rov qab - nws nyob qhov twg?
YN: - Ntawm Clickhouse host, nws tsis sau dab tsi rau lub disk.
Z: - Cia peb xav.
Thawj: – Koj puas txaus siab? Peb puas tuaj yeem muab nyiaj hli rau koj?
Z: - Yog, koj ua tau. Qhov tseeb, muaj ntau tus ntoo khaub lig thiaj li yuav tau txais tib yam, thiab tam sim no - cov lus teb dhau los ntawm lub ntsiab lus ntawm TCP tsis sib haum, hauv kuv lub tswv yim, qhov xwm txheej no. Nws tsuas yog xav zoo li txhua yam tuaj yeem ua tiav ntawm kuv lub hauv caug hauv lub sijhawm tsawg dua.
YN: - Thiab kuj yog vim li cas kuv tsis xav siv Kafka, vim tias muaj ntau qhov kev tsis txaus siab hauv Clickhouse Telegram tham uas, piv txwv li, cov lus los ntawm Kafka tau ploj lawm. Tsis yog los ntawm Kafka nws tus kheej, tab sis nyob rau hauv kev koom ua ke ntawm Kafka thiab Clickhaus; los yog ib yam dab tsi tsis txuas rau ntawd. Hais lus kwv yees, nws yuav tsim nyog sau tus neeg siv khoom rau Kafka ces. Kuv tsis xav tias yuav muaj kev daws teeb meem yooj yim dua lossis txhim khu kev qha.
Z: - Qhia rau kuv, vim li cas koj ho tsis sim cov kab lossis qee yam tsheb npav? Txij li thaum koj hais tias nrog asynchrony koj tuaj yeem xa cov cav lawv tus kheej los ntawm cov kab thiab tau txais cov lus teb asynchronously los ntawm cov kab?
YN: - Thov qhia seb cov kab twg tuaj yeem siv tau?
Z: - Txhua yam, txawm tias tsis muaj kev lees paub tias lawv nyob hauv kev txiav txim. Qee hom Redis, RMQ ...
YN: - Kuv muaj kev xav tias Redis feem ntau yuav tsis muaj peev xwm rub tau qhov ntim ntawm qhov ntxig txawm tias ntawm ib tus tswv tsev (hauv kev nkag siab ntawm ntau lub servers) uas rub tawm Clickhouse. Kuv tsis tuaj yeem thim qhov no nrog cov pov thawj (Kuv tsis tau ntsuas nws), tab sis nws zoo li kuv tias Redis tsis yog qhov kev daws teeb meem zoo tshaj plaws ntawm no. Raws li txoj cai, qhov system no tuaj yeem suav hais tias yog ib qho kev hais lus tsis zoo, tab sis qhov twg yog qhov tsim nyog rau "Clickhouse" nkaus xwb.
Thawj: - Yuri, ua tsaug ntau. Kuv thov kom xaus cov lus nug thiab lus teb ntawm no thiab hais tias leej twg ntawm cov neeg nug cov lus nug peb yuav muab phau ntawv rau.
YN: – Kuv xav muab ib phau ntawv rau thawj tus neeg nug ib lo lus nug.
Thawj: - Zoo kawg! Zoo heev! Zoo heev! Ua tsaug ntau!
Ib co ads 🙂
Ua tsaug uas koj tau nyob nrog peb. Koj puas nyiam peb cov ntawv? Xav pom cov ntsiab lus nthuav ntxiv? Txhawb nqa peb los ntawm kev tso ib qho kev txiav txim lossis qhia rau cov phooj ywg, , ib qho tshwj xeeb analogue ntawm nkag-theem servers, uas tau tsim los ntawm peb rau koj: (muaj nrog RAID1 thiab RAID10, mus txog 24 cores thiab mus txog 40GB DDR4).
Dell R730xd 2x pheej yig dua hauv Equinix Tier IV data center hauv Amsterdam? Tsuas yog nyob ntawm no hauv Netherlands! Dell R420 - 2x E5-2430 2.2Ghz 6C 128GB DDR3 2x960GB SSD 1Gbps 100TB - los ntawm $ 99! Nyeem txog
Tau qhov twg los: www.hab.com