Tsiv mus rau ClickHouse: 3 xyoos tom qab

Peb xyoos dhau los Viktor Tarnavsky thiab Alexey Milovidov los ntawm Yandex ntawm theem HighLoad ++ hais, ClickHouse zoo npaum li cas, thiab nws tsis qeeb. Thiab nyob rau theem tom ntej muaj Alexander Zaitsev с qhia hais txog tsiv mus Nyem Tsev los ntawm lwm qhov kev tshuaj ntsuam DBMS thiab nrog rau qhov xaus ntawd Nyem Tsev, ntawm chav kawm, zoo, tab sis tsis yooj yim heev. Thaum xyoo 2016 lub tuam txhab LifeStreet, qhov twg Alexander ces ua hauj lwm, tau hloov ib tug multi-petabyte analytical system rau Nyem Tsev, nws yog qhov txaus nyiam "daj cib txoj kev" tag nrho ntawm kev phom sij tsis paub - Nyem Tsev ces nws zoo li lub minefield.

Peb xyoos tom qab Nyem Tsev ua tau zoo dua - lub sijhawm no Alexander tau tsim lub tuam txhab Altinity, uas tsis tsuas yog pab tib neeg tsiv mus rau Nyem Tsev kaum ob ntawm cov haujlwm, tab sis kuj txhim kho cov khoom nws tus kheej ua ke nrog cov npoj yaig los ntawm Yandex. Tam sim no Nyem Tsev tseem tsis yog txoj kev taug kev tsis tu ncua, tab sis tsis muaj minefield lawm.

Alexander tau ua haujlwm nrog cov tshuab faib txij li xyoo 2003, tsim cov haujlwm loj ntawm MySQL, Oracle и Vertica. Thaum kawg HighLoad++ 2019 Alexander, ib tug ntawm cov pioneers ntawm kev siv Nyem Tsev, hais tias DBMS no yog dab tsi tam sim no. Peb yuav kawm txog cov yam ntxwv tseem ceeb Nyem Tsev: nws txawv li cas ntawm lwm lub tshuab thiab nyob rau hauv dab tsi nws yog qhov zoo dua los siv nws. Siv cov piv txwv, peb yuav saib tsis ntev los no thiab qhov project-tested xyaum rau lub tsev systems raws li Nyem Tsev.

Retrospective: dab tsi tshwm sim 3 xyoo dhau los

Peb xyoos dhau los peb tau hloov lub tuam txhab LifeStreet rau Nyem Tsev los ntawm lwm qhov kev txheeb xyuas database, thiab ad network analytics migration zoo li no:

• Lub Rau Hli 2016. In Qhib qhov chaw nyob Nyem Tsev thiab peb qhov project pib;
• Lub yim hli ntuj. Pov Thawj Ntawm Lub Tswv Yim: loj advertising network, infrastructure thiab 200-300 terabytes ntawm cov ntaub ntawv;
• Kaum Hli. Thawj cov ntaub ntawv ntau lawm;
• Kaum Ob Hlis. Tag nrho cov khoom thauj khoom yog 10-50 billion txheej xwm ib hnub.
• Lub Rau Hli 2017. Kev vam meej ntawm cov neeg siv mus Nyem Tsev, 2,5 petabytes ntawm cov ntaub ntawv ntawm ib pawg ntawm 60 servers.

Thaum txoj kev tsiv teb tsaws chaw, muaj kev nkag siab ntau zuj zus tuaj Nyem Tsev Nws yog qhov zoo tshaj plaws rau kev ua haujlwm nrog, tab sis qhov no yog qhov project sab hauv ntawm Yandex. Yog li ntawd, muaj cov nuances: Yandex yuav thawj zaug nrog nws tus kheej cov neeg siv khoom sab hauv thiab tsuas yog tom qab ntawd nrog cov zej zog thiab cov kev xav tau ntawm cov neeg siv sab nraud, thiab ClickHouse tom qab ntawd tsis mus txog qib kev lag luam hauv ntau qhov chaw ua haujlwm. Yog vim li cas peb thiaj nrhiav tau Altinity hauv lub Peb Hlis 2017 los ua Nyem Tsev Txawm sai dua thiab yooj yim dua tsis yog rau Yandex xwb, tab sis kuj rau lwm tus neeg siv. Thiab tam sim no peb:

• Peb cob qhia thiab pab tsim cov kev daws teeb meem raws li Nyem Tsev kom cov neeg siv khoom tsis txhob poob rau hauv qhov teeb meem, thiab kom cov kev daws teeb meem thaum kawg ua haujlwm;
• Peb muab kev txhawb nqa 24/7 Nyem Tsev- kev teeb tsa;
• Peb tsim peb tus kheej ecosystem tej yaam num;
• Peb nquag cog lus rau peb tus kheej Nyem Tsev, teb cov lus thov los ntawm cov neeg siv uas xav pom qee yam nta.

Thiab ntawm chav kawm, peb pab nrog tsiv mus Nyem Tsev с MySQL, Vertica, Oracle, Greenplum, Redshift thiab lwm yam systems. Peb tau koom nrog ntau yam kev txav, thiab lawv txhua tus tau ua tiav.

Vim li cas txav mus rau Nyem Tsev

Tsis qeeb! Qhov no yog qhov laj thawj tseem ceeb. Nyem Tsev - ceev heev database rau txawv scenarios:

Random quotes los ntawm cov neeg uas tau ua hauj lwm nrog cov neeg tau ntev Nyem Tsev.

Scalability. Ntawm qee qhov chaw khaws ntaub ntawv koj tuaj yeem ua tiav kev ua haujlwm zoo ntawm ib qho khoom kho vajtse, tab sis Nyem Tsev Koj tuaj yeem ntsuas tsis tau tsuas yog vertically, tab sis kuj horizontally, tsuas yog ntxiv cov servers. Txhua yam tsis ua haujlwm zoo li peb xav tau, tab sis nws ua haujlwm. Koj tuaj yeem nthuav qhov system raws li koj txoj kev lag luam loj hlob. Nws yog ib qho tseem ceeb uas peb tsis txwv los ntawm kev daws tam sim no thiab yeej ib txwm muaj peev xwm rau kev txhim kho.

Portability. Tsis muaj kev txuas rau ib yam. Piv txwv li, nrog Amazon RedShift Nws nyuaj rau txav mus rau qhov chaw. A Nyem Tsev Koj tuaj yeem nruab nws ntawm koj lub laptop, server, xa mus rau huab, mus rau Kubernetes - tsis muaj kev txwv rau kev ua haujlwm ntawm kev tsim kho vaj tse. Qhov no yog qhov yooj yim rau txhua tus, thiab qhov no yog qhov txiaj ntsig zoo uas ntau lwm cov ntaub ntawv zoo sib xws tsis tuaj yeem khav ntawm.

Flexibility. Nyem Tsev tsis nres ntawm ib yam, piv txwv li, Yandex.Metrica, tab sis tsim thiab siv nyob rau hauv ntau thiab ntau txawv tej yaam num thiab industries. Nws tuaj yeem nthuav dav los ntawm kev ntxiv cov peev txheej tshiab los daws cov teeb meem tshiab. Piv txwv li, nws ntseeg tau tias kev khaws cov cav hauv ib lub database yog tus cwj pwm phem, yog li lawv tuaj nrog Elasticsearch. Tab sis ua tsaug rau yooj Nyem Tsev, koj tuaj yeem khaws cov cav hauv nws, thiab feem ntau qhov no tseem zoo dua hauv Elasticsearch - ntawm Nyem Tsev qhov no yuav tsum tau 10 npaug tsawg hlau.

Dawb qhib Tau qhov twg los. Koj tsis tas yuav them rau dab tsi. Tsis tas yuav tham txog kev tso cai rau nruab qhov system ntawm koj lub laptop lossis server. Tsis muaj tus nqi zais. Nyob rau tib lub sijhawm, tsis muaj lwm yam Open Source database technology tuaj yeem sib tw nrawm nrog Nyem Tsev. MySQL, MariaDB, Greenplum - lawv tag nrho cov qeeb qeeb.

Community, tsav thiab kev lom zemCov. Ntawm Nyem Tsev Cov zej zog zoo heev: kev sib ntsib, kev sib tham thiab Alexey Milovidov, uas them peb txhua tus nrog nws lub zog thiab kev cia siab.

Hloov mus rau ClickHouse

Mus rau Nyem Tsev rau qee yam, koj tsuas xav tau peb yam:

• To taub cov kev txwv Nyem Tsev thiab dab tsi nws tsis haum rau.
• Coj kom zoo dua technology thiab nws lub zog loj tshaj plaws.
• Kev sim. Txawm tias nkag siab nws ua haujlwm li cas Nyem Tsev, Nws tsis yog ib txwm ua tau los twv seb thaum twg yuav nrawm dua, thaum twg yuav qeeb, thaum twg nws yuav zoo, thiab thaum twg nws yuav phem dua. Yog li sim nws.

Tsiv teeb meem

Tsuas muaj ib qho "tab sis": yog tias koj tsiv mus Nyem Tsev los ntawm lwm yam, ces feem ntau ib yam dab tsi mus tsis ncaj ncees lawm. Peb tau ua raws li qee qhov kev coj ua thiab tej yam uas ua haujlwm hauv peb cov ntaub ntawv nyiam. Piv txwv li, leej twg ua haujlwm nrog SQL-databases xav txog cov txheej txheem hauv qab no yuav tsum tau ua:

• kev lag luam;
• txwv;
• sib xws;
• qhov ntsuas;
• UPDATE/DELETE;
• NULLs;
• milliseconds;
• tsis siv neeg hom cam khwb cia;
• ntau qhov sib koom;
• arbitrary partitions;
• cov cuab yeej tswj pawg.

Recruitment yog yuav tsum tau, tab sis peb xyoos dhau los nyob rau hauv Nyem Tsev Tsis muaj cov haujlwm no muaj! Tam sim no tsawg tshaj li ib nrab ntawm qhov uas tsis tau muab coj los siv tseem tshuav: kev hloov pauv, kev txwv, Kev sib raug zoo, milliseconds thiab hom casting.

Thiab qhov tseem ceeb yog qhov hauv Nyem Tsev qee qhov kev coj ua thiab kev coj ua tsis ua haujlwm lossis ua haujlwm txawv dua li peb tau siv los. Txhua yam uas tshwm hauv Nyem Tsev, sib raug rau "ClickHouse txoj kev", i.e. kev ua haujlwm txawv ntawm lwm cov ntaub ntawv. Piv txwv li:

• Index tsis raug xaiv, tab sis hla.
• UPDATE/DELETE tsis synchronous, tab sis asynchronous.
• Muaj ntau qhov sib koom ua ke, tab sis tsis muaj lus nug npaj. Yuav ua li cas lawv thiaj li ua tau yog feem ntau tsis meej rau cov neeg los ntawm lub ntiaj teb database.

ClickHouse Scripts

Nyob rau hauv 1960, ib tug American mathematician ntawm Hungarian keeb kwm Wigner EP sau ib tsab xov xwm "Qhov tsis tsim nyog ua tau zoo ntawm kev ua lej hauv natural sciences” (“Lub Incomprehensible Effectiveness of Mathematics in the Natural Sciences”) uas lub ntiaj teb nyob ib ncig ntawm peb yog vim li cas tau piav qhia los ntawm kev cai lej. Kev ua lej yog kev tshawb fawb txog kev paub daws teeb meem, thiab cov cai ntawm lub cev uas qhia ua lej tsis tseem ceeb, thiab Wigner EP hais tias qhov no txawv heev.

Los ntawm kuv qhov kev xav, Nyem Tsev - tib yam txawv. Txhawm rau rov hais dua Wigner, peb tuaj yeem hais qhov no: kev ua haujlwm tsis txaus ntseeg yog qhov xav tsis thoob Nyem Tsev nyob rau hauv ntau yam kev siv tshuaj ntsuam xyuas!

Piv txwv li, cia peb ua Real-Time Data Warehouse, nyob rau hauv uas cov ntaub ntawv yog loaded yuav luag tsis tu ncua. Peb xav tau txais kev thov los ntawm nws nrog ncua sijhawm thib ob. Thov - siv nws Nyem Tsev, vim qhov no yog qhov scenario nws tau tsim los rau. Nyem Tsev qhov no yog raws nraim li cas nws yog siv tsis tau tsuas yog nyob rau hauv lub vev xaib, tab sis kuj nyob rau hauv kev lag luam thiab nyiaj txiag analytics, AdTech, as ib Nrhiav kev dag ntxiasn. IN Real-time Data Warehouse ib txoj kev tsim qauv zoo li "hnub qub" lossis "snowflake" yog siv, ntau lub rooj nrog SIB THAM (qee zaum ntau), thiab cov ntaub ntawv feem ntau yog khaws cia thiab hloov pauv hauv qee lub tshuab.

Cia peb ua lwm yam scenario - Sijhawm Series: saib xyuas cov khoom siv, tes hauj lwm, kev siv txheeb cais, Internet ntawm Yam. Ntawm no peb ntsib cov xwm txheej yooj yim ncaj ncees raws sijhawm. Nyem Tsev tsis yog thawj zaug tsim rau qhov no, tab sis tau pom nws tus kheej ua haujlwm zoo, uas yog vim li cas cov tuam txhab loj siv Nyem Tsev raws li ib tug repository rau kev soj ntsuam cov ntaub ntawv. Tshawb nrhiav seb nws puas tsim nyog Nyem Tsev rau lub sij hawm-series, peb tau ua tus qauv ntsuas raws li txoj hauv kev thiab cov txiaj ntsig InfluxDB и TimescaleDB - tshwj xeeb sij hawm-series databases. Nws muab tawm, uas Nyem Tsev, txawm tias tsis muaj kev ua kom zoo rau cov haujlwm zoo li no, yeej ntawm kev ua haujlwm txawv teb chaws:

В sij hawm-series Feem ntau siv lub rooj nqaim - ob peb kab me me. Ntau cov ntaub ntawv tuaj yeem los ntawm kev saib xyuas-ntau lab cov ntaub ntawv ib ob-thiab feem ntau lawv tuaj yeem tawg me me (tiag tiag-lub sij hawm streaming). Yog li ntawd, xav tau ib tsab ntawv sau sib txawv, thiab cov lus nug lawv tus kheej muaj lawv tus kheej tshwj xeeb.

Log Management. Sau cov cav rau hauv ib lub database feem ntau yog phem, tab sis Nyem Tsev qhov no tuaj yeem ua tiav nrog qee cov lus pom raws li tau piav qhia saum toj no. Ntau lub tuam txhab siv Nyem Tsev raws nraim rau lub hom phiaj no. Hauv qhov no, peb siv lub rooj dav dav uas peb khaws tag nrho cov cav (piv txwv li, hauv daim ntawv JSON), los yog txiav rau hauv pieces. Cov ntaub ntawv feem ntau yog thauj khoom hauv cov khoom loj (cov ntaub ntawv), thiab peb tshawb nrhiav los ntawm qee qhov chaw.

Rau txhua qhov haujlwm no, cov ntaub ntawv tshwj xeeb feem ntau yog siv. Nyem Tsev ib tug tuaj yeem ua tau txhua yam thiab zoo heev uas nws ua tau zoo tshaj lawv. Cia wb mus saib ze dua sij hawm-series scenario, thiab yuav ua li cas "ua noj" kom raug Nyem Tsev rau qhov xwm txheej no.

Sijhawm-Series

Tam sim no qhov no yog lub ntsiab scenario rau uas Nyem Tsev suav tias yog tus qauv kev daws teeb meem. Sijhawm-series yog txheej txheej xwm txheej tau txiav txim rau lub sijhawm, sawv cev rau cov kev hloov pauv hauv qee qhov txheej txheem dhau sijhawm. Piv txwv li, qhov no tuaj yeem yog lub plawv dhia ib hnub lossis cov txheej txheem hauv lub cev. Txhua yam uas muab lub sijhawm zuam nrog qee qhov ntev yog sij hawm-series:

Feem ntau ntawm cov xwm txheej no yog los ntawm kev saib xyuas. Qhov no tuaj yeem tsis tsuas yog saib xyuas lub vev xaib xwb, tab sis kuj tseem muaj cov khoom siv tiag tiag: tsheb, kev siv tshuab, IoT, factories los yog unmanned tsheb tavxij, nyob rau hauv lub cev ntawm Yandex twb muab tso Nyem Tsev- server.

Piv txwv li, muaj cov tuam txhab uas sau cov ntaub ntawv los ntawm nkoj. Txhua ob peb vib nas this, sensors ntawm lub thawv nkoj xa ntau pua qhov sib txawv. Cov kws tsim qauv kawm lawv, tsim qauv thiab sim nkag siab tias lub nkoj siv tau zoo npaum li cas, vim tias lub nkoj ntim khoom yuav tsum tsis txhob nyob twj ywm rau ib pliag. Txhua lub sijhawm poob yog poob nyiaj, yog li nws yog ib qho tseem ceeb uas yuav tsum tau twv seb txoj hauv kev kom nres tsawg kawg nkaus.

Niaj hnub no muaj kev loj hlob ntawm cov ntaub ntawv tshwj xeeb uas ntsuas sij hawm-series. Ntawm qhov chaw DB-Xyaw Cov databases sib txawv yog qee qhov qeb duas, thiab koj tuaj yeem saib lawv los ntawm hom:

Hom loj hlob sai tshaj plaws yog sij hawm seriess. Graph databases kuj loj hlob, tab sis sij hawm seriess tau loj hlob sai dua ob peb xyoos dhau los. Cov neeg sawv cev ntawm tsev neeg ntawm cov ntaub ntawv no yog InfluxDB, Prometheus, KDB, TimescaleDB (ua rau PostgreSQL), kev daws teeb meem los ntawm Amazon. Nyem Tsev tuaj yeem siv ntawm no ib yam nkaus, thiab siv tau. Cia kuv muab ob peb yam piv txwv rau pej xeem.

Ib tug ntawm cov pioneers yog lub tuam txhab CloudFlare (CDN- tus muab). Lawv saib xyuas lawv CDN los ntawm Nyem Tsev (DNS- thov, HTTP-queries) nrog rau cov khoom loj - 6 lab txheej xwm ib ob. Txhua yam dhau mus Kafka, mus Nyem Tsev, uas muab lub peev xwm los saib dashboards ntawm cov xwm txheej hauv qhov system hauv lub sijhawm.

Comcast - ib qho ntawm cov thawj coj hauv kev sib txuas lus hauv Asmeskas: Internet, digital TV, xov tooj. Lawv tsim ib qho kev tswj hwm zoo sib xws CDN nyob rau hauv lub moj khaum qhib Tau qhov twg los peb tes num Apache Traffic Control ua haujlwm nrog koj cov ntaub ntawv loj. Nyem Tsev siv los ua tus backend rau kev txheeb xyuas.

percona ua hauv Nyem Tsev hauv koj PMMkhaws cia saib xyuas ntau yam MySQL.

Cov Lus Qhia Tshwj Xeeb

Time-series databases muaj lawv tus kheej cov cai tshwj xeeb.

• Kev nrawm nrawm los ntawm ntau tus neeg sawv cev. Peb yuav tsum tau ntxig cov ntaub ntawv los ntawm ntau cov kwj deg sai heev. Nyem Tsev Nws ua qhov no zoo vim tias tag nrho nws cov ntawv ntxig tsis yog thaiv. Ib qho ntxig rau yog cov ntaub ntawv tshiab ntawm disk, thiab cov ntxig me me tuaj yeem raug buffered hauv ib txoj kev lossis lwm qhov. IN Nyem Tsev Nws yog qhov zoo dua los ntxig cov ntaub ntawv hauv cov khoom loj es tsis yog ib kab ntawm ib lub sijhawm.
• Cov tswv yim yooj yim. Cov sij hawm-series peb feem ntau tsis paub cov ntaub ntawv qauv kiag li. Nws muaj peev xwm tsim tau ib qho kev soj ntsuam rau ib daim ntawv thov tshwj xeeb, tab sis tom qab ntawd nws nyuaj rau siv rau lwm daim ntawv thov. Qhov no yuav tsum muaj lub tswv yim hloov tau yooj yim dua. Nyem Tsev, tso cai rau koj ua qhov no, txawm tias nws yog lub hauv paus muaj zog heev.
• Kev khaws cia zoo thiab tsis nco qab cov ntaub ntawv. Feem ntau hauv sij hawm-series ib tug loj npaum li cas ntawm cov ntaub ntawv, yog li nws yuav tsum tau muab khaws cia kom zoo li sai tau. Piv txwv li, ntawm InfluxDB zoo compression yog nws lub ntsiab feature. Tab sis dua li khaws cia, koj kuj yuav tsum muaj peev xwm "tsis nco qab" cov ntaub ntawv qub thiab ua qee yam downsampling - tsis siv neeg suav ntawm aggregates.
• Cov lus nug ceev ceev ntawm cov ntaub ntawv sib sau. Qee lub sij hawm nws yog qhov nthuav kom saib 5 feeb kawg nrog qhov tseeb ntawm milliseconds, tab sis nyob rau txhua hli cov ntaub ntawv feeb lossis thib ob granularity yuav tsis xav tau - kev txheeb cais dav dav txaus. Kev txhawb nqa ntawm hom no yog qhov tsim nyog, txwv tsis pub qhov kev thov rau 3 lub hlis yuav siv sijhawm ntev heev los ua kom tiav txawm tias nyob rau hauv Nyem Tsev.
• Thov zoo li "kawg point, as of». Cov no yog ib yam rau sij hawm-series Cov lus nug: saib qhov ntsuas kawg lossis lub xeev ntawm lub kaw lus nyob rau lub sijhawm t. Cov no tsis yog cov lus nug qab ntxiag heev rau cov ntaub ntawv, tab sis koj kuj yuav tsum muaj peev xwm ua tau.
• "Gluing" lub sijhawm series. Sijhawm-series yog lub sijhawm series. Yog hais tias muaj ob lub sij hawm series, lawv feem ntau yuav tsum tau kev cob cog rua thiab correlated. Nws tsis yooj yim los ua qhov no ntawm txhua lub databases, tshwj xeeb tshaj yog nrog cov sij hawm tsis sib haum: ntawm no yog qee lub sijhawm cov ntsiab lus, muaj lwm yam. Koj tuaj yeem xav txog qhov nruab nrab, tab sis tam sim ntawd tseem yuav muaj lub qhov nyob ntawd, yog li nws tsis meej.

Cia peb saib yuav ua li cas cov kev cai no tau ntsib hauv Nyem Tsev.

Lub tswv yim

В Nyem Tsev tswvyim rau sij hawm-series tuaj yeem ua tau ntau txoj hauv kev, nyob ntawm qib ntawm cov ntaub ntawv tsis tu ncua. Nws muaj peev xwm tsim tau ib qho system ntawm cov ntaub ntawv tsis tu ncua thaum peb paub tag nrho cov kev ntsuas ua ntej. Piv txwv li, kuv tau ua qhov no CloudFlare nrog saib xyuas CDN yog ib tug zoo optimized system. Koj tuaj yeem tsim kom muaj ntau qhov system uas saib xyuas tag nrho cov txheej txheem thiab ntau yam kev pabcuam. Nyob rau hauv rooj plaub ntawm cov ntaub ntawv tsis raug, peb tsis paub ua ntej peb saib xyuas dab tsi - thiab qhov no yog qhov feem ntau tshwm sim.

Cov ntaub ntawv tsis tu ncua. Cov kab. Cov txheej txheem yog yooj yim - kab nrog rau hom yuav tsum tau:

CREATE TABLE cpu (
  created_date Date DEFAULT today(),  
  created_at DateTime DEFAULT now(),  
  time String,  
  tags_id UInt32,  /* join to dim_tag */
  usage_user Float64,  
  usage_system Float64,  
  usage_idle Float64,  
  usage_nice Float64,  
  usage_iowait Float64,  
  usage_irq Float64,  
  usage_softirq Float64,  
  usage_steal Float64,  
  usage_guest Float64,  
  usage_guest_nice Float64
) ENGINE = MergeTree(created_date, (tags_id, created_at), 8192);

Qhov no yog lub rooj tsis tu ncua uas saib xyuas qee yam kev ua haujlwm hauv lub cev (cov neeg siv, system, dawb, zoo). Yooj yim thiab yooj yim, tab sis tsis hloov pauv. Yog tias peb xav tau lub tswv yim hloov tau yooj yim, ces peb tuaj yeem siv arrays.

Cov ntaub ntawv tsis raug. Arrays:

CREATE TABLE cpu_alc (
  created_date Date,  
  created_at DateTime,  
  time String,  
  tags_id UInt32,  
  metrics Nested(
    name LowCardinality(String),  
    value Float64
  )
) ENGINE = MergeTree(created_date, (tags_id, created_at), 8192);

SELECT max(metrics.value[indexOf(metrics.name,'usage_user')]) FROM ...

Qauv Zes yog ob arrays: npe metrics и metrics.value. Ntawm no koj tuaj yeem khaws cov ntaub ntawv saib xyuas tsis txaus ntseeg raws li cov npe ntawm cov npe thiab cov kev ntsuas ntsuas rau txhua qhov xwm txheej. Rau kev ua kom zoo ntxiv, tsis yog ib qho qauv zoo li no, koj tuaj yeem ua ntau yam. Piv txwv li, ib qho rau ntab-value, lwm - rau rau cov menyuam-lub ntsiab lus vim rau cov menyuam Kuv xav khaws cia kom zoo dua.

Tab sis cov qauv zoo li no nyuaj rau kev nkag mus. Koj yuav tau siv cov kev tsim tshwj xeeb, siv cov haujlwm tshwj xeeb los rub tawm cov txiaj ntsig ntawm thawj qhov ntsuas thiab tom qab ntawd cov array:

SELECT max(metrics.value[indexOf(metrics.name,'usage_user')]) FROM ...

Tab sis nws tseem ua haujlwm sai heev. Lwm txoj hauv kev los khaws cov ntaub ntawv tsis tu ncua yog los ntawm kab.

Cov ntaub ntawv tsis raug. Cov hlua. Hauv cov qauv no, tsis muaj arrays, cov npe thiab cov txiaj ntsig tau khaws cia ib txhij. Yog tias 5 kev ntsuas los ntawm ib lub cuab yeej ib zaug, 000 kab raug tsim tawm hauv cov ntaub ntawv:

CREATE TABLE cpu_rlc (
  created_date Date,  
  created_at DateTime,  
  time String,  
  tags_id UInt32,  
  metric_name LowCardinality(String),  
  metric_value Float64
) ENGINE = MergeTree(created_date, (metric_name, tags_id, created_at), 8192);


SELECT 
    maxIf(metric_value, metric_name = 'usage_user'),
    ... 
FROM cpu_r
WHERE metric_name IN ('usage_user', ...)

Nyem Tsev copes nrog qhov no - nws muaj tshwj xeeb extensions Nyem Tsev SQLCov. Piv txwv maxif - lub luag haujlwm tshwj xeeb uas suav qhov siab tshaj plaws los ntawm kev ntsuas thaum muaj qee qhov xwm txheej. Koj tuaj yeem sau ntau cov lus qhia hauv ib qhov kev thov thiab xam tam sim tus nqi rau ntau qhov ntsuas.

Cia peb piv peb txoj hauv kev:

Детали

Ntawm no kuv tau ntxiv "Disk Data Size" rau qee cov ntaub ntawv xeem. Hauv cov kab ntawv, peb muaj cov ntaub ntawv me me tshaj plaws: qhov siab tshaj plaws compression, siab tshaj plaws kev nug ceev, tab sis peb them los ntawm kev sau txhua yam ib zaug.

Nyob rau hauv cov ntaub ntawv ntawm arrays, txhua yam yog me ntsis zuj zus. Cov ntaub ntawv tseem zoo compressed thiab cov qauv tsis zoo tuaj yeem khaws cia. Tab sis Nyem Tsev - columnar database, thiab thaum peb pib khaws txhua yam hauv ib qho array, nws hloov mus ua ib kab, thiab peb them rau kev hloov pauv nrog kev ua haujlwm zoo. Rau ib qho kev ua haujlwm, koj yuav tsum tau nyeem tag nrho cov array rau hauv lub cim xeeb, tom qab ntawd nrhiav cov khoom xav tau hauv nws - thiab yog tias cov array loj tuaj, ces qhov ceev degrades.

Hauv ib lub tuam txhab uas siv txoj hauv kev no (piv txwv li, Uber), arrays raug txiav rau hauv daim ntawm 128 yam. Cov ntaub ntawv los ntawm ntau txhiab qhov kev ntsuas nrog qhov ntim ntawm 200 TB ntawm cov ntaub ntawv / hnub yog khaws cia tsis nyob hauv ib qho array, tab sis hauv 10 lossis 30 arrays nrog tshwj xeeb cia logic.

Txoj kev yooj yim tshaj plaws yog nrog cov hlua. Tab sis cov ntaub ntawv tsis zoo compressed, lub rooj loj loj, thiab txawm tias cov lus nug raws li ntau qhov kev ntsuas, ClickHouse tsis ua haujlwm zoo.

Hybrid tswvyim

Cia peb xav tias peb tau xaiv ib qho array circuit. Tab sis yog tias peb paub tias feem ntau ntawm peb cov dashboards tsuas yog qhia cov neeg siv thiab cov kev ntsuas qhov system, peb tuaj yeem ntxiv cov kev ntsuas no rau hauv txhua kab los ntawm ib qho array ntawm qib lus hauv txoj kev no:

CREATE TABLE cpu_alc (
  created_date Date,  
  created_at DateTime,  
  time String,  
  tags_id UInt32,  
  metrics Nested(
    name LowCardinality(String),  
    value Float64
  ),
  usage_user Float64 
             MATERIALIZED metrics.value[indexOf(metrics.name,'usage_user')],
  usage_system Float64 
             MATERIALIZED metrics.value[indexOf(metrics.name,'usage_system')]
) ENGINE = MergeTree(created_date, (tags_id, created_at), 8192);

Thaum ntxig Nyem Tsev yuav suav lawv. Txoj kev no koj tuaj yeem sib xyaw ua lag luam nrog kev txaus siab: cov tswv yim hloov pauv tau yooj yim thiab dav, tab sis peb rub tawm cov kab uas nquag siv. Nco ntsoov tias qhov no tsis tas yuav hloov qhov ntxig thiab ETLuas txuas ntxiv mus ntxig arrays rau hauv lub rooj. Peb nyuam qhuav ua ALTER LUB ROOJ, ntxiv ob peb tus neeg hais lus thiab peb tau txais cov txheej txheem hybrid thiab nrawm dua uas koj tuaj yeem pib siv tam sim ntawd.

Codecs thiab compression

rau sij hawm-series Nws tseem ceeb npaum li cas koj ntim cov ntaub ntawv vim tias cov ntaub ntawv tuaj yeem loj heev. IN Nyem Tsev Muaj cov cuab yeej siv kom ua tiav cov nyhuv compression ntawm 1:10, 1:20, thiab qee zaum ntau dua. Qhov no txhais tau hais tias 1 TB ntawm cov ntaub ntawv tsis tau ntim ntawm lub disk siv 50-100 GB. Qhov loj me me yog qhov zoo, cov ntaub ntawv tuaj yeem nyeem thiab ua tiav sai dua.

Yuav kom ua tiav qib siab ntawm compression, Nyem Tsev txhawb cov codecs hauv qab no:

Piv txwv rooj:

CREATE TABLE benchmark.cpu_codecs_lz4 (
    created_date Date DEFAULT today(), 
    created_at DateTime DEFAULT now() Codec(DoubleDelta, LZ4), 
    tags_id UInt32, 
    usage_user Float64 Codec(Gorilla, LZ4), 
    usage_system Float64 Codec(Gorilla, LZ4), 
    usage_idle Float64 Codec(Gorilla, LZ4), 
    usage_nice Float64 Codec(Gorilla, LZ4), 
    usage_iowait Float64 Codec(Gorilla, LZ4), 
    usage_irq Float64 Codec(Gorilla, LZ4), 
    usage_softirq Float64 Codec(Gorilla, LZ4), 
    usage_steal Float64 Codec(Gorilla, LZ4), 
    usage_guest Float64 Codec(Gorilla, LZ4), 
    usage_guest_nice Float64 Codec(Gorilla, LZ4), 
    additional_tags String DEFAULT ''
)
ENGINE = MergeTree(created_date, (tags_id, created_at), 8192);

Ntawm no peb txhais lub codec DoubleDelta hauv ib rooj plaub, hauv qhov thib ob - Liab twm hawj, thiab peb yuav twv yuav raug hu ntxiv LZ4 compression. Raws li qhov tshwm sim, qhov loj ntawm cov ntaub ntawv ntawm disk tau txo qis heev:

Qhov no qhia tau hais tias ntau qhov chaw tib cov ntaub ntawv yuav siv li cas, tab sis siv cov codecs sib txawv thiab compressions:

• hauv cov ntaub ntawv GZIP ntawm disk;
• hauv ClickHouse tsis muaj codecs, tab sis nrog ZSTD compression;
• hauv ClickHouse nrog codecs thiab compression LZ4 thiab ZSTD.

Nws tuaj yeem pom tias cov ntxhuav nrog codecs siv ntau qhov chaw tsawg dua.

Qhov loj me muaj teeb meem

Tsis tseem ceeb tsawg xaiv hom ntaub ntawv raug:

Hauv txhua qhov piv txwv saum toj no kuv tau siv Float 64. Tab sis yog peb xaiv Float 32, ces qhov ntawd yuav zoo dua. Qhov no tau pom zoo los ntawm cov txiv neej los ntawm Perkona hauv kab lus txuas saum toj no. Nws yog ib qho tseem ceeb uas yuav tsum tau siv ntau hom kev cog lus uas tsim nyog rau txoj haujlwm: txawm tias tsawg dua rau disk loj dua rau kev nug ceev. Nyem Tsev rhiab heev rau qhov no.

Yog koj siv tau intxnumx es tsis txhob intxnumx, ces cia siab tias yuav luag ob npaug ntawm kev ua tau zoo. Cov ntaub ntawv siv lub cim xeeb tsawg, thiab tag nrho cov "arithmetic" ua haujlwm sai dua. Nyem Tsev internally nws yog ib tug nruj heev ntaus ntawv system; nws ua rau siab tshaj plaws siv ntawm tag nrho cov possibilities uas niaj hnub systems muab.

Aggregation thiab Khoom Siv Views

Aggregation thiab materialized views tso cai rau koj los tsim aggregates rau ntau zaus:

Piv txwv li, koj tuaj yeem muaj cov ntaub ntawv tsis sib sau ua ke, thiab koj tuaj yeem txuas ntau yam kev pom zoo rau lawv nrog kev suav tsis siv neeg los ntawm lub cav tshwj xeeb SummingMergeTree (SMT). SMT yog ib tug tshwj xeeb aggregating cov ntaub ntawv qauv uas xam aggregates txiav. Cov ntaub ntawv nyoos tau muab tso rau hauv cov ntaub ntawv, nws tau txais kev sib sau ua ke, thiab cov dashboards tuaj yeem siv tam sim ntawm nws.

TTL - "tsis nco qab" cov ntaub ntawv qub

Yuav ua li cas "tsis nco qab" cov ntaub ntawv uas tsis xav tau ntxiv lawm? Nyem Tsev paub ua li no. Thaum tsim cov ntxhuav, koj tuaj yeem qhia meej TTL Cov lus qhia: piv txwv li, tias peb khaws cov ntaub ntawv feeb rau ib hnub, cov ntaub ntawv txhua hnub rau 30 hnub, thiab tsis txhob kov cov ntaub ntawv txhua lub lim tiam lossis txhua hli:

CREATE TABLE aggr_by_minute
…
TTL time + interval 1 day

CREATE TABLE aggr_by_day
…
TTL time + interval 30 day

CREATE TABLE aggr_by_week
…
/* no TTL */

Multi-tier - faib cov ntaub ntawv hla disks

Siv lub tswv yim no ntxiv, cov ntaub ntawv tuaj yeem khaws cia rau hauv Nyem Tsev nyob rau hauv qhov chaw sib txawv. Piv txwv tias peb xav khaws cov ntaub ntawv kub rau lub lim tiam dhau los ntawm lub zos ceev heev SSD, thiab peb muab cov ntaub ntawv keeb kwm ntau ntxiv rau lwm qhov chaw. IN Nyem Tsev qhov no yog tam sim no ua tau:

Koj tuaj yeem teeb tsa txoj cai cia khoom (txoj cai cia) Yog li ntawd Nyem Tsev yuav cia li hloov cov ntaub ntawv thaum ncav cuag tej yam kev mob rau lwm cia.

Tab sis tsis yog tag nrho. Nyob rau theem ntawm ib lub rooj tshwj xeeb, koj tuaj yeem txhais cov cai raws nraim thaum cov ntaub ntawv nkag mus rau hauv qhov txias txias. Piv txwv li, cov ntaub ntawv khaws cia rau ntawm lub disk ceev heev rau 7 hnub, thiab txhua yam uas laus dua yog pauv mus rau qhov qeeb. Qhov no yog qhov zoo vim tias nws tso cai rau koj khaws lub kaw lus ntawm qhov ua tau zoo tshaj plaws, thaum tswj cov nqi thiab tsis txhob nkim nyiaj rau cov ntaub ntawv txias:

CREATE TABLE 
... 
TTL date + INTERVAL 7 DAY TO VOLUME 'cold_volume', 
    date + INTERVAL 180 DAY DELETE

Cov sijhawm tshwj xeeb Nyem Tsev

Hauv yuav luag txhua yam Nyem Tsev Muaj xws li "highlights", tab sis lawv yog offset los ntawm exclusivity - ib yam dab tsi uas tsis yog nyob rau hauv lwm yam databases. Piv txwv li, ntawm no yog qee qhov tshwj xeeb nta Nyem Tsev:

• Arrays. Cov Nyem Tsev kev txhawb nqa zoo heev rau arrays, nrog rau lub peev xwm los ua cov lej nyuaj rau lawv.
• Aggregating Data Structures. Qhov no yog ib qho ntawm "killer nta" Nyem Tsev. Txawm tias muaj tseeb hais tias cov txiv neej los ntawm Yandex hais tias peb tsis xav kom sau cov ntaub ntawv, txhua yam yog sau nyob rau hauv Nyem Tsev, vim nws ceev thiab yooj yim.
• Materialized Views. Ua ke nrog cov ntaub ntawv sib sau ua ke, cov ntaub ntawv pom tau tso cai rau koj kom yooj yim tiag tiag-lub sij hawm kev sib sau ua ke.
• ClickHouse SQL. Qhov no yog ib hom lus extension SQL nrog ib co ntxiv thiab tshwj xeeb nta uas tsuas muaj nyob rau hauv Nyem Tsev. Yav dhau los, nws zoo li kev nthuav dav ntawm ib sab, thiab qhov tsis zoo ntawm lwm tus. Tam sim no yuav luag txhua qhov tsis zoo piv rau SQL ib 92 peb tshem tawm nws, tam sim no nws tsuas yog kev txuas ntxiv.
• Lambda-cov lus qhia. Puas yog lawv tseem nyob hauv ib qho database?
• ML- kev txhawb nqa. Qhov no muaj nyob rau hauv cov ntaub ntawv sib txawv, qee qhov zoo dua, qee qhov phem dua.
• qhib qhov chaw. Peb tuaj yeem nthuav dav Nyem Tsev ua ke. Tam sim no nyob rau hauv Nyem Tsev kwv yees li 500 tus neeg koom nrog, thiab tus lej no tau nce mus tas li.

Cov lus nug tsis yooj yim

В Nyem Tsev muaj ntau yam sib txawv los ua ib yam. Piv txwv li, koj tuaj yeem rov qab tus nqi kawg los ntawm ib lub rooj hauv peb txoj kev sib txawv rau CPU (tseem muaj plaub, tab sis nws tseem txawv dua).

Thawj qhov qhia tau hais tias nws yooj yim npaum li cas los ua hauv Nyem Tsev cov lus nug thaum koj xav tshawb xyuas qhov ntawd tuple tug muaj nyob rau hauv subquery. Qhov no yog ib yam dab tsi uas kuv tus kheej yeej nco nyob rau hauv lwm yam databases. Yog hais tias kuv xav muab piv ib yam dab tsi nrog ib tug subquery, ces nyob rau hauv lwm yam databases tsuas yog ib tug scalar yuav muab piv nrog nws, tab sis rau ob peb kab kuv yuav tsum tau sau. SIB THAM. Cov Nyem Tsev Koj tuaj yeem siv tuple:

SELECT *
  FROM cpu 
 WHERE (tags_id, created_at) IN 
    (SELECT tags_id, max(created_at)
        FROM cpu 
        GROUP BY tags_id)

Qhov thib ob txoj kev ua tib yam tab sis siv ib qho kev sib sau ua ke argMax:

SELECT 
    argMax(usage_user), created_at),
    argMax(usage_system), created_at),
...
 FROM cpu 

В Nyem Tsev muaj ob peb lub kaum sib sau ua haujlwm, thiab yog tias koj siv cov combinators, ces raws li txoj cai ntawm combinatorics koj yuav tau txais txog ib txhiab ntawm lawv. ArgMax - ib qho ntawm cov haujlwm uas suav tus nqi siab tshaj plaws: qhov kev thov rov qab tus nqi siv_user, thaum tus nqi siab tshaj plaws mus txog tsim_at:

SELECT now() as created_at,
       cpu.*
  FROM (SELECT DISTINCT tags_id from cpu) base 
  ASOF LEFT JOIN cpu USING (tags_id, created_at)

ASOF JOIN - "gluing" kab nrog lub sijhawm sib txawv. Qhov no yog ib tug tshwj xeeb feature rau databases uas tsuas muaj nyob rau hauv kdb+. Yog tias muaj ob lub sijhawm sib txawv nrog lub sijhawm sib txawv, ASOF JOIN tso cai rau koj txav mus thiab muab lawv tso rau hauv ib qho kev thov. Rau txhua tus nqi nyob rau hauv ib lub sij hawm series, qhov ze tshaj tus nqi nyob rau hauv lwm yam yog pom, thiab lawv raug xa rov qab rau tib kab:

Analytic Functions

Hauv tus qauv SQL-2003 koj sau tau li no:

SELECT origin,
       timestamp,
       timestamp -LAG(timestamp, 1) OVER (PARTITION BY origin ORDER BY timestamp) AS duration,
       timestamp -MIN(timestamp) OVER (PARTITION BY origin ORDER BY timestamp) AS startseq_duration,
       ROW_NUMBER() OVER (PARTITION BY origin ORDER BY timestamp) AS sequence,
       COUNT() OVER (PARTITION BY origin ORDER BY timestamp) AS nb
  FROM mytable
ORDER BY origin, timestamp;

В Nyem Tsev Koj tsis tuaj yeem ua li ntawd - nws tsis txhawb tus qauv SQL-2003 thiab tej zaum yuav tsis ua li ntawd. Hloov chaw, hauv Nyem Tsev Nws yog kev cai sau li no:

Kuv tau cog lus tias lambdas - ntawm no lawv nyob!

Qhov no yog ib qho analogue ntawm cov lus nug analytical hauv tus qauv SQL-2003: nws suav qhov sib txawv ntawm ob timestamp, lub sijhawm, ordinal number - txhua yam uas peb feem ntau xav txog kev ua haujlwm analytical. IN Nyem Tsev Peb suav lawv los ntawm arrays: ua ntej peb tsoo cov ntaub ntawv rau hauv ib qho array, tom qab ntawd peb ua txhua yam peb xav tau ntawm array, thiab tom qab ntawd peb nthuav nws rov qab. Nws tsis yooj yim heev, nws xav tau kev hlub ntawm kev ua haujlwm ntawm qhov tsawg kawg nkaus, tab sis nws hloov tau yooj yim heev.

Tshwj xeeb nta

Tsis tas li ntawd, hauv Nyem Tsev ntau yam tshwj xeeb. Piv txwv li, yuav ua li cas los txiav txim seb muaj pes tsawg lub rooj sib tham ua ib txhij? Ib txoj haujlwm saib xyuas ib txwm yog los txiav txim siab qhov siab tshaj plaws nrog ib qho kev thov. IN Nyem Tsev Nws muaj lub luag haujlwm tshwj xeeb rau lub hom phiaj no:

Feem ntau, ClickHouse muaj cov haujlwm tshwj xeeb rau ntau lub hom phiaj:

• runDifference, runAccumulate, nyob ze;
• sumMap(key, tus nqi);
• timeSeriesGroupSum(uid, timestamp, nqi);
• timeSeriesGroupRateSum(uid, timestamp, nqi);
• skewPop, skewSamp, kurtPop, kurtSamp;
• NTAUB NTAWV / NROG TIES;
• yooj yimLinearRegression, stochasticLinearRegression.

Qhov no tsis yog tag nrho cov npe ntawm cov haujlwm, muaj 500-600 nyob rau hauv tag nrho. Lus Cim: txhua qhov haujlwm hauv Nyem Tsev yog nyob rau hauv cov lus system (tsis yog tag nrho cov ntaub ntawv, tab sis txhua yam yog nthuav):

select * from system.functions order by name

Nyem Tsev nws khaws ntau cov ntaub ntawv hais txog nws tus kheej, suav nrog log rooj, query_log, trace log, log of operations with data blocks (part_log), metrics cav, thiab lub kaw lus, uas feem ntau nws sau rau disk. Log metrics yog sij hawm-series в Nyem Tsev qhov tseeb Nyem Tsev: Lub database nws tus kheej tuaj yeem ua lub luag haujlwm sij hawm-series databases, yog li "devouring" nws tus kheej.

Qhov no kuj yog ib qho tshwj xeeb - txij li thaum peb ua haujlwm zoo rau sij hawm-series, vim li cas peb tsis tuaj yeem khaws txhua yam peb xav tau hauv peb tus kheej? Peb tsis tas yuav Prometheus, peb khaws txhua yam rau peb tus kheej. Txuas nrog ua grafana thiab peb saib xyuas peb tus kheej. Txawm li cas los xij, yog tias Nyem Tsev poob, peb yuav tsis pom vim li cas, yog li lawv feem ntau tsis ua li ntawd.

Cov pawg loj lossis ntau qhov me me Nyem Tsev

Dab tsi yog qhov zoo dua - ib pawg loj lossis ntau lub ClickHouses me? Kev lig kev cai mus kom ze DWH yog ib pawg loj nyob rau hauv uas circuits tau faib rau txhua daim ntawv thov. Peb tuaj rau tus thawj tswj hwm database - muab ib daim duab rau peb, thiab lawv muab ib qho rau peb:

В Nyem Tsev koj tuaj yeem ua nws txawv. Koj tuaj yeem ua txhua daim ntawv thov koj tus kheej Nyem Tsev:

Peb tsis xav tau lub monstrous loj lawm DWH thiab intractable admins. Peb tuaj yeem muab txhua daim ntawv thov nws tus kheej Nyem Tsev, thiab tus tsim tawm tuaj yeem ua nws tus kheej, txij li thaum Nyem Tsev yooj yim heev rau nruab thiab tsis xav tau txoj kev tswj hwm:

Tab sis yog tias peb muaj ntau Nyem Tsev, thiab koj yuav tsum tau nruab nws ntau zaus, ces koj xav automate cov txheej txheem no. Rau qhov no peb tuaj yeem, piv txwv li, siv Kubernetes и clickhouse- tus neeg ua haujlwm. IN Kubernetes ClickHouse Koj tuaj yeem tso nws "on-click": Kuv tuaj yeem nyem lub pob, khiav lub manifest thiab cov ntaub ntawv tau npaj txhij. Kuv tuaj yeem tsim daim duab tam sim ntawd, pib xa cov ntsuas ntsuas nyob ntawd, thiab hauv 5 feeb kuv muaj lub dashboard npaj ua grafana. Nws yooj yim heev!

Qhov tshwm sim yog dab tsi?

Thiab yog li ntawd, Nyem Tsev - Qhov no:

• Ceev ceev. Sawv daws paub qhov no.
• Cia li. Kev tsis sib haum xeeb me ntsis, tab sis kuv ntseeg tias nws nyuaj rau kev cob qhia, yooj yim hauv kev sib ntaus sib tua. Yog koj nkag siab li cas Nyem Tsev nws ua haujlwm, ces txhua yam yog yooj yim heev.
• Kev kawm. Nws yog haum rau txawv scenarios: DWH, Sijhawm Series, Log Cia. Tab sis nws tsis yog OLTP database, yog li tsis txhob sim ua luv luv thiab nyeem muaj.
• Nthuav. Tej zaum tus uas ua haujlwm nrog Nyem Tsev, tau ntsib ntau lub sijhawm nthuav dav hauv qhov zoo thiab qhov tsis zoo. Piv txwv li, kev tso tawm tshiab tawm, txhua yam tsis ua haujlwm. Los yog thaum koj tawm tsam nrog ib txoj haujlwm rau ob hnub, tab sis tom qab nug cov lus nug hauv Telegram tham, txoj haujlwm tau daws hauv ob feeb. Los yog zoo li ntawm lub rooj sib tham ntawm tsab ntawv ceeb toom ntawm Lesha Milovidov, ib qho screenshot ntawm Nyem Tsev tsoo qhov kev tshaj tawm HighLoad ++. Tej yam zoo li no tshwm sim txhua lub sijhawm thiab ua rau peb lub neej nyuaj. Nyem Tsev kaj thiab nthuav!

Koj tuaj yeem saib qhov kev nthuav qhia no.

Lub rooj sib tham ntev tos ntawm developers ntawm high-load systems ntawm HighLoad ++ yuav muaj nyob rau lub Kaum Ib Hlis 9 thiab 10 hauv Skolkovo. Thaum kawg, qhov no yuav yog lub rooj sib tham offline (txawm tias nrog txhua qhov kev ceev faj hauv qhov chaw), vim tias lub zog ntawm HighLoad ++ tsis tuaj yeem ntim hauv online.

Rau lub rooj sab laj, peb pom thiab qhia koj cov xwm txheej txog lub peev xwm siab tshaj plaws ntawm thev naus laus zis: HighLoad ++ yog, thiab yuav yog tib qho chaw uas koj tuaj yeem kawm hauv ob hnub li cas Facebook, Yandex, VKontakte, Google thiab Amazon ua haujlwm.

Tau tuav peb lub rooj sib tham tsis muaj kev cuam tshuam txij li xyoo 2007, xyoo no peb yuav sib ntsib zaum 14. Lub sijhawm no, lub rooj sib tham tau loj hlob 10 zaug; xyoo tas los, qhov kev lag luam tseem ceeb tau coj los ua ke 3339 tus neeg tuaj koom, 165 tus neeg hais lus, cov lus ceeb toom thiab kev sib tham, thiab 16 lem tau khiav ib txhij.
Xyoo tas los muaj 20 lub tsheb npav, 5280 litres tshuaj yej thiab kas fes, 1650 litres dej qab zib thiab 10200 fwj dej. Thiab lwm 2640 kilograms zaub mov, 16 daim phiaj thiab 000 khob. Los ntawm txoj kev, nrog cov nyiaj tau los ntawm cov ntawv rov ua dua, peb cog 25 oak seedlings :)

Koj tuaj yeem yuav daim pib no, tau xov xwm txog lub rooj sab laj - no, thiab tham hauv txhua qhov social network: Telegram, Facebook, Vkontakte и Twitter.

Tau qhov twg los: www.hab.com