Nyob zoo txhua tus, kuv lub npe yog Alexander, Kuv ua haujlwm ntawm CIAN ua tus kws tshaj lij thiab koom nrog kev tswj hwm kev tswj hwm thiab kev ua haujlwm ntawm cov txheej txheem txheej txheem. Hauv cov lus hais rau ib qho ntawm cov kab lus dhau los, peb tau hais kom qhia qhov twg peb tau txais 4 TB ntawm cov cav hauv ib hnub thiab peb ua dab tsi nrog lawv. Yog lawm, peb muaj ntau lub cav, thiab cov txheej txheem cais tau tsim los ua cov txheej txheem, uas tso cai rau peb daws teeb meem sai. Nyob rau hauv tsab xov xwm no kuv yuav tham txog yuav ua li cas peb yoog nws nyob rau hauv ib lub xyoo los ua hauj lwm nrog ib tug puas-loj hlob ntws ntawm cov ntaub ntawv.
Peb pib qhov twg?
Ob peb xyoos dhau los, kev thauj khoom ntawm cian.ru tau loj hlob sai heev, thiab los ntawm peb lub hlis twg ntawm 2018, kev siv peev txheej mus txog 11.2 lab tus neeg siv tshwj xeeb hauv ib hlis. Lub sijhawm ntawd, thaum lub sijhawm tseem ceeb peb poob txog 40% ntawm cov cav, uas yog vim li cas peb tsis tuaj yeem daws qhov xwm txheej sai thiab siv sijhawm ntau thiab siv zog los daws lawv. Peb kuj feem ntau nrhiav tsis tau qhov laj thawj ntawm qhov teeb meem, thiab nws yuav rov tshwm sim tom qab qee lub sijhawm. Nws yog ntuj txiag teb tsaus thiab ib yam dab tsi yuav tsum tau ua txog nws.
Lub sijhawm ntawd, peb siv pawg ntawm 10 cov ntaub ntawv nrog ElasticSearch version 5.5.2 nrog cov qauv ntsuas ntsuas los khaws cov cav. Nws tau tshaj tawm ntau tshaj li ib xyoos dhau los ua ib qho kev daws teeb meem nrov thiab pheej yig: tom qab ntawd qhov ntws ntawm cov cav ntoo tsis loj heev, tsis muaj qhov taw qhia los ntawm cov txheej txheem tsis zoo.
Kev ua tiav ntawm cov cav nkag tau muab los ntawm Logstash ntawm cov chaw nres nkoj sib txawv ntawm tsib tus neeg ua haujlwm ElasticSearch. Ib qhov ntsuas, tsis hais qhov loj me, muaj tsib shards. Ib teev thiab txhua hnub kev sib hloov tau teeb tsa, vim li ntawd, muaj txog 100 tus tshiab shards tshwm sim hauv pawg txhua teev. Txawm hais tias tsis muaj ntau lub cav, pawg tswj hwm tau zoo thiab tsis muaj leej twg them rau nws qhov chaw.
Cov teeb meem ntawm kev loj hlob sai
Qhov ntim ntawm cov cav tsim tawm tau loj hlob sai heev, vim tias ob txoj kev sib tshooj sib tshooj. Ntawm qhov tod tes, tus naj npawb ntawm cov neeg siv cov kev pabcuam tau loj hlob. Ntawm qhov tod tes, peb pib nquag hloov mus rau microservice architecture, pom peb cov qub monoliths hauv C # thiab Python. Ntau lub kaum os tshiab microservices uas hloov qhov chaw ntawm lub monolith generated ho ntau log rau infrastructure pawg.
Nws tau scaling uas coj peb mus rau qhov chaw uas pawg tau los ua ib qho kev tswj tsis tau. Thaum cov cav pib tuaj txog ntawm tus nqi ntawm 20 txhiab cov lus ib ob, tsis tshua muaj txiaj ntsig kev sib hloov tau nce tus naj npawb ntawm shards mus rau 6 txhiab, thiab muaj ntau tshaj 600 shards ib ntawm.
Qhov no ua rau muaj teeb meem nrog kev faib tawm ntawm RAM, thiab thaum lub pob tawg, tag nrho cov shards pib txav mus ib txhij, sib faib cov tsheb thiab thauj lwm cov nodes, uas ua rau nws yuav luag tsis tuaj yeem sau cov ntaub ntawv rau pawg. Thiab thaum lub sij hawm no peb tau tawm tsis muaj cav. Thiab yog tias muaj teeb meem nrog lub server, peb pib poob 1/10 ntawm pawg. Ib tug loj tus naj npawb ntawm me indexes ntxiv complexity.
Yog tsis muaj cov cav, peb tsis nkag siab qhov laj thawj ntawm qhov xwm txheej thiab tuaj yeem sai lossis tom qab los ntawm qhov qub qub dua, thiab hauv kev xav ntawm peb pab pawg no yog qhov tsis txaus ntseeg, vim tias tag nrho peb cov txheej txheem kev ua haujlwm tau tsim los ua qhov sib txawv - tsis txhob rov ua dua. tib yam teeb meem. Txhawm rau ua qhov no, peb xav tau tag nrho cov ntim ntawm cov cav thiab lawv cov khoom xa tuaj yuav luag nyob rau lub sijhawm, txij li pab pawg ntawm cov kws ua haujlwm ua haujlwm tau saib xyuas cov lus ceeb toom tsis yog los ntawm kev ntsuas, tab sis kuj los ntawm cov cav. Txhawm rau kom nkag siab qhov ntsuas ntawm qhov teeb meem, lub sijhawm ntawd tag nrho cov ntaub ntawv teev tseg yog kwv yees li 2 TB ib hnub.
Peb teem lub hom phiaj kom tshem tawm tag nrho cov kev poob ntawm cov cav thiab txo lub sij hawm ntawm lawv cov khoom xa mus rau ELK pawg kom siab tshaj 15 feeb thaum lub sij hawm quab yuam majeure (peb tom qab tso siab rau daim duab no ua ib qho KPI sab hauv).
Tshiab kev sib hloov mechanism thiab kub-sov nodes
Peb pib qhov hloov pauv pawg los ntawm kev hloov kho ElasticSearch version ntawm 5.5.2 txog 6.4.3. Ib zaug ntxiv peb version 5 pawg tau tuag, thiab peb txiav txim siab kaw nws thiab hloov kho nws tag nrho - tseem tsis muaj cov ntawv teev tseg. Yog li peb tau ua qhov kev hloov pauv no hauv ob peb teev xwb.
Qhov kev hloov pauv loj tshaj plaws nyob rau theem no yog kev siv Apache Kafka ntawm peb lub nodes nrog tus neeg koom tes ua qhov tsis sib xws. Cov lus broker tau cawm peb ntawm kev poob lub cav thaum muaj teeb meem nrog ElasticSearch. Nyob rau tib lub sijhawm, peb ntxiv 2 nodes rau pawg thiab hloov mus rau qhov kub-sov architecture nrog peb "kub" nodes nyob rau hauv sib txawv racks nyob rau hauv cov ntaub ntawv chaw. Peb tau xa cov ntawv xa mus rau lawv siv lub npog ntsej muag uas yuav tsum tsis txhob poob nyob rau hauv ib qho xwm txheej - nginx, nrog rau daim ntawv thov yuam kev. Cov ntaub ntawv me me raug xa mus rau qhov seem ntawm cov nodes - debug, ceeb toom, thiab lwm yam, thiab tom qab 24 teev, "tseem ceeb" cav los ntawm "kub" nodes raug xa mus.
Txhawm rau kom tsis txhob nce tus naj npawb ntawm cov ntsuas me me, peb hloov ntawm lub sijhawm hloov mus rau lub tshuab rollover. Muaj ntau cov ntaub ntawv nyob rau hauv lub rooj sab laj hais tias kev sib hloov los ntawm qhov Performance index loj yog heev unreliable, yog li ntawd peb txiav txim siab siv kev sib hloov los ntawm cov ntaub ntawv nyob rau hauv lub Performance index. Peb txheeb xyuas txhua qhov ntsuas thiab sau cov lej ntawm cov ntaub ntawv tom qab qhov kev sib hloov yuav tsum ua haujlwm. Yog li, peb tau mus txog qhov zoo tshaj plaws shard loj - tsis ntau tshaj 50 GB.
Cluster optimization
Txawm li cas los xij, peb tsis tau tshem tag nrho cov teeb meem. Hmoov tsis zoo, qhov ntsuas me me tseem tshwm sim: lawv tsis ncav cuag lub ntim ntim, tsis tau tig, thiab raug tshem tawm los ntawm kev ntxuav thoob ntiaj teb ntawm cov ntsuas qis dua peb hnub, txij li peb tshem tawm kev sib hloov los ntawm hnub tim. Qhov no ua rau cov ntaub ntawv poob vim qhov tseeb tias qhov Performance index ntawm pawg tau ploj mus tag nrho, thiab kev sim sau ntawv mus rau qhov tsis muaj qhov Performance index tsoo lub logic ntawm tus curator uas peb siv los tswj. Alias ββββrau kev sau ntawv tau hloov dua siab tshiab rau hauv qhov ntsuas thiab rhuav tshem lub logic logic, ua rau tsis muaj kev tswj hwm kev loj hlob ntawm qee qhov ntsuas txog li 600 GB.
Piv txwv li, rau kev sib hloov config:
Ρurator-elk-rollover.yaml
---
actions:
1:
action: rollover
options:
name: "nginx_write"
conditions:
max_docs: 100000000
2:
action: rollover
options:
name: "python_error_write"
conditions:
max_docs: 10000000
Yog tias tsis muaj rollover alias, qhov yuam kev tshwm sim:
ERROR alias "nginx_write" not found.
ERROR Failed to complete action: rollover. <type 'exceptions.ValueError'>: Unable to perform index rollover with alias "nginx_write".
Peb tau tso txoj kev daws teeb meem rau qhov teeb meem no rau qhov kev rov ua dua tom ntej thiab coj mus rau lwm qhov teeb meem: peb hloov mus rau lub logic rub tawm ntawm Logstash, uas ua cov ntaub ntawv nkag (tshem tawm cov ntaub ntawv tsis tsim nyog thiab txhawb nqa). Peb muab tso rau hauv docker, uas peb tso tawm ntawm docker-compose, thiab peb kuj tso logstash-exporter nyob rau ntawd, uas xa metrics mus rau Prometheus rau kev ua hauj lwm saib xyuas ntawm lub cav kwj. Txoj kev no peb tau muab lub sijhawm rau peb tus kheej kom hloov pauv tus lej ntawm cov logstash lub luag haujlwm rau kev ua txhua hom cav.
Thaum peb tab tom txhim kho pawg, cian.ru cov tsheb khiav mus rau 12,8 lab tus neeg siv tshwj xeeb hauv ib hlis. Raws li qhov tshwm sim, nws tau pom tias peb cov kev hloov pauv me ntsis tom qab cov kev hloov pauv hauv kev tsim khoom, thiab peb tau ntsib nrog qhov tseeb tias "sov" nodes tsis tuaj yeem tiv nrog lub nra thiab ua rau tag nrho cov khoom xa tuaj. Peb tau txais cov ntaub ntawv "kub" yam tsis muaj qhov ua tsis tiav, tab sis peb yuav tsum tau cuam tshuam rau kev xa khoom ntawm tus so thiab ua phau ntawv rollover txhawm rau txhawm rau faib cov lej sib npaug.
Nyob rau tib lub sij hawm, scaling thiab hloov qhov chaw ntawm logstash piv txwv nyob rau hauv pawg yog nyuaj los ntawm qhov tseeb hais tias nws yog ib tug hauv zos docker-compose, thiab tag nrho cov ua tau ua manually (kom ntxiv cov tshiab xaus, nws yog tsim nyog los manually mus rau tag nrho cov. cov servers thiab ua docker-compose li -d txhua qhov chaw).
Kev faib tawm ntawm lub log
Thaum lub Cuaj Hlis ntawm lub xyoo no, peb tseem tab tom txiav cov monolith, cov khoom thauj ntawm pawg tau nce, thiab qhov ntws ntawm cov cav tau nce mus txog 30 txhiab lus ib ob.
Peb pib qhov iteration tom ntej nrog kho vajtse hloov tshiab. Peb hloov los ntawm tsib tus neeg koom tes mus rau peb, hloov cov ntaub ntawv nodes thiab yeej ntawm cov nyiaj thiab qhov chaw cia. Rau cov nodes peb siv ob configurations:
- Rau "kub" nodes: E3-1270 v6 / 960Gb SSD / 32 Gb x 3 x 2 (3 rau Hot1 thiab 3 rau Hot2).
- Rau "sov" nodes: E3-1230 v6 / 4Tb SSD / 32 Gb x 4.
Ntawm qhov kev rov hais dua no, peb tau txav qhov Performance index nrog cov ntaub ntawv nkag ntawm microservices, uas siv tib qhov chaw raws li cov kab hauv ntej nginx cav, mus rau pawg thib ob ntawm peb "kub" nodes. Tam sim no peb khaws cov ntaub ntawv ntawm "kub" nodes rau 20 teev, thiab tom qab ntawd hloov lawv mus rau "sov" nodes mus rau seem ntawm cov cav.
Peb tau daws qhov teeb meem ntawm qhov ntsuas me me ploj los ntawm kev teeb tsa lawv cov kev sib hloov. Tam sim no cov indexes tau tig txhua 23 teev nyob rau hauv txhua rooj plaub, txawm tias muaj cov ntaub ntawv me me nyob ntawd. Qhov no me ntsis nce tus naj npawb ntawm shards (muaj txog 800 ntawm lawv), tab sis los ntawm qhov pom ntawm kev ua tau zoo ntawm pawg nws yog tolerable.
Yog li ntawd, muaj rau "kub" thiab tsuas yog plaub "sov" nodes hauv pawg. Qhov no ua rau qeeb me ntsis ntawm kev thov nyob rau lub sijhawm ntev, tab sis nce tus naj npawb ntawm cov nodes yav tom ntej yuav daws qhov teeb meem no.
Qhov kev rov ua dua no kuj tau kho qhov teeb meem ntawm qhov tsis muaj qhov tsis siv neeg qhov ntsuas qhov ntsuas. Txhawm rau ua qhov no, peb tau siv cov txheej txheem Nomad pawg - zoo ib yam li peb twb tau xa mus rau hauv kev tsim khoom. Txog tam sim no, tus nqi ntawm Logstash tsis tuaj yeem hloov pauv nyob ntawm qhov thauj khoom, tab sis peb yuav tuaj rau qhov no.
Cov tswv yim rau yav tom ntej
Cov kev teeb tsa kev teeb tsa tau ua tiav zoo kawg nkaus, thiab tam sim no peb khaws 13,3 TB ntawm cov ntaub ntawv - tag nrho cov ntawv teev tseg rau 4 hnub, uas yog qhov tsim nyog rau kev soj ntsuam xwm txheej ceev ntawm kev ceeb toom. Peb hloov qee cov cav rau hauv cov ntsuas, uas peb ntxiv rau Graphite. Txhawm rau ua kom txoj haujlwm ntawm cov kws ua haujlwm tau yooj yim dua, peb muaj cov ntsuas kev ntsuas rau cov txheej txheem hauv pawg thiab cov ntawv sau rau kev kho ib nrab ntawm cov teeb meem tshwm sim. Tom qab nce tus naj npawb ntawm cov ntaub ntawv nodes, uas tau npaj rau xyoo tom ntej, peb yuav hloov mus rau cov ntaub ntawv khaws cia ntawm 4 mus rau 7 hnub. Qhov no yuav txaus rau kev ua haujlwm ua haujlwm, txij li peb ib txwm sim tshawb xyuas qhov xwm txheej sai li sai tau, thiab rau kev tshawb nrhiav mus sij hawm ntev muaj cov ntaub ntawv telemetry.
Thaum Lub Kaum Hli 2019, kev khiav mus rau cian.ru twb tau nce mus rau 15,3 lab tus neeg siv tshwj xeeb hauv ib hlis. Qhov no tau dhau los ua qhov kev sim loj ntawm kev tsim vaj tsev rau kev xa cov cav.
Tam sim no peb tab tom npaj hloov kho ElasticSearch rau version 7. Txawm li cas los xij, rau qhov no peb yuav tau hloov kho daim ntawv qhia ntawm ntau qhov ntsuas hauv ElasticSearch, txij li lawv tau tsiv tawm ntawm version 5.5 thiab tau tshaj tawm tias tsis muaj nyob hauv version 6 (lawv tsuas yog tsis muaj nyob hauv version. 7). Qhov no txhais tau hais tias thaum lub sijhawm hloov tshiab yuav muaj qee yam kev quab yuam majeure, uas yuav tawm hauv peb yam tsis muaj cav thaum qhov teeb meem raug daws. Ntawm version 7, peb tab tom tos ntsoov rau Kibana nrog kev txhim kho interface thiab cov ntxaij lim dej tshiab.
Peb tau ua tiav peb lub hom phiaj tseem ceeb: peb nres poob cov cav thiab txo cov sij hawm poob ntawm cov txheej txheem hauv pawg los ntawm 2-3 kev sib tsoo hauv ib lub lis piam mus rau ob peb teev ntawm kev saib xyuas haujlwm hauv ib hlis. Tag nrho cov haujlwm no hauv kev tsim khoom yuav luag tsis pom. Txawm li cas los xij, tam sim no peb tuaj yeem txiav txim siab tias yuav ua li cas nrog peb cov kev pabcuam, peb tuaj yeem ua sai sai rau hauv hom ntsiag to thiab tsis txhawj xeeb tias cov cav yuav ploj. Feem ntau, peb txaus siab, zoo siab thiab npaj rau kev siv tshiab, uas peb yuav tham txog tom qab.
Tau qhov twg los: www.hab.com