Cov log yog ib feem tseem ceeb ntawm lub kaw lus, tso cai rau koj nkag siab tias nws ua haujlwm (lossis tsis ua haujlwm) raws li qhov xav tau. Raws li cov xwm txheej ntawm microservice architecture, kev ua hauj lwm nrog cov cav yuav ua ib qho kev qhuab qhia ntawm Special Olympiad. Muaj ntau yam teeb meem uas yuav tsum tau daws:
- yuav ua li cas sau cov cav los ntawm daim ntawv thov;
- qhov twg sau cov cav;
- yuav ua li cas xa cov cav rau kev khaws cia thiab ua tiav;
- yuav ua li cas thiab khaws cov cav.
Kev siv cov tshuab ntim khoom nrov tam sim no ntxiv cov xuab zeb nyob rau sab saum toj ntawm rake hauv thaj chaw ntawm kev daws teeb meem.
Tsuas yog hais txog qhov no yog cov ntawv sau tseg ntawm tsab ntawv ceeb toom los ntawm Yuri Bushmelev "Daim ntawv qhia ntawm lub rake hauv kev sau thiab xa cov cav"
Leej twg mob siab, thov nyob rau hauv tus miv.
Kuv lub npe yog Yuri Bushmelev. Kuv ua haujlwm rau Lazada. Hnub no kuv yuav tham txog yuav ua li cas peb ua peb cov cav, peb khaws lawv li cas, thiab peb sau dab tsi rau ntawd.
Peb nyob qhov twg? Peb yog leej twg? Lazada yog tus #1 cov khw muag khoom online hauv rau lub tebchaws nyob rau sab hnub tuaj Asia. Tag nrho cov teb chaws no tau muab faib rau hauv cov chaw khaws ntaub ntawv. Tam sim no muaj 4 lub chaw khaws ntaub ntawv tag nrho, vim li cas qhov no tseem ceeb? Vim tias qee qhov kev txiav txim siab yog vim muaj qhov sib txuas tsis muaj zog heev ntawm cov chaw zov me nyuam. Peb muaj microservice architecture. Kuv xav tsis thoob thaum pom tias peb twb muaj 80 microservices. Thaum kuv pib ua haujlwm nrog cov cav, tsuas muaj 20 ntawm lawv. Ntxiv rau, muaj ib qho loj heev ntawm PHP legacy, uas kuv kuj yuav tsum tau nyob nrog thiab muab tso rau. Tag nrho cov no ua rau peb tam sim no ntau dua 6 lab cov lus hauv ib feeb rau lub kaw lus tag nrho. Ntxiv mus, kuv yuav qhia tias peb tab tom sim ua neej nyob nrog qhov no, thiab vim li cas qhov no yog li ntawd.
Koj yuav tsum nyob nrog 6 lab cov lus no. Peb yuav ua li cas nrog lawv? 6 lab cov lus xav tau:
- xa los ntawm app
- txais khoom xa tuaj
- xa mus rau kev txheeb xyuas thiab khaws cia.
- txheeb xyuas
- khaws qee yam.
Thaum muaj peb lab cov lus, kuv muaj txog qhov zoo ib yam. Vim peb pib nrog ib co pennies. Nws yog qhov tseeb tias daim ntawv thov teev npe tau sau rau ntawd. Piv txwv li, tsis tuaj yeem txuas mus rau database, tuaj yeem txuas mus rau database, tab sis nyeem tsis tau ib yam dab tsi. Tab sis dua li qhov no, txhua qhov ntawm peb cov microservices kuj sau ib qho kev nkag nkag. Txhua qhov kev thov uas tuaj txog ntawm microservice poob rau hauv lub cav. Vim li cas peb thiaj ua li no? Cov neeg tsim khoom xav kom muaj peev xwm taug qab. Txhua qhov nkag nkag muaj cov traceid teb, raws li qhov tshwj xeeb interface ces unwinds tag nrho cov saw thiab zoo nkauj qhia cov kab. Cov kab mob qhia tau hais tias qhov kev thov mus li cas, thiab qhov no yuav pab peb cov neeg tsim khoom lag luam nrog cov khib nyiab tsis paub sai dua.
Yuav ua li cas nrog nws? Tam sim no kuv yuav piav qhia luv luv txog thaj chaw ntawm kev xaiv - qhov teeb meem no feem ntau daws tau li cas. Yuav daws qhov teeb meem ntawm kev sau, hloov pauv thiab khaws cov cav li cas.
Yuav sau li cas los ntawm daim ntawv thov? Nws yog qhov tseeb tias muaj ntau txoj hauv kev. Tshwj xeeb, muaj kev xyaum zoo tshaj plaws, raws li cov phooj ywg fashionable qhia peb. Muaj ob hom tsev kawm qub, raws li yawg hais. Muaj lwm txoj kev.
Nrog rau kev sau cov cav, qhov xwm txheej yog kwv yees li qub. Tsis muaj ntau txoj hauv kev los daws qhov tshwj xeeb no. Muaj ntau ntawm lawv, tab sis tsis muaj ntau.
Tab sis nrog kev xa khoom thiab kev soj ntsuam tom qab, tus naj npawb ntawm kev hloov pauv pib tawg. Kuv yuav tsis piav txog txhua qhov kev xaiv tam sim no. Kuv xav tias cov kev xaiv tseem ceeb yog paub zoo rau txhua tus neeg uas nyiam lub ntsiab lus.
Kuv mam li qhia koj seb peb tau ua li cas ntawm Lazada thiab nws pib li cas.
Ib xyoo dhau los, kuv tuaj rau Lazada thiab raug xa mus rau qhov project log. Nws zoo li no nyob ntawd. Lub cav los ntawm daim ntawv thov raug sau rau stdout thiab stderr. Txhua yam tau ua tiav hauv txoj kev zam. Tab sis tom qab ntawd cov neeg tsim khoom pov nws tawm ntawm tus qauv kwj, thiab tom qab ntawd cov kws tshaj lij kev tsim kho vaj tsev yuav txiav txim siab nws li cas. Nruab nrab ntawm cov kws tshaj lij kev tsim kho vaj tse thiab cov neeg tsim khoom, kuj tseem muaj cov neeg tshaj tawm uas tau hais tias: "uh ... zoo, cia li qhwv lawv rau hauv cov ntaub ntawv nrog lub plhaub, thiab qhov ntawd yog nws." Thiab txij li tag nrho cov no yog nyob rau hauv ib lub thawv, lawv qhwv nws txoj cai nyob rau hauv lub thawv nws tus kheej, mapped lub directory nyob rau hauv thiab muab tso rau nws. Kuv xav tias nws yog qhov pom tseeb zoo rau txhua tus uas tau tshwm sim.
Cia peb saib ntxiv me ntsis. Yuav ua li cas peb xa cov cav no. Ib tug neeg tau xaiv td-tus neeg sawv cev, uas yog qhov ua tau zoo tab sis tsis tau zoo heev. Kuv tseem tsis nkag siab txog kev sib raug zoo ntawm ob txoj haujlwm no, tab sis lawv zoo li hais txog tib yam. Thiab qhov no tau zoo, sau rau hauv Ruby, nyeem cov ntaub ntawv teev tseg, txheeb xyuas lawv rau hauv JSON siv qee cov lus hais tsis tu ncua. Tom qab ntawd lawv raug xa mus rau Kafka. Ntxiv mus, hauv Kafka, peb muaj 4 cais cov ncauj lus rau txhua API. Vim li cas 4? Vim muaj nyob, muaj staging, thiab vim muaj stdout thiab stderr. Cov neeg tsim khoom tsim lawv, thiab cov neeg ua haujlwm tsim vaj tsev yuav tsum tsim lawv hauv Kafka. Ntxiv mus, Kafka tau tswj hwm los ntawm lwm lub tuam tsev. Yog li ntawd, nws yog ib qho tsim nyog los tsim ib daim pib kom lawv tsim 4 lub ntsiab lus muaj rau txhua api. Sawv daws tsis nco qab txog nws. Feem ntau, nws yog thoob khib nyiab thiab pov tseg.
Peb tau ua dab tsi ntxiv nrog nws? Peb xa nws mus rau kafka. Tsis tas li ntawd los ntawm Kafka, ib nrab ntawm lub cav ya mus rau Logstash. Lwm ib nrab ntawm cov cav tau muab faib. Ib txhia ya mus rau ib tug Greylog, ib txhia mus rau lwm Graylog. Yog li ntawd, tag nrho cov no tau ya mus rau hauv ib pawg Elasticsearch. Ntawd yog, tag nrho cov kev ntxhov siab no poob rau hauv qhov kawg. Koj tsis tas yuav ua li ntawd!
Qhov no yog qhov zoo li thaum saib los ntawm saum toj no. Koj tsis tas yuav ua li ntawd! Ntawm no, thaj chaw teeb meem raug cim tam sim nrog cov lej. Muaj ntau ntau ntawm lawv, tab sis 6 yog cov teeb meem tiag tiag, uas ib yam dab tsi yuav tsum tau ua. Kuv yuav qhia txog lawv cais tam sim no.
Ntawm no (1,2,3) peb sau cov ntaub ntawv thiab, raws li, muaj peb rakes ntawm no ib zaug.
Thawj (1) yog tias peb yuav tsum tau sau lawv qhov chaw. Nws tsis yog ib txwm xav muab API lub peev xwm los sau ncaj qha rau cov ntaub ntawv. Nws yog qhov xav tau tias API raug cais tawm hauv ib lub thawv, thiab tseem zoo dua, uas nws yuav tsum tau nyeem nkaus xwb. Kuv yog tus tswj hwm qhov system, yog li kuv muaj lwm qhov kev pom me ntsis ntawm cov khoom no.
Lub ntsiab lus thib ob (2,3) yog tias peb muaj ntau qhov kev thov tuaj rau API. API sau ntau cov ntaub ntawv rau ib cov ntaub ntawv. Cov ntaub ntawv loj hlob tuaj. Peb yuav tsum tig lawv. Vim tsis li ntawd koj yuav tsis muaj peev xwm txuag tau ib qho discs nyob ntawd. Tig lawv yog qhov tsis zoo vim tias lawv raug xa rov qab los ntawm lub plhaub mus rau ib daim ntawv teev npe. Tsis muaj txoj hauv kev uas peb tuaj yeem tig nws. Koj tsis tuaj yeem qhia rau daim ntawv thov kom rov qhib cov tes. Vim tias cov neeg tsim khoom yuav saib koj zoo li tus neeg ruam: "Dab tsi piav qhia? Peb feem ntau sau rau stdout. Lub moj khaum ua copytruncate rau hauv logrotate, uas tsuas yog ua ib daim qauv ntawm cov ntaub ntawv thiab trunks tus thawj. Raws li, ntawm cov txheej txheem theej no, qhov chaw disk feem ntau khiav tawm.
(4) Peb muaj cov qauv sib txawv hauv APIs sib txawv. Lawv txawv me ntsis, tab sis regexp yuav tsum tau sau txawv. Txij li thaum nws tau tswj hwm los ntawm Puppet, muaj ntau pawg ntawm cov chav kawm nrog lawv tus kheej cov kab laum. Ntxiv rau, td-tus neeg sawv cev feem ntau tuaj yeem noj nco, ua neeg ruam, nws tuaj yeem ua txuj tias nws ua haujlwm thiab tsis ua dab tsi. Outwardly, nws tsis tuaj yeem nkag siab tias nws tsis ua dab tsi. Qhov zoo tshaj plaws, nws yuav poob, thiab ib tug neeg yuav tuaj tos nws tom qab. Ntau precisely, ib tug ceeb toom yuav ya nyob rau hauv, thiab ib tug neeg yuav mus thiab tsa nws nrog lawv txhais tes.
(6) Thiab feem ntau cov thoob khib nyiab thiab pov tseg - nws yog elasticsearch. Vim nws yog ib tug qub version. Vim tias peb tsis muaj tus tswv rau lub sijhawm ntawd. Peb muaj heterogeneous cav uas nws teb yuav sib tshooj. Cov cav sib txawv ntawm cov ntawv thov sib txawv tuaj yeem sau nrog tib lub npe, tab sis tib lub sijhawm tuaj yeem muaj cov ntaub ntawv sib txawv hauv. Ntawd yog, ib lub cav tuaj nrog Integer hauv ib daim teb, piv txwv li, qib. Lwm lub cav los nrog ib txoj hlua nyob rau hauv theem teb. Nyob rau hauv tsis muaj daim ntawv qhia zoo li qub, xws li ib qho zoo kawg nkaus hloov tawm. Yog tias, tom qab qhov ntsuas kev sib hloov, cov lus nrog ib txoj hlua tuaj txog ua ntej hauv elasticsearch, ces peb nyob li qub. Thiab yog tias thawj tus tuaj txog nrog Integer, ces tag nrho cov lus tom ntej uas tuaj txog nrog String tsuas yog muab pov tseg. Vim tias hom teb tsis sib xws.
Peb pib nug cov lus nug no. Peb txiav txim siab tsis mus nrhiav cov neeg txhaum.
Tab sis ib yam dab tsi yuav tsum tau ua! Qhov pom tseeb yog tias peb yuav tsum tsim cov qauv. Peb twb muaj ib co qauv. Ib txhia peb coj me ntsis tom qab. Hmoov zoo, ib daim ntawv teev npe rau txhua tus APIs twb tau pom zoo nyob rau lub sijhawm ntawd. Nws yog sau ncaj qha rau hauv cov qauv kev pabcuam kev sib tshuam. Yog li ntawd, cov neeg uas xav tau txais cov cav yuav tsum sau lawv hauv hom ntawv no. Yog hais tias ib tug neeg tsis sau cav hauv hom ntawv no, ces peb tsis lav dab tsi.
Tsis tas li ntawd, kuv xav kom muaj ib qho qauv rau cov txheej txheem ntawm kev sau, xa thiab sau cov cav. Qhov tseeb, qhov twg sau lawv, thiab yuav xa lawv li cas. Qhov xwm txheej zoo tshaj yog thaum tej yaam num siv tib lub tsev qiv ntawv. Muaj ib lub tsev qiv ntawv cais rau Go, muaj ib lub tsev qiv ntawv cais rau PHP. Peb txhua tus muaj, txhua tus yuav tsum siv lawv. Tam sim no, kuv xav hais tias peb ua tiav 80 feem pua. Tab sis ib txhia tseem noj cacti.
Thiab muaj (ntawm tus swb) "SLA rau kev xa khoom" nyuam qhuav pib tshwm sim. Nws tseem tsis tau muaj, tab sis peb tab tom ua haujlwm rau nws. Vim tias nws yooj yim heev thaum infra hais tias yog tias koj sau cov ntawv zoo li no rau qhov chaw zoo li no thiab tsis pub ntau tshaj N cov lus ib ob, ces peb yuav xa tuaj rau ntawd. Nws yuav siv ntau qhov mob taub hau. Yog tias muaj SLA, ces nws zoo heev!
Peb pib daws qhov teeb meem li cas? Lub ntsiab rake yog nrog td-tus neeg sawv cev. Nws tsis paub meej tias peb cov cav yuav mus qhov twg. Puas yog lawv xa? Lawv puas mus? Lawv nyob qhov twg lawm? Yog li ntawd, nws tau txiav txim siab los hloov td-tus neeg sawv cev nrog thawj yam khoom. Kev xaiv rau dab tsi los hloov nws nrog, Kuv tau piav qhia luv luv ntawm no.
Fluentd. Ua ntej, kuv tau hla nws ntawm txoj haujlwm dhau los, thiab nws kuj tau poob rau ntawd. Qhov thib ob, qhov no yog qhov qub, tsuas yog hauv profile.
filebeat. Nws zoo li cas rau peb? Qhov tseeb tias nws nyob hauv Go, thiab peb muaj kev txawj ntse hauv Go. Raws li, yog tias muaj dab tsi, peb tuaj yeem ua qee yam ntxiv rau peb tus kheej. Yog vim li cas peb tsis coj nws. Yog li ntawd yuav tsis muaj kev sim siab los pib rov sau dua rau koj tus kheej.
Txoj kev daws teeb meem rau sysadmin yog txhua hom syslogs hauv qhov ntau (syslog-ng/rsyslog/nxlog).
Los yog sau ib yam dab tsi ntawm koj tus kheej, tab sis peb muab pov tseg, nrog rau filebeat. Yog tias koj sau ib yam dab tsi, ces nws yog qhov zoo dua los sau qee yam muaj txiaj ntsig rau kev lag luam. Txhawm rau xa cov cav, nws yog qhov zoo dua los nqa ib yam dab tsi npaj txhij.
Yog li ntawd, qhov kev xaiv tau nqis los rau kev xaiv ntawm syslog-ng thiab rsyslog. Kuv leaned ntawm rsyslog tsuas yog vim peb twb muaj cov chav kawm rau rsyslog hauv Puppet, thiab kuv tsis pom qhov txawv ntawm lawv. syslog yog dab tsi, dab tsi yog syslog. Yog, qee cov ntaub ntawv phem dua, qee qhov zoo dua. Nws paub li no, thiab nws ua nws txawv.
Thiab me ntsis txog rsyslog. Ua ntej, nws txias vim nws muaj ntau modules. Nws muaj tib neeg-nyeem tau RainerScript (hnub no configuration lus). Ib qho txiaj ntsig zoo yog tias peb tuaj yeem ua raws li tus cwj pwm ntawm td-tus neeg sawv cev nrog nws cov cuab yeej txheem, thiab tsis muaj dab tsi hloov pauv rau daim ntawv thov. Ntawd yog, peb hloov td-tus neeg sawv cev rau rsyslog, thiab tsis txhob kov txhua yam tsis tau. Thiab tam sim ntawd peb tau txais kev xa khoom ua haujlwm. Tom ntej no, mmnormalize yog qhov txias tshaj plaws txog rsyslog. Nws tso cai rau koj txheeb xyuas cov cav, tab sis tsis nrog Grok thiab regexp. Nws ua ib tsob ntoo abstract syntax. Nws parses cav nyob rau hauv ntau tib txoj kev uas ib tug compiler parses qhov chaws. Qhov no tso cai rau koj ua haujlwm nrawm heev, noj CPU me me, thiab, feem ntau, nws tsuas yog qhov txias heev. Muaj ib pawg ntawm lwm cov nyiaj tshwj xeeb. Kuv yuav tsis nyob ntawm lawv.
rsyslog muaj ntau qhov tsis zoo. Lawv hais txog tib yam li nyiaj tshwj xeeb. Cov teeb meem tseem ceeb yog tias koj yuav tsum muaj peev xwm ua noj, thiab koj yuav tsum xaiv ib qho version.
Peb txiav txim siab tias peb yuav sau cov cav hauv ib lub qhov (socket) unix. Thiab tsis nyob rau hauv / dev / log, vim hais tias muaj peb muaj ib tug mess ntawm lub cav cav, muaj cov phau ntawv journald nyob rau hauv no pipeline. Yog li cia peb sau mus rau lub qhov (socket) kev cai. Peb yuav muab nws tso rau hauv ib qho kev cai cais. Cia peb tsis txhob cuam tshuam dab tsi. Txhua yam yuav yog pob tshab thiab nkag siab. Yog li peb tau ua tiag tiag. Daim ntawv teev npe nrog cov qhov (socket) no yog tus qauv thiab xa mus rau txhua lub thawv. Cov thawv tuaj yeem pom lub qhov (socket) uas lawv xav tau, qhib thiab sau rau nws.
Vim li cas ho tsis ua ntaub ntawv? Vim sawv daws tau nyeem , uas tau sim xa cov ntaub ntawv mus rau docker, thiab pom tias tom qab rov pib rsyslog, cov ntaub ntawv piav qhia hloov pauv, thiab docker poob cov ntaub ntawv no. Nws qhib ib yam dab tsi ntxiv, tab sis tsis yog tib lub qhov (socket) uas lawv sau. Peb txiav txim siab tias peb yuav hla qhov teeb meem no, thiab, tib lub sijhawm, hla qhov teeb meem thaiv.
Rsyslog ua cov haujlwm uas tau teev tseg rau ntawm tus swb thiab xa cov cav mus rau relay lossis Kafka. Kafka ua raws li txoj kev qub. Rayleigh - Kuv sim siv cov ntshiab rsyslog xa cov cav. Tsis muaj Message Queue, siv cov cuab yeej txheem rsyslog. Yeej, nws ua haujlwm.
Tab sis muaj nuances nrog yuav ua li cas muab lawv tom qab rau hauv qhov no (Logstash/Graylog/ES). Qhov no (rsyslog-rsyslog) yog siv ntawm datacenters. Ntawm no yog qhov txuas tcp compressed, uas tso cai rau koj kom txuag tau bandwidth thiab, raws li, qee qhov yuav ua rau peb yuav tau txais qee cov cav los ntawm lwm qhov chaw cov ntaub ntawv thaum lub channel puv. Vim peb muaj Indonesia, qhov twg txhua yam yog phem. Qhov ntawd yog qhov teeb meem tas li nyob.
Peb xav txog yuav ua li cas peb saib xyuas, nrog qhov tshwm sim ntawm cov cav uas peb sau los ntawm daim ntawv thov mus txog qhov kawg? Peb txiav txim siab pib metrics. Rsyslog muaj nws tus kheej cov ntaub ntawv sau tseg, uas muaj qee yam ntawm cov txee. Piv txwv li, nws tuaj yeem qhia koj qhov luaj li cas ntawm cov kab, lossis pes tsawg cov lus tuaj rau hauv kev ua haujlwm zoo li no. Koj twb tau txais ib yam dab tsi los ntawm lawv. Ntxiv rau, nws muaj cov txee kev cai uas koj tuaj yeem teeb tsa, thiab nws yuav qhia koj, piv txwv li, cov lus uas qee qhov API tau sau tseg. Tom ntej no, kuv sau rsyslog_exporter hauv Python, thiab peb xa tag nrho rau Prometheus thiab npaj. Peb yeej xav tau Graylog metrics, tab sis txog tam sim no peb tsis muaj sijhawm los teeb tsa lawv.
Cov teeb meem yog dab tsi? Qhov teeb meem tshwm sim nrog qhov tseeb tias peb pom (TSUM TSUM!) uas peb nyob APIs sau 50k lus ib ob. Qhov no tsuas yog Live API yam tsis muaj kev tshaj tawm. Thiab Graylog tsuas qhia peb 12 txhiab lus ib ob. Thiab ib lo lus nug tsim nyog tau tshwm sim, cov seem nyob qhov twg? Los ntawm qhov peb tau xaus tias Graylog tsuas tuaj yeem tiv tsis tau. Peb tau saib, thiab, qhov tseeb, Greylog nrog Elasticsearch tsis tau paub qhov ntws no.
Tom ntej no, lwm yam discoveries peb tau ua raws txoj kev.
Sau rau lub qhov (socket) raug thaiv. Nws tshwm sim li cas? Thaum kuv siv rsyslog rau kev xa khoom, qee lub sijhawm peb tsoo cov channel ntawm cov chaw khaws ntaub ntawv. Kev xa khoom tau nce rau hauv ib qho chaw, kev xa khoom tau nce mus rau lwm qhov. Tag nrho cov no tau nqis los rau lub tshuab nrog APIs uas sau rau rsyslog socket. Muaj ib txoj kab. Tom qab ntawd cov kab ntawv sau rau lub qhov (socket) unix sau, uas yog lub neej ntawd yog 128 pob ntawv. Thiab tom ntej no sau () hauv daim ntawv thov blocks. Thaum peb saib lub tsev qiv ntawv uas peb siv hauv Go daim ntawv thov, nws tau sau rau ntawd tias kev sau ntawv rau lub qhov (socket) tshwm sim hauv hom tsis thaiv. Peb paub tseeb tias tsis muaj dab tsi thaiv. Vim peb tau nyeem leej twg sau txog nws. Tab sis muaj ib pliag. Kuj tseem muaj lub voj tsis kawg nyob ib ncig ntawm qhov kev hu no, nyob rau hauv uas muaj kev sim mus tas li thawb cov lus rau hauv lub qhov (socket). Peb tsis pom nws. Kuv yuav tsum rov sau lub tsev qiv ntawv. Txij thaum ntawd los, nws tau hloov ntau zaus, tab sis tam sim no peb tau tshem ntawm cov xauv hauv txhua qhov subsystems. Yog li ntawd, koj tuaj yeem nres rsyslog thiab tsis muaj dab tsi yuav poob.
Nws yog ib qho tsim nyog los saib xyuas qhov luaj li cas ntawm cov kab, uas yuav pab kom tsis txhob mus rau ntawm qhov rake no. Ua ntej, peb tuaj yeem saib xyuas thaum peb pib poob cov lus. Qhov thib ob, peb tuaj yeem saib xyuas tias peb pib muaj teeb meem nrog kev xa khoom.
Thiab lwm lub sijhawm tsis zoo - kev nthuav dav los ntawm 10 zaug hauv microservice architecture yog qhov yooj yim heev. Peb tsis muaj ntau qhov kev thov tuaj, tab sis vim yog cov duab kos uas cov lus no khiav mus ntxiv, vim tias cov ntawv nkag mus, peb tau nce cov load ntawm cov cav los ntawm kaum zaus. Hmoov tsis zoo, kuv tsis muaj sijhawm los suav cov lej, tab sis microservices yog dab tsi. Qhov no yuav tsum tau khaws cia rau hauv siab. Nws hloov tawm tias tam sim no lub log sau subsystem yog qhov thauj khoom tshaj plaws hauv Lazada.
Yuav daws qhov teeb meem elasticsearch li cas? Yog tias koj xav tau kom ceev cov cav nyob rau hauv ib qho chaw, kom tsis txhob khiav thoob plaws txhua lub tshuab thiab sau lawv rau ntawd, siv cov ntaub ntawv khaws cia. Qhov no yog guaranteed ua hauj lwm. Nws yog ua los ntawm txhua lub server. Koj tsuas yog yuav tsum tau lo disks nyob ntawd thiab muab syslog. Tom qab ntawd, koj tau lees tias muaj tag nrho cov cav nyob rau hauv ib qho chaw. Tom qab ntawd nws yuav ua tau maj mam teeb tsa elasticsearch, greylog, lossis lwm yam. Tab sis koj yuav twb muaj tag nrho cov cav, thiab, ntxiv rau, koj tuaj yeem khaws lawv, kom deb li deb muaj cov disk arrays txaus.
Thaum lub sijhawm kuv tshaj tawm, cov tswv yim pib zoo li no. Peb xyaum tsis sau ntawv mus rau cov ntaub ntawv. Tam sim no, feem ntau yuav, peb yuav tua cov seem. Ntawm cov tshuab hauv zos uas khiav API, peb yuav tsum tsis txhob sau rau cov ntaub ntawv. Ua ntej, muaj cov ntaub ntawv khaws cia, uas ua haujlwm zoo heev. Qhov thib ob, cov tshuab no tas li khiav tawm ntawm qhov chaw, koj yuav tsum tau saib xyuas nws tas li.
Qhov no nrog Logstash thiab Greylog, nws yeej soars. Yog li ntawd, koj yuav tsum tau tshem ntawm nws. Koj yuav tsum xaiv ib qho.
Peb txiav txim siab tso Logstash thiab Kibana. Vim peb muaj lub chaw ruaj ntseg. Kev sib txuas yog dab tsi? Kev sib txuas yog tias Kibana tsis muaj X-Pack thiab tsis muaj Shield tsis tso cai rau koj los sib txawv txoj cai nkag mus rau cov cav. Yog li ntawd, lawv coj Graylog. Nws muaj tag nrho. Kuv tsis nyiam nws, tab sis nws ua haujlwm. Peb yuav cov khoom siv tshiab, teeb tsa Greylog tshiab nyob rau ntawd, thiab txav tag nrho cov cav nrog cov qauv nruj rau ib qho Greylog cais. Peb tau daws qhov teeb meem nrog ntau hom kev sib txawv ntawm tib lub koom haum.
Dab tsi yog suav nrog hauv Graylog tshiab. Peb tsuas yog sau txhua yam hauv docker. Peb coj ib pawg ntawm cov servers, dov tawm peb lub sijhawm Kafka, 7 Graylog servers version 2.3 (vim kuv xav tau Elasticsearch version 5). Tag nrho cov no tau raug tsa los ntawm kev tawm tsam los ntawm HDD. Peb pom qhov ntsuas qhov ntsuas txog li 100 txhiab lus ib ob. Peb pom daim duab uas 140 terabytes ntawm cov ntaub ntawv ib lub lim tiam.
Thiab dua ib tug rake! Peb muaj ob qhov kev muag khoom tuaj. Peb tau tsiv dhau 6 lab tus ncej. Peb Graylog tsis muaj sijhawm los zom. Cas koj yuav tsum ciaj sia dua.
Qhov no yog li cas peb ciaj sia. Ntxiv ob peb ntxiv servers thiab SSDs. Lub caij peb nyob li no. Tam sim no peb twb zom 160k lus ib ob. Peb tseem tsis tau tsoo qhov txwv, yog li nws tsis paub meej npaum li cas peb tuaj yeem tawm ntawm nws.
Cov no yog peb cov kev npaj rau yav tom ntej. Ntawm cov no, tiag tiag, qhov tseem ceeb tshaj plaws yog tej zaum muaj siab. Peb tseem tsis tau muaj. Ob peb lub tsheb tau teeb tsa tib txoj kev, tab sis kom deb li deb txhua yam mus dhau ib lub tsheb. Nws yog ib qho tsim nyog yuav tsum siv sij hawm los teeb tsa qhov kev poob qis ntawm lawv.
Sau cov metrics los ntawm Graylog.
Ua tus nqi txwv kom peb muaj ib qho API vwm uas tsis tua peb bandwidth thiab txhua yam.
Thiab thaum kawg, kos npe rau qee yam SLA nrog cov neeg tsim khoom kom peb tuaj yeem ua haujlwm ntau. Yog koj sau ntxiv thov txim.
Thiab sau cov ntaub ntawv.
Luv luv, cov txiaj ntsig ntawm txhua yam peb tau ntsib. Ua ntej, cov qauv. Qhov thib ob, syslog yog ncuav mog qab zib. Thib peb, rsyslog ua haujlwm raws nraim li nws tau sau rau ntawm tus swb. Thiab cia peb mus rau cov lus nug.
Lus nug.
Koj lo lus nug: Vim li cas lawv thiaj txiav txim siab tsis coj ... (filebeat?)
Teb: Yuav tsum tau sau rau ib daim ntawv. Kuv yeej tsis xav tau. Thaum koj API sau ntau txhiab cov lus hauv ib ob, txawm tias koj tig ib zaug ib teev, qhov no tseem tsis yog qhov kev xaiv. Koj tuaj yeem sau rau cov yeeb nkab. Rau qhov uas cov neeg tsim khoom nug kuv: "Yuav ua li cas yog tias cov txheej txheem uas peb sau poob"? Kuv nyuam qhuav nrhiav tsis tau dab tsi los teb lawv, thiab hais tias: "Zoo, ok, cia peb tsis ua li ntawd."
Koj lo lus nug: Vim li cas koj tsis cia li sau cov cav rau HDFS?
TebA: Nov yog kauj ruam tom ntej. Peb xav txog nws thaum pib, tab sis txij li tsis muaj peev txheej los daws nws tam sim no, nws tau dai rau hauv peb txoj kev daws teeb meem ntev.
Koj lo lus nug: Ib hom kab ke yuav tsim nyog dua.
Teb: Kuv nkag siab. Peb yog "rau" nrog ob txhais tes.
Koj lo lus nug: Koj sau rau rsyslog. Ob leeg TCP thiab UDP muaj nyob rau ntawd. Tab sis yog UDP, ces koj yuav lav li cas xa khoom?
TebA: Muaj ob lub ntsiab lus. Ua ntej, kuv qhia txhua tus neeg tam sim ntawd tias peb tsis lav qhov kev xa khoom ntawm cov cav. Vim tias thaum cov neeg tsim khoom tuaj thiab hais tias: "Cia peb pib sau cov ntaub ntawv nyiaj txiag rau ntawd, thiab koj yuav muab tso rau qhov chaw rau peb thaum muaj qee yam tshwm sim," peb teb rau lawv, "Zoo heev! Cia peb pib thaiv kev sau ntawv rau lub qhov (socket), thiab ua nws hauv kev lag luam, kom koj tau lees tias muab tso rau hauv lub qhov (socket) rau peb thiab xyuas kom meej tias peb tau txais los ntawm lwm sab. Thiab tam sim no, txhua tus neeg tam sim ntawd ua tsis tsim nyog. Thiab yog tias tsis yog, peb muaj lus nug dab tsi? Yog tias koj tsis xav kom lav ib daim ntawv sau rau lub qhov (socket), ces vim li cas peb thiaj li lav qhov khoom xa tuaj? Peb tabtom ua qhov zoo tshaj plaws. Peb yeej sim xa ntau npaum li sai tau thiab zoo tshaj plaws, tab sis peb tsis muab 100% lav. Yog li ntawd, koj tsis tas yuav sau cov ntaub ntawv nyiaj txiag muaj. Muaj kev hloov pauv databases rau qhov no.
Koj lo lus nug: Thaum API tsim qee cov lus rau lub cav thiab hloov chaw tswj mus rau microservices, koj puas tau ntsib cov teeb meem uas cov lus los ntawm cov microservices sib txawv tuaj txog qhov tsis ncaj ncees lawm? Vim li no, tsis meej pem tshwm sim.
TebA: Nws yog ib txwm uas lawv tuaj nyob rau hauv ib qho kev txiav txim sib txawv. Koj yuav tsum tau npaj rau qhov no. Vim tias ib qho kev xa khoom hauv network tsis lav kev txiav txim rau koj, lossis koj yuav tsum siv cov peev txheej tshwj xeeb ntawm qhov no. Yog tias peb muab cov ntaub ntawv khaws cia, ces txhua API khaws cov cav rau nws tus kheej cov ntaub ntawv. Es tsis txhob, rsyslog decomposes lawv mus rau hauv cov npe muaj. Txhua API muaj nws tus kheej cov cav nyob rau ntawd, qhov twg koj tuaj yeem mus thiab saib, thiab tom qab ntawd koj tuaj yeem sib piv lawv siv lub sijhawm nyob hauv lub cav no. Yog tias lawv mus saib hauv Graylog, ces lawv yuav raug txheeb xyuas raws sijhawm. Txhua yam yuav zoo nyob ntawd.
Koj lo lus nug: Timestamp yuav txawv ntawm milliseconds.
Teb: Lub timestamp yog tsim los ntawm API nws tus kheej. Qhov no, qhov tseeb, yog tag nrho cov ntsiab lus. Peb muaj NTP. API tsim cov ntawv teev sijhawm nyob rau hauv cov lus nws tus kheej. Nws tsis yog ntxiv los ntawm rsyslog.
Koj lo lus nug: Kev sib cuam tshuam ntawm cov chaw khaws ntaub ntawv tsis meej heev. Nyob rau hauv lub moj khaum ntawm cov ntaub ntawv chaw, nws yog tseeb hais tias cov cav tau sau thiab ua tiav li cas. Kev sib cuam tshuam ntawm cov chaw zov me nyuam li cas? Los yog txhua qhov chaw khaws ntaub ntawv ua nws lub neej?
Teb: Yuav luag. Peb muaj txhua lub teb chaws nyob hauv ib lub chaw ntaub ntawv. Tam sim no peb tsis muaj kev sib kis, yog li ntawd ib lub tebchaws tau muab tso rau hauv cov chaw sib txawv. Yog li ntawd, tsis tas yuav tsum ua ke nrog lawv. Hauv txhua qhov chaw muaj lub Log Relay. Qhov no yog Rsyslog server. Qhov tseeb, ob lub tshuab tswj. Lawv raug teeb tsa tib yam nkaus. Tab sis tam sim no, kev khiav tsheb tsuas yog hla ib qho ntawm lawv. Nws sau txhua yam sib sau ua ke. Nws muaj ib tug disk queue xwb nyob rau hauv rooj plaub. Nws nias lub cav thiab xa lawv mus rau qhov chaw nruab nrab cov ntaub ntawv (Singapore), qhov twg ntxiv lawv twb raug tshuaj lom hauv Graylog. Thiab txhua qhov chaw khaws ntaub ntawv muaj nws cov ntaub ntawv khaws cia. Yog hais tias peb poob kev twb kev txuas, peb muaj tag nrho cov cav nyob rau ntawd. Lawv yuav nyob ntawd. Lawv yuav muab khaws cia rau ntawd.
Koj lo lus nug: Koj puas tau txais cov cav los ntawm qhov ntawd thaum muaj xwm txheej txawv txav?
Teb: Koj tuaj yeem mus rau qhov ntawd (rau cov ntaub ntawv khaws cia) thiab saib.
Koj lo lus nug: Yuav ua li cas koj saib xyuas koj tsis poob cav?
Teb: Peb yeej poob lawv, thiab peb saib xyuas nws. Kev tshuaj xyuas pib ib hlis dhau los. Lub tsev qiv ntawv uas Go APIs siv muaj cov ntsuas. Nws tuaj yeem suav tau pes tsawg zaus nws tsis sau ntawv rau lub qhov (socket). Nyob rau tam sim no muaj ib lo lus nug heuristic. Muaj ib tug tsis nyob ntawd. Nws sim sau cov lus los ntawm nws mus rau lub qhov (socket). Yog hais tias lub buffer overflows, nws pib poob lawv. Thiab nws suav pes tsawg tus uas nws tso lawv tseg. Yog tias cov txee pib dhau mus rau qhov ntawd, peb yuav paub txog nws. Tam sim no lawv tseem tuaj rau prometheus, thiab koj tuaj yeem pom cov duab hauv Grafana. Koj tuaj yeem teeb tsa kev ceeb toom. Tab sis tseem tsis tau paub meej tias yuav xa lawv mus rau leej twg.
Koj lo lus nug: Hauv elasticsearch, koj khaws cov cav nrog redundancy. Koj muaj pes tsawg replicas?
Teb: Ib replica.
Koj lo lus nug: Puas yog ib kab?
Teb: Qhov no yog tus tswv thiab replica. Cov ntaub ntawv yog muab cia rau hauv duplicate.
Koj lo lus nug: Koj puas tau tweak qhov loj ntawm rsyslog tsis ua li cas?
Teb: Peb sau datagrams rau ib qho kev cai unix socket. Qhov no tam sim ntawd txwv qhov kev txwv ntawm 128 kilobytes rau peb. Peb tsis tuaj yeem sau ntxiv rau hauv nws. Peb tau sau qhov no rau hauv tus qauv. Leej twg xav tau mus rau hauv cia, lawv sau 128 kilobytes. Cov tsev qiv ntawv, ntxiv rau, txiav tawm, thiab muab tus chij uas cov lus raug txiav tawm. Peb muaj ib qho tshwj xeeb teb nyob rau hauv tus qauv ntawm cov lus nws tus kheej, uas qhia tau hais tias nws raug txiav tawm thaum lub sij hawm kaw los yog tsis. Yog li peb muaj sijhawm los taug qab lub sijhawm no.
Lus nug: Koj puas sau JSON tawg?
Teb: tawg JSON yuav raug muab pov tseg thaum lub sij hawm relay vim lub pob ntawv loj dhau. Los yog Greylog yuav poob, vim nws yuav tsis tuaj yeem txheeb xyuas JSON. Tab sis muaj cov nuances ntawm no uas yuav tsum tau kho, thiab lawv feem ntau khi rau rsyslog. Kuv twb tau sau ob peb qhov teeb meem muaj, uas tseem yuav tsum tau ua haujlwm.
Lus nug: Vim li cas Kafka? Koj puas tau sim RabbitMQ? Greylog tsis ntxiv nyob rau hauv xws li loads?
Teb: Nws tsis ua haujlwm nrog Graylog. Thiab Graylog tab tom ua haujlwm zoo. Nws yog ib qho teeb meem tiag tiag rau nws. Nws yog ib yam dab tsi. Thiab, qhov tseeb, nws tsis xav tau. Kuv xav sau los ntawm rsyslog ncaj qha mus rau elasticsearch thiab tom qab ntawd saib Kibana. Tab sis peb yuav tsum tau daws qhov teeb meem nrog cov neeg saib xyuas kev ruaj ntseg. Qhov no yog qhov sib txawv ntawm peb txoj kev loj hlob thaum peb pov tawm Graylog thiab siv Kibana. Logstash yuav tsis muaj txiaj ntsig. Vim kuv tuaj yeem ua ib yam nrog rsyslog. Thiab nws muaj ib qho module sau rau elasticsearch. Nrog Graylog peb tab tom sim ua neej nyob qee yam. Peb txawm tweaked nws me ntsis. Tab sis tseem muaj chaw rau kev txhim kho.
Hais txog Kafka. Qhov ntawd yog li cas nws tshwm sim keeb kwm. Thaum kuv tuaj txog, nws twb muaj lawm, thiab cov ntawv teev lus twb tau sau rau nws. Peb tsuas yog tsa peb pawg thiab txav cov cav rau hauv nws. Peb tswj hwm nws, peb paub tias nws xav li cas. Raws li RabbitMQ ... peb muaj teeb meem nrog RabbitMQ. Thiab RabbitMQ tab tom txhim kho rau peb. Peb muaj nws nyob rau hauv ntau lawm, thiab muaj teeb meem nrog nws. Tam sim no, ua ntej kev muag khoom, nws yuav shamanized, thiab nws yuav pib ua haujlwm ib txwm. Tab sis ua ntej ntawd, kuv tsis tau npaj tso tawm rau hauv kev tsim khoom. Muaj ib qho ntxiv. Graylog tuaj yeem nyeem AMQP 0.9 version thiab rsyslog tuaj yeem sau AMQP 1.0 version. Thiab tsis muaj ib qho kev daws teeb meem uas tuaj yeem ua ob qho tib si hauv nruab nrab. Muaj ib tug los yog lwm yam. Yog li tam sim no tsuas yog Kafka. Tab sis kuj muaj nuances. Vim hais tias omkafka ntawm lub version ntawm rsyslog uas peb siv tau poob tag nrho cov lus tsis hais tias nws scooped los ntawm rsyslog. Tsuav yog peb muab tso tseg.
Lus nug: Koj puas siv Kafka vim koj muaj nws? Tsis siv rau lwm lub hom phiaj?
Teb: Kafka, uas tau siv los ntawm pab pawg Data Science. Qhov no yog ib qhov project cais kiag li, txog qhov uas kuv, hmoov tsis, tsis tuaj yeem hais dab tsi. kuv tsis paub. Nws tau khiav los ntawm pab pawg Data Science. Thaum lub cav pib, lawv txiav txim siab siv nws, thiaj li tsis tso lawv tus kheej. Tam sim no peb tau hloov kho Graylog, thiab peb tau poob kev sib raug zoo, vim tias muaj qhov qub version ntawm Kafka. Peb yuav tsum ua peb tus kheej. Tib lub sijhawm, peb tau tshem ntawm plaub lub ntsiab lus no rau txhua API. Peb ua ib qho dav saum toj kawg nkaus rau txhua tus nyob, ib lub dav dav saum toj kawg nkaus rau txhua qhov kev ua yeeb yam thiab peb tsuas yog tua txhua yam nyob ntawd. Greylog rakes tag nrho cov no tawm nyob rau hauv parallel.
Lus nug: Vim li cas peb thiaj xav tau qhov shamanism no nrog lub qhov (socket)? Koj puas tau sim siv syslog log tsav rau ntim.
Teb: Thaum lub sijhawm peb nug cov lus nug no, peb muaj kev sib raug zoo nrog tus docker. Nws yog docker 1.0 lossis 0.9. Docker nws tus kheej yog qhov txawv. Qhov thib ob, yog tias koj tseem shove cav rau hauv nws ... Kuv muaj ib qho kev xav tsis thoob uas nws hla tag nrho cov cav los ntawm nws tus kheej, los ntawm docker daemon. Yog tias peb muaj ib qho API mus vwm, ces qhov seem ntawm APIs yuav khiav mus rau qhov tseeb tias lawv tsis tuaj yeem xa stdout thiab stderr. Kuv tsis paub qhov no yuav coj mus rau qhov twg. Kuv muaj qhov tsis txaus ntseeg ntawm qib kev xav tias tsis tas yuav siv tus tsav tsheb docker syslog hauv qhov chaw no. Peb lub chaw kuaj xyuas ua haujlwm muaj nws tus kheej Graylog pawg nrog cov cav. Lawv siv docker log tsav tsheb thiab txhua yam zoo li zoo nyob ntawd. Tab sis lawv tam sim ntawd sau GELF rau Graylog. Lub sijhawm thaum peb pib tag nrho cov no, peb xav tau nws tsuas yog ua haujlwm xwb. Tej zaum tom qab ntawd, thaum ib tug neeg tuaj thiab hais tias nws tau ua haujlwm ib txwm tau ib puas xyoo, peb yuav sim.
Lus nug: Koj xa ntawm cov chaw zov me nyuam siv rsyslog. Vim li cas ho tsis nyob rau Kafka?
Teb: Peb ua li no, thiab qhov no yog qhov nws tiag tiag. Rau ob qho laj thawj. Yog hais tias lub channel yog tag nrho tuag, ces tag nrho peb cov cav, txawm nyob rau hauv ib tug compressed daim ntawv, yuav tsis nce los ntawm nws. Thiab kafka tso cai rau lawv yooj yim poob hauv cov txheej txheem. Nyob rau hauv no txoj kev, peb tshem tawm cov sticking ntawm no cav. Peb tsuas yog siv Kafka hauv qhov no ncaj qha. Yog hais tias peb muaj ib tug zoo channel thiab xav pub dawb, ces peb siv lawv rsyslog. Tab sis qhov tseeb, koj tuaj yeem teeb tsa nws kom nws poob qhov nws tsis tau dhau los. Tam sim no peb tsuas yog siv rsyslog xa ncaj qha rau qhov chaw, qhov chaw Kafka.
Tau qhov twg los: www.hab.com