Nws yog 2019, thiab peb tseem tsis muaj cov qauv kev daws teeb meem rau kev sib sau ua ke hauv Kubernetes. Hauv tsab xov xwm no, peb xav, siv cov piv txwv los ntawm kev xyaum tiag tiag, qhia peb cov kev tshawb nrhiav, teeb meem ntsib thiab lawv cov kev daws teeb meem.
Txawm li cas los xij, ua ntej, kuv yuav ua qhov tshwj xeeb uas cov neeg siv khoom sib txawv nkag siab ntau yam sib txawv los ntawm kev sau cov cav:
- ib tug neeg xav pom kev ruaj ntseg thiab tshawb xyuas cov cav;
- ib tug neeg - centralized txiav ntawm tag nrho cov infrastructure;
- thiab rau qee qhov, nws yog qhov txaus los sau tsuas yog daim ntawv thov cav, tsis suav nrog, piv txwv li, balancers.
Hauv qab no yog qhov txiav hauv qab no hais txog yuav ua li cas peb tau siv ntau yam "cov npe xav tau" thiab cov teeb meem uas peb ntsib.
Kev xav: hais txog cov cuab yeej txiav
Keeb kwm ntawm lub Cheebtsam ntawm ib tug loging system
Kev txiav tawm tau los ntev, vim tias cov txheej txheem rau kev sau thiab txheeb xyuas cov cav tau tsim, uas yog qhov peb siv niaj hnub no. Rov qab rau xyoo 1950, Fortran tau qhia txog qhov piv txwv ntawm cov qauv nkag / tso tawm cov kwj deg, uas pab tus programmer debug nws qhov program. Cov no yog thawj lub computer cav uas ua rau lub neej yooj yim dua rau programmers ntawm lub sijhawm ntawd. Niaj hnub no peb pom nyob rau hauv lawv thawj feem ntawm kev txiav txim siab - qhov chaw lossis "tus tsim khoom" ntawm cov cav.
Kev tshawb fawb hauv computer tsis tau sawv ntsug: lub computer network tau tshwm sim, thawj pawg ... Cov txheej txheem nyuaj uas muaj ntau lub khoos phis tawj pib ua haujlwm. Tam sim no cov thawj tswj hwm raug yuam kom sau cov cav los ntawm ntau lub tshuab, thiab tshwj xeeb lawv tuaj yeem ntxiv OS ntsiav cov lus nyob rau hauv rooj plaub uas lawv xav tau los tshawb xyuas qhov tsis ua haujlwm. Txhawm rau piav qhia txog kev sau cov cav hauv nruab nrab, thaum ntxov 2000s nws tau luam tawm , uas standardized remote_syslog. Nov yog li cas lwm qhov tseem ceeb tshwm sim: log sau thiab lawv cia.
Nrog rau qhov nce ntawm qhov ntim ntawm cov cav thiab cov kev qhia dav dav ntawm lub vev xaib thev naus laus zis, cov lus nug tshwm sim ntawm cov cav yuav tsum tau yooj yim qhia rau cov neeg siv. Cov cuab yeej console yooj yim (awk / sed / grep) tau hloov los ntawm cov khoom siab dua log saib - peb feem.
Vim tias qhov nce hauv ntim ntawm cov cav, ib yam dab tsi tau pom meej: cov cav yuav tsum tau, tab sis tsis yog txhua tus ntawm lawv. Thiab cov cav sib txawv yuav tsum muaj ntau theem ntawm kev khaws cia: qee qhov tuaj yeem ploj hauv ib hnub, thaum lwm tus yuav tsum tau khaws cia rau 5 xyoos. Yog li, ib qho kev tivthaiv rau kev lim dej thiab xa cov ntaub ntawv ntws tau ntxiv rau lub kaw lus kaw lus - cia peb hu nws lub lim.
Kev khaws cia kuj tau ua kom muaj kev vam meej: los ntawm cov ntaub ntawv tsis tu ncua mus rau cov ntaub ntawv sib txheeb, thiab tom qab ntawd mus rau cov ntaub ntawv khaws cia (piv txwv li, Elasticsearch). Yog li ntawd, qhov chaw khaws cia tau muab cais tawm ntawm lub collector.
Thaum kawg, lub tswv yim ntawm lub cav tau nthuav dav mus rau ib hom kev paub daws teeb meem ntawm cov xwm txheej uas peb xav khaws cia rau keeb kwm. Los yog theej, yog tias koj yuav tsum tau ua qhov kev tshawb nrhiav lossis kos ib daim ntawv qhia kev tshuaj ntsuam ...
Raws li qhov tshwm sim, nyob rau lub sijhawm luv luv, kev sau cov cav tau tsim rau hauv ib qho tseem ceeb subsystem, uas tuaj yeem raug hu ua ib qho ntawm cov ntu hauv Cov Ntaub Ntawv Loj.
Yog tias ib zaug ib zaug cov ntawv luam tawm tuaj yeem txaus rau "kev kaw lus", tam sim no qhov xwm txheej tau hloov pauv ntau.
Kubernetes thiab cav
Thaum Kubernetes tuaj rau hauv kev tsim kho, qhov teeb meem uas twb muaj lawm ntawm kev sau cov cav tsis hla nws ib yam. Hauv qee txoj hauv kev, nws tau dhau los ua qhov mob hnyav dua: kev tswj hwm cov txheej txheem kev tsim kho tsis yog tsuas yog yooj yim, tab sis kuj nyuaj rau tib lub sijhawm. Ntau cov kev pabcuam qub tau pib tsiv mus rau microservices. Nyob rau hauv cov ntsiab lus ntawm cov cav, qhov no yog xav txog nyob rau hauv ntau zuj zus ntawm cav qhov chaw, lawv tshwj xeeb lub neej voj voog, thiab yuav tsum tau mus taug qab cov kev sib raug zoo ntawm tag nrho cov system Cheebtsam los ntawm cav ...
Saib ua ntej, kuv tuaj yeem hais tias tam sim no, hmoov tsis, tsis muaj kev xaiv txiav txiav ua qauv rau Kubernetes uas yuav muab piv rau lwm tus. Cov txheej txheem nrov tshaj plaws hauv zej zog yog raws li hauv qab no:
- ib tug unrolls pawg EFK (Elasticsearch, Fluentd, Kibana);
- ib tug neeg sim qhov nyuam qhuav tso tawm los yog siv ;
- peb (thiab tej zaum tsis yog peb xwb?..) Kuv txaus siab rau kuv tus kheej txoj kev loj hlob - ...
Raws li txoj cai, peb siv cov khoom hauv qab no hauv K8s pawg (rau kev daws teeb meem rau tus kheej):
- ;
- .
Txawm li cas los xij, kuv yuav tsis nyob ntawm cov lus qhia rau lawv qhov kev teeb tsa thiab teeb tsa. Hloov chaw, kuv yuav tsom mus rau lawv qhov tsis txaus thiab ntau qhov kev txiav txim siab thoob ntiaj teb txog qhov xwm txheej nrog cov ntaub ntawv dav dav.
Xyaum nrog cov cav hauv K8s
"Txhua hnub log", koj muaj pes tsawg tus?..
Centralized sau cov cav los ntawm ib qho chaw loj loj yuav tsum muaj peev txheej ntau, uas yuav siv rau kev sau, khaws cia thiab ua cov cav. Thaum lub sijhawm ua haujlwm ntawm ntau yam haujlwm, peb tau ntsib ntau yam kev xav tau thiab cov teeb meem kev ua haujlwm tshwm sim los ntawm lawv.
Wb sim ClickHouse
Cia peb saib ntawm lub hauv paus cia ntawm ib qhov project nrog ib daim ntawv thov uas tsim cov cav ntoo heev: ntau tshaj 5000 kab ib ob. Cia peb pib ua haujlwm nrog nws cov cav, ntxiv rau ClickHouse.
Sai li qhov siab tshaj plaws ntawm lub sijhawm yuav tsum tau, 4-core server nrog ClickHouse yuav twb dhau los ntawm lub disk subsystem:
Hom kev thauj khoom no yog vim qhov tseeb tias peb tab tom sim sau hauv ClickHouse kom sai li sai tau. Thiab lub database reacts rau qhov no nrog nce disk load, uas yuav ua rau cov nram qab no yuam kev:
DB::Exception: Too many parts (300). Merges are processing significantly slower than inserts
Point yog tias hauv ClickHouse (lawv muaj cov ntaub ntawv teev tseg) muaj lawv tus kheej nyuaj thaum sau ntawv. Cov ntaub ntawv muab tso rau hauv lawv tsim ib ntus muab faib, uas yog tom qab ntawd merged nrog lub ntsiab lus. Raws li qhov tshwm sim, cov ntaubntawv povthawj hloov mus rau qhov xav tau heev ntawm lub disk, thiab nws tseem yuav raug txwv rau qhov peb tau txais daim ntawv ceeb toom saum toj no: tsis pub ntau tshaj 1 subpartitions tuaj yeem ua ke hauv 300 thib ob (qhov tseeb, qhov no yog 300 qhov ntxig. per second).
Txhawm rau zam qhov kev coj cwj pwm no, hauv cov khoom loj li sai tau thiab tsis pub ntau tshaj 1 zaug txhua 2 vib nas this. Txawm li cas los xij, kev sau ntawv hauv qhov tawg loj qhia tias peb yuav tsum sau tsawg zaus hauv ClickHouse. Qhov no, nyob rau hauv lem, yuav ua rau ib tug tsis overflow thiab poob ntawm cav. Txoj kev daws teeb meem yog txhawm rau nce qhov Fluentd tsis, tab sis tom qab ntawd lub cim xeeb noj kuj yuav nce ntxiv.
ΠΡΠΈΠΌΠ΅ΡΠ°Π½ΠΈΠ΅: Lwm qhov teeb meem ntawm peb cov kev daws teeb meem nrog ClickHouse tau hais txog qhov tseeb tias kev faib tawm hauv peb rooj plaub (loghouse) yog siv los ntawm cov rooj sab nraud txuas nrog. . Qhov no ua rau lub fact tias thaum sampling loj lub sij hawm ncua, ntau RAM yuav tsum tau, txij li thaum lub metatable iterates los ntawm tag nrho cov partitions - txawm cov uas pom tseeb tsis muaj cov ntaub ntawv tsim nyog. Txawm li cas los xij, tam sim no txoj hauv kev no tuaj yeem muaj kev nyab xeeb tshaj tawm tias tsis siv rau tam sim no ntawm ClickHouse (c ).
Raws li qhov tshwm sim, nws tau pom tseeb tias tsis yog txhua qhov project muaj peev txheej txaus los sau cov cav hauv lub sijhawm tiag tiag hauv ClickHouse (ntau qhov tseeb, lawv qhov kev faib tawm yuav tsis tsim nyog). Tsis tas li ntawd, koj yuav tau siv roj teeb, uas peb yuav rov qab los. Cov ntaub ntawv uas tau piav saum toj no yog qhov tseeb. Thiab thaum lub sijhawm ntawd peb tsis tuaj yeem muab cov kev daws teeb meem ruaj khov thiab ruaj khov uas yuav haum rau cov neeg siv khoom thiab tso cai rau peb khaws cov cav nrog tsawg kawg ...
Yuav ua li cas yog Elasticsearch?
Elasticsearch paub tias ua haujlwm hnyav. Cia peb sim nws hauv tib qhov project. Tam sim no lub load zoo li no:
Elasticsearch tuaj yeem zom cov ntaub ntawv ntws, txawm li cas los xij, sau cov ntim rau nws zoo heev siv CPU. Qhov no txiav txim siab los ntawm kev teeb tsa ib pawg. Technically, qhov no tsis yog ib qho teeb meem, tab sis nws hloov tawm hais tias tsuas yog ua haujlwm lub cav sau peb twb siv txog 8 cores thiab muaj ib qho ntxiv loaded tivthaiv nyob rau hauv lub system ...
Cov kab hauv qab: qhov kev xaiv no tuaj yeem tsim nyog, tab sis tsuas yog tias qhov project loj thiab nws cov kev tswj hwm tau npaj los siv cov peev txheej tseem ceeb ntawm lub hauv paus kev txiav txim siab.
Ces ib lo lus nug ntuj tshwm sim:
Dab tsi log yog xav tau tiag tiag?
Cia peb sim hloov txoj hauv kev nws tus kheej: cov cav yuav tsum ib txhij yuav qhia thiab tsis npog txhua kev tshwm sim hauv qhov system.
Cia peb hais tias peb muaj kev vam meej hauv khw online. Cov ntawv teev dab tsi tseem ceeb? Kev sau cov ntaub ntawv ntau li ntau tau, piv txwv li, los ntawm lub qhov rooj them nyiaj, yog ib lub tswv yim zoo. Tab sis tsis yog tag nrho cov cav los ntawm cov duab slicing kev pab cuam nyob rau hauv cov khoom catalog yog ib qho tseem ceeb rau peb: tsuas yog qhov yuam kev thiab kev saib xyuas siab heev yog txaus (piv txwv li, qhov feem pua ββββntawm 500 yuam kev uas cov khoom no generates).
Yog li peb tau los xaus qhov ntawd centralized logging tsis yog ib txwm tsim nyog. Feem ntau tus neeg siv xav sau tag nrho cov cav nyob rau hauv ib qho chaw, txawm hais tias qhov tseeb, los ntawm tag nrho cov cav, tsuas yog muaj 5% ntawm cov lus uas tseem ceeb rau kev lag luam yog yuav tsum tau:
- Qee lub sij hawm nws txaus los teeb tsa, hais, tsuas yog qhov loj ntawm lub thawv ntim thiab cov khoom siv yuam kev (piv txwv li, Sentry).
- Kev ceeb toom yuam kev thiab lub cav loj hauv zos nws tus kheej feem ntau yuav txaus los tshawb xyuas qhov xwm txheej.
- Peb muaj cov haujlwm uas tau ua nrog cov kev xeem ua haujlwm tshwj xeeb thiab cov khoom sau ua yuam kev. Tus tsim tawm tsis xav tau cov cav li ntawd - lawv pom txhua yam los ntawm qhov yuam kev.
Piv txwv ntawm lub neej
Lwm zaj dab neeg tuaj yeem ua piv txwv zoo. Peb tau txais kev thov los ntawm pab pawg kev ruaj ntseg ntawm ib qho ntawm peb cov neeg siv khoom uas twb tau siv cov kev lag luam daws teeb meem uas tau tsim los ntev ua ntej kev taw qhia ntawm Kubernetes.
Nws yog qhov tsim nyog los "ua phooj ywg" ntawm lub hauv paus kev sau cov cav nrog lub tuam txhab teeb meem nrhiav pom - QRadar. Cov kab ke no tuaj yeem tau txais cov ntawv teev npe ntawm syslog raws tu qauv thiab muab lawv los ntawm FTP. Txawm li cas los xij, nws tsis tau tam sim ntawd los ua ke nrog cov remote_syslog plugin rau fluentd (raws li nws muab tawm, ). Teeb meem nrog kev teeb tsa QRadar tau dhau los ua ib sab ntawm pab pawg neeg siv khoom ruaj ntseg.
Raws li qhov tshwm sim, ib feem ntawm kev lag luam-tseem ceeb cav raug xa mus rau FTP QRadar, thiab lwm qhov tau xa rov qab los ntawm cov chaw taws teeb syslog ncaj qha los ntawm cov nodes. Rau qhov no peb txawm sau - tej zaum nws yuav pab tau ib tug neeg daws qhov teeb meem zoo sib xws ... Ua tsaug rau cov tswv yim tsim tawm, tus neeg siv khoom nws tus kheej tau txais thiab txheeb xyuas cov ntaub ntawv tseem ceeb (siv cov cuab yeej uas nws nyiam), thiab peb tuaj yeem txo tus nqi ntawm lub kaw lus, txuag tsuas yog lub lub hli tas los.
Lwm qhov piv txwv yog qhov qhia tau hais tias tsis ua dab tsi. Ib qho ntawm peb cov neeg siv khoom rau kev ua haujlwm ntawm txhua tus txheej xwm los ntawm tus neeg siv, ua multiline unstructured tso zis cov ntaub ntawv hauv lub log. Raws li koj tuaj yeem kwv yees, cov cav no tsis yooj yim heev rau kev nyeem thiab khaws cia.
Cov txheej txheem rau cov log
Xws li cov piv txwv ua rau kev txiav txim siab uas ntxiv rau kev xaiv ib lub cav sau, koj yuav tsum tau kuj tsim cov cav lawv tus kheej! Cov kev cai ntawm no yog dab tsi?
- Cov cav yuav tsum nyob hauv lub tshuab nyeem tau (piv txwv li, JSON).
- Cov cav yuav tsum compact thiab muaj peev xwm los hloov cov qib ntawm kev txiav txim siab kom debug cov teeb meem tshwm sim. Nyob rau tib lub sijhawm, nyob rau hauv ib puag ncig ntau lawm koj yuav tsum khiav cov tshuab nrog rau qib kev txiav txim siab zoo li Ceeb toom los yog kev ua yuam kev.
- Cov cav yuav tsum tau normalized, uas yog, nyob rau hauv ib lub cav kwv, txhua kab yuav tsum muaj tib hom teb.
Cov cav tsis sib xws tuaj yeem ua rau muaj teeb meem nrog kev thauj khoom rau hauv qhov chaw cia thiab nres ua tiav hauv lawv cov txheej txheem. Raws li ib qho piv txwv, ntawm no yog ib qho piv txwv nrog kev ua yuam kev 400, uas muaj ntau yam tau ntsib hauv cov cav uas muaj peev xwm:
2019-10-29 13:10:43 +0000 [warn]: dump an error event: error_class=Fluent::Plugin::ElasticsearchErrorHandler::ElasticsearchError error="400 - Rejected by Elasticsearch"
Qhov yuam kev txhais tau hais tias koj tab tom xa ib daim teb uas nws hom tsis ruaj khov rau qhov ntsuas nrog kev npaj ua tiav. Qhov piv txwv yooj yim tshaj plaws yog daim teb hauv nginx cav nrog qhov sib txawv $upstream_status. Nws tuaj yeem muaj tus lej lossis ib txoj hlua. Piv txwv li:
{ "ip": "1.2.3.4", "http_user": "-", "request_id": "17ee8a579e833b5ab9843a0aca10b941", "time": "29/Oct/2019:16:18:57 +0300", "method": "GET", "uri": "/staffs/265.png", "protocol": "HTTP/1.1", "status": "200", "body_size": "906", "referrer": "https://example.com/staff", "user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.70 Safari/537.36", "request_time": "0.001", "cache_status": "-", "upstream_response_time": "0.001, 0.007", "upstream_addr": "127.0.0.1:9000", "upstream_status": "200", "upstream_response_length": "906", "location": "staff"}
{ "ip": "1.2.3.4", "http_user": "-", "request_id": "47fe42807f2a7d8d5467511d7d553a1b", "time": "29/Oct/2019:16:18:57 +0300", "method": "GET", "uri": "/staff", "protocol": "HTTP/1.1", "status": "200", "body_size": "2984", "referrer": "-", "user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.70 Safari/537.36", "request_time": "0.010", "cache_status": "-", "upstream_response_time": "0.001, 0.007", "upstream_addr": "10.100.0.10:9000, 10.100.0.11:9000", "upstream_status": "404, 200", "upstream_response_length": "0, 2984", "location": "staff"}
Cov ntawv teev cia qhia tias server 10.100.0.10 teb nrog 404 yuam kev thiab qhov kev thov raug xa mus rau lwm cov ntsiab lus khaws cia. Raws li qhov tshwm sim, tus nqi hauv cov cav tau zoo li no:
"upstream_response_time": "0.001, 0.007"
Qhov xwm txheej no tshwm sim heev uas nws tseem tsim nyog tau txais kev sib cais .
Yuav ua li cas txog kev ntseeg tau?
Muaj qee lub sij hawm thaum tag nrho cov cav tsis muaj qhov tshwj xeeb yog qhov tseem ceeb. Thiab nrog rau qhov no, cov txheej txheem sau cov txheej txheem rau K8s tau thov / sib tham saum toj no muaj teeb meem.
Piv txwv li, fluentd tsis tuaj yeem sau cov cav los ntawm cov thawv ntim khoom luv luv. Nyob rau hauv ib qho ntawm peb cov haujlwm, cov ntaub ntawv nkag mus hauv lub thawv nyob rau tsawg dua 4 vib nas this thiab tom qab ntawd raug tshem tawm - raws li cov lus piav qhia sib xws:
"helm.sh/hook-delete-policy": hook-succeeded
Vim li no, lub log ua haujlwm tsiv teb tsaws tsis suav nrog hauv kev cia. Kev nom kev tswv tuaj yeem pab tau qhov no. before-hook-creation.
Lwm qhov piv txwv yog Docker log tig. Cia peb hais tias muaj ib daim ntawv thov uas nquag sau rau cov cav. Raws li ib txwm muaj, peb tswj kom ua tiav tag nrho cov cav, tab sis sai li sai tau thaum muaj teeb meem tshwm sim - piv txwv li, raws li tau piav qhia saum toj no nrog rau hom ntawv tsis raug - ua tiav, thiab Docker tig cov ntaub ntawv. Qhov tshwm sim yog tias kev lag luam-tseem ceeb cav yuav poob.
Yog vim li cas Nws yog ib qho tseem ceeb kom cais cov kwj deg, embedding xa cov tseem ceeb tshaj plaws ncaj qha rau hauv daim ntawv thov kom ntseeg tau tias lawv muaj kev nyab xeeb. Tsis tas li ntawd, nws yuav tsis yog superfluous los tsim ib co "accumulator" ntawm cov cav, uas tuaj yeem muaj sia nyob luv luv tsis muaj nyob rau thaum txuag cov lus tseem ceeb.
Thaum kawg, peb yuav tsum tsis txhob hnov ββqab qhov ntawd Nws yog ib qho tseem ceeb uas yuav tsum tau saib xyuas txhua qhov subsystem kom raug. Txwv tsis pub, nws yog ib qho yooj yim khiav mus rau hauv ib qho teeb meem uas fluentd nyob rau hauv ib lub xeev CrashLoopBackOff thiab tsis xa dab tsi, thiab qhov no cog lus tias yuav poob cov ntaub ntawv tseem ceeb.
tshawb pom
Hauv kab lus no, peb tsis saib SaaS cov kev daws teeb meem zoo li Datadog. Ntau yam teeb meem uas tau piav qhia ntawm no tau raug daws los ntawm ib txoj hauv kev los yog lwm qhov los ntawm cov tuam txhab lag luam tshwj xeeb hauv kev sau cov cav, tab sis tsis yog txhua tus tuaj yeem siv SaaS rau ntau yam laj thawj. (cov tseem ceeb yog tus nqi thiab ua raws li 152-FZ).
Centralized log sau thaum xub thawj zoo li ib txoj haujlwm yooj yim, tab sis nws tsis yog tag nrho. Nws yog ib qho tseem ceeb kom nco ntsoov tias:
- Tsuas yog cov khoom tseem ceeb yuav tsum tau nkag mus rau hauv kev nthuav dav, thaum saib xyuas thiab kev sau ua yuam kev tuaj yeem raug teeb tsa rau lwm lub tshuab.
- Cov cav hauv kev tsim khoom yuav tsum tau khaws cia tsawg kawg yog li tsis txhob ntxiv cov khoom tsis tsim nyog.
- Cov cav yuav tsum yog tshuab nyeem tau, normalized, thiab muaj hom ntawv nruj.
- Cov cav tseem ceeb tiag tiag yuav tsum raug xa mus rau hauv ib qho kev sib cais, uas yuav tsum tau muab cais tawm ntawm cov ntsiab lus tseem ceeb.
- Nws yog ib qho tsim nyog xav txog lub cav accumulator, uas tuaj yeem txuag koj los ntawm kev tawg ntawm cov khoom hnyav thiab ua rau cov khoom ntim ntawm cov khoom sib xws.
Cov kev cai yooj yim no, yog siv rau txhua qhov chaw, yuav tso cai rau cov circuits uas tau piav saum toj no ua haujlwm - txawm tias lawv tsis muaj cov khoom tseem ceeb (lub roj teeb). Yog tias koj tsis ua raws li cov hauv paus ntsiab lus no, txoj haujlwm yuav yooj yim coj koj thiab cov cuab yeej cuab tam mus rau lwm qhov kev thauj khoom siab heev (thiab tib lub sijhawm tsis muaj txiaj ntsig) feem ntawm lub system.
PS
Nyeem kuj ntawm peb blog:
- «»;
- «»;
- Β«".
Tau qhov twg los: www.hab.com