Niaj hnub no, ntxiv rau txoj cai monolithic, peb qhov project suav nrog ntau lub microservices. Txhua tus ntawm lawv yuav tsum tau saib xyuas. Ua qhov no ntawm qhov ntsuas siv DevOps engineers yog teeb meem. Peb tau tsim ib qho kev saib xyuas uas ua haujlwm raws li kev pabcuam rau cov neeg tsim khoom. Lawv tuaj yeem sau cov metrics ntawm nws tus kheej rau hauv kev saib xyuas, siv lawv, tsim cov dashboards raws li lawv, thiab xa cov lus ceeb toom rau lawv uas yuav tshwm sim thaum cov nqi pib mus txog. Rau DevOps engineers, tsuas yog infrastructure thiab cov ntaub ntawv.
Cov ncej no yog ib daim ntawv teev lus ntawm kuv cov lus nrog peb ntawm RIT++. Ntau tus neeg tau thov kom peb ua cov ntawv nyeem ntawm cov lus ceeb toom los ntawm qhov ntawd. Yog tias koj nyob hauv lub rooj sab laj lossis saib cov yeeb yaj kiab, koj yuav tsis pom dab tsi tshiab. Thiab txhua leej txhua tus - txais tos rau miv. Kuv yuav qhia koj seb peb tuaj rau qhov system li cas, nws ua haujlwm li cas thiab peb npaj yuav hloov kho nws li cas.
Yav dhau los: schemes thiab kev npaj
Peb tuaj txog ntawm qhov kev saib xyuas tam sim no li cas? Txhawm rau teb cov lus nug no, koj yuav tsum mus rau xyoo 2015. Nov yog qhov nws zoo li ntawd:
Peb muaj kwv yees li 24 qhov ntawm lub luag haujlwm saib xyuas. Muaj ib pob tag nrho ntawm cov yas sib txawv, cov ntawv sau, daemons uas saib xyuas qee yam, xa lus, thiab ua haujlwm. Peb xav tias qhov ntxiv peb mus, qhov tsis muaj peev xwm zoo li no yuav yog. Tsis muaj teeb meem hauv kev tsim nws: nws yog cumbersome dhau lawm.
Peb txiav txim siab xaiv cov kev saib xyuas uas peb yuav khaws thiab txhim kho, thiab cov uas peb yuav tso tseg. Muaj 19 tus ntawm lawv, tsuas yog graphites, aggregators thiab Grafana ua lub dashboard tseem nyob. Tab sis qhov system tshiab yuav zoo li cas? Zoo li no:
Peb muaj kev ntsuas kev ntsuas: cov no yog cov graphites, uas yuav ua raws li SSD tsav ceev, cov no yog qee qhov sib sau ua ke rau kev ntsuas. Tom ntej no - Grafana rau displaying dashboards thiab Moira rau ceeb toom. Peb kuj xav tsim ib qho system rau kev tshawb nrhiav qhov tsis txaus ntseeg.
Standard: Saib xyuas 2.0
Qhov no yog qhov kev npaj zoo li xyoo 2015. Tab sis peb yuav tsum tau npaj tsis yog tsuas yog kev tsim kho vaj tse thiab kev pabcuam nws tus kheej xwb, tab sis kuj yog cov ntaub ntawv rau nws. Peb tau tsim cov qauv kev lag luam rau peb tus kheej, uas peb hu ua kev soj ntsuam 2.0. Dab tsi yog qhov yuav tsum tau ua rau qhov system?
- muaj nyob tas li;
- metrics cia lub sijhawm = 10 vib nas this;
- txhim khu kev qha cia ntawm metrics thiab dashboards;
- SLA > 99,99%
- sau cov xwm txheej ntsuas ntawm UDP (!).
Peb xav tau UDP vim tias peb muaj kev khiav tsheb loj thiab cov xwm txheej uas tsim kev ntsuas. Yog tias koj sau tag nrho rau hauv graphite ib zaug, qhov chaw cia yuav vau. Peb kuj tau xaiv thawj theem ua ntej rau txhua qhov ntsuas.
Txhua tus prefixes muaj qee yam khoom. Muaj cov kev ntsuas rau cov servers, networks, ntim, cov peev txheej, kev siv, thiab lwm yam. Clear, nruj, typed filtering tau raug muab coj los siv, qhov twg peb txais thawj-theem metrics thiab cias tso tseg. Qhov no yog li cas peb npaj qhov system no hauv 2015. Tam sim no yog dab tsi?
Tam sim no: daim duab ntawm kev sib cuam tshuam ntawm kev saib xyuas cov khoom
Ua ntej tshaj plaws, peb saib xyuas cov ntawv thov: peb cov lej PHP, cov ntawv thov thiab microservices - luv luv, txhua yam uas peb cov neeg tsim khoom sau. Txhua daim ntawv thov xa cov ntsuas ntawm UDP mus rau Brubeck aggregator (statsd, rov sau dua hauv C). Nws tau dhau los ua qhov ceev tshaj plaws hauv kev sim hluavtaws. Thiab nws xa cov metrics sib sau ua ke rau Graphite ntawm TCP.
Nws muaj ib hom kev ntsuas hu ua timers. Qhov no yog ib qho yooj yim heev. Piv txwv li, rau txhua tus neeg siv kev sib txuas rau qhov kev pabcuam, koj xa ib qho kev ntsuas nrog lub sijhawm teb rau Brubeck. Ib lab cov lus teb tuaj, tab sis tus sau rov qab tsuas yog 10 qhov ntsuas. Koj muaj pes tsawg tus neeg tuaj, qhov siab tshaj plaws, yam tsawg kawg nkaus thiab nruab nrab lub sij hawm teb, qhov nruab nrab thiab 4 feem pua. Tom qab ntawd cov ntaub ntawv raug xa mus rau Graphite thiab peb pom nws tag nrho nyob.
Peb kuj muaj kev sib sau ua ke rau kev ntsuas ntawm cov khoom siv kho vajtse, software, kev ntsuas qhov system thiab peb qhov qub Munin xyuas qhov system (nws ua haujlwm rau peb txog xyoo 2015). Peb sau tag nrho cov no los ntawm C daemon CollectD (nws muaj tag nrho pawg ntawm cov plugins sib txawv tsim rau hauv nws, nws tuaj yeem tshawb xyuas tag nrho cov peev txheej ntawm tus tswv tsev uas nws tau teeb tsa, tsuas yog qhia hauv kev teeb tsa qhov twg los sau cov ntaub ntawv) thiab sau cov ntaub ntawv rau Graphite los ntawm nws. Nws kuj tseem txhawb nqa python plugins thiab cov ntawv plhaub, yog li koj tuaj yeem sau koj tus kheej cov kev cai daws teeb meem: CollectD yuav sau cov ntaub ntawv no los ntawm tus tswv tsev lossis chaw taws teeb (xws li Curl) thiab xa mus rau Graphite.
Tom qab ntawd peb xa tag nrho cov kev ntsuas uas peb tau sau rau Carbon-c-relay. Qhov no yog cov tshuaj Carbon Relay los ntawm Graphite, hloov kho hauv C. Qhov no yog lub router uas sau tag nrho cov metrics uas peb xa los ntawm peb cov aggregators thiab xa lawv mus rau cov nodes. Tsis tas li nyob rau theem routing, nws tshawb xyuas qhov siv tau ntawm cov ntsuas. Ua ntej, lawv yuav tsum tau ua raws li cov txheej txheem ua ntej uas kuv tau pom ua ntej thiab, qhov thib ob, lawv siv tau rau graphite. Txwv tsis pub lawv yuav poob.
Carbon-c-relay ces xa cov metrics mus rau Graphite pawg. Peb siv Carbon-cache, rewritten nyob rau hauv Go, raws li lub ntsiab cia ntawm metrics. Go-carbon, vim nws txoj kev sib txuas ntau, deb tshaj Carbon-cache. Nws tau txais cov ntaub ntawv thiab sau nws rau disks siv lub pob ntxhi (tus qauv, sau hauv python). Txhawm rau nyeem cov ntaub ntawv los ntawm peb cov chaw khaws cia, peb siv Graphite API. Nws yog nrawm dua li tus qauv Graphite WEB. Yuav ua li cas rau cov ntaub ntawv tom ntej?
Lawv mus rau Grafana. Peb siv peb cov graphite pawg ua cov ntaub ntawv tseem ceeb, ntxiv rau peb muaj Grafana ua lub vev xaib sib cuam tshuam rau kev nthuav tawm cov ntsuas ntsuas thiab lub tsev dashboards. Rau txhua qhov ntawm lawv cov kev pabcuam, cov neeg tsim khoom tsim lawv tus kheej dashboard. Tom qab ntawd lawv tsim cov duab raws li lawv, uas tso saib cov metrics lawv sau los ntawm lawv daim ntawv thov. Ntxiv rau Grafana, peb kuj muaj SLAM. Qhov no yog python dab uas suav SLA raws li cov ntaub ntawv los ntawm graphite. Raws li kuv twb tau hais lawm, peb muaj ntau lub kaum os microservices, txhua tus muaj nws tus kheej xav tau. Siv SLAM, peb mus rau hauv cov ntaub ntawv thiab muab piv nrog qhov muaj nyob hauv Graphite thiab sib piv cov kev xav tau zoo npaum li cas ntawm peb cov kev pabcuam.
Cia peb mus ntxiv: ceeb toom. Nws yog npaj siv lub zog muaj zog - Moira. Nws yog ywj siab vim nws muaj nws tus kheej Graphite nyob rau hauv lub hood. Tsim los ntawm cov hais mav los ntawm SKB "Kontur", sau nyob rau hauv python thiab mus, kiag li qhib qhov chaw. Moira tau txais tib qhov ntws mus rau hauv graphites. Yog tias vim li cas koj qhov chaw cia tuag, koj qhov kev ceeb toom tseem yuav ua haujlwm.
Peb xa Moira hauv Kubernetes; nws siv pawg ntawm Redis servers ua cov ntaub ntawv tseem ceeb. Qhov tshwm sim yog ib qho txhaum-tolerant system. Nws piv cov kwj ntawm metrics nrog cov npe ntawm qhov ua rau: yog tias tsis muaj cov lus hais hauv nws, ces nws poob qhov ntsuas. Yog li nws muaj peev xwm zom cov gigabytes ntawm metrics ib feeb.
Peb kuj tau txuas ib lub tuam txhab LDAP rau nws, nrog kev pab los ntawm txhua tus neeg siv ntawm lub tuam txhab kev lag luam tuaj yeem tsim cov ntawv ceeb toom rau lawv tus kheej raws li cov uas twb muaj lawm (los yog tsim tshiab) ua rau. Txij li Moira muaj Graphite, nws txhawb nqa tag nrho nws cov nta. Yog li koj thawj zaug coj kab thiab luam rau hauv Grafana. Saib yuav ua li cas cov ntaub ntawv nthuav tawm ntawm daim duab. Thiab ces koj coj tib kab thiab luam rau hauv Moira. Koj dai nws nrog txwv thiab tau txais kev ceeb toom ntawm qhov tso zis. Ua txhua yam no, koj tsis tas yuav muaj kev paub tshwj xeeb. Moira tuaj yeem ceeb toom ntawm SMS, email, Jira, Slack... Nws kuj txhawb kev ua tiav ntawm cov ntawv sau. Thaum qhov tshwm sim tshwm sim rau nws, thiab nws tau sau npe rau cov ntawv sau los yog binary, nws khiav nws thiab xa JSON rau stdin rau qhov binary no. Yog li, koj qhov kev zov me nyuam yuav tsum txheeb xyuas nws. Koj yuav ua li cas nrog JSON no yog nyob ntawm koj. Yog tias koj xav tau, xa mus rau Telegram, yog tias koj xav tau, qhib cov haujlwm hauv Jira, ua dab tsi.
Peb kuj siv peb tus kheej txoj kev loj hlob rau kev ceeb toom - Imagotag. Peb hloov kho lub vaj huam sib luag, uas feem ntau yog siv rau cov nqi hluav taws xob hauv khw muag khoom, kom haum rau peb cov kev xav tau. Peb nqa cov txiaj ntsig los ntawm Moira rau nws. Nws qhia tias lawv nyob hauv xeev dab tsi thiab thaum lawv tshwm sim. Qee qhov kev txhim kho hais mav tso tseg cov ntawv ceeb toom hauv Slack thiab email hauv kev pom zoo ntawm lub vaj huam sib luag no.
Zoo, txij li peb yog ib lub tuam txhab kev vam meej, peb kuj tau saib xyuas Kubernetes hauv qhov system no. Peb suav nrog nws hauv qhov system siv Heapster, uas peb tau teeb tsa hauv pawg, nws sau cov ntaub ntawv thiab xa mus rau Graphite. Yog li ntawd, daim duab zoo li no:
Saib xyuas cov Cheebtsam
Ntawm no yog ib daim ntawv teev cov txuas mus rau cov khoom uas peb tau siv rau txoj haujlwm no. Tag nrho cov ntawm lawv yog qhib qhov chaw.
Graphite:
- carbon hlau:
- ntxhi:
- graphite-pab:
Cov pa roj carbon-c-relay:
Brubeck:
Sau:
Moira:
Grafana:
Heapster:
Π‘ΡΠ°ΡΠΈΡΡΠΈΠΊΠ°
Thiab ntawm no yog qee tus lej hais txog qhov system ua haujlwm li cas rau peb.
Aggregator (Brubeck)
Tus lej ntsuas: ~ 300 / sec
Lub sijhawm xa cov metrics rau Graphite: 30 sec
Kev siv neeg rau zaub mov: ~ 6% CPU (peb tab tom tham txog cov servers puv npo); ~ 1 Gb RAM; ~ 3 Mbps LAN
Graphite (mus-carbon)
Tus lej ntsuas: ~ 1 / min
Metrics hloov tshiab ncua sij hawm: 30 sec
Metrics cia tswv yim: 30sec 35d, 5min 90d, 10min 365d (ua rau koj nkag siab txog dab tsi tshwm sim rau cov kev pab cuam nyob rau lub sij hawm ntev)
Kev siv neeg rau zaub mov: ~ 10% CPU; ~ 20Gb RAM; ~ 30 Mbps LAN
Flexibility
Peb ntawm Avito yeej muaj txiaj ntsig zoo hauv peb cov kev pabcuam saib xyuas. Vim li cas nws thiaj ua tau zoo li no? Ua ntej, nws cov khoom sib hloov pauv tau: ob qho tib si ntawm lawv tus kheej thiab lawv cov qauv. Thib ob, kev txhawb nqa. Txij li thaum tag nrho qhov project yog qhib qhov chaw, koj tuaj yeem hloov kho cov cai ntawm koj tus kheej, hloov pauv, thiab siv cov haujlwm uas tsis muaj nyob hauv lub thawv. Cov pawg ntau heev yog siv, feem ntau yog Mus thiab Python, yog li qhov no ua tiav yooj yim heev.
Ntawm no yog ib qho piv txwv ntawm qhov teeb meem tiag tiag. Ib qho metric hauv Graphite yog cov ntaub ntawv. Nws muaj lub npe. Cov ntaub ntawv npe = metric lub npe. Thiab muaj ib txoj kev mus rau qhov ntawd. Cov npe ntaub ntawv hauv Linux raug txwv rau 255 cim. Thiab peb muaj (raws li "cov neeg siv khoom sab hauv") cov txiv neej los ntawm chav haujlwm database. Lawv qhia peb: βPeb xav saib xyuas peb cov lus nug SQL. Thiab lawv tsis yog 255 cim, tab sis 8 MB txhua. Peb xav muab tso rau hauv Grafana, saib qhov tsis muaj rau qhov kev thov no, thiab txawm zoo dua, peb xav pom cov ntawv thov saum toj no. Nws yuav zoo heev yog tias nws tshwm sim hauv lub sijhawm. Nws yuav yog qhov txias heev kom muab lawv tso rau hauv kev ceeb toom. "
Piv txwv SQL query yog coj los ua piv txwv los ntawm
Peb teeb tsa Redis server thiab siv peb cov plugins sau, uas mus rau Postgres thiab nqa tag nrho cov ntaub ntawv los ntawm qhov ntawd, xa cov ntsuas mus rau Graphite. Tab sis peb hloov lub npe metric nrog hashes. Peb ib txhij xa tib lub hash rau Redis ua tus yuam sij, thiab tag nrho cov lus nug SQL raws li tus nqi. Txhua yam peb yuav tsum ua yog xyuas kom Grafana tuaj yeem mus rau Redis thiab nqa cov ntaub ntawv no. Peb tab tom qhib Graphite API vim ... qhov no yog lub ntsiab interface rau kev sib cuam tshuam ntawm tag nrho cov saib xyuas Cheebtsam nrog graphite, thiab peb nkag mus rau ib tug tshiab muaj nuj nqi muaj hu ua aliasByHash() - los ntawm Grafana peb tau txais lub npe ntawm lub metric, thiab siv nws nyob rau hauv ib tug thov rau Redis raws li ib tug tseem ceeb, nyob rau hauv cov ntaub ntawv no. teb peb tau txais tus nqi ntawm tus yuam sij, uas yog peb "SQL lus nug" " Yog li, peb tau tso tawm hauv Grafana cov lus nug ntawm SQL, uas nyob rau hauv txoj kev xav yog tsis yooj yim sua kom pom muaj, nrog rau cov txheeb cais ntawm nws (hu, kab, total_time, ...).
Cov txiaj ntsim tau los
Qhov Txaj Muag Peb cov kev pabcuam saib xyuas muaj 24/7 los ntawm txhua daim ntawv thov thiab txhua qhov chaws. Yog tias koj nkag mus rau cov chaw khaws cia, koj tuaj yeem sau cov ntaub ntawv mus rau qhov kev pabcuam. Cov lus tsis tseem ceeb, kev txiav txim siab tsis tseem ceeb. Koj tsuas yog yuav tsum paub yuav ua li cas qhib lub qhov (socket), muab lub ntsuas ntsuas rau ntawd thiab kaw lub qhov (socket).
Kev ntseeg Tag nrho cov khoom yog ua txhaum cai thiab tswj peb cov khoom zoo.
Tsawg barrier rau kev nkag. Txhawm rau siv qhov system no, koj tsis tas yuav kawm cov lus programming thiab cov lus nug hauv Grafana. Tsuas yog qhib koj daim ntawv thov, nkag mus rau hauv lub qhov (socket) uas yuav xa cov ntsuas rau Graphite, kaw nws, qhib Grafana, tsim dashboards nyob ntawd thiab saib tus cwj pwm ntawm koj qhov ntsuas, tau txais cov ntawv ceeb toom los ntawm Moira.
Kev ywj pheej. Koj tuaj yeem ua txhua yam no koj tus kheej, yam tsis muaj kev pab los ntawm DevOps engineers. Thiab qhov no yog qhov zoo dua, vim tias koj tuaj yeem saib xyuas koj qhov project tam sim no, koj tsis tas yuav nug leej twg - txawm pib ua haujlwm lossis hloov pauv.
Peb lub hom phiaj yog dab tsi?
Txhua yam uas tau teev tseg hauv qab no tsis yog tsuas yog kev xav paub daws teeb xwb, tab sis qee yam ntawm qhov tsawg kawg yog thawj kauj ruam tau ua.
- Tsis pom kev detector. Peb xav tsim ib qho kev pabcuam uas yuav mus rau peb cov Graphite cia thiab xyuas txhua qhov kev ntsuas uas siv ntau yam algorithms. Muaj twb muaj algorithms uas peb xav mus saib, muaj cov ntaub ntawv, peb paub yuav ua li cas ua hauj lwm nrog nws.
- Metadata. Peb muaj ntau yam kev pabcuam, lawv hloov pauv raws sijhawm, ib yam li cov neeg ua haujlwm nrog lawv. Kev tswj cov ntaub ntawv tas li manually tsis yog ib qho kev xaiv. Tias yog vim li cas peb tam sim no embedding metadata rau hauv peb microservices. Nws hais tias leej twg tsim nws, cov lus nws cuam tshuam nrog, SLA cov cai, qhov twg thiab rau leej twg yuav tsum tau xa cov ntawv ceeb toom. Thaum siv ib qho kev pabcuam, txhua qhov chaw cov ntaub ntawv raug tsim los ntawm nws tus kheej. Raws li qhov tshwm sim, koj tau txais ob qhov txuas - ib qho los ua rau, lwm qhov rau dashboards hauv Grafana.
- Saib xyuas hauv txhua lub tsev. Peb ntseeg tias txhua tus neeg tsim khoom yuav tsum siv cov txheej txheem zoo li no. Hauv qhov no, koj ib txwm nkag siab tias koj txoj kev tsheb khiav mus rau qhov twg, qhov twg tshwm sim, nws ntog, qhov twg nws qhov tsis muaj zog. Yog hais tias, piv txwv li, ib yam dab tsi los thiab poob koj cov kev pab cuam, ces koj yuav kawm txog nws tsis yog thaum lub sij hawm hu los ntawm tus thawj tswj, tab sis los ntawm ib tug ceeb toom, thiab koj muaj peev xwm tam sim ntawd qhib cov ntaub ntawv tshiab thiab saib dab tsi tshwm sim nyob rau ntawd.
- Kev ua haujlwm siab. Peb qhov project yog tas li loj hlob, thiab niaj hnub no nws ua txog 2 metric nqi ib feeb. Ib xyoo dhau los, daim duab no yog 000. Thiab qhov kev loj hlob txuas ntxiv, thiab qhov no txhais tau hais tias tom qab qee lub sijhawm Graphite (ntxhi) yuav pib hnyav rau lub disk subsystem. Raws li kuv twb tau hais lawm, qhov kev tshuaj xyuas no yog qhov dav dav vim muaj kev sib hloov ntawm cov khoom. Qee tus tswj hwm thiab txuas ntxiv txuas ntxiv lawv cov kev tsim kho tshwj xeeb rau Graphite, tab sis peb txiav txim siab mus rau lwm txoj hauv kev: siv raws li ib tug repository rau peb metrics. Qhov kev hloov pauv no yuav luag tiav, thiab sai sai no kuv yuav qhia rau koj paub ntau ntxiv tias qhov no tau ua li cas: qhov nyuaj li cas thiab lawv tau kov yeej li cas, cov txheej txheem tsiv teb tsaws mus li cas, Kuv yuav piav qhia txog cov khoom uas tau xaiv los ua kev khi thiab lawv cov teeb tsa.
Ua tsaug rau koj mloog! Nug koj cov lus nug ntawm lub ncauj lus, Kuv yuav sim teb ntawm no lossis hauv cov lus hauv qab no. Tej zaum ib tug neeg muaj kev paub txog kev tsim cov kev soj ntsuam zoo sib xws lossis hloov mus rau Clickhouse hauv qhov xwm txheej zoo sib xws - qhia nws hauv cov lus.
Tau qhov twg los: www.hab.com