Kuv lub npe yog Anton Baderin. Kuv ua haujlwm ntawm High Technology Center thiab ua kev tswj hwm. Ib lub hlis dhau los, peb lub tuam txhab kev sib tham tau xaus, qhov uas peb tau qhia peb cov kev paub dhau los nrog IT zej zog ntawm peb lub nroog. Kuv tham txog kev saib xyuas cov ntawv thov web. Cov khoom siv tau npaj rau qib qis lossis nruab nrab, uas tsis tsim cov txheej txheem no los ntawm kos.
Lub hauv paus tseem ceeb ntawm txhua qhov kev saib xyuas yog daws teeb meem kev lag luam. Kev saib xyuas rau kev saib xyuas tsis muaj txiaj ntsig rau leej twg. Kev lag luam xav tau dab tsi? Yog li ntawd txhua yam ua haujlwm sai thiab tsis muaj qhov yuam kev. Cov lag luam xav kom muaj kev ua haujlwm, kom peb tus kheej txheeb xyuas cov teeb meem hauv kev pabcuam thiab kho lawv sai li sai tau. Cov no, qhov tseeb, yog cov teeb meem uas kuv tau daws tag nrho xyoo tas los ntawm ib qhov project rau ib qho ntawm peb cov neeg siv khoom.
Hais txog qhov project
Qhov project yog ib qho ntawm cov kev pab cuam loj tshaj plaws hauv lub tebchaws. Peb pab cov khw muag khoom lag luam nce ntau zaus ntawm kev muag khoom los ntawm ntau yam khoom lag luam xws li daim npav rho nyiaj. Nyob rau hauv tag nrho, qhov project suav nrog 14 daim ntawv thov uas khiav ntawm kaum servers.
Thaum lub sijhawm xam phaj, kuv rov pom dua tias cov thawj coj tsis ib txwm mus saib xyuas lub vev xaib kom raug: ntau tus tseem tsom mus rau kev ntsuas kev ua haujlwm thiab qee zaus saib xyuas cov kev pabcuam.
Hauv kuv qhov xwm txheej, cov neeg siv khoom saib xyuas qhov system yav dhau los ua raws li Icinga. Nws tsis tau daws cov teeb meem saum toj no hauv txhua txoj kev. Feem ntau tus neeg siv khoom nws tus kheej qhia peb txog cov teeb meem, thiab feem ntau tsis yog, peb tsuas yog tsis muaj cov ntaub ntawv txaus kom nkag mus rau hauv qab ntawm qhov laj thawj.
Tsis tas li ntawd, muaj kev nkag siab meej txog qhov tsis muaj txiaj ntsig ntawm nws txoj kev txhim kho ntxiv. Kuv xav tias cov uas paub txog Icinga yuav nkag siab kuv. Yog li, peb tau txiav txim siab los tsim kho tag nrho lub vev xaib kev tshuaj xyuas qhov system rau qhov project.
Prometheus
Peb xaiv Prometheus raws li peb lub ntsiab lus qhia:
- Ib tug loj tus naj npawb ntawm cov metrics muaj. Hauv peb cov ntaub ntawv muaj 60 txhiab ntawm lawv. Tau kawg, nws tsim nyog sau cia tias peb tsis siv feem ntau ntawm lawv (tej zaum txog 95%). Ntawm qhov tod tes, lawv txhua tus kuj pheej yig. Rau peb, qhov no yog lwm qhov huab piv rau yav dhau los siv Icinga. Nyob rau hauv nws, ntxiv metrics yog ib qho mob tshwj xeeb: cov uas twb muaj lawm yog kim (tsuas yog saib ntawm qhov chaws ntawm ib qho plugin). Ib qho plugin yog tsab ntawv hauv Bash lossis Python, qhov kev tshaj tawm uas kim heev ntawm cov peev txheej siv.
- Qhov system no siv cov peev txheej me me. 600 MB ntawm RAM, 15% ntawm ib qho tseem ceeb thiab ob peb lub kaum os IOPS yog txaus rau tag nrho peb cov kev ntsuas. Tau kawg, koj yuav tsum khiav metrics exporters, tab sis lawv txhua tus tau sau rau hauv Go thiab kuj tsis muaj zog heev tshaib plab. Kuv tsis xav tias hauv qhov tseeb niaj hnub no yog qhov teeb meem.
- Muab lub peev xwm los mus rau Kubernetes. Xav txog cov neeg siv khoom cov phiaj xwm, qhov kev xaiv yog pom tseeb.
ELK
Yav dhau los, peb tsis tau sau lossis txheej txheem cav. Cov shortcomings yog meej rau txhua leej txhua tus. Peb xaiv ELK vim peb twb muaj kev paub txog qhov system no lawm. Peb tsuas khaws cov ntawv thov nkag rau ntawd. Cov txheej txheem xaiv tseem ceeb yog kev tshawb nrhiav cov ntawv nyeem thiab nws qhov nrawm.
Lub tsev
Thaum xub thawj, qhov kev xaiv poob rau InfluxDB. Peb pom tau hais tias yuav tsum tau sau Nginx cav, txheeb cais los ntawm pg_stat_statements, thiab khaws cov ntaub ntawv keeb kwm ntawm Prometheus. Peb tsis nyiam Influx vim tias nws pib noj ntau lub cim xeeb thiab poob. Tsis tas li ntawd, kuv xav ua pab pawg lus nug los ntawm remote_addr, tab sis kev sib koom ua ke hauv DBMS no tsuas yog los ntawm cov cim npe. Cov cim yog kim (nco), lawv tus lej muaj cai txwv.
Peb pib tshawb nrhiav dua. Qhov xav tau yog qhov kev txheeb xyuas cov ntaub ntawv nrog cov peev txheej tsawg, nyiam dua nrog cov ntaub ntawv compression ntawm disk.
Clickhouse ua tau raws li tag nrho cov qauv no, thiab peb yeej tsis khuv xim peb qhov kev xaiv. Peb tsis sau cov ntaub ntawv ntau ntxiv rau hauv nws (tus naj npawb ntawm cov ntawv ntxig tsuas yog li tsib txhiab ib feeb xwb).
NewRelic
NewRelic muaj keeb kwm nrog peb vim nws yog tus neeg siv khoom xaiv. Peb siv nws ua APM.
Zabbix
Peb siv Zabbix nkaus xwb los saib xyuas lub thawv Dub ntawm ntau yam APIs.
Defining ib txoj kev saib xyuas
Peb xav kom decompose txoj haujlwm thiab yog li systematize txoj hauv kev los saib xyuas.
Txhawm rau ua qhov no, kuv faib peb qhov system rau hauv cov theem hauv qab no:
- hardware thiab VMS;
- kev khiav hauj lwm qhov system;
- kev pab cuam, software pawg;
- daim ntawv thov;
- kev lag luam logic.
Vim li cas txoj kev no yooj yim:
- peb paub tias leej twg yog tus ua haujlwm ntawm txhua qib thiab, raws li qhov no, peb tuaj yeem xa cov lus ceeb toom;
- peb tuaj yeem siv cov qauv thaum txwv kev ceeb toom - nws yuav yog qhov txawv txav kom xa cov lus ceeb toom txog qhov tsis muaj database thaum lub tshuab virtual tag nrho tsis muaj.
Txij li thaum peb txoj haujlwm yog txhawm rau txheeb xyuas qhov ua txhaum cai hauv kev ua haujlwm ntawm lub kaw lus, peb yuav tsum nyob rau txhua theem qhia txog qee qhov kev ntsuas uas tsim nyog yuav tsum tau them rau thaum sau ntawv ceeb toom cov cai. Tom ntej no, cia peb mus dhau theem "VMS", "Operating system" thiab "System services, software stack".
Cov tshuab virtual
Hosting faib peb lub processor, disk, nco thiab network. Thiab peb muaj teeb meem nrog thawj ob. Yog li, metrics:
CPU nyiag lub sij hawm - thaum koj yuav lub tshuab virtual ntawm Amazon (t2.micro, piv txwv li), koj yuav tsum nkag siab tias koj tsis tau faib tag nrho cov tub ntxhais processor, tab sis tsuas yog quota ntawm nws lub sijhawm. Thiab thaum koj tso nws, processor yuav raug tshem tawm ntawm koj.
Qhov kev ntsuas no tso cai rau koj taug qab lub sijhawm zoo li no thiab txiav txim siab. Piv txwv li, nws puas yuav tsum tau noj cov nqi se fatter lossis faib cov kev ua haujlwm tom qab thiab API thov rau cov servers sib txawv?
IOPS + CPU iowait lub sijhawm - rau qee qhov laj thawj, ntau huab hostings kev txhaum los ntawm tsis muab IOPS txaus. Ntxiv mus, lub sijhawm nrog IOPS qis tsis yog qhov kev sib cav rau lawv. Yog li ntawd, nws tsim nyog sau CPU iowait. Nrog rau ob daim duab no - nrog IOPS qis thiab siab I / O tos - koj tuaj yeem tham nrog hosting thiab daws qhov teeb meem.
operating system
Operating System metrics:
- tus nqi ntawm lub cim xeeb muaj nyob hauv %;
- swap kev ua haujlwm: vmstat swapin, swapout;
- tus naj npawb ntawm inodes muaj thiab qhov chaw dawb ntawm cov ntaub ntawv hauv %
- nruab nrab load;
- tus naj npawb ntawm kev sib txuas hauv lub xeev tw;
- conntrack lub rooj puv;
- Qhov zoo ntawm lub network tuaj yeem saib xyuas siv cov khoom siv hluav taws xob ss, iproute2 pob - tau txais qhov taw qhia ntawm RTT kev sib txuas los ntawm nws cov zis thiab pab pawg los ntawm qhov chaw nres nkoj dest.
Tsis tas li ntawd nyob rau theem kev khiav hauj lwm peb muaj xws li ib qho chaw raws li cov txheej txheem. Nws yog ib qho tseem ceeb los txheeb xyuas cov txheej txheem ntawm cov txheej txheem uas ua lub luag haujlwm tseem ceeb hauv nws txoj haujlwm. Yog tias, piv txwv li, koj muaj ob peb pgpools, ces koj yuav tsum tau sau cov ntaub ntawv rau lawv txhua tus.
Cov txheej txheem metrics yog raws li nram no:
- CPUs;
- lub cim xeeb yog nyob rau hauv feem ntau;
- IO - nyiam dua hauv IOPS;
- FileFd - qhib thiab txwv;
- cov nplooj ntawv tseem ceeb tsis ua haujlwm - txoj kev no koj tuaj yeem nkag siab tias cov txheej txheem twg raug hloov pauv.
Peb xa tag nrho cov kev soj ntsuam hauv Docker, thiab peb siv Tus Kws Pab Tswv Yim los sau cov ntaub ntawv ntsuas. Ntawm lwm lub tshuab peb siv cov txheej txheem-exporter.
Cov kev pab cuam, software pawg
Txhua daim ntawv thov muaj nws tus kheej tshwj xeeb, thiab nws yog ib qho nyuaj rau ib qho kev qhia tshwj xeeb.
Cov txheej txheem universal yog:
- thov tus nqi;
- tus lej yuam kev;
- latency;
- saturation
Peb cov piv txwv zoo tshaj plaws ntawm kev saib xyuas ntawm qib no yog Nginx thiab PostgreSQL.
Qhov kev pabcuam thauj khoom tshaj plaws hauv peb lub cev yog lub database. Yav dhau los, peb feem ntau muaj teeb meem nrhiav seb lub database ua dab tsi.
Peb pom lub siab load ntawm cov disks, tab sis cov cav qeeb tsis tshua pom dab tsi. Peb daws qhov teeb meem no siv pg_stat_statements, ib qho kev pom uas sau cov lus nug txheeb cais.
Qhov ntawd yog tag nrho cov admin xav tau.
Peb tsim cov duab ntawm cov haujlwm ntawm kev nyeem thiab sau ntawv thov:
Txhua yam yog yooj yim thiab meej, txhua qhov kev thov muaj nws tus kheej xim.
Ib qho piv txwv zoo sib xws yog Nginx cav. Nws tsis yog qhov xav tsis thoob tias ob peb tus neeg txheeb xyuas lawv lossis hais lawv hauv cov npe yuav tsum muaj. Cov qauv qauv tsis yog cov ntaub ntawv ntau thiab yuav tsum tau nthuav dav.
Tus kheej, Kuv ntxiv thov_time, upstream_response_time, body_bytes_sent, request_length, request_id. Peb npaj cov sij hawm teb thiab cov lej yuam kev:
Peb tsim cov duab ntawm lub sijhawm teb thiab tus lej ntawm qhov yuam kev. Nco ntsoov? Kuv puas tau tham txog kev ua lag luam? Yuav kom sai thiab tsis muaj qhov yuam kev? Peb twb tau them cov teeb meem no nrog ob daim duab. Thiab koj tuaj yeem hu rau cov thawj coj ntawm lub luag haujlwm siv lawv.
Tab sis ib qho teeb meem ntxiv tseem nyob - los xyuas kom meej qhov kev tshem tawm sai ntawm qhov ua rau ntawm qhov teeb meem.
Kev daws teeb meem
Tag nrho cov txheej txheem los ntawm kev txheeb xyuas txog kev daws teeb meem tuaj yeem muab faib ua ntau theem:
- txheeb xyuas qhov teeb meem;
- ceeb toom rau tus thawj tswj haujlwm;
- teb rau qhov xwm txheej;
- tshem tawm vim li cas.
Nws yog ib qho tseem ceeb uas peb yuav tsum ua qhov no kom sai li sai tau. Thiab yog tias nyob rau theem ntawm kev txheeb xyuas qhov teeb meem thiab xa cov ntawv ceeb toom peb tsis tuaj yeem tau txais ntau lub sijhawm - ob feeb yuav siv rau lawv nyob rau hauv txhua rooj plaub, tom qab ntawd cov tom ntej tsuas yog unplowed teb rau kev txhim kho.
Cia li xav tias tus tub ceev xwm lub xov tooj nrov. Nws yuav ua li cas? Nrhiav cov lus teb rau cov lus nug - dab tsi tawg, qhov twg nws tawg, yuav ua li cas? Nov yog qhov peb teb cov lus nug no:
Peb tsuas suav tag nrho cov ntaub ntawv no hauv cov ntawv ceeb toom, muab nws txuas rau nplooj ntawv wiki uas piav qhia yuav ua li cas teb cov teeb meem no, yuav daws li cas thiab nce ntxiv.
Kuv tseem tsis tau hais dab tsi txog daim ntawv thov txheej thiab kev lag luam logic. Hmoov tsis zoo, peb cov ntawv thov tseem tsis tau siv cov metrics sau. Tsuas yog qhov chaw ntawm cov ntaub ntawv los ntawm cov qib no yog cov cav.
Ob peb lub ntsiab lus.
Ua ntej, sau cov txheej txheem cav. Tsis tas yuav suav nrog cov ntsiab lus hauv cov ntawv ntawm cov lus. Qhov no ua rau lawv nyuaj rau pawg thiab txheeb xyuas. Logstash siv sijhawm ntev los normalize tag nrho cov no.
Qhov thib ob, siv cov qib hnyav kom raug. Txhua yam lus muaj nws tus qauv. Tus kheej, kuv paub qhov txawv plaub theem:
- tsis muaj qhov yuam kev;
- tus neeg sab nrauv yuam kev;
- qhov yuam kev yog nyob ntawm peb sab, peb tsis poob nyiaj, peb tsis ris kev pheej hmoo;
- Qhov yuam kev ntawm peb sab, peb poob nyiaj.
Cia kuv piav qhia. Koj yuav tsum sim tsim kev soj ntsuam raws li kev lag luam logic. Sim saib xyuas daim ntawv thov nws tus kheej thiab ua haujlwm nrog cov kev ntsuas xws li tus naj npawb ntawm kev muag khoom, tus naj npawb ntawm cov neeg siv npe tshiab, tus naj npawb ntawm cov neeg siv tam sim no, thiab lwm yam.
Yog tias koj lub lag luam tag nrho yog ib lub pob hauv qhov browser, koj yuav tsum tau saib xyuas seb nws clicks thiab ua haujlwm zoo. Tag nrho cov seem tsis muaj teeb meem.
Yog tias koj tsis muaj qhov no, koj tuaj yeem sim ntes nws hauv daim ntawv thov, Nginx cav, thiab lwm yam, raws li peb tau ua. Koj yuav tsum nyob ze rau ntawm daim ntawv thov li sai tau.
Kev ntsuas kev ua haujlwm yog qhov tseem ceeb, tab sis kev lag luam tsis txaus siab rau lawv, peb tsis tau them nyiaj rau lawv.
Tau qhov twg los: www.hab.com