Dzina langa ndine Anton Baderin. Ndimagwira ntchito ku High Technology Center ndikuchita kasamalidwe kazinthu. Mwezi wapitawu, msonkhano wathu wamakampani udatha, pomwe tidagawana zomwe takumana nazo ndi gulu la IT mumzinda wathu. Ndinayankhula za kuyang'anira ntchito zapaintaneti. Nkhaniyi idapangidwira kwa achichepere kapena apakati, omwe sanapange izi kuyambira pachiyambi.
Mwala wapangodya womwe umayang'anira njira iliyonse yowunikira ndikuthetsa mavuto abizinesi. Kuyang'anira ndi cholinga chowunika sikukhudza aliyense. Kodi bizinesi ikufuna chiyani? Kotero kuti zonse zimagwira ntchito mofulumira komanso popanda zolakwika. Mabizinesi amafuna kukhala okhazikika, kotero kuti ife tokha tizindikire zovuta muutumiki ndikuzikonza mwachangu momwe tingathere. Izi, kwenikweni, ndizovuta zomwe ndidazithetsa chaka chathachi pantchito ya m'modzi mwa makasitomala athu.
Ponena za polojekiti
Ntchitoyi ndi imodzi mwamapulogalamu akuluakulu a kukhulupirika mdziko muno. Timathandizira maunyolo ogulitsa kukulitsa kuchuluka kwa malonda kudzera mu zida zosiyanasiyana zotsatsa monga makhadi a bonasi. Ponseponse, polojekitiyi ikuphatikizapo mapulogalamu 14 omwe amayendetsa ma seva khumi.
Panthawi yofunsa mafunso, ndidazindikira mobwerezabwereza kuti ma admins samayandikira njira zowunikira momwe mawebusayiti amayendera: ambiri amangoyang'anabe makina ogwiritsira ntchito ndipo nthawi zina amawunika ntchito.
Kwa ine, njira yowunikira kasitomala idakhazikitsidwa kale ndi Icinga. Sizinathetse mavuto omwe ali pamwambawa m'njira iliyonse. Nthawi zambiri kasitomala mwiniwakeyo amatiuza za mavuto, ndipo nthawi zambiri, tinalibe deta yokwanira kuti tipeze chifukwa chake.
Kuonjezera apo, panali kumvetsetsa bwino za kupanda pake kwa chitukuko chake chowonjezereka. Ndikuganiza kuti omwe amandidziwa bwino Icinga andimvetsetsa. Chifukwa chake, tidaganiza zokonzanso dongosolo loyang'anira ntchito pa intaneti.
Prometheus
Tinasankha Prometheus kutengera zizindikiro zazikulu zitatu:
- Chiwerengero chachikulu cha ma metric omwe alipo. Kwa ife pali 60 zikwi za iwo. Zachidziwikire, ndikofunikira kudziwa kuti sitigwiritsa ntchito ambiri aiwo (mwina pafupifupi 95%). Kumbali ina, onse ndi otsika mtengo. Kwa ife, ichi ndi china chonyanyira poyerekeza ndi Icinga yomwe idagwiritsidwa ntchito kale. Mmenemo, kuwonjezera ma metrics kunali kupweteka kwapadera: zomwe zinalipo zinali zodula (ingoyang'anani pa code source ya plugin iliyonse). Pulagi iliyonse inali script ku Bash kapena Python, kukhazikitsidwa kwake komwe kumakhala kokwera mtengo malinga ndi zomwe zimagwiritsidwa ntchito.
- Dongosololi limawononga ndalama zochepa. 600 MB ya RAM, 15% ya core imodzi ndi ma IOPS angapo ndi okwanira ma metric athu onse. Zachidziwikire, muyenera kuyendetsa ma metrics otumiza kunja, koma onse amalembedwa mu Go ndipo alibenso njala yamphamvu. Sindikuganiza kuti muzochitika zamakono izi ndizovuta.
- Amapereka mwayi wosamukira ku Kubernetes. Poganizira mapulani a kasitomala, kusankha ndi koonekeratu.
mbawala
M'mbuyomu, sitinatole kapena kukonza zipika. Zolakwazo zimawonekera kwa aliyense. Tinasankha ELK chifukwa tinali ndi chidziwitso kale ndi dongosolo lino. Timangosunga zolemba zamapulogalamu pamenepo. Zosankha zazikuluzikulu zinali kufufuza zolemba zonse ndi liwiro lake.
Π‘lickhouse
Poyamba, chisankhocho chinagwera pa InfluxDB. Tinazindikira kufunika kosonkhanitsa zipika za Nginx, ziwerengero kuchokera ku pg_stat_statements, ndikusunga mbiri yakale ya Prometheus. Sitinakonde Influx chifukwa nthawi ndi nthawi idayamba kukumbukira zinthu zambiri ndikuwonongeka. Kuphatikiza apo, ndimafuna kuyika mafunso ndi remote_addr, koma kupanga magulu mu DBMS iyi ndi ma tag okha. Ma tag ndi okwera mtengo (kukumbukira), chiwerengero chawo ndi chochepa.
Tinayambanso kufufuza kwathu. Chomwe chinkafunika chinali nkhokwe yowunikira yokhala ndi zinthu zochepa, makamaka ndi kukakamiza kwa data pa disk.
Clickhouse imakwaniritsa izi zonse, ndipo sitinadandaulepo ndi zomwe tasankha. Sitilemba kuchuluka kwazinthu zachilendo mmenemo (chiwerengero cha zoyikapo ndi pafupifupi zikwi zisanu pa mphindi imodzi).
NewRelic
NewRelic idakhala nafe mbiri yakale chifukwa chinali chisankho cha kasitomala. Timagwiritsa ntchito ngati APM.
Zabbix
Timagwiritsa ntchito Zabbix kuyang'anira Black Box ya ma API osiyanasiyana.
Kufotokozera Njira Yowunikira
Tinkafuna kuti tiwononge ntchitoyo ndikukonzekera njira yowunikira.
Kuti tichite izi, ndinagawa dongosolo lathu m'magulu otsatirawa:
- hardware ndi VMS;
- opareting'i sisitimu;
- ntchito zamakina, mapulogalamu okwana;
- ntchito;
- bizinesi logic.
Chifukwa chiyani njira iyi ndi yabwino:
- timadziwa yemwe ali ndi udindo pa ntchito ya mlingo uliwonse ndipo, potengera izi, tikhoza kutumiza machenjezo;
- titha kugwiritsa ntchito kapangidwe kake popondereza zidziwitso - zingakhale zachilendo kutumiza chenjezo za kusapezeka kwa database pomwe makina onse sakupezeka.
Popeza ntchito yathu ndikuzindikira zophwanya machitidwe a dongosololi, tiyenera pamlingo uliwonse kuwunikira ma metrics ena omwe ndi oyenera kusamala polemba malamulo ochenjeza. Kenako, tiyeni tidutse magawo "VMS", "Operating System" ndi "System services, software stack".
Makina a Virtual
Hosting imatipatsa purosesa, disk, memory ndi network. Ndipo tinali ndi mavuto ndi awiri oyambirira. Choncho, ma metrics:
CPU yabedwa nthawi - mukagula makina enieni pa Amazon (t2.micro, mwachitsanzo), muyenera kumvetsetsa kuti simunapatsidwe purosesa yonse, koma gawo la nthawi yake. Ndipo mukamaliza, purosesa idzachotsedwa kwa inu.
Metric iyi imakupatsani mwayi wolondolera nthawi ngati izi ndikupanga zisankho. Mwachitsanzo, kodi ndikofunikira kutenga mitengo yotsika kwambiri kapena kugawa kukonzanso kwa ntchito zakumbuyo ndi zopempha za API kumaseva osiyanasiyana?
IOPS + CPU iowait nthawi - pazifukwa zina, osungira mitambo ambiri amachimwa posapereka ma IOPS okwanira. Kuphatikiza apo, ndandanda yokhala ndi ma IOPS otsika si mkangano kwa iwo. Chifukwa chake, ndikofunikira kusonkhanitsa CPU iowait. Ndi ma graph awa - okhala ndi ma IOPS otsika komanso kudikirira kwa I / O - mutha kulankhula kale ndi omwe akuchititsayo ndikuthetsa vutoli.
opaleshoni dongosolo
Ma metric ogwiritsira ntchito:
- kuchuluka kwa kukumbukira komwe kulipo mu%;
- kusinthana kwa ntchito: vmstat swapin, swapout;
- chiwerengero cha ma innode omwe alipo ndi malo aulere pamafayilo mu%
- katundu wapakati;
- chiwerengero cha maulumikizidwe mu tw state;
- contrack tebulo chidzalo;
- Ubwino wa netiweki ukhoza kuyang'aniridwa pogwiritsa ntchito ss, phukusi la iproute2 - pezani chizindikiro cha kulumikizana kwa RTT kuchokera pazotulutsa zake ndikuyika gulu ndi doko la dest.
Komanso pamlingo wogwiritsa ntchito tili ndi gulu ngati njira. Ndikofunika kuzindikira mu dongosolo la ndondomeko zomwe zimagwira ntchito yofunika kwambiri pakugwira ntchito kwake. Ngati, mwachitsanzo, muli ndi ma pgpools angapo, ndiye kuti muyenera kusonkhanitsa zambiri za aliyense wa iwo.
Seti ya metrics ili motere:
- CPUs;
- kukumbukira kumakhala kokhazikika;
- IO - makamaka mu IOPS;
- FileFd - tsegulani ndi malire;
- zolephera zazikulu zamasamba - mwanjira iyi mutha kumvetsetsa zomwe zikusinthidwa.
Timatumiza zowunikira zonse ku Docker, ndipo timagwiritsa ntchito Advisor kusonkhanitsa deta ya ma metrics. Pa makina ena timagwiritsa ntchito process-exporter.
Services System, mapulogalamu okwana
Ntchito iliyonse ili ndi zake zake, ndipo ndizovuta kusankha ma metrics enaake.
Universal set ndi:
- pempho mlingo;
- kuchuluka kwa zolakwika;
- kuchedwa;
- kuchuluka.
Zitsanzo zathu zochititsa chidwi kwambiri zowunikira pamlingo uwu ndi Nginx ndi PostgreSQL.
Ntchito yodzaza kwambiri m'dongosolo lathu ndi database. M'mbuyomu, nthawi zambiri tinkavutika kudziwa zomwe database ikuchita.
Tinawona katundu wapamwamba pa disks, koma zipika zapang'onopang'ono sizinasonyeze kalikonse. Tinathetsa vutoli pogwiritsa ntchito pg_stat_statements, mawonedwe omwe amasonkhanitsa ziwerengero zamafunso.
Ndizo zonse zomwe admin amafunikira.
Timapanga ma graph a ntchito yowerenga ndi kulemba:
Chilichonse ndi chosavuta komanso chomveka, pempho lililonse lili ndi mtundu wake.
Chitsanzo chochititsa chidwi ndi zolemba za Nginx. Ndizosadabwitsa kuti ndi anthu ochepa omwe amawawerengera kapena kuwatchula pamndandanda wazinthu zomwe ziyenera kukhala nazo. Mtundu wokhazikika siwophunzitsa kwambiri ndipo uyenera kukulitsidwa.
Inemwini, ndinawonjezera request_time, upstream_response_time, body_bytes_sent, request_length, request_id. Timakonza nthawi yoyankha ndi kuchuluka kwa zolakwika:
Timamanga ma graph a nthawi yoyankha ndi kuchuluka kwa zolakwika. Mukukumbukira? Kodi ndidalankhula zantchito zamabizinesi? Kuti mwachangu komanso popanda zolakwika? Nkhanizi tazifotokoza kale ndi ma chart awiri. Ndipo mutha kuyimbira kale olamulira omwe ali pantchito kuwagwiritsa ntchito.
Koma vuto linanso lidakalipo - kuonetsetsa kuthetsa mwamsanga zomwe zimayambitsa zochitikazo.
Kusintha kwa zochitika
Njira yonse kuyambira pakuzindikiritsa mpaka kuthetsa vuto ikhoza kugawidwa m'njira zingapo:
- kuzindikira vuto;
- chidziwitso kwa woyang'anira ntchito;
- kuyankha ku chochitika;
- kuthetsa zifukwa.
Mβpofunika kuti tichite zimenezi mwamsanga. Ndipo ngati pazigawo zozindikiritsa vuto ndi kutumiza zidziwitso sitingathe kupeza nthawi yochuluka - mphindi ziwiri zidzathera pa iwo mulimonsemo, ndiye kuti zotsatilazo ndi gawo losalimidwa kuti lisinthe.
Tiyerekeze kuti foni ya mkulu wa ntchitoyo inalira. Kodi adzachita chiyani? Yang'anani mayankho a mafunso - ndi chiyani chinasweka, chinasweka kuti, momwe mungachitire? Umu ndi momwe timayankhira mafunso awa:
Timangophatikiza zonse izi m'mawu achidziwitso, perekani ulalo wa tsamba la wiki lomwe limafotokoza momwe mungayankhire vutoli, momwe mungalithetsere ndikulikulitsa.
Sindinanenebe chilichonse chokhudza gawo la ntchito komanso malingaliro abizinesi. Tsoka ilo, mapulogalamu athu sanagwiritsebe ntchito kusonkhanitsa ma metric. Gwero lokha la chidziwitso chilichonse kuchokera mumagulu awa ndi zolemba.
Mfundo zingapo.
Choyamba, lembani zipika zokhazikika. Palibe chifukwa chophatikiza nkhani m'mawu a uthengawo. Izi zimapangitsa kuti zikhale zovuta kuziyika m'magulu ndi kuzisanthula. Logstash imatenga nthawi yayitali kuti izi zisinthe.
Chachiwiri, gwiritsani ntchito milingo yowuma bwino. Chilankhulo chilichonse chili ndi muyezo wake. Inemwini, ndimasiyanitsa magawo anayi:
- palibe cholakwika;
- cholakwika cha mbali ya kasitomala;
- kulakwitsa kuli kumbali yathu, sititaya ndalama, sitimakhala ndi zoopsa;
- Kulakwitsa kuli kumbali yathu, timataya ndalama.
Ndiloleni ndifotokoze mwachidule. Muyenera kuyesa kupanga zowunikira potengera malingaliro abizinesi. Yesani kuyang'anira pulogalamuyo yokha ndikugwira ntchito ndi ma metric monga kuchuluka kwa malonda, kuchuluka kwa olembetsa atsopano, kuchuluka kwa omwe akugwiritsa ntchito, ndi zina zotero.
Ngati bizinesi yanu yonse ndi batani limodzi mumsakatuli, muyenera kuyang'ana ngati ikudina ndikugwira ntchito moyenera. Zina zonse zilibe kanthu.
Ngati mulibe izi, mutha kuyesa kuzipeza muzolemba zamapulogalamu, zipika za Nginx, ndi zina zotero, monga tinachitira. Muyenera kukhala pafupi ndi pulogalamuyi momwe mungathere.
Miyezo yamakina opangira ntchito ndiyofunikira, koma bizinesi ilibe chidwi ndi iwo, sitimalipidwa.
Source: www.habr.com