Yi la'akari da ra'ayin Kubernetes saka idanu, ku san kayan aikin Prometheus, kuma kuyi magana game da faɗakarwa.
Batun saka idanu yana da yawa, ba za a iya rarraba shi a cikin labarin ɗaya ba. Manufar wannan rubutu ita ce samar da bayyani na kayan aiki, dabaru da hanyoyin da za a bi.
Abubuwan labarin matsi ne daga . Idan kuna son yin cikakken kwas - yi rajista don kwas a kan .
Abin da ake sa ido a cikin gungu na Kubernetes
sabobin jiki. Idan gungun Kubernetes aka tura akan sabar sa, kuna buƙatar saka idanu akan lafiyarsu. Zabbix yana gudanar da wannan aikin; idan kun yi aiki tare da shi, to, ba ku buƙatar ƙi, ba za a sami rikici ba. Zabbix ce ke sa ido kan yanayin sabobin mu.
Mu ci gaba zuwa sa ido a matakin tari.
Abubuwan Gudanar da Jirgin Sama: API, Mai tsarawa da sauransu. Aƙalla, kuna buƙatar tabbatar da cewa API na sabobin ko da dai sauransu ya fi 0. Etcd na iya dawo da ma'auni da yawa: ta faifan diski ɗin da yake jujjuya shi, ta hanyar lafiyar gungun sa da sauransu, da sauransu.
Docker ya bayyana da daɗewa kuma kowa yana sane da matsalolinsa: yawancin kwantena suna haifar da daskarewa da sauran matsaloli. Don haka, Docker kanta, a matsayin tsarin, yakamata a sarrafa shi, aƙalla don samuwa.
dns. Idan DNS ya faɗi a cikin gungu, to, duk sabis ɗin Gano zai faɗi bayansa, kira daga kwasfa zuwa kwas ɗin zai daina aiki. A cikin aikina, babu irin waɗannan matsalolin, amma wannan baya nufin cewa yanayin DNS baya buƙatar kulawa. Neman latency da wasu ma'auni za a iya bin diddigin su akan CoreDNS.
Shiga Wajibi ne don sarrafa wadatar ingresses (ciki har da Mai Gudanar da Ingress) a matsayin wuraren shigarwa zuwa aikin.
An wargaza manyan abubuwan da ke cikin gungu - yanzu bari mu gangara zuwa matakin abstractions.
Zai yi kama da aikace-aikacen suna gudana a cikin kwasfa, wanda ke nufin suna buƙatar sarrafawa, amma a zahiri ba haka bane. Pods ne ephemeral: a yau suna gudu a kan wani uwar garken, gobe a kan wani; yau akwai 10 daga cikinsu, gobe 2. Don haka, babu mai sa ido akan kwasfa. A cikin gine-ginen microservice, yana da mahimmanci don sarrafa samuwar aikace-aikacen gaba ɗaya. Musamman, duba samuwa na ƙarshen sabis: shin wani abu yana aiki? Idan aikace-aikacen yana samuwa, to, abin da ke faruwa a baya, yawancin kwafi yanzu - waɗannan tambayoyi ne na tsari na biyu. Babu buƙatar saka idanu akan al'amuran mutum ɗaya.
A matakin ƙarshe, kuna buƙatar sarrafa aikin aikace-aikacen kanta, ɗaukar ma'aunin kasuwanci: adadin umarni, halayen mai amfani, da sauransu.
Prometheus
Mafi kyawun tsarin kula da gungu shine . Ban san kowane kayan aiki da zai iya dacewa da Prometheus dangane da inganci da sauƙin amfani ba. Yana da kyau ga kayan aiki masu sassauƙa, don haka lokacin da suka ce "Kubernetes saka idanu", yawanci suna nufin Prometheus.
Akwai zaɓuɓɓuka biyu don farawa tare da Prometheus: ta amfani da Helm, zaku iya shigar da Prometheus na yau da kullun ko Prometheus Operator.
- Prometheus na yau da kullun. Komai yana da kyau tare da shi, amma kuna buƙatar saita ConfigMap - a zahiri, rubuta fayilolin daidaitawa na rubutu, kamar yadda muka yi a baya, kafin ƙirar microservice.
- Prometheus Operator ya ɗan ɗanɗana, ɗan ƙaramin rikitarwa dangane da dabaru na ciki, amma yana da sauƙin aiki tare da shi: akwai abubuwa daban-daban, an ƙara abstractions zuwa gungu, don haka sun fi dacewa don sarrafawa da daidaitawa.
Don fahimtar samfurin, Ina ba da shawarar shigar da Prometheus na yau da kullun da farko. Dole ne ku saita komai ta hanyar saitin, amma wannan zai yi amfani: zaku gano abin da ke cikin menene kuma yadda aka saita shi. A cikin Prometheus Operator, nan da nan za ku tashi zuwa mafi girman abstraction, kodayake kuna iya shiga cikin zurfin idan kuna so.
Prometheus yana da haɗin gwiwa tare da Kubernetes: yana iya samun dama da yin hulɗa tare da API Server.
Prometheus ya shahara, wanda shine dalilin da ya sa ɗimbin aikace-aikace da harsunan shirye-shirye ke goyan bayan shi. Ana buƙatar tallafi, tun da Prometheus yana da nasa tsarin awo, kuma don canja wurin shi, kuna buƙatar ko dai ɗakin karatu a cikin aikace-aikacen ko mai fitarwa da aka shirya. Kuma akwai ƴan irin waɗannan masu fitar da kayayyaki. Misali, akwai PostgreSQL Exporter: yana ɗaukar bayanai daga PostgreSQL kuma ya canza shi zuwa tsarin Prometheus don Prometheus zai iya aiki da shi.
Prometheus architecture
Prometheus Server shine ƙarshen baya, kwakwalwar Prometheus. Ana adana awo kuma ana sarrafa su anan.
Ana adana ma'auni a cikin jerin bayanan lokaci (TSDB). TSDB ba bayanai bane daban, amma fakiti ne a cikin yaren Go wanda ke cikin Prometheus. Kusan magana, komai yana cikin binary guda ɗaya.
Kar a adana bayanai a cikin TSDB na dogon lokaci
Kayan aikin Prometheus bai dace da adana awo na dogon lokaci ba. Matsakaicin lokacin riƙewa shine kwanaki 15. Kuna iya wuce wannan iyaka, amma ku tuna: yawan bayanan da kuke adanawa a cikin TSDB kuma tsawon lokacin da kuke yi, yawan albarkatun zai cinye. Ajiye bayanan tarihi a cikin Prometheus ana ɗaukar mummunan aiki.
Idan kuna da babban zirga-zirga, adadin ma'auni shine ɗaruruwan dubunnan a sakan daya, to yana da kyau a iyakance ajiyar su ta sararin faifai ko ta lokaci. Yawancin lokaci, "bayanan zafi" ana adana su a cikin TSDB, ma'auni a cikin 'yan sa'o'i kadan. Don adana tsawon lokaci, ana amfani da ma'ajiyar waje a cikin waɗancan bayanan da suka dace da wannan, misali, InfluxDB, ClickHouse, da sauransu. Na ga ƙarin kyawawan sake dubawa game da ClickHouse.
Prometheus Server yana aiki akan ƙirar jawo: ya je nemo ma'auni zuwa ga wuraren da muka ba shi. Suka ce: "Je zuwa API Server", kuma yana zuwa wurin kowane adadin n-th na daƙiƙa kuma yana ɗaukar awo.
Don abubuwan da ke da ɗan gajeren rayuwa (aiki ko aikin cron) waɗanda zasu iya bayyana tsakanin lokutan gogewa, akwai bangaren Pushgateway. Ana tura ma'auni daga abubuwa na ɗan gajeren lokaci zuwa cikinsa: aikin ya tashi, yayi wani aiki, aika awo zuwa Pushgateway kuma an gama. Bayan ɗan lokaci, Prometheus zai sauko a kan kansa kuma ya ɗauki waɗannan ma'auni daga Pushgateway.
Don saita sanarwa a cikin Prometheus akwai wani sashi daban - Mai sarrafa faɗakarwa. Da kuma ka'idojin faɗakarwa. Misali, kuna buƙatar ƙirƙirar faɗakarwa idan API ɗin uwar garken shine 0. Lokacin da abin ya faru, ana aika faɗakarwa ga mai sarrafa faɗakarwa don ƙarin aikawa. Manajan faɗakarwa yana da saitunan daidaitawa masu sassauƙa: ana iya aika rukuni ɗaya na faɗakarwa zuwa tattaunawar telegram na admins, wani zuwa tattaunawar masu haɓakawa, da na uku zuwa tattaunawar ma'aikatan kayan more rayuwa. Ana iya aika sanarwar zuwa Slack, Telegram, imel, da sauran tashoshi.
Kuma a ƙarshe, zan gaya muku game da fasalin Killer Prometheus - ganowa. Lokacin aiki tare da Prometheus, ba kwa buƙatar ƙayyade takamaiman adireshi na abubuwa don saka idanu, ya isa ya saita nau'in su. Wato, ba kwa buƙatar rubuta "a nan ne adireshin IP, ga tashar jiragen ruwa - duba", maimakon haka, kuna buƙatar ƙayyade ta waɗanne ka'idoji don nemo waɗannan abubuwan (hari - raga). Prometheus kanta, dangane da waɗanne abubuwa ne ke aiki a halin yanzu, yana jan abubuwan da suka dace kuma yana ƙara su zuwa saka idanu.
Wannan hanya ta dace da tsarin Kubernetes, inda duk abin da ke iyo: a yau akwai sabobin 10, gobe 3. Don kada a ƙayyade adireshin IP na uwar garken kowane lokaci, sun rubuta sau ɗaya yadda za a gano shi - kuma Discovering zai yi shi. .
Ana kiran yaren Prometheus PromQL. Yin amfani da wannan harshe, zaku iya samun ƙimar takamaiman ma'auni sannan ku canza su, gina ƙididdiga na nazari akan su.
https://prometheus.io/docs/prometheus/latest/querying/basics/
Простой запрос
container_memory_usage_bytes
Математические операции
container_memory_usage_bytes / 1024 / 1024
Встроенные функции
sum(container_memory_usage_bytes) / 1024 / 1024
Уточнение запроса
100 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m]) * 100)Shafin yanar gizo na Prometheus
Prometheus yana da nasa, ƙaƙƙarfan ƙa'idodin gidan yanar gizo. Ya dace kawai don gyara kuskure ko nuni.
A cikin layin Magana, zaku iya rubuta tambaya a cikin yaren PromQL.
Shafin Faɗakarwa ya ƙunshi ƙa'idodin faɗakarwa, kuma suna da matsayi uku:
- rashin aiki - idan faɗakarwar ba ta aiki a halin yanzu, wato, duk abin yana da kyau tare da shi, kuma bai yi aiki ba;
- jiran - wannan shine idan faɗakarwar ta yi aiki, amma aika bai riga ya wuce ba. An saita jinkiri don ramawa don kyaftawar hanyar sadarwa: idan ƙayyadadden sabis ɗin ya tashi a cikin minti ɗaya, to bai kamata a ƙara ƙararrawa ba tukuna;
- harbe-harbe shine matsayi na uku lokacin da faɗakarwar ta haskaka da aika saƙonni.
A cikin Menu na Matsayi za ku sami damar yin amfani da bayanai game da abin da Prometheus yake. Har ila yau, akwai sauyi zuwa maƙasudai (manufa), wanda muka yi magana a sama.
Don ƙarin bayyani na Prometheus interface, duba .
Haɗuwa da Grafana
A cikin haɗin yanar gizo na Prometheus, ba za ku sami hotuna masu kyau da fahimta ba waɗanda za ku iya zana ƙarshe game da yanayin tari. Don gina su, an haɗa Prometheus tare da Grafana. Muna samun irin waɗannan dashboards.
Kafa haɗin kai na Prometheus da Grafana ba shi da wahala ko kaɗan, zaka iya samun umarni a cikin takardun: To, zan ƙare da wannan.
A cikin kasidu masu zuwa, za mu ci gaba da batun sa ido: za mu yi magana game da tattarawa da kuma nazarin rajistan ayyukan ta amfani da Grafana Loki da madadin kayan aikin.
Mawallafi: Marcel Ibraev, ƙwararren mai kula da Kubernetes, injiniya mai aiki a cikin kamfanin , lasifika da kuma darasi developer Slurm.
source: www.habr.com