🥇Prometey: Blackbox eksportchisi orqali HTTP monitoringi

Hammaga salom. May oyida OTUS ishga tushadi monitoring va ro'yxatga olish bo'yicha seminar, Zabbix, Prometey, Grafana va ELK-dan foydalanadigan infratuzilma va ilovalar. Shu munosabat bilan biz an'anaviy tarzda mavzu bo'yicha foydali materiallarni baham ko'ramiz.

Blackbox eksportchisi Prometey uchun HTTP, HTTPS, DNS, TCP, ICMP orqali tashqi xizmatlar monitoringini amalga oshirish imkonini beradi. Ushbu maqolada men Blackbox eksportchisi yordamida HTTP/HTTPS monitoringini qanday o'rnatishni ko'rsataman. Kubernetesda Blackbox eksportchisini ishga tushiramiz.

Atrof-muhit

Bizga quyidagilar kerak bo'ladi:

Kubernetes
Prometey operatori

Eksport qiluvchi qora quti konfiguratsiyasi

Blackbox orqali sozlash ConfigMap sozlash uchun http veb-xizmatlar monitoringi moduli.

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-blackbox-exporter
  labels:
    app: prometheus-blackbox-exporter
data:
  blackbox.yaml: |
    modules:
      http_2xx:
        http:
          no_follow_redirects: false
          preferred_ip_protocol: ip4
          valid_http_versions:
          - HTTP/1.1
          - HTTP/2
          valid_status_codes: []
        prober: http
        timeout: 5s

Modul http_2xx veb-xizmati HTTP 2xx holat kodini qaytarishini tekshirish uchun ishlatiladi. Qora quti eksportchisi konfiguratsiyasi batafsilroq tavsiflangan hujjatlar.

Qora quti eksportchisini Kubernetes klasteriga joylashtirish

Ta'riflang Deployment и Service Kubernetesda joylashtirish uchun.

---
kind: Service
apiVersion: v1
metadata:
  name: prometheus-blackbox-exporter
  labels:
    app: prometheus-blackbox-exporter
spec:
  type: ClusterIP
  ports:
    - name: http
      port: 9115
      protocol: TCP
  selector:
    app: prometheus-blackbox-exporter

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus-blackbox-exporter
  labels:
    app: prometheus-blackbox-exporter
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus-blackbox-exporter
  template:
    metadata:
      labels:
        app: prometheus-blackbox-exporter
    spec:
      restartPolicy: Always
      containers:
        - name: blackbox-exporter
          image: "prom/blackbox-exporter:v0.15.1"
          imagePullPolicy: IfNotPresent
          securityContext:
            readOnlyRootFilesystem: true
            runAsNonRoot: true
            runAsUser: 1000
          args:
            - "--config.file=/config/blackbox.yaml"
          resources:
            {}
          ports:
            - containerPort: 9115
              name: http
          livenessProbe:
            httpGet:
              path: /health
              port: http
          readinessProbe:
            httpGet:
              path: /health
              port: http
          volumeMounts:
            - mountPath: /config
              name: config
        - name: configmap-reload
          image: "jimmidyson/configmap-reload:v0.2.2"
          imagePullPolicy: "IfNotPresent"
          securityContext:
            runAsNonRoot: true
            runAsUser: 65534
          args:
            - --volume-dir=/etc/config
            - --webhook-url=http://localhost:9115/-/reload
          resources:
            {}
          volumeMounts:
            - mountPath: /etc/config
              name: config
              readOnly: true
      volumes:
        - name: config
          configMap:
            name: prometheus-blackbox-exporter

Blackbox eksportchisi quyidagi buyruq yordamida joylashtirilishi mumkin. Ismlar maydoni monitoring Prometey operatoriga ishora qiladi.

kubectl --namespace=monitoring apply -f blackbox-exporter.yaml

Quyidagi buyruq yordamida barcha xizmatlar ishlayotganligiga ishonch hosil qiling:

kubectl --namespace=monitoring get all --selector=app=prometheus-blackbox-exporter

Qora qutini tekshirish

Siz Blackbox eksportchisi veb-interfeysiga kirishingiz mumkin port-forward:

kubectl --namespace=monitoring port-forward svc/prometheus-blackbox-exporter 9115:9115

Veb-brauzer orqali Blackbox eksportchisi veb-interfeysiga ulaning localhost: 9115.

Manzilga borsangiz http://localhost:9115/probe?module=http_2xx&target=https://www.google.com, siz belgilangan URLni tekshirish natijasini ko'rasiz (https://www.google.com).

Metrik qiymat probe_success 1 ga teng bo'lsa, muvaffaqiyatli tekshirish. 0 qiymati xatoni bildiradi.

Prometeyni sozlash

BlackBox eksportchisini joylashtirgandan so'ng, biz Prometeyni sozlaymiz prometheus-additional.yaml.

- job_name: 'kube-api-blackbox'
  scrape_interval: 1w
  metrics_path: /probe
  params:
    module: [http_2xx]
  static_configs:
   - targets:
      - https://www.google.com
      - http://www.example.com
      - https://prometheus.io
  relabel_configs:
   - source_labels: [__address__]
     target_label: __param_target
   - source_labels: [__param_target]
     target_label: instance
   - target_label: __address__
     replacement: prometheus-blackbox-exporter:9115 # The blackbox exporter.

Biz hosil qilamiz Secretquyidagi buyruq yordamida.

PROMETHEUS_ADD_CONFIG=$(cat prometheus-additional.yaml | base64)
cat << EOF | kubectl --namespace=monitoring apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: additional-scrape-configs
type: Opaque
data:
  prometheus-additional.yaml: $PROMETHEUS_ADD_CONFIG
EOF

Belgilash additional-scrape-configs Prometey operatoridan foydalanish uchun additionalScrapeConfigs.

kubectl --namespace=monitoring edit prometheuses k8s
...
spec:
  additionalScrapeConfigs:
    key: prometheus-additional.yaml
    name: additional-scrape-configs

Biz Prometey veb-interfeysiga o'tamiz va ko'rsatkichlar va maqsadlarni tekshiramiz.

kubectl --namespace=monitoring port-forward svc/prometheus-k8s 9090:9090

Biz Blackbox ko'rsatkichlari va maqsadlarini ko'ramiz.

Bildirishnomalar uchun qoidalar qo'shish (ogohlantirish)

Blackbox eksportchisidan bildirishnomalarni olish uchun biz Prometey Operatoriga qoidalar qo'shamiz.

kubectl --namespace=monitoring edit prometheusrules prometheus-k8s-rules
...
  - name: blackbox-exporter
    rules:
    - alert: ProbeFailed
      expr: probe_success == 0
      for: 5m
      labels:
        severity: error
      annotations:
        summary: "Probe failed (instance {{ $labels.instance }})"
        description: "Probe failedn  VALUE = {{ $value }}n  LABELS: {{ $labels }}"
    - alert: SlowProbe
      expr: avg_over_time(probe_duration_seconds[1m]) > 1
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "Slow probe (instance {{ $labels.instance }})"
        description: "Blackbox probe took more than 1s to completen  VALUE = {{ $value }}n  LABELS: {{ $labels }}"
    - alert: HttpStatusCode
      expr: probe_http_status_code <= 199 OR probe_http_status_code >= 400
      for: 5m
      labels:
        severity: error
      annotations:
        summary: "HTTP Status Code (instance {{ $labels.instance }})"
        description: "HTTP status code is not 200-399n  VALUE = {{ $value }}n  LABELS: {{ $labels }}"
    - alert: SslCertificateWillExpireSoon
      expr: probe_ssl_earliest_cert_expiry - time() < 86400 * 30
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "SSL certificate will expire soon (instance {{ $labels.instance }})"
        description: "SSL certificate expires in 30 daysn  VALUE = {{ $value }}n  LABELS: {{ $labels }}"
    - alert: SslCertificateHasExpired
      expr: probe_ssl_earliest_cert_expiry - time()  <= 0
      for: 5m
      labels:
        severity: error
      annotations:
        summary: "SSL certificate has expired (instance {{ $labels.instance }})"
        description: "SSL certificate has expired alreadyn  VALUE = {{ $value }}n  LABELS: {{ $labels }}"
    - alert: HttpSlowRequests
      expr: avg_over_time(probe_http_duration_seconds[1m]) > 1
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "HTTP slow requests (instance {{ $labels.instance }})"
        description: "HTTP request took more than 1sn  VALUE = {{ $value }}n  LABELS: {{ $labels }}"
    - alert: SlowPing
      expr: avg_over_time(probe_icmp_duration_seconds[1m]) > 1
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "Slow ping (instance {{ $labels.instance }})"
        description: "Blackbox ping took more than 1sn  VALUE = {{ $value }}n  LABELS: {{ $labels }}"

Prometey veb-interfeysida Status => Rules-ga o'ting va blackbox-exporter uchun ogohlantirish qoidalarini toping.

Kubernetes API Server SSL sertifikatining amal qilish muddati tugashi haqida bildirishnomalarni sozlash

Kubernetes API Server SSL sertifikatining amal qilish muddati monitoringini sozlaylik. U haftada bir marta bildirishnomalarni yuboradi.

Kubernetes API Server autentifikatsiyasi uchun Blackbox eksportyor modulini qo'shish.

kubectl --namespace=monitoring edit configmap prometheus-blackbox-exporter
...
      kube-api:
        http:
          method: GET
          no_follow_redirects: false
          preferred_ip_protocol: ip4
          tls_config:
            insecure_skip_verify: false
            ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
          valid_http_versions:
          - HTTP/1.1
          - HTTP/2
          valid_status_codes: []
        prober: http
        timeout: 5s

Prometey skrep konfiguratsiyasini qo'shish

- job_name: 'kube-api-blackbox'
  metrics_path: /probe
  params:
    module: [kube-api]
  static_configs:
   - targets:
      - https://kubernetes.default.svc/api
  relabel_configs:
   - source_labels: [__address__]
     target_label: __param_target
   - source_labels: [__param_target]
     target_label: instance
   - target_label: __address__
     replacement: prometheus-blackbox-exporter:9115 # The blackbox exporter.

Prometey siridan foydalanish

PROMETHEUS_ADD_CONFIG=$(cat prometheus-additional.yaml | base64)
cat << EOF | kubectl --namespace=monitoring apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: additional-scrape-configs
type: Opaque
data:
  prometheus-additional.yaml: $PROMETHEUS_ADD_CONFIG
EOF

Ogohlantirish qoidalarini qo'shish

kubectl --namespace=monitoring edit prometheusrules prometheus-k8s-rules
...
  - name: k8s-api-server-cert-expiry
    rules:
    - alert: K8sAPIServerSSLCertExpiringAfterThreeMonths
      expr: probe_ssl_earliest_cert_expiry{job="kube-api-blackbox"} - time() < 86400 * 90 
      for: 1w
      labels:
        severity: warning
      annotations:
        summary: "Kubernetes API Server SSL certificate will expire after three months (instance {{ $labels.instance }})"
        description: "Kubernetes API Server SSL certificate expires in 90 daysn  VALUE = {{ $value }}n  LABELS: {{ $labels }}"

Foydali havolalar

Docker-da monitoring va tizimga kirish

Manba: www.habr.com