ProHoster > Блог > Gudanarwa > Prometheus: Sa ido na HTTP ta hanyar mai fitar da Blackbox

Prometheus: Sa ido na HTTP ta hanyar mai fitar da Blackbox

Assalamu alaikum. A watan Mayu OTUS ya ƙaddamar taron bita akan sa ido da shiga, duka kayan aiki da aikace-aikace ta amfani da Zabbix, Prometheus, Grafana da ELK. Dangane da wannan, a al'ada muna raba abubuwa masu amfani akan batun.

Blackbox mai fitarwa don Prometheus yana ba ku damar aiwatar da saka idanu na ayyukan waje ta hanyar HTTP, HTTPS, DNS, TCP, ICMP. A cikin wannan labarin, zan nuna muku yadda ake saita HTTP/HTTPS saka idanu ta amfani da Blackbox Exporter. Za mu ƙaddamar da mai fitar da Blackbox a Kubernetes.

Muhalli

Za mu buƙaci abubuwa masu zuwa:

  • Kubernetes
  • Prometheus Operator

Tsarin akwatin akwatin mai fitarwa

Ana saita Blackbox ta hanyar ConfigMap don saituna http tsarin kula da ayyukan yanar gizo.

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-blackbox-exporter
  labels:
    app: prometheus-blackbox-exporter
data:
  blackbox.yaml: |
    modules:
      http_2xx:
        http:
          no_follow_redirects: false
          preferred_ip_protocol: ip4
          valid_http_versions:
          - HTTP/1.1
          - HTTP/2
          valid_status_codes: []
        prober: http
        timeout: 5s

Module http_2xx ana amfani dashi don bincika cewa sabis ɗin gidan yanar gizon yana dawo da lambar matsayi na HTTP 2xx. An yi bayanin daidaitawar mai fitar da akwatin blackbox daki-daki a ciki takardun.

Ana tura mai fitar da akwatin baki zuwa gungu na Kubernetes

Bayyana Deployment и Service don turawa a Kubernetes.

---
kind: Service
apiVersion: v1
metadata:
  name: prometheus-blackbox-exporter
  labels:
    app: prometheus-blackbox-exporter
spec:
  type: ClusterIP
  ports:
    - name: http
      port: 9115
      protocol: TCP
  selector:
    app: prometheus-blackbox-exporter

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus-blackbox-exporter
  labels:
    app: prometheus-blackbox-exporter
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus-blackbox-exporter
  template:
    metadata:
      labels:
        app: prometheus-blackbox-exporter
    spec:
      restartPolicy: Always
      containers:
        - name: blackbox-exporter
          image: "prom/blackbox-exporter:v0.15.1"
          imagePullPolicy: IfNotPresent
          securityContext:
            readOnlyRootFilesystem: true
            runAsNonRoot: true
            runAsUser: 1000
          args:
            - "--config.file=/config/blackbox.yaml"
          resources:
            {}
          ports:
            - containerPort: 9115
              name: http
          livenessProbe:
            httpGet:
              path: /health
              port: http
          readinessProbe:
            httpGet:
              path: /health
              port: http
          volumeMounts:
            - mountPath: /config
              name: config
        - name: configmap-reload
          image: "jimmidyson/configmap-reload:v0.2.2"
          imagePullPolicy: "IfNotPresent"
          securityContext:
            runAsNonRoot: true
            runAsUser: 65534
          args:
            - --volume-dir=/etc/config
            - --webhook-url=http://localhost:9115/-/reload
          resources:
            {}
          volumeMounts:
            - mountPath: /etc/config
              name: config
              readOnly: true
      volumes:
        - name: config
          configMap:
            name: prometheus-blackbox-exporter

Ana iya tura mai fitar da Blackbox ta amfani da umarni mai zuwa. Wurin suna monitoring yana nufin Prometheus Operator.

kubectl --namespace=monitoring apply -f blackbox-exporter.yaml

Tabbatar cewa duk ayyuka suna gudana ta amfani da umarni mai zuwa:

kubectl --namespace=monitoring get all --selector=app=prometheus-blackbox-exporter

Blackbox rajistan shiga

Kuna iya samun damar shiga yanar gizo na Blackbox mai fitarwa ta amfani da shi port-forward:

kubectl --namespace=monitoring port-forward svc/prometheus-blackbox-exporter 9115:9115

Haɗa zuwa cibiyar yanar gizon mai fitar da Blackbox ta hanyar mai binciken gidan yanar gizo a Localhost: 9115.

Prometheus: Sa ido na HTTP ta hanyar mai fitar da Blackbox

Idan kun je adireshin http://localhost:9115/probe?module=http_2xx&target=https://www.google.com, za ku ga sakamakon duba URL ɗin da aka ƙayyade (https://www.google.com).

Prometheus: Sa ido na HTTP ta hanyar mai fitar da Blackbox

Ƙimar awo probe_success daidai da 1 yana nufin dubawa mai nasara. Ƙimar 0 tana nuna kuskure.

Saita Prometheus

Bayan tura mai fitar da BlackBox, mun saita Prometheus a ciki prometheus-additional.yaml.

- job_name: 'kube-api-blackbox'
  scrape_interval: 1w
  metrics_path: /probe
  params:
    module: [http_2xx]
  static_configs:
   - targets:
      - https://www.google.com
      - http://www.example.com
      - https://prometheus.io
  relabel_configs:
   - source_labels: [__address__]
     target_label: __param_target
   - source_labels: [__param_target]
     target_label: instance
   - target_label: __address__
     replacement: prometheus-blackbox-exporter:9115 # The blackbox exporter.

Muna samarwa Secretta amfani da umarni mai zuwa.

PROMETHEUS_ADD_CONFIG=$(cat prometheus-additional.yaml | base64)
cat << EOF | kubectl --namespace=monitoring apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: additional-scrape-configs
type: Opaque
data:
  prometheus-additional.yaml: $PROMETHEUS_ADD_CONFIG
EOF

Mun nuna additional-scrape-configs don Prometheus Operator amfani additionalScrapeConfigs.

kubectl --namespace=monitoring edit prometheuses k8s
...
spec:
  additionalScrapeConfigs:
    key: prometheus-additional.yaml
    name: additional-scrape-configs

Mun je zuwa shafin yanar gizon Prometheus kuma mu duba ma'auni da raga.

kubectl --namespace=monitoring port-forward svc/prometheus-k8s 9090:9090

Prometheus: Sa ido na HTTP ta hanyar mai fitar da Blackbox

Prometheus: Sa ido na HTTP ta hanyar mai fitar da Blackbox

Muna ganin ma'auni da burin Blackbox.

Ƙara ƙa'idodi don sanarwa (jijjiga)

Don karɓar sanarwa daga mai fitar da Blackbox, za mu ƙara dokoki zuwa Prometheus Operator.

kubectl --namespace=monitoring edit prometheusrules prometheus-k8s-rules
...
  - name: blackbox-exporter
    rules:
    - alert: ProbeFailed
      expr: probe_success == 0
      for: 5m
      labels:
        severity: error
      annotations:
        summary: "Probe failed (instance {{ $labels.instance }})"
        description: "Probe failedn  VALUE = {{ $value }}n  LABELS: {{ $labels }}"
    - alert: SlowProbe
      expr: avg_over_time(probe_duration_seconds[1m]) > 1
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "Slow probe (instance {{ $labels.instance }})"
        description: "Blackbox probe took more than 1s to completen  VALUE = {{ $value }}n  LABELS: {{ $labels }}"
    - alert: HttpStatusCode
      expr: probe_http_status_code <= 199 OR probe_http_status_code >= 400
      for: 5m
      labels:
        severity: error
      annotations:
        summary: "HTTP Status Code (instance {{ $labels.instance }})"
        description: "HTTP status code is not 200-399n  VALUE = {{ $value }}n  LABELS: {{ $labels }}"
    - alert: SslCertificateWillExpireSoon
      expr: probe_ssl_earliest_cert_expiry - time() < 86400 * 30
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "SSL certificate will expire soon (instance {{ $labels.instance }})"
        description: "SSL certificate expires in 30 daysn  VALUE = {{ $value }}n  LABELS: {{ $labels }}"
    - alert: SslCertificateHasExpired
      expr: probe_ssl_earliest_cert_expiry - time()  <= 0
      for: 5m
      labels:
        severity: error
      annotations:
        summary: "SSL certificate has expired (instance {{ $labels.instance }})"
        description: "SSL certificate has expired alreadyn  VALUE = {{ $value }}n  LABELS: {{ $labels }}"
    - alert: HttpSlowRequests
      expr: avg_over_time(probe_http_duration_seconds[1m]) > 1
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "HTTP slow requests (instance {{ $labels.instance }})"
        description: "HTTP request took more than 1sn  VALUE = {{ $value }}n  LABELS: {{ $labels }}"
    - alert: SlowPing
      expr: avg_over_time(probe_icmp_duration_seconds[1m]) > 1
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "Slow ping (instance {{ $labels.instance }})"
        description: "Blackbox ping took more than 1sn  VALUE = {{ $value }}n  LABELS: {{ $labels }}"

A cikin haɗin yanar gizon Prometheus, je zuwa Matsayi => Dokoki kuma nemo ƙa'idodin faɗakarwa don blackbox-exporter.

Prometheus: Sa ido na HTTP ta hanyar mai fitar da Blackbox

Yana Haɓaka Kubernetes API Sabar Sabar SSL Fadakarwar Karewa Takaddun shaida

Bari mu daidaita Kubernetes API Sabar Sabar SSL takardar shedar karewa. Zai aika sanarwa sau ɗaya a mako.

Ƙara tsarin mai fitar da Blackbox don Tabbatar da Sabar API na Kubernetes.

kubectl --namespace=monitoring edit configmap prometheus-blackbox-exporter
...
      kube-api:
        http:
          method: GET
          no_follow_redirects: false
          preferred_ip_protocol: ip4
          tls_config:
            insecure_skip_verify: false
            ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
          valid_http_versions:
          - HTTP/1.1
          - HTTP/2
          valid_status_codes: []
        prober: http
        timeout: 5s

Ƙara Tsarin Scrape Prometheus

- job_name: 'kube-api-blackbox'
  metrics_path: /probe
  params:
    module: [kube-api]
  static_configs:
   - targets:
      - https://kubernetes.default.svc/api
  relabel_configs:
   - source_labels: [__address__]
     target_label: __param_target
   - source_labels: [__param_target]
     target_label: instance
   - target_label: __address__
     replacement: prometheus-blackbox-exporter:9115 # The blackbox exporter.

Amfani da Asirin Prometheus

PROMETHEUS_ADD_CONFIG=$(cat prometheus-additional.yaml | base64)
cat << EOF | kubectl --namespace=monitoring apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: additional-scrape-configs
type: Opaque
data:
  prometheus-additional.yaml: $PROMETHEUS_ADD_CONFIG
EOF

Ƙara dokokin faɗakarwa

kubectl --namespace=monitoring edit prometheusrules prometheus-k8s-rules
...
  - name: k8s-api-server-cert-expiry
    rules:
    - alert: K8sAPIServerSSLCertExpiringAfterThreeMonths
      expr: probe_ssl_earliest_cert_expiry{job="kube-api-blackbox"} - time() < 86400 * 90 
      for: 1w
      labels:
        severity: warning
      annotations:
        summary: "Kubernetes API Server SSL certificate will expire after three months (instance {{ $labels.instance }})"
        description: "Kubernetes API Server SSL certificate expires in 90 daysn  VALUE = {{ $value }}n  LABELS: {{ $labels }}"

hanyoyi masu amfani

Saka idanu da shiga cikin Docker

source: www.habr.com