ProHoster > Blog > Administrasyon an > Prometheus: HTTP siveyans atravè ekspòtatè Blackbox

Prometheus: HTTP siveyans atravè ekspòtatè Blackbox

Bonjou tout moun. Nan mwa me OTUS lanse atelye sou siveyans ak anrejistreman, tou de enfrastrikti ak aplikasyon lè l sèvi avèk Zabbix, Prometheus, Grafana ak ELK. Nan sans sa a, nou tradisyonèlman pataje materyèl itil sou sijè a.

Ekspòtatè Blackbox pou Prometheus pèmèt ou aplike siveyans sèvis ekstèn atravè HTTP, HTTPS, DNS, TCP, ICMP. Nan atik sa a, mwen pral montre w kouman yo mete kanpe HTTP/HTTPS siveyans lè l sèvi avèk Blackbox ekspòtatè. Nou pral lanse ekspòtatè a Blackbox nan Kubernetes.

Environ

Nou pral bezwen bagay sa yo:

  • Kubernetes
  • Operatè Prometheus

Konfigirasyon Blackbox ekspòtatè

Konfigirasyon Blackbox atravè ConfigMap pou paramèt http modil siveyans sèvis entènèt.

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-blackbox-exporter
  labels:
    app: prometheus-blackbox-exporter
data:
  blackbox.yaml: |
    modules:
      http_2xx:
        http:
          no_follow_redirects: false
          preferred_ip_protocol: ip4
          valid_http_versions:
          - HTTP/1.1
          - HTTP/2
          valid_status_codes: []
        prober: http
        timeout: 5s

Modil http_2xx yo itilize pou tcheke si sèvis entènèt la retounen yon kòd estati HTTP 2xx. Konfigirasyon ekspòtatè blackbox la dekri an plis detay nan dokiman.

Deplwaye yon ekspòtatè blackbox nan yon gwoup Kubernetes

Dekri Deployment и Service pou deplwaman nan Kubernetes.

---
kind: Service
apiVersion: v1
metadata:
  name: prometheus-blackbox-exporter
  labels:
    app: prometheus-blackbox-exporter
spec:
  type: ClusterIP
  ports:
    - name: http
      port: 9115
      protocol: TCP
  selector:
    app: prometheus-blackbox-exporter

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus-blackbox-exporter
  labels:
    app: prometheus-blackbox-exporter
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus-blackbox-exporter
  template:
    metadata:
      labels:
        app: prometheus-blackbox-exporter
    spec:
      restartPolicy: Always
      containers:
        - name: blackbox-exporter
          image: "prom/blackbox-exporter:v0.15.1"
          imagePullPolicy: IfNotPresent
          securityContext:
            readOnlyRootFilesystem: true
            runAsNonRoot: true
            runAsUser: 1000
          args:
            - "--config.file=/config/blackbox.yaml"
          resources:
            {}
          ports:
            - containerPort: 9115
              name: http
          livenessProbe:
            httpGet:
              path: /health
              port: http
          readinessProbe:
            httpGet:
              path: /health
              port: http
          volumeMounts:
            - mountPath: /config
              name: config
        - name: configmap-reload
          image: "jimmidyson/configmap-reload:v0.2.2"
          imagePullPolicy: "IfNotPresent"
          securityContext:
            runAsNonRoot: true
            runAsUser: 65534
          args:
            - --volume-dir=/etc/config
            - --webhook-url=http://localhost:9115/-/reload
          resources:
            {}
          volumeMounts:
            - mountPath: /etc/config
              name: config
              readOnly: true
      volumes:
        - name: config
          configMap:
            name: prometheus-blackbox-exporter

Blackbox ekspòtatè ka deplwaye lè l sèvi avèk lòd sa a. Espas non monitoring refere a Operatè Prometheus.

kubectl --namespace=monitoring apply -f blackbox-exporter.yaml

Asire w ke tout sèvis yo ap kouri lè l sèvi avèk lòd sa a:

kubectl --namespace=monitoring get all --selector=app=prometheus-blackbox-exporter

Tcheke Blackbox

Ou ka jwenn aksè nan koòdone entènèt ekspòtatè Blackbox lè l sèvi avèk port-forward:

kubectl --namespace=monitoring port-forward svc/prometheus-blackbox-exporter 9115:9115

Konekte nan koòdone entènèt Blackbox ekspòtatè atravè yon navigatè entènèt nan lokalost: 9115.

Prometheus: HTTP siveyans atravè ekspòtatè Blackbox

Si ou ale nan adrès la http://localhost:9115/probe?module=http_2xx&target=https://www.google.com, ou pral wè rezilta tcheke URL espesifye a (https://www.google.com).

Prometheus: HTTP siveyans atravè ekspòtatè Blackbox

Valè metrik probe_success egal a 1 vle di chèk siksè. Yon valè 0 endike yon erè.

Mete kanpe Prometheus

Apre deplwaye ekspòtatè BlackBox la, nou konfigirasyon Prometheus nan prometheus-additional.yaml.

- job_name: 'kube-api-blackbox'
  scrape_interval: 1w
  metrics_path: /probe
  params:
    module: [http_2xx]
  static_configs:
   - targets:
      - https://www.google.com
      - http://www.example.com
      - https://prometheus.io
  relabel_configs:
   - source_labels: [__address__]
     target_label: __param_target
   - source_labels: [__param_target]
     target_label: instance
   - target_label: __address__
     replacement: prometheus-blackbox-exporter:9115 # The blackbox exporter.

Nou jenere Secretlè l sèvi avèk lòd sa a.

PROMETHEUS_ADD_CONFIG=$(cat prometheus-additional.yaml | base64)
cat << EOF | kubectl --namespace=monitoring apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: additional-scrape-configs
type: Opaque
data:
  prometheus-additional.yaml: $PROMETHEUS_ADD_CONFIG
EOF

Nou endike additional-scrape-configs pou Prometheus Operator itilize additionalScrapeConfigs.

kubectl --namespace=monitoring edit prometheuses k8s
...
spec:
  additionalScrapeConfigs:
    key: prometheus-additional.yaml
    name: additional-scrape-configs

Nou ale nan koòdone entènèt Prometheus la epi tcheke mezi ak objektif yo.

kubectl --namespace=monitoring port-forward svc/prometheus-k8s 9090:9090

Prometheus: HTTP siveyans atravè ekspòtatè Blackbox

Prometheus: HTTP siveyans atravè ekspòtatè Blackbox

Nou wè mezi ak objektif Blackbox.

Ajoute règ pou notifikasyon (alèt)

Pou resevwa notifikasyon nan men ekspòtatè Blackbox, nou pral ajoute règ nan Operatè Prometheus.

kubectl --namespace=monitoring edit prometheusrules prometheus-k8s-rules
...
  - name: blackbox-exporter
    rules:
    - alert: ProbeFailed
      expr: probe_success == 0
      for: 5m
      labels:
        severity: error
      annotations:
        summary: "Probe failed (instance {{ $labels.instance }})"
        description: "Probe failedn  VALUE = {{ $value }}n  LABELS: {{ $labels }}"
    - alert: SlowProbe
      expr: avg_over_time(probe_duration_seconds[1m]) > 1
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "Slow probe (instance {{ $labels.instance }})"
        description: "Blackbox probe took more than 1s to completen  VALUE = {{ $value }}n  LABELS: {{ $labels }}"
    - alert: HttpStatusCode
      expr: probe_http_status_code <= 199 OR probe_http_status_code >= 400
      for: 5m
      labels:
        severity: error
      annotations:
        summary: "HTTP Status Code (instance {{ $labels.instance }})"
        description: "HTTP status code is not 200-399n  VALUE = {{ $value }}n  LABELS: {{ $labels }}"
    - alert: SslCertificateWillExpireSoon
      expr: probe_ssl_earliest_cert_expiry - time() < 86400 * 30
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "SSL certificate will expire soon (instance {{ $labels.instance }})"
        description: "SSL certificate expires in 30 daysn  VALUE = {{ $value }}n  LABELS: {{ $labels }}"
    - alert: SslCertificateHasExpired
      expr: probe_ssl_earliest_cert_expiry - time()  <= 0
      for: 5m
      labels:
        severity: error
      annotations:
        summary: "SSL certificate has expired (instance {{ $labels.instance }})"
        description: "SSL certificate has expired alreadyn  VALUE = {{ $value }}n  LABELS: {{ $labels }}"
    - alert: HttpSlowRequests
      expr: avg_over_time(probe_http_duration_seconds[1m]) > 1
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "HTTP slow requests (instance {{ $labels.instance }})"
        description: "HTTP request took more than 1sn  VALUE = {{ $value }}n  LABELS: {{ $labels }}"
    - alert: SlowPing
      expr: avg_over_time(probe_icmp_duration_seconds[1m]) > 1
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "Slow ping (instance {{ $labels.instance }})"
        description: "Blackbox ping took more than 1sn  VALUE = {{ $value }}n  LABELS: {{ $labels }}"

Nan koòdone entènèt Prometheus, ale nan Status => Règ epi jwenn règ alèt yo pou blackbox-exporter.

Prometheus: HTTP siveyans atravè ekspòtatè Blackbox

Konfigirasyon Notifikasyon ekspirasyon Sètifika SSL sèvè Kubernetes API

Ann konfigure siveyans ekspirasyon sètifika SSL sèvè Kubernetes API. Li pral voye notifikasyon yon fwa pa semèn.

Ajoute modil ekspòtatè Blackbox pou Otantifikasyon sèvè API Kubernetes.

kubectl --namespace=monitoring edit configmap prometheus-blackbox-exporter
...
      kube-api:
        http:
          method: GET
          no_follow_redirects: false
          preferred_ip_protocol: ip4
          tls_config:
            insecure_skip_verify: false
            ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
          valid_http_versions:
          - HTTP/1.1
          - HTTP/2
          valid_status_codes: []
        prober: http
        timeout: 5s

Ajoute konfigirasyon Prometheus grate

- job_name: 'kube-api-blackbox'
  metrics_path: /probe
  params:
    module: [kube-api]
  static_configs:
   - targets:
      - https://kubernetes.default.svc/api
  relabel_configs:
   - source_labels: [__address__]
     target_label: __param_target
   - source_labels: [__param_target]
     target_label: instance
   - target_label: __address__
     replacement: prometheus-blackbox-exporter:9115 # The blackbox exporter.

Sèvi ak Prometheus sekrè

PROMETHEUS_ADD_CONFIG=$(cat prometheus-additional.yaml | base64)
cat << EOF | kubectl --namespace=monitoring apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: additional-scrape-configs
type: Opaque
data:
  prometheus-additional.yaml: $PROMETHEUS_ADD_CONFIG
EOF

Ajoute règ alèt

kubectl --namespace=monitoring edit prometheusrules prometheus-k8s-rules
...
  - name: k8s-api-server-cert-expiry
    rules:
    - alert: K8sAPIServerSSLCertExpiringAfterThreeMonths
      expr: probe_ssl_earliest_cert_expiry{job="kube-api-blackbox"} - time() < 86400 * 90 
      for: 1w
      labels:
        severity: warning
      annotations:
        summary: "Kubernetes API Server SSL certificate will expire after three months (instance {{ $labels.instance }})"
        description: "Kubernetes API Server SSL certificate expires in 90 daysn  VALUE = {{ $value }}n  LABELS: {{ $labels }}"

lyen itil

Siveyans ak antre nan Docker

Sous: www.habr.com