āĻšāĻžāĻ āĻ¸āĻŦ. āĻŽā§ āĻŽāĻžāĻ¸ā§ OTUS āĻāĻžāĻ˛ā§ āĻšāĻ¯āĻŧ
āĻĒā§āĻ°āĻ¤āĻŋāĻŦā§āĻļ
āĻāĻŽāĻžāĻĻā§āĻ° āĻ¨āĻŋāĻŽā§āĻ¨āĻ˛āĻŋāĻāĻŋāĻ¤ āĻĒā§āĻ°āĻ¯āĻŧā§āĻāĻ¨ āĻšāĻŦā§:
- Kubernetes
- āĻĒā§āĻ°āĻŽāĻŋāĻĨāĻŋāĻāĻ¸ āĻ āĻĒāĻžāĻ°ā§āĻāĻ°
āĻ°āĻĒā§āĻ¤āĻžāĻ¨āĻŋāĻāĻžāĻ°āĻ āĻŦā§āĻ˛ā§āĻ¯āĻžāĻāĻŦāĻā§āĻ¸ āĻāĻ¨āĻĢāĻŋāĻāĻžāĻ°ā§āĻļāĻ¨
āĻāĻ° āĻŽāĻžāĻ§ā§āĻ¯āĻŽā§ āĻŦā§āĻ˛ā§āĻ¯āĻžāĻāĻŦāĻā§āĻ¸ āĻāĻ¨āĻĢāĻŋāĻāĻžāĻ° āĻāĻ°āĻž āĻšāĻā§āĻā§ ConfigMap
āĻ¸ā§āĻāĻŋāĻāĻ¸ā§āĻ° āĻāĻ¨ā§āĻ¯ http
āĻāĻ¯āĻŧā§āĻŦ āĻ¸āĻžāĻ°ā§āĻāĻŋāĻ¸ āĻŽāĻ¨āĻŋāĻāĻ°āĻŋāĻ āĻŽāĻĄāĻŋāĻāĻ˛āĨ¤
apiVersion: v1
kind: ConfigMap
metadata:
name: prometheus-blackbox-exporter
labels:
app: prometheus-blackbox-exporter
data:
blackbox.yaml: |
modules:
http_2xx:
http:
no_follow_redirects: false
preferred_ip_protocol: ip4
valid_http_versions:
- HTTP/1.1
- HTTP/2
valid_status_codes: []
prober: http
timeout: 5s
āĻŽāĻĄāĻŋāĻāĻ˛ http_2xx
āĻāĻ¯āĻŧā§āĻŦ āĻĒāĻ°āĻŋāĻˇā§āĻŦāĻžāĻāĻŋ āĻāĻāĻāĻŋ HTTP 2xx āĻ¸ā§āĻā§āĻ¯āĻžāĻāĻžāĻ¸ āĻā§āĻĄ āĻĒā§āĻ°āĻĻāĻžāĻ¨ āĻāĻ°ā§ āĻāĻŋāĻ¨āĻž āĻ¤āĻž āĻĒāĻ°ā§āĻā§āĻˇāĻž āĻāĻ°āĻ¤ā§ āĻŦā§āĻ¯āĻŦāĻšā§āĻ¤ āĻšāĻ¯āĻŧāĨ¤ āĻŦā§āĻ˛ā§āĻ¯āĻžāĻāĻŦāĻā§āĻ¸ āĻ°āĻĒā§āĻ¤āĻžāĻ¨āĻŋāĻāĻžāĻ°āĻ āĻāĻ¨āĻĢāĻŋāĻāĻžāĻ°ā§āĻļāĻ¨ āĻāĻ°āĻ āĻŦāĻŋāĻļāĻĻā§ āĻŦāĻ°ā§āĻŖāĻ¨āĻž āĻāĻ°āĻž āĻšāĻ¯āĻŧā§āĻā§
āĻāĻāĻāĻŋ Kubernetes āĻā§āĻ˛āĻžāĻ¸ā§āĻāĻžāĻ°ā§ āĻāĻāĻāĻŋ āĻŦā§āĻ˛ā§āĻ¯āĻžāĻāĻŦāĻā§āĻ¸ āĻ°āĻĒā§āĻ¤āĻžāĻ¨āĻŋāĻāĻžāĻ°āĻ āĻ¸ā§āĻĨāĻžāĻĒāĻ¨ āĻāĻ°āĻž āĻšāĻā§āĻā§
āĻŦāĻ°ā§āĻŖāĻ¨āĻž āĻāĻ°ā§āĻ¨ Deployment
и Service
āĻā§āĻŦāĻžāĻ°āĻ¨ā§āĻāĻ¸ā§ āĻ¸ā§āĻĨāĻžāĻĒāĻ¨āĻžāĻ° āĻāĻ¨ā§āĻ¯āĨ¤
---
kind: Service
apiVersion: v1
metadata:
name: prometheus-blackbox-exporter
labels:
app: prometheus-blackbox-exporter
spec:
type: ClusterIP
ports:
- name: http
port: 9115
protocol: TCP
selector:
app: prometheus-blackbox-exporter
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: prometheus-blackbox-exporter
labels:
app: prometheus-blackbox-exporter
spec:
replicas: 1
selector:
matchLabels:
app: prometheus-blackbox-exporter
template:
metadata:
labels:
app: prometheus-blackbox-exporter
spec:
restartPolicy: Always
containers:
- name: blackbox-exporter
image: "prom/blackbox-exporter:v0.15.1"
imagePullPolicy: IfNotPresent
securityContext:
readOnlyRootFilesystem: true
runAsNonRoot: true
runAsUser: 1000
args:
- "--config.file=/config/blackbox.yaml"
resources:
{}
ports:
- containerPort: 9115
name: http
livenessProbe:
httpGet:
path: /health
port: http
readinessProbe:
httpGet:
path: /health
port: http
volumeMounts:
- mountPath: /config
name: config
- name: configmap-reload
image: "jimmidyson/configmap-reload:v0.2.2"
imagePullPolicy: "IfNotPresent"
securityContext:
runAsNonRoot: true
runAsUser: 65534
args:
- --volume-dir=/etc/config
- --webhook-url=http://localhost:9115/-/reload
resources:
{}
volumeMounts:
- mountPath: /etc/config
name: config
readOnly: true
volumes:
- name: config
configMap:
name: prometheus-blackbox-exporter
āĻŦā§āĻ˛ā§āĻ¯āĻžāĻāĻŦāĻā§āĻ¸ āĻ°āĻĒā§āĻ¤āĻžāĻ¨āĻŋāĻāĻžāĻ°ā§ āĻ¨āĻŋāĻŽā§āĻ¨āĻ˛āĻŋāĻāĻŋāĻ¤ āĻāĻŽāĻžāĻ¨ā§āĻĄ āĻŦā§āĻ¯āĻŦāĻšāĻžāĻ° āĻāĻ°ā§ āĻ¸ā§āĻĨāĻžāĻĒāĻ¨ āĻāĻ°āĻž āĻ¯ā§āĻ¤ā§ āĻĒāĻžāĻ°ā§āĨ¤ āĻ¨āĻžāĻŽāĻ¸ā§āĻĨāĻžāĻ¨ monitoring
āĻĒā§āĻ°āĻŽāĻŋāĻĨāĻŋāĻāĻ¸ āĻ
āĻĒāĻžāĻ°ā§āĻāĻ° āĻŦā§āĻāĻžāĻ¯āĻŧāĨ¤
kubectl --namespace=monitoring apply -f blackbox-exporter.yaml
āĻ¨āĻŋāĻŽā§āĻ¨āĻ˛āĻŋāĻāĻŋāĻ¤ āĻāĻŽāĻžāĻ¨ā§āĻĄ āĻŦā§āĻ¯āĻŦāĻšāĻžāĻ° āĻāĻ°ā§ āĻ¸āĻŽāĻ¸ā§āĻ¤ āĻĒāĻ°āĻŋāĻˇā§āĻŦāĻž āĻāĻ˛āĻā§ āĻ¤āĻž āĻ¨āĻŋāĻļā§āĻāĻŋāĻ¤ āĻāĻ°ā§āĻ¨:
kubectl --namespace=monitoring get all --selector=app=prometheus-blackbox-exporter
āĻŦā§āĻ˛ā§āĻ¯āĻžāĻāĻŦāĻā§āĻ¸ āĻā§āĻ
āĻāĻĒāĻ¨āĻŋ āĻŦā§āĻ¯āĻŦāĻšāĻžāĻ° āĻāĻ°ā§ āĻŦā§āĻ˛ā§āĻ¯āĻžāĻāĻŦāĻā§āĻ¸ āĻāĻā§āĻ¸āĻĒā§āĻ°ā§āĻāĻžāĻ° āĻāĻ¯āĻŧā§āĻŦ āĻāĻ¨ā§āĻāĻžāĻ°āĻĢā§āĻ¸ āĻ
ā§āĻ¯āĻžāĻā§āĻ¸ā§āĻ¸ āĻāĻ°āĻ¤ā§ āĻĒāĻžāĻ°ā§āĻ¨ port-forward
:
kubectl --namespace=monitoring port-forward svc/prometheus-blackbox-exporter 9115:9115
āĻāĻāĻāĻŋ āĻāĻ¯āĻŧā§āĻŦ āĻŦā§āĻ°āĻžāĻāĻāĻžāĻ°ā§āĻ° āĻŽāĻžāĻ§ā§āĻ¯āĻŽā§ āĻŦā§āĻ˛ā§āĻ¯āĻžāĻāĻŦāĻā§āĻ¸ āĻāĻā§āĻ¸āĻĒā§āĻ°ā§āĻāĻžāĻ° āĻāĻ¯āĻŧā§āĻŦ āĻāĻ¨ā§āĻāĻžāĻ°āĻĢā§āĻ¸ā§āĻ° āĻ¸āĻžāĻĨā§ āĻ¸āĻāĻ¯ā§āĻ āĻāĻ°ā§āĻ¨
āĻ āĻŋāĻāĻžāĻ¨āĻžāĻ¯āĻŧ āĻā§āĻ˛ā§
āĻŽā§āĻā§āĻ°āĻŋāĻ āĻŽāĻžāĻ¨ probe_success
1 āĻāĻ° āĻ¸āĻŽāĻžāĻ¨ āĻŽāĻžāĻ¨ā§ āĻ¸āĻĢāĻ˛ āĻā§āĻāĨ¤ 0 āĻāĻ° āĻŽāĻžāĻ¨ āĻāĻāĻāĻŋ āĻ¤ā§āĻ°ā§āĻāĻŋ āĻ¨āĻŋāĻ°ā§āĻĻā§āĻļ āĻāĻ°ā§āĨ¤
āĻĒā§āĻ°āĻŽāĻŋāĻĨāĻŋāĻāĻ¸ āĻ¸ā§āĻĨāĻžāĻĒāĻ¨ āĻāĻ°āĻž āĻšāĻā§āĻā§
āĻŦā§āĻ˛ā§āĻ¯āĻžāĻāĻŦāĻā§āĻ¸ āĻ°āĻĒā§āĻ¤āĻžāĻ¨āĻŋāĻāĻžāĻ°āĻ āĻŽā§āĻ¤āĻžāĻ¯āĻŧā§āĻ¨ āĻāĻ°āĻžāĻ° āĻĒāĻ°ā§, āĻāĻŽāĻ°āĻž āĻĒā§āĻ°āĻŽāĻŋāĻĨāĻŋāĻāĻ¸āĻā§ āĻāĻ¨āĻĢāĻŋāĻāĻžāĻ° āĻāĻ°āĻŋā§ˇ prometheus-additional.yaml
.
- job_name: 'kube-api-blackbox'
scrape_interval: 1w
metrics_path: /probe
params:
module: [http_2xx]
static_configs:
- targets:
- https://www.google.com
- http://www.example.com
- https://prometheus.io
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: prometheus-blackbox-exporter:9115 # The blackbox exporter.
āĻāĻŽāĻ°āĻž āĻā§āĻĒāĻ¨ā§āĻ¨ āĻāĻ°āĻŋ Secret
āĻ¨āĻŋāĻŽā§āĻ¨āĻ˛āĻŋāĻāĻŋāĻ¤ āĻāĻŽāĻžāĻ¨ā§āĻĄ āĻŦā§āĻ¯āĻŦāĻšāĻžāĻ° āĻāĻ°ā§āĨ¤
PROMETHEUS_ADD_CONFIG=$(cat prometheus-additional.yaml | base64)
cat << EOF | kubectl --namespace=monitoring apply -f -
apiVersion: v1
kind: Secret
metadata:
name: additional-scrape-configs
type: Opaque
data:
prometheus-additional.yaml: $PROMETHEUS_ADD_CONFIG
EOF
āĻāĻŽāĻ°āĻž āĻāĻā§āĻāĻŋāĻ¤ additional-scrape-configs
āĻŦā§āĻ¯āĻŦāĻšāĻžāĻ° āĻāĻ°ā§ āĻĒā§āĻ°āĻŽāĻŋāĻĨāĻŋāĻāĻ¸ āĻ
āĻĒāĻžāĻ°ā§āĻāĻ°ā§āĻ° āĻāĻ¨ā§āĻ¯ additionalScrapeConfigs
.
kubectl --namespace=monitoring edit prometheuses k8s
...
spec:
additionalScrapeConfigs:
key: prometheus-additional.yaml
name: additional-scrape-configs
āĻāĻŽāĻ°āĻž āĻĒā§āĻ°āĻŽāĻŋāĻĨāĻŋāĻāĻ¸ āĻāĻ¯āĻŧā§āĻŦ āĻāĻ¨ā§āĻāĻžāĻ°āĻĢā§āĻ¸ā§ āĻ¯āĻžāĻ āĻāĻŦāĻ āĻŽā§āĻā§āĻ°āĻŋāĻā§āĻ¸ āĻāĻŦāĻ āĻ˛āĻā§āĻˇā§āĻ¯āĻā§āĻ˛āĻŋ āĻĒāĻ°ā§āĻā§āĻˇāĻž āĻāĻ°āĻŋāĨ¤
kubectl --namespace=monitoring port-forward svc/prometheus-k8s 9090:9090
āĻāĻŽāĻ°āĻž āĻŦā§āĻ˛ā§āĻ¯āĻžāĻāĻŦāĻā§āĻ¸ā§āĻ° āĻŽā§āĻā§āĻ°āĻŋāĻā§āĻ¸ āĻāĻŦāĻ āĻ˛āĻā§āĻˇā§āĻ¯āĻā§āĻ˛āĻŋ āĻĻā§āĻāĻŋāĨ¤
āĻŦāĻŋāĻā§āĻāĻĒā§āĻ¤āĻŋāĻ° āĻāĻ¨ā§āĻ¯ āĻ¨āĻŋāĻ¯āĻŧāĻŽ āĻ¯ā§āĻ āĻāĻ°āĻž (āĻ¸āĻ¤āĻ°ā§āĻāĻ¤āĻž)
āĻŦā§āĻ˛ā§āĻ¯āĻžāĻāĻŦāĻā§āĻ¸ āĻ°āĻĒā§āĻ¤āĻžāĻ¨āĻŋāĻāĻžāĻ°āĻ āĻĨā§āĻā§ āĻŦāĻŋāĻā§āĻāĻĒā§āĻ¤āĻŋ āĻĒā§āĻ¤ā§, āĻāĻŽāĻ°āĻž āĻĒā§āĻ°āĻŽāĻŋāĻĨāĻŋāĻāĻ¸ āĻ āĻĒāĻžāĻ°ā§āĻāĻ°ā§ āĻ¨āĻŋāĻ¯āĻŧāĻŽ āĻ¯ā§āĻ āĻāĻ°āĻŦāĨ¤
kubectl --namespace=monitoring edit prometheusrules prometheus-k8s-rules
...
- name: blackbox-exporter
rules:
- alert: ProbeFailed
expr: probe_success == 0
for: 5m
labels:
severity: error
annotations:
summary: "Probe failed (instance {{ $labels.instance }})"
description: "Probe failedn VALUE = {{ $value }}n LABELS: {{ $labels }}"
- alert: SlowProbe
expr: avg_over_time(probe_duration_seconds[1m]) > 1
for: 5m
labels:
severity: warning
annotations:
summary: "Slow probe (instance {{ $labels.instance }})"
description: "Blackbox probe took more than 1s to completen VALUE = {{ $value }}n LABELS: {{ $labels }}"
- alert: HttpStatusCode
expr: probe_http_status_code <= 199 OR probe_http_status_code >= 400
for: 5m
labels:
severity: error
annotations:
summary: "HTTP Status Code (instance {{ $labels.instance }})"
description: "HTTP status code is not 200-399n VALUE = {{ $value }}n LABELS: {{ $labels }}"
- alert: SslCertificateWillExpireSoon
expr: probe_ssl_earliest_cert_expiry - time() < 86400 * 30
for: 5m
labels:
severity: warning
annotations:
summary: "SSL certificate will expire soon (instance {{ $labels.instance }})"
description: "SSL certificate expires in 30 daysn VALUE = {{ $value }}n LABELS: {{ $labels }}"
- alert: SslCertificateHasExpired
expr: probe_ssl_earliest_cert_expiry - time() <= 0
for: 5m
labels:
severity: error
annotations:
summary: "SSL certificate has expired (instance {{ $labels.instance }})"
description: "SSL certificate has expired alreadyn VALUE = {{ $value }}n LABELS: {{ $labels }}"
- alert: HttpSlowRequests
expr: avg_over_time(probe_http_duration_seconds[1m]) > 1
for: 5m
labels:
severity: warning
annotations:
summary: "HTTP slow requests (instance {{ $labels.instance }})"
description: "HTTP request took more than 1sn VALUE = {{ $value }}n LABELS: {{ $labels }}"
- alert: SlowPing
expr: avg_over_time(probe_icmp_duration_seconds[1m]) > 1
for: 5m
labels:
severity: warning
annotations:
summary: "Slow ping (instance {{ $labels.instance }})"
description: "Blackbox ping took more than 1sn VALUE = {{ $value }}n LABELS: {{ $labels }}"
āĻĒā§āĻ°āĻŽāĻŋāĻĨāĻŋāĻāĻ¸ āĻāĻ¯āĻŧā§āĻŦ āĻāĻ¨ā§āĻāĻžāĻ°āĻĢā§āĻ¸ā§, āĻ¸ā§āĻā§āĻ¯āĻžāĻāĻžāĻ¸ => āĻ¨āĻŋāĻ¯āĻŧāĻŽāĻā§āĻ˛āĻŋāĻ¤ā§ āĻ¯āĻžāĻ¨ āĻāĻŦāĻ āĻŦā§āĻ˛ā§āĻ¯āĻžāĻāĻŦāĻā§āĻ¸-āĻ°āĻĒā§āĻ¤āĻžāĻ¨āĻŋāĻāĻžāĻ°āĻā§āĻ° āĻāĻ¨ā§āĻ¯ āĻ¸āĻ¤āĻ°ā§āĻāĻ¤āĻžāĻ° āĻ¨āĻŋāĻ¯āĻŧāĻŽāĻā§āĻ˛āĻŋ āĻā§āĻāĻā§āĻ¨āĨ¤
Kubernetes API āĻ¸āĻžāĻ°ā§āĻāĻžāĻ° SSL āĻļāĻāĻ¸āĻžāĻĒāĻ¤ā§āĻ°ā§āĻ° āĻŽā§āĻ¯āĻŧāĻžāĻĻ āĻļā§āĻˇ āĻšāĻāĻ¯āĻŧāĻžāĻ° āĻŦāĻŋāĻā§āĻāĻĒā§āĻ¤āĻŋ āĻāĻ¨āĻĢāĻŋāĻāĻžāĻ° āĻāĻ°āĻž āĻšāĻā§āĻā§
āĻāĻ¸ā§āĻ¨ Kubernetes API āĻ¸āĻžāĻ°ā§āĻāĻžāĻ° SSL āĻļāĻāĻ¸āĻžāĻĒāĻ¤ā§āĻ°ā§āĻ° āĻŽā§āĻ¯āĻŧāĻžāĻĻ āĻļā§āĻˇ āĻšāĻāĻ¯āĻŧāĻžāĻ° āĻĒāĻ°ā§āĻ¯āĻŦā§āĻā§āĻˇāĻŖ āĻāĻ¨āĻĢāĻŋāĻāĻžāĻ° āĻāĻ°āĻŋāĨ¤ āĻāĻāĻŋ āĻ¸āĻĒā§āĻ¤āĻžāĻšā§ āĻāĻāĻŦāĻžāĻ° āĻŦāĻŋāĻā§āĻāĻĒā§āĻ¤āĻŋ āĻĒāĻžāĻ āĻžāĻŦā§āĨ¤
Kubernetes API āĻ¸āĻžāĻ°ā§āĻāĻžāĻ° āĻĒā§āĻ°āĻŽāĻžāĻŖā§āĻāĻ°āĻŖā§āĻ° āĻāĻ¨ā§āĻ¯ āĻŦā§āĻ˛ā§āĻ¯āĻžāĻāĻŦāĻā§āĻ¸ āĻāĻā§āĻ¸āĻĒā§āĻ°ā§āĻāĻžāĻ° āĻŽāĻĄāĻŋāĻāĻ˛ āĻ¯ā§āĻ āĻāĻ°āĻž āĻšāĻā§āĻā§āĨ¤
kubectl --namespace=monitoring edit configmap prometheus-blackbox-exporter
...
kube-api:
http:
method: GET
no_follow_redirects: false
preferred_ip_protocol: ip4
tls_config:
insecure_skip_verify: false
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
valid_http_versions:
- HTTP/1.1
- HTTP/2
valid_status_codes: []
prober: http
timeout: 5s
āĻĒā§āĻ°āĻŽāĻŋāĻĨāĻŋāĻāĻ¸ āĻ¸ā§āĻā§āĻ°ā§āĻ¯āĻžāĻĒ āĻāĻ¨āĻĢāĻŋāĻāĻžāĻ°ā§āĻļāĻ¨ āĻ¯ā§āĻ āĻāĻ°āĻž āĻšāĻā§āĻā§
- job_name: 'kube-api-blackbox'
metrics_path: /probe
params:
module: [kube-api]
static_configs:
- targets:
- https://kubernetes.default.svc/api
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: prometheus-blackbox-exporter:9115 # The blackbox exporter.
āĻĒā§āĻ°āĻŽāĻŋāĻĨāĻŋāĻāĻ¸ āĻ¸āĻŋāĻā§āĻ°ā§āĻ āĻŦā§āĻ¯āĻŦāĻšāĻžāĻ° āĻāĻ°ā§
PROMETHEUS_ADD_CONFIG=$(cat prometheus-additional.yaml | base64)
cat << EOF | kubectl --namespace=monitoring apply -f -
apiVersion: v1
kind: Secret
metadata:
name: additional-scrape-configs
type: Opaque
data:
prometheus-additional.yaml: $PROMETHEUS_ADD_CONFIG
EOF
āĻ¸āĻ¤āĻ°ā§āĻāĻ¤āĻž āĻ¨āĻŋāĻ¯āĻŧāĻŽ āĻ¯ā§āĻ āĻāĻ°āĻž āĻšāĻā§āĻā§
kubectl --namespace=monitoring edit prometheusrules prometheus-k8s-rules
...
- name: k8s-api-server-cert-expiry
rules:
- alert: K8sAPIServerSSLCertExpiringAfterThreeMonths
expr: probe_ssl_earliest_cert_expiry{job="kube-api-blackbox"} - time() < 86400 * 90
for: 1w
labels:
severity: warning
annotations:
summary: "Kubernetes API Server SSL certificate will expire after three months (instance {{ $labels.instance }})"
description: "Kubernetes API Server SSL certificate expires in 90 daysn VALUE = {{ $value }}n LABELS: {{ $labels }}"
āĻĻāĻ°āĻāĻžāĻ°ā§ āĻ˛āĻŋāĻā§āĻ
āĻāĻ¤ā§āĻ¸: www.habr.com