Sveiki visiem. MaijÄ sÄk darboties OTUS
Vide
Mums būs nepiecieŔams:
- Kubernetes
- Prometejs operators
EksportÄtÄja melnÄs kastes konfigurÄcija
Blackbox konfigurÄÅ”ana, izmantojot ConfigMap
iestatījumiem http
tīmekļa pakalpojumu uzraudzības modulis.
apiVersion: v1
kind: ConfigMap
metadata:
name: prometheus-blackbox-exporter
labels:
app: prometheus-blackbox-exporter
data:
blackbox.yaml: |
modules:
http_2xx:
http:
no_follow_redirects: false
preferred_ip_protocol: ip4
valid_http_versions:
- HTTP/1.1
- HTTP/2
valid_status_codes: []
prober: http
timeout: 5s
Modulis http_2xx
izmanto, lai pÄrbaudÄ«tu, vai tÄ«mekļa pakalpojums atgriež HTTP 2xx statusa kodu. Blackbox eksportÄtÄja konfigurÄcija ir sÄ«kÄk aprakstÄ«ta
Blackbox eksportÄtÄja izvietoÅ”ana Kubernetes klasterÄ«
Aprakstiet Deployment
Šø Service
izvietoŔanai Kubernetes.
---
kind: Service
apiVersion: v1
metadata:
name: prometheus-blackbox-exporter
labels:
app: prometheus-blackbox-exporter
spec:
type: ClusterIP
ports:
- name: http
port: 9115
protocol: TCP
selector:
app: prometheus-blackbox-exporter
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: prometheus-blackbox-exporter
labels:
app: prometheus-blackbox-exporter
spec:
replicas: 1
selector:
matchLabels:
app: prometheus-blackbox-exporter
template:
metadata:
labels:
app: prometheus-blackbox-exporter
spec:
restartPolicy: Always
containers:
- name: blackbox-exporter
image: "prom/blackbox-exporter:v0.15.1"
imagePullPolicy: IfNotPresent
securityContext:
readOnlyRootFilesystem: true
runAsNonRoot: true
runAsUser: 1000
args:
- "--config.file=/config/blackbox.yaml"
resources:
{}
ports:
- containerPort: 9115
name: http
livenessProbe:
httpGet:
path: /health
port: http
readinessProbe:
httpGet:
path: /health
port: http
volumeMounts:
- mountPath: /config
name: config
- name: configmap-reload
image: "jimmidyson/configmap-reload:v0.2.2"
imagePullPolicy: "IfNotPresent"
securityContext:
runAsNonRoot: true
runAsUser: 65534
args:
- --volume-dir=/etc/config
- --webhook-url=http://localhost:9115/-/reload
resources:
{}
volumeMounts:
- mountPath: /etc/config
name: config
readOnly: true
volumes:
- name: config
configMap:
name: prometheus-blackbox-exporter
Blackbox eksportÄtÄju var izvietot, izmantojot Å”Ädu komandu. VÄrdtelpa monitoring
attiecas uz Prometheus operatoru.
kubectl --namespace=monitoring apply -f blackbox-exporter.yaml
PÄrliecinieties, vai visi pakalpojumi darbojas, izmantojot Å”o komandu:
kubectl --namespace=monitoring get all --selector=app=prometheus-blackbox-exporter
Blackbox pÄrbaude
Blackbox eksportÄtÄja tÄ«mekļa saskarnei varat piekļūt, izmantojot port-forward
:
kubectl --namespace=monitoring port-forward svc/prometheus-blackbox-exporter 9115:9115
Izveidojiet savienojumu ar Blackbox eksportÄtÄja tÄ«mekļa saskarni, izmantojot tÄ«mekļa pÄrlÅ«kprogrammu vietnÄ
Ja dodaties uz adresi
MetriskÄ vÄrtÄ«ba probe_success
vienÄds ar 1 nozÄ«mÄ veiksmÄ«gu pÄrbaudi. VÄrtÄ«ba 0 norÄda kļūdu.
Prometeja iestatīŔana
PÄc BlackBox eksportÄtÄja izvietoÅ”anas mÄs konfigurÄjam Prometheus iekÅ”Ä prometheus-additional.yaml
.
- job_name: 'kube-api-blackbox'
scrape_interval: 1w
metrics_path: /probe
params:
module: [http_2xx]
static_configs:
- targets:
- https://www.google.com
- http://www.example.com
- https://prometheus.io
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: prometheus-blackbox-exporter:9115 # The blackbox exporter.
MÄs Ä£enerÄjam Secret
izmantojot Å”Ädu komandu.
PROMETHEUS_ADD_CONFIG=$(cat prometheus-additional.yaml | base64)
cat << EOF | kubectl --namespace=monitoring apply -f -
apiVersion: v1
kind: Secret
metadata:
name: additional-scrape-configs
type: Opaque
data:
prometheus-additional.yaml: $PROMETHEUS_ADD_CONFIG
EOF
NorÄdiet additional-scrape-configs
Prometheus Operatoram, izmantojot additionalScrapeConfigs
.
kubectl --namespace=monitoring edit prometheuses k8s
...
spec:
additionalScrapeConfigs:
key: prometheus-additional.yaml
name: additional-scrape-configs
MÄs ejam uz Prometheus tÄ«mekļa saskarni un pÄrbaudÄm metriku un mÄrÄ·us.
kubectl --namespace=monitoring port-forward svc/prometheus-k8s 9090:9090
MÄs redzam Blackbox metriku un mÄrÄ·us.
PaziÅojumu noteikumu pievienoÅ”ana (brÄ«dinÄjums)
Lai saÅemtu paziÅojumus no Blackbox eksportÄtÄja, mÄs pievienosim Prometheus Operator noteikumus.
kubectl --namespace=monitoring edit prometheusrules prometheus-k8s-rules
...
- name: blackbox-exporter
rules:
- alert: ProbeFailed
expr: probe_success == 0
for: 5m
labels:
severity: error
annotations:
summary: "Probe failed (instance {{ $labels.instance }})"
description: "Probe failedn VALUE = {{ $value }}n LABELS: {{ $labels }}"
- alert: SlowProbe
expr: avg_over_time(probe_duration_seconds[1m]) > 1
for: 5m
labels:
severity: warning
annotations:
summary: "Slow probe (instance {{ $labels.instance }})"
description: "Blackbox probe took more than 1s to completen VALUE = {{ $value }}n LABELS: {{ $labels }}"
- alert: HttpStatusCode
expr: probe_http_status_code <= 199 OR probe_http_status_code >= 400
for: 5m
labels:
severity: error
annotations:
summary: "HTTP Status Code (instance {{ $labels.instance }})"
description: "HTTP status code is not 200-399n VALUE = {{ $value }}n LABELS: {{ $labels }}"
- alert: SslCertificateWillExpireSoon
expr: probe_ssl_earliest_cert_expiry - time() < 86400 * 30
for: 5m
labels:
severity: warning
annotations:
summary: "SSL certificate will expire soon (instance {{ $labels.instance }})"
description: "SSL certificate expires in 30 daysn VALUE = {{ $value }}n LABELS: {{ $labels }}"
- alert: SslCertificateHasExpired
expr: probe_ssl_earliest_cert_expiry - time() <= 0
for: 5m
labels:
severity: error
annotations:
summary: "SSL certificate has expired (instance {{ $labels.instance }})"
description: "SSL certificate has expired alreadyn VALUE = {{ $value }}n LABELS: {{ $labels }}"
- alert: HttpSlowRequests
expr: avg_over_time(probe_http_duration_seconds[1m]) > 1
for: 5m
labels:
severity: warning
annotations:
summary: "HTTP slow requests (instance {{ $labels.instance }})"
description: "HTTP request took more than 1sn VALUE = {{ $value }}n LABELS: {{ $labels }}"
- alert: SlowPing
expr: avg_over_time(probe_icmp_duration_seconds[1m]) > 1
for: 5m
labels:
severity: warning
annotations:
summary: "Slow ping (instance {{ $labels.instance }})"
description: "Blackbox ping took more than 1sn VALUE = {{ $value }}n LABELS: {{ $labels }}"
Prometheus tÄ«mekļa saskarnÄ dodieties uz Statuss => Noteikumi un atrodiet blackbox-eksportÄtÄja brÄ«dinÄjumu noteikumus.
Kubernetes API servera SSL sertifikÄta beigu paziÅojumu konfigurÄÅ”ana
KonfigurÄsim Kubernetes API servera SSL sertifikÄta derÄ«guma termiÅa uzraudzÄ«bu. Tas sÅ«tÄ«s paziÅojumus reizi nedÄļÄ.
Blackbox eksportÄtÄja moduļa pievienoÅ”ana Kubernetes API servera autentifikÄcijai.
kubectl --namespace=monitoring edit configmap prometheus-blackbox-exporter
...
kube-api:
http:
method: GET
no_follow_redirects: false
preferred_ip_protocol: ip4
tls_config:
insecure_skip_verify: false
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
valid_http_versions:
- HTTP/1.1
- HTTP/2
valid_status_codes: []
prober: http
timeout: 5s
Prometheus skrÄpÄÅ”anas konfigurÄcijas pievienoÅ”ana
- job_name: 'kube-api-blackbox'
metrics_path: /probe
params:
module: [kube-api]
static_configs:
- targets:
- https://kubernetes.default.svc/api
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: prometheus-blackbox-exporter:9115 # The blackbox exporter.
Izmantojot Prometheus Secret
PROMETHEUS_ADD_CONFIG=$(cat prometheus-additional.yaml | base64)
cat << EOF | kubectl --namespace=monitoring apply -f -
apiVersion: v1
kind: Secret
metadata:
name: additional-scrape-configs
type: Opaque
data:
prometheus-additional.yaml: $PROMETHEUS_ADD_CONFIG
EOF
BrÄ«dinÄjuma noteikumu pievienoÅ”ana
kubectl --namespace=monitoring edit prometheusrules prometheus-k8s-rules
...
- name: k8s-api-server-cert-expiry
rules:
- alert: K8sAPIServerSSLCertExpiringAfterThreeMonths
expr: probe_ssl_earliest_cert_expiry{job="kube-api-blackbox"} - time() < 86400 * 90
for: 1w
labels:
severity: warning
annotations:
summary: "Kubernetes API Server SSL certificate will expire after three months (instance {{ $labels.instance }})"
description: "Kubernetes API Server SSL certificate expires in 90 daysn VALUE = {{ $value }}n LABELS: {{ $labels }}"
Noderīgas saites
Avots: www.habr.com