🥇Monitoring and logging of external services in a Kubernetes cluster

Greetings to everyone.

I couldn’t find a comprehensive guide on logging and collecting metrics from external services in systems deployed in Kubernetes. I am publishing my solution. This article assumes that you already have a working Prometheus and other services. As an example of a data source from an external stateful service, we will use the database PostgreSQL in a container Docker. The company uses a package manager Helm, below are examples based on it. For the entire solution, we prepare our own chart, which includes nested charts of all the used services.

Logging

Many companies use the technology stack for collecting, viewing, and centralizing logs Elasticsearch + Logstash + kibana, abbreviated as ELK. In our case, there’s no need to index the content, and I applied a more lightweight Loki. It is available as a Helm package; we added it as a subchart, modifying values for ingress and pv to suit our system.

values.yaml

ingress:
  enabled: true
  annotations:
     kubernetes.io/ingress.class: nginx
  hosts:
    - host: kube-loki.example.domain
      paths: 
        - /
  tls: []

....

persistence:
  type: pvc
  enabled: true
  accessModes:
    - ReadWriteOnce
  size: 100Gi
  finalizers:
    - kubernetes.io/pvc-protection
  existingClaim: "pv-loki"

To send logs to the instance Loki use Loki Docker Logging Driver.

It is necessary to install this extension on all Docker hosts from which you want to receive logs. There are several ways to instruct the daemon on how to use the extension. I specify the driver choice in yaml Docker Compose, which is part of Ansible playbook.

postgres.yaml

    - name: Run containers
      docker_compose:
        project_name: main-postgres
        definition:
          version: '3.7'
          services:
            p:
              image: "{{ postgres_version }}"
              container_name: postgresql
              restart: always
              volumes:
                - "{{ postgres_dir }}/data:/var/lib/postgresql/data"
                - "{{ postgres_dir }}/postgres_init_scripts:/docker-entrypoint-initdb.d"
              environment:
                POSTGRES_PASSWORD: {{ postgres_pass }}
                POSTGRES_USER: {{ postgres_user }}
              ports:
                - "{{ postgres_ip }}:{{ postgres_port }}:5432"
              logging:
                driver: "loki"
                options:
                  loki-url: "{{ loki_url }}"
                  loki-batch-size: "{{ loki_batch_size }}"
                  loki-retries: "{{ loki_retries }}"
...

where loki_url: kube-loki.example.domain/loki/api/v1/push

Metrics

Metrics are collected from PostgreSQL using postgres_exporter for Prometheus. Continuation of the above file Ansible playbook.

postgres.yaml

...
            pexp:
              image: "wrouesnel/postgres_exporter"
              container_name: pexporter
              restart: unless-stopped
              environment:
                DATA_SOURCE_NAME: "postgresql://{{ postgres_user }}:{{ postgres_pass }}@p:5432/postgres?sslmode=disable"
              ports:
                - "{{ postgres_ip }}:{{ postgres_exporter_port }}:9187"
              logging:
                driver: "json-file"
                options:
                  max-size: "5m"
...

To better illustrate the names of external stateful services, we will define them through Endpoints.

postgres-service.yaml

apiVersion: v1
kind: Endpoints
metadata:
  name: postgres-exporter
subsets:
  - addresses:
      - ip: {{ .Values.service.postgres.ip }}
    ports:
      - port: {{ .Values.service.postgres.port }}
        protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
  name: postgres-exporter
  labels:
    chart:  "{{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}"
spec:
  ports:
    - protocol: TCP
      port: {{ .Values.service.postgres.port }}
      targetPort: {{ .Values.service.postgres.port }}

Configuring Prometheus to collect data from postgres_exporter is done by editing values in the subchart.

values.yaml

scrape_configs:
...
  - job_name: postgres-exporter
    static_configs:
      - targets: 
         - postgres-exporter.applicationnamespace.svc.cluster.local:9187
        labels:
          alias: postgres
...

To visualize the collected data, install the corresponding Dashboard in
Grafana and configure the data sources. This can also be done through values in the Grafana subchart.

What it looks like

I hope this brief article has helped you understand the main ideas behind this solution and will save you time when setting up monitoring and logging of external services for Loki/Prometheus in a Kubernetes cluster.

Source: habr.com

Monitoring and logging of external services in Kubernetes cluster

Logging

Metrics