Monitoring a Kubernetes Cluster: An Overview and Introduction to Prometheus

Consider the concept of Kubernetes monitoring, get acquainted with the Prometheus tool, and talk about alerting.

The topic of monitoring is voluminous, it cannot be disassembled in one article. The purpose of this text is to provide an overview of the tools, concepts and approaches.

The material of the article is a squeeze from open lecture of the school "Slurm". If you want to take a full course - sign up for a course on Monitoring and logging infrastructure in Kubernetes.

What is monitored in a Kubernetes cluster

physical servers. If the Kubernetes cluster is deployed on its servers, you need to monitor their health. Zabbix handles this task; if you work with him, then you do not need to refuse, there will be no conflicts. It is Zabbix that monitors the state of our servers.

Let's move on to monitoring at the cluster level.

Control Plane components: API, Scheduler and others. At a minimum, you need to make sure that the API of servers or etcd is greater than 0. Etcd can return a lot of metrics: by the disks on which it is spinning, by the health of its etcd cluster, and others.

Docker appeared a long time ago and everyone is well aware of its problems: a lot of containers generate freezes and other problems. Therefore, Docker itself, as a system, should also be controlled, at least for availability.

DNS If DNS falls off in the cluster, then the entire Discovery service will fall off after it, calls from pods to pods will stop working. In my practice, there were no such problems, but this does not mean that the state of the DNS does not need to be monitored. Request latency and some other metrics can be tracked on CoreDNS.

Ingress. It is necessary to control the availability of ingresses (including the Ingress Controller) as entry points to the project.

The main components of the cluster have been dismantled - now let's go down to the level of abstractions.

It would seem that applications run in pods, which means they need to be controlled, but in reality they are not. Pods are ephemeral: today they run on one server, tomorrow on another; today there are 10 of them, tomorrow 2. Therefore, no one monitors the pods. Within a microservice architecture, it is more important to control the availability of the application as a whole. In particular, check the availability of service endpoints: does anything work? If the application is available, then what happens behind it, how many replicas are now - these are questions of the second order. There is no need to monitor individual instances.

At the last level, you need to control the operation of the application itself, take business metrics: the number of orders, user behavior, and so on.

Prometheus

The best system for monitoring a cluster is Prometheus. I don't know of any tool that can match Prometheus in terms of quality and ease of use. It is great for flexible infrastructure, so when they say “Kubernetes monitoring”, they usually mean Prometheus.

There are a couple of options for getting started with Prometheus: using Helm, you can install a regular Prometheus or Prometheus Operator.

1. Regular Prometheus. Everything is fine with him, but you need to configure ConfigMap - in fact, write text-based configuration files, as we did before, before the microservice architecture.
2. Prometheus Operator is a little more spread out, a little more complicated in terms of internal logic, but it’s easier to work with it: there are separate objects, abstractions are added to the cluster, so they are much more convenient to control and configure.

To understand the product, I recommend installing the regular Prometheus first. You will have to configure everything through the config, but this will be beneficial: you will figure out what belongs to what and how it is configured. In Prometheus Operator, you immediately rise to an abstraction higher, although you can also delve into the depths if you wish.

Prometheus is well integrated with Kubernetes: it can access and interact with the API Server.

Prometheus is popular, which is why a large number of applications and programming languages ​​support it. Support is needed, since Prometheus has its own metrics format, and to transfer it, you need either a library inside the application or a ready-made exporter. And there are quite a few such exporters. For example, there is PostgreSQL Exporter: it takes data from PostgreSQL and converts it to Prometheus format so that Prometheus can work with it.

Prometheus architecture

Prometheus Server is the back end, the brain of Prometheus. Metrics are stored and processed here.

The metrics are stored in the time series database (TSDB). TSDB is not a separate database, but a package in the Go language that is embedded in Prometheus. Roughly speaking, everything is in one binary.

Don't store data in TSDB for a long time

The Prometheus infrastructure is not suitable for long-term storage of metrics. The default retention period is 15 days. You can exceed this limit, but keep in mind: the more data you store in TSDB and the longer you do it, the more resources it will consume. Storing historical data in Prometheus is considered bad practice.

If you have huge traffic, the number of metrics is hundreds of thousands per second, then it is better to limit their storage by disk space or by period. Usually, “hot data” is stored in TSDB, metrics in just a few hours. For longer storage, external storage is used in those databases that are really suitable for this, for example, InfluxDB, ClickHouse, and so on. I saw more good reviews about ClickHouse.

Prometheus Server works on the model pull: he goes for metrics to those endpoints that we gave him. They said: “go to the API Server”, and he goes there every n-th number of seconds and takes the metrics.

For objects with a short lifetime (job or cron job) that can appear between scraping periods, there is a Pushgateway component. Metrics from short-term objects are pushed into it: the job has risen, performed an action, sent metrics to Pushgateway and completed. After a while, Prometheus will come down at its own pace and pick up these metrics from Pushgateway.

To configure notifications in Prometheus there is a separate component - Alertmanager. And the alerting rules. For example, you need to create an alert if the server API is 0. When the event fires, the alert is passed to the alert manager for further dispatch. The alert manager has quite flexible routing settings: one group of alerts can be sent to the admins' telegram chat, another to the developers' chat, and a third to the infrastructure workers' chat. Notifications can be sent to Slack, Telegram, email, and other channels.

And finally, I’ll tell you about the Prometheus killer feature - Discovering. When working with Prometheus, you do not need to specify specific addresses of objects for monitoring, it is enough to set their type. That is, you don’t need to write “here is the IP address, here is the port - monitor”, instead, you need to determine by what principles to find these objects (targets - goals). Prometheus itself, depending on which objects are currently active, pulls up the necessary ones and adds them to monitoring.

This approach fits well with the Kubernetes structure, where everything also floats: today there are 10 servers, tomorrow 3. In order not to specify the IP address of the server each time, they wrote once how to find it - and Discovering will do it.

The Prometheus language is called PromQL. Using this language, you can get the values ​​​​of specific metrics and then convert them, build analytical calculations based on them.

https://prometheus.io/docs/prometheus/latest/querying/basics/

Простой запрос

    container_memory_usage_bytes

Математические операции

    container_memory_usage_bytes / 1024 / 1024

Встроенные функции

    sum(container_memory_usage_bytes) / 1024 / 1024

Уточнение запроса

    100 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m]) * 100)

Prometheus web interface

Prometheus has its own, fairly minimalistic web interface. Only suitable for debug or demonstration.

In the Expression line, you can write a query in the PromQL language.

The Alerts tab contains alerting rules, and they have three statuses:

1. inactive - if the alert is not active at the moment, that is, everything is fine with it, and it did not work;
2. pending - this is if the alert worked, but the sending has not yet passed. The delay is set to compensate for network blinking: if the specified service has risen within a minute, then the alarm should not be sounded yet;
3. firing is the third status when the alert lights up and sends messages.

In the Status menu you will find access to information about what Prometheus is. There is also a transition to the targets (targets), which we talked about above.

For a more detailed overview of the Prometheus interface, see in Slurm's lecture on monitoring a Kubernetes cluster.

Integration with Grafana

In the Prometheus web interface, you will not find beautiful and understandable graphs from which you can draw a conclusion about the state of the cluster. To build them, Prometheus is integrated with Grafana. We get such dashboards.

Setting up the integration of Prometheus and Grafana is not difficult at all, you can find instructions in the documentation: GRAFANA SUPPORT FOR PROMETHEUSWell, I'll end with this.

In the following articles, we will continue the topic of monitoring: we will talk about collecting and analyzing logs using Grafana Loki and alternative tools.

Author: Marcel Ibraev, certified Kubernetes administrator, practicing engineer in the company Southbridge, speaker and course developer Slurm.

Source: habr.com