I created Kube Eagle — a Prometheus exporter. It turned out to be a great tool that helps better understand the resources of small and medium clusters. In the end, I saved hundreds of dollars by selecting the right instance types and configuring application resource limits according to workloads.
I will talk about the advantages of , but first, let me explain why there was confusion and why quality monitoring was needed.
I managed several clusters with 4–50 nodes. Each cluster had up to 200 microservices and applications. To make more efficient use of the available hardware, most deployments were configured with burstable memory and CPU resources. This way, pods could utilize available resources when necessary without interfering with other applications on the same node. Isn't it great?
And even though the cluster consumed relatively little CPU (8%) and memory (40%), we constantly faced eviction issues when pods needed more memory than what was available on the node. At that time, we only had one dashboard for monitoring Kubernetes resources. It looked like this:
Grafana dashboard with cAdvisor metrics only
With such a dashboard, it is easy to see nodes that consume a lot of memory and CPU. The problem is figuring out the cause. To keep the pods in place, we could have configured guaranteed resources for all pods (requested resources equal to limits). But that's not the smartest use of resources. The cluster had several hundred gigabytes of memory, while some nodes were starving and others had 4–10 GB available.
It turns out that the Kubernetes scheduler distributed workloads unevenly across available resources. The Kubernetes scheduler considers various configurations: affinity rules, taints and tolerations, node selectors that can restrict available nodes. But in my case, nothing like that was present, and pods were scheduled based on the requested resources on each node.
A node with the most free resources that meets the request conditions was chosen for the pod. We found that the requested resources on nodes did not match the actual usage, and that's where Kube Eagle came in to help with its resource monitoring capabilities.
I had almost all Kubernetes clusters monitored only with and . Node Exporter provides statistics on input-output, disk usage, CPU, and memory usage, while Kube State Metrics shows metrics for Kubernetes objects, such as CPU and memory requests and limits.
We need to combine usage metrics with request and limit metrics in Grafana, and then we'll get the full picture of the issue. It sounds simple, but in reality, the labels in these two tools are named differently, and some metrics don't even have metadata labels at all. Kube Eagle does everything automatically, and the dashboard looks like this:
We managed to solve many resource problems and save hardware:
- Some developers were unaware of how many resources their microservices needed (or simply didn’t care). We had no way of identifying incorrect resource requests — to do that, you need to know consumption plus requests and limits. Now they can see Prometheus metrics, monitor actual usage, and adjust requests and limits accordingly.
- JVM applications take as much memory as they can carry. The garbage collector only frees up memory if it's over 75% utilized. Since most services have burstable memory, the JVM was always occupying it. As a result, all these Java services consumed significantly more memory than expected.
- Some applications were requesting too much memory, and the Kubernetes scheduler wasn't allocating those nodes to other applications, even though they were actually freer than other nodes. One developer accidentally added an extra digit in the request and grabbed a large chunk of memory: 20 GB instead of 2. No one noticed. The application had 3 replicas, so 3 nodes were affected.
- We imposed resource limits, reallocated pods with proper requests, and achieved an ideal balance of hardware utilization across all nodes. A couple of nodes could even be shut down. Then we realized that we had the wrong machines (CPU-oriented rather than memory-oriented). We changed the type and removed several more nodes.
Summary
With burstable resources in the cluster, you make better use of existing hardware, but the Kubernetes scheduler schedules pods based on resource requests, which can be problematic. To kill two birds with one stone: to avoid issues and fully utilize resources—a good monitoring system is necessary. This is where it comes in. (Prometheus exporter and Grafana dashboard).
Source: habr.com
