Created Kube Eagle — a Prometheus exporter. It turned out to be a cool tool that helps better understand the resources of small and medium clusters. In the end, I saved quite a bit of money because I picked the right machine types and set application resource limits according to workloads.
I will describe the advantages , but first, I'll explain what caused the fuss and why quality monitoring was needed.
I managed several clusters with 4–50 nodes. Each cluster contained up to 200 microservices and applications. To use the available hardware more efficiently, most deployments were configured with burstable RAM and CPU resources. This way, pods could take available resources when needed without affecting other applications on that node. Isn't that great?
And although the cluster consumed relatively little CPU (8%) and RAM (40%), we constantly faced issues with pod eviction when they tried to allocate more memory than available on the node. At that time, we had only one panel for monitoring Kubernetes resources. Like this:
Grafana panel with metrics from cAdvisor
With such a panel, it's not a problem to see nodes that consume a lot of memory and CPU. The challenge is to figure out the reason. To keep the pods in place, we could have set guaranteed resources on all pods (requested resources equal to limits). But that's not the smartest way to utilize hardware. The cluster had several hundred gigabytes of memory, while some nodes were starving and others had 4–10 GB in reserve.
It turns out that the Kubernetes scheduler unevenly distributed workloads across available resources. The Kubernetes scheduler takes into account various configurations: affinity rules, taints and tolerations, node selectors that can limit available nodes. But in my case, there was none of that, and pods were scheduled based on the resources requested on each node.
For a pod, a node was chosen that had the most free resources and satisfied the request conditions. We found that the requested resources on the nodes did not match the actual usage, and that's where Kube Eagle came to the rescue with its resource monitoring capabilities.
Almost all of my Kubernetes clusters were monitored only with in . Node Exporter provides statistics on I/O and disk usage, CPU, and memory, while Kube State Metrics shows metrics for Kubernetes objects such as CPU and memory requests and limits.
We need to combine usage metrics with request and limit metrics in Grafana, and then we'll get all the information about the issue. It sounds simple, but in these two tools, labels are named differently, and some metrics don’t have metadata labels at all. Kube Eagle automates everything, and the dashboard looks like this:
We managed to resolve many resource issues and preserve hardware:
- Some developers didn’t know how many resources microservices needed (or simply didn’t care). We had no means to identify incorrect resource requests - we needed to know consumption along with requests and limits. Now they can see Prometheus metrics, monitor actual usage, and adjust requests and limits.
- JVM applications use as much memory as they can. The garbage collector frees up memory only when usage exceeds 75%. Since most services have burstable memory, it was always occupied by the JVM. Consequently, all these Java services consumed much more memory than expected.
- Some applications requested too much memory, and the Kubernetes scheduler didn’t allocate these nodes to other applications, even though they were less loaded than other nodes. One developer accidentally added an extra digit to the request and captured a large chunk of memory: 20 GB instead of 2. Nobody noticed. The application had 3 replicas, so 3 nodes were affected.
- We set resource limits, reconfigured pods with accurate requests, and achieved an optimal balance of hardware usage across all nodes. A couple of nodes could even be shut down. Then we realized we had the wrong machines (CPU-oriented instead of memory-oriented). We changed the type and removed a few more nodes.
Zaključki
With burstable resources in the cluster, you utilize existing hardware more efficiently; however, the Kubernetes scheduler schedules pods based on resource requests, which can be problematic. To kill two birds with one stone: to avoid issues and fully utilize resources, good monitoring is essential. That's where Kube Eagle comes in. (Prometheus exporter and Grafana monitoring panel).
Izvor: habr.com
