ProHoster > Blog > Administration > Kubernetes: why it is crucial to set up system resource management?

Kubernetes: why it is crucial to set up system resource management?

Typically, there is always a need to provide a dedicated pool of resources for an application to ensure its proper and stable operation. But what if several applications are running on the same infrastructure at the same time? How can we ensure that each of them has the minimally necessary resources? How can resource consumption be limited? How can the load be distributed between nodes? How can horizontal scaling be managed in case of increased load on the applications?

Kubernetes: why it is crucial to set up system resource management?

We should start by identifying the main types of resources present in the system — specifically, CPU time and RAM. In k8s manifests, these resource types are measured in the following units:

  • CPU — in cores
  • RAM — in bytes

Moreover, for each resource, there is an option to set two types of requests — requests and limits. Requests describe the minimum resource requirements of a node to run a container (and the pod as a whole), while limits impose a strict cap on the resources available to the container.

It is important to understand that in the manifest it is not necessary to explicitly define both types, and the behavior will be as follows:

  • If only limits for a resource are explicitly defined, then requests for that resource will automatically take on the value equal to limits (this can be confirmed by calling the describe function on the entity). That is, the actual operation of the container will be limited to the same amount of resources that it requires for its start.
  • If only requests for a resource are explicitly set, then there are no upper limits imposed on that resource — meaning the container is limited only by the resources of the node itself.

There is also an option to configure resource management not only at the container level but also at the namespace level using the following entities:

  • LimitRange — describes the resource limitation policy at the container/pod level in the namespace and is needed to outline default limits for the container/pod, as well as to prevent the creation of excessively large containers/pods (or vice versa), limit their quantity, and define possible variations in limits and requests.
  • ResourceQuotas – describe the resource quota policy generally across all containers in the namespace and is typically used to allocate resources across environments (useful when environments are not strictly isolated at the node level)

Below are examples of manifests where resource limits are set:

  • At the level of a specific container:

    containers:
- name: app-nginx
  image: nginx
  resources:
    requests:
      memory: 1Gi
    limits:
      cpu: 200m

    That is, in this case, to run a container with nginx, at least 1Gi of memory and 0.2 CPU must be available on the node, while the maximum that the container can consume is 0.2 CPU and all available memory on the node.

  • At the namespace level:

    apiVersion: v1
kind: ResourceQuota
metadata:
  name: nxs-test
spec:
  hard:
    requests.cpu: 300m
    requests.memory: 1Gi
    limits.cpu: 700m
    limits.memory: 2Gi

    That is, the total of all container requests in the default namespace cannot exceed 300m for CPU and 1Gi for memory, while the total of all limits cannot exceed 700m for CPU and 2Gi for memory.

  • Default limits for containers in the namespace:

    apiVersion: v1
kind: LimitRange
metadata:
  name: nxs-limit-per-container
spec:
 limits:
   - type: Container
     defaultRequest:
       cpu: 100m
       memory: 1Gi
     default:
       cpu: 1
       memory: 2Gi
     min:
       cpu: 50m
       memory: 500Mi
     max:
       cpu: 2
       memory: 4Gi

    That is, in the default namespace, all containers will have a default request of 100m for CPU and 1Gi for memory, and a limit of 1 CPU and 2Gi. There is also a limit on the possible values in request/limit for CPU (50m < x < 2) and RAM (500Mi < x < 4Gi).

  • Limits at the pod level in the namespace:

    apiVersion: v1
kind: LimitRange
metadata:
 name: nxs-limit-pod
spec:
 limits:
 - type: Pod
   max:
     cpu: 4
     memory: 1Gi

    That is, for each pod in the default namespace, a limit of 4 vCPU and 1Gi will be set.

Now, I would like to discuss the advantages that setting these limits can provide.

The load balancing mechanism between nodes

As is known, the distribution of pods across nodes is handled by a k8s component called scheduler, which operates according to a specific algorithm. This algorithm goes through two stages in choosing the optimal node for launching:

  1. Filtering
  2. Ranking

That is, according to the described policy, initially nodes are selected where the pod can be launched based on a set of predicates (including checking whether there are enough resources on the node to launch the pod — PodFitsResources), and then for each of these nodes, according to priorities points are awarded (the more available resources the node has, the more points it is assigned — LeastResourceAllocation/LeastRequestedPriority/BalancedResourceAllocation) and it runs on the node with the highest number of points (if multiple nodes meet this condition, one is selected at random).

It's important to understand that the scheduler evaluates the available resources of a node based on data stored in etcd — that is, on the sum of requested/limit resources of each pod running on that node, not on the actual resource consumption. This information can be obtained from the output of the command kubectl describe node $NODE, for example:

# kubectl describe nodes nxs-k8s-s1
..
Non-terminated Pods:         (9 in total)
  Namespace                  Name                                         CPU Requests  CPU Limits  Memory Requests  Memory Limits  AGE
  ---------                  ----                                         ------------  ----------  ---------------  -------------  ---
  ingress-nginx              nginx-ingress-controller-754b85bf44-qkt2t    0 (0%)        0 (0%)      0 (0%)           0 (0%)         233d
  kube-system                kube-flannel-26bl4                           150m (0%)     300m (1%)   64M (0%)         500M (1%)      233d
  kube-system                kube-proxy-exporter-cb629                    0 (0%)        0 (0%)      0 (0%)           0 (0%)         233d
  kube-system                kube-proxy-x9fsc                             0 (0%)        0 (0%)      0 (0%)           0 (0%)         233d
  kube-system                nginx-proxy-k8s-worker-s1                    25m (0%)      300m (1%)   32M (0%)         512M (1%)      233d
  nxs-monitoring             alertmanager-main-1                          100m (0%)     100m (0%)   425Mi (1%)       25Mi (0%)      233d
  nxs-logging                filebeat-lmsmp                               100m (0%)     0 (0%)      100Mi (0%)       200Mi (0%)     233d
  nxs-monitoring             node-exporter-v4gdq                          112m (0%)     122m (0%)   200Mi (0%)       220Mi (0%)     233d
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests           Limits
  --------           --------           ------
  cpu                487m (3%)          822m (5%)
  memory             15856217600 (2%)  749976320 (3%)
  ephemeral-storage  0 (0%)             0 (0%)

Here we see all the pods running on the specific node, as well as the resources requested by each pod. Below is how the scheduler logs look when the pod cronjob-cron-events-1573793820-xt6q9 is launched (this information will appear in the scheduler log when the 10th level of logging is set in the command start arguments —v=10):

log

I1115 07:57:21.637791       1 scheduling_queue.go:908] About to try and schedule pod nxs-stage/cronjob-cron-events-1573793820-xt6q9
I1115 07:57:21.637804       1 scheduler.go:453] Attempting to schedule pod: nxs-stage/cronjob-cron-events-1573793820-xt6q9
I1115 07:57:21.638285       1 predicates.go:829] Schedule Pod nxs-stage/cronjob-cron-events-1573793820-xt6q9 on Node nxs-k8s-s5 is allowed, Node is running only 16 out of 110 Pods.
I1115 07:57:21.638300       1 predicates.go:829] Schedule Pod nxs-stage/cronjob-cron-events-1573793820-xt6q9 on Node nxs-k8s-s6 is allowed, Node is running only 20 out of 110 Pods.
I1115 07:57:21.638322       1 predicates.go:829] Schedule Pod nxs-stage/cronjob-cron-events-1573793820-xt6q9 on Node nxs-k8s-s3 is allowed, Node is running only 20 out of 110 Pods.
I1115 07:57:21.638322       1 predicates.go:829] Schedule Pod nxs-stage/cronjob-cron-events-1573793820-xt6q9 on Node nxs-k8s-s4 is allowed, Node is running only 17 out of 110 Pods.
I1115 07:57:21.638334       1 predicates.go:829] Schedule Pod nxs-stage/cronjob-cron-events-1573793820-xt6q9 on Node nxs-k8s-s10 is allowed, Node is running only 16 out of 110 Pods.
I1115 07:57:21.638365       1 predicates.go:829] Schedule Pod nxs-stage/cronjob-cron-events-1573793820-xt6q9 on Node nxs-k8s-s12 is allowed, Node is running only 9 out of 110 Pods.
I1115 07:57:21.638334       1 predicates.go:829] Schedule Pod nxs-stage/cronjob-cron-events-1573793820-xt6q9 on Node nxs-k8s-s11 is allowed, Node is running only 11 out of 110 Pods.
I1115 07:57:21.638385       1 predicates.go:829] Schedule Pod nxs-stage/cronjob-cron-events-1573793820-xt6q9 on Node nxs-k8s-s1 is allowed, Node is running only 19 out of 110 Pods.
I1115 07:57:21.638402       1 predicates.go:829] Schedule Pod nxs-stage/cronjob-cron-events-1573793820-xt6q9 on Node nxs-k8s-s2 is allowed, Node is running only 21 out of 110 Pods.
I1115 07:57:21.638383       1 predicates.go:829] Schedule Pod nxs-stage/cronjob-cron-events-1573793820-xt6q9 on Node nxs-k8s-s9 is allowed, Node is running only 16 out of 110 Pods.
I1115 07:57:21.638335       1 predicates.go:829] Schedule Pod nxs-stage/cronjob-cron-events-1573793820-xt6q9 on Node nxs-k8s-s8 is allowed, Node is running only 18 out of 110 Pods.
I1115 07:57:21.638408       1 predicates.go:829] Schedule Pod nxs-stage/cronjob-cron-events-1573793820-xt6q9 on Node nxs-k8s-s13 is allowed, Node is running only 8 out of 110 Pods.
I1115 07:57:21.638478       1 predicates.go:1369] Schedule Pod nxs-stage/cronjob-cron-events-1573793820-xt6q9 on Node nxs-k8s-s10 is allowed, existing pods anti-affinity terms satisfied.
I1115 07:57:21.638505       1 predicates.go:1369] Schedule Pod nxs-stage/cronjob-cron-events-1573793820-xt6q9 on Node nxs-k8s-s8 is allowed, existing pods anti-affinity terms satisfied.
I1115 07:57:21.638577       1 predicates.go:1369] Schedule Pod nxs-stage/cronjob-cron-events-1573793820-xt6q9 on Node nxs-k8s-s9 is allowed, existing pods anti-affinity terms satisfied.
I1115 07:57:21.638583       1 predicates.go:829] Schedule Pod nxs-stage/cronjob-cron-events-1573793820-xt6q9 on Node nxs-k8s-s7 is allowed, Node is running only 25 out of 110 Pods.
I1115 07:57:21.638932       1 resource_allocation.go:78] cronjob-cron-events-1573793820-xt6q9 -> nxs-k8s-s10: BalancedResourceAllocation, capacity 39900 millicores 66620178432 memory bytes, total request 2343 millicores 9640186880 memory bytes, score 9
I1115 07:57:21.638946       1 resource_allocation.go:78] cronjob-cron-events-1573793820-xt6q9 -> nxs-k8s-s10: LeastResourceAllocation, capacity 39900 millicores 66620178432 memory bytes, total request 2343 millicores 9640186880 memory bytes, score 8
I1115 07:57:21.638961       1 resource_allocation.go:78] cronjob-cron-events-1573793820-xt6q9 -> nxs-k8s-s9: BalancedResourceAllocation, capacity 39900 millicores 66620170240 memory bytes, total request 4107 millicores 11307422720 memory bytes, score 9
I1115 07:57:21.638971       1 resource_allocation.go:78] cronjob-cron-events-1573793820-xt6q9 -> nxs-k8s-s8: BalancedResourceAllocation, capacity 39900 millicores 66620178432 memory bytes, total request 5847 millicores 24333637120 memory bytes, score 7
I1115 07:57:21.638975       1 resource_allocation.go:78] cronjob-cron-events-1573793820-xt6q9 -> nxs-k8s-s9: LeastResourceAllocation, capacity 39900 millicores 66620170240 memory bytes, total request 4107 millicores 11307422720 memory bytes, score 8
I1115 07:57:21.638990       1 resource_allocation.go:78] cronjob-cron-events-1573793820-xt6q9 -> nxs-k8s-s8: LeastResourceAllocation, capacity 39900 millicores 66620178432 memory bytes, total request 5847 millicores 24333637120 memory bytes, score 7
I1115 07:57:21.639022       1 generic_scheduler.go:726] cronjob-cron-events-1573793820-xt6q9_nxs-stage -> nxs-k8s-s10: TaintTolerationPriority, Score: (10)
I1115 07:57:21.639030       1 generic_scheduler.go:726] cronjob-cron-events-1573793820-xt6q9_nxs-stage -> nxs-k8s-s8: TaintTolerationPriority, Score: (10)
I1115 07:57:21.639034       1 generic_scheduler.go:726] cronjob-cron-events-1573793820-xt6q9_nxs-stage -> nxs-k8s-s9: TaintTolerationPriority, Score: (10)
I1115 07:57:21.639041       1 generic_scheduler.go:726] cronjob-cron-events-1573793820-xt6q9_nxs-stage -> nxs-k8s-s10: NodeAffinityPriority, Score: (0)
I1115 07:57:21.639053       1 generic_scheduler.go:726] cronjob-cron-events-1573793820-xt6q9_nxs-stage -> nxs-k8s-s8: NodeAffinityPriority, Score: (0)
I1115 07:57:21.639059       1 generic_scheduler.go:726] cronjob-cron-events-1573793820-xt6q9_nxs-stage -> nxs-k8s-s9: NodeAffinityPriority, Score: (0)
I1115 07:57:21.639061       1 interpod_affinity.go:237] cronjob-cron-events-1573793820-xt6q9 -> nxs-k8s-s10: InterPodAffinityPriority, Score: (0)
I1115 07:57:21.639063       1 selector_spreading.go:146] cronjob-cron-events-1573793820-xt6q9 -> nxs-k8s-s10: SelectorSpreadPriority, Score: (10)
I1115 07:57:21.639073       1 interpod_affinity.go:237] cronjob-cron-events-1573793820-xt6q9 -> nxs-k8s-s8: InterPodAffinityPriority, Score: (0)
I1115 07:57:21.639077       1 selector_spreading.go:146] cronjob-cron-events-1573793820-xt6q9 -> nxs-k8s-s8: SelectorSpreadPriority, Score: (10)
I1115 07:57:21.639085       1 interpod_affinity.go:237] cronjob-cron-events-1573793820-xt6q9 -> nxs-k8s-s9: InterPodAffinityPriority, Score: (0)
I1115 07:57:21.639088       1 selector_spreading.go:146] cronjob-cron-events-1573793820-xt6q9 -> nxs-k8s-s9: SelectorSpreadPriority, Score: (10)
I1115 07:57:21.639103       1 generic_scheduler.go:726] cronjob-cron-events-1573793820-xt6q9_nxs-stage -> nxs-k8s-s10: SelectorSpreadPriority, Score: (10)
I1115 07:57:21.639109       1 generic_scheduler.go:726] cronjob-cron-events-1573793820-xt6q9_nxs-stage -> nxs-k8s-s8: SelectorSpreadPriority, Score: (10)
I1115 07:57:21.639114       1 generic_scheduler.go:726] cronjob-cron-events-1573793820-xt6q9_nxs-stage -> nxs-k8s-s9: SelectorSpreadPriority, Score: (10)
I1115 07:57:21.639127       1 generic_scheduler.go:781] Host nxs-k8s-s10 -> Score 100037
I1115 07:57:21.639150       1 generic_scheduler.go:781] Host nxs-k8s-s8 -> Score 100034
I1115 07:57:21.639154       1 generic_scheduler.go:781] Host nxs-k8s-s9 -> Score 100037
I1115 07:57:21.639267       1 scheduler_binder.go:269] AssumePodVolumes for pod "nxs-stage/cronjob-cron-events-1573793820-xt6q9", node "nxs-k8s-s10"
I1115 07:57:21.639286       1 scheduler_binder.go:279] AssumePodVolumes for pod "nxs-stage/cronjob-cron-events-1573793820-xt6q9", node "nxs-k8s-s10": all PVCs bound and nothing to do
I1115 07:57:21.639333       1 factory.go:733] Attempting to bind cronjob-cron-events-1573793820-xt6q9 to nxs-k8s-s10

Here we see that the scheduler initially filters and forms a list of 3 nodes on which launching is possible (nxs-k8s-s8, nxs-k8s-s9, nxs-k8s-s10). It then counts the scores based on several parameters (including BalancedResourceAllocation, LeastResourceAllocation) for each of these nodes to determine the most suitable node. Ultimately, the pod is scheduled on the node with the highest score (in this case, two nodes have the same score of 100037, so one is selected randomly — nxs-k8s-s10).

OutputIf pods are running on the node without specified limits, then for k8s (in terms of resource consumption), this would be equivalent to the node having no such pods at all. Therefore, if you have a pod with a resource-intensive process (for example, wowza) and no limits set for it, a situation may arise where this pod consumes all the node's resources, while for k8s, this node is considered underutilized and will receive the same number of points during ranking (particularly in terms of the available resources evaluation) as a node with no running pods, which may ultimately lead to an uneven load distribution between nodes.

Pod eviction

As we know, each pod is assigned one of 3 QoS classes:

  1. guaranteed — assigned when each container in the pod has both a request and a limit set for memory and CPU, and these values must match
  2. burstable — at least one container in the pod has a request and limit defined, with the condition that request < limit
  3. best effort — when no container in the pod is limited in resources

When there is a resource shortage (disk, memory) on the node, the kubelet starts to rank and evict pods based on a specific algorithm that considers the pod's priority and its QoS class. For example, regarding RAM, points are awarded based on QoS class using the following principle:

  • Guaranteed: -998
  • BestEffort: 1000
  • Burstable: min(max(2, 1000 - (1000 * memoryRequestBytes) / machineMemoryCapacityBytes), 999)

That is, with the same priority, the kubelet will first evict the pods with the best effort QoS class from the node.

Output: if you want to reduce the likelihood of evicting an important pod from the node in case of resource shortage, then alongside priority, you also need to ensure that request/limit are set for it.

Horizontal Pod Autoscaling Mechanism (HPA)

When the task is to automatically increase or decrease the number of pods depending on resource usage (system resources — CPU/RAM or user resources — rps), such an entity in k8s can help: HPA (Horizontal Pod Autoscaler). Its algorithm is as follows:

  1. The current readings of the observed resource (currentMetricValue) are determined.
  2. Desired values for the resource (desiredMetricValue) are established, which for system resources are set using request.
  3. The current number of replicas (currentReplicas) is determined.
  4. The desired number of replicas (desiredReplicas) is calculated using the following formula:
    desiredReplicas = [ currentReplicas * ( currentMetricValue / desiredMetricValue )]

Scaling will not occur when the ratio (currentMetricValue / desiredMetricValue) is close to 1 (the allowable error can be set by us, by default it is 0.1).

Let's consider the operation of HPA using the example of the app-test application (described as a Deployment), where it is necessary to change the number of replicas depending on CPU consumption:

  • Application Manifest

    kind: Deployment
apiVersion: apps/v1beta2
metadata:
 name: app-test
spec:
 selector:
   matchLabels:
     app: app-test
 replicas: 2
 template:
   metadata:
     labels:
       app: app-test
   spec:
     containers:
     - name: nginx
       image: registry.nixys.ru/generic-images/nginx
       imagePullPolicy: Always
       resources:
         requests:
           cpu: 60m
       ports:
       - name: http
         containerPort: 80
       - name: nginx-exporter
         image: nginx/nginx-prometheus-exporter
         resources:
           requests:
             cpu: 30m
         ports:
         - name: nginx-exporter
           containerPort: 9113
         args:
         - -nginx.scrape-uri
         - http://127.0.0.1:80/nginx-status

    That is, we see that the pod with the application is initially launched in two instances, each containing two containers, nginx and nginx-exporter, for each of which the requests for CPU.

  • HPA Manifest

    apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
 name: app-test-hpa
spec:
 maxReplicas: 10
 minReplicas: 2
 scaleTargetRef:
   apiVersion: extensions/v1beta1
   kind: Deployment
   name: app-test
 metrics:
 - type: Resource
   resource:
     name: cpu
     target:
       type: Utilization
       averageUtilization: 30

    That is, we have created an HPA that will monitor the Deployment app-test and regulate the number of pods with the application based on the CPU metric (we expect that the pod should consume 30% of the requested CPU), while the number of replicas is in the range of 2-10.

    Now, let's consider how the HPA mechanism works if we load one of the pods:

     # kubectl top pod
 NAME                                                   CPU(cores)   MEMORY(bytes)
 app-test-78559f8f44-pgs58            101m         243Mi
 app-test-78559f8f44-cj4jz            4m           240Mi

In total, we have the following:

  • The desired value (desiredMetricValue) — according to hpa settings, is set to 30%.
  • The current value (currentMetricValue) — for calculation, the controller-manager computes the average resource consumption in %, i.e., it performs the following:
    1. It obtains the absolute metric values of the pods from the metric server, i.e., 101m and 4m.
    2. It calculates the average absolute value, i.e., (101m + 4m) / 2 = 53m.
    3. It retrieves the absolute value for the desired resource consumption (by summing the requests of all containers) 60m + 30m = 90m.
    4. It computes the average percentage of CPU consumption relative to the pod's requests, i.e., 53m / 90m * 100% = 59%.

Now we have everything needed to determine if we should change the number of replicas, for this we calculate the ratio:

ratio = 59% / 30% = 1.96.

This means that the number of replicas should be increased by about 2 times to [2 * 1.96] = 4.

Output: As can be noted, for this mechanism to work, it is also necessary to have requests for all containers in the observed pod.

The horizontal node autoscaling mechanism (Cluster Autoscaler)

To mitigate the negative impact on the system during load spikes, having a configured hpa may not be sufficient. For example, according to hpa settings, the controller manager decides that the number of replicas needs to be doubled, but there are no available resources on the nodes to run that number of pods (i.e., the node cannot provide the requested pod resources). Therefore, these pods enter a Pending state.

In this case, if the provider has the relevant IaaS/PaaS (e.g., GKE/GCE, AKS, EKS, etc.), a tool like Node Autoscalercan assist. It allows you to set a maximum and minimum number of nodes in the cluster and automatically adjust the current number of nodes (by calling the cloud provider's API to add/remove a node) when resource shortages are observed in the cluster and pods cannot be scheduled (they are in a Pending state).

Output: To enable node autoscaling, it is necessary to specify requests in the pod containers so that k8s can accurately assess the load on the nodes and appropriately report that there are insufficient resources in the cluster to launch the next pod.

Conclusion

It should be noted that setting resource limits for a container is not a strict requirement for the successful launch of an application; however, it is still advisable for the following reasons:

  1. To improve the accuracy of the scheduler in balancing the load between k8s nodes
  2. To reduce the likelihood of an eviction event for the pod
  3. To enable horizontal pod autoscaling (HPA) for the application
  4. To facilitate horizontal node autoscaling (Cluster Autoscaling) for cloud providers

Also, read other articles in our blog:

Source: habr.com

Buy reliable website hosting with DDoS protection, VPS VDS servers 🔥 Buy reliable website hosting with DDoS protection, VPS VDS servers | ProHoster