When creating a Kubernetes cluster, questions may arise: how many worker nodes to set up and of what type? What is better for an on-premise cluster: to buy several powerful servers or to utilize a dozen old machines in your data center? And in the cloud, is it better to take eight single-core instances or two quad-core instances?
The answers to these questions are in the article translated by the team .
Cluster Capacity
In general, a Kubernetes cluster can be considered as a large 'supernode'. Its total computing power is the sum of the capabilities of all constituent nodes.
There are several ways to achieve the desired target capacity of the cluster. For example, we need a cluster with a total capacity of 8 processor cores and 32 GB of RAM, because the application set requires such resources. Then, we can set up two nodes with 16 GB of memory each or four nodes with 8 GB of memory, two quad-core processors or four dual-core processors.
Here are just two possible ways to create a cluster:
Both options yield a cluster with the same capacity, but the configuration below has four smaller nodes, while the configuration above has two larger ones.
Which option is better?
To answer this question, let's consider the advantages of both options. We summarized them in a table.
Several large nodes
Many small nodes
Easier management of the cluster (if it's on-premise)
Smooth auto-scaling
Cheaper (if on-premise)
Cost is quite similar (in the cloud)
Can run resource-intensive applications
Full replication
Resources are utilized more efficiently (less overhead for system daemons)
Higher cluster fault tolerance
Note that we are only discussing worker nodes. The choice of the number and size of master nodes is a completely different topic.
So, let's discuss each point from the table in more detail.
First option: several large nodes
The most extreme option is a single worker node for the entire capacity of the cluster. In the example above, this would be one worker node with 16 CPU cores and 16 GB of RAM.
Advantages
Plus #1. Easier management
It's easier to manage multiple machines than an entire fleet. Updates and fixes are rolled out faster and synchronization is simpler. The number of failures in absolute terms is also reduced.
Note that all of the above relates to your own hardware, your own servers, and not to cloud instances.
In the cloud, the situation is different. The management is handled by the cloud service provider. Thus, managing ten nodes in the cloud doesn't significantly differ from managing a single node.
Traffic routing and load balancing between pods in the cloud : incoming internet traffic is directed to the main load balancer, which directs traffic to the port of one of the nodes (the NodePort service exposes a port in the range of 30000-32767 on each node in the cluster). The rules set by kube-proxy redirect traffic from the node to the pod. Here’s what it looks like for ten pods on two nodes:
Plus #2. Lower costs per node
A powerful machine is more expensive, but the price increase is not necessarily linear. In other words, one ten-core server with 10 GB of memory is usually cheaper than ten single-core servers with the same amount of memory.
However, note that this rule typically does not apply to cloud services. In current pricing schemes from all major cloud service providers, prices increase linearly with capacity.
Thus, in the cloud, it is generally not possible to save on more powerful servers.
Plus #3. You can run resource-intensive applications
Some applications require powerful servers in the cluster. For example, if a machine learning system needs 8 GB of memory, you cannot run it on nodes with 1 GB; you need at least one large working node.
Cons
Minus #1. Many pods per node
If the same task is performed on fewer nodes, there will naturally be more pods on each.
This can become a problem.
The reason is that each module adds some overhead to the container runtime (e.g., Docker), as well as kubelet and cAdvisor.
For example, kubelet regularly probes the health of all containers on the node — the more containers there are, the more work kubelet has to do.
CAdvisor collects resource usage statistics for all containers on the node, while kubelet regularly requests this information and provides it via the API. The more containers there are, the more work there is for both cAdvisor and kubelet.
If the number of modules increases, it can slow down the system and even undermine its reliability.
In the Kubernetes repository, some , that nodes jump between Ready/NotReady statuses as kubelet's regular checks of all containers on the node take too much time.
For this reason, Kubernetes . Depending on the node's performance, you may run more pods on a node, but it's hard to predict whether issues will arise or everything will work well. It’s advisable to test the performance in advance.
Drawback #2. Replication Limitations
A small number of nodes limits the effective replication degree of applications. For example, if you have a highly available application with five replicas but only two nodes, the effective replication degree of the application decreases to two.
Five replicas can only be distributed across two nodes, and if one of them fails, it immediately disables multiple replicas.
If you have five nodes or more, each replica will run on a separate node, and the failure of one node will remove at most one replica.
Thus, high availability requirements may necessitate a certain minimum number of nodes in the cluster.
Drawback #3. Worse Failure Consequences
With a small number of nodes, each failure carries more serious consequences. For example, if you have only two nodes and one of them fails, half of your modules disappear immediately.
Of course, Kubernetes will move the workload from the failed node to others. But if there are few nodes, there may not be enough free capacity. As a result, some of your applications will be unavailable until you bring the failed node back online.
Thus, the more nodes there are, the less impact hardware failures have.
Drawback #4. Larger Autoscaling Steps
In Kubernetes, there is a cluster auto-scaling system for cloud infrastructure that allows for the automatic addition or removal of nodes based on current needs. With larger nodes, auto-scaling becomes more abrupt and unwieldy. For example, adding an additional node in a two-node setup will increase the cluster capacity by 50% immediately. You will have to pay for these resources even if you do not need them.
Therefore, if you plan to use cluster auto-scaling, the smaller the nodes, the more flexible and cost-effective scaling you will achieve.
Now let's consider the advantages and disadvantages of having many small nodes.
Option two: multiple small nodes
The advantages of this approach essentially stem from the disadvantages of the opposite variant with a few large nodes.
Advantages
Plus #1. Lesser failure impact
The more nodes there are, the fewer pods on each node. For example, if you have a hundred modules across ten nodes, there will be an average of ten modules on each node.
Thus, if one of the nodes fails, you only lose 10% of the workload. It is likely that only a small number of replicas are affected, and the applications as a whole remain operational.
Moreover, the remaining nodes will likely have enough free resources for the workload of the failed node, so Kubernetes can freely reschedule the pods, and your applications will relatively quickly return to a functional state.
Plus #2. Good replication
If there are enough nodes, the Kubernetes scheduler can assign different nodes to all replicas. Thus, in the case of a node failure, only one replica will be affected, and the application will remain available.
Cons
Minus #1. Harder management
A large number of nodes is more difficult to manage. For example, each Kubernetes node must interact with all others, meaning that the number of connections grows quadratically, and all these links need to be tracked.
The node controller in the Kubernetes controller manager regularly checks all nodes in the cluster for health — the more nodes, the greater the load on the controller.
The load on the etcd database also increases — each kubelet and kube-proxy calls for etcd (via API), to which etcd must relay object updates.
In general, each worker node adds extra load on the system components of the master nodes.
Officially, Kubernetes supports clusters with However, in practice, even 500 nodes .
To manage a large number of worker nodes, it is advisable to choose more powerful master nodes. For example, kube-up the appropriate VM size for the master node based on the number of worker nodes. This means the more worker nodes you have, the more powerful the master nodes should be.
To address these specific issues, there are specialized solutions like This system allows bypassing limitations and building clusters with a huge number of worker nodes.
Drawback #2. More overhead.
On each Kubernetes worker node, a set of system daemons run — these include the container runtime (e.g., Docker), kube-proxy, and kubelet, including cAdvisor. Collectively, they consume a fixed amount of resources.
If you have many small nodes, the share of this overhead on each node is higher. For example, imagine that all system daemons of one node together consume 0.1 CPU core and 0.1 GB of memory. If you have one ten-core node with 10 GB of memory, then daemons consume 1% of the cluster's capacity. On the other hand, on ten single-core nodes with 1 GB of memory each, daemons will take up 10% of the cluster's capacity.
Thus, the fewer nodes there are, the more efficiently the infrastructure is utilized.
Drawback #3. Inefficient resource usage.
On small nodes, there may be a situation where the remaining fragments of resources are too small to allocate any workload, so they remain unused.
For example, each pod requires 0.75 GB of memory. If you have ten nodes, and each has 1 GB of memory, you can run ten pods — resulting in 0.25 GB of unused memory on each node.
This means that 25% of the memory of the entire cluster is wasted.
On a large node with 10 GB of memory, you can run 13 such pods — and only one fragment of 0.25 GB will remain unused.
In this case, only 2.5% of the memory is wasted.
Thus, resources are utilized more efficiently on larger nodes.
Several large nodes or many small ones?
So, what is better: several large nodes in a cluster or many small ones? As always, there is no definitive answer. Much depends on the type of application.
For example, if an application requires 10 GB of memory, the choice for larger nodes is evident. However, if the application requires tenfold replication for high availability, it is unlikely to be wise to risk placing replicas on only two nodes—there should be at least ten nodes in the cluster.
In intermediate situations, make your choice based on the advantages and disadvantages of each option. Some arguments may be more relevant to your situation than others.
It is also not necessary to make all nodes the same size. There is nothing preventing you from starting with nodes of one size and then adding nodes of another size, combining them in the cluster. The working nodes in a Kubernetes cluster can be fully heterogeneous. So, it's possible to try to combine the advantages of both approaches.
There is no single recipe, and each situation has its nuances; only production will reveal the truth.
Translation prepared by the cloud platform team .
More about Kubernetes: .
Source: habr.com
