Welcome to the series of brief guides on Kubernetes. This is a regular column featuring the most interesting questions we receive online and during our training sessions, answered by a Kubernetes expert.
Today's expert is Daniele Polencic (). Daniele works as an instructor and software developer at .
If you want to get an answer to your question in the next post, or in .
Missed the previous posts? .
How to connect Kubernetes clusters in different data centers?
Summary: , and I also recommend reading about and .
Often, infrastructure is replicated and distributed across various regions, especially in controlled environments.
If one region becomes unavailable, traffic is redirected to another to avoid outages.
With Kubernetes, a similar strategy can be employed to distribute workloads across different regions.
You may have one or several clusters per team, region, environment, or a combination of these elements.
Your clusters can be hosted across various clouds and on-premises environments.
But how do you plan infrastructure for such geographic dispersion?
Should you create one large cluster across multiple cloud environments over a single network?
Or establish many small clusters and find a way to control and synchronize them?
One guiding cluster
Creating a single cluster over a single network is not that straightforward.
Imagine, you have an outage, and connectivity is lost between cluster segments.
If you have one master server, half of the resources won't be able to receive new commands because they can't connect to the master.
Meanwhile, you have old routing tables (kube-proxy can't load new ones) and no additional pods (kubelet can't request updates).
Worse yet, if Kubernetes can't see a node, it marks it as lost and redistributes the missing pods across existing nodes.
As a result, you have twice as many pods.
If you set up one master server for each region, there will be issues with the consensus algorithm in the etcd database.Note: In fact, the etcd database does not necessarily have to be on the master servers. It can be run on a separate group of servers in one region. However, this results in a single point of failure for the cluster. But it's fast.)
etcd uses , to agree on a value before writing it to disk.
This means that a majority of instances must reach consensus before the state can be written to etcd.
If the latency between etcd instances sharply increases, as in the case of three etcd instances in different regions, it takes a lot of time to agree on a value and write it to disk.
This is also reflected in Kubernetes controllers.
The controller manager takes longer to learn about a change and write the response to the database.
And since there is not just one controller, but several, it results in a chain reaction, causing the entire cluster to slow down significantly..
etcd is so sensitive to latency that .
Currently, there are no good examples of a large network for a single cluster.
Primarily, the developer community and the SIG-cluster group are trying to figure out how to orchestrate clusters in the same way that Kubernetes orchestrates containers.
Option 1: cluster federation with kubefed
The official response from SIG-cluster is that .
Attempted to manage a collection of clusters as a single object using the kube federation tool for the first time.
The beginning was promising, but ultimately kube federation never gained popularity because it did not support all resources.
It supported aggregated deployments and services, but did not support, for example, StatefulSets.
Additionally, the federation configuration was passed as annotations and lacked flexibility.
Imagine how one could describe the distribution of replicas for each cluster in the federation using just annotations.
It resulted in complete chaos.
SIG-cluster did a lot of work after kubefed v1 and decided to approach the problem from a different angle.
Instead of annotations, they decided to release a controller that is installed on the clusters. It can be configured using Custom Resource Definitions (CRD).
For each resource that will be part of the federation, you have a custom CRD definition consisting of three sections:
- a standard resource definition, such as a deployment;
- section
placement, where you define how the resource will be distributed in the federation; - section
override, where you can override the weight and parameters from placement for a specific resource.
Here is an example of a federated deployment with placement and override sections.
apiVersion: types.federation.k8s.io/v1alpha1
kind: FederatedDeployment
metadata:
name: test-deployment
namespace: test-namespace
spec:
template:
metadata:
labels:
app: nginx
spec:
replicas: 3
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- image: nginx
name: nginx
placement:
clusterNames:
- cluster2
- cluster1
overrides:
- clusterName: cluster2
clusterOverrides:
- path: spec.replicas
value: 5As you can see, the deployment is distributed across two clusters: cluster1 and cluster2.
The first cluster supplies three replicas, while the second one has a value of 5 set.
If you need more control over the number of replicas, kubefed2 provides a new object called ReplicaSchedulingPreference, where replicas can be distributed by weight:
apiVersion: scheduling.federation.k8s.io/v1alpha1
kind: ReplicaSchedulingPreference
metadata:
name: test-deployment
namespace: test-ns
spec:
targetKind: FederatedDeployment
totalReplicas: 9
clusters:
A:
weight: 1
B:
weight: 2The CRD structure and API are still not completely ready, and active work is underway in the official project repository.
Keep an eye on kubefed2, but remember, it is not yet suitable for production environments.
Learn more about kubefed2 from in the Kubernetes blog and in the .
Option 2: Cluster Federation in the Style of Booking.com
Booking.com developers have not dealt with kubefed v2, but they came up with Shipper — an operator for deployment across multiple clusters, in different regions, and in various clouds.
Somewhat similar to kubefed2.
Both tools allow you to configure a deployment strategy across multiple clusters (which clusters are used and how many replicas they have).
However, the goal of Shipper is to reduce the risk of deployment errors.
In Shipper, you can define a series of steps that describe the splitting of replicas between the previous and current deployments and the volume of incoming traffic.
When you send a resource to a cluster, the Shipper controller gradually deploys this change across all federated clusters.
Moreover, Shipper is very limited.
For example, It takes Helm charts as input. and does not support vanilla resources.
In general, Shipper works as follows.
Instead of the standard delivery, you need to create an application resource that includes a Helm chart:
apiVersion: shipper.booking.com/v1alpha1
kind: Application
metadata:
name: super-server
spec:
revisionHistoryLimit: 3
template:
chart:
name: nginx
repoUrl: https://storage.googleapis.com/shipper-demo
version: 0.0.1
clusterRequirements:
regions:
- name: local
strategy:
steps:
- capacity:
contender: 1
incumbent: 100
name: staging
traffic:
contender: 0
incumbent: 100
- capacity:
contender: 100
incumbent: 0
name: full on
traffic:
contender: 100
incumbent: 0
values:
replicaCount: 3Shipper is a good option for managing multiple clusters, but its close integration with Helm is a hindrance.
What if we all switch from Helm to or ?
Learn more about Shipper and its philosophy in .
If you want to dive into the code, .
Option 3: 'magical' cluster merging
Kubefed v2 and Shipper work with cluster federation, providing clusters with new resources through custom resource definitions.
But what if you don’t want to rewrite all deliveries, StatefulSets, DaemonSets, etc., for merging?
How to include an existing cluster in the federation without changing the YAML?
, which deals with scheduling workloads in clusters.
But instead of creating a new way to interact with the cluster and wrapping resources in custom definitions, the multi-cluster-scheduler integrates into the standard Kubernetes lifecycle and intercepts all calls that create pods.
Every created pod is immediately replaced with a placeholder.
multi-cluster-scheduler uses , to intercept the call and create an idle pod placeholder.
The original pod goes through another scheduling cycle, where a decision is made on placement after polling the entire federation.
Finally, the pod is delivered to the target cluster.
As a result, you have an extra pod that does nothing, just takes up space.
The advantage is that you didn’t have to write new resources for merging deliveries.
Every resource that creates a pod is automatically ready for merging.
It's interesting how you suddenly have supplies distributed across several regions, and you didn't even notice. However, this is quite risky, as everything here relies on magic.
But while Shipper primarily tries to mitigate the effects of deliveries, the multi-cluster-scheduler performs more general tasks and may be better suited for batch jobs.
It lacks an advanced mechanism for gradual delivery.
You can learn more about the multi-cluster-scheduler on .
If you want to read about the multi-cluster-scheduler in action, Admiralty has an — workflows, events, CI, and CD in Kubernetes.
Other tools and solutions
Connecting multiple clusters and managing them is a complex task; there is no universal solution.
If you wish to explore this topic further, here are some resources:
- — a tool that connects overlay networks of different Kubernetes clusters.
- The Target retail chain uses .
- Try using IPV6 and .
- You can use a service mesh, for example, .
- Cilium, a container network interface plugin, offers , which allows several clusters to be combined.
That's all for today.
I tried not to write too much, but in my opinion, the article still turned out to be quite long. The other features of unRAID are quite simple to configure, especially since everything can be set up with a mouse.
If you know a more effective way to connect multiple clusters, .
We'll add your method to the links.
Special thanks to Chris Nesbitt-Smith () and Vincent De Smet () (reliability engineer at ) for reading the article and sharing valuable information about how federation works.
Source: habr.com
