{"id":34825,"date":"2019-10-31T22:00:35","date_gmt":"2019-10-31T19:00:35","guid":{"rendered":"https:\/\/prohoster.info\/blog\/kak-soedinit-klastery-kubernetes-v-raznyh-data-tsentrah\/"},"modified":"2019-10-31T22:00:35","modified_gmt":"2019-10-31T19:00:35","slug":"kak-soedinit-klastery-kubernetes-v-raznyh-data-tsentrah","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/kak-soedinit-klastery-kubernetes-v-raznyh-data-tsentrah","title":{"rendered":"How to connect Kubernetes clusters in different data centers","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

\"How
\nWelcome to the series of brief guides on Kubernetes.<\/strong> This is a regular column featuring the most interesting questions we receive online and during our training sessions, answered by a Kubernetes expert.<\/p>\n

<\/p>\n

Today's expert is Daniele Polencic (Daniele Polencic<\/a><\/noindex>). Daniele works as an instructor and software developer at Learnk8s<\/a><\/noindex>.<\/p><\/blockquote>\n

If you want to get an answer to your question in the next post, contact us via email<\/a><\/noindex> or in Twitter: @learnk8s<\/a><\/noindex>.<\/p>\n

<\/p>\n

Missed the previous posts? You can find them here<\/a><\/noindex>.<\/p>\n

<\/p>\n

How to connect Kubernetes clusters in different data centers?<\/h3>\n

<\/p>\n

Summary<\/strong>: Kubefed v2 is coming soon<\/a><\/noindex>, and I also recommend reading about Shipper<\/a><\/noindex> and the multi-cluster-scheduler project.<\/a><\/noindex>.
\n<\/a><\/noindex><\/p>\n

Often, infrastructure is replicated and distributed across various regions, especially in controlled environments.<\/p><\/blockquote>\n

If one region becomes unavailable, traffic is redirected to another to avoid outages.<\/strong><\/p>\n

<\/p>\n

With Kubernetes, a similar strategy can be employed to distribute workloads across different regions.<\/p>\n

<\/p>\n

You may have one or several clusters per team, region, environment, or a combination of these elements.<\/p>\n

<\/p>\n

Your clusters can be hosted across various clouds and on-premises environments.<\/p>\n

<\/p>\n

But how do you plan infrastructure for such geographic dispersion?
\nShould you create one large cluster across multiple cloud environments over a single network?
\nOr establish many small clusters and find a way to control and synchronize them?<\/em><\/p>\n

<\/p>\n

One guiding cluster<\/h3>\n

<\/p>\n

Creating a single cluster over a single network is not that straightforward.<\/em><\/p>\n

<\/p>\n

Imagine, you have an outage, and connectivity is lost between cluster segments.<\/p>\n

<\/p>\n

If you have one master server, half of the resources won't be able to receive new commands because they can't connect to the master.<\/p>\n

<\/p>\n

Meanwhile, you have old routing tables (kube-proxy<\/code> can't load new ones) and no additional pods (kubelet can't request updates).<\/p>\n

<\/p>\n

Worse yet, if Kubernetes can't see a node, it marks it as lost and redistributes the missing pods across existing nodes.<\/p>\n

<\/p>\n

As a result, you have twice as many pods.<\/strong><\/p>\n

<\/p>\n

If you set up one master server for each region, there will be issues with the consensus algorithm in the etcd database.Note: In fact, the etcd database does not necessarily have to be on the master servers. It can be run on a separate group of servers in one region. However, this results in a single point of failure for the cluster. But it's fast.<\/em>)<\/p>\n

<\/p>\n

etcd uses the Raft algorithm<\/a><\/noindex>, to agree on a value before writing it to disk.
\nThis means that a majority of instances must reach consensus before the state can be written to etcd.<\/p>\n

<\/p>\n

If the latency between etcd instances sharply increases, as in the case of three etcd instances in different regions, it takes a lot of time to agree on a value and write it to disk.
\nThis is also reflected in Kubernetes controllers.<\/p>\n

<\/p>\n

The controller manager takes longer to learn about a change and write the response to the database.<\/p>\n

<\/p>\n

And since there is not just one controller, but several, it results in a chain reaction, causing the entire cluster to slow down significantly.<\/strong>.<\/p>\n

<\/p>\n

etcd is so sensitive to latency that the official documentation recommends using SSDs instead of regular hard drives.<\/a><\/noindex>.<\/p>\n

<\/p>\n

Currently, there are no good examples of a large network for a single cluster.<\/strong><\/p>\n

<\/p>\n

Primarily, the developer community and the SIG-cluster group are trying to figure out how to orchestrate clusters in the same way that Kubernetes orchestrates containers.<\/p>\n

<\/p>\n

Option 1: cluster federation with kubefed<\/h3>\n

<\/p>\n

The official response from SIG-cluster is that kubefed2, the new version of the original client and kube federation operator,<\/a><\/noindex>.<\/p>\n

<\/p>\n

Attempted to manage a collection of clusters as a single object using the kube federation tool for the first time.<\/p>\n

<\/p>\n

The beginning was promising, but ultimately kube federation never gained popularity because it did not support all resources.<\/p>\n

<\/p>\n

It supported aggregated deployments and services, but did not support, for example, StatefulSets.
\nAdditionally, the federation configuration was passed as annotations and lacked flexibility.<\/p>\n

<\/p>\n

Imagine how one could describe the distribution of replicas for each cluster in the federation using just annotations.<\/em><\/p>\n

<\/p>\n

It resulted in complete chaos.<\/strong><\/p>\n

<\/p>\n

SIG-cluster did a lot of work after kubefed v1 and decided to approach the problem from a different angle.<\/p>\n

<\/p>\n

Instead of annotations, they decided to release a controller that is installed on the clusters. It can be configured using Custom Resource Definitions (CRD).<\/strong><\/p>\n

<\/p>\n

For each resource that will be part of the federation, you have a custom CRD definition consisting of three sections:<\/p>\n

<\/p>\n