🥇 Our experience of working with data in etcd of a Kubernetes cluster directly (without K8s API)

More and more clients are approaching us with requests to provide access to the Kubernetes cluster in order to connect to services within the cluster: to be able to directly connect to a specific database or service, to link a local application with applications inside the cluster...

For example, there is a need to connect from your local machine to the service memcached.staging.svc.cluster.local. We provide this capability through a VPN within the cluster, which the client connects to. To do this, we announce the subnets of the pods, services, and push the cluster DNS to the client. Thus, when the client tries to connect to the service memcached.staging.svc.cluster.local, the request goes to the cluster's DNS and it responds with the address of this service from the cluster's service network or the address of the pod.

We configure K8s clusters using kubeadm, where by default the service subnet is 192.168.0.0/16, and the pod network is 10.244.0.0/16. Usually, everything works well, but there are a couple of issues:

The subnet 192.168.*.* is often used in clients' office networks, and even more frequently — in developers' home networks. And then we encounter conflicts: home routers operate in this subnet and the VPN pushes these subnets from the cluster to the client.
We have several clusters (production, stage, and/or several dev clusters). Thus, by default, they will all have identical subnets for pods and services, which creates significant difficulties for working simultaneously with services across multiple clusters.

We have long adopted the practice of using different subnets for services and pods within a single project — generally, so that all clusters have different networks. However, there are a large number of clusters in operation that we would prefer not to roll out from scratch, as many services, stateful applications, etc. are running in them.

So we posed the question: how can we change the subnet in an existing cluster?

Searching for solutions

The most common practice is to recreate all services of the ClusterIP type. Alternatively, they might suggest the following:

The following process has a problem: after everything is configured, the pods come up with the old IP as a DNS nameserver in /etc/resolv.conf.
Since I still did not find the solution, I had to reset the entire cluster with kubeadm reset and initialize it again.

However, this does not work for everyone... Here are more detailed inputs for our case:

Flannel is used;
There are clusters both in the cloud and on-premises;
We would like to avoid redeploying all services in the cluster;
There is a need to do everything with the minimum amount of problems;
Kubernetes version — 1.16.6 (however, further actions will be similar for other versions);
The main task boils down to replacing the service subnet in the cluster, deployed with kubeadm 192.168.0.0/16, replacing it with 172.24.0.0/16.

And it just so happened that we have long been interested in looking at what and how is stored in Kubernetes in etcd, and what can be done with it... So we thought, "Why not simply update the data in etcd by replacing old IP addresses (subnet) with new ones??»

After searching for ready-made tools for working with data in etcd, we found nothing that fully resolved the task at hand. (By the way, if you know of any utilities for working with data directly in etcd — we'd appreciate the links.) However, a good starting point was etcdhelper from OpenShift (thanks to its authors!).

This utility can connect to etcd using certificates and read data from there with the commands ls, get, dump.

Let's supplement etcdhelper

The next thought is logical: "What prevents us from extending this utility by adding the ability to write data to etcd?"

This led to a modified version of etcdhelper with two new functions changeServiceCIDR and changePodCIDR. Its code can be viewed here.

What do the new functions do? The algorithm changeServiceCIDR:

creates a deserializer;
compiles a regular expression for replacing CIDR;
iterates over all services of type ClusterIP in the cluster:
- decodes the value from etcd into a Go object;
- uses a regular expression to replace the first two bytes of the address;
- assigns the service an IP address from the new subnet;
- creates a serializer, converts the Go object to protobuf, and writes the new data to etcd.

Function changePodCIDR is essentially similar changeServiceCIDR — only instead of editing service specifications, we do it for the node and change .spec.PodCIDR to the new subnet.

Practice

Changing serviceCIDR

The plan to implement the task is very simple, but involves downtime during the recreation of all pods in the cluster. After outlining the main steps, we will also share thoughts on how to theoretically minimize this downtime.

Preparatory actions:

installing the necessary software and compiling the patched etcdhelper;
backup etcd and /etc/kubernetes.

Brief action plan for changing serviceCIDR:

modifying the manifests for the apiserver and controller-manager;
reissuance of certificates;
modification of ClusterIP services in etcd;
restart all pods in the cluster.

Below is the complete sequence of actions in detail.

1. Install etcd-client for data dumping:

apt install etcd-client

2. Build etcdhelper:

Install golang:

GOPATH=/root/golang
mkdir -p $GOPATH/local
curl -sSL https://dl.google.com/go/go1.14.1.linux-amd64.tar.gz | tar -xzvC $GOPATH/local
echo "export GOPATH="$GOPATH"" >> ~/ .bashrc
echo 'export GOROOT="$GOPATH/local/go"' >> ~/ .bashrc
echo 'export PATH="$PATH:$GOPATH/local/go/bin"' >> ~/ .bashrc

Save etcdhelper.go, download dependencies, and build:

wget https://raw.githubusercontent.com/flant/examples/master/2020/04-etcdhelper/etcdhelper.go
go get go.etcd.io/etcd/clientv3 k8s.io/kubectl/pkg/scheme k8s.io/apimachinery/pkg/runtime
go build -o etcdhelper etcdhelper.go

3. Create a backup of etcd:

backup_dir=/root/backup
mkdir ${backup_dir}
cp -rL /etc/kubernetes ${backup_dir}
ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt --key=/etc/kubernetes/pki/etcd/server.key --cert=/etc/kubernetes/pki/etcd/server.crt --endpoints https://192.168.199.100:2379 snapshot save ${backup_dir}/etcd.snapshot

4. Change the service subnet in the Kubernetes control plane manifests. In the files /etc/kubernetes/manifests/kube-apiserver.yaml and /etc/kubernetes/manifests/kube-controller-manager.yaml modify the parameter --service-cluster-ip-range to the new subnet: 172.24.0.0/16 instead of 192.168.0.0/16.

5. Since we are changing the service subnet for which kubeadm issues certificates for the apiserver (among others), they need to be reissued:

Let's check what domains and IP addresses the current certificate has been issued for:

openssl x509 -noout -ext subjectAltName </etc/kubernetes/pki/apiserver.crt
X509v3 Subject Alternative Name:
    DNS:dev-1-master, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, DNS:apiserver, IP Address:192.168.0.1, IP Address:10.0.0.163, IP Address:192.168.199.100

Prepare a minimal config for kubeadm:

cat kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
networking:
  podSubnet: "10.244.0.0/16"
  serviceSubnet: "172.24.0.0/16"
apiServer:
  certSANs:
  - "192.168.199.100" # Master node IP address

Delete the old crt and key, as the new certificate will not be issued without this:
```
rm /etc/kubernetes/pki/apiserver.{key,crt}
```

Reissue certificates for the API server:

kubeadm init phase certs apiserver --config=kubeadm-config.yaml

Check that the certificate has been issued for the new subnet:

openssl x509 -noout -ext subjectAltName </etc/kubernetes/pki/apiserver.crt
X509v3 Subject Alternative Name:
    DNS:kube-2-master, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, IP Address:172.24.0.1, IP Address:10.0.0.163, IP Address:192.168.199.100

After reissuing the API server certificate, restart its container:

docker ps | grep k8s_kube-apiserver | awk '{print $1}' | xargs docker restart

Regenerate the config for admin.conf:
```
kubeadm alpha certs renew admin.conf
```
Edit the data in etcd:
```
./etcdhelper -cacert /etc/kubernetes/pki/etcd/ca.crt -cert /etc/kubernetes/pki/etcd/server.crt -key /etc/kubernetes/pki/etcd/server.key -endpoint https://127.0.0.1:2379 change-service-cidr 172.24.0.0/16 
```
Attention! At this moment, domain resolution stops working in the cluster because the existing pods have the old address of CoreDNS (kube-dns), and kube-proxy has changed the iptables rules from the old subnet to the new one. The article goes on to discuss possible options to minimize downtime. /etc/resolv.conf Let's update the ConfigMaps in the namespace
kubectl -n kube-system edit cm kubelet-config-1.16 kube-system:
```
— here we will replace
```
clusterDNS with the new IP address of the kube-dns service: kubectl -n kube-system get svc kube-dns kubectl -n kube-system edit cm kubeadm-config.
```
— we will correct
```
data.ClusterConfiguration.networking.serviceSubnet Since the kube-dns address has changed, it is necessary to update the kubelet config on all nodes: to the new subnet.
kubeadm upgrade node phase kubelet-config && systemctl restart kubelet
```
Now, we need to restart all pods in the cluster:
```
kubectl get pods --no-headers=true --all-namespaces | sed -r 's/(S+)s+(S+).*/kubectl --namespace 1 delete pod 2/e'
```
Minimizing Downtime
```

Thoughts on how to minimize downtime:

After changing the control plane manifests, create a new kube-dns service, for example, named

kube-dns-tmp and with a new address Make 172.24.0.10.
in etcdhelper, which will not modify the kube-dns service. if Replace the address in all kubelets
ClusterDNS with the new one, while the old service continues to operate simultaneously with the new one. Wait for the application pods to roll out, either on their own due to natural reasons or at a scheduled time.
Delete the service
and change and with a new address serviceSubnetCIDR for the kube-dns service. This plan will minimize downtime to approximately one minute — during the service deletion

and subnet replacement for the service and with a new address kube-dns Modifying podNetwork.

Additionally, we decided to see how to modify podNetwork using the obtained etcdhelper. The sequence of actions is as follows:

we correct configs in

we amend the kube-controller-manager manifest; kube-system;
we change podCIDR directly in etcd;
we restart all cluster nodes.
Now more details about these actions:

1. We modify the ConfigMaps in the namespace

— we correct kube-system:

— we will correct

data.ClusterConfiguration.networking.podSubnet to the new subnet kubectl -n kube-system edit cm kube-proxy 10.55.0.0/16.

data.config.conf.clusterCIDR: 10.55.0.0/16

data.ClusterConfiguration.networking.podSubnet 2. We modify the controller-manager manifest:.

vim /etc/kubernetes/manifests/kube-controller-manager.yaml

--cluster-cidr=10.55.0.0/16

data.ClusterConfiguration.networking.podSubnet 3. We check the current values.

.spec.podCIDR .spec.podCIDRs, .InternalIP, .status.addresses, for all cluster nodes: kubectl get no -o json | jq '[.items[] | {"name": .metadata.name, "podCIDR": .spec.podCIDR, "podCIDRs": .spec.podCIDRs, "InternalIP": (.status.addresses[] | select(.type == "InternalIP") | .address)}]'

kubectl get no -o json | jq '[.items[] | {"name": .metadata.name, "podCIDR": .spec.podCIDR, "podCIDRs": .spec.podCIDRs, "InternalIP": (.status.addresses[] | select(.type == "InternalIP") | .address)}]'

[
  {
    "name": "kube-2-master",
    "podCIDR": "10.244.0.0/24",
    "podCIDRs": [
      "10.244.0.0/24"
    ],
    "InternalIP": "192.168.199.2"
  },
  {
    "name": "kube-2-master",
    "podCIDR": "10.244.0.0/24",
    "podCIDRs": [
      "10.244.0.0/24"
    ],
    "InternalIP": "10.0.1.239"
  },
  {
    "name": "kube-2-worker-01f438cf-579f9fd987-5l657",
    "podCIDR": "10.244.1.0/24",
    "podCIDRs": [
      "10.244.1.0/24"
    ],
    "InternalIP": "192.168.199.222"
  },
  {
    "name": "kube-2-worker-01f438cf-579f9fd987-5l657",
    "podCIDR": "10.244.1.0/24",
    "podCIDRs": [
      "10.244.1.0/24"
    ],
    "InternalIP": "10.0.4.73"
  }
]

4. We will replace the podCIDR by making changes directly in etcd:

./etcdhelper -cacert /etc/kubernetes/pki/etcd/ca.crt -cert /etc/kubernetes/pki/etcd/server.crt -key /etc/kubernetes/pki/etcd/server.key -endpoint https://127.0.0.1:2379 change-pod-cidr 10.55.0.0/16

5. Let's check that the podCIDR has indeed changed:

kubectl get no -o json | jq '[.items[] | {"name": .metadata.name, "podCIDR": .spec.podCIDR, "podCIDRs": .spec.podCIDRs, "InternalIP": (.status.addresses[] | select(.type == "InternalIP") | .address)}]'

[
  {
    "name": "kube-2-master",
    "podCIDR": "10.55.0.0/24",
    "podCIDRs": [
      "10.55.0.0/24"
    ],
    "InternalIP": "192.168.199.2"
  },
  {
    "name": "kube-2-master",
    "podCIDR": "10.55.0.0/24",
    "podCIDRs": [
      "10.55.0.0/24"
    ],
    "InternalIP": "10.0.1.239"
  },
  {
    "name": "kube-2-worker-01f438cf-579f9fd987-5l657",
    "podCIDR": "10.55.1.0/24",
    "podCIDRs": [
      "10.55.1.0/24"
    ],
    "InternalIP": "192.168.199.222"
  },
  {
    "name": "kube-2-worker-01f438cf-579f9fd987-5l657",
    "podCIDR": "10.55.1.0/24",
    "podCIDRs": [
      "10.55.1.0/24"
    ],
    "InternalIP": "10.0.4.73"
  }
]

6. We will reboot all nodes in the cluster one by one.

7. If at least one node retains the old podCIDR, then kube-controller-manager will fail to start and the pods in the cluster will not be scheduled.

In fact, the podCIDR can be changed more easily (for example, like this). But we wanted to learn how to work directly with etcd, because there are cases when editing Kubernetes objects in etcd is the only feasible option. (For example, you cannot simply change the field spec.clusterIP of a Service without downtime..)

Summary

This article discusses the possibility of working with data in etcd directly, i.e., bypassing the Kubernetes API. Sometimes this approach allows for "clever tricks". The operations presented in the text were tested on real K8s clusters. However, their readiness for widespread use is PoC (proof of concept). Therefore, if you want to use a modified version of the etcdhelper utility on your clusters, do so at your own risk.