ProHoster > Блог > Isakoso > Iriri wa pẹlu data ni etcd Kubernetes iṣupọ taara (laisi K8s API)

Iriri wa pẹlu data ni etcd Kubernetes iṣupọ taara (laisi K8s API)

Npọ sii, awọn alabara n beere lọwọ wa lati pese iraye si iṣupọ Kubernetes lati ni anfani lati wọle si awọn iṣẹ laarin iṣupọ: ki wọn le sopọ taara si data data tabi iṣẹ kan, lati so ohun elo agbegbe kan pọ pẹlu awọn ohun elo laarin iṣupọ…

Iriri wa pẹlu data ni etcd Kubernetes iṣupọ taara (laisi K8s API)

Fun apẹẹrẹ, iwulo wa lati sopọ lati ẹrọ agbegbe rẹ si iṣẹ kan memcached.staging.svc.cluster.local. A pese agbara yii nipa lilo VPN laarin iṣupọ eyiti alabara sopọ si. Lati ṣe eyi, a kede awọn subnets ti pods, awọn iṣẹ ati titari iṣupọ DNS si alabara. Nitorinaa, nigbati alabara kan gbiyanju lati sopọ si iṣẹ naa memcached.staging.svc.cluster.local, ibeere naa lọ si iṣupọ DNS ati ni idahun gba adirẹsi iṣẹ yii lati inu nẹtiwọki iṣẹ iṣupọ tabi adirẹsi adarọ-ese.

A tunto awọn iṣupọ K8s nipa lilo kubeadm, nibiti subnet iṣẹ aiyipada jẹ 192.168.0.0/16, ati awọn nẹtiwọki ti pods ni 10.244.0.0/16. Nigbagbogbo ohun gbogbo ṣiṣẹ daradara, ṣugbọn awọn aaye meji wa:

  • Subnet 192.168.*.* nigbagbogbo lo ninu awọn nẹtiwọọki ọfiisi alabara, ati paapaa diẹ sii nigbagbogbo ni awọn nẹtiwọọki ile idagbasoke. Ati lẹhinna a pari pẹlu awọn ija: awọn olulana ile ṣiṣẹ lori subnet yii ati VPN titari awọn subnet wọnyi lati iṣupọ si alabara.
  • A ni ọpọlọpọ awọn iṣupọ (igbejade, ipele ati/tabi ọpọlọpọ awọn iṣupọ dev). Lẹhinna, nipasẹ aiyipada, gbogbo wọn yoo ni awọn subnets kanna fun awọn adarọ-ese ati awọn iṣẹ, eyiti o ṣẹda awọn iṣoro nla fun iṣẹ nigbakanna pẹlu awọn iṣẹ ni ọpọlọpọ awọn iṣupọ.

A ti gba aṣa ti lilo oriṣiriṣi awọn subnets fun awọn iṣẹ ati awọn adarọ-ese laarin iṣẹ akanṣe kanna - ni gbogbogbo, ki gbogbo awọn iṣupọ ni awọn nẹtiwọọki oriṣiriṣi. Sibẹsibẹ, nọmba nla ti awọn iṣupọ wa ni iṣẹ ti Emi kii yoo fẹ lati yiyi pada lati ibere, nitori wọn nṣiṣẹ ọpọlọpọ awọn iṣẹ, awọn ohun elo ipinlẹ, ati bẹbẹ lọ.

Ati lẹhinna a beere lọwọ ara wa: bawo ni a ṣe le yi subnet pada ninu iṣupọ ti o wa tẹlẹ?

Wiwa ti awọn ipinnu

Iwa ti o wọpọ julọ ni lati tun ṣe gbogbo awọn iṣẹ pẹlu iru ClusterIP. Bi aṣayan, le ni imọran ati eyi:

Ilana atẹle naa ni iṣoro: lẹhin ohun gbogbo ti tunto, awọn adarọ-ese wa pẹlu IP atijọ bi olupin orukọ DNS ni /etc/resolv.conf.
Niwọn igba ti Emi ko tun rii ojutu naa, Mo ni lati tun gbogbo iṣupọ naa tunto pẹlu kubeadm atunkọ ati tun bẹrẹ lẹẹkansi.

Ṣugbọn eyi ko dara fun gbogbo eniyan… Eyi ni awọn ifihan alaye diẹ sii fun ọran wa:

  • Flannel ti lo;
  • Awọn iṣupọ wa mejeeji ninu awọn awọsanma ati lori ohun elo;
  • Emi yoo fẹ lati yago fun tun-ṣiṣẹ gbogbo awọn iṣẹ inu iṣupọ;
  • O nilo lati ṣe ohun gbogbo pẹlu nọmba ti o kere julọ ti awọn iṣoro;
  • Ẹya Kubernetes jẹ 1.16.6 (sibẹsibẹ, awọn igbesẹ siwaju yoo jẹ iru fun awọn ẹya miiran);
  • Iṣẹ akọkọ ni lati rii daju pe ninu iṣupọ ti a fi ranṣẹ nipa lilo kubeadm pẹlu subnet iṣẹ kan 192.168.0.0/16, rọpo rẹ pẹlu 172.24.0.0/16.

Ati pe o ṣẹlẹ pe a ti nifẹ lati rii kini ati bii o ṣe fipamọ Kubernetes ni etcd, kini a le ṣe pẹlu rẹ… Nitorina a ronu: “Kilode ti kii ṣe imudojuiwọn data nikan ni etcd, rọpo awọn adirẹsi IP atijọ (subnet) pẹlu awọn tuntun? »

Lẹhin wiwa awọn irinṣẹ ti a ti ṣetan fun ṣiṣẹ pẹlu data ni etcd, a ko rii ohunkohun ti o yanju iṣoro naa patapata. (Ni ọna, ti o ba mọ nipa eyikeyi awọn ohun elo fun ṣiṣẹ pẹlu data taara ni etcd, a yoo ni riri fun awọn ọna asopọ.) Sibẹsibẹ, ibẹrẹ ti o dara ni etcdhhelper lati OpenShift (O ṣeun si awọn onkọwe rẹ!).

IwUlO yii le sopọ si etcd nipa lilo awọn iwe-ẹri ati ka data lati ibẹ nipa lilo awọn aṣẹ ls, get, dump.

Fi etcdhhelper kun

Ero ti o tẹle jẹ ọgbọn: “Kini o ṣe idiwọ fun ọ lati ṣafikun ohun elo yii nipa fifi agbara lati kọ data si ati bẹbẹ lọ?”

O di ẹya ti a tunṣe ti etcdhhelper pẹlu awọn iṣẹ tuntun meji changeServiceCIDR и changePodCIDR. lori re o le wo koodu naa nibi.

Kini awọn ẹya tuntun ṣe? Algoridimu changeServiceCIDR:

  • ṣẹda deserializer;
  • ṣajọ ikosile deede lati rọpo CIDR;
  • a lọ nipasẹ gbogbo awọn iṣẹ pẹlu iru ClusterIP ninu iṣupọ:
    • pinnu iye lati etcd sinu ohun Go;
    • lilo ikosile deede a rọpo awọn baiti meji akọkọ ti adirẹsi naa;
    • fi adiresi IP kan fun iṣẹ naa lati inu subnet tuntun;
    • ṣẹda a serializer, iyipada awọn Go ohun sinu protobuf, kọ titun data to ati be be lo.

Išẹ changePodCIDR pataki iru changeServiceCIDR - nikan dipo ti satunkọ awọn sipesifikesonu iṣẹ, a se o fun ipade ati ayipada .spec.PodCIDR si titun kan subnet.

Ṣaṣeṣe

Yi iṣẹ CID pada

Eto fun imuse iṣẹ naa rọrun pupọ, ṣugbọn o kan akoko isunmi lakoko ti gbogbo awọn adarọ-ese ti o wa ninu iṣupọ naa tun ṣe. Lẹhin ti n ṣalaye awọn igbesẹ akọkọ, a yoo tun pin awọn ero lori bii, ni imọ-jinlẹ, akoko idinku le dinku.

Awọn igbesẹ igbaradi:

  • fifi sọfitiwia pataki ati apejọ patched etcdhelper;
  • afẹyinti etcd ati /etc/kubernetes.

Eto iṣe kukuru fun iyipada iṣẹCIDR:

  • iyipada apiserver ati oludari-oluṣakoso farahan;
  • reissue ti awọn iwe-ẹri;
  • iyipada awọn iṣẹ ClusterIP ni etcd;
  • tun bẹrẹ gbogbo awọn podu inu iṣupọ.

Atẹle ni pipe ti awọn iṣe ni kikun.

1. Fi sori ẹrọ etcd-onibara fun idalẹnu data:

apt install etcd-client

2. Kọ etcdhhelper:

  • Fi sori ẹrọ golang: 
    GOPATH=/root/golang
mkdir -p $GOPATH/local
curl -sSL https://dl.google.com/go/go1.14.1.linux-amd64.tar.gz | tar -xzvC $GOPATH/local
echo "export GOPATH="$GOPATH"" >> ~/.bashrc
echo 'export GOROOT="$GOPATH/local/go"' >> ~/.bashrc
echo 'export PATH="$PATH:$GOPATH/local/go/bin"' >> ~/.bashrc
  • A fipamọ fun ara wa etcdhelper.go, ṣe igbasilẹ awọn igbẹkẹle, gba: 
    wget https://raw.githubusercontent.com/flant/examples/master/2020/04-etcdhelper/etcdhelper.go
go get go.etcd.io/etcd/clientv3 k8s.io/kubectl/pkg/scheme k8s.io/apimachinery/pkg/runtime
go build -o etcdhelper etcdhelper.go

3. Ṣe afẹyinti etcd:

backup_dir=/root/backup
mkdir ${backup_dir}
cp -rL /etc/kubernetes ${backup_dir}
ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt --key=/etc/kubernetes/pki/etcd/server.key --cert=/etc/kubernetes/pki/etcd/server.crt --endpoints https://192.168.199.100:2379 snapshot save ${backup_dir}/etcd.snapshot

4. Yi subnet iṣẹ pada ni Kubernetes iṣakoso ofurufu farahan. Ninu awọn faili /etc/kubernetes/manifests/kube-apiserver.yaml и /etc/kubernetes/manifests/kube-controller-manager.yaml yi paramita --service-cluster-ip-range si subnet tuntun kan: 172.24.0.0/16 dipo 192.168.0.0/16.

5. Niwọn bi a ti n yi subnet iṣẹ pada si eyiti kubeadm fun awọn iwe-ẹri fun apiserver (pẹlu), wọn nilo lati tun gbejade:

  1. Jẹ ki a wo iru awọn ibugbe ati awọn adirẹsi IP ti ijẹrisi lọwọlọwọ ti fun: 
    openssl x509 -noout -ext subjectAltName </etc/kubernetes/pki/apiserver.crt
X509v3 Subject Alternative Name:
    DNS:dev-1-master, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, DNS:apiserver, IP Address:192.168.0.1, IP Address:10.0.0.163, IP Address:192.168.199.100
  2. Jẹ ki a mura atunto pọọku fun kubeadm: 
    cat kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
networking:
  podSubnet: "10.244.0.0/16"
  serviceSubnet: "172.24.0.0/16"
apiServer:
  certSANs:
  - "192.168.199.100" # IP-адрес мастер узла
  3. Jẹ ki a pa crt ati bọtini atijọ rẹ, nitori laisi eyi ijẹrisi tuntun kii yoo funni: 
    rm /etc/kubernetes/pki/apiserver.{key,crt}
  4. Jẹ ki a tun awọn iwe-ẹri fun olupin API naa jade: 
    kubeadm init phase certs apiserver --config=kubeadm-config.yaml
  5. Jẹ ki a ṣayẹwo pe a ti fun ijẹrisi naa fun subnet tuntun: 
    openssl x509 -noout -ext subjectAltName </etc/kubernetes/pki/apiserver.crt
X509v3 Subject Alternative Name:
    DNS:kube-2-master, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, IP Address:172.24.0.1, IP Address:10.0.0.163, IP Address:192.168.199.100
  6. Lẹhin ti tun-iṣẹjade ijẹrisi olupin API, tun eiyan rẹ bẹrẹ: 
    docker ps | grep k8s_kube-apiserver | awk '{print $1}' | xargs docker restart
  7. Jẹ ká regenerate awọn konfigi fun admin.conf: 
    kubeadm alpha certs renew admin.conf
  8. Jẹ ki a ṣatunkọ data ni etcd: 
    ./etcdhelper -cacert /etc/kubernetes/pki/etcd/ca.crt -cert /etc/kubernetes/pki/etcd/server.crt -key /etc/kubernetes/pki/etcd/server.key -endpoint https://127.0.0.1:2379 change-service-cidr 172.24.0.0/16

    Išọra Ni akoko yii, ipinnu agbegbe duro ṣiṣẹ ninu iṣupọ, nitori ninu awọn adarọ-ese ti o wa ninu /etc/resolv.conf Adirẹsi CoreDNS atijọ (kube-dns) ti forukọsilẹ, ati kube-aṣoju yi awọn ofin iptables pada lati inu subnet atijọ si tuntun. Siwaju sii ninu nkan naa o ti kọ nipa awọn aṣayan ti o ṣeeṣe lati dinku akoko isinmi.

  9. Jẹ ki a ṣatunṣe ConfigMap ni aaye orukọ kube-system: 
    kubectl -n kube-system edit cm kubelet-config-1.16

    - ropo nibi clusterDNS si adiresi IP tuntun ti iṣẹ kube-dns: kubectl -n kube-system get svc kube-dns.

    kubectl -n kube-system edit cm kubeadm-config

    - a yoo ṣe atunṣe data.ClusterConfiguration.networking.serviceSubnet si titun kan subnet.

  10. Niwọn igba ti adirẹsi kube-dns ti yipada, o jẹ dandan lati ṣe imudojuiwọn atunto kubelet lori gbogbo awọn apa: 
    kubeadm upgrade node phase kubelet-config && systemctl restart kubelet
  11. Gbogbo ohun ti o ku ni lati tun bẹrẹ gbogbo awọn podu inu iṣupọ: 
    kubectl get pods --no-headers=true --all-namespaces |sed -r 's/(S+)s+(S+).*/kubectl --namespace 1 delete pod 2/e'

Din akoko idaduro

Awọn ero lori bi o ṣe le dinku akoko idaduro:

  1. Lẹhin iyipada awọn ifihan ofurufu iṣakoso, ṣẹda iṣẹ kube-dns tuntun, fun apẹẹrẹ, pẹlu orukọ kube-dns-tmp ati titun adirẹsi 172.24.0.10.
  2. Ṣe if ni etcdhelper, eyiti kii yoo ṣe atunṣe iṣẹ kube-dns.
  3. Rọpo adirẹsi ni gbogbo kubelets ClusterDNS si titun kan, nigba ti atijọ iṣẹ yoo tesiwaju lati ṣiṣẹ ni nigbakannaa pẹlu awọn titun kan.
  4. Duro titi awọn adarọ-ese pẹlu awọn ohun elo yipo boya nipasẹ ara wọn fun awọn idi adayeba tabi ni akoko adehun.
  5. Paarẹ iṣẹ kube-dns-tmp ati iyipada serviceSubnetCIDR fun kube-dns iṣẹ.

Eto yii yoo gba ọ laaye lati dinku akoko isinmi si ~ iṣẹju kan - fun iye akoko yiyọ iṣẹ naa kube-dns-tmp ati iyipada subnet fun iṣẹ naa kube-dns.

Ayipada podNetwork

Ni akoko kanna, a pinnu lati wo bi o ṣe le yipada podNetwork ni lilo abajade etcdhelper. Ilana ti awọn iṣe jẹ bi atẹle:

  • ojoro configs ni kube-system;
  • ojoro awọn kube-iṣakoso-oluṣakoso farahan;
  • yi podCIDR taara ni etcd;
  • atunbere gbogbo awọn apa iṣupọ.

Bayi diẹ sii nipa awọn iṣe wọnyi:

1. Ṣatunṣe ConfigMaps ni aaye orukọ kube-system:

kubectl -n kube-system edit cm kubeadm-config

- atunse data.ClusterConfiguration.networking.podSubnet si titun kan subnet 10.55.0.0/16.

kubectl -n kube-system edit cm kube-proxy

- atunse data.config.conf.clusterCIDR: 10.55.0.0/16.

2. Ṣatunṣe ifihan oludari-oluṣakoso:

vim /etc/kubernetes/manifests/kube-controller-manager.yaml

- atunse --cluster-cidr=10.55.0.0/16.

3. Wo awọn iye ti o wa lọwọlọwọ .spec.podCIDR, .spec.podCIDRs, .InternalIP, .status.addresses fun gbogbo awọn apa iṣupọ:

kubectl get no -o json | jq '[.items[] | {"name": .metadata.name, "podCIDR": .spec.podCIDR, "podCIDRs": .spec.podCIDRs, "InternalIP": (.status.addresses[] | select(.type == "InternalIP") | .address)}]'

[
  {
    "name": "kube-2-master",
    "podCIDR": "10.244.0.0/24",
    "podCIDRs": [
      "10.244.0.0/24"
    ],
    "InternalIP": "192.168.199.2"
  },
  {
    "name": "kube-2-master",
    "podCIDR": "10.244.0.0/24",
    "podCIDRs": [
      "10.244.0.0/24"
    ],
    "InternalIP": "10.0.1.239"
  },
  {
    "name": "kube-2-worker-01f438cf-579f9fd987-5l657",
    "podCIDR": "10.244.1.0/24",
    "podCIDRs": [
      "10.244.1.0/24"
    ],
    "InternalIP": "192.168.199.222"
  },
  {
    "name": "kube-2-worker-01f438cf-579f9fd987-5l657",
    "podCIDR": "10.244.1.0/24",
    "podCIDRs": [
      "10.244.1.0/24"
    ],
    "InternalIP": "10.0.4.73"
  }
]

4. Rọpo podCIDR nipa ṣiṣe awọn ayipada taara si etcd:

./etcdhelper -cacert /etc/kubernetes/pki/etcd/ca.crt -cert /etc/kubernetes/pki/etcd/server.crt -key /etc/kubernetes/pki/etcd/server.key -endpoint https://127.0.0.1:2379 change-pod-cidr 10.55.0.0/16

5. Jẹ ki a ṣayẹwo pe podCIDR ti yipada gaan:

kubectl get no -o json | jq '[.items[] | {"name": .metadata.name, "podCIDR": .spec.podCIDR, "podCIDRs": .spec.podCIDRs, "InternalIP": (.status.addresses[] | select(.type == "InternalIP") | .address)}]'

[
  {
    "name": "kube-2-master",
    "podCIDR": "10.55.0.0/24",
    "podCIDRs": [
      "10.55.0.0/24"
    ],
    "InternalIP": "192.168.199.2"
  },
  {
    "name": "kube-2-master",
    "podCIDR": "10.55.0.0/24",
    "podCIDRs": [
      "10.55.0.0/24"
    ],
    "InternalIP": "10.0.1.239"
  },
  {
    "name": "kube-2-worker-01f438cf-579f9fd987-5l657",
    "podCIDR": "10.55.1.0/24",
    "podCIDRs": [
      "10.55.1.0/24"
    ],
    "InternalIP": "192.168.199.222"
  },
  {
    "name": "kube-2-worker-01f438cf-579f9fd987-5l657",
    "podCIDR": "10.55.1.0/24",
    "podCIDRs": [
      "10.55.1.0/24"
    ],
    "InternalIP": "10.0.4.73"
  }
]

6. Jẹ ki a tun atunbere gbogbo awọn apa iṣupọ ọkan nipasẹ ọkan.

7. Ti o ba lọ kuro ni o kere kan ipade podCIDR atijọ, lẹhinna kube-controller-manager kii yoo ni anfani lati bẹrẹ, ati pe awọn pods ninu iṣupọ ko ni ṣeto.

Ni otitọ, iyipada podCIDR le ṣee ṣe paapaa rọrun (fun apẹẹrẹ, bẹ). Ṣugbọn a fẹ lati kọ ẹkọ bi a ṣe le ṣiṣẹ pẹlu etcd taara, nitori pe awọn ọran wa nigba ṣiṣatunṣe awọn nkan Kubernetes ni bbl - awọn nikan ṣee ṣe iyatọ. (Fun apẹẹrẹ, o ko le yipada aaye Iṣẹ nikan laisi akoko idaduro spec.clusterIP.)

Abajade

Nkan naa jiroro lori iṣeeṣe ti ṣiṣẹ pẹlu data ni etcd taara, i.e. fori Kubernetes API. Nigba miiran ọna yii gba ọ laaye lati ṣe "awọn ohun ti o ni ẹtan". A ṣe idanwo awọn iṣẹ ṣiṣe ti a fun ni ọrọ lori awọn iṣupọ K8s gidi. Sibẹsibẹ, ipo imurasilẹ wọn fun lilo ni ibigbogbo jẹ PoC (ẹri ti imọran). Nitorinaa, ti o ba fẹ lo ẹya ti a tunṣe ti ohun elo etcdhelper lori awọn iṣupọ rẹ, ṣe bẹ ninu ewu tirẹ.

PS

Ka tun lori bulọọgi wa:

orisun: www.habr.com

Fi ọrọìwòye kun