ProHoster > Blog > Tsamaiso > Boiphihlelo ba rona ka data ho etcd Kubernetes cluster ka kotloloho (ntle le K8s API)

Boiphihlelo ba rona ka data ho etcd Kubernetes cluster ka kotloloho (ntle le K8s API)

Ka mokhoa o ntseng o eketseha, bareki ba re kopa ho fana ka phihlello ho sehlopha sa Kubernetes ho khona ho fumana lits'ebeletso ka har'a sehlopha: ho khona ho hokela ka kotloloho ho database kapa ts'ebeletso e itseng, ho hokahanya kopo ea lehae le lits'ebetso ka har'a sehlopha...

Boiphihlelo ba rona ka data ho etcd Kubernetes cluster ka kotloloho (ntle le K8s API)

Ka mohlala, ho na le tlhokahalo ea ho hokahanya ho tloha mochine oa sebaka sa hau ho ea ho tšebeletso memcached.staging.svc.cluster.local. Re fana ka bokhoni bona re sebelisa VPN ka har'a sehlopha seo moreki a hokelang ho sona. Ho etsa sena, re phatlalatsa li-subnets tsa li-pods, lits'ebeletso le ho sutumelletsa sehlopha sa DNS ho moreki. Kahoo, ha moreki a leka ho hokela tšebeletso memcached.staging.svc.cluster.local, kopo e ea ho sehlopha sa DNS 'me ka karabo e fumana aterese ea tšebeletso ena ho tsoa ho marang-rang a tšebeletso ea lihlopha kapa atereseng ea pod.

Re lokisa lihlopha tsa K8s re sebelisa kubeadm, moo subnet ea tšebeletso ea kamehla e leng teng 192.168.0.0/16, le marang-rang a li-pods ke 10.244.0.0/16. Hangata, ntho e 'ngoe le e' ngoe e sebetsa hantle, empa ho na le lintlha tse 'maloa:

  • Subnet 192.168.*.* hangata e sebelisoa ho marang-rang a liofisi tsa bareki, esita le hangata ho marang-rang a lehae a nts'etsopele. 'Me joale re fumana likhohlano: li-routers tsa lapeng li sebetsa ho subnet ena mme VPN e sutumelletsa li-subnets tsena ho tloha sehlopheng ho ea ho moreki.
  • Re na le lihlopha tse 'maloa (tlhahiso, sethala le/kapa lihlopha tse ngata tsa dev). Joale, ka ho sa feleng, kaofela ha bona ba tla ba le li-subnets tse tšoanang bakeng sa li-pods le litšebeletso, e leng se bakang mathata a maholo bakeng sa ho sebetsa ka nako e le 'ngoe le litšebeletso ka lihlopha tse' maloa.

Ke khale re amohetse tloaelo ea ho sebelisa li-subnet tse fapaneng bakeng sa lits'ebeletso le li-pods ka har'a morero o le mong - ka kakaretso, e le hore lihlopha tsohle li be le marang-rang a fapaneng. Leha ho le joalo, ho na le lihlopha tse ngata tse sebetsang tseo ke sa rateng ho li qeta ho tloha qalong, kaha li tsamaisa litšebeletso tse ngata, likopo tse hlakileng, joalo-joalo.

'Me joale re ile ra ipotsa: mokhoa oa ho fetola subnet ka sehlopha se teng?

Ho batla liqeto

Mokhoa o tloaelehileng haholo ke oa ho bopa bocha tsohle litšebeletso tse nang le mofuta oa ClusterIP. E le khetho, e ka eletsa le sena:

Mokhoa o latelang o na le bothata: ka mor'a hore ntho e 'ngoe le e' ngoe e lokisoe, li-pods li tla le IP ea khale e le DNS nameserver ho /etc/resolv.conf.
Kaha ke ne ke ntse ke sa fumane tharollo, ke ile ka tlameha ho hlophisa sehlopha sohle ka kubeadm reset ebe ke e qala hape.

Empa sena ha sea lokela motho e mong le e mong ... Mona ke li-introductions tse qaqileng bakeng sa nyeoe ea rona:

  • Flannel e sebelisoa;
  • Ho na le lihlopha ka bobeli marung le ho hardware;
  • Ke kopa ho qoba ho romela litšebeletso tsohle ka har'a sehlopha;
  • Ho na le tlhokahalo ea ho etsa ntho e 'ngoe le e' ngoe ka kakaretso ka palo e fokolang ea mathata;
  • Phetolelo ea Kubernetes ke 1.16.6 (leha ho le joalo, mehato e meng e tla tšoana le liphetolelo tse ling);
  • Mosebetsi o ka sehloohong ke ho netefatsa hore sehlopheng se sebelisoang ho sebelisoa kubeadm ka subnet ea litšebeletso 192.168.0.0/16, e nkele sebaka ka 172.24.0.0/16.

'Me ho ile ha etsahala hore ebe re ne re e-na le nako e telele re thahasella ho bona hore na ho na le eng ho Kubernetes e bolokiloeng joalo-joalo, ke eng e ka etsoang ka eona ... Kahoo re ile ra nahana: "Hobaneng o sa ntlafatse data ho etcd, o nkela liaterese tsa khale tsa IP (subnet) ka tse ncha? »

Ha re se re batlile lisebelisoa tse seng li entsoe tsa ho sebetsa le data ho etcd, ha rea ​​ka ra fumana letho le rarollotseng bothata ka botlalo. (Ka tsela, haeba u tseba ka lisebelisoa life kapa life tsa ho sebetsa ka data ka kotloloho ho etcd, re ka thabela lihokelo.) Leha ho le joalo, ntlha e ntle ea ho qala ke joalo-joalo ho tsoa ho OpenShift (ke leboha bangoli ba eona!).

Sesebelisoa sena se ka hokela ho etcd ka ho sebelisa litifikeiti le ho bala data ho tsoa moo ho sebelisa litaelo ls, get, dump.

Kenya joalo-joalo

Mohopolo o latelang oa utloahala: "Ke eng e u sitisang ho eketsa ts'ebeliso ee ka ho eketsa bokhoni ba ho ngola data ho etcd?"

E ile ea fetoha mofuta o fetotsoeng oa etcdhelper ka mesebetsi e 'meli e mecha changeServiceCIDR и changePodCIDR. ho yena o ka bona khoutu mona.

Likarolo tse ncha li etsa eng? Algorithm changeServiceCIDR:

  • etsa deserializer;
  • bokella polelo e tloaelehileng ho nkela CIDR sebaka;
  • re feta lits'ebeletso tsohle ka mofuta oa ClusterIP sehlopheng:
    • khetha boleng ho tsoa ho etcd ho ntho ea Go;
    • re sebelisa polelo e tloaelehileng re nkela li-byte tse peli tsa pele tsa aterese sebaka;
    • abela tšebeletso aterese ea IP ho tsoa subnet e ncha;
    • re theha serializer, re fetolela ntho ea Go ho protobuf, re ngola data e ncha ho joalo-joalo.

Mosebetsi changePodCIDR e tšoanang hantle changeServiceCIDR - feela sebakeng sa ho hlophisa litlhaloso tsa tšebeletso, re e etsetsa node le phetoho .spec.PodCIDR ho subnet e ncha.

Itloaetse

Fetola tšebeletso ea CIDR

Morero oa ho kenya ts'ebetsong mosebetsi o bonolo haholo, empa o kenyelletsa nako ea ho theoha nakong ea ho tsosolosoa ha li-pods tsohle ka har'a sehlopha. Ka mor'a ho hlalosa mehato e meholo, re tla boela re arolelane maikutlo a hore na, ka khopolo, nako ena ea ho theoha e ka fokotsoa joang.

Mehato ea ho itokisa:

  • ho kenya software e hlokahalang le ho bokella patched etcdhelper;
  • bekapo etcd le /etc/kubernetes.

Moralo o khuts'oane oa ts'ebetso ea ho fetola litšebeletsoCIDR:

  • ho fetola li-apiserver le li-controller-manager li bonts'a;
  • ho ntšoa bocha ha litifikeiti;
  • ho fetola litšebeletso tsa ClusterIP ho joalo-joalo;
  • ho qala bocha ha li-pods tsohle sehlopheng.

Se latelang ke tatellano e felletseng ea liketso ka botlalo.

1. Kenya etcd-client bakeng sa ho lahla data:

apt install etcd-client

2. Haha etcdhelper:

  • Kenya golang: 
    GOPATH=/root/golang
mkdir -p $GOPATH/local
curl -sSL https://dl.google.com/go/go1.14.1.linux-amd64.tar.gz | tar -xzvC $GOPATH/local
echo "export GOPATH="$GOPATH"" >> ~/.bashrc
echo 'export GOROOT="$GOPATH/local/go"' >> ~/.bashrc
echo 'export PATH="$PATH:$GOPATH/local/go/bin"' >> ~/.bashrc
  • Re ipolokela rona etcdhelper.go, download itšetlehile ka, bokella: 
    wget https://raw.githubusercontent.com/flant/examples/master/2020/04-etcdhelper/etcdhelper.go
go get go.etcd.io/etcd/clientv3 k8s.io/kubectl/pkg/scheme k8s.io/apimachinery/pkg/runtime
go build -o etcdhelper etcdhelper.go

3. Etsa bekapo etcd:

backup_dir=/root/backup
mkdir ${backup_dir}
cp -rL /etc/kubernetes ${backup_dir}
ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt --key=/etc/kubernetes/pki/etcd/server.key --cert=/etc/kubernetes/pki/etcd/server.crt --endpoints https://192.168.199.100:2379 snapshot save ${backup_dir}/etcd.snapshot

4. Fetola subnet ea tšebeletso ka sefofaneng sa taolo sa Kubernetes se bonahatsa. Lifaeleng /etc/kubernetes/manifests/kube-apiserver.yaml и /etc/kubernetes/manifests/kube-controller-manager.yaml fetola parameter --service-cluster-ip-range ho subnet e ncha: 172.24.0.0/16 sebakeng sa 192.168.0.0/16.

5. Kaha re ntse re fetola subnet ea lits'ebeletso eo kubeadm e fanang ka litifikeiti tsa apiserver (ho kenyeletsoa), li hloka ho hlahisoa hape:

  1. Ha re boneng hore na setifikeiti sa hajoale se fuoe libaka life le liaterese tsa IP: 
    openssl x509 -noout -ext subjectAltName </etc/kubernetes/pki/apiserver.crt
X509v3 Subject Alternative Name:
    DNS:dev-1-master, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, DNS:apiserver, IP Address:192.168.0.1, IP Address:10.0.0.163, IP Address:192.168.199.100
  2. Ha re lokisetseng tlhophiso e nyane bakeng sa kubeadm: 
    cat kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
networking:
  podSubnet: "10.244.0.0/16"
  serviceSubnet: "172.24.0.0/16"
apiServer:
  certSANs:
  - "192.168.199.100" # IP-адрес мастер узла
  3. Ha re hlakole crt le senotlolo sa khale, kaha ntle le sena setifikeiti se secha se ke ke sa ntšoa: 
    rm /etc/kubernetes/pki/apiserver.{key,crt}
  4. Ha re ntše setifikeiti hape bakeng sa seva sa API: 
    kubeadm init phase certs apiserver --config=kubeadm-config.yaml
  5. Ha re hlahlobeng hore na setifikeiti se fanoe bakeng sa subnet e ncha: 
    openssl x509 -noout -ext subjectAltName </etc/kubernetes/pki/apiserver.crt
X509v3 Subject Alternative Name:
    DNS:kube-2-master, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, IP Address:172.24.0.1, IP Address:10.0.0.163, IP Address:192.168.199.100
  6. Kamora ho fana ka setifikeiti sa seva sa API hape, qala setshelo sa eona hape: 
    docker ps | grep k8s_kube-apiserver | awk '{print $1}' | xargs docker restart
  7. Ha re nchafatse config bakeng sa admin.conf: 
    kubeadm alpha certs renew admin.conf
  8. Ha re fetole data ho etcd: 
    ./etcdhelper -cacert /etc/kubernetes/pki/etcd/ca.crt -cert /etc/kubernetes/pki/etcd/server.crt -key /etc/kubernetes/pki/etcd/server.key -endpoint https://127.0.0.1:2379 change-service-cidr 172.24.0.0/16

    Ela hloko ka kopo! Ka nako ena, qeto ea domain e emisa ho sebetsa sehlopheng, kaha ho li-pods tse teng /etc/resolv.conf aterese ea khale ea CoreDNS (kube-dns) e ngolisitsoe, 'me kube-proxy e fetola melao ea iptables ho tloha subnet ea khale ho ea ho e ncha. Ho feta moo sehloohong se ngotsoeng ka mekhoa e ka khonehang ea ho fokotsa nako ea ho theoha.

  9. Ha re lokise ConfigMap sebakeng sa mabitso kube-system: 
    kubectl -n kube-system edit cm kubelet-config-1.16

    - nka sebaka mona clusterDNS ho aterese e ncha ea IP ea tšebeletso ea kube-dns: kubectl -n kube-system get svc kube-dns.

    kubectl -n kube-system edit cm kubeadm-config

    - re tla e lokisa data.ClusterConfiguration.networking.serviceSubnet ho subnet e ncha.

  10. Kaha aterese ea kube-dns e fetohile, hoa hlokahala hore u ntlafatse tlhophiso ea kubelet libakeng tsohle: 
    kubeadm upgrade node phase kubelet-config && systemctl restart kubelet
  11. Sohle se setseng ke ho qala li-pods tsohle ka har'a sehlopha: 
    kubectl get pods --no-headers=true --all-namespaces |sed -r 's/(S+)s+(S+).*/kubectl --namespace 1 delete pod 2/e'

Fokotsa nako ea ho theoha

Mehopolo ea ho fokotsa nako ea ho fokotsa nako:

  1. Kamora ho fetola ponahalo ea sefofane sa taolo, theha tšebeletso e ncha ea kube-dns, mohlala, ka lebitso kube-dns-tmp le aterese e ncha 172.24.0.10.
  2. Ho etsa if ho etcdhelper, e ke keng ea fetola ts'ebeletso ea kube-dns.
  3. Tlosa aterese ho li-kubelets tsohle ClusterDNS ho e ncha, ha tšebeletso ea khale e tla tsoela pele ho sebetsa ka nako e le ’ngoe le e ncha.
  4. Ema ho fihlela li-pods tse nang le lits'ebetso li pitika ka botsona ka mabaka a tlhaho kapa ka nako eo ho lumellanoeng ka eona.
  5. Hlakola tšebeletso kube-dns-tmp le phetoho serviceSubnetCIDR bakeng sa ts'ebeletso ea kube-dns.

Morero ona o tla u lumella ho fokotsa nako ea ho theoha ho ~ motsotso - bakeng sa nako ea ho tlosoa ha tšebeletso kube-dns-tmp le ho fetola subnet bakeng sa tshebeletso kube-dns.

Phetoho podNetwork

Ka nako e ts'oanang, re ile ra etsa qeto ea ho sheba mokhoa oa ho fetola podNetwork ho sebelisa sephetho sa etcdhelper. Tatelano ea liketso ke e latelang:

  • ho lokisa li-configs kube-system;
  • ho lokisa lebe-controller-manager manifest;
  • fetola podCIDR ka kotloloho ho etcd;
  • qala bocha li-node tsohle tsa cluster.

Joale ho eketsehileng ka liketso tsena:

1. Fetola ConfigMap sebakeng sa mabitso kube-system:

kubectl -n kube-system edit cm kubeadm-config

- ho lokisa data.ClusterConfiguration.networking.podSubnet ho subnet e ncha 10.55.0.0/16.

kubectl -n kube-system edit cm kube-proxy

- ho lokisa data.config.conf.clusterCIDR: 10.55.0.0/16.

2. Fetola ponahatso ea molaoli-molaoli:

vim /etc/kubernetes/manifests/kube-controller-manager.yaml

- ho lokisa --cluster-cidr=10.55.0.0/16.

3. Sheba litekanyetso tsa hona joale .spec.podCIDR, .spec.podCIDRs, .InternalIP, .status.addresses bakeng sa li-cluster node tsohle:

kubectl get no -o json | jq '[.items[] | {"name": .metadata.name, "podCIDR": .spec.podCIDR, "podCIDRs": .spec.podCIDRs, "InternalIP": (.status.addresses[] | select(.type == "InternalIP") | .address)}]'

[
  {
    "name": "kube-2-master",
    "podCIDR": "10.244.0.0/24",
    "podCIDRs": [
      "10.244.0.0/24"
    ],
    "InternalIP": "192.168.199.2"
  },
  {
    "name": "kube-2-master",
    "podCIDR": "10.244.0.0/24",
    "podCIDRs": [
      "10.244.0.0/24"
    ],
    "InternalIP": "10.0.1.239"
  },
  {
    "name": "kube-2-worker-01f438cf-579f9fd987-5l657",
    "podCIDR": "10.244.1.0/24",
    "podCIDRs": [
      "10.244.1.0/24"
    ],
    "InternalIP": "192.168.199.222"
  },
  {
    "name": "kube-2-worker-01f438cf-579f9fd987-5l657",
    "podCIDR": "10.244.1.0/24",
    "podCIDRs": [
      "10.244.1.0/24"
    ],
    "InternalIP": "10.0.4.73"
  }
]

4. Kenya sebaka sa podCIDR ka ho etsa liphetoho ka kotloloho ho etcd:

./etcdhelper -cacert /etc/kubernetes/pki/etcd/ca.crt -cert /etc/kubernetes/pki/etcd/server.crt -key /etc/kubernetes/pki/etcd/server.key -endpoint https://127.0.0.1:2379 change-pod-cidr 10.55.0.0/16

5. Ha re hlahlobeng hore na podCIDR e hlile e fetohile.

kubectl get no -o json | jq '[.items[] | {"name": .metadata.name, "podCIDR": .spec.podCIDR, "podCIDRs": .spec.podCIDRs, "InternalIP": (.status.addresses[] | select(.type == "InternalIP") | .address)}]'

[
  {
    "name": "kube-2-master",
    "podCIDR": "10.55.0.0/24",
    "podCIDRs": [
      "10.55.0.0/24"
    ],
    "InternalIP": "192.168.199.2"
  },
  {
    "name": "kube-2-master",
    "podCIDR": "10.55.0.0/24",
    "podCIDRs": [
      "10.55.0.0/24"
    ],
    "InternalIP": "10.0.1.239"
  },
  {
    "name": "kube-2-worker-01f438cf-579f9fd987-5l657",
    "podCIDR": "10.55.1.0/24",
    "podCIDRs": [
      "10.55.1.0/24"
    ],
    "InternalIP": "192.168.199.222"
  },
  {
    "name": "kube-2-worker-01f438cf-579f9fd987-5l657",
    "podCIDR": "10.55.1.0/24",
    "podCIDRs": [
      "10.55.1.0/24"
    ],
    "InternalIP": "10.0.4.73"
  }
]

6. Ha re qaleng li-node tsohle tsa cluster ka bonngoe.

7. Haeba u siea bonyane node e le 'ngoe PodCIDR ea khale, joale kube-controller-manager e ke ke ea khona ho qala, 'me li-pods ka har'a sehlopha li ke ke tsa hlophisoa.

Ebile, ho fetola podCIDR ho ka etsoa habonolo le ho feta (mohlala, kahoo). Empa re ne re batla ho ithuta ho sebetsa le etcd ka kotloloho, hobane ho na le maemo ha u hlophisa lintho tsa Kubernetes ho etcd - eona feela phapang e ka khonehang. (Ka mohlala, u ke ke ua fetola tšimo ea Tšebeletso ntle le nako ea ho phomola spec.clusterIP.)

Phello

Sehlooho se bua ka monyetla oa ho sebetsa le data ho etcd ka ho toba, ke hore. ho feta Kubernetes API. Ka linako tse ling mokhoa ona o u lumella ho etsa "lintho tse qhekellang". Re lekile ts'ebetso e fanoeng sengolong ho lihlopha tsa 'nete tsa K8s. Leha ho le joalo, boemo ba bona ba ho itokisetsa ho sebelisoa hohle ho PoC (bopaki ba mohopolo). Ka hona, haeba u batla ho sebelisa mofuta o fetotsoeng oa lisebelisoa tsa etcdhelper lihlopha tsa hau, etsa joalo ka kotsi ea hau.

PES

Bala hape ho blog ea rona:

Source: www.habr.com

Eketsa ka tlhaloso