I-Calico yokuxhumana e-Kubernetes: isingeniso nolwazi oluncane

Inhloso yalesi sihloko ukwethula umfundi ezintweni eziyisisekelo zokuxhumana nokuphatha izinqubomgomo zenethiwekhi ku-Kubernetes, kanye ne-plugin yenkampani yangaphandle ye-Calico enweba amakhono ajwayelekile. Ngokuhamba kwesikhathi, ukucushwa okulula nezinye izici kuzoboniswa kusetshenziswa izibonelo zangempela ezivela kokuhlangenwe nakho kwethu kokusebenza.

Isingeniso esisheshayo ku-Kubernetes networking appliance

Iqoqo le-Kubernetes alinakucatshangwa ngaphandle kwenethiwekhi. Sesivele sishicilele izinto ezisetshenziswayo ezisekelweni zazo: “Umhlahlandlela onemifanekiso wokuxhumana e-Kubernetes"Futhi"Isingeniso se-Kubernetes Network Policies for Security Professionals".

Kumongo walesi sihloko, kubalulekile ukuqaphela ukuthi i-K8s ngokwayo ayinasibopho sokuxhumana kwenethiwekhi phakathi kweziqukathi namanodi: kulokhu, okuhlukahlukene. Ama-plugins we-CNI (Isixhumi Esibonakalayo Sokuxhumana Kweziqukathi). Okuningi mayelana nalo mqondo thina babuye bangitshele.

Isibonelo, okuvame kakhulu kulawa ma-plugin Flannel - inikeza ukuxhumana okugcwele kwenethiwekhi phakathi kwawo wonke ama-cluster node ngokuphakamisa amabhuloho endaweni ngayinye, ukwabela i-subnet kuyo. Nokho, ukufinyeleleka okuphelele nokungalawulwa akuzuzisi ngaso sonke isikhathi. Ukuze unikeze uhlobo oluthile lokuhlukaniswa okuncane kuqoqo, kuyadingeka ukungenelela ekucushweni kwe-firewall. Esimweni esivamile, ibekwe ngaphansi kokulawulwa kwe-CNI efanayo, yingakho noma yikuphi ukungenelela komuntu wesithathu kuma-iptables kungahunyushwa ngokungalungile noma kunganakwa ngokuphelele.

Futhi "ngaphandle kwebhokisi" lokuhlela ukuphathwa kwenqubomgomo yenethiwekhi kuqoqo le-Kubernetes kunikezwa I-NetworkPolicy API. Lesi sisetshenziswa, esatshalaliswa ezindaweni ezikhethiwe, singaqukatha imithetho yokuhlukanisa ukufinyelela kusuka kolunye uhlelo kuya kolunye. Iphinde ikuvumela ukuthi ulungiselele ukufinyeleleka phakathi kwama-pod athile, indawo (izikhala zamagama) noma amabhlogo wamakheli e-IP:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: test-network-policy
  namespace: default
spec:
  podSelector:
    matchLabels:
      role: db
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - ipBlock:
        cidr: 172.17.0.0/16
        except:
        - 172.17.1.0/24
    - namespaceSelector:
        matchLabels:
          project: myproject
    - podSelector:
        matchLabels:
          role: frontend
    ports:
    - protocol: TCP
      port: 6379
  egress:
  - to:
    - ipBlock:
        cidr: 10.0.0.0/24
    ports:
    - protocol: TCP
      port: 5978

Lesi akusona isibonelo sakudala kakhulu imibhalo esemthethweni ingase isithene amandla unomphela isifiso sokuqonda umqondo wokuthi izinqubomgomo zenethiwekhi zisebenza kanjani. Nokho, sisazozama ukuqonda izimiso eziyisisekelo nezindlela zokucubungula ukugeleza kwethrafikhi sisebenzisa izinqubomgomo zenethiwekhi...

Kunengqondo ukuthi kunezinhlobo ezi-2 zethrafikhi: ukungena ku-pod (I-Ingress) nokuphuma kuyo (Egress).

Empeleni, ipolitiki ihlukaniswe ngalezi zigaba ezi-2 ngokusekelwe ekuqondisweni kokunyakaza.

Isibaluli esilandelayo esidingekayo yisikhethi; lowo umthetho osebenza kuye. Lokhu kungaba i-pod (noma iqembu lama-pods) noma indawo (okungukuthi indawo yamagama). Imininingwane ebalulekile: zombili izinhlobo zalezi zinto kufanele zibe nelebula (ilebula ku-Kubernetes terminology) - yilawa osopolitiki abasebenza ngawo.

Ngokungeziwe enanini elilinganiselwe labakhethi abahlanganiswe uhlobo oluthile lwelebula, kungenzeka ukubhala imithetho efana nokuthi “Vumela/phika yonke into/wonke umuntu” ngokuhlukahluka okuhlukile. Ngale njongo, ukwakhiwa kwefomu kusetshenziswa:

  podSelector: {}
  ingress: []
  policyTypes:
  - Ingress

- kulesi sibonelo, wonke ama-pods endaweni avinjelwe kuthrafikhi engenayo. Ukuziphatha okuphambene kungafinyelelwa ngokwakhiwa okulandelayo:

  podSelector: {}
  ingress:
  - {}
  policyTypes:
  - Ingress

Ngokufanayo kokuphumayo:

  podSelector: {}
  policyTypes:
  - Egress

- ukuyicisha. Futhi nakhu ongakufaka:

  podSelector: {}
  egress:
  - {}
  policyTypes:
  - Egress

Ukubuyela ekukhethweni kwe-plugin ye-CNI yeqoqo, kufanelekile ukuqaphela lokho akuwona wonke ama-plugin enethiwekhi asekela i-NetworkPolicy. Isibonelo, i-Flannel eshiwo kakade ayikwazi ukumisa izinqubomgomo zenethiwekhi, okuyinto kushiwo ngqo endaweni yokugcina esemthethweni. Okunye futhi kuyashiwo lapho - iphrojekthi yomthombo ovulekile UCalico, enweba kakhulu isethi evamile ye-Kubernetes APIs ngokuya ngezinqubomgomo zenethiwekhi.

Ukwazi uCalico: ithiyori

I-plugin ye-Calico ingasetshenziswa ekuhlanganisweni ne-Flannel (i-subproject Canal) noma ngokuzimela, okuhlanganisa kokubili ukuxhumana kwenethiwekhi namandla okuphatha ukutholakala.

Imaphi amathuba anikezwayo ngokusebenzisa isixazululo “sebhokisi” ye-K8s kanye nesethi ye-API evela ku-Calico?

Nakhu okwakhelwe ku-NetworkPolicy:

• osopolitiki banqunyelwe imvelo;
• izinqubomgomo zisetshenziswa kuma-pods amakwe ngamalebula;
• imithetho ingasetshenziswa kuma-pods, izindawo noma ama-subnets;
• Imithetho ingaqukatha izivumelwano, ezinegama noma izicaciso zembobo engokomfanekiso.

Nansi indlela i-Calico enweba ngayo le misebenzi:

• izinqubomgomo zingasetshenziswa kunoma iyiphi into: i-pod, isitsha, umshini obonakalayo noma isixhumi esibonakalayo;
• imithetho ingaqukatha isenzo esithile (ukwenqatshelwa, imvume, ukugawulwa kwemithi);
• okuqondiwe noma umthombo wemithetho kungaba imbobo, uhla lwamachweba, izivumelwano, izibaluli ze-HTTP noma ze-ICMP, i-IP noma i-subnet (isizukulwane sesi-4 noma sesi-6), noma yiziphi izikhethi (ama-node, ababungazi, izindawo);
• Ukwengeza, ungakwazi ukulawula ukudlula kwethrafikhi usebenzisa izilungiselelo ze-DNAT nezinqubomgomo zokudlulisa ithrafikhi.

Owokuqala wenza ku-GitHub endaweni yenqolobane yaseCalico emuva kukaJulayi 2016, kwathi ngonyaka olandelayo iphrojekthi yathatha isikhundla esiphambili ekuhleleni ukuxhumana kwenethiwekhi ye-Kubernetes - lokhu kufakazelwa, ngokwesibonelo, ngemiphumela yocwaningo, eqhutshwa yi-New Stack:

Izixazululo eziningi ezinkulu eziphethwe ngama-K8s, njenge I-Amazon EKS, I-Azure AKS, Google GKE nabanye baqala ukuyincoma ukuthi isetshenziswe.

Ngokuqondene nokusebenza, konke kuhle lapha. Ekuhloleni umkhiqizo wabo, ithimba lokuthuthukisa i-Calico libonise ukusebenza kwezinkanyezi, lisebenzisa iziqukathi ezingaphezu kuka-50000 kuma-node abonakalayo angu-500 anenani lokudala leziqukathi ezingu-20 ngomzuzwana. Azikho izinkinga ezihlonzwe ngokukala. Imiphumela enjalo zamenyezelwa kakade ekumenyezweni kwenguqulo yokuqala. Izifundo ezizimele ezigxile ekusebenzeni nasekusetshenzisweni kwezinsiza nazo ziqinisekisa ukuthi ukusebenza kukaCalico kucishe kufane nokukaFlannel. Isibonelo:

Iphrojekthi ithuthuka ngokushesha okukhulu, isekela umsebenzi kuzixazululo ezidumile eziphethwe ama-K8s, i-OpenShift, i-OpenStack, kungenzeka ukusebenzisa i-Calico lapho uphakela iqoqo usebenzisa. khahlela, kukhona izinkomba ekwakhiweni kwamanethiwekhi we-Service Mesh (nasi isibonelo esetshenziswe ngokubambisana ne-Istio).

Prakthiza nge-Calico

Esimweni esijwayelekile sokusebenzisa i-vanilla Kubernetes, ukufaka i-CNI kwehla ekusebenziseni ifayela calico.yaml, ilandwe kusizindalwazi esisemthethweni, ngokusebenzisa kubectl apply -f.

Njengomthetho, inguqulo yamanje ye-plugin ihambisana nezinguqulo zakamuva ze-2-3 ze-Kubernetes: ukusebenza ezinguqulweni ezindala akuvivinywa futhi akuqinisekisiwe. Ngokusho kwabathuthukisi, i-Calico isebenza kuma-kernel e-Linux ngaphezulu kwe-3.10 esebenzisa i-CentOS 7, Ubuntu 16 noma i-Debian 8, phezu kwama-iptables noma i-IPVS.

Ukuzihlukanisa phakathi kwemvelo

Ukuze uthole ukuqonda okujwayelekile, ake sibheke isimo esilula ukuze siqonde ukuthi izinqubomgomo zenethiwekhi ekuphawulweni kwe-Calico zihluka kanjani kwezijwayelekile nokuthi indlela yokudala imithetho ikwenza kube lula ukufundeka kwayo nokuvumelana nezimo zokucushwa:

Kunezinhlelo zokusebenza zewebhu ezi-2 ezifakwe kuqoqo: ku-Node.js ne-PHP, eyodwa yazo esebenzisa i-Redis. Ukuvimba ukufinyelela ku-Redis kusuka ku-PHP, ngenkathi ugcina ukuxhumana ne-Node.js, vele usebenzise inqubomgomo elandelayo:

kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: allow-redis-nodejs
spec:
  podSelector:
    matchLabels:
      service: redis
  ingress:
  - from:
    - podSelector:
        matchLabels:
          service: nodejs
    ports:
    - protocol: TCP
      port: 6379

Empeleni sivumele ithrafikhi engenayo ethekwini le-Redis ukusuka ku-Node.js. Futhi ngokusobala abazange bavimbele noma yini enye. Lapho nje i-NetworkPolicy ibonakala, zonke izikhethi ezishiwo kuyo ziqala ukuhlukaniswa, ngaphandle uma kuchazwe ngenye indlela. Kodwa-ke, imithetho yokuhlukanisa ayisebenzi kwezinye izinto ezingahlanganiswa yisikhethi.

Isibonelo sisetshenziswa apiVersion Kubernetes ngaphandle kwebhokisi, kodwa akukho okukuvimbela ukuthi uyisebenzise insiza yegama elifanayo kusukela ekulethweni kwe-Calico. I-syntax lapho inemininingwane eminingi, ngakho-ke uzodinga ukuthi uphinde ubhale umthetho wecala elingenhla ngale ndlela elandelayo:

apiVersion: crd.projectcalico.org/v1
kind: NetworkPolicy
metadata:
  name: allow-redis-nodejs
spec:
  selector: service == 'redis'
  ingress:
  - action: Allow
    protocol: TCP
    source:
      selector: service == 'nodejs'
    destination:
      ports:
      - 6379

Izakhiwo ezibalwe ngenhla zokuvumela noma ukwenqaba yonke i-traffic nge-NetworkPolicy API evamile iqukethe izakhiwo ezinamabakaki okunzima ukukuqonda nokuzikhumbula. Endabeni ye-Calico, ukuze uguqule i-logic yomthetho we-firewall iye kokuphambene, vele ushintshe action: Allow on action: Deny.

Ukuhlukaniswa nemvelo

Manje cabanga ngesimo lapho uhlelo lokusebenza lukhiqiza amamethrikhi ebhizinisi ukuze aqoqwe e-Prometheus nokuhlaziya okwengeziwe kusetshenziswa i-Grafana. Okulayishiwe kungase kuqukathe idatha ebucayi, engaphinda ibukwe esidlangalaleni ngokuzenzakalela. Masifihle le datha ukuze abantu bangaboni emehlweni:

I-Prometheus, njengomthetho, ibekwe endaweni ehlukile yesevisi - esibonelweni kuzoba indawo yegama kanje:

apiVersion: v1
kind: Namespace
metadata:
  labels:
    module: prometheus
  name: kube-prometheus

Insimu metadata.labels lokhu akwenzekanga ngengozi. Njengoba kushiwo ngenhla, namespaceSelector (kanye ne podSelector) isebenza ngamalebula. Ngakho-ke, ukuze uvumele amamethrikhi ukuthi athathwe kuwo wonke ama-pods embotsheni ethile, kuzodingeka wengeze uhlobo oluthile lwelebula (noma uthathe kwalukhona kakade), bese usebenzisa ukumisa okufana nalokhu:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-metrics-prom
spec:
  podSelector: {}
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          module: prometheus
    ports:
    - protocol: TCP
      port: 9100

Futhi uma usebenzisa izinqubomgomo ze-Calico, i-syntax izoba kanje:

apiVersion: crd.projectcalico.org/v1
kind: NetworkPolicy
metadata:
  name: allow-metrics-prom
spec:
  ingress:
  - action: Allow
    protocol: TCP
    source:
      namespaceSelector: module == 'prometheus'
    destination:
      ports:
      - 9100

Ngokuvamile, ngokwengeza lezi zinhlobo zezinqubomgomo zezidingo ezithile, ungavikela ekuphazamisekeni okunonya noma ngephutha ekusebenzeni kwezinhlelo zokusebenza kuqoqo.

Umkhuba omuhle kakhulu, ngokusho kwabadali beCalico, indlela "Vimba yonke into futhi uvule ngokusobala okudingayo", ebhalwe ku. imibhalo esemthethweni (abanye balandela indlela efanayo - ikakhulukazi, ku isihloko esishiwo kakade).

Ukusebenzisa Izinto Ze-Calico Ezengeziwe

Ake ngikukhumbuze ukuthi ngesethi enwetshiwe ye-Calico APIs ungakwazi ukulawula ukutholakala kwamanodi, hhayi kuphela kuma-pods. Esibonelweni esilandelayo usebenzisa GlobalNetworkPolicy ikhono lokudlulisa izicelo ze-ICMP kuqoqo livaliwe (ngokwesibonelo, ama-pings ukusuka ku-pod kuya endaweni, phakathi kwama-pods, noma ukusuka ku-node kuya ku-IP pod):

apiVersion: crd.projectcalico.org/v1
kind: GlobalNetworkPolicy
metadata:
  name: block-icmp
spec:
  order: 200
  selector: all()
  types:
  - Ingress
  - Egress
  ingress:
  - action: Deny
    protocol: ICMP
  egress:
  - action: Deny
    protocol: ICMP

Esimeni esingenhla, kusengenzeka ukuthi ama-cluster node "afinyelele" komunye nomunye nge-ICMP. Futhi lolu daba luxazululwa ngezindlela GlobalNetworkPolicy, kusetshenziswe ebhizinisini HostEndpoint:

apiVersion: crd.projectcalico.org/v1
kind: GlobalNetworkPolicy
metadata:
  name: deny-icmp-kube-02
spec:
  selector: "role == 'k8s-node'"
  order: 0
  ingress:
  - action: Allow
    protocol: ICMP
  egress:
  - action: Allow
    protocol: ICMP
---
apiVersion: crd.projectcalico.org/v1
kind: HostEndpoint
metadata:
  name: kube-02-eth0
  labels:
    role: k8s-node
spec:
  interfaceName: eth0
  node: kube-02
  expectedIPs: ["192.168.2.2"]

Ikesi le-VPN

Okokugcina, ngizonikeza isibonelo sangempela sokusebenzisa imisebenzi ye-Calico esimweni sokusebenzisana eduze kweqoqo, lapho isethi evamile yezinqubomgomo inganele. Ukuze ufinyelele uhlelo lwewebhu, amaklayenti asebenzisa umhubhe we-VPN, futhi lokhu kufinyelela kulawulwa ngokuqinile futhi kukhawulelwe ohlwini oluthile lwezinsizakalo ezivunyelwe ukusetshenziswa:

Amaklayenti axhumeka ku-VPN ngembobo evamile ye-UDP engu-1194 futhi, lapho exhunyiwe, athola imizila eya ku-subnet yeqoqo lama-pods namasevisi. Wonke ama-subnet ayaphushwa ukuze angalahlekelwa yizinsizakalo ngesikhathi sokuqalisa kabusha nokushintshwa kwekheli.

Ichweba ekucushweni lijwayelekile, elibeka ama-nuances athile enqubweni yokumisa uhlelo nokuludlulisela kuqoqo le-Kubernetes. Isibonelo, ku-AWS LoadBalancer efanayo ye-UDP ivele ngokoqobo ekupheleni konyaka odlule ohlwini olulinganiselwe lwezifunda, futhi i-NodePort ayikwazi ukusetshenziswa ngenxa yokudluliselwa kwayo kuwo wonke ama-cluster node futhi akunakwenzeka ukukala inani lezehlakalo zeseva izinhloso zokubekezelela amaphutha. Futhi, kuzodingeka ushintshe ububanzi obuzenzakalelayo bezimbobo...

Njengomphumela wokucwaninga ngezixazululo ezingaba khona, okulandelayo kwakhethwa:

1. Amaphodi ane-VPN ahlelelwe indawo ngayinye hostNetwork, okungukuthi, ku-IP yangempela.
2. Isevisi ithunyelwa ngaphandle ClusterIP. Imbobo ifakwe ngokoqobo ku-node, efinyeleleka ngaphandle ngokubhukha okuncane (ukuba khona okunemibandela kwekheli le-IP langempela).
3. Ukunquma indawo lapho i-pod rose ingaphezu kobubanzi bendaba yethu. Ngizosho nje ukuthi ungakwazi ukufaka i-hardwire isevisi endaweni noma ubhale isevisi ye-sidecar encane ezoqapha ikheli le-IP lamanje lesevisi ye-VPN futhi uhlele amarekhodi e-DNS abhaliswe namakhasimende - noma ubani onokucabanga okwanele.

Ngokombono womzila, singakwazi ukuhlonza ngokukhethekile iklayenti le-VPN ngekheli lalo le-IP elikhishwe iseva ye-VPN. Ngezansi isibonelo sakudala sokukhawulela ukufinyelela kwekhasimende elinjalo ezinsizeni, esiboniswe ku-Redis eshiwo ngenhla:

apiVersion: crd.projectcalico.org/v1
kind: HostEndpoint
metadata:
  name: vpnclient-eth0
  labels:
    role: vpnclient
    environment: production
spec:
  interfaceName: "*"
  node: kube-02
  expectedIPs: ["172.176.176.2"]
---
apiVersion: crd.projectcalico.org/v1
kind: GlobalNetworkPolicy
metadata:
  name: vpn-rules
spec:
  selector: "role == 'vpnclient'"
  order: 0
  applyOnForward: true
  preDNAT: true
  ingress:
  - action: Deny
    protocol: TCP
    destination:
      ports: [6379]
  - action: Allow
    protocol: UDP
    destination:
      ports: [53, 67]

Lapha, ukuxhuma ku-port 6379 akuvunyelwe ngokuqinile, kodwa ngesikhathi esifanayo ukusebenza kwesevisi ye-DNS kuyagcinwa, ukusebenza kwayo okuvame ukuhlupheka lapho udweba imithetho. Ngoba, njengoba kushiwo ngaphambili, lapho isikhethi sivela, inqubomgomo yokwenqaba ezenzakalelayo isetshenziswa kuso ngaphandle kwalapho kucaciswe ngenye indlela.

Imiphumela

Ngakho-ke, usebenzisa i-API ethuthukisiwe ye-Calico, ungakwazi ukulungisa kalula futhi uguqule ngokushintshashintshayo umzila ngaphakathi naseduze kweqoqo. Ngokuvamile, ukusetshenziswa kwayo kungase kubukeke njengokudubula ondlunkulu ngenganono, futhi ukusebenzisa inethiwekhi ye-L3 enemigudu ye-BGP ne-IP-IP kubukeka kubi kakhulu ekufakweni okulula kwe-Kubernetes kunethiwekhi eyisicaba... Nokho, ngaphandle kwalokho ithuluzi libukeka lisebenza futhi liwusizo. .

Ukuhlukanisa iqoqo ukuze kuhlangatshezwane nezidingo zokuphepha kungase kungenzeki ngaso sonke isikhathi, futhi kulapho u-Calico (noma isisombululo esifanayo) esiza khona. Izibonelo ezinikezwe kulesi sihloko (ezinezinguquko ezincane) zisetshenziswa ekufakweni okumbalwa kwamakhasimende ethu ku-AWS.

PS

Funda futhi kubhulogi yethu:

• «Isingeniso se-Kubernetes Network Policies for Security Professionals";
• "Umhlahlandlela Onemifanekiso Wokuxhumana Nenethiwekhi ku-Kubernetes": izingxenye 1 kanye no-2 (imodeli yenethiwekhi, amanethiwekhi ambondelanayo), ingxenye 3 (amasevisi kanye nokucutshungulwa kwethrafikhi);
• «I-Container Networking Interface (CNI) - isixhumi esibonakalayo senethiwekhi nezinga le-Linux".

Source: www.habr.com