ABC na Tsaro a Kubernetes: Tabbatarwa, izini, dubawa

Ba da dade ko ba dade, a cikin aiki na kowane tsari, batun tsaro ya taso: tabbatar da tabbatarwa, raba haƙƙoƙi, dubawa da sauran ayyuka. An riga an ƙirƙira don Kubernetes mafita da yawa, wanda ke ba ka damar cimma daidaituwa tare da ma'auni har ma a cikin yanayin da ake bukata ... Irin wannan abu yana da mahimmanci ga mahimman abubuwan tsaro da aka aiwatar a cikin hanyoyin da aka gina na K8s. Da farko, zai zama da amfani ga waɗanda suka fara fahimtar Kubernetes - a matsayin mafari don nazarin batutuwan da suka shafi tsaro.

Gasktawa

Akwai nau'ikan masu amfani guda biyu a cikin Kubernetes:

• Asusun Sabis - asusun da Kubernetes API ke sarrafawa;
• Masu amfani - masu amfani da "na al'ada" ta waje, ayyuka masu zaman kansu.

Babban bambanci tsakanin waɗannan nau'ikan shine don Asusun Sabis akwai abubuwa na musamman a cikin Kubernetes API (ana kiran su cewa - ServiceAccounts), waɗanda aka ɗaure da sararin suna da saitin bayanan izini da aka adana a cikin gungu a cikin abubuwa na nau'in Asirin. Irin waɗannan masu amfani (Asusun Sabis) an yi niyya da farko don sarrafa haƙƙin samun dama ga Kubernetes API na tafiyar matakai da ke gudana a cikin gungu na Kubernetes.

Masu amfani na yau da kullun ba su da shigarwar a cikin Kubernetes API: dole ne a sarrafa su ta hanyoyin waje. An yi su ne don mutane ko matakai da ke zaune a waje da tari.

Kowane buƙatun API yana da alaƙa da ko dai Asusun Sabis, Mai amfani, ko kuma ana ɗaukarsa ba a san shi ba.

Bayanan mai amfani sun haɗa da:

• Sunan mai amfani - sunan mai amfani (masu mahimmanci!);
• UID - igiyar tantance mai amfani da za ta iya karanta na'ura wacce ta kasance "mafi daidaituwa kuma na musamman fiye da sunan mai amfani";
• Groups - jerin ƙungiyoyin da mai amfani ke ciki;
• karin - ƙarin filayen da za a iya amfani da su ta hanyar ba da izini.

Kubernetes na iya amfani da ɗimbin hanyoyin tabbatarwa: Takaddun shaida na X509, Alamu masu ɗaukar nauyi, wakili mai tabbatarwa, HTTP Basic Auth. Yin amfani da waɗannan hanyoyin, zaku iya aiwatar da babban adadin tsare-tsaren izini: daga babban fayil tare da kalmomin shiga zuwa OpenID OAuth2.

Bugu da ƙari, yana yiwuwa a yi amfani da tsare-tsaren izini da yawa lokaci guda. Ta hanyar tsoho, gungu yana amfani da:

• alamun asusun sabis - don Asusun Sabis;
• X509 - don Masu amfani.

Tambayar game da sarrafa ServiceAccounts ya wuce iyakar wannan labarin, amma ga waɗanda suke so su fahimci kansu da wannan batu daki-daki, Ina ba da shawarar farawa da. shafukan takardun aiki. Za mu yi nazari sosai kan batun yadda takaddun takaddun X509 ke aiki.

Takaddun shaida ga masu amfani (X.509)

Hanyar gargajiya ta aiki tare da takaddun shaida ta ƙunshi:

• key tsara:

  mkdir -p ~/mynewuser/.certs/
  openssl genrsa -out ~/.certs/mynewuser.key 2048

• samar da buƙatar takardar shaida:

  openssl req -new -key ~/.certs/mynewuser.key -out ~/.certs/mynewuser.csr -subj "/CN=mynewuser/O=company"

• sarrafa buƙatun takardar shedar ta amfani da maɓallan cluster CA na Kubernetes, samun takardar shaidar mai amfani (don samun takaddun shaida, dole ne ku yi amfani da asusu wanda ke da damar zuwa maɓallin Kubernetes cluster CA, wanda ta tsohuwa yana cikin /etc/kubernetes/pki/ca.key):

  openssl x509 -req -in ~/.certs/mynewuser.csr -CA /etc/kubernetes/pki/ca.crt -CAkey /etc/kubernetes/pki/ca.key -CAcreateserial -out ~/.certs/mynewuser.crt -days 500

• ƙirƙirar fayil ɗin sanyi:
  • Bayanin gungu (ayyana adireshi da wurin fayil ɗin takardar shedar CA don takamaiman shigarwar gungu):

    kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/pki/ca.crt --server=https://192.168.100.200:6443

  • ko yaya bazaɓin da aka ba da shawarar - ba lallai ne ka saka takardar shaidar tushe ba (sannan kubectl ba zai bincika daidaitaccen sabar api-uwar garke ba):

    kubectl config set-cluster kubernetes  --insecure-skip-tls-verify=true --server=https://192.168.100.200:6443

  • ƙara mai amfani zuwa fayil ɗin daidaitawa:

    kubectl config set-credentials mynewuser --client-certificate=.certs/mynewuser.crt  --client-key=.certs/mynewuser.key

  • ƙara mahallin:

    kubectl config set-context mynewuser-context --cluster=kubernetes --namespace=target-namespace --user=mynewuser

  • aikin mahallin tsoho:

    kubectl config use-context mynewuser-context

Bayan magudin da ke sama, a cikin fayil ɗin .kube/config za a ƙirƙiri config kamar haka:

apiVersion: v1
clusters:
- cluster:
    certificate-authority: /etc/kubernetes/pki/ca.crt
    server: https://192.168.100.200:6443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    namespace: target-namespace
    user: mynewuser
  name: mynewuser-context
current-context: mynewuser-context
kind: Config
preferences: {}
users:
- name: mynewuser
  user:
    client-certificate: /home/mynewuser/.certs/mynewuser.crt
    client-key: /home/mynewuser/.certs/mynewuser.key

Don sauƙaƙe don canja wurin saitin tsakanin asusu da sabobin, yana da amfani don gyara ƙimar maɓallan masu zuwa:

• certificate-authority
• client-certificate
• client-key

Don yin wannan, zaku iya ɓoye fayilolin da aka ƙayyade a cikin su ta amfani da base64 kuma kuyi rajistar su a cikin saitin, ƙara ƙarawa zuwa sunan maɓallan. -data, i.e. bayan sun karba certificate-authority-data da sauransu.

Takaddun shaida tare da kubeadm

Tare da sakin Kubernetes 1.15 Yin aiki tare da takaddun shaida ya zama mafi sauƙi godiya ga nau'in alpha na goyon bayansa a ciki kubeadm utility. Misali, wannan shine abin da samar da fayil ɗin sanyi tare da maɓallan mai amfani zai iya zama kamar:

kubeadm alpha kubeconfig user --client-name=mynewuser --apiserver-advertise-address 192.168.100.200

NB: Ana bukata tallata adireshin ana iya samuwa a cikin saitin uwar garken api, wanda ta tsohuwa yana cikin /etc/kubernetes/manifests/kube-apiserver.yaml.

Za a fitar da tsarin da aka samu zuwa stdout. Yana buƙatar ajiyewa a ciki ~/.kube/config asusun mai amfani ko zuwa fayil da aka kayyade a cikin canjin yanayi KUBECONFIG.

Yi zurfi

Ga masu son fahimtar batutuwan da aka bayyana su sosai:

• raba labarin akan aiki tare da takaddun shaida a cikin takaddun Kubernetes na hukuma;
• labari mai kyau daga Bitnami, wanda aka tabo batun takaddun shaida ta fuskar aiki.
• janar takardun a kan tabbaci a Kubernetes.

Izini

Tsohuwar asusu mai izini bashi da haƙƙin yin aiki akan tari. Don ba da izini, Kubernetes yana aiwatar da hanyar ba da izini.

Kafin sigar 1.6, Kubernetes yayi amfani da nau'in izini da ake kira ABAC (Ikon samun dama na tushen sifa). Ana iya samun cikakkun bayanai game da shi a takardun shaida. Ana ɗaukar wannan hanyar a halin yanzu gado ce, amma har yanzu kuna iya amfani da ita tare da sauran nau'ikan tantancewa.

Ana kiran hanyar yanzu (kuma mafi sassauƙa) na raba haƙƙin samun dama ga gungu RBAC (Ikon tushen tushen aiki). An ayyana kwanciyar hankali tun sigar Kubernetes 1.8. RBAC tana aiwatar da tsarin haƙƙin wanda a cikinsa aka haramta duk abin da ba a ba da izini ba.
Don kunna RBAC, kuna buƙatar fara Kubernetes api-uwar garken tare da siga --authorization-mode=RBAC. An saita sigogi a cikin bayyanuwa tare da saitin uwar garken api, wanda ta tsohuwa yana kan hanyar /etc/kubernetes/manifests/kube-apiserver.yaml, a sashe command. Koyaya, an riga an kunna RBAC ta tsohuwa, don haka da alama bai kamata ku damu da shi ba: zaku iya tabbatar da wannan ta ƙimar. authorization-mode (a cikin da aka ambata kube-apiserver.yaml). Af, a cikin ma'anarsa ana iya samun wasu nau'ikan izini (node, webhook, always allow), amma za mu bar la'akarinsu a waje da iyakokin kayan aiki.

Af, mun riga mun buga labarin tare da cikakken bayanin ƙa'idodi da fasali na aiki tare da RBAC, don haka zan ƙara taƙaita kaina ga taƙaitaccen jeri na asali da misalai.

Ana amfani da abubuwan API masu zuwa don sarrafa shiga cikin Kubernetes ta hanyar RBAC:

• Role и ClusterRole - rawar da ke aiki don bayyana haƙƙin shiga:
• Role yana ba ku damar bayyana haƙƙoƙin cikin sararin suna;
• ClusterRole - a cikin gungu, gami da takamaiman abubuwa kamar su nodes, url marasa albarkatu (watau ba su da alaƙa da albarkatun Kubernetes - misali, /version, /logs, /api*);
• RoleBinding и ClusterRoleBinding - amfani da ɗauri Role и ClusterRole zuwa mai amfani, rukunin mai amfani ko Account Account.

Abubuwan Role da RoleBinding an iyakance su ta sararin suna, watau. dole ne ya kasance a cikin sararin suna. Koyaya, RoleBinding na iya yin la'akari da ClusterRole, wanda ke ba ku damar ƙirƙirar saitin izini na gabaɗaya da sarrafa damar amfani da su.

Matsayin yana bayyana haƙƙoƙin ta amfani da jerin ƙa'idodi waɗanda suka ƙunshi:

• Ƙungiyoyin API - duba takardun hukuma ta apiGroups da fitarwa kubectl api-resources;
• albarkatun (Albarkatun: pod, namespace, deployment da sauransu.);
• Kalmomi (karin magana: set, update da sauransu.).
• albarkatun albarkatu (resourceNames) - don yanayin lokacin da kuke buƙatar samar da damar yin amfani da takamaiman albarkatu, kuma ba ga duk albarkatun irin wannan ba.

Ana iya samun ƙarin cikakken bincike na izini a cikin Kubernetes akan shafin takardun shaida. Maimakon haka (ko ma dai, ban da wannan), zan ba da misalan da ke kwatanta aikinta.

Misalan abubuwan RBAC

Sauƙi Role, wanda ke ba ka damar samun jeri da matsayi na kwasfan fayiloli da saka idanu a cikin sunan suna target-namespace:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: target-namespace
  name: pod-reader
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "watch", "list"]

Alal misali: ClusterRole, wanda ke ba ka damar samun jeri da matsayi na kwas ɗin da saka idanu a cikin tari:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  # секции "namespace" нет, так как ClusterRole задействует весь кластер
  name: secret-reader
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get", "watch", "list"]

Alal misali: RoleBinding, wanda ke ba da damar mai amfani mynewuser "karanta" kwasfan fayiloli a cikin sararin suna my-namespace:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: read-pods
  namespace: target-namespace
subjects:
- kind: User
  name: mynewuser # имя пользователя зависимо от регистра!
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role # здесь должно быть “Role” или “ClusterRole”
  name: pod-reader # имя Role, что находится в том же namespace,
                   # или имя ClusterRole, использование которой
                   # хотим разрешить пользователю
  apiGroup: rbac.authorization.k8s.io

Binciken taron

Tsarin tsari, ana iya wakilta gine-ginen Kubernetes kamar haka:

Maɓallin ɓangaren Kubernetes da ke da alhakin sarrafa buƙatun shine api-uwar garke. Duk ayyukan da ke kan gungu suna tafiya ta cikinsa. Kuna iya karanta ƙarin game da waɗannan hanyoyin na ciki a cikin labarin "Me ke faruwa a Kubernetes lokacin da kuke gudu kubectl?".

Binciken tsarin abu ne mai ban sha'awa a Kubernetes, wanda aka kashe ta tsohuwa. Yana ba ku damar shiga duk kira zuwa Kubernetes API. Kamar yadda zaku iya tsammani, duk ayyukan da suka shafi sa ido da canza yanayin gungu ana yin su ta wannan API. Ana iya samun kyakkyawan bayanin iyawarsa (kamar yadda aka saba) a ciki takardun shaida K8s. Na gaba, zan yi ƙoƙarin gabatar da batun cikin harshe mai sauƙi.

Sabili da haka, don ba da damar dubawa, muna buƙatar wuce sigogi guda uku da ake buƙata zuwa akwati a cikin uwar garken api, waɗanda aka bayyana dalla-dalla a ƙasa:

• --audit-policy-file=/etc/kubernetes/policies/audit-policy.yaml
• --audit-log-path=/var/log/kube-audit/audit.log
• --audit-log-format=json

Baya ga waɗannan sigogi guda uku masu mahimmanci, akwai ƙarin ƙarin saituna da yawa masu alaƙa da dubawa: daga jujjuyawar log zuwa kwatancen gidan yanar gizo. Misalin sigogin jujjuyawar log:

• --audit-log-maxbackup=10
• --audit-log-maxsize=100
• --audit-log-maxage=7

Amma ba za mu yi magana da su dalla-dalla ba - zaku iya samun duk cikakkun bayanai a ciki takardun kube-apiserver.

Kamar yadda aka ambata, duk sigogi an saita su a cikin bayyanuwa tare da saitin uwar garken api (ta tsohuwa /etc/kubernetes/manifests/kube-apiserver.yaml), a cikin sashe command. Bari mu koma kan sigogi guda 3 da ake buƙata kuma mu bincika su:

1. audit-policy-file - hanyar zuwa fayil ɗin YAML da ke kwatanta manufofin duba. Za mu koma abubuwan da ke ciki daga baya, amma a yanzu zan lura cewa fayil ɗin dole ne a karanta shi ta hanyar api-server. Saboda haka, wajibi ne a saka shi a cikin akwati, wanda zaka iya ƙara lambar mai zuwa zuwa sassan da suka dace na saitin:

     volumeMounts:
       - mountPath: /etc/kubernetes/policies
         name: policies
         readOnly: true
     volumes:
     - hostPath:
         path: /etc/kubernetes/policies
         type: DirectoryOrCreate
       name: policies

2. audit-log-path - hanyar zuwa fayil ɗin log. Tilas ne hanyar kuma ta kasance mai isa ga tsarin api-uwar garke, don haka muna bayyana hawansa kamar haka:

     volumeMounts:
       - mountPath: /var/log/kube-audit
         name: logs
         readOnly: false
     volumes:
     - hostPath:
         path: /var/log/kube-audit
         type: DirectoryOrCreate
       name: logs

3. audit-log-format - duba log format. Tsohuwar ita ce json, amma tsarin rubutun gado kuma yana samuwa (legacy).

Manufar Audit

Yanzu game da fayil ɗin da aka ambata yana kwatanta manufofin shiga. Manufar farko ta manufofin duba ita ce level, matakin shiga. Gasu kamar haka:

• None - kada ku shiga;
• Metadata - metadata buƙatun log: mai amfani, lokacin buƙata, albarkatun manufa (pod, sarari suna, da sauransu), nau'in aiki (fi'ili), da sauransu;
• Request - log metadata da buƙatar jiki;
• RequestResponse - log metadata, buƙatar jiki da jikin amsawa.

Mataki na biyu na ƙarshe (Request и RequestResponse) kar a yi rajistar buƙatun da ba su sami damar albarkatu ba (hanyoyi zuwa abubuwan da ake kira url marasa albarkatu).

Hakanan duk buƙatun suna wucewa matakai da yawa:

• RequestReceived - matakin lokacin da mai sarrafa buƙatun ya karɓi buƙatun kuma har yanzu ba a watsa shi gaba da jerin abubuwan sarrafawa ba;
• ResponseStarted - Ana aika masu taken amsa, amma kafin a aika da martanin. An ƙirƙira don tambayoyin dogon gudu (misali, watch);
• ResponseComplete - An aika da hukumar mayar da martani, ba za a aika ƙarin bayani ba;
• Panic - abubuwan da ke faruwa suna faruwa lokacin da aka gano wani yanayi mara kyau.

Don tsallake kowane matakai da za ku iya amfani da su omitStages.

A cikin fayil ɗin tsari, zamu iya kwatanta sassa da yawa tare da matakan shiga daban-daban. Za a yi amfani da ƙa'idar daidaitawa ta farko da aka samo a cikin bayanin manufofin.

Kubelet daemon yana lura da canje-canje a cikin bayyanar tare da daidaitawar uwar garken api kuma, idan an gano wasu, sake kunna akwati tare da sabar api. Amma akwai wani muhimmin daki-daki: canje-canje a cikin fayil ɗin manufofin ba za a yi watsi da shi ba. Bayan yin canje-canje ga fayil ɗin manufofin, kuna buƙatar sake kunna uwar garken api da hannu. Tunda api-uwar garke kamar yadda a tsaye, tawaga kubectl delete ba zai sa ta sake farawa ba. Dole ne ku yi shi da hannu docker stop akan kube-masters, inda aka canza manufofin duba:

docker stop $(docker ps | grep k8s_kube-apiserver | awk '{print $1}')

Lokacin kunna dubawa, yana da mahimmanci a tuna da hakan nauyin da ke kan kube-apiserver yana ƙaruwa. Musamman, amfani da ƙwaƙwalwar ajiya don adana mahallin buƙatu yana ƙaruwa. Shiga yana farawa ne kawai bayan an aika da taken martani. Har ila yau, lodin ya dogara da tsarin manufofin duba.

Misalai na manufofi

Bari mu dubi tsarin fayilolin manufofin ta amfani da misalai.

Ga fayil mai sauƙi policydon shiga komai a matakin Metadata:

apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: Metadata

A cikin manufofin za ku iya ƙayyade jerin masu amfani (Users и ServiceAccounts) da kungiyoyin masu amfani. Misali, wannan shine yadda zamu yi watsi da masu amfani da tsarin, amma shiga duk wani abu a matakin Request:

apiVersion: audit.k8s.io/v1
kind: Policy
rules:
  - level: None
    userGroups:
      - "system:serviceaccounts"
      - "system:nodes"
    users:
      - "system:anonymous"
      - "system:apiserver"
      - "system:kube-controller-manager"
      - "system:kube-scheduler"
  - level: Request

Hakanan yana yiwuwa a bayyana maƙasudin:

• wuraren suna (namespaces);
• Kalmomi (karin magana: get, update, delete da sauransu);
• albarkatun (Albarkatun, wato: pod, configmaps da dai sauransu) da kungiyoyin albarkatu (apiGroups).

Kula! Za a iya samun albarkatu da ƙungiyoyin albarkatu (ƙungiyoyin API, watau apiGroups), da nau'ikan su da aka shigar a cikin gungu, ta amfani da umarni:

kubectl api-resources
kubectl api-versions

An bayar da manufofin duba mai zuwa azaman nunin mafi kyawun ayyuka a ciki Alibaba Cloud Dokokin:

apiVersion: audit.k8s.io/v1beta1
kind: Policy
# Не логировать стадию RequestReceived
omitStages:
  - "RequestReceived"
rules:
  # Не логировать события, считающиеся малозначительными и не опасными:
  - level: None
    users: ["system:kube-proxy"]
    verbs: ["watch"]
    resources:
      - group: "" # это api group с пустым именем, к которому относятся
                  # базовые ресурсы Kubernetes, называемые “core”
        resources: ["endpoints", "services"]
  - level: None
    users: ["system:unsecured"]
    namespaces: ["kube-system"]
    verbs: ["get"]
    resources:
      - group: "" # core
        resources: ["configmaps"]
  - level: None
    users: ["kubelet"]
    verbs: ["get"]
    resources:
      - group: "" # core
        resources: ["nodes"]
  - level: None
    userGroups: ["system:nodes"]
    verbs: ["get"]
    resources:
      - group: "" # core
        resources: ["nodes"]
  - level: None
    users:
      - system:kube-controller-manager
      - system:kube-scheduler
      - system:serviceaccount:kube-system:endpoint-controller
    verbs: ["get", "update"]
    namespaces: ["kube-system"]
    resources:
      - group: "" # core
        resources: ["endpoints"]
  - level: None
    users: ["system:apiserver"]
    verbs: ["get"]
    resources:
      - group: "" # core
        resources: ["namespaces"]
  # Не логировать обращения к read-only URLs:
  - level: None
    nonResourceURLs:
      - /healthz*
      - /version
      - /swagger*
  # Не логировать сообщения, относящиеся к типу ресурсов “события”:
  - level: None
    resources:
      - group: "" # core
        resources: ["events"]
  # Ресурсы типа Secret, ConfigMap и TokenReview могут содержать  секретные данные,
  # поэтому логируем только метаданные связанных с ними запросов
  - level: Metadata
    resources:
      - group: "" # core
        resources: ["secrets", "configmaps"]
      - group: authentication.k8s.io
        resources: ["tokenreviews"]
  # Действия типа get, list и watch могут быть ресурсоёмкими; не логируем их
  - level: Request
    verbs: ["get", "list", "watch"]
    resources:
      - group: "" # core
      - group: "admissionregistration.k8s.io"
      - group: "apps"
      - group: "authentication.k8s.io"
      - group: "authorization.k8s.io"
      - group: "autoscaling"
      - group: "batch"
      - group: "certificates.k8s.io"
      - group: "extensions"
      - group: "networking.k8s.io"
      - group: "policy"
      - group: "rbac.authorization.k8s.io"
      - group: "settings.k8s.io"
      - group: "storage.k8s.io"
  # Уровень логирования по умолчанию для стандартных ресурсов API
  - level: RequestResponse
    resources:
      - group: "" # core
      - group: "admissionregistration.k8s.io"
      - group: "apps"
      - group: "authentication.k8s.io"
      - group: "authorization.k8s.io"
      - group: "autoscaling"
      - group: "batch"
      - group: "certificates.k8s.io"
      - group: "extensions"
      - group: "networking.k8s.io"
      - group: "policy"
      - group: "rbac.authorization.k8s.io"
      - group: "settings.k8s.io"
      - group: "storage.k8s.io"
  # Уровень логирования по умолчанию для всех остальных запросов
  - level: Metadata

Wani kyakkyawan misali na manufofin duba shine profile amfani a GCE.

Don amsa da sauri ga abubuwan dubawa, yana yiwuwa bayyana webhook. An rufe wannan batu a ciki takardun shaida, Zan bar shi a waje da iyakar wannan labarin.

Sakamakon

Labarin yana ba da bayyani na ainihin hanyoyin tsaro a cikin gungu na Kubernetes, waɗanda ke ba ku damar ƙirƙirar asusun mai amfani na keɓaɓɓen, raba haƙƙoƙin su, da yin rikodin ayyukansu. Ina fatan zai kasance da amfani ga waɗanda ke fuskantar irin waɗannan batutuwa a cikin tunani ko a aikace. Har ila yau, ina ba da shawarar ku karanta jerin sauran kayan da ke kan batun tsaro a Kubernetes, wanda aka ba a cikin "PS" - watakila daga cikinsu za ku sami cikakkun bayanai game da matsalolin da suka dace da ku.

PS

Karanta kuma a kan shafinmu:

• «33+ Kubernetes kayan aikin tsaro";
• «Gabatarwa ga Manufofin Sadarwar Kubernetes don Ma'aikatan Tsaro";
• «Fahimtar RBAC a cikin Kubernetes";
• «9 Mafi kyawun Ayyuka na Tsaro Kubernetes";
• «Hanyoyi 11 don (Ba) Samun Hacked a Kubernetes".

source: www.habr.com