The ABC of Security in Kubernetes: Authentication, Authorization, Auditing

Sooner or later, in the operation of any system, the issue of security arises: providing authentication, separation of rights, auditing and other tasks. Already built for Kubernetes many solutions, which allow you to achieve standards compliance even in very demanding environments ... The same material is devoted to the basic security aspects implemented within the built-in K8s mechanisms. First of all, it will be useful to those who are starting to get acquainted with Kubernetes - as a starting point for studying issues related to security.

Authentication

There are two types of users in Kubernetes:

• Service Accounts - accounts managed by the Kubernetes API;
• Users - "normal" users managed by external, independent services.

The main difference between these types is that there are special objects for Service Accounts in the Kubernetes API (they are called like this - ServiceAccounts) that are bound to a namespace and a set of authorization data stored in the cluster in objects of type Secrets. Such users (Service Accounts) are mainly intended to manage access rights to the Kubernetes API of processes running in a Kubernetes cluster.

Ordinary Users, on the other hand, do not have entries in the Kubernetes API: they must be managed by external mechanisms. They are intended for people or processes living outside the cluster.

Each API request is tied to either a Service Account or a User, or is considered anonymous.

User authentication data includes:

• Username — username (case sensitive!);
• UID - a machine-readable user identification string that is "more consistent and unique than a username";
• Groups — list of groups to which the user belongs;
• Extras - additional fields that can be used by the authorization mechanism.

Kubernetes can use a large number of authentication mechanisms: X509 certificates, Bearer tokens, authentication proxy, HTTP Basic Auth. Using these mechanisms, you can implement a large number of authorization schemes: from a static password file to OpenID OAuth2.

Moreover, multiple authorization schemes can be used at the same time. By default, the cluster uses:

• service account tokens - for Service Accounts;
• X509 - for Users.

The issue of managing ServiceAccounts is beyond the scope of this article, and for those wishing to learn more about this issue, I recommend starting with official documentation pages. We will take a closer look at the issue of X509 certificates.

Certificates for Users (X.509)

The classic way to work with certificates involves:

• key generation:

  mkdir -p ~/mynewuser/.certs/
  openssl genrsa -out ~/.certs/mynewuser.key 2048

• generating a certificate request:

  openssl req -new -key ~/.certs/mynewuser.key -out ~/.certs/mynewuser.csr -subj "/CN=mynewuser/O=company"

• processing the certificate request using the CA keys of the Kubernetes cluster, obtaining a user certificate (to obtain a certificate, you need to use an account that has access to the key of the Kubernetes cluster CA, which by default is located in /etc/kubernetes/pki/ca.key):

  openssl x509 -req -in ~/.certs/mynewuser.csr -CA /etc/kubernetes/pki/ca.crt -CAkey /etc/kubernetes/pki/ca.key -CAcreateserial -out ~/.certs/mynewuser.crt -days 500

• creating a config file:
  • description of the cluster (specify the address and location of the CA certificate file of the specific cluster installation):

    kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/pki/ca.crt --server=https://192.168.100.200:6443

  • or how notrecommended option - you can not specify the root certificate (then kubectl will not check the correctness of the cluster api-server):

    kubectl config set-cluster kubernetes  --insecure-skip-tls-verify=true --server=https://192.168.100.200:6443

  • adding a user to the config file:

    kubectl config set-credentials mynewuser --client-certificate=.certs/mynewuser.crt  --client-key=.certs/mynewuser.key

  • adding context:

    kubectl config set-context mynewuser-context --cluster=kubernetes --namespace=target-namespace --user=mynewuser

  • default context assignment:

    kubectl config use-context mynewuser-context

After the above manipulations, in the file .kube/config the view config will be created:

apiVersion: v1
clusters:
- cluster:
    certificate-authority: /etc/kubernetes/pki/ca.crt
    server: https://192.168.100.200:6443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    namespace: target-namespace
    user: mynewuser
  name: mynewuser-context
current-context: mynewuser-context
kind: Config
preferences: {}
users:
- name: mynewuser
  user:
    client-certificate: /home/mynewuser/.certs/mynewuser.crt
    client-key: /home/mynewuser/.certs/mynewuser.key

To facilitate the transfer of the config between accounts and servers, it is useful to edit the values ​​of the following keys:

• certificate-authority
• client-certificate
• client-key

To do this, you can encode the files specified in them using base64 and register them in the config, adding the suffix to the name of the keys -data, i.e. received certificate-authority-data etc.

Certificates with kubeadm

With release Kubernetes 1.15 working with certificates has become much easier thanks to the alpha version of its support in kubeadm utility. For example, here's what the generation of a configuration file with user keys might look like now:

kubeadm alpha kubeconfig user --client-name=mynewuser --apiserver-advertise-address 192.168.100.200

NB: Required advertise address can be viewed in the api-server config, which is located by default in /etc/kubernetes/manifests/kube-apiserver.yaml.

The resulting config will be printed to stdout. It must be kept in ~/.kube/config user account or to a file specified in an environment variable KUBECONFIG.

Dig deeper

For those who want to take a closer look at the issues described:

• A separate article on working with certificates in the official Kubernetes documentation;
• good article from Bitnami, in which the issue of certificates is touched upon from a practical point of view.
• general documentation for authentication in Kubernetes.

Authorization

An authorized account, by default, does not have rights to act on the cluster. To grant permissions, Kubernetes implements an authorization mechanism.

Prior to version 1.6, Kubernetes used an authorization type called ABAC (Attribute-based access control). Details about it can be found in official documentation. This approach is currently considered legacy, but you can still use it at the same time as other authorization types.

The actual (and more flexible) way of separating access rights to a cluster is called RBAC (Role-based access control). It has been declared stable since version Kubernetes 1.8. RBAC implements a rights model that disallows everything that is not explicitly allowed.
To enable RBAC, you need to start Kubernetes api-server with the parameter --authorization-mode=RBAC. The parameters are set in the manifest with the api-server configuration, which by default is located along the path /etc/kubernetes/manifests/kube-apiserver.yaml, in section command. However, by default, RBAC is already enabled, so most likely you should not worry about it: you can verify this by the value authorization-mode (in the already mentioned kube-apiserver.yaml). By the way, among its values ​​there may be other types of authorization (node, webhook, always allow), but we will leave their consideration beyond the scope of the material.

By the way, we have already published Article with a fairly detailed story about the principles and features of working with RBAC, so further I will limit myself to a brief listing of the basics and examples.

The following API entities are used to control access to Kubernetes via RBAC:

• Role и ClusterRole - roles that serve to describe access rights:
• Role allows you to describe the rights within the namespace;
• ClusterRole - within the cluster, including to cluster-specific objects such as nodes, non-resources urls (that is, not related to Kubernetes resources - for example, /version, /logs, /api*);
• RoleBinding и ClusterRoleBinding - used for binding Role и ClusterRole to a user, user group, or ServiceAccount.

The Role and RoleBinding entities are namespace bound, i.e. must be within the same namespace. However, a RoleBinding can refer to a ClusterRole, which allows you to create a set of generic permissions and control access with them.

Roles describe rights using sets of rules containing:

• API groups - see official documentation by apiGroups and output kubectl api-resources;
• resources (Resources: pod, namespace, deployment etc.);
• Verbs (verbs: set, update etc.).
• resource names (resourceNames) - for the case when you need to provide access to a specific resource, and not to all resources of this type.

A more detailed breakdown of authorization in Kubernetes can be found on the page official documentation. Instead (or rather, in addition to this), I will give examples that illustrate its work.

RBAC Entity Examples

Simple Role, which allows you to get a list and status of pods and keep track of them in the namespace target-namespace:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: target-namespace
  name: pod-reader
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "watch", "list"]

Example ClusterRole, which allows you to get a list and status of pods and monitor them throughout the cluster:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  # секции "namespace" нет, так как ClusterRole задействует весь кластер
  name: secret-reader
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get", "watch", "list"]

Example RoleBinding, which allows the user mynewuser "read" pods in namespace my-namespace:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: read-pods
  namespace: target-namespace
subjects:
- kind: User
  name: mynewuser # имя пользователя зависимо от регистра!
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role # здесь должно быть “Role” или “ClusterRole”
  name: pod-reader # имя Role, что находится в том же namespace,
                   # или имя ClusterRole, использование которой
                   # хотим разрешить пользователю
  apiGroup: rbac.authorization.k8s.io

Event Audit

Schematically, the architecture of Kubernetes can be represented as follows:

The key Kubernetes component responsible for processing requests is − api-server. All operations on the cluster go through it. You can read more about these internal mechanisms in the article "What happens in Kubernetes when you run kubectl run?».

System Auditing is an interesting feature in Kubernetes that is disabled by default. It allows you to log all calls to the Kubernetes API. As you might guess, all actions related to the control and change of the state of the cluster are performed through this API. A good description of its capabilities can (as usual) be found in official documentation K8s. In what follows, I will try to explain the topic in a simpler way.

So, to enable auditing, we need to pass three required parameters to the container in api-server, which are described in more detail below:

• --audit-policy-file=/etc/kubernetes/policies/audit-policy.yaml
• --audit-log-path=/var/log/kube-audit/audit.log
• --audit-log-format=json

In addition to these three required parameters, there are many additional settings related to auditing: from log rotation to webhook descriptions. Example of log rotation parameters:

• --audit-log-maxbackup=10
• --audit-log-maxsize=100
• --audit-log-maxage=7

But we will not dwell on them in more detail - you can find all the details in kube-apiserver documentation.

As already mentioned, all parameters are set in the manifest with the api-server configuration (by default /etc/kubernetes/manifests/kube-apiserver.yaml), in the section command. Let's go back to the 3 required parameters and analyze them:

1. audit-policy-file - the path to the YAML file with a description of the policy (policy) of the audit. We will return to its contents, but for now I will note that the file must be readable by the api-server process. Therefore, you need to mount it inside the container, for which you can add the following code to the appropriate sections of the config:

     volumeMounts:
       - mountPath: /etc/kubernetes/policies
         name: policies
         readOnly: true
     volumes:
     - hostPath:
         path: /etc/kubernetes/policies
         type: DirectoryOrCreate
       name: policies

2. audit-log-path - path to the log file. The path must also be available to the api-server process, so we describe its mounting in the same way:

     volumeMounts:
       - mountPath: /var/log/kube-audit
         name: logs
         readOnly: false
     volumes:
     - hostPath:
         path: /var/log/kube-audit
         type: DirectoryOrCreate
       name: logs

3. audit-log-format — audit log format. The default is json, but the legacy text format is also available (legacy).

Audit Policy

Now about the mentioned file with a description of the logging policy. The first concept of an audit policy is level, logging level. They are as follows:

• None - do not log;
• Metadata — log request metadata: user, request time, target resource (pod, namespace, etc.), action type (verb), etc.;
• Request - log metadata and request body;
• RequestResponse - log metadata, request body and response body.

The last two levelsRequest и RequestResponse) do not log requests that did not access resources (references to the so-called non-resources urls).

Also, all requests go through several stages:

• RequestReceived - the stage when the request is received by the handler and has not yet been transferred further along the chain of handlers;
• ResponseStarted - The response headers are sent, but before the response body is sent. Generated for long running queries (for example, watch);
• ResponseComplete - the response body has been sent, no more information will be sent;
• Panic — Events are generated when an abnormal situation is detected.

To skip any stages, you can use omitStages.

In the policy file, we can describe several sections with different logging levels. The first matching rule found in the policy description will be applied.

The kubelet daemon listens for a change in the manifest with the api-server configuration and, if any, restarts the api-server container. But there is an important detail: changes in the policy file will be ignored. After making changes to the policy file, you will need to restart the api-server manually. Since api-server is started as static pod, team kubectl delete will not restart it. Will have to do it manually docker stop on kube-masters where the audit policy has been changed:

docker stop $(docker ps | grep k8s_kube-apiserver | awk '{print $1}')

When enabling auditing, it is important to remember that load increases on kube-apiserver. In particular, the memory consumption for storing the query context increases. Logging starts only after the response header has been sent. The load also depends on the configuration of the audit policy.

Policy examples

Let's analyze the structure of policy files using examples.

Here is a simple file policyto log everything at the level Metadata:

apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: Metadata

In policy, you can specify a list of users (Users и ServiceAccounts) and user groups. For example, this is how we will ignore system users, but log everything else at the level Request:

apiVersion: audit.k8s.io/v1
kind: Policy
rules:
  - level: None
    userGroups:
      - "system:serviceaccounts"
      - "system:nodes"
    users:
      - "system:anonymous"
      - "system:apiserver"
      - "system:kube-controller-manager"
      - "system:kube-scheduler"
  - level: Request

It is also possible to describe the target:

• namespaces (namespaces);
• Verbs (verbs: get, update, delete and others);
• resources (Resources, As follows: pod, configmaps etc.) and resource groups (apiGroups).

Pay attention! Resources and resource groups (API groups, i.e. apiGroups), as well as their versions installed in the cluster, can be obtained using the commands:

kubectl api-resources
kubectl api-versions

The following audit policy is provided as a demonstration of the best practices in Alibaba Cloud documentation:

apiVersion: audit.k8s.io/v1beta1
kind: Policy
# Не логировать стадию RequestReceived
omitStages:
  - "RequestReceived"
rules:
  # Не логировать события, считающиеся малозначительными и не опасными:
  - level: None
    users: ["system:kube-proxy"]
    verbs: ["watch"]
    resources:
      - group: "" # это api group с пустым именем, к которому относятся
                  # базовые ресурсы Kubernetes, называемые “core”
        resources: ["endpoints", "services"]
  - level: None
    users: ["system:unsecured"]
    namespaces: ["kube-system"]
    verbs: ["get"]
    resources:
      - group: "" # core
        resources: ["configmaps"]
  - level: None
    users: ["kubelet"]
    verbs: ["get"]
    resources:
      - group: "" # core
        resources: ["nodes"]
  - level: None
    userGroups: ["system:nodes"]
    verbs: ["get"]
    resources:
      - group: "" # core
        resources: ["nodes"]
  - level: None
    users:
      - system:kube-controller-manager
      - system:kube-scheduler
      - system:serviceaccount:kube-system:endpoint-controller
    verbs: ["get", "update"]
    namespaces: ["kube-system"]
    resources:
      - group: "" # core
        resources: ["endpoints"]
  - level: None
    users: ["system:apiserver"]
    verbs: ["get"]
    resources:
      - group: "" # core
        resources: ["namespaces"]
  # Не логировать обращения к read-only URLs:
  - level: None
    nonResourceURLs:
      - /healthz*
      - /version
      - /swagger*
  # Не логировать сообщения, относящиеся к типу ресурсов “события”:
  - level: None
    resources:
      - group: "" # core
        resources: ["events"]
  # Ресурсы типа Secret, ConfigMap и TokenReview могут содержать  секретные данные,
  # поэтому логируем только метаданные связанных с ними запросов
  - level: Metadata
    resources:
      - group: "" # core
        resources: ["secrets", "configmaps"]
      - group: authentication.k8s.io
        resources: ["tokenreviews"]
  # Действия типа get, list и watch могут быть ресурсоёмкими; не логируем их
  - level: Request
    verbs: ["get", "list", "watch"]
    resources:
      - group: "" # core
      - group: "admissionregistration.k8s.io"
      - group: "apps"
      - group: "authentication.k8s.io"
      - group: "authorization.k8s.io"
      - group: "autoscaling"
      - group: "batch"
      - group: "certificates.k8s.io"
      - group: "extensions"
      - group: "networking.k8s.io"
      - group: "policy"
      - group: "rbac.authorization.k8s.io"
      - group: "settings.k8s.io"
      - group: "storage.k8s.io"
  # Уровень логирования по умолчанию для стандартных ресурсов API
  - level: RequestResponse
    resources:
      - group: "" # core
      - group: "admissionregistration.k8s.io"
      - group: "apps"
      - group: "authentication.k8s.io"
      - group: "authorization.k8s.io"
      - group: "autoscaling"
      - group: "batch"
      - group: "certificates.k8s.io"
      - group: "extensions"
      - group: "networking.k8s.io"
      - group: "policy"
      - group: "rbac.authorization.k8s.io"
      - group: "settings.k8s.io"
      - group: "storage.k8s.io"
  # Уровень логирования по умолчанию для всех остальных запросов
  - level: Metadata

Another good example of an audit policy is profile used in GCE.

For prompt response to audit events, it is possible to describe webhook. This question is covered in official documentationI will leave it outside the scope of this article.

Results

This article provides an overview of the basic security mechanisms in Kubernetes clusters that allow creating personalized user accounts, separating their rights, and logging their actions. I hope it will be useful to those who are faced with such issues in theory or already in practice. I also recommend that you familiarize yourself with the list of other materials on the topic of security in Kubernetes, which is given in “PS”, perhaps among them you will find the necessary details on the problems that are relevant to you.

PS

Read also on our blog:

• «33+ Kubernetes security tools";
• «An Introduction to Kubernetes Network Policies for Security Professionals";
• «Understanding RBAC in Kubernetes";
• «9 Kubernetes Security Best Practices";
• «11 Ways to (Not) Get Hacked in Kubernetes».

Source: habr.com