🥇Kubernetes authentication with GitHub OAuth and Dex

I present to your attention a tutorial for generating access to a Kubernetes cluster using Dex, dex-k8s-authenticator and GitHub.

Local meme from the Russian-language Kubernetes chat in Telegram

Introduction

We use Kubernetes to create dynamic environments for the development team and QA. So we want to give them access to the cluster for both the dashboard and kubectl. Unlike the same OpenShift, vanilla Kubernetes does not have native authentication, so we use third-party tools for this.

In this configuration we use:

dex-k8s-authenticator - web application for generating kubectl config
Dex - OpenID Connect provider
GitHub - simply because we use GitHub in our company

We tried to use Google OIDC, but unfortunately we failed to start them with groups, so the integration with GitHub suited us just fine. Without group mapping, it will not be possible to create group-based RBAC policies.

So, how does our Kubernetes authorization process work in a visual representation:

Authorization process

A little more detail and point by point:

User logs into dex-k8s-authenticator (login.k8s.example.com)
dex-k8s-authenticator redirects the request to Dex (dex.k8s.example.com)
Dex redirects to GitHub login page
GitHub generates the required authorization information and returns it to Dex
Dex passes the received information to dex-k8s-authenticator
User gets OIDC token from GitHub
dex-k8s-authenticator adds token to kubeconfig
kubectl passes token to KubeAPIServer
KubeAPIServer based on the passed token returns access to kubectl
User accesses from kubectl

Preparatory Actions

Of course, we already have a Kubernetes cluster installed (k8s.example.com), as well as HELM pre-installed. We also have an organization on GitHub (super-org).
If you don't have HELM, install it very simple.

First we need to set up GitHub.

Go to the organization settings page, (https://github.com/organizations/super-org/settings/applications) and create a new application (Authorized OAuth App):

Create a new app on GitHub

Fill in the fields with the required URLs, for example:

Homepage URL: https://dex.k8s.example.com
Authorization callback URL: https://dex.k8s.example.com/callback

Be careful with links, it is important not to lose slashes.

In response to the completed form, GitHub will generate Client ID и Client secret, save them in a safe place, they will be useful to us (for example, we use Vault for keeping secrets):

Client ID: 1ab2c3d4e5f6g7h8
Client secret: 98z76y54x32w1

Prepare DNS records for subdomains login.k8s.example.com и dex.k8s.example.com, as well as SSL certificates for ingresses.

Let's create SSL certificates:

cat <<EOF | kubectl create -f -
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: cert-auth-dex
  namespace: kube-system
spec:
  secretName: cert-auth-dex
  dnsNames:
    - dex.k8s.example.com
  acme:
    config:
    - http01:
        ingressClass: nginx
      domains:
      - dex.k8s.example.com
  issuerRef:
    name: le-clusterissuer
    kind: ClusterIssuer
---
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: cert-auth-login
  namespace: kube-system
spec:
  secretName: cert-auth-login
  dnsNames:
    - login.k8s.example.com
  acme:
    config:
    - http01:
        ingressClass: nginx
      domains:
      - login.k8s.example.com
  issuerRef:
    name: le-clusterissuer
    kind: ClusterIssuer
EOF
kubectl describe certificates cert-auth-dex -n kube-system
kubectl describe certificates cert-auth-login -n kube-system

ClusterIssuer with title le-clusterissuer should already exist, if not, create it using HELM:

helm install --namespace kube-system -n cert-manager stable/cert-manager
cat << EOF | kubectl create -f -
apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
  name: le-clusterissuer
  namespace: kube-system
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: [email protected]
    privateKeySecretRef:
      name: le-clusterissuer
    http01: {}
EOF

KubeAPIServer configuration

For kubeAPIServer to work, you need to configure OIDC and upgrade the cluster:

kops edit cluster
...
  kubeAPIServer:
    anonymousAuth: false
    authorizationMode: RBAC
    oidcClientID: dex-k8s-authenticator
    oidcGroupsClaim: groups
    oidcIssuerURL: https://dex.k8s.example.com/
    oidcUsernameClaim: email
kops update cluster --yes
kops rolling-update cluster --yes

We use since to expand clusters, but it works the same way for other cluster managers.

Dex configuration and dex-k8s-authenticator

For Dex to work, you need to have a certificate and a key from the Kubernetes master, we will pull it out from there:

sudo cat /srv/kubernetes/ca.{crt,key}
-----BEGIN CERTIFICATE-----
AAAAAAAAAAABBBBBBBBBBCCCCCC
-----END CERTIFICATE-----
-----BEGIN RSA PRIVATE KEY-----
DDDDDDDDDDDEEEEEEEEEEFFFFFF
-----END RSA PRIVATE KEY-----

Clone the dex-k8s-authenticator repository:

git clone [email protected]:mintel/dex-k8s-authenticator.git
cd dex-k8s-authenticator/

With the help of values-files, we can flexibly set up variables for our HELM charts.

Let's describe the configuration for Dex:

cat << EOF > values-dex.yml
global:
  deployEnv: prod
tls:
  certificate: |-
    -----BEGIN CERTIFICATE-----
    AAAAAAAAAAABBBBBBBBBBCCCCCC
    -----END CERTIFICATE-----
  key: |-
    -----BEGIN RSA PRIVATE KEY-----
    DDDDDDDDDDDEEEEEEEEEEFFFFFF
    -----END RSA PRIVATE KEY-----
ingress:
  enabled: true
  annotations:
    kubernetes.io/ingress.class: nginx
    kubernetes.io/tls-acme: "true"
  path: /
  hosts:
    - dex.k8s.example.com
  tls:
    - secretName: cert-auth-dex
      hosts:
        - dex.k8s.example.com
serviceAccount:
  create: true
  name: dex-auth-sa
config: |
  issuer: https://dex.k8s.example.com/
  storage: # https://github.com/dexidp/dex/issues/798
    type: sqlite3
    config:
      file: /var/dex.db
  web:
    http: 0.0.0.0:5556
  frontend:
    theme: "coreos"
    issuer: "Example Co"
    issuerUrl: "https://example.com"
    logoUrl: https://example.com/images/logo-250x25.png
  expiry:
    signingKeys: "6h"
    idTokens: "24h"
  logger:
    level: debug
    format: json
  oauth2:
    responseTypes: ["code", "token", "id_token"]
    skipApprovalScreen: true
  connectors:
  - type: github
    id: github
    name: GitHub
    config:
      clientID: $GITHUB_CLIENT_ID
      clientSecret: $GITHUB_CLIENT_SECRET
      redirectURI: https://dex.k8s.example.com/callback
      orgs:
      - name: super-org
        teams:
        - team-red
  staticClients:
  - id: dex-k8s-authenticator
    name: dex-k8s-authenticator
    secret: generatedLongRandomPhrase
    redirectURIs:
      - https://login.k8s.example.com/callback/
envSecrets:
  GITHUB_CLIENT_ID: "1ab2c3d4e5f6g7h8"
  GITHUB_CLIENT_SECRET: "98z76y54x32w1"
EOF

And for dex-k8s-authenticator:

cat << EOF > values-auth.yml
global:
  deployEnv: prod
dexK8sAuthenticator:
  clusters:
  - name: k8s.example.com
    short_description: "k8s cluster"
    description: "Kubernetes cluster"
    issuer: https://dex.k8s.example.com/
    k8s_master_uri: https://api.k8s.example.com
    client_id: dex-k8s-authenticator
    client_secret: generatedLongRandomPhrase
    redirect_uri: https://login.k8s.example.com/callback/
    k8s_ca_pem: |
      -----BEGIN CERTIFICATE-----
      AAAAAAAAAAABBBBBBBBBBCCCCCC
      -----END CERTIFICATE-----
ingress:
  enabled: true
  annotations:
    kubernetes.io/ingress.class: nginx
    kubernetes.io/tls-acme: "true"
  path: /
  hosts:
    - login.k8s.example.com
  tls:
    - secretName: cert-auth-login
      hosts:
        - login.k8s.example.com
EOF

Install Dex and dex-k8s-authenticator:

helm install -n dex --namespace kube-system --values values-dex.yml charts/dex
helm install -n dex-auth --namespace kube-system --values values-auth.yml charts/dex-k8s-authenticator

Let's check the serviceability of the services (Dex should return code 400, and dex-k8s-authenticator should return code 200):

curl -sI https://dex.k8s.example.com/callback | head -1
HTTP/2 400
curl -sI https://login.k8s.example.com/ | head -1
HTTP/2 200

RBAC configuration

Create a ClusterRole for the group, in our case with read-only access:

cat << EOF | kubectl create -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: cluster-read-all
rules:
  -
    apiGroups:
      - ""
      - apps
      - autoscaling
      - batch
      - extensions
      - policy
      - rbac.authorization.k8s.io
      - storage.k8s.io
    resources:
      - componentstatuses
      - configmaps
      - cronjobs
      - daemonsets
      - deployments
      - events
      - endpoints
      - horizontalpodautoscalers
      - ingress
      - ingresses
      - jobs
      - limitranges
      - namespaces
      - nodes
      - pods
      - pods/log
      - pods/exec
      - persistentvolumes
      - persistentvolumeclaims
      - resourcequotas
      - replicasets
      - replicationcontrollers
      - serviceaccounts
      - services
      - statefulsets
      - storageclasses
      - clusterroles
      - roles
    verbs:
      - get
      - watch
      - list
  - nonResourceURLs: ["*"]
    verbs:
      - get
      - watch
      - list
  - apiGroups: [""]
    resources: ["pods/exec"]
    verbs: ["create"]
EOF

Let's create a configuration for ClusterRoleBinding:

cat <<EOF | kubectl create -f -
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: dex-cluster-auth
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-read-all
subjects:
  kind: Group
  name: "super-org:team-red"
EOF

Now we are ready for testing.

Tests

Go to login pagehttps://login.k8s.example.com) and log in with a GitHub account:

Authorization page

Authorization page redirected to GitHub

Follow the generated instructions to gain access

After copy-pasting from the web page, we can use kubectl to manage our cluster resources:

kubectl get po
NAME                READY   STATUS    RESTARTS   AGE
mypod               1/1     Running   0          3d

kubectl delete po mypod
Error from server (Forbidden): pods "mypod" is forbidden: User "[email protected]" cannot delete pods in the namespace "default"

And it works, all GitHub users in our organization can see resources and log into pods, but they don't have permission to change them.

Source: habr.com

Kubernetes Authentication with GitHub OAuth and Dex