{"id":86902,"date":"2020-07-01T07:42:28","date_gmt":"2020-07-01T05:42:28","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/udalyaem-ustarevshuyu-feature-branch-v-kubernetes-klastere"},"modified":"2020-07-01T07:42:28","modified_gmt":"2020-07-01T05:42:28","slug":"udalyaem-ustarevshuyu-feature-branch-v-kubernetes-klastere","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/udalyaem-ustarevshuyu-feature-branch-v-kubernetes-klastere","title":{"rendered":"Removing the outdated feature branch in the Kubernetes cluster","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

\"Removing<\/p>\n

Hello! Feature branch<\/b> (aka deploy preview, review app) \u2014 this is when not only the master branch is deployed, but every pull request is also deployed to a unique URL. You can check if the code works in a production environment, and the feature can be demonstrated to other developers or product managers. While you are working in a pull request, each new commit deletes the current deploy for the old code, and a new deploy for the new code is launched. Questions may arise when you merge the pull request into the master branch. The feature branch is no longer needed, but the Kubernetes resources are still in the cluster.<\/p>\n

<\/a><\/noindex><\/p>\n

More about feature branches<\/h2>\n

<\/p>\n

One approach to creating feature branches in Kubernetes is to use namespaces. Briefly, the production configuration looks like this:<\/p>\n

<\/p>\n

kind: Namespace\napiVersion: v1\nmetadata:\n  name: habr-back-end\n...\n\nkind: Deployment\napiVersion: apps\/v1\nmetadata:\n  namespace: habr-back-end\nspec:\n  replicas: 3\n...<\/code><\/pre>\n
<\/p>\n
A namespace is created for the feature branch with its identifier (for instance, the pull request number) and some prefix\/suffix (for example, -pr-<\/b>):<\/p>\n
<\/p>\n
kind: Namespace\napiVersion: v1\nmetadata:\n  name: habr-back-end-pr-17\n...\n\nkind: Deployment\napiVersion: apps\/v1\nmetadata:\n  namespace: habr-back-end-pr-17\nspec:\n  replicas: 1\n...<\/code><\/pre>\n
<\/p>\n
Overall, I wrote Kubernetes Operator<\/b> (an application that has access to cluster resources), link to the project on Github<\/a><\/noindex>). It deletes namespaces related to old feature branches. In Kubernetes, if you delete a namespace, other resources in that namespace are also automatically deleted.<\/p>\n
<\/p>\n
$ kubectl get pods --all-namespaces | grep -e \"-pr-\"\nNAMESPACE            ... AGE\nhabr-back-end-pr-264 ... 4d8h\nhabr-back-end-pr-265 ... 5d7h<\/code><\/pre>\n
<\/p>\n
You can read about how to implement feature branches in the cluster here<\/a><\/noindex> and here<\/a><\/noindex>.<\/p>\n
<\/p>\n
Motivation<\/h2>\n
<\/p>\n
Let's look at a typical pull request lifecycle with continuous integration () so that when developers make changes to the code, they can be sure that nothing is broken.<\/code>):<\/p>\n
<\/p>\n
    \n
  1. We push a new commit to the branch.<\/li>\n
  2. During the build, linters and\/or tests are run.<\/li>\n
  3. Kubernetes configurations for the pull request are formed on the fly (for instance, its number is substituted into a ready template).<\/li>\n
  4. Using kubectl apply, the configurations are applied to the cluster (deploy).<\/li>\n
  5. The pull request is merged into the master branch.<\/li>\n<\/ol>\n
    <\/p>\n
    While you are working in a pull request, each new commit deletes the current deploy for the old code, and a new deploy for the new code is launched. But when the pull request is merged into the master branch, only the master branch will be built. As a result, we end up forgetting about the pull request, while its Kubernetes resources are still in the cluster.<\/p>\n
    <\/p>\n
    How to use<\/h2>\n
    <\/p>\n
    Install the project with the command below:<\/p>\n
    <\/p>\n
    $ kubectl apply -f https:\/\/raw.githubusercontent.com\/dmytrostriletskyi\/stale-feature-branch-operator\/master\/configs\/production.yml<\/code><\/pre>\n
    <\/p>\n
    Create a file with the following content and install via kubectl apply -f<\/code>:<\/p>\n
    <\/p>\n
    apiVersion: feature-branch.dmytrostriletskyi.com\/v1\nkind: StaleFeatureBranch\nmetadata:\n  name: stale-feature-branch\nspec:\n  namespaceSubstring: -pr-\n  afterDaysWithoutDeploy: 3<\/code><\/pre>\n
    <\/p>\n
    Parameter namespaceSubstring<\/b> needed to filter namespaces for pull requests from other namespaces. For example, if there are the following namespaces in the cluster: habr-back-end<\/code>, habr-front-end<\/code>, habr-back-end-pr-17<\/code>, habr-back-end-pr-33<\/code>, then the candidates for deletion will be habr-back-end-pr-17<\/code>, habr-back-end-pr-33<\/code>.<\/p>\n
    <\/p>\n
    Parameter afterDaysWithoutDeploy<\/b> needed to delete old namespaces. For instance, if a namespace is created 3 days 1 hour<\/code> ago, and the parameter specifies 3 days<\/code>, this namespace will be deleted. This works in reverse as well; if a namespace was created 2 days 23 hours<\/code> ago, and the parameter specifies 3 days<\/code>, this namespace will not be deleted.<\/p>\n
    <\/p>\n
    There is another parameter that determines how frequently to scan all namespaces and check for days without deployment \u2014 checkEveryMinutes<\/b>. By default, it is set to 30 minutes<\/code>.<\/p>\n
    <\/p>\n
    How it works<\/h2>\n
    <\/p>\n
    In practice, you will need:<\/p>\n
    <\/p>\n
      \n
    1. Docker<\/a><\/noindex> to work in an isolated environment.<\/li>\n
    2. Minikube<\/a><\/noindex> will launch a Kubernetes cluster locally.<\/li>\n
    3. kubectl<\/a><\/noindex> \u2014 command-line interface for managing the cluster.<\/li>\n<\/ol>\n
      <\/p>\n
      Launching a Kubernetes cluster locally:<\/p>\n
      <\/p>\n
      $ minikube start --vm-driver=docker\nminikube v1.11.0 on Darwin 10.15.5\nUsing the docker driver based on existing profile.\nStarting control plane node minikube in cluster minikube.<\/code><\/pre>\n
      <\/p>\n
      Specify kubectl<\/code> to use the local cluster by default:<\/p>\n
      <\/p>\n
      $ kubectl config use-context minikube\nSwitched to context \"minikube\".<\/code><\/pre>\n
      <\/p>\n
      Downloading configurations for the production environment:<\/p>\n
      <\/p>\n
      $ curl https:\/\/raw.githubusercontent.com\/dmytrostriletskyi\/stale-feature-branch-operator\/master\/configs\/production.yml > stale-feature-branch-production-configs.yml<\/code><\/pre>\n
      <\/p>\n
      Since the production configurations are set to check old namespaces, and in our newly created cluster there are none, we will replace the environment variable IS_DEBUG<\/code> to true<\/code>. With this value, the parameter afterDaysWithoutDeploy<\/code> is not taken into account and namespaces are not checked for days without deployment, only for substring inclusion (-pr-<\/code>).<\/p>\n
      <\/p>\n
      If you are in Linux<\/code>:<\/p>\n
      <\/p>\n
      $ sed -i 's|false|true|g' stale-feature-branch-production-configs.yml<\/code><\/pre>\n
      <\/p>\n
      If you are in macOS<\/code>:<\/p>\n
      <\/p>\n
      $ sed -i \"\" 's|false|true|g' stale-feature-branch-production-configs.yml<\/code><\/pre>\n
      <\/p>\n
      Installing the project:<\/p>\n
      <\/p>\n
      $ kubectl apply -f stale-feature-branch-production-configs.yml<\/code><\/pre>\n
      <\/p>\n
      Checking that the resource has appeared in the cluster StaleFeatureBranch<\/code>:<\/p>\n
      <\/p>\n
      $ kubectl api-resources | grep stalefeaturebranches\nNAME                 ... APIGROUP                             ... KIND\nstalefeaturebranches ... feature-branch.dmytrostriletskyi.com ... StaleFeatureBranch<\/code><\/pre>\n
      <\/p>\n
      Checking that the operator has appeared in the cluster:<\/p>\n
      <\/p>\n
      $ kubectl get pods --namespace stale-feature-branch-operator\nNAME                                           ... STATUS  ... AGE\nstale-feature-branch-operator-6bfbfd4df8-m7sch ... Running ... 38s<\/code><\/pre>\n
      <\/p>\n
      If you look at its logs, it is ready to process resources StaleFeatureBranch<\/code>:<\/p>\n
      <\/p>\n
      $ kubectl logs stale-feature-branch-operator-6bfbfd4df8-m7sch -n stale-feature-branch-operator\n... \"msg\":\"Operator Version: 0.0.1\"}\n...\n... \"msg\":\"Starting EventSource\", ... , \"source\":\"kind source: \/, Kind=\"}\n... \"msg\":\"Starting Controller\", ...}\n... \"msg\":\"Starting workers\", ..., \"worker count\":1}<\/code><\/pre>\n
      <\/p>\n
      Installing the prepared fixtures<\/code> (ready configurations for modeling cluster resources) for the resource StaleFeatureBranch<\/code>:<\/p>\n
      <\/p>\n
      $ kubectl apply -f https:\/\/raw.githubusercontent.com\/dmytrostriletskyi\/stale-feature-branch-operator\/master\/fixtures\/stale-feature-branch.yml<\/code><\/pre>\n
      <\/p>\n
      The configurations specify searching for namespaces with the substring -pr-<\/code> once in 1 minute<\/code>.:<\/p>\n
      <\/p>\n
      apiVersion: feature-branch.dmytrostriletskyi.com\/v1\nkind: StaleFeatureBranch\nmetadata:\n  name: stale-feature-branch\nspec:\n  namespaceSubstring: -pr-\n  afterDaysWithoutDeploy: 1 \n  checkEveryMinutes: 1<\/code><\/pre>\n
      <\/p>\n
      The operator reacted and is ready to check namespaces:<\/p>\n
      <\/p>\n
      $ kubectl logs stale-feature-branch-operator-6bfbfd4df8-m7sch -n stale-feature-branch-operator\n... \"msg\":\"Stale feature branch is being processed.\",\"namespaceSubstring\":\"-pr-\",\"afterDaysWithoutDeploy\":1,\"checkEveryMinutes\":1,\"isDebug\":\"true\"}<\/code><\/pre>\n
      <\/p>\n
      Installing fixtures<\/code>, containing two namespaces (project-pr-1<\/code>, project-pr-2<\/code>) and their deployments<\/code>, services<\/code>, ingress<\/code>, and so on:<\/p>\n
      <\/p>\n
      $ kubectl apply -f https:\/\/raw.githubusercontent.com\/dmytrostriletskyi\/stale-feature-branch-operator\/master\/fixtures\/first-feature-branch.yml -f https:\/\/raw.githubusercontent.com\/dmytrostriletskyi\/stale-feature-branch-operator\/master\/fixtures\/second-feature-branch.yml\n...\nnamespace\/project-pr-1 created\ndeployment.apps\/project-pr-1 created\nservice\/project-pr-1 created\nhorizontalpodautoscaler.autoscaling\/project-pr-1 created\nsecret\/project-pr-1 created\nconfigmap\/project-pr-1 created\nIngress.extensions\/project-pr-1 created\nnamespace\/project-pr-2 created\ndeployment.apps\/project-pr-2 created\nservice\/project-pr-2 created\nhorizontalpodautoscaler.autoscaling\/project-pr-2 created\nsecret\/project-pr-2 created\nconfigmap\/project-pr-2 created\nIngress.extensions\/project-pr-2 created<\/code><\/pre>\n
      <\/p>\n
      Check that all the above resources have been successfully created:<\/p>\n
      <\/p>\n
      $ kubectl get namespace,pods,deployment,service,horizontalpodautoscaler,configmap,ingress -n project-pr-1 && kubectl get namespace,pods,deployment,service,horizontalpodautoscaler,configmap,ingress -n project-pr-2\n...\nNAME                              ... READY ... STATUS  ... AGE\npod\/project-pr-1-848d5fdff6-rpmzw ... 1\/1   ... Running ... 67s\n\nNAME                         ... READY ... AVAILABLE ... AGE\ndeployment.apps\/project-pr-1 ... 1\/1   ... 1         ... 67s\n...<\/code><\/pre>\n
      <\/p>\n
      Since we have enabled debug<\/code>, namespaces project-pr-1<\/code> and project-pr-2<\/code>, consequently, all other resources should be deleted immediately without considering the parameter afterDaysWithoutDeploy<\/code>. This is evident in the operator's logs:<\/p>\n
      <\/p>\n
      $ kubectl logs stale-feature-branch-operator-6bfbfd4df8-m7sch -n stale-feature-branch-operator\n... \"msg\":\"Namespace should be deleted because debug mode is enabled.\",\"namespaceName\":\"project-pr-1\"}\n... \"msg\":\"Namespace is being processed.\",\"namespaceName\":\"project-pr-1\",\"namespaceCreationTimestamp\":\"2020-06-16 18:43:58 +0300 EEST\"}\n... \"msg\":\"Namespace has been deleted.\",\"namespaceName\":\"project-pr-1\"}\n... \"msg\":\"Namespace should be deleted because debug mode is enabled.\",\"namespaceName\":\"project-pr-2\"}\n... \"msg\":\"Namespace is being processed.\",\"namespaceName\":\"project-pr-2\",\"namespaceCreationTimestamp\":\"2020-06-16 18:43:58 +0300 EEST\"}\n... \"msg\":\"Namespace has been deleted.\",\"namespaceName\":\"project-pr-2\"}<\/code><\/pre>\n
      <\/p>\n
      If you check for the existence of resources, they will be in the status Terminating<\/code> (deletion process) or already deleted (the command output is empty).<\/p>\n
      <\/p>\n
      $ kubectl get namespace,pods,deployment,service,horizontalpodautoscaler,configmap,ingress -n project-pr-1 && kubectl get namespace,pods,deployment,service,horizontalpodautoscaler,configmap,ingress -n project-pr-2\n...<\/code><\/pre>\n
      <\/p>\n
      You can repeat the creation process fixtures<\/code> multiple times and ensure they will be deleted within a minute.<\/p>\n
      <\/p>\n
      Alternatives<\/h2>\n
      <\/p>\n
      What can be done instead of an operator that works in a cluster? There are several approaches, all of which are imperfect (and their shortcomings are subjective), and everyone decides what works best for their particular project:<\/p>\n
      <\/p>\n
        \n
      1. \n
        Delete the feature branch during the build of the master branch in continuous integration.<\/p>\n
        <\/p>\n