{"id":94977,"date":"2020-09-24T07:43:00","date_gmt":"2020-09-24T05:43:00","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/nashi-vyvody-za-god-migraczii-gitlab-com-na-kubernetes"},"modified":"2020-09-24T07:43:00","modified_gmt":"2020-09-24T05:43:00","slug":"nashi-vyvody-za-god-migraczii-gitlab-com-na-kubernetes","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/nashi-vyvody-za-god-migraczii-gitlab-com-na-kubernetes","title":{"rendered":"Our Findings on the Year of Migration of GitLab.com to Kubernetes","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

Note: translation.<\/b>: adapting Kubernetes in GitLab is considered one of the two main factors contributing to the company\u2019s growth. However, until recently, the infrastructure of the online service GitLab.com was built on virtual machines, and only about a year ago did the migration to K8s begin, which is still ongoing. We are pleased to present a translation of a recent article by a GitLab SRE engineer about how this is happening and what conclusions the engineers involved in the project are making.<\/i><\/p>\n

\"Our
\n
\nFor about a year now, our infrastructure department has been migrating all services operating on GitLab.com to Kubernetes. During this time, we have encountered issues related not only to moving services to Kubernetes but also to managing a hybrid deployment during the transition. This article will discuss the valuable lessons we have learned.<\/a><\/noindex><\/p>\n

From the very beginning, GitLab.com servers operated in the cloud on virtual machines. These virtual machines are managed by Chef, and their installation occurs through our official Linux package.<\/a><\/noindex>. Deployment Strategy<\/a><\/noindex> in case an application needs updating, consists of simply updating the server pool in a coordinated sequential manner using a CI pipeline. This method\u2014though slow and a bit tedious\u2014<\/a><\/noindex> ensures that GitLab.com applies the same installation and configuration methods as users of standalone (self-managed)<\/i> GitLab installations that use our Linux packages for this purpose.<\/p>\n

We use this method because it is crucial to experience all the joys and frustrations that ordinary community members face when installing and configuring their GitLab instances. This approach worked well for a while, but when the number of projects on GitLab exceeded 10 million, we realized it no longer met our needs for scaling and deployment.<\/p>\n

First Steps Towards Kubernetes and Cloud-Native GitLab<\/h2>\n

\nIn 2017, the GitLab Charts project was created.<\/a><\/noindex> to prepare GitLab for deployment in the cloud, as well as to allow users to install GitLab in Kubernetes clusters. At that time, we knew that migrating GitLab to Kubernetes would enhance the scalability of the SaaS platform, simplify deployments, and improve computational resource efficiency. However, many functions of our application depended on mounted NFS volumes, which slowed the transition from virtual machines.<\/p>\n

The push towards cloud-native and Kubernetes enabled our engineers to plan a gradual transition, during which we eliminated some of the application\u2019s dependencies on network storage while simultaneously continuing to develop new features. Since we began planning the migration in the summer of 2019, many of these constraints have been addressed, and the process of transitioning GitLab.com to Kubernetes is now in full swing!<\/p>\n

Features of GitLab.com running on Kubernetes<\/h2>\n

\nFor GitLab.com, we use a single regional GKE cluster that handles all application traffic. To minimize the complexity of the already intricate migration, we focus on services that do not rely on local storage or NFS. GitLab.com primarily utilizes a monolithic codebase on Rails, and we direct traffic based on workload characteristics to different endpoints, isolated into their own node pools.<\/p>\n

For the frontend, these types are divided into requests to web, API, Git SSH\/HTTPS, and Registry. For the backend, we split jobs in the queue based on various characteristics according to predetermined resource boundaries<\/a><\/noindex>, which allow us to set Service Level Objectives (SLOs) for various loads.<\/p>\n

All these services for GitLab.com are configured using an unmodified GitLab Helm chart. Configuration is done in sub-charts that can be selectively included as we gradually migrate services to the cluster. Even though it was decided not to include some of our stateful services, such as Redis, Postgres, GitLab Pages, and Gitaly, in the migration, using Kubernetes drastically reduces the number of VMs currently managed by Chef.<\/p>\n

Transparency and configuration management of Kubernetes<\/h2>\n

\nAll settings are managed by GitLab itself. We use three configuration projects based on Terraform and Helm for this purpose. We strive to use GitLab itself wherever possible to run GitLab, but for operational tasks, we operate a separate GitLab installation. This is necessary to avoid dependence on the availability of GitLab.com during deployments and updates of GitLab.com.<\/p>\n

Although our pipelines for the Kubernetes cluster operate on a separate GitLab installation, the code repositories have mirrors publicly accessible at the following addresses:<\/p>\n