We at GitLab handle software fixes in two ways - manually and automatically. Read on to learn more about the release manager's job of creating and delivering important updates with automatic deployment to gitlab.com, as well as fixes for users who work with their installations.
I recommend setting a reminder on your smartwatch: every month on the 22nd of the month, users working with GitLab at their facilities can see updates to the current version of our product. In the monthly release, you can find new features, enhancements to existing ones, and often see the end result of community tooling or merge requests.
But, as practice shows, software development is rarely without flaws. When a security bug or vulnerability is found, the release manager on the delivery team creates a patch for our users with their own setups. Gitlab.com is updated during the CD process. We call this CD process auto-deployment so that there is no confusion with the CD feature in GitLab. This process can include suggestions from pull requests submitted by users, clients, and our internal development team, so the boring problem of releasing patches is solved in two very different ways.
Β«We ensure that everything the developers have made is deployed daily to all environments before rolling it out to GitLab.comβ, explains , Senior Technical Manager, Infrastructure Department. "Think of releases for your installations like snapshots for a gitlab.com deployment, for which we've added separate build steps so that our users can use it to install on their facilitiesΒ».
Regardless of the bug or vulnerability, gitlab.com customers will receive fixes shortly after they are posted, which is a benefit of the automatic CD process. Fixes for users with custom installations require separate preparation by the release manager.
The delivery team is hard at work on automating most of the processes involved in creating releases to reduce (mean time to production, i.e. the time spent on production), the period of time from the processing of a merge request by the developer to deployment to gitlab.com.
Β«The purpose of the delivery team is to make sure we can work faster as a company, or at least speed up the delivery people, right?β says Marin.
Both gitlab.com customers and users of their installations benefit from the delivery team's efforts to reduce cycle times and speed up deployments. In this article, we will explain the similarities and differences between these two methods. , as well as how our delivery team prepares patches for in-house users, and how we keep gitlab.com up to date with automatic deployment.
What does a release manager do?
Team members monthly our releases to users at their facilities, including fixes and security releases that may occur between releases. They are also responsible for the company's transition to automatic, continuous deployment.
The in-house releases and the gitlab.com releases use similar workflows but run at different times, Marin explains.
First of all, the release manager, regardless of the type of release, ensures the availability and security of GitLab from the moment the application is launched on gitlab.com, including ensuring that the same problems do not fall into the infrastructure of customers with their own capacities.
Once a bug or vulnerability is marked fixed in GitLab, the release manager should evaluate that it will be included in patches or security updates for users with their installations. If he decides that a bug or vulnerability deserves an update, preparatory work begins.
The release manager has to decide whether to prepare a fix or when to deploy it - and this is highly dependent on the context of the situation, "in the meantime, machines are not as good at managing context as peopleMarin says.
All about fixes
What are fixes and why do we need them?
The release manager decides whether to release a fix based on the severity of the bug.
Errors vary according to their severity. So S4 or S3 errors can be stylistic, such as pixel or icon misalignment. It's just as important, but it doesn't have much of an impact on someone's workflow, which means it's unlikely that a fix will be created for these S3 or S4 bugs, Marin explains.
However, S1 or S2 vulnerabilities mean that the user must not upgrade to the latest version, or there is a significant bug affecting the user's workflow. If they get into the tracker, many users have encountered them, so the release manager immediately starts preparing the fix.
Once a patch for S1 or S2 vulnerabilities is ready, the release manager triggers the release of the patch.
For example, the GitLab 12.10.1 patch was created after several blocking issues were identified and the developers fixed the underlying issue that was causing them. The release manager evaluated the correctness of the assigned severity levels, and after confirmation, the process of releasing the fix was launched, which was ready within XNUMX hours after blocking problems were discovered.
When a lot of S4, S3 and S2 accumulate, the release manager looks at the context to determine the urgency of releasing a fix, and when a certain number of them are reached, they are all combined and released. Post-release fixes or security updates are briefly described in blog posts.
How the release manager creates patches
We use GitLab CI and other features such as our ChatOps to create patches. The release manager triggers a hotfix release by activating the ChatOps team on our internal channel #releases in Slack.
/chatops run release prepare 12.10.1ChatOps works in Slack to trigger different events, which are then processed and executed by GitLab. For example, the delivery team set up ChatOps to automate things to release patches.
Once the release manager launches the ChatOps team in Slack, the rest of the work happens automatically in GitLab using CICD. There is a two-way communication between ChatOps in Slack and GitLab during the release process as the release manager activates some of the main steps in the process.
The video below provides a technical process for preparing a patch for GitLab.
How gitlab.com auto-deploy works
The process and tools used to update gitlab.com are similar to those used to create patches. Updating gitlab.com requires less manual work from the release manager's point of view.
Instead of running a deployment using ChatOps, we use CI features like , with which the release manager can schedule certain actions to be performed at the required time. Instead of a manual process, there is a pipeline that runs periodically once an hour, which downloads the new changes made to GitLab projects, packages them and schedules deployment, and automatically starts testing, QA and other necessary steps.
βSo we have a lot of deployments running in different environments before gitlab.com, and after those environments are in good shape and testing shows good results, the release manager kicks off the gitlab.com deployment activities,β says Marin.
The CICD technology to support gitlab.com updates automates the whole process up to the point where the release manager has to manually start deploying the production environment to gitlab.com.
Marin details the process of updating gitlab.com in the video below.
What else does the delivery team do
The main difference between updating gitlab.com and releasing patches to clients at their own facilities is that the latter process requires more time and more manual work from the release manager.
βSometimes we delay releasing patches to customers with their installations because of reported issues, issues with tools, and because there are many nuances to consider when releasing a single patch,β says Marin.
One of the short-term goals of the delivery team is to reduce the amount of manual work on the part of the release manager to speed up the release. The team is working on simplifying, streamlining, and automating the release process to get rid of fixes for low-severity issues (S3 and S4, approx. translator). Focusing on speed is a key performance indicator: you need to reduce MTTP - the time from accepting a pull request to deploying the result to gitlab.com - from the current 50 hours to 8 hours.
The delivery team is also working on migrating gitlab.com to a Kubernetes-based infrastructure.
Editor's n.b.: If you have already heard about Kubernetes technology (and I have no doubt that you have heard), but have not yet felt it with your hands, I recommend participating in online intensives , which will be held September 28-30, and which will be held October 14-16. This will allow you to confidently navigate the technology and work with it.
They are two approaches that share the same goal: fast delivery of updates to both gitlab.com and clients at their facilities.
Any ideas or recommendations for us?
Everyone can take part in the development of GitLab, we welcome feedback from our readers. If you have ideas for our delivery team, feel free to with notice team: Delivery.
Source: habr.com