🥇The (Almost) Complete Beginner's Guide to CI/CD in GitLab

Or how to get beautiful badges for your project in one evening of easy coding

Probably, every developer who has at least one pet project at some point has an itch about beautiful badges with statuses, code coverage, package versions in nuget ... And this itch led me to write this article. In preparation for writing it, I got this beauty in one of my projects:

This article will walk you through the basic setup of continuous integration and delivery for a .Net Core class library project in GitLab, publishing documentation to GitLab Pages, and pushing built packages to a private feed in Azure DevOps.

VS Code was used as the development environment with the extension GitLab Workflow (for validating the settings file directly from the development environment).

Brief introduction

CD - is it when you just pushed, and everything has already fallen on the client?

What is CI / CD and why you need it - you can easily google it. Find complete documentation on configuring pipelines in GitLab also easy. Here I will briefly and, if possible, without flaws describe the process of the system from a bird's eye view:

the developer sends a commit to the repository, creates a merge request through the site, or in some other way, explicitly or implicitly starts the pipeline,
all tasks are selected from the configuration, the conditions of which allow them to be launched in the given context,
tasks are organized according to their stages,
stages are executed in turn - i.e. side-by-side all tasks of this stage are completed,
if the stage fails (i.e., at least one of the tasks of the stage fails), the pipeline stops (almost always),
if all stages are completed successfully, the pipeline is considered successful.

Thus, we have:

pipeline - a set of tasks organized into stages in which you can build, test, package code, deploy a finished build to a cloud service, etc.,
stage (training) — pipeline organization unit, contains 1+ task,
task (job) is a unit of work in the pipeline. It consists of a script (mandatory), launch conditions, settings for publishing/caching artifacts, and much more.

Accordingly, the task when setting up CI / CD comes down to creating a set of tasks that implement all the necessary actions for building, testing and publishing code and artifacts.

Before starting: why?

Why Gitlab?

Because when it became necessary to create private repositories for pet projects, they were paid on GitHub, and I was greedy. The repositories have become free, but so far this is not enough reason for me to move to GitHub.

Why not Azure DevOps Pipelines?

Because there the setting is elementary - knowledge of the command line is not even required. Integration with external git providers - in a couple of clicks, import of SSH keys to send commits to the repository - too, the pipeline is easily configured even not from a template.

Starting position: what you have and what you want

We have:

repository in GitLab.

We want:

automatic assembly and testing for each merge request,
building packages for each merge request and pushing to the master, provided that there is a certain line in the commit message,
sending built packages to a private feed in Azure DevOps,
assembly of documentation and publication in GitLab Pages,
badges!11

The described requirements organically fall on the following pipeline model:

Stage 1 - Assembly
- We collect the code, publish the output files as artifacts
Stage 2 - testing
- We get artifacts from the build stage, run tests, collect code coverage data
Stage 3 - Submit
- Task 1 - build the nuget package and send it to Azure DevOps
- Task 2 - we collect the site from xmldoc in the source code and publish it in GitLab Pages

Let's get started!

Collecting the configuration

Preparing accounts

Create an account in Microsoft Azure
We pass to Azure DevOps
We create a new project
1. Name - any
2. Visibility - any
When you click on the Create button, the project will be created and you will be redirected to its page. On this page, you can disable unnecessary features by going to the project settings (lower link in the list on the left -> Overview -> Azure DevOps Services block)
Go to Atrifacts, click Create feed
1. Enter the name of the source
2. Choose visibility
3. Uncheck Include packages from common public sources, so that the source does not turn into a dump nuget clone
Click Connect to feed, select Visual Studio, copy Source from the Machine Setup block
Go to account settings, select Personal Access Token
Create a new access token
1. Name - arbitrary
2. Organization - Current
3. Valid for a maximum of 1 year
4. Scope - Packaging/Read & Write
Copy the created token - after the modal window is closed, the value will be unavailable
Go to the repository settings in GitLab, select the CI / CD settings
Expand the Variables block, add a new one
1. Name - any without spaces (will be available in the command shell)
2. Value - access token from paragraph 9
3. Select Mask variable

This completes the pre-configuration.

Preparing the configuration framework

By default, CI/CD configuration in GitLab uses the file .gitlab-ci.yml from the root of the repository. You can set an arbitrary path to this file in the repository settings, but in this case it is not necessary.

As you can see from the extension, the file contains a configuration in the format YAML. The documentation details which keys can be contained at the top level of the configuration, and at each of the nested levels.

First, let's add a link to the docker image in the configuration file, in which the tasks will be performed. For this we find .Net Core images page on Docker Hub. As for the GitHub there is a detailed guide on which image to choose for different tasks. An image with .Net Core 3.1 is suitable for us to build, so feel free to add the first line to the configuration

image: mcr.microsoft.com/dotnet/core/sdk:3.1

Now, when the pipeline is launched from the Microsoft image repository, the specified image will be downloaded, in which all tasks from the configuration will be executed.

The next step is to add training's. By default, GitLab defines 5 stages:

.pre - performed up to all stages,
.post - performed after all stages,
build - first after .pre stage,
test - second phase,
deploy - the third stage.

Nothing prevents you from declaring them explicitly, however. The order in which the steps are listed affects the order in which they are performed. For completeness, let's add to the configuration:

stages:
  - build
  - test
  - deploy

For debugging, it makes sense to get information about the environment in which the tasks are executed. Let's add a global set of commands that will be executed before each task with before_script:

before_script:
  - $PSVersionTable.PSVersion
  - dotnet --version
  - nuget help | select-string Version

It remains to add at least one task so that the pipeline starts when the commits are sent. For now, let's add an empty task to demonstrate:

dummy job:
  script:
    - echo ok

We start validation, we get a message that everything is fine, we commit, we push, we look at the results on the site ... And we get a script error - bash: .PSVersion: command not found. wtf?

Everything is logical - by default, runners (responsible for executing task scripts and provided by GitLab) use bash to execute commands. You can fix this by explicitly specifying in the task description what tags the executing pipeline runner should have:

dummy job on windows:
  script:
    - echo ok
  tags:
    - windows

Great! The pipeline is now running.

An attentive reader, having repeated the indicated steps, will notice that the task was completed in the stage test, although we did not specify the stage. As you might guess test is the default step.

Let's continue creating the configuration skeleton by adding all the tasks described above:

build job:
  script:
    - echo "building..."
  tags:
    - windows
  stage: build

test and cover job:
  script:
    - echo "running tests and coverage analysis..."
  tags:
    - windows
  stage: test

pack and deploy job:
  script:
    - echo "packing and pushing to nuget..."
  tags:
    - windows
  stage: deploy

pages:
  script:
    - echo "creating docs..."
  tags:
    - windows
  stage: deploy

We got a not particularly functional, but nevertheless correct pipeline.

Setting up triggers

Due to the fact that no trigger filters are specified for any of the tasks, the pipeline will fully be executed every time a commit is pushed to the repository. Since this is not the desired behavior in general, we will set up trigger filters for tasks.

Filters can be configured in two formats: only/except и rules. Briefly, only/except allows you to configure filters by triggers (merge_request, for example - sets the task to be executed each time a pull request is created and each time commits are sent to the branch that is the source in the merge request) and branch names (including using regular expressions); rules allows you to customize a set of conditions and, optionally, change the task execution condition depending on the success of previous tasks (when in GitLab CI/CD).

Let's recall a set of requirements - assembly and testing only for merge request, packaging and sending to Azure DevOps - for merge request and pushes to the master, documentation generation - for pushes to the master.

First, let's set up the code build task by adding a rule that fires only on merge request:

build job:
  # snip
  only:
    - merge_request

Now let's set up the packaging task to fire on the merge request and add commits to the master:

pack and deploy job:
  # snip
  only:
    - merge_request
    - master

As you can see, everything is simple and straightforward.

You can also set the task to fire only if a merge request is created with a specific target or source branch:

  rules:
    - if: $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "master"

Under conditions, you can use variables listed here; rules rules incompatible with the rules only/except.

Configuring Artifact Saving

During a task build job we will have build artifacts that can be reused in subsequent tasks. To do this, you need to add the paths to the task configuration, the files along which you will need to save and reuse in the following tasks, to the key artifacts:

build job:
  # snip
  artifacts:
    paths:
      - path/to/build/artifacts
      - another/path
      - MyCoolLib.*/bin/Release/*

Paths support wildcards, which definitely makes them easier to set.

If a task creates artifacts, then each subsequent task will be able to access them - they will be located along the same paths relative to the repository root that were collected from the original task. Artifacts are also available for download on the site.

Now that we have a configuration framework ready (and tested), we can proceed to actually writing scripts for tasks.

We write scripts

Perhaps, once upon a time, in a galaxy far, far away, building projects (including those on .net) from the command line was a pain. Now you can build, test and publish the project in 3 teams:

dotnet build
dotnet test
dotnet pack

Naturally, there are some nuances due to which we will complicate the commands somewhat.

We want a release build, not a debug build, so we add to each command -c Release
When testing, we want to collect code coverage data, so we need to include a coverage analyzer in the test libraries:
1. Add the package to all test libraries coverlet.msbuild: dotnet add package coverlet.msbuild from project folder
2. Add to the test run command /p:CollectCoverage=true
3. Add a key to the test task configuration to get coverage results (see below)
When packing the code into nuget packages, set the output directory for the packages: -o .

Collecting code coverage data

After running the tests, Coverlet prints run statistics to the console:

Calculating coverage result...
  Generating report 'C:Usersxxxsourcereposmy-projectmyProject.testscoverage.json'

+-------------+--------+--------+--------+
| Module      | Line   | Branch | Method |
+-------------+--------+--------+--------+
| project 1   | 83,24% | 66,66% | 92,1%  |
+-------------+--------+--------+--------+
| project 2   | 87,5%  | 50%    | 100%   |
+-------------+--------+--------+--------+
| project 3   | 100%   | 83,33% | 100%   |
+-------------+--------+--------+--------+

+---------+--------+--------+--------+
|         | Line   | Branch | Method |
+---------+--------+--------+--------+
| Total   | 84,27% | 65,76% | 92,94% |
+---------+--------+--------+--------+
| Average | 90,24% | 66,66% | 97,36% |
+---------+--------+--------+--------+

GitLab allows you to specify a regular expression to get statistics, which can then be obtained in the form of a badge. The regular expression is specified in the task settings with the key coverage; the expression must contain a capture group, the value of which will be passed to the badge:

test and cover job:
  # snip
  coverage: /|s*Totals*|s*(d+[,.]d+%)/

Here we get statistics from a line with total line coverage.

Publish packages and documentation

Both actions are scheduled for the last stage of the pipeline - since the assembly and tests have passed, we can share our developments with the world.

First, consider publishing to the package source:

If the project does not have a nuget configuration file (nuget.config), create a new one: dotnet new nugetconfig
Why: the image may not have write access to global (user and machine) configurations. In order not to catch errors, we simply create a new local configuration and work with it.
Let's add a new package source to the local configuration: nuget sources add -name <name> -source <url> -username <organization> -password <gitlab variable> -configfile nuget.config -StorePasswordInClearText
1. name - local source name, not critical
2. url - URL of the source from the stage "Preparing accounts", p. 6
3. organization - organization name in Azure DevOps
4. gitlab variable - the name of the variable with the access token added to GitLab ("Preparing accounts", p. 11). Naturally, in the format $variableName
5. -StorePasswordInClearText - a hack to bypass the access denied error (I'm not the first to step on this rake)
6. In case of errors, it may be useful to add -verbosity detailed
Sending the package to the source: nuget push -source <name> -skipduplicate -apikey <key> *.nupkg
1. We send all packages from the current directory, so *.nupkg.
2. name - from the step above.
3. key - any line. In Azure DevOps, in the Connect to feed window, the example is always the line az.
4. -skipduplicate - when trying to send an already existing package without this key, the source will return an error 409 Conflict; with the key, sending will be skipped.

Now let's set up the creation of documentation:

First, in the repository, in the master branch, we initialize the docfx project. To do this, run the command from the root docfx init and interactively set the key parameters for building documentation. Detailed description of the minimum project setup here.
1. When configuring, it is important to specify the output directory ..public - GitLab by default takes the contents of the public folder in the root of the repository as a source for Pages. Because the project will be located in a folder nested in the repository - add an output to the level up in the path.
Let's push the changes to GitLab.
Add a task to the pipeline configuration pages (reserved word for site publishing tasks in GitLab Pages):
1. Script:
  1. nuget install docfx.console -version 2.51.0 - install docfx; the version is specified to ensure that the package installation paths are correct.
  2. .docfx.console.2.51.0toolsdocfx.exe .docfx_projectdocfx.json - collecting documentation
2. Node artifacts:

pages:
  # snip
  artifacts:
    paths:
      - public

Lyrical digression about docfx

Previously, when setting up a project, I specified the code source for the documentation as a solution file. The main disadvantage is that documentation is also created for test projects. In case this is not necessary, you can set this value to the node metadata.src:

{
  "metadata": [
    {
      "src": [
        {
          "src": "../",
          "files": [
            "**/*.csproj"
          ],
          "exclude":[
            "*.tests*/**"
          ]
        }
      ],
      // --- snip ---
    },
    // --- snip ---
  ],
  // --- snip ---
}

metadata.src.src: "../" - we go one level up relative to the location docfx.json, because in patterns, searching up the directory tree does not work.
metadata.src.files: ["**/*.csproj"] - a global pattern, we collect all C # projects from all directories.
metadata.src.exclude: ["*.tests*/**"] - global pattern, exclude everything from folders with .tests In the title

Subtotal

Such a simple configuration can be created in just half an hour and a couple of cups of coffee, which will allow you to check that the code is built and the tests pass, build a new package, update the documentation and please the eye with beautiful badges in the README of the project with each merge request and sending to the master.

Final .gitlab-ci.yml

image: mcr.microsoft.com/dotnet/core/sdk:3.1

before_script:
  - $PSVersionTable.PSVersion
  - dotnet --version
  - nuget help | select-string Version

stages:
  - build
  - test
  - deploy

build job:
  stage: build
  script:
    - dotnet build -c Release
  tags:
    - windows
  only:
    - merge_requests
    - master
  artifacts:
    paths:
      - your/path/to/binaries

test and cover job:
  stage: test
  tags:
    - windows
  script:
    - dotnet test -c Release /p:CollectCoverage=true
  coverage: /|s*Totals*|s*(d+[,.]d+%)/
  only:
    - merge_requests
    - master

pack and deploy job:
  stage: deploy
  tags:
    - windows
  script:
    - dotnet pack -c Release -o .
    - dotnet new nugetconfig
    - nuget sources add -name feedName -source https://pkgs.dev.azure.com/your-organization/_packaging/your-feed/nuget/v3/index.json -username your-organization -password $nugetFeedToken -configfile nuget.config -StorePasswordInClearText
    - nuget push -source feedName -skipduplicate -apikey az *.nupkg
  only:
    - master

pages:
  tags:
    - windows
  stage: deploy
  script:
    - nuget install docfx.console -version 2.51.0
    - $env:path = "$env:path;$($(get-location).Path)"
    - .docfx.console.2.51.0toolsdocfx.exe .docfxdocfx.json
  artifacts:
    paths:
      - public
  only:
    - master

Speaking of badges

Because of them, after all, everything was started!

Badges with pipeline statuses and code coverage are available in GitLab in the CI/CD settings in the Gtntral pipelines block:

I created a badge with a link to the documentation on the platform shields.io - everything is quite straightforward there, you can create your own badge and receive it using a request.

![Пример с Shields.io](https://img.shields.io/badge/custom-badge-blue)

Azure DevOps Artifacts also allows you to create badges for packages with the latest version. To do this, in the source on the Azure DevOps site, you need to click on Create badge for the selected package and copy the markdown markup:

Adding beauty

Highlighting Common Configuration Fragments

While writing the configuration and searching through the documentation, I came across an interesting feature of YAML - reusing fragments.

As you can see from the task settings, they all require the tag windows at the runner, and are triggered when a merge request is sent to the master/created (except documentation). Let's add this to the fragment that we will reuse:

.common_tags: &common_tags
  tags:
    - windows
.common_only: &common_only
  only:
    - merge_requests
    - master

And now we can insert the fragment declared earlier in the task description:

build job:
  <<: *common_tags
  <<: *common_only

Fragment names must begin with a dot, so as not to be interpreted as a task.

Package versioning

When creating a package, the compiler checks the command line switches, and in their absence, the project files; when it finds a Version node, it takes its value as the version of the package being built. It turns out that in order to build a package with a new version, you need to either update it in the project file or pass it as a command line argument.

Let's add one more Wishlist - let the minor two numbers in the version be the year and build date of the package, and add prerelease versions. Of course, you can add this data to the project file and check before each submission - but you can also do it in the pipeline, collecting the package version from the context and passing it through the command line argument.

Let's agree that if the commit message contains a line like release (v./ver./version) <version number> (rev./revision <revision>)?, then we will take the version of the package from this line, supplement it with the current date and pass it as an argument to the command dotnet pack. In the absence of a line, we simply will not collect the package.

The following script solves this problem:

# регулярное выражение для поиска строки с версией
$rx = "releases+(v.?|ver.?|version)s*(?<maj>d+)(?<min>.d+)?(?<rel>.d+)?s*((rev.?|revision)?s+(?<rev>[a-zA-Z0-9-_]+))?"
# ищем строку в сообщении коммита, передаваемом в одной из предопределяемых GitLab'ом переменных
$found = $env:CI_COMMIT_MESSAGE -match $rx
# совпадений нет - выходим
if (!$found) { Write-Output "no release info found, aborting"; exit }
# извлекаем мажорную и минорную версии
$maj = $matches['maj']
$min = $matches['min']
# если строка содержит номер релиза - используем его, иначе - текущий год
if ($matches.ContainsKey('rel')) { $rel = $matches['rel'] } else { $rel = ".$(get-date -format "yyyy")" }
# в качестве номера сборки - текущие месяц и день
$bld = $(get-date -format "MMdd")
# если есть данные по пререлизной версии - включаем их в версию
if ($matches.ContainsKey('rev')) { $rev = "-$($matches['rev'])" } else { $rev = '' }
# собираем единую строку версии
$version = "$maj$min$rel.$bld$rev"
# собираем пакеты
dotnet pack -c Release -o . /p:Version=$version

Adding a script to a task pack and deploy job and observe the assembly of packages strictly in the presence of a given string in the commit message.

Summary

After spending about half an hour or an hour writing the configuration, debugging in the local powershell and, possibly, a couple of unsuccessful launches, we got a simple configuration for automating routine tasks.

Of course, GitLab CI / CD is much more extensive and multifaceted than it might seem after reading this guide - this is completely wrong. There even Auto DevOps isallowing

automatically detect, build, test, deploy, and monitor your applications

Now the plans are to configure a pipeline for deploying applications to Azure, using Pulumi and automatically determining the target environment, which will be covered in the next article.

Source: habr.com