Dynamic assembly and deployment of Docker images with werf using versioned documentation site as an example

We have shared information about our GitOps tool multiple times. werf, and this time we would like to share our experience in building a website along with the documentation of the project itself — werf.io (its Russian version is — ru.werf.io). This is a regular static website, but its construction is interesting because it is built using a dynamic number of artifacts.

We won't delve into the details of the website structure: generating a common menu for all versions, pages with information about releases, etc. Instead, let's focus on the questions and features of dynamic assembly and a bit on related CI/CD processes.

Introduction: How the Website Works

To begin with, the documentation for werf is stored alongside its code. This imposes certain requirements on development that generally go beyond the scope of this article, but at the very least, we can say that:

• New features of werf must not be released without updating the documentation, and vice versa, any changes in the documentation imply the release of a new version of werf;
• The project has quite intensive development: new versions can be released several times a day;
• Any manual operations for deploying the website with the new version of documentation are at least tedious;
• The project adopts a semantic approach versioning, with 5 stability channels. The release process implies a sequential passage of versions through the channels in order of increasing stability: from alpha to rock-solid;
• The website has a Russian version that "lives and develops" (i.e., its content is updated) in parallel with the primary (i.e., English) version.

To hide all this "internal work" from the user while offering them something that "just works", we created a separate tool for installing and updating werf — this is multiwerf. Just specify the release number and the stability channel you are willing to use, and multiwerf will check if there is a new version available on the channel and download it if necessary.

In the version selection menu on the website, the latest versions of werf are available in each channel. By default, at the address werf.io/documentation , the version of the most stable channel for the latest release opens — which is also indexed by search engines. Documentation for the channel is available at separate addresses (for example, werf.io/v1.0-beta/documentation for the beta release 1.0).

In total, the website has the following versions available:

1. root (opens by default),
2. for each active update channel of each release (for example, werf.io/v1.0-beta).

To generate a specific version of the site in general, it is sufficient to compile it using Jekyll, running the corresponding command in the /docs werf repository (jekyll build), switching to the Git tag of the required version beforehand.

It only remains to add that:

• the utility itself (werf) is used for the build;
• CI/CD processes are built on GitLab CI;
• and all of this, of course, works in Kubernetes.

Since I have already learned to "somewhat" port QEMU to JavaScript, this time it was decided to do it wisely and not repeat past mistakes.

Now let's formulate tasks that take into account all the described specifics:

1. After changing the version of werf on any update channel, the documentation on the site must be automatically updated..
2. For development, it needs to be possible to sometimes view preview versions of the site..

Recompilation of the site must be performed after changing the version on any channel from the corresponding Git tags, but during the image build process, we will encounter the following features:

• Since the list of versions on the channels changes, it is necessary to rebuild only the documentation for the channels where the version has changed. After all, rebuilding everything from scratch is not very elegant.
• The set of channels for releases may change. At some point in time, for example, there may not be a version on the channels more stable than the early-access release 1.1, but over time they will appear — should we manually change the build in this case?

This means that the build depends on changing external data..

Implementation

Selection of the approach

As an option, you can run each necessary version in a separate pod in Kubernetes. This option implies a larger number of objects in the cluster, which will grow as the number of stable releases of werf increases. This, in turn, implies more complex maintenance: each version has its own HTTP server, albeit with a small load. Of course, this incurs greater resource costs.

We chose the path of building all the necessary versions in one image. The compiled static files for all site versions are located in a container with NGINX, and traffic to the corresponding Deployment comes through NGINX Ingress. A simple structure — stateless application — allows easy scaling of the Deployment (depending on the load) using Kubernetes itself.

To be more precise, we create two images: one for the production environment and the second as an additional one for the dev environment. The additional image is used (launched) only in the dev environment together with the main one and contains the version of the site from the review commit, with routing between them carried out using Ingress resources.

werf vs git clone and artifacts

As mentioned earlier, to generate the static site for a specific version of the documentation, you need to build it by switching to the corresponding tag in the repository. This could also be done by cloning the repository each time during the build, selecting the appropriate tags from the list. However, this is quite a resource-intensive operation and also requires writing non-trivial instructions... Another significant drawback is that with this approach, there is no opportunity to cache anything during the build.

Here, the werf utility comes to our aid, implementing smart caching and allowing the use of external repositories. Using werf to add code from the repository will significantly speed up the build since werf essentially clones the repository once, and then performs only fetch as needed. Additionally, when adding data from the repository, we can select only the necessary directories (in our case, it is the directory docs), which significantly reduces the amount of data added.

Since Jekyll is a tool meant for compiling static sites and is not needed in the final image, it would make sense to perform the compilation in the werf artifact, and import only the compilation result into the final image. Writing werf.yaml.

So, we have decided that we will compile each version in a separate werf artifact. However, we

do not know how many of these artifacts will exist during the build , so we cannot write a fixed build configuration (strictly speaking, we could, but it wouldn't be very efficient).werf allows the use of

Go templates in its configuration file ( ), which allows us towerf.yamlgenerate the config "on the fly" depending on external data (just what we need!). The external data in our case is information about versions and releases, based on which we compile the necessary number of artifacts and end up with two images: werf-doc werf-dev and werf-dev to be run on different environments.

External data is passed through environment variables. Here is their composition:

• RELEASES — a string listing the releases and their corresponding current version of werf, formatted as a space-separated list of values in the form of %. Example: 1.0%v1.0.4-beta.20
• CHANNELS — a string listing the channels and their corresponding current version of werf, formatted as a space-separated list of values in the form of %. Example: 1.0-beta%v1.0.4-beta.20 1.0-alpha%v1.0.5-alpha.22
• ROOT_VERSION — the release version of werf to be displayed by default on the site (it's not always necessary to show documentation for the highest release number). Example: v1.0.4-beta.20
• REVIEW_SHA — the hash of the review commit from which the version for the testing environment should be assembled.

These variables will be filled in the GitLab CI pipeline, and how exactly is described below.

First of all, for convenience, we will define in werf.yaml Go template variables, assigning them values from the environment variables:

{{ $_ := set . "WerfVersions" (cat (env "CHANNELS") (env "RELEASES") | splitList " ") }}
{{ $Root := . }}
{{ $_ := set . "WerfRootVersion" (env "ROOT_VERSION") }}
{{ $_ := set . "WerfReviewCommit" (env "REVIEW_SHA") }}

The artifact description for compiling the static version of the site is generally the same for all our necessary cases (including generating the root version and the version for the dev environment). Therefore, we will extract it into a separate block using the function define — for subsequent reuse using include. The following arguments will be passed to the template:

• Version — the generated version (tag name);
• Channel — the name of the update channel for which the artifact is generated;
• Commit — the commit hash if the artifact is generated for a review commit;
• context.

Artifact template description

{{- define "doc_artifact" -}}
{{- $Root := index . "Root" -}}
artifact: doc-{{ .Channel }}
from: jekyll/builder:3
mount:
- from: build_dir
  to: /usr/local/bundle
ansible:
  install:
  - shell: |
      export PATH=/usr/jekyll/bin/:$PATH
  - name: "Install Dependencies"
    shell: bundle install
    args:
      executable: /bin/bash
      chdir: /app/docs
  beforeSetup:
{{- if .Commit }}
  - shell: echo "Review SHA - {{ .Commit }}."
{{- end }}
{{- if eq .Channel "root" }}
  - name: "releases.yml HASH: {{ $Root.Files.Get "releases.yml" | sha256sum }}"
    copy:
      content: |
{{ $Root.Files.Get "releases.yml" | indent 8 }}
      dest:  /app/docs/_data/releases.yml
{{- else }}
  - file:
      path: /app/docs/_data/releases.yml
      state: touch
{{- end }}
  - file:
      path: "{{`{{ item }}`}}"
      state: directory
      mode: 0777
    with_items:
    - /app/main_site/
    - /app/ru_site/
  - file:
      dest: /app/docs/pages_ru/cli
      state: link
      src: /app/docs/pages/cli
  - shell: |
      echo -e "werfVersion: {{ .Version }}\nwerfChannel: {{ .Channel }}" > /tmp/_config_additional.yml
      export PATH=/usr/jekyll/bin/:$PATH
{{- if and (ne .Version "review") (ne .Channel "root") }}
{{- $_ := set . "BaseURL" ( printf "v%s" .Channel ) }}
{{- else if ne .Channel "root" }}
{{- $_ := set . "BaseURL" .Channel }}
{{- end }}
      jekyll build -s /app/docs  -d /app/_main_site/{{ if .BaseURL }} --baseurl /{{ .BaseURL }}{{ end }} --config /app/docs/_config.yml,/tmp/_config_additional.yml
      jekyll build -s /app/docs  -d /app/_ru_site/{{ if .BaseURL }} --baseurl /{{ .BaseURL }}{{ end }} --config /app/docs/_config.yml,/app/docs/_config_ru.yml,/tmp/_config_additional.yml
    args:
      executable: /bin/bash
      chdir: /app/docs
git:
- url: https://github.com/flant/werf.git
  to: /app/
  owner: jekyll
  group: jekyll
{{- if .Commit }}
  commit: {{ .Commit }}
{{- else }}
  tag: {{ .Version }}
{{- end }}
  stageDependencies:
    install: ['docs/Gemfile','docs/Gemfile.lock']
    beforeSetup: '**/*'
  includePaths: 'docs'
  excludePaths: '**/*.sh'
{{- end }}

The artifact name must be unique. We can achieve this, for example, by adding the channel name (the value of the variable .Channel) as a suffix to the artifact name: artifact: doc-{{ .Channel }}. However, it should be understood that when importing from artifacts, references will need to be made to the same names.

When describing the artifact, a feature of werf is used, specifically mounting. Mounting with the specification of the service directory build_dir allows caching of Jekyll between pipeline runs, which significantly speeds up the rebuild process.

You may have also noticed the use of the file releases.yml — this is a YAML file containing data about releases requested with github.com (the artifact obtained when executing the pipeline). It is needed when compiling the site, but in the context of the article, it's interesting to us because its state affects the rebuilding of only a single artifact — the artifact of the root version site (it is not needed in other artifacts).

This is implemented using the conditional operator if Go templates and the construct {{ $Root.Files.Get "releases.yml" | sha256sum }} in the stage stageIt works as follows: when assembling an artifact for the root version (variable .Channel equal to root) the file hash releases.yml affects the signature of the entire stage, as it is part of the name of the Ansible task (parameter name). Thus, when changing the content file releases.yml the corresponding artifact will be rebuilt.

Also note the work with the external repository. In the artifact image from the werf repository, only the directory /docsis added, depending on the parameters passed, data of the required tag or review commit is added right away.

To use the artifact template to generate descriptions of artifacts for the passed versions of channels and releases, we organize a loop over the variable .WerfVersions downward API support (simultaneously with this in werf.yaml:

{{ range .WerfVersions -}}
{{ $VersionsDict := splitn "%" 2 . -}}
{{ dict "Version" $VersionsDict._1 "Channel" $VersionsDict._0 "Root" $Root | include "doc_artifact" }}
---
{{ end -}}

Since the loop will generate several artifacts (we hope so), it is necessary to account for the separator between them — the sequence --- (for more on the configuration file syntax see the documentation). As defined earlier, when calling the template in a loop, we pass the version, URL, and root context parameters.

Similarly, but now without the loop, we call the artifact template for "special cases": for the root version, as well as for the version from the review commit:

{{ dict "Version" .WerfRootVersion "Channel" "root" "Root" $Root  | include "doc_artifact" }}
---
{{- if .WerfReviewCommit }}
{{ dict "Version" "review" "Channel" "review" "Commit" .WerfReviewCommit "Root" $Root  | include "doc_artifact" }}
{{- end }}

Please note that the artifact for the review commit will only be built if the variable is set .WerfReviewCommit.

Artifacts are ready — it's time to focus on the import!

The final image intended for running in Kubernetes is a standard NGINX, with an added server configuration file nginx.conf and static files from the artifacts. Besides the root version artifact of the site, we need to repeat the loop over the variable .WerfVersions for importing the artifacts of channel versions and releases + adhere to the artifact naming rule we established earlier. Since each artifact stores site versions for two languages, we will import them into the locations specified by the configuration.

Description of the final werf-doc image

image: werf-doc
from: nginx:stable-alpine
ansible:
  setup:
  - name: "Setup /etc/nginx/nginx.conf"
    copy:
      content: |
{{ .Files.Get ".werf/nginx.conf" | indent 8 }}
      dest: /etc/nginx/nginx.conf
  - file:
      path: "{{`{{ item }}`}}"
      state: directory
      mode: 0777
    with_items:
    - /app/main_site/assets
    - /app/ru_site/assets
import:
- artifact: doc-root
  add: /app/_main_site
  to: /app/main_site
  before: setup
- artifact: doc-root
  add: /app/_ru_site
  to: /app/ru_site
  before: setup
{{ range .WerfVersions -}}
{{ $VersionsDict := splitn "%" 2 . -}}
{{ $Channel := $VersionsDict._0 -}}
{{ $Version := $VersionsDict._1 -}}
- artifact: doc-{{ $Channel }}
  add: /app/_main_site
  to: /app/main_site/v{{ $Channel }}
  before: setup
{{ end -}}
{{ range .WerfVersions -}}
{{ $VersionsDict := splitn "%" 2 . -}}
{{ $Channel := $VersionsDict._0 -}}
{{ $Version := $VersionsDict._1 -}}
- artifact: doc-{{ $Channel }}
  add: /app/_ru_site
  to: /app/ru_site/v{{ $Channel }}
  before: setup
{{ end -}}

An additional image that runs alongside the main one on the dev contour contains only two versions of the site: the version from the review commit and the root version of the site (where common assets and, if you remember, release data are stored). Thus, the additional image will differ from the main one only in the import section (and, of course, in the name):

image: werf-dev
...
import:
- artifact: doc-root
  add: /app/_main_site
  to: /app/main_site
  before: setup
- artifact: doc-root
  add: /app/_ru_site
  to: /app/ru_site
  before: setup
{{- if .WerfReviewCommit  }}
- artifact: doc-review
  add: /app/_main_site
  to: /app/main_site/review
  before: setup
- artifact: doc-review
  add: /app/_ru_site
  to: /app/ru_site/review
  before: setup
{{- end }}

As mentioned above, the artifact for the review commit will only be generated when the specified environment variable is set. REVIEW_SHAWe could actually skip generating the werf-dev image if there is no environment variable. REVIEW_SHA, but for the sake of policy clearance for Docker images in werf to work for the werf-dev image, we will keep it building only with the artifact of the root version (it is already built), to simplify the pipeline structure.

The build is ready! Let's move on to CI/CD and important nuances.

Pipeline in GitLab CI and features of dynamic builds

When starting the build, we need to set the environment variables used in werf.yaml. This does not concern the REVIEW_SHA variable, which we will set when triggering the pipeline from a GitHub hook.

The necessary external data generation will be moved to a Bash script generate_artifacts, which will generate two artifacts for the GitLab pipeline:

• file releases.yml with release data,
• file common_envs.sh, containing environment variables for export.

Contents of the file generate_artifacts you will find in our repository with examples. The data retrieval itself is not the subject of this article, but the file common_envs.sh is important to us, as it affects the operation of werf. An example of its contents:

export RELEASES='1.0%v1.0.6-4'
export CHANNELS='1.0-alpha%v1.0.7-1 1.0-beta%v1.0.7-1 1.0-ea%v1.0.6-4 1.0-stable%v1.0.6-4 1.0-rock-solid%v1.0.6-4'
export ROOT_VERSION='v1.0.6-4'

You can use the output of such a script, for example, with a Bash function source.

And now for the most interesting part. For both the build and deployment of the application to work correctly, it is necessary to ensure that werf.yaml was the same at least within a single pipeline. If this condition is not met, the stage signatures that werf calculates during the build and, for example, deployment will differ. This will lead to a deployment error, as the image required for deployment will be missing.

In other words, if during the build of the site image the information about releases and versions is one thing, but at the time of deployment a new version comes out and the environment variables have different values, then the deployment will end in error: because the artifact of the new version has not yet been built.

If the generation werf.yaml depends on external data (for example, a list of current versions, as in our case), then the composition and values of such data must be fixed within the pipeline. This is especially important if the external parameters change quite often.

We will receive and fix external data in the first stage of the pipeline in GitLab (Prebuild) and pass them further as a GitLab CI artifact. This will allow running and rerunning the pipeline jobs (build, deployment, cleanup) with the same configuration in werf.yaml.

Content of the stage Prebuild file .gitlab-ci.yml:

Prebuild:
  stage: prebuild
  script:
    - bash .\/generate_artifacts 1> common_envs.sh
    - cat .\/common_envs.sh
  artifacts:
    paths:
      - releases.yml
      - common_envs.sh
    expire_in: 2 week

Having fixed the external data in the artifact, one can perform the build and deployment using the standard GitLab CI pipeline stages: Build and Deploy. The pipeline itself is triggered by hooks from the GitHub repository werf (i.e., when changes occur in the GitHub repository). The data for them can be found in the project properties in GitLab under CI / CD Settings -> Pipeline triggers, and then we will create the corresponding Webhook in GitHub (Settings -> Webhooks).

The build stage will look as follows:

Build:
  stage: build
  script:
    - type multiwerf && . $(multiwerf use 1.0 alpha --as-file)
    - type werf && source <(werf ci-env gitlab --tagging-strategy tag-or-branch --verbose)
    - source common_envs.sh
    - werf build-and-publish --stages-storage :local
  except:
    refs:
      - schedules
  dependencies:
    - Prebuild

GitLab will add two artifacts from the stage to the build stage Prebuild, so we export variables with the prepared input data using the construction source common_envs.shWe initiate the build stage in all cases except for running the pipeline on a schedule. For the schedule, our pipeline will launch for cleanup — no build is needed in this case.

During the deployment stage, we will describe two tasks — separately for deployment to production and dev environments, using a YAML template:

.base_deploy: &base_deploy
  stage: deploy
  script:
    - type multiwerf && . $(multiwerf use 1.0 alpha --as-file)
    - type werf && source <(werf ci-env gitlab --tagging-strategy tag-or-branch --verbose)
    - source common_envs.sh
    - werf deploy --stages-storage :local
  dependencies:
    - Prebuild
  except:
    refs:
      - schedules

Deploy to Production:
  <<: *base_deploy
  variables:
    WERF_KUBE_CONTEXT: prod
  environment:
    name: production
    url: werf.io
  only:
    refs:
      - master
  except:
    variables:
      - $REVIEW_SHA
    refs:
      - schedules

Deploy to Test:
  <<: *base_deploy
  variables:
    WERF_KUBE_CONTEXT: dev
  environment:
    name: test
    url: werf.test.flant.com
  except:
    refs:
      - schedules
  only:
    variables:
      - $REVIEW_SHA

The tasks essentially differ only by specifying the cluster context to which werf should perform the deployment (WERF_KUBE_CONTEXT), and setting environment variables for the environment (environment.name and environment.url), which are then used in the Helm chart templates. We will not detail the contents of the templates since there is nothing interesting for the topic at hand, but you can find them in the repository linked to the article.

Final Touch

As new versions of werf are released quite frequently, new images will also be built often, resulting in the Docker Registry continuously growing. Therefore, it is essential to set up automatic image cleanup policies. This is very straightforward.

To implement this, you will need:

• Add a cleanup stage in .gitlab-ci.yml;
• Add a periodic cleanup job execution;
• Set up an environment variable with the write access token.

Adding a cleanup stage in .gitlab-ci.yml:

Cleanup:
  stage: cleanup
  script:
    - type multiwerf && . $(multiwerf use 1.0 alpha --as-file)
    - type werf && source <(werf ci-env gitlab --tagging-strategy tag-or-branch --verbose)
    - source common_envs.sh
    - docker login -u nobody -p ${WERF_IMAGES_CLEANUP_PASSWORD} ${WERF_IMAGES_REPO}
    - werf cleanup --stages-storage :local
  only:
    refs:
      - schedules

We have seen almost all of this earlier — only for cleanup you need to first authenticate to the Docker Registry with a token that has permissions to delete images in the Docker Registry (the automatically issued token for GitLab CI jobs does not have such permissions). The token needs to be set up in GitLab in advance and its value specified in the environment variable WERF_IMAGES_CLEANUP_PASSWORD project (CI/CD Settings -> Variables).

The cleanup job with the required schedule is added in CI/CD ->
Schedules.

Everything: the project in Docker Registry will no longer grow constantly from unused images.

In conclusion of the practical part, I would like to remind you that the complete listings from the article are available in Git:

• .gitlab-ci.yml;
• werf.yaml.

Result

1. We have obtained a logical build structure: one artifact per version.
2. The build is universal and does not require manual changes when new versions of werf are released: documentation on the website is automatically updated.
3. Two images are built for different environments.
4. It works quickly since caching is maximally utilized — when a new version of werf is released or a GitHub hook is triggered for a review commit, only the corresponding artifact with the changed version is rebuilt.
5. There is no need to worry about removing unused images: cleanup according to werf policies will maintain order in the Docker Registry.

Conclusions

• Using werf allows the build to work quickly due to caching of both the build itself and when working with external repositories.
• Working with external Git repositories eliminates the need to clone the repository fully each time or to invent a bicycle with intricate optimization logic. werf uses cache and clones only once, afterward it uses fetch and only when necessary.
• The ability to use Go templates in the build configuration file werf.yaml allows describing a build whose result depends on external data.
• Using mounting in werf greatly speeds up the artifact build — due to cache being shared across all pipelines.
• werf makes it easy to configure cleanup, which is particularly relevant in dynamic builds.

P.S.

Also read in our blog:

• «Running commands during the delivery process of a new application release to Kubernetes»;
• «Building and deploying identical microservices with werf and GitLab CI»;
• «Using werf for deploying complex Helm charts»;
• «Introducing werf 1.0 stable: what does GitOps, status, and plans have to do with it?».

Source: habr.com