ProHoster > Blog > Administration > You can now build Docker images in werf using a regular Dockerfile

You can now build Docker images in werf using a regular Dockerfile

Better late than never. Or how we almost made a serious mistake by not having support for regular Dockerfiles to build application images.

You can now build Docker images in werf using a regular Dockerfile

It will be about yard — A GitOps utility that integrates with any CI/CD system and manages the entire application life cycle, allowing you to:

  • collect and publish images,
  • deploy applications to Kubernetes,
  • delete unused images using special policies.


The philosophy of the project is to collect low-level tools into a single unified system that gives DevOps engineers control over applications. If possible, existing utilities (like Helm and Docker) should be used. If there is no solution to some problem, we can create and maintain everything necessary for this.

Background: your own image collector

This is what happened with the image builder in werf: we lacked the usual Dockerfile. If you briefly plunge into the history of the project, then this problem manifested itself already in the first versions of werf (then known as dapp).

While creating a tool for building applications into Docker images, we quickly realized that the Dockerfile was not suitable for us for some very specific tasks:

  1. The need to assemble typical small web applications according to the following standard scheme:
    • install the application's system-wide dependencies,
    • install a bundle of application dependency libraries,
    • collect assets,
    • and most importantly, update the code in the image quickly and efficiently.
  2. When changes are made to project files, the builder must quickly create a new layer by patching the changed files.
  3. If certain files have changed, then the corresponding dependent stage must be rebuilt.

Today, there are many other possibilities in our faucet, but the initial desires and urges were as follows.

In general, without thinking twice, we armed ourselves with the programming language used (see below) and hit the road - to implement own DSL! In accordance with the tasks set, it was intended to describe the build process in stages and determine the dependencies of these stages on files. And supplemented it own faucet, which turned DSL into the final goal - the assembled image. At first DSL was in Ruby, but as switching to Golang - the config of our collector began to be described in the YAML file.

You can now build Docker images in werf using a regular Dockerfile
Old config for dapp in Ruby

You can now build Docker images in werf using a regular Dockerfile
Actual config for werf on YAML

The mechanism of the collector also changed over time. At first, we simply generated some temporary Dockerfile from our configuration on the fly, and then we started running build instructions in temporary containers and making a commit.

NB: At the moment, our builder, which works with its own config (in YAML) and is called the Stapel builder, has already developed into a fairly powerful tool. Its detailed description deserves separate articles, and the main details can be found in documentation.

Awareness of the problem

But we realized, and not immediately, that we had made one mistake: we did not add the ability build images through a standard Dockerfile and integrate them into the same end-to-end application management infrastructure (i.e. collect images, deploy and clean them). How could you make a deployment tool in Kubernetes and not implement Dockerfile support, i.e. a standard way to describe images for most projects?..

Instead of answering such a question, we offer its solution. What if you already have a Dockerfile (or a set of Dockerfiles) and want to use werf?

NB: By the way, why would you even want to use werf? The main features boil down to the following:

  • full application management cycle including cleaning images;
  • the ability to manage the assembly of several images at once from a single config;
  • improved deployment process for Helm compatible charts.

A more complete list can be found at project page.

So, if earlier we would suggest rewriting the Dockerfile to our config, now we are happy to say: “Let werf build your Dockerfiles!”

How to use?

The full implementation of this feature appeared in the release werf v1.0.3-beta.1. The general principle is simple: the user specifies the path to an existing Dockerfile in the werf config, and then runs the command werf build... and that's it - werf will build the image. Let's consider an abstract example.

Let's announce the next Dockerfile at the root of the project:

FROM ubuntu:18.04
RUN echo Building ...

And we'll announce werf.yamlwhich uses this Dockerfile:

configVersion: 1
project: dockerfile-example
---
image: ~
dockerfile: ./Dockerfile

All! Left run werf build:

You can now build Docker images in werf using a regular Dockerfile

In addition, you can declare the following werf.yaml to build several images at once from different Dockerfiles:

configVersion: 1
project: dockerfile-example
---
image: backend
dockerfile: ./dockerfiles/Dockerfile-backend
---
image: frontend
dockerfile: ./dockerfiles/Dockerfile-frontend

Finally, passing additional build parameters is also supported - such as --build-arg и --add-host - via the werf config. A complete description of the Dockerfile image configuration is available at documentation page.

How does it work?

During the build process, the standard cache of local layers in Docker functions. However, importantly, werf also integrates the Dockerfile configuration into its infrastructure. What does this mean?

  1. Each image built from a Dockerfile consists of a single stage called dockerfile (more about what stages are in werf, you can read here).
  2. For stage dockerfile werf calculates a signature that depends on the contents of the Dockerfile configuration. Changing the Dockerfile configuration changes the stage signature dockerfile and werf will initiate a rebuild of that stage with the new Dockerfile config. If the signature does not change, then werf takes the image from the cache (more about the use of signatures in werf was described in this report).
  3. Further, the collected images can be published with the command werf publish (or werf build-and-publish) and use it to deploy to Kubernetes. Published images in the Docker Registry will be cleaned with standard werf cleanup tools, i.e. there will be an automatic cleanup of old images (older than N days), images associated with non-existent Git branches, and other policies.

More details about the points described here can be found in the documentation:

Notes and Precautions

1. External URL in ADD is not supported

The use of an external URL in a directive is currently not supported. ADD. Werf will not trigger a rebuild when a resource at the specified URL changes. This feature is planned to be added soon.

2. You can't add .git to an image

Generally speaking, adding a directory .git into an image is a vicious bad practice, and here's why:

  1. If .git remains in the final image, this violates the principles 12 factor app: since the resulting image must be associated with a single commit, it should not be possible to do git checkout arbitrary commit.
  2. .git increases the size of the image (the repository can be large due to the fact that large files were once added to it and then deleted). The size of the work-tree associated only with a particular commit will not depend on the history of operations in Git. In this case, the addition and subsequent removal .git from the final image will not work: the image will still acquire an extra layer - this is how Docker works.
  3. Docker may trigger an unnecessary rebuild even if the same commit is being built from different work-trees. For example, GitLab creates separate cloned directories in /home/gitlab-runner/builds/HASH/[0-N]/yourproject with parallel assembly enabled. An extra rebuild will be due to the fact that the directory .git different in different cloned versions of the same repository, even if the same commit is built.

The last point also has consequences when using werf. Werf requires the built cache to be present when running some commands (for example, werf deploy). During the execution of such commands, werf calculates stage signatures for the images specified in werf.yaml, and they must be in the build cache, otherwise the command will not be able to continue. If the signature of the stages depends on the content .git, then we get a cache that is unstable to changes in irrelevant files, and werf will not be able to forgive such an oversight (for more details, see documentation).

Generally adding only certain required files through instructions ADD in any case increases the efficiency and reliability of the written Dockerfile, and also improves the stability of the cache collected by this Dockerfile, to irrelevant changes in Git.

Сonclusion

Our initial path with writing our own builder for specific needs was hard, honest and straightforward: instead of using crutches on top of the standard Dockerfile, we wrote our own solution with custom syntax. And this gave its advantages: the Stapel-collector does its job perfectly.

However, in the process of writing our own builder, we lost sight of the support for already existing Dockerfiles. Now this shortcoming has been fixed, and in the future we plan to develop Dockerfile support along with our custom Stapel builder for distributed builds and for building using Kubernetes (i.e. building on runners inside Kubernetes, as done in kaniko).

So, if you suddenly have a couple of Dockerfiles lying around ... try yard!

PS List of documentation on the topic

Read also on our blog:werf - our tool for CI / CD in Kubernetes (overview and video report)».

Source: habr.com

Add a comment