{"id":81600,"date":"2020-05-15T01:42:41","date_gmt":"2020-05-14T23:42:41","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/neskolko-sovetov-o-tom-kak-uskorit-sborku-docker-obrazov-naprimer-do-30-sekund"},"modified":"2020-05-15T01:42:41","modified_gmt":"2020-05-14T23:42:41","slug":"neskolko-sovetov-o-tom-kak-uskorit-sborku-docker-obrazov-naprimer-do-30-sekund","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/neskolko-sovetov-o-tom-kak-uskorit-sborku-docker-obrazov-naprimer-do-30-sekund","title":{"rendered":"A few tips on how to speed up Docker image builds. For example, in up to 30 seconds.","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

Before a feature goes into production, in our era of complex orchestrators and CI\/CD, it has to go through a long journey from commit to testing and delivery. In the past, you could just throw new files via FTP (which no one does anymore, right?), and the 'deployment' process took seconds. Now you need to create a merge request and wait a considerable time for the feature to reach users.<\/p>\n

<\/p>\n

Part of this journey is building the Docker image. Sometimes the build takes minutes, and sometimes \u2014 dozens of minutes, which is hard to call normal. In this article, we will take a simple application, package it into an image, apply several methods to speed up the build, and discuss the nuances of these methods.<\/p>\n

<\/p>\n

\"A<\/p>\n

<\/a><\/noindex><\/p>\n

We have considerable experience in creating and maintaining media websites: TASS<\/a><\/noindex>, The Bell<\/a><\/noindex>, \"Novaya Gazeta\"<\/a><\/noindex>, Republic<\/a><\/noindex>\u2026 Not long ago we expanded our portfolio by releasing the production website Reminder<\/a><\/noindex>. And while we quickly polished new features and fixed old bugs, slow deployment became a big problem.<\/p>\n

<\/p>\n

We deploy on GitLab. We build images, push them to the GitLab Registry, and roll them out in production. In this list, the longest part is building images. For example, without optimization, each backend build took 14 minutes.<\/p>\n

<\/p>\n

\"A<\/p>\n

<\/p>\n

Eventually, it became clear that this couldn't go on, and we sat down to figure out why the images were taking so long to build. In the end, we managed to reduce the build time to 30 seconds!<\/p>\n

<\/p>\n

\"A<\/p>\n

<\/p>\n

For this article, to avoid tying it to the Reminder environment, let's look at an example of creating a blank application in Angular. So, let's create our application:<\/p>\n

<\/p>\n

ng n app<\/code><\/pre>\n
<\/p>\n
We add PWA to it (after all, we are progressive):<\/p>\n
<\/p>\n
ng add @angular\/pwa --project app<\/code><\/pre>\n
<\/p>\n
While a million npm packages are being downloaded, let\u2019s understand how a Docker image is structured. Docker provides the ability to package applications and run them in an isolated environment called a container. Thanks to this isolation, you can run many containers on one server simultaneously. Containers are significantly lighter than virtual machines, as they run directly on the system's kernel. To run a container with our application, we first need to create an image that packages everything necessary for our application's operation. Essentially, an image is a snapshot of the file system. For example, let's take a Dockerfile:<\/p>\n
<\/p>\n
FROM node:12.16.2\nWORKDIR \/app\nCOPY . .\nRUN npm ci\nRUN npm run build --prod<\/code><\/pre>\n
<\/p>\n
A Dockerfile is a set of instructions; as each is executed, Docker saves changes to the file system and applies them to the previous ones. Each command creates its own layer. A finished image is a combination of these layers.<\/p>\n
<\/p>\n
What is important to know: each layer can be cached by Docker. If nothing has changed since the last build, Docker will use the existing layer instead of executing the command. Since the main speed gain in building will come from using the cache, we will focus on measuring the speed of the build with a ready cache. So, let's break it down step by step:<\/p>\n
<\/p>\n
    \n
  1. We will remove images locally so that previous runs do not affect the test.
    \ndocker rmi $(docker images -q)<\/code><\/li>\n
  2. We start the build for the first time.
    \ntime docker build -t app .<\/code><\/li>\n
  3. We modify the file src\/index.html \u2014 simulating a programmer's work.<\/li>\n
  4. We start the build a second time.
    \ntime docker build -t app .<\/code><\/li>\n<\/ol>\n
    <\/p>\n
    If the environment for building images is set up correctly (which will be covered shortly), Docker will already have a bunch of caches on board when the build starts. Our task is to learn to use the cache so that the build proceeds as quickly as possible. Since we assume that the build without the cache occurs only once \u2014 the very first time \u2014 we can ignore how slow this first attempt was. In our tests, the important part is the second build when the caches are warmed up and we are ready to bake our pie. Nevertheless, some tips will also affect the first build.<\/p>\n
    <\/p>\n
    Let's place the Dockerfile described above in the project folder and run the build. All of the listings provided have been shortened for readability.<\/p>\n
    <\/p>\n
    $ time docker build -t app .\nSending build context to Docker daemon 409MB\nStep 1\/5 : FROM node:12.16.2\nStatus: Downloaded newer image for node:12.16.2\nStep 2\/5 : WORKDIR \/app\nStep 3\/5 : COPY . .\nStep 4\/5 : RUN npm ci\nadded 1357 packages in 22.47s\nStep 5\/5 : RUN npm run build --prod\nDate: 2020-04-16T19:20:09.664Z - Hash: fffa0fddaa3425c55dd3 - Time: 37581ms\nSuccessfully built c8c279335f46\nSuccessfully tagged app:latest\n\nreal 5m4.541s\nuser 0m0.000s\nsys 0m0.000s<\/code><\/pre>\n
    <\/p>\n
    We change the contents of src\/index.html and run it a second time.<\/p>\n
    <\/p>\n
    $ time docker build -t app .\nSending build context to Docker daemon 409MB\nStep 1\/5 : FROM node:12.16.2\nStep 2\/5 : WORKDIR \/app\n ---> Using cache\nStep 3\/5 : COPY . .\nStep 4\/5 : RUN npm ci\nadded 1357 packages in 22.47s\nStep 5\/5 : RUN npm run build --prod\nDate: 2020-04-16T19:26:26.587Z - Hash: fffa0fddaa3425c55dd3 - Time: 37902ms\nSuccessfully built 79f335df92d3\nSuccessfully tagged app:latest\n\nreal 3m33.262s\nuser 0m0.000s\nsys 0m0.000s<\/code><\/pre>\n
    <\/p>\n
    To check if we obtained the image, we will run the command docker images<\/code>:<\/p>\n
    <\/p>\n
    REPOSITORY   TAG      IMAGE ID       CREATED              SIZE\napp          latest   79f335df92d3   About a minute ago   1.74GB<\/code><\/pre>\n
    <\/p>\n
    Before building, Docker takes all the files in the current context and sends them to its daemon. Sending build context to Docker daemon 409MB<\/code>. The build context is specified as the last argument of the build command. In our case, this is the current directory \u2014 \".\", \u2014 and Docker pulls everything we have in this folder. 409 MB is a lot: let's think about how to fix this.<\/p>\n
    <\/p>\n
    Reducing the context<\/h2>\n
    <\/p>\n
    To reduce the context, there are two options. Either place all the files needed for the build in a separate folder and specify that folder as the context for Docker. This might not always be convenient, so there is an option to specify exclusions: what should not be included in the context. For this, we will put a .dockerignore file in the project and specify what is not needed for the build:<\/p>\n
    <\/p>\n
    .git\n\/node_modules<\/code><\/pre>\n
    <\/p>\n
    and run the build again:<\/p>\n
    <\/p>\n
    $ time docker build -t app .\nSending build context to Docker daemon 607.2kB\nStep 1\/5 : FROM node:12.16.2\nStep 2\/5 : WORKDIR \/app\n ---> Using cache\nStep 3\/5 : COPY . .\nStep 4\/5 : RUN npm ci\nadded 1357 packages in 22.47s\nStep 5\/5 : RUN npm run build --prod\nDate: 2020-04-16T19:33:54.338Z - Hash: fffa0fddaa3425c55dd3 - Time: 37313ms\nSuccessfully built 4942f010792a\nSuccessfully tagged app:latest\n\nreal 1m47.763s\nuser 0m0.000s\nsys 0m0.000s<\/code><\/pre>\n
    <\/p>\n
    607.2 KB is much better than 409 MB. Plus, we reduced the image size from 1.74 to 1.38 GB:<\/p>\n
    <\/p>\n
    REPOSITORY   TAG      IMAGE ID       CREATED         SIZE\napp          latest   4942f010792a   3 minutes ago   1.38GB<\/code><\/pre>\n
    <\/p>\n
    Let's try to further reduce the image size.<\/p>\n
    <\/p>\n
    Using Alpine<\/h2>\n
    <\/p>\n
    Another way to save on image size is to use a smaller parent image. The parent image is the image on which our image is based. The base layer is specified with the command FROM<\/code> in the Dockerfile. In our case, we are using an image based on Ubuntu, which already has nodejs installed. And it weighs \u2026<\/p>\n
    <\/p>\n
    $ docker images -a | grep node\nnode 12.16.2 406aa3abbc6c 17 minutes ago 916MB<\/code><\/pre>\n
    <\/p>\n
    ... almost a gigabyte. We can significantly reduce the size by using an image based on Alpine Linux. Alpine is a very small Linux. The Docker image for nodejs based on Alpine weighs only 88.5 MB. So let's replace our bulky base image:<\/p>\n
    <\/p>\n
    FROM node:12.16.2-alpine3.11\nRUN apk --no-cache --update --virtual build-dependencies add \n    python \n    make \n    g++\nWORKDIR \/app\nCOPY . .\nRUN npm ci\nRUN npm run build --prod<\/code><\/pre>\n
    <\/p>\n
    We had to install some things that are necessary for building the application. Yes, Angular won't build without Python \u00af(\u00b0_o)\u2044\u00af<\/p>\n
    <\/p>\n
    But the image size dropped by 150 MB:<\/p>\n
    <\/p>\n
    REPOSITORY   TAG      IMAGE ID       CREATED          SIZE\napp          latest   aa031edc315a   22 minutes ago   761MB<\/code><\/pre>\n
    <\/p>\n
    Let's go even further.<\/p>\n
    <\/p>\n
    Multi-stage build<\/h2>\n
    <\/p>\n
    Not everything in the image is needed for production.<\/p>\n
    <\/p>\n
    $ docker run app ls -lah\ntotal 576K\ndrwxr-xr-x 1 root root 4.0K Apr 16 19:54 .\ndrwxr-xr-x 1 root root 4.0K Apr 16 20:00 ..\n-rwxr-xr-x 1 root root 19 Apr 17 2020 .dockerignore\n-rwxr-xr-x 1 root root 246 Apr 17 2020 .editorconfig\n-rwxr-xr-x 1 root root 631 Apr 17 2020 .gitignore\n-rwxr-xr-x 1 root root 181 Apr 17 2020 Dockerfile\n-rwxr-xr-x 1 root root 1020 Apr 17 2020 README.md\n-rwxr-xr-x 1 root root 3.6K Apr 17 2020 angular.json\n-rwxr-xr-x 1 root root 429 Apr 17 2020 browserslist\ndrwxr-xr-x 3 root root 4.0K Apr 16 19:54 dist\ndrwxr-xr-x 3 root root 4.0K Apr 17 2020 e2e\n-rwxr-xr-x 1 root root 1015 Apr 17 2020 karma.conf.js\n-rwxr-xr-x 1 root root 620 Apr 17 2020 ngsw-config.json\ndrwxr-xr-x 1 root root 4.0K Apr 16 19:54 node_modules\n-rwxr-xr-x 1 root root 494.9K Apr 17 2020 package-lock.json\n-rwxr-xr-x 1 root root 1.3K Apr 17 2020 package.json\ndrwxr-xr-x 5 root root 4.0K Apr 17 2020 src\n-rwxr-xr-x 1 root root 210 Apr 17 2020 tsconfig.app.json\n-rwxr-xr-x 1 root root 489 Apr 17 2020 tsconfig.json\n-rwxr-xr-x 1 root root 270 Apr 17 2020 tsconfig.spec.json\n-rwxr-xr-x 1 root root 1.9K Apr 17 2020 tslint.json<\/code><\/pre>\n
    <\/p>\n
    Using docker run app ls -lah<\/code> We launched a container based on our image app<\/code> and executed the command within it ls -lah<\/code>, after which the container finished its work.<\/p>\n
    <\/p>\n
    In production, we only need the folder dist<\/code>. However, we need a way to expose the files. We can run an HTTP server using Node.js. But let's simplify things. Can you guess the Russian word that contains four letters '\u044b'? Correct! \u042b\u043d\u0436\u044b\u043d\u044b\u043a\u0441\u044b. We'll take an image with nginx and place the folder dist<\/code> and a small configuration:<\/p>\n
    <\/p>\n
    server {\n    listen 80 default_server;\n    server_name localhost;\n    charset utf-8;\n    root \/app\/dist;\n\n    location \/ {\n        try_files $uri $uri\/ \/index.html;\n    }\n}<\/code><\/pre>\n
    <\/p>\n
    This can all be accomplished with a multi-stage build. Let's modify our Dockerfile:<\/p>\n
    <\/p>\n
    FROM node:12.16.2-alpine3.11 as builder\nRUN apk --no-cache --update --virtual build-dependencies add \n    python \n    make \n    g++\nWORKDIR \/app\nCOPY . .\nRUN npm ci\nRUN npm run build --prod\n\nFROM nginx:1.17.10-alpine\nRUN rm \/etc\/nginx\/conf.d\/default.conf\nCOPY nginx\/static.conf \/etc\/nginx\/conf.d\nCOPY --from=builder \/app\/dist\/app .<\/code><\/pre>\n
    <\/p>\n
    Now we have two instructions FROM<\/code> in the Dockerfile, each running its own build stage. We named the first one builder<\/code>, and from the last FROM onwards, our final image will be prepared. The final step is to copy the artifact from our previous build stage to the final image with nginx. The image size has significantly decreased:<\/p>\n
    <\/p>\n
    REPOSITORY   TAG      IMAGE ID       CREATED          SIZE\napp          latest   2c6c5da07802   29 minutes ago   36MB<\/code><\/pre>\n
    <\/p>\n
    Let's run a container with our image and make sure everything works:<\/p>\n
    <\/p>\n
    docker run -p8080:80 app<\/code><\/pre>\n
    <\/p>\n
    With the option -p8080:80 we mapped port 8080 on our host machine to port 80 inside the container, where nginx is running. We open in the browser http:\/\/localhost:8080\/<\/a><\/noindex> and see our application. Everything works!<\/p>\n
    <\/p>\n
    \"A<\/p>\n
    <\/p>\n
    Reducing the image size from 1.74 GB to 36 MB significantly shortens the time it takes to deliver your application to production. But let's return to build time.<\/p>\n
    <\/p>\n
    $ time docker build -t app .\nSending build context to Docker daemon 608.8kB\nStep 1\/11 : FROM node:12.16.2-alpine3.11 as builder\nStep 2\/11 : RUN apk --no-cache --update --virtual build-dependencies add python make g++\n ---> Using cache\nStep 3\/11 : WORKDIR \/app\n ---> Using cache\nStep 4\/11 : COPY . .\nStep 5\/11 : RUN npm ci\nadded 1357 packages in 47.338s\nStep 6\/11 : RUN npm run build --prod\nDate: 2020-04-16T21:16:03.899Z - Hash: fffa0fddaa3425c55dd3 - Time: 39948ms\n ---> 27f1479221e4\nStep 7\/11 : FROM nginx:stable-alpine\nStep 8\/11 : WORKDIR \/app\n ---> Using cache\nStep 9\/11 : RUN rm \/etc\/nginx\/conf.d\/default.conf\n ---> Using cache\nStep 10\/11 : COPY nginx\/static.conf \/etc\/nginx\/conf.d\n ---> Using cache\nStep 11\/11 : COPY --from=builder \/app\/dist\/app .\nSuccessfully built d201471c91ad\nSuccessfully tagged app:latest\n\nreal 2m17.700s\nuser 0m0.000s\nsys 0m0.000s<\/code><\/pre>\n
    <\/p>\n
    We are changing the order of layers<\/h2>\n
    <\/p>\n
    The first three steps were cached (hint Using cache<\/code>). In the fourth step, all project files are copied, and in the fifth step, dependencies are installed RUN npm ci<\/code> \u2014 a total of 47.338s. Why reinstall dependencies every time when they change very rarely? Let's figure out why they were not cached. The thing is, Docker checks layer by layer to see if the command and related files have changed. In the fourth step, we copy all the files of our project, and among them, of course, there are changes, so Docker not only does not take this layer from the cache, but all subsequent ones as well! Let\u2019s make some minor changes to the Dockerfile.<\/p>\n
    <\/p>\n
    FROM node:12.16.2-alpine3.11 as builder\nRUN apk --no-cache --update --virtual build-dependencies add \n    python \n    make \n    g++\nWORKDIR \/app\nCOPY package*.json .\/\nRUN npm ci\nCOPY . .\nRUN npm run build --prod\n\nFROM nginx:1.17.10-alpine\nRUN rm \/etc\/nginx\/conf.d\/default.conf\nCOPY nginx\/static.conf \/etc\/nginx\/conf.d\nCOPY --from=builder \/app\/dist\/app .<\/code><\/pre>\n
    <\/p>\n
    First, package.json and package-lock.json are copied, then the dependencies are installed, and only after that the entire project is copied. As a result:<\/p>\n
    <\/p>\n
    $ time docker build -t app .\nSending build context to Docker daemon 608.8kB\nStep 1\/12 : FROM node:12.16.2-alpine3.11 as builder\nStep 2\/12 : RUN apk --no-cache --update --virtual build-dependencies add python make g++\n ---> Using cache\nStep 3\/12 : WORKDIR \/app\n ---> Using cache\nStep 4\/12 : COPY package*.json .\/\n ---> Using cache\nStep 5\/12 : RUN npm ci\n ---> Using cache\nStep 6\/12 : COPY . .\nStep 7\/12 : RUN npm run build --prod\nDate: 2020-04-16T21:29:44.770Z - Hash: fffa0fddaa3425c55dd3 - Time: 38287ms\n ---> 1b9448c73558\nStep 8\/12 : FROM nginx:stable-alpine\nStep 9\/12 : WORKDIR \/app\n ---> Using cache\nStep 10\/12 : RUN rm \/etc\/nginx\/conf.d\/default.conf\n ---> Using cache\nStep 11\/12 : COPY nginx\/static.conf \/etc\/nginx\/conf.d\n ---> Using cache\nStep 12\/12 : COPY --from=builder \/app\/dist\/app .\nSuccessfully built a44dd7c217c3\nSuccessfully tagged app:latest\n\nreal 0m46.497s\nuser 0m0.000s\nsys 0m0.000s<\/code><\/pre>\n
    <\/p>\n
    46 seconds instead of 3 minutes \u2014 significantly better! The correct order of layers is important: first, we copy what doesn't change, then what changes rarely, and finally \u2014 what changes frequently.<\/p>\n
    <\/p>\n
    Next, a few words about building images in CI\/CD systems.<\/p>\n
    <\/p>\n
    Using previous images as cache<\/h2>\n
    <\/p>\n
    If we are using some SaaS solution for building, then the local Docker cache may be clean and fresh. To allow Docker to retrieve baked layers, provide it with the previously built image.<\/p>\n
    <\/p>\n
    Let\u2019s consider, for example, the build of our application in GitHub Actions. We will use the following config<\/p>\n
    <\/p>\n
    on:\n  push:\n    branches:\n      - master\n\nname: Test docker build\n\njobs:\n  deploy:\n    name: Build\n    runs-on: ubuntu-latest\n    env:\n      IMAGE_NAME: docker.pkg.github.com\/${{ github.repository }}\\\/app\n      IMAGE_TAG: ${{ github.sha }}\n\n    steps:\n    - name: Checkout\n      uses: actions\\\/checkout@v2\n\n    - name: Login to GitHub Packages\n      env:\n        TOKEN: ${{ secrets.GITHUB_TOKEN }}\n      run: |\n        docker login docker.pkg.github.com -u $GITHUB_ACTOR -p $TOKEN\n\n    - name: Build\n      run: |\n        docker build \n          -t $IMAGE_NAME:$IMAGE_TAG \n          -t $IMAGE_NAME:latest \n          .\n\n    - name: Push image to GitHub Packages\n      run: |\n        docker push $IMAGE_NAME:latest\n        docker push $IMAGE_NAME:$IMAGE_TAG\n\n    - name: Logout\n      run: |\n        docker logout docker.pkg.github.com<\/code><\/pre>\n
    <\/p>\n
    The image is built and pushed to GitHub Packages in two minutes and 20 seconds:<\/p>\n
    <\/p>\n
    \"A<\/p>\n
    <\/p>\n
    Now let's modify the build to utilize cache based on previously built images:<\/p>\n
    <\/p>\n
    on:\n  push:\n    branches:\n      - master\n\nname: Test docker build\n\njobs:\n  deploy:\n    name: Build\n    runs-on: ubuntu-latest\n    env:\n      IMAGE_NAME: docker.pkg.github.com\/${{ github.repository }}\\\/app\n      IMAGE_TAG: ${{ github.sha }}\n\n    steps:\n    - name: Checkout\n      uses: actions\\\/checkout@v2\n\n    - name: Login to GitHub Packages\n      env:\n        TOKEN: ${{ secrets.GITHUB_TOKEN }}\n      run: |\n        docker login docker.pkg.github.com -u $GITHUB_ACTOR -p $TOKEN\n\n    - name: Pull latest images\n      run: |\n        docker pull $IMAGE_NAME:latest || true\n        docker pull $IMAGE_NAME-builder-stage:latest || true\n\n    - name: Images list\n      run: |\n        docker images\n\n    - name: Build\n      run: |\n        docker build \n          --target builder \n          --cache-from $IMAGE_NAME-builder-stage:latest \n          -t $IMAGE_NAME-builder-stage \n          .\n        docker build \n          --cache-from $IMAGE_NAME-builder-stage:latest \n          --cache-from $IMAGE_NAME:latest \n          -t $IMAGE_NAME:$IMAGE_TAG \n          -t $IMAGE_NAME:latest \n          .\n\n    - name: Push image to GitHub Packages\n      run: |\n        docker push $IMAGE_NAME-builder-stage:latest\n        docker push $IMAGE_NAME:latest\n        docker push $IMAGE_NAME:$IMAGE_TAG\n\n    - name: Logout\n      run: |\n        docker logout docker.pkg.github.com<\/code><\/pre>\n
    <\/p>\n
    First, it is necessary to explain why two commands are executed. build<\/code>The reason is that in a multi-stage build, the resulting image will consist of layers from the last stage. Layers from previous stages will not be included in the image. Therefore, when using the final image from the previous build, Docker will not be able to find the prepared layers for building the image with nodejs (builder stage). To solve this issue, an intermediate image is created. $IMAGE_NAME-builder-stage<\/code> and is sent to GitHub Packages so it can be used in subsequent builds as a cache source.<\/p>\n
    <\/p>\n
    \"A<\/p>\n
    <\/p>\n
    Total build time has been reduced to one and a half minutes. Half a minute is spent pulling previous images.<\/p>\n
    <\/p>\n
    Pre-building images<\/h2>\n
    <\/p>\n
    Another way to resolve the issue of Docker's clean cache is to move some layers to a different Dockerfile, build it separately, push it to the Container Registry, and use it as a parent.<\/p>\n
    <\/p>\n
    We create our own nodejs image for building an Angular application. We create Dockerfile.node in the project.<\/p>\n
    <\/p>\n
    FROM node:12.16.2-alpine3.11\nRUN apk --no-cache --update --virtual build-dependencies add \n    python \n    make \n    g++<\/code><\/pre>\n
    <\/p>\n
    We build and push the public image to Docker Hub:<\/p>\n
    <\/p>\n
    docker build -t exsmund\/node-for-angular -f Dockerfile.node .\ndocker push exsmund\/node-for-angular:latest<\/code><\/pre>\n
    <\/p>\n
    Now we use the prepared image in our main Dockerfile:<\/p>\n
    <\/p>\n
    FROM exsmund\/node-for-angular:latest as builder\n...<\/code><\/pre>\n
    <\/p>\n
    In our example, the build time did not decrease, but pre-built images can be useful if you have many projects and need to install the same dependencies in each.<\/p>\n
    <\/p>\n
    \"A<\/p>\n
    <\/p>\n
    We have discussed several methods for speeding up Docker image builds. If you want fast deployment, try applying the following in your project:<\/p>\n
    <\/p>\n