Docker Compose: From Development to Production

The transcript translation of the podcast has been prepared ahead of the course launch Linux Administrator

Docker Compose is an amazing tool for creating a working
environment for the stack used in your application. It allows you to define
each component of your application in a clear and simple syntax using YAML files
With the advent of.

docker compose v3 these YAML files can be used directly in a production environment when working with a cluster
But does this mean you can use the same docker-compose file in the development process and in the production environment? Or use the same file for.

staging? Well, generally — yes, but for this functionality we need the following:
Variable interpolation: using environment variables for certain
values that change in each environment.

• Configuration override: the ability to define a second (or any
  subsequent) docker-compose file that modifies something from
• the first, and docker compose will take care of merging both files.
  Differences between development and production files
  During development, you will likely want to check code changes in

real time. For this, it is common to mount the source code volume in

the container where your application runtime resides. However, for a production environment
this approach is not suitable.
In production, you have a cluster with many nodes, and the volume is local to
the node where your container (or service) runs, so you cannot

mount the source code without complex operations involving
code synchronization, signals, etc.
Instead, we typically want to create an image with a specific version of your code.
This is usually tagged accordingly (you can use semantic

versioning or another system at your discretion).
Configuration Override
Given the differences and the fact that your dependencies may vary across development and production scenarios, it's clear that we'll need different configuration files.

Docker compose supports merging various compose files to

achieve the final configuration. How this works can be seen in an example:
development and production, it is clear that we will need different configuration files.

Docker Compose supports the merging of different compose files to
achieve the final configuration. How this works can be seen in the example:

$ cat docker-compose.yml
version: "3.2"

services:
  whale:
    image: docker/whalesay
    command: ["cowsay", "hello!"]
$ docker-compose up
Creating network "composeconfigs_default" with the default driver
Starting composeconfigs_whale_1
Attaching to composeconfigs_whale_1
whale_1  |  ________
whale_1  | 
whale_1  |  --------
whale_1  |     
whale_1  |      
whale_1  |       
whale_1  |                     ##        .
whale_1  |               ## ## ##       ==
whale_1  |            ## ## ## ##      ===
whale_1  |        /""""""""""""""""___/ ===
whale_1  |   ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~
whale_1  |        ______ o          __/
whale_1  |                     __/
whale_1  |           __________/
composeconfigs_whale_1 exited with code 0

As mentioned, docker compose supports merging multiple compose files,
allowing you to override various parameters in the second file. For example:

$ cat docker-compose.second.yml
version: "3.2"
services:
  whale:
    command: ["cowsay", "bye!"]

$ docker-compose -f docker-compose.yml -f docker-compose.second.yml up
Creating composeconfigs_whale_1
Attaching to composeconfigs_whale_1
whale_1  |  ______
whale_1  | 
whale_1  |  ------
whale_1  |     
whale_1  |      
whale_1  |       
whale_1  |                     ##        .
whale_1  |               ## ## ##       ==
whale_1  |            ## ## ## ##      ===
whale_1  |        /""""""""""""""""___/ ===
whale_1  |   ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~
whale_1  |        ______ o          __/
whale_1  |                     __/
whale_1  |           __________/
composeconfigs_whale_1 exited with code 0

This syntax is not very convenient during development when the command
needs to be executed multiple times.

Fortunately, docker compose automatically looks for a special file named
docker-compose.override.yml to override values docker-compose.yml. If
you rename the second file, you will achieve the same result, but using the original command:

$ mv docker-compose.second.yml docker-compose.override.yml
$ docker-compose up
Starting composeconfigs_whale_1
Attaching to composeconfigs_whale_1
whale_1  |  ______
whale_1  | 
whale_1  |  ------
whale_1  |     
whale_1  |      
whale_1  |       
whale_1  |                     ##        .
whale_1  |               ## ## ##       ==
whale_1  |            ## ## ## ##      ===
whale_1  |        /""""""""""""""""___/ ===
whale_1  |   ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~
whale_1  |        ______ o          __/
whale_1  |                     __/
whale_1  |           __________/
composeconfigs_whale_1 exited with code 0

That's easier to remember.

Variable interpolation

Configuration files support interpolation
of variables and default values. This means you can do the following:

services:
  my-service:
    build:
      context: .
    image: private.registry.mine/my-stack/my-service:${MY_SERVICE_VERSION:-latest}
...

And if you run docker-compose build (or push) without the environment variable
$MY_SERVICE_VERSION, the value will be latest, but if you set the environment variable before building, it will be used during the build or push
to the registry
private.registry.mine My principles.

My Principles

The approaches that work for me might be useful for you as well. I follow these
simple rules:

• All my stacks for production, development (or other environments) are defined through
  docker-compose files.
• Configuration files necessary to cover all my environments try to
  minimize duplication.
• I need one straightforward command to work in each environment.
• The main configuration is defined in the file docker-compose.yml.
• Environment variables are used to define image tags or other
  variables that may change from environment to environment (staging, integration,
  production).
• The variable values for production are used as default values, minimizing risks if the stack is run in production without
  the environment variable set.
  To run the service in the production environment, the command used is
• docker stack deploy --compose-file docker-compose.yml --with-registry-auth my-stack-name The working environment is started with the command.
• Let's look at a simple example. docker-compose up -d.

And

# docker-compose.yml
...
services:
  my-service:
    build:
      context: .
    image: private.registry.mine/my-stack/my-service:${MY_SERVICE_VERSION:-latest}
    environment:
      API_ENDPOINT: ${API_ENDPOINT:-https://production.my-api.com}
...

I can use

# docker-compose.override.yml
...
services:
  my-service:
    ports: # This is needed for development!
      - 80:80
    environment:
      API_ENDPOINT: https://devel.my-api.com
    volumes:
      - ./:/project/src
...

docker-compose (docker-compose up) to start the stack indevelopment mode with the source code mounted in
I can use these same files in production! And I could use exactly /project/src.

the same file
for staging. To deploy this to docker-compose.yml production, I just need to build and push the image with a predefined tag
during the CI stage:
export MY_SERVICE_VERSION=1.2.3 docker-compose -f docker-compose.yml build docker-compose -f docker-compose.yml push

In production, this can be run using the following commands:

export MY_SERVICE_VERSION=1.2.3 docker stack deploy my-stack --compose-file docker-compose.yml --with-registry-auth

And if you want to do the same on staging, you just need to define

the necessary environment variables to work in the staging environment:
export MY_SERVICE_VERSION=1.2.3 export API_ENDPOINT=http://staging.my-api.com docker stack deploy my-stack --compose-file docker-compose.yml --with-registry-auth

As a result, we used two different docker-compose files that can be used for any of your environments without

duplicate configurations!
Learn more about the course

Theory and practice of using ClickHouse in real applications. Alexander Zaitsev (2018) Linux Administrator

Source: habr.com