Docker Compose: from development to production

Translation of the transcription of the podcast prepared in anticipation of the start of the course "Linux Administrator"

Docker Compose is an amazing tool for creating desktop
environment for the stack used in your application. It allows you to define
every component of your application, following a clear and simple syntax in Yaml-
files.

With the advent of docker compose v3 these YAML files can be used directly in the production environment, when working with
cluster Docker swarm.

But does that mean you can use the same docker-compose file in
development process and production environment? Or use the same file for
staging? Well, in general, yes, but for such functionality we need the following:

• Variable interpolation: using environment variables for some
  values ​​that change in each environment.
• Configuration override: the ability to define a second (or any
  another follow-up) docker-compose file that will change something regarding
  the first one, and docker compose will take care of merging both files.

Differences between development and production files

During development, you will most likely want to check for code changes in
real time mode. To do this, usually, the volume with the source code is mounted in
a container that contains the runtime for your application. But for a production environment
this way is not suitable.

In production you have a multi-node cluster and the volume is local
relative to the host your container (or service) is running on, so you don't
you can mount the source code without complex operations, which include
code synchronization, signals, etc.

Instead, we usually want to create an image with a specific version of your code.
It is customary to mark it with the appropriate tag (you can use semantic
versioning or another system of your choice).

Configuration override

Given the differences and that your dependencies may differ in scenarios
development and production, it is clear that we will need different configuration files.

Docker compose supports merging different compose files to
get the final configuration. How it works can be seen in an example:

$ cat docker-compose.yml
version: "3.2"

services:
  whale:
    image: docker/whalesay
    command: ["cowsay", "hello!"]
$ docker-compose up
Creating network "composeconfigs_default" with the default driver
Starting composeconfigs_whale_1
Attaching to composeconfigs_whale_1
whale_1  |  ________
whale_1  | < hello! >
whale_1  |  --------
whale_1  |     
whale_1  |      
whale_1  |       
whale_1  |                     ##        .
whale_1  |               ## ## ##       ==
whale_1  |            ## ## ## ##      ===
whale_1  |        /""""""""""""""""___/ ===
whale_1  |   ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~
whale_1  |        ______ o          __/
whale_1  |                     __/
whale_1  |           __________/
composeconfigs_whale_1 exited with code 0

As said, docker compose supports merging multiple compose-
files, this allows you to override various options in the second file. For example:

$ cat docker-compose.second.yml
version: "3.2"
services:
  whale:
    command: ["cowsay", "bye!"]

$ docker-compose -f docker-compose.yml -f docker-compose.second.yml up
Creating composeconfigs_whale_1
Attaching to composeconfigs_whale_1
whale_1  |  ______
whale_1  | < bye! >
whale_1  |  ------
whale_1  |     
whale_1  |      
whale_1  |       
whale_1  |                     ##        .
whale_1  |               ## ## ##       ==
whale_1  |            ## ## ## ##      ===
whale_1  |        /""""""""""""""""___/ ===
whale_1  |   ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~
whale_1  |        ______ o          __/
whale_1  |                     __/
whale_1  |           __________/
composeconfigs_whale_1 exited with code 0

This syntax is not very convenient in the development process, when the command
need to be done multiple times.

Luckily, docker compose automatically looks for a special file called
Docker-compose.override.yml to override values docker-compose.yml. If
rename the second file, you will get the same result, only using the original command:

$ mv docker-compose.second.yml docker-compose.override.yml
$ docker-compose up
Starting composeconfigs_whale_1
Attaching to composeconfigs_whale_1
whale_1  |  ______
whale_1  | < bye! >
whale_1  |  ------
whale_1  |     
whale_1  |      
whale_1  |       
whale_1  |                     ##        .
whale_1  |               ## ## ##       ==
whale_1  |            ## ## ## ##      ===
whale_1  |        /""""""""""""""""___/ ===
whale_1  |   ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~
whale_1  |        ______ o          __/
whale_1  |                     __/
whale_1  |           __________/
composeconfigs_whale_1 exited with code 0

Okay, that's easier to remember.

Variable Interpolation

Configuration files support interpolation
variables and default values. That is, you can do the following:

services:
  my-service:
    build:
      context: .
    image: private.registry.mine/my-stack/my-service:${MY_SERVICE_VERSION:-latest}
...

And if you do docker-compose build (or push) without environment variable
$MY_SERVICE_VERSION, the value will be used latestbut if you set
the value of the environment variable before the build, it will be used when building or pushing
to register private.registry.mine.

My principles

Approaches that are convenient for me may be useful for you. I follow this
simple rules:

• All my stacks for production, development (or other environments) are defined through
  docker-compose files.
• The configuration files needed to cover all my environments, max.
  avoid duplication.
• I need one simple command to work in every environment.
• The main configuration is defined in the file docker-compose.yml.
• Environment variables are used to define image tags or other
  variables that can change from environment to environment (staging, integration,
  production).
• The values ​​of variables for production are used as values ​​by
  by default, this minimizes the risks if you run the stack in production without
  set environment variable.
• To start a service in a production environment, use the command docker stack deploy - compose-file docker-compose.yml - with-registry-auth my-stack-name.
• The working environment is started with the command Docker -Compose up -D.

Let's look at a simple example.

# docker-compose.yml
...
services:
  my-service:
    build:
      context: .
    image: private.registry.mine/my-stack/my-service:${MY_SERVICE_VERSION:-latest}
    environment:
      API_ENDPOINT: ${API_ENDPOINT:-https://production.my-api.com}
...

И

# docker-compose.override.yml
...
services:
  my-service:
    ports: # This is needed for development!
      - 80:80
    environment:
      API_ENDPOINT: https://devel.my-api.com
    volumes:
      - ./:/project/src
...

I can use docker-compose (docker-compose up)to run the stack at
development mode with source code mounted in /Project/SRC.

I can use the same files in production! And I could use exactly
same file docker-compose.yml for staging. To expand this to
production, I just need to build and send the image with a predefined tag
at the CI stage:

export MY_SERVICE_VERSION=1.2.3
docker-compose -f docker-compose.yml build
docker-compose -f docker-compose.yml push

In production, this can be run with the following commands:

export MY_SERVICE_VERSION=1.2.3
docker stack deploy my-stack --compose-file docker-compose.yml --with-registry-auth

And if you want to do the same on stage, you just need to define
necessary environment variables to work in the staging environment:

export MY_SERVICE_VERSION=1.2.3
export API_ENDPOINT=http://staging.my-api.com
docker stack deploy my-stack --compose-file docker-compose.yml --with-registry-auth

As a result, we used two different docker-compose files, which, without
duplicate configurations can be used for any of your environments!

Learn more about the course "Linux Administrator"

Source: habr.com