In this article, we will use , , and to organize a seamless deployment of a web application. is a technique that allows you to update an application instantly without dropping any requests. It is one of the strategies for zero downtime deployment and is best suited for applications with a single instance but the possibility of loading a second, fully functional instance alongside it.
Let’s say you have a web application that many users actively rely on, and it absolutely must not go down even for a couple of seconds. However, you need to deploy an update, fix a bug, or introduce a cool new feature. In a typical scenario, you would need to stop the application, replace it, and then restart it. In the case of Docker, you can first replace it and then restart, but there will still be a period when requests to the application won't be processed, as it usually takes some time for the application to load initially. And what if it starts but turns out to be non-functional? Let's tackle this task with minimal tools and maximum elegance.
DISCLAIMER: Most of this article is presented in an experimental format — as a recording of a console session. I hope it's not too difficult to digest, and this code sufficiently documents itself. To create atmosphere, imagine that these are not just code snippets, but paper from an 'iron' teleprinter.
Interesting techniques that are hard to find just by reading code are explained at the beginning of each section. If something is unclear — look it up on (thankfully, it's working again due to the unblocking of Telegram). For anything that isn't listed online — feel free to ask in the comments. I'd be happy to add to the relevant section 'Interesting Techniques.'
Let's get started.
$ mkdir blue-green-deployment && cd $_The service
Let's create a test service and put it in a container.
Interesting Techniques
cat < file-name( + ) — a way to create a multi-line file with a single command. Everything bash reads from/dev/stdinthis line to the lineEOFwill be written tofile-name.wget -qO- URL() — output the document received via HTTP to/dev/stdout(analogous tocurl URL).
Output
I intentionally break the snippet to enable highlighting for Python. There will be another such piece at the end. Consider that in these places the paper was cut to be sent to the highlighting department (where code was manually colored with highlighters), and then these pieces were glued back.
$ cat < uptimer.pyfrom http.server import BaseHTTPRequestHandler, HTTPServer
from time import monotonic
app_version = 1
app_name = f'Uptimer v{app_version}.0'
loading_seconds = 15 - app_version * 5
class Handler(BaseHTTPRequestHandler):
def do_GET(self):
if self.path == '/':
try:
t = monotonic() - server_start
if t < loading_seconds:
self.send_error(503)
else:
self.send_response(200)
self.send_header('Content-Type', 'text/html')
self.end_headers()
response = f'<h2>{app_name} is running for {t:3.1f} seconds.</h2>n'
self.wfile.write(response.encode('utf-8'))
except Exception:
self.send_error(500)
else:
self.send_error(404)
httpd = HTTPServer(('', 8080), Handler)
server_start = monotonic()
print(f'{app_name} (loads in {loading_seconds} sec.) started.')
httpd.serve_forever()EOF
$ cat << EOF > Dockerfile
FROM python:alpine
EXPOSE 8080
COPY uptimer.py app.py
CMD [ "python", "-u", ".\/app.py" ]
EOF
$ docker build --tag uptimer .
Sending build context to Docker daemon 39.42kB
Step 1\/4 : FROM python:alpine
---> 8ecf5a48c789
Step 2\/4 : EXPOSE 8080
---> Using cache
---> cf92d174c9d3
Step 3\/4 : COPY uptimer.py app.py
---> a7fbb33d6b7e
Step 4\/4 : CMD [ "python", "-u", ".\/app.py" ]
---> Running in 1906b4bd9fdf
Removing intermediate container 1906b4bd9fdf
---> c1655b996fe8
Successfully built c1655b996fe8
Successfully tagged uptimer:latest
$ docker run --rm --detach --name uptimer --publish 8080:8080 uptimer
8f88c944b8bf78974a5727070a94c76aa0b9bb2b3ecf6324b784e782614b2fbf
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
8f88c944b8bf uptimer "python -u .\/app.py" 3 seconds ago Up 5 seconds 0.0.0.0:8080->8080\/tcp uptimer
$ docker logs uptimer
Uptimer v1.0 (loads in 10 sec.) started.
$ wget -qSO- http:\/\/localhost:8080
HTTP\/1.0 503 Service Unavailable
Server: BaseHTTP\/0.6 Python\/3.8.3
Date: Sat, 22 Aug 2020 19:52:40 GMT
Connection: close
Content-Type: text\/html;charset=utf-8
Content-Length: 484
$ wget -qSO- http:\/\/localhost:8080
HTTP\/1.0 200 OK
Server: BaseHTTP\/0.6 Python\/3.8.3
Date: Sat, 22 Aug 2020 19:52:45 GMT
Content-Type: text\/html
<h2>Uptimer v1.0 has been running for 15.4 seconds.</h2>
$ docker rm --force uptimer
uptimerReverse proxy
In order for our application to be able to switch seamlessly, there needs to be some entity in front of it that will hide its replacement. This can be a web server. downward API support (simultaneously with this in . The reverse proxy is set up between the client and the application. It receives requests from clients and forwards them to the application, while directing the application’s responses back to the clients.
The application and reverse proxy can be linked within Docker using . This way, the application container doesn’t even need to expose a port to the host system, allowing you to isolate the application from external threats as much as possible.
If the reverse proxy will be on a different host, you will have to abandon docker network and connect the application to the reverse proxy through the host network by exposing a port. of the application parameter --publish, just like at the first launch and as with the reverse proxy.
We will run the reverse proxy on port 80, as it is the entity that should listen to the outside. If port 80 is occupied on your test host, change the parameter. --publish 80:80 to --publish ANY_FREE_PORT:80.
Interesting Techniques
- “In user-defined docker networks, containers can be linked not only by IP address. The container name is also resolved to its IP.” (, item 5 of the docker guidelines).
Output
$ docker network create web-gateway
5dba128fb3b255b02ac012ded1906b7b4970b728fb7db3dbbeccc9a77a5dd7bd
$ docker run --detach --rm --name uptimer --network web-gateway uptimer
a1105f1b583dead9415e99864718cc807cc1db1c763870f40ea38bc026e2d67f
$ docker run --rm --network web-gateway alpine wget -qO- http://uptimer:8080
<h2>Uptimer v1.0 has been running for 11.5 seconds.</h2>
$ docker run --detach --publish 80:80 --network web-gateway --name reverse-proxy nginx:alpine
80695a822c19051260c66bf60605dcb4ea66802c754037704968bc42527bf120
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
80695a822c19 nginx:alpine "/docker-entrypoint.…" 27 seconds ago Up 25 seconds 0.0.0.0:80->80/tcp reverse-proxy
a1105f1b583d uptimer "python -u ./app.py" About a minute ago Up About a minute 8080/tcp uptimer
$ cat << EOF > uptimer.conf
server {
listen 80;
location / {
proxy_pass http://uptimer:8080;
}
}
EOF
$ docker cp ./uptimer.conf reverse-proxy:/etc/nginx/conf.d/default.conf
$ docker exec reverse-proxy nginx -s reload
2020/06/23 20:51:03 [notice] 31#31: signal process started
$ wget -qSO- http://localhost
HTTP/1.1 200 OK
Server: nginx/1.19.0
Date: Sat, 22 Aug 2020 19:56:24 GMT
Content-Type: text/html
Transfer-Encoding: chunked
Connection: keep-alive
<h2>Uptimer v1.0 has been running for 104.1 seconds.</h2>Seamless deployment
We will release a new version of the application (with a twofold boost in startup performance) and try to deploy it seamlessly.
Interesting Techniques
echo 'my text' | docker exec -i my-container sh -c 'cat > /my-file.txt'— Write textmy textin the file/my-file.txtinside the containermy-container.cat > /my-file.txt— Write the contents of standard input to a file/dev/stdin.
Output
$ sed -i "s/app_version = 1/app_version = 2/" uptimer.py
$ docker build --tag uptimer .
Sending build context to Docker daemon 39.94kB
Step 1/4 : FROM python:alpine
---> 8ecf5a48c789
Step 2/4 : EXPOSE 8080
---> Using cache
---> cf92d174c9d3
Step 3/4 : COPY uptimer.py app.py
---> 3eca6a51cb2d
Step 4/4 : CMD [ "python", "-u", ".\/app.py" ]
---> Running in 8f13c6d3d9e7
Removing intermediate container 8f13c6d3d9e7
---> 1d56897841ec
Successfully built 1d56897841ec
Successfully tagged uptimer:latest
$ docker run --detach --rm --name uptimer_BLUE --network web-gateway uptimer
96932d4ca97a25b1b42d1b5f0ede993b43f95fac3c064262c5c527e16c119e02
$ docker logs uptimer_BLUE
Uptimer v2.0 (loads in 5 sec.) started.
$ docker run --rm --network web-gateway alpine wget -qO- http://uptimer_BLUE:8080
<h2>Uptimer v2.0 has been running for 23.9 seconds.</h2>
$ sed s/uptimer/uptimer_BLUE/ uptimer.conf | docker exec --interactive reverse-proxy sh -c 'cat > /etc/nginx/conf.d/default.conf'
$ docker exec reverse-proxy cat /etc/nginx/conf.d/default.conf
server {
listen 80;
location / {
proxy_pass http://uptimer_BLUE:8080;
}
}
$ docker exec reverse-proxy nginx -s reload
2020/06/25 21:22:23 [notice] 68#68: signal process started
$ wget -qO- http://localhost
<h2>Uptimer v2.0 has been running for 63.4 seconds.</h2>
$ docker rm -f uptimer
uptimer
$ wget -qO- http://localhost
<h2>Uptimer v2.0 has been running for 84.8 seconds.</h2>
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
96932d4ca97a uptimer "python -u .\/app.py" About a minute ago Up About a minute 8080/tcp uptimer_BLUE
80695a822c19 nginx:alpine "/docker-entrypoint.…" 8 minutes ago Up 8 minutes 0.0.0.0:80->80/tcp reverse-proxyAt this stage, the image is being built directly on the server, which requires the application sources to be there and also burdens the server with unnecessary work. The next step will be to allocate the image building to a separate machine (for example, in a CI system), with subsequent transfer to the server.
Image transfer
Unfortunately, transferring images from localhost to localhost is pointless, so this section can only be explored if you have two hosts with Docker at hand. At a minimum, it looks something like this:
$ ssh production-server docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
$ docker image save uptimer | ssh production-server 'docker image load'
Loaded image: uptimer:latest
$ ssh production-server docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
uptimer latest 1d56897841ec 5 minutes ago 78.9MBThe command docker save saves the image data to a .tar archive, meaning it weighs about 1.5 times more than it would in compressed form. So let’s compress it in the name of saving time and traffic:
$ docker image save uptimer | gzip | ssh production-server 'zcat | docker image load'
Loaded image: uptimer:latestAdditionally, you can monitor the transfer process (though for this, you need an external utility):
$ docker image save uptimer | gzip | pv | ssh production-server 'zcat | docker image load'
25.7MiB 0:01:01 [ 425KiB/s] [ ]
Loaded image: uptimer:latestTip: If you require a bunch of parameters to connect to the server via SSH, you might not be using a file
~/.ssh/config.
Transfer of the image via docker image save/load — is the most minimalist method, but not the only one. There are others:
- Container Registry (industry standard).
- Connect to the Docker daemon of the server from another host:
- Environment variable
DOCKER_HOST. - Command-line option
-Hor--hosttooldocker-compose. docker context
- Environment variable
The second method (with three ways to implement it) is well described in the article .
deploy.sh
Now let's gather everything we've done manually into one script. We’ll start with the top-level function and then look at the others used within it.
Interesting Techniques
${parameter?err_msg}— is one of the spells of bash magic (aka ). Ifparameteris not specified, displayerr_msgand exit with code 1.docker --log-driver journald— by default, the Docker logging driver is a text file without any rotation. With this approach, logs quickly fill up the entire disk, so for production environments, it's necessary to switch to a smarter driver.
Deployment script
deploy() {
local usage_msg="Usage: ${FUNCNAME[0]} image_name"
local image_name=${1?$usage_msg}
ensure-reverse-proxy || return 2
if get-active-slot $image_name
then
local OLD=${image_name}_BLUE
local new_slot=GREEN
else
local OLD=${image_name}_GREEN
local new_slot=BLUE
fi
local NEW=${image_name}_${new_slot}
echo "Deploying '$NEW' in place of '$OLD'..."
docker run
--detach
--restart always
--log-driver journald
--name $NEW
--network web-gateway
$image_name || return 3
echo "Container started. Checking health..."
for i in {1..20}
do
sleep 1
if get-service-status $image_name $new_slot
then
echo "New '$NEW' service seems OK. Switching heads..."
sleep 2 # Ensure service is ready
set-active-slot $image_name $new_slot || return 4
echo "'$NEW' service is live!"
sleep 2 # Ensure all requests were processed
echo "Killing '$OLD'..."
docker rm -f $OLD
docker image prune -f
echo "Deployment successful!"
return 0
fi
echo "New '$NEW' service is not ready yet. Waiting ($i)..."
done
echo "New '$NEW' service did not raise, killing it. Failed to deploy T_T"
docker rm -f $NEW
return 5
}Used functions:
ensure-reverse-proxy— Ensures that the reverse proxy is running (useful for the first deployment)get-active-slot service_name— Determines which slot is currently active for the specified service (BLUEorGREEN)get-service-status service_name deployment_slot— Checks if the service is ready to handle incoming requestsset-active-slot service_name deployment_slot— Changes the nginx configuration in the reverse proxy container
In order:
ensure-reverse-proxy() {
is-container-up reverse-proxy && return 0
echo "Deploying reverse-proxy..."
docker network create web-gateway
docker run
--detach
--restart always
--log-driver journald
--name reverse-proxy
--network web-gateway
--publish 80:80
nginx:alpine || return 1
docker exec --interactive reverse-proxy sh -c "> /etc/nginx/conf.d/default.conf"
docker exec reverse-proxy nginx -s reload
}
is-container-up() {
local container=${1?"Usage: ${FUNCNAME[0]} container_name"}
[ -n "$(docker ps -f name=${container} -q)" ]
return $?
}
get-active-slot() {
local service=${1?"Usage: ${FUNCNAME[0]} service_name"}
if is-container-up ${service}_BLUE && is-container-up ${service}_GREEN; then
echo "Collision detected! Stopping ${service}_GREEN..."
docker rm -f ${service}_GREEN
return 0 # BLUE
fi
if is-container-up ${service}_BLUE && ! is-container-up ${service}_GREEN; then
return 0 # BLUE
fi
if ! is-container-up ${service}_BLUE; then
return 1 # GREEN
fi
}
get-service-status() {
local usage_msg="Usage: ${FUNCNAME[0]} service_name deployment_slot"
local service=${1?usage_msg}
local slot=${2?$usage_msg}
case $service in
# Add specific healthcheck paths for your services here
*) local health_check_port_path=":8080/" ;;
esac
local health_check_address="http://${service}_${slot}${health_check_port_path}"
echo "Requesting '$health_check_address' within the 'web-gateway' docker network:"
docker run --rm --network web-gateway alpine
wget --timeout=1 --quiet --server-response $health_check_address
return $?
}
set-active-slot() {
local usage_msg="Usage: ${FUNCNAME[0]} service_name deployment_slot"
local service=${1?$usage_msg}
local slot=${2?$usage_msg}
[ "$slot" == BLUE ] || [ "$slot" == GREEN ] || return 1
get-nginx-config $service $slot | docker exec --interactive reverse-proxy sh -c "cat > /etc/nginx/conf.d/$service.conf"
docker exec reverse-proxy nginx -t || return 2
docker exec reverse-proxy nginx -s reload
}Function get-active-slot requires a bit of clarification:
Why does it return a number instead of outputting a string?
Still, in the calling function, we check its execution result, and checking the exit code through bash is much simpler than parsing a string. Moreover, obtaining a string from it is quite easy:
get-active-slot service && echo BLUE || echo GREEN.
Are three conditions really enough to distinguish all states?
Even two are sufficient; the last one is just for completeness, to avoid writing else.
The only remaining undefined function is the one returning nginx configs: get-nginx-config service_name deployment_slot. By analogy with the health check, any config can be specified for any service here. Interestingly, only cat <<- EOF, which allows you to remove all tabs at the beginning. However, the price of neat formatting is mixed tabs with spaces, which is considered very poor practice today. But bash enforces tabs, and it would also be nice to have proper formatting in the nginx config. In short, mixing tabs with spaces seems like the best solution out of the worst options here. However, you won’t see this in the snippet below, as Habr 'does it well,' replacing all tabs with 4 spaces and making EOF invalid. .
To avoid repeating myself, I’ll tell you about
cat << 'EOF', which will come up later. If you simply writecat << EOF, interpolation of the string occurs within the heredoc (variables are expanded ($foo), command calls ($(bar)), etc.), and if you enclose the end-of-document marker in single quotes, then interpolation is disabled and the character$is output as is. This is necessary for inserting a script inside another script.
get-nginx-config() {
local usage_msg="Usage: ${FUNCNAME[0]} service_name deployment_slot"
local service=${1?$usage_msg}
local slot=${2?$usage_msg}
[ "$slot" == BLUE ] || [ "$slot" == GREEN ] || return 1
local container_name=${service}_${slot}
case $service in
# Add specific nginx configs for your services here
*) nginx-config-simple-service $container_name:8080 ;;
esac
}
nginx-config-simple-service() {
local usage_msg="Usage: ${FUNCNAME[0]} proxy_pass"
local proxy_pass=${1?$usage_msg}
cat << EOF
server {
listen 80;
location / {
proxy_pass http://$proxy_pass;
}
}
EOF
}This is the entire script. And here is for download via wget or curl.
Executing parameterized scripts on a remote server
It's time to reach out to the target server. This time localhost will do perfectly:
$ ssh-copy-id localhost
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
himura@localhost's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'localhost'"
and check to make sure that only the key(s) you wanted were added.We wrote a deployment script that transfers a pre-built image to the target server and seamlessly replaces the service container, but how can we execute it on a remote machine? The script has arguments because it is versatile and can deploy multiple services under one reverse proxy (nginx configurations can be adjusted to determine which service is available under which URL). The script cannot be stored on the server because we would not be able to automatically update it (for bug fixes and adding new services), and in general, state = evil.
Solution 1: Store the script on the server, but copy it each time via scp. Then connect via ssh and execute the script with the necessary arguments.
Cons:
- Two actions instead of one.
- The place where you copy may not exist, or may be inaccessible, or the script may execute at the time of replacement.
- It is advisable to clean up after yourself (delete the script).
- Now three actions.
Solution 2:
- Keep only function definitions in the script and do not execute anything.
- Using
sedAppend a function call at the end. - Send all this directly into shh through a pipe (
|)
Pros:
- Truly stateless.
- No boilerplate entities.
- Feeling cool.
But let's avoid Ansible. Yes, it’s all been thought of. Yes, a wheel. Look at how simple, elegant, and minimalist this wheel is:
$ cat < deploy.sh#!/bin/bash
usage_msg="Usage: $0 ssh_address local_image_tag"
ssh_address=${1?$usage_msg}
image_name=${2?$usage_msg}
echo "Connecting to '$ssh_address' via ssh to seamlessly deploy '$image_name'..."
( sed "$a deploy $image_name" | ssh -T $ssh_address ) << 'END_OF_SCRIPT'
deploy() {
echo "Yay! The '${FUNCNAME[0]}' function is executing on '$(hostname)' with argument '$1'"
}
END_OF_SCRIPTEOF
$ chmod +x deploy.sh
$ ./deploy.sh localhost magic-porridge-pot
Connecting to localhost...
Yay! The 'deploy' function is executing on 'hut' with argument 'magic-porridge-pot'.However, we cannot be sure that there is a proper bash on the remote host, so let's add a small check at the beginning (instead of ):
if [ "$SHELL" != "/bin/bash" ]
then
echo "The '$SHELL' shell is not supported by 'deploy.sh'. Set a '/bin/bash' shell for '$USER@$HOSTNAME'."
exit 1
fiAnd now everything is for real:
$ docker exec reverse-proxy rm /etc/nginx/conf.d/default.conf
$ wget -qO deploy.sh https://git.io/JUURc
$ chmod +x deploy.sh
$ ./deploy.sh localhost uptimer
Sending gzipped image 'uptimer' to 'localhost' via ssh...
Loaded image: uptimer:latest
Connecting to 'localhost' via ssh to seamlessly deploy 'uptimer'...
Deploying 'uptimer_GREEN' in place of 'uptimer_BLUE'...
06f5bc70e9c4f930e7b1f826ae2ca2f536023cc01e82c2b97b2c84d68048b18a
Container started. Checking health...
Requesting 'http://uptimer_GREEN:8080/' within the 'web-gateway' docker network:
HTTP/1.0 503 Service Unavailable
wget: server returned error: HTTP/1.0 503 Service Unavailable
New 'uptimer_GREEN' service is not ready yet. Waiting (1)...
Requesting 'http://uptimer_GREEN:8080/' within the 'web-gateway' docker network:
HTTP/1.0 503 Service Unavailable
wget: server returned error: HTTP/1.0 503 Service Unavailable
New 'uptimer_GREEN' service is not ready yet. Waiting (2)...
Requesting 'http://uptimer_GREEN:8080/' within the 'web-gateway' docker network:
HTTP/1.0 200 OK
Server: BaseHTTP/0.6 Python/3.8.3
Date: Sat, 22 Aug 2020 20:15:50 GMT
Content-Type: text/html
New 'uptimer_GREEN' service seems OK. Switching heads...
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
2020/08/22 20:15:54 [notice] 97#97: signal process started
The 'uptimer_GREEN' service is live!
Killing 'uptimer_BLUE'...
uptimer_BLUE
Total reclaimed space: 0B
Deployment successful!Now you can open in the browser, run the deployment again and verify that it proceeds seamlessly by refreshing the page frequently during the rollout.
Don't forget to clean up after work :3
$ docker rm -f uptimer_GREEN reverse-proxy
uptimer_GREEN
reverse-proxy
$ docker network rm web-gateway
web-gateway
$ cd ..
$ rm -r blue-green-deploymentSource: habr.com
