I wrote my first websites in the late '90s. Back then, getting them up and running was very simple. There was an Apache server on some shared hosting, and you could log into this server via FTP by typing something like ftp://ftp.example.com. Then you had to enter a username and password and upload files to the server. Those were different times; everything was easier back then than it is now.
Over the past two decades, everything has changed quite a bit. Websites have become more complex; they need to be built before being released into production. A single server has transformed into multiple servers working behind load balancers, and using version control systems has become commonplace.
For my personal project, I had a special configuration. I knew that I needed the ability to deploy the site to production with just one action: pushing code to the branch master on GitHub. Moreover, I also knew that to keep my small web application running, I didn't want to manage a huge Kubernetes cluster, use Docker Swarm technology, or maintain a fleet of servers with pods, agents, and various other complexities. To achieve my goal of maximizing simplicity, I needed to get acquainted with CI/CD.
If you have a small project (in our case, a Node.js project) and you would like to know how to automate the deployment of this project, ensuring that what is stored in the repository exactly matches what is running in production, then this article may interest you.
Prerequisites
It is expected that the reader of this article has basic knowledge of working with the command line and writing Bash scripts. Additionally, you will need accounts for and .
Objectives
I won't say that this article can be unequivocally called a 'tutorial.' It is more of a document where I share what I have learned and describe the process for testing and deploying code in production that I find satisfactory, executed in a single automated pass.
This is what my workflow ultimately looks like.
For code pushed to any branch of the repository except master, the following actions are carried out:
- The project build is triggered on Travis CI.
- All unit, integration, and end-to-end tests are executed.
Only for the code that falls into master, the following is executed:
- Everything mentioned above, plus…
- Building a Docker image based on the current code, settings, and environment.
- Placing the image on Docker Hub.
- Connecting to the production server.
- Pulling the image from Docker Hub to the server.
- Stopping the current container and starting a new one based on the new image.
If you know nothing about Docker, images, and containers—don't worry. I will explain all of this to you.
What is CI/CD?
The abbreviation CI/CD stands for "continuous integration/continuous deployment".
▍Continuous Integration
Continuous integration is the process in which developers commit to the main code repository of the project (usually to the branch master). The quality of the code is ensured through automated testing.
▍Continuous Deployment
Continuous deployment is the frequent automated deployment of code to production. The second part of the CI/CD abbreviation is sometimes revealed as "continuous delivery." This is generally the same as "continuous deployment," but "continuous delivery" implies the need for manual approval of changes before initiating the project's deployment process.
Getting Started
The application where I learned all of this is called . This is a web project I am working on, designed for taking notes. Initially, I tried to make a -project, or just a frontend application without a server, to take advantage of the standard hosting and deployment features offered by . As the complexity of the application grew, I needed to create its backend portion, which meant I needed to formulate my own strategy for automated integration and automated deployment of the project.
In my case, the application consists of an Express server running in a Node.js environment, serving a single-page React application and supporting a secure server-side API. This architecture follows a strategy found in full-stack authentication guide.
I consulted with , who is an automation expert, and asked him what I needed to do for everything to work as I needed. He suggested an idea of how the automated workflow should look, as outlined in the Goals section of this article. Setting such goals for myself meant I needed to figure out how to use Docker.
Docker
Docker is a tool that, thanks to containerization technology, allows for easy distribution of applications, as well as their deployment and execution in the same environment, even if Docker itself is running in various environments. To start, I needed to get the Docker command-line tools (CLI). for installing Docker cannot be called very clear and understandable, but from it, you can learn that to take the first step in installation, you need to download Docker Desktop (for Mac or Windows).
Docker Hub is roughly the same as for git repositories, or a registry for JavaScript packages. It is an online repository for Docker images. This is where Docker Desktop connects.
So, to start working with Docker, you need to do two things:
- Install .
- Sign up at .
After that, you can check if Docker CLI is working by running the following command to check the Docker version:
docker -vNext, log in to Docker Hub by entering your username and password when prompted:
docker loginTo use Docker, you must understand the concepts of images and containers.
▍Images
An image is somewhat like a blueprint containing instructions for building a container. It is an immutable snapshot of the file system and application settings. Developers can easily share images.
# Вывод сведений обо всех образах
docker imagesThis command will output a table with the following header:
REPOSITORY TAG IMAGE ID CREATED SIZE
---Next, we will consider some command examples in the same format — the command comes first with a comment, followed by an example of what it might output.
▍Containers
A container is an executable package that includes everything needed to run an application. With this approach, the application will always run consistently, regardless of the infrastructure: in an isolated environment and in the same environment. This means that instances of the same image are launched in different environments.
# Перечисление всех контейнеров
docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
---▍Tags
A tag indicates a specific version of an image.
▍Brief Overview of Docker Commands
Here is an overview of some commonly used Docker commands.
The command
Context
Action
Type
Building an image from a Dockerfile
Type
Tagging an image
Type
Listing images
Container
Running a container based on an image
Type
Pushing an image to a registry
Type
Pulling an image from a registry
Container
Listing containers
Image/Container
Removing unused containers and images
▍Dockerfile
I know how to locally run an application for production. I have a Webpack configuration designed to build a ready-to-use React application. Next, I have a command that runs a server based on Node.js on port 5000. It looks like this:
npm i # installing dependencies
npm run build # building the React application
npm run start # starting the Node serverIt is worth noting that I do not have a sample application for this material. However, for experimentation, any simple Node application will suffice.
To make use of the container, you will need to give Docker instructions. This is done through a file called Dockerfile, located in the root directory of the project. Initially, this file may seem quite obscure.
But what it contains simply describes, with specific commands, something akin to setting up a working environment. Here are some of these commands:
- — This command initiates the file. It specifies the base image upon which the container is built.
- — Copying files from a local source into the container.
- — Setting the working directory for the following commands.
- — Running commands.
- — Setting up the port.
- — Specifying the command to execute.
Dockerfile might look something like this:
# Загрузить базовый образ
FROM node:12-alpine
# Скопировать файлы из текущей директории в директорию app/
COPY . app/
# Использовать app/ в роли рабочей директории
WORKDIR app/
# Установить зависимости (команда npm ci похожа npm i, но используется для автоматизированных сборок)
RUN npm ci --only-production
# Собрать клиентское React-приложение для продакшна
RUN npm run build
# Прослушивать указанный порт
EXPOSE 5000
# Запустить Node-сервер
ENTRYPOINT npm run startDepending on the selected base image, you may need to install additional dependencies. The fact is that some base images (like Node Alpine Linux) are created to be as compact as possible. As a result, they may lack certain programs that you rely on.
▍Building, tagging, and running the container
Local build and run of the container — this is, after we have Dockerfile, fairly straightforward tasks. Before pushing the image to Docker Hub, it needs to be tested locally.
▍Build
First, you need to build , specifying a name, and optionally a tag (if no tag is specified, the system will automatically assign a tag to the image latest).
# Сборка образа
docker build -t <image>:<tag> .After executing this command, you can see how Docker builds the image.
Sending build context to Docker daemon 2.88MB
Step 1/9 : FROM node:12-alpine
---> ...running build steps...
Successfully built 123456789123
Successfully tagged : Building may take a few minutes — it all depends on how many dependencies you have. Once the build is complete, you can execute the command docker images and take a look at the description of your new image.
REPOSITORY TAG IMAGE ID CREATED SIZE
latest 123456789123 About a minute ago x.xxGB▍Running
The image is created. This means that you can run a container based on it. Since I want to be able to access the application running in the container at localhost:5000, I specified in the left part of the pair 5000:5000 in the next command. 5000On the right side is the container port.
# Запуск с использованием локального порта 5000 и порта контейнера 5000
docker run -p 5000:5000 <image>:<tag> Now that the container is created and running, you can use the command docker ps to view the details of this container (or you can use the command docker ps -a, which shows details about all containers, not just the running ones).
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
987654321234 "\/bin\/sh -c 'npm run…" 6 seconds ago Up 6 seconds 0.0.0.0:5000->5000\/tcp stoic_darwin If you navigate to localhost:5000 — you can see the page of the running application, which looks exactly like the page of the application running in the production environment.
▍Assigning a tag and publishing
To use one of the created images on a production server, we need to be able to upload this image to Docker Hub. This means that we first need to create a repository for the project on Docker Hub. After this, we will have a location to which we can push the image. The image should be renamed so that its name starts with our username on Docker Hub, followed by the repository name. At the end of the name, any tag may be placed. Below is an example of how to name images according to this scheme.
Now we can build the image, giving it a new name and executing the command docker push to push it to the Docker Hub repository.
docker build -t /: .
docker tag /: /:latest
docker push /:
# In practice, this might look like:
docker build -t user/app:v1.0.0 .
docker tag user/app:v1.0.0 user/app:latest
docker push user/app:v1.0.0If everything goes as planned, the image will be available on Docker Hub, and it will be easy to download it to the server or share it with other developers.
Next Steps
By this point, we have confirmed that the application, in the form of a Docker container, works locally. We have uploaded the container to Docker Hub. All of this means that we have already made significant progress towards our goal. Now we need to address two more issues:
- Setting up a CI tool for testing and deploying the code.
- Configuring the production server so that it can download and run our code.
In our case, the CI/CD solution used is . As the server — .
It should be noted that another combination of services can also be used here. For example, instead of Travis CI, one could use CircleCI or Github Actions. And instead of DigitalOcean — AWS or Linode.
We decided to work with Travis CI, and I already have some settings configured in this service. Therefore, I will briefly explain how to prepare it for use.
Travis CI
Travis CI is a tool for testing and deploying code. I wouldn't like to delve into the specifics of configuring Travis CI, as each project is unique, and that wouldn't bring much benefit. However, I will talk about the basics that will help you get started in case you decide to use Travis CI. Regardless of what you choose — Travis CI, CircleCI, Jenkins, or something else, similar configuration methods will be applied.
To get started with Travis CI, go to and create an account. Then, integrate Travis CI with your GitHub account. During the setup process, you'll need to specify the repository you want to automate and enable access to it. (I use GitHub but I'm sure Travis CI can integrate with BitBucket, GitLab, and other similar services as well).
Every time Travis CI starts working, a server is launched to execute the commands specified in the configuration file, including deployment of the corresponding repository branches.
▍Job Lifecycle
The Travis CI configuration file, called .travis.yml and stored in the root directory of the project, supports the concept of events of the job. Here are these events, listed in the order they occur:
apt addonscache componentsbefore_installinstallbefore_scriptscriptbefore_cacheafter_success or after_failurebefore_deploydeployafter_deployafter_script
▍Testing
In the configuration file, I am going to set up a local Travis CI server. I chose Node version 12 as the language and instructed the system to install the dependencies necessary for using Docker.
Everything listed in .travis.yml, will be executed for all pull requests to all repository branches unless stated otherwise. This is a useful feature since it means we can test all code coming into the repository. It allows us to know whether the code is ready to be merged into the branch master, and whether it will disrupt the build process. In this global configuration, I set everything locally, run the Webpack development server in the background (this is part of my workflow), and run the tests.
If you want your repository to display badges with code coverage information, you can find a brief guide on using Jest, Travis CI, and Coveralls to collect and display this information.
So, here is the contents of the file .travis.yml:
# Установить язык
language: node_js
# Установить версию Node.js
node_js:
- '12'
services:
# Использовать командную строку Docker
- docker
install:
# Установить зависимости для тестов
- npm ci
before_script:
# Запустить сервер и клиент для тестов
- npm run dev &
script:
# Запустить тесты
- npm run testThis marks the end of the actions performed for all branches of the repository and for pull requests.
▍Deployment
Assuming that all automated tests have completed successfully, we can optionally deploy the code to the production server. Since we want to do this only for code from the branch master, we provide the appropriate instructions in the deployment settings. Before you try to use the code we will discuss next in your project, I would like to warn you that you need to have a real script invoked for deployment.
deploy:
# Build the Docker container and push it to Docker Hub
provider: script
script: bash deploy.sh
on:
branch: masterThe deployment script solves two tasks:
- Building, tagging, and pushing the image to Docker Hub using a CI tool (in our case, Travis CI).
- Downloading the image on the server, stopping the old container, and starting the new one (in our case, the server runs on the DigitalOcean platform).
First, you need to set up an automatic process for building, tagging, and pushing the image to Docker Hub. This is very similar to what we have done manually, except that we need a strategy for assigning unique tags to the images and automating the login process. I encountered difficulties with some details of the deployment script, such as the tagging strategy, login, SSH key encoding, and establishing an SSH connection. Fortunately, my boyfriend is very skilled with bash, as well as many other things. He helped me write this script.
So, the first part of the script is uploading the image to Docker Hub. It’s relatively simple to do. The tagging scheme I used involves combining the git hash and the git tag if it exists. This ensures the creation of a unique tag and simplifies the identification of the build it is based on. DOCKER_USERNAME and DOCKER_PASSWORD — these are custom environment variables that can be set through the Travis CI interface. Travis CI will automatically handle sensitive data so that it doesn’t fall into the wrong hands.
Here is the first part of the script deploy.sh.
#!/bin/sh
set -e # Остановить скрипт при наличии ошибок
IMAGE="<username>/<repository>" # Образ Docker
GIT_VERSION=$(git describe --always --abbrev --tags --long) # Git-хэш и теги
# Сборка и тегирование образа
docker build -t ${IMAGE}:${GIT_VERSION} .
docker tag ${IMAGE}:${GIT_VERSION} ${IMAGE}:latest
# Вход в Docker Hub и выгрузка образа
echo "${DOCKER_PASSWORD}" | docker login -u "${DOCKER_USERNAME}" --password-stdin
docker push ${IMAGE}:${GIT_VERSION} What the second part of the script will be completely depends on which host you are using and how the connection to it is organized. In my case, since I am using DigitalOcean, the commands used to connect to the server are . When working with AWS, the utility aws, and so on.
Setting up the server was not particularly difficult. I configured a droplet based on the basic image. It's worth noting that the system I chose requires a one-time manual installation of Docker and a one-time manual start of Docker. For the installation of Docker, I used Ubuntu 18.04, so if you're also using Ubuntu, you can simply follow the simple guide.
I'm not talking about specific commands for the service, as this aspect can vary widely in different cases. I'll just outline a general plan of action performed after connecting via SSH to the server on which the project will be deployed:
- You need to find the container that is currently running and stop it.
- Then, you need to start a new container in the background.
- You will need to set the local server port to
80— this will allow accessing the site at an address likeexample.com, without specifying the port, rather than using an address likeexample.com:5000. - And finally, you need to remove all old containers and images.
Here’s the continuation of the script.
# Найти ID работающего контейнера
CONTAINER_ID=$(docker ps | grep takenote | cut -d" " -f1)
# Остановить старый контейнер, запустить новый, очистить систему
docker stop ${CONTAINER_ID}
docker run --restart unless-stopped -d -p 80:5000 ${IMAGE}:${GIT_VERSION}
docker system prune -a -fSome things to pay attention to
When you connect to the server via SSH from Travis CI, you may see a warning that will prevent the installation from continuing, as the system will wait for user input.
The authenticity of host ' ()' can't be established.
RSA key fingerprint is .
Are you sure you want to continue connecting (yes/no)? I learned that the string key can be encoded in base64 so that it can be saved in a format that is convenient and secure to work with. At the installation stage, the public key can be decoded and written to the file known_hosts to get rid of the aforementioned error.
echo | base64 # outputsIn practice, this command might look like:
echo "123.45.67.89 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAklOUpkDHrfHY17SbrmTIpNLTGK9Tjom/BWDSU
GPl+nafzlHDTYW7hdI4yZ5ew18JH4JW9jbhUFrviQzM7xlELEVf4h9lFX5QVkbPppSwg0cda3
Pbv7kOdJ/MTyBlWXFCR+HAo3FXRitBqxiX1nKhXpHAZsMciLq8V6RjsNAQwdsdMFvSlVK/7XA
t3FaoJoAsncM1Q9x5+3V0Ww68/eIFmb1zuUFljQJKprrX88XypNDvjYNby6vw/Pb0rwert/En
mZ+AW4OZPnTPI89ZPmVMLuayrD2cE86Z/il8b+gw3r3+1nKatmIkjn2so1d01QraTlMqVSsbx
NrRFi9wrf+M7Q== you@example.com" | base64And here's what it outputs — a base64 encoded string:
123.45.67.89 ssh-rsa AABBA3NzaC1yc2EAAABBQW5sUmNrWm9nLjk3eHk1Y3JlQUFBQUJJd0FBQVFFQWtsT1Vwa0RIcmZIWTE3U2JybVRJcE5MVEdLOVRqb20vQldEU1UKR1BsK25hZnpsSERUWVc3aGRJNHlaNWV3MThKSDRKVzlqYmhVRnJ2aVF6TTd4bEVMRVZmNGg5bEZYNVFWa2JQcHBTd2cwY2RhMwpQYnY3a09kSi9NVHlCbFdYRkNSK0hBbzNGWFJpdEJxeGlYMW5LaFhwSEFac01jaUxxOFY2UmpzTkFRd2RzZE1GdlNsVksvN1hBCnQzRmFvSm9Bc25jTTFROXg1KzNWMFd3NjgvZUlGbWIxenVVRmxqUUpLcHJyWDg4WHlwTkR2allOYnk2dncvUGIwcndlcnQvRW4KbVorQVc0T1pQblRQSTg5WlBtVk1MdWF5ckQyY0U4NlovaWw4YitndzNyMysxbkthdG1Ja2puMnNvMWQwMVFyYVRsTXFWU3NieApOclJGaTl3cmYrTTdRPT0geW91QGV4YW1wbGUuY29tCg==Here's the command mentioned above
install:
- echo | base64 -d >> $HOME/.ssh/known_hostsThe same approach can be used with a private key when establishing a connection, as you might need the private key to access the server. When working with the key, you just need to ensure its secure storage in a Travis CI environment variable, and that it's not output anywhere.
Another thing to consider is that you might need to run the entire deployment script as a single line, for example — using doctl. This may require some additional effort.
doctl compute ssh --ssh-command "all commands will go here && here"TLS/SSL and load balancing
After I completed everything mentioned above, the last issue I faced was that the server did not have SSL. Since I use a Node.js server, to enable reverse proxy Nginx and Let’s Encrypt, it requires some effort.
I really didn’t want to go through all those SSL configurations manually, so I simply created a load balancer and recorded its details in DNS. In the case of DigitalOcean, for example, creating a self-signed certificate that auto-renews on the load balancer is a simple, free, and quick procedure. This approach also has the added benefit of easily allowing SSL configuration on multiple servers behind the load balancer, enabling the servers themselves to not have to 'worry' about SSL, while still using port 80. So, configuring SSL on the load balancer is much easier and more convenient than alternative SSL configuration methods.
Now you can close all ports on the server that accept incoming connections — except for port 80, which is used for communication with the load balancer, and port 22 for SSH. As a result, attempts to directly access the server through any ports except these two will fail.
Summary
After following all the steps outlined in this material, I was no longer intimidated by the Docker platform or the concepts of automated CI/CD pipelines. I was able to set up a continuous integration chain, during which code testing occurs before it reaches production, along with automatic code deployment on the server. Everything is still relatively new to me, and I am confident that there are ways to improve my automated workflow and make it more efficient. So if you have any ideas on this matter, please let me know. I hope this article has helped you in your endeavors. I like to believe that by reading it, you learned as much as I did while figuring out everything discussed here.
P.S. In our we have an image , which can be installed with a single click. You can check the operation of containers on . All new clients receive 3 days of free testing.
Dear readers! Do you use CI/CD technologies in your projects?
Source: habr.com
