CI/CD in Github Actions for a Flask+Angular project
In this article, I will share my experience of setting up CI/CD using the Plesk Control Panel and Github Actions. Today we will learn how to deploy a simple project with the uncomplicated name "Helloworld". It is written in the Flask Python framework, with Celery workers and an Angular 8 frontend.
In the first part of the article, we will look at our project and its parts. In the second, we will figure out how to set up Plesk and install the necessary extensions and components (DB, RabbitMQ, Redis, Docker, etc.).
In the third part, we will finally figure out how to set up a pipeline for deploying our project to a server in a dev and prod environment. And then we will launch the site on the server.
And yes, I forgot to introduce myself. My name is Oleg Borzov, I'm a fullstack developer in the CRM team for mortgage managers at Domclick.
project Overview
First, let's look at two project repositories - backend and front - and go over the code.
Backend: Flask+Celery
For the back part, I took a bunch that is quite popular among Python developers: the Flask framework (for the API) and Celery (for the task queue). SQLAchemy is used as ORM. Alembic is used for migrations. For JSON validation in handles - Marshmallow.
Π repositories there is a Readme.md file with a detailed description of the structure and instructions for running the project.
Web Part API quite uncomplicated, consists of 6 pens:
/ping - to check availability;
handles for registration, authorization, de-authorization and obtaining an authorized user;
an email handle that puts a task in the Celery queue.
Celery part even easier, there is only one problem send_mail_task.
Main page with a form for sending email and an exit button.
Login page.
Registration page.
The main page looks ascetic:
There are two files at the root Dockerfile ΠΈ docker-compose.yml, as well as the familiar folder .ci-cd with slightly fewer scripts than in the back repository (removed scripts for running tests).
Starting a project in Plesk
Let's start by setting up Plesk and creating a subscription for our site.
Installing extensions
In Plesk, we need four extensions:
Docker to manage and visually display the status of containers in the Plesk admin panel;
Git to configure the deployment step on the server;
Let's Encrypt to generate (and auto-renew) free TLS certificates;
Firewall to configure filtering of incoming traffic.
You can install them through the Plesk admin panel in the Extensions section:
We will not consider the detailed settings for extensions, the default settings will do for our demo purposes.
Create a subscription and site
Next, we need to create a subscription for our helloworld.ru website and add the dev.helloworld.ru subdomain there.
Create a subscription for the helloworld.ru domain and specify the login-password for the system user:
Check the box at the bottom of the page Secure the domain with Let's Encryptif we want to set up HTTPS for the site:
Next, in this subscription, create a subdomain dev.helloworld.ru (for which you can also issue a free TLS certificate):
Installing Server Components
We have a server with OS Debian Stretch 9.12 and installed control panel Plesk Obsidian 18.0.27.
We need to install and configure for our project:
PostgreSQL (in our case, there will be one server with two databases for dev and prod environments).
RabbitMQ (same, same instance with different vhosts for environments).
Two Redis instances (for dev and prod environments).
Docker Registry (for local storage of built Docker images).
UI for Docker registry.
PostgreSQL
Plesk already comes with PostgreSQL DBMS, but not the latest version (at the time of writing Plesk Obsidian supported Postgres versions 8.4β10.8). We want the latest version for our application (12.3 at the time of this writing), so we will install it manually.
There are a lot of detailed instructions for installing Postgres on Debian on the net (example), so I wonβt describe them in detail, Iβll just give the commands:
Considering that PostgreSQL has rather mediocre default settings, it is necessary to correct the configuration. This will help us calculator: you need to drive in the parameters of your server and replace the settings in the file /etc/postgresql/12/main/postgresql.confto those offered. It should be noted here that such calculators are not a magic bullet, and the base should be tuned more precisely, based on your hardware, application, and query complexity. But this is enough to get started.
In addition to the settings proposed by the calculator, we also change in postgresql.confthe default port 5432 to another (in our example - 53983).
After changing the configuration file, restart postgresql-server with the command:
service postgresql restart
We have installed and configured PostgreSQL. Now let's create a database, users for dev and prod environments, and give users rights to manage the database:
$ su - postgres
postgres:~$ create database hw_dev_db_name;
CREATE DATABASE
postgres:~$ create user hw_dev_db_user with password 'hw_dev_db_password';
CREATE ROLE
postgres:~$ grant ALL privileges ON database hw_dev_db_name to hw_dev_db_user;
GRANT
postgres:~$ create database hw_prod_db_name;
CREATE DATABASE
postgres:~$ create user hw_prod_db_user with password 'hw_prod_db_password';
CREATE ROLE
postgres:~$ grant ALL privileges ON database hw_prod_db_name to hw_prod_db_user;
GRANT
Rabbit MQ
Let's move on to installing RabbitMQ, a message broker for Celery. Installing it on Debian is quite simple:
Now let's install and configure the last component for our application - Redis. It will be used as a backend for storing the results of Celery tasks.
We will raise two Docker containers with Redis for dev and prod environments using the extension Docker for Plesk.
We go to Plesk, go to the Extensions section, look for the Docker extension and install it (we need a free version):
Go to the installed extension, find the image through the search redis bitnami and install the latest version:
We go into the downloaded container and adjust the configuration: specify the port, the maximum allocated RAM size, the password in the environment variables, and mount the volume:
We perform steps 2-3 for the prod container, in the settings we only change the parameters: port, password, RAM size and path to the volume folder on the server:
Docker Registry
In addition to basic services, it would be nice to put your own Docker image repository on the server. Fortunately, server space is now quite cheap (certainly cheaper than a DockerHub subscription), and the process of setting up a private repository is very simple.
Let's create two subdomains in Plesk in our subscription: docker.helloworld.ru and docker-ui.helloworld.ru, and configure Let's Encrypt certificates for them.
Add the file to the docker.helloworld.ru subdomain folder docker-compose.yml with content like this:
And we need to redirect Nginx to our containers. This can be done through Plesk.
The following steps need to be done for the docker.helloworld.ru and docker-ui.helloworld.ru subdomains:
In section DevTools our site go to Docker Proxy Rules:
And add a rule to proxy incoming traffic to our container:
We check that we can log in to our container from the local machine:
$ docker login docker.helloworld.ru -u hw_docker_admin -p hw_docker_password
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Login Succeeded
Let's also check the operation of the docker-ui.helloworld.ru subdomain:
When you click on Browse repositories, the browser will display an authorization window where you will need to enter the username and password for the repository. After that, we will be transferred to a page with a list of repositories (for now, it will be empty for you):
Opening ports in Plesk Firewall
After installing and configuring the components, we need to open ports so that the components are accessible from Docker containers and the external network.
Let's see how to do this using the Firewall extension for Plesk that we installed earlier.
We pass to Tools & Settings > Settings > Firewall:
We pass to Modify Plesk Firewall Rules > Add Custom Rule and open the following TCP ports for the Docker subnet (172.0.0.0 / 8):
RabbitMQ: 1883, 4369, 5671-5672, 25672, 61613-61614
Redis: 32785, 32786
We will also add a rule that will open PostgreSQL ports and RabbitMQ management panels to the outside world:
Apply the rules using the Apply Changes button:
Setting up CI/CD in Github Actions
Let's get down to the most interesting part - setting up a continuous integration pipeline and delivering our project to the server.
This pipeline will consist of two parts:
building an image and running tests (for the backend) - on the Github side;
running migrations (for the backend) and deploying containers - on the server.
Deploy to Plesk
Let's deal with the second point first (because the first one depends on it).
We will configure the deployment process using the Git extension for Plesk.
Consider an example with a Prod environment for a Backend repository.
We go to the subscription of our Helloworld website and go to the Git subsection:
Insert a link to our Github repository into the "Remote Git repository" field and change the default folder httpdocs to another (eg. /httpdocs/hw_back):
Copy the SSH Public key from the previous step and add its in Github settings.
Click OK on the screen in step 2, after which we will be redirected to the repository page in Plesk. Now we need to configure the repository to be updated on commits to the master branch. To do this, go to Repository Settings and save the value Webhook URL (we will need it later when setting up Github Actions):
In the Actions field on the screen from the previous paragraph, enter the script to launch the deployment:
cd {REPOSITORY_ABSOLUTE_PATH}
.ci-cd/deploy.sh {ENV} {DOCKER_REGISTRY_HOST} {DOCKER_USER} {DOCKER_PASSWORD} {TG_BOT_TOKEN} {TG_CHAT_ID}
where:
{REPOSITORY_ABSOLUTE_PATH} - path to the prod folder of the backend repository on the server; {ENV} - environment (dev / prod), in our case prod; {DOCKER_REGISTRY_HOST} - the host of our docker repository {TG_BOT_TOKEN} β Telegram bot token; {TG_CHAT_ID} β ID of the chat/channel for sending notifications.
Sample script:
cd /var/www/vhosts/helloworld.ru/httpdocs/hw_back/
.ci-cd/deploy.sh dev docker.helloworld.ru docker_user docker_password 12345678:AAbcdEfghCH1vGbCasdfSAs0K5PALDsaw -1001234567890
We add a user from our subscription to the Docker group (so that he can manage containers):
sudo usermod -aG docker helloworld_admin
The dev environment for the backend repository and the frontend are set up in the same way.
Deployment pipeline in Github Actions
Let's move on to setting up the first part of our CI/CD pipeline in Github Actions.
But before parsing it, let's fill in the Secret variables we need in Github. To do this, go to Settings -> Secrets:
DOCKER_REGISTRY - the host of our Docker repository (docker.helloworld.ru);
DOCKER_LOGIN - login to the Docker repository;
DOCKER_PASSWORD - password to it;
DEPLOY_HOST β host where the Plesk admin panel is available (example: helloworld.com: 8443 or 123.4.56.78:8443);
DEPLOY_BACK_PROD_TOKEN - a token for deployment to the prod-repository on the server (we got it in Deployment in Plesk p. 4);
DEPLOY_BACK_DEV_TOKEN - token for deployment to the dev repository on the server.
The deployment process is simple and consists of three main steps:
building and publishing the image in our repository;
running tests in a container based on a freshly built image;
deployment to the desired environment depending on the branch (dev/master).
Frontend
The deploy.yml file for the front repository little different from Beck's. It lacks a step with running tests and changes the names of tokens for deployment. Secrets for the front repository, by the way, need to be filled out separately.
Site setup
Proxying traffic through Nginx
Well, we've come to the end. It remains only to configure the proxying of incoming and outgoing traffic to our container through Nginx. We have already covered this process in step 5 of the Docker Registry setup. The same should be repeated for the back and front parts in dev and prod environments.
I will provide screenshots of the settings.
BACKEND
Frontend
IMPORTANT UPDATE. All URLs will be proxied to the frontend container, except those starting with /api/ - they will be proxied to the back container (so in the back container, all handlers must start with /api/).
Results
Now our site should be available at helloworld.ru and dev.helloworld.ru (prod- and dev-environments, respectively).
In total, we learned how to prepare a simple application in Flask and Angular and set up a pipeline in Github Actions to roll it out to a server running Plesk.
I will duplicate the links to the repositories with the code: backend, front end.