In Chelyabinsk, meetups of Sysadminka system administrators are taking place, and at the last of them I made a report on our solution for running applications on 1C-Bitrix in Kubernetes.
Bitrix, Kubernetes, Ceph - a great mix?
I’ll tell you how we put together a working solution from all this.
Let's go!
The mitap took place on April 18 in Chelyabinsk. You can read about our meetups in and look at .
If you want to come to us with a report or as a listener - welcome, write to [email protected] and in Telegram t.me/vadimisakanov.
My report
Solution "Bitrix in Kubernetes, version Southbridge 1.0"
I will talk about our solution in the “for dummies in Kubernetes” format, as it was done at the mitap. But I assume that you know the words Bitrix, Docker, Kubernetes, Ceph at least at the level of Wikipedia articles.
What is ready-made about Bitrix in Kubernetes?
There is very little information on the entire Internet about the operation of applications on Bitrix in Kubernetes.
I found only such materials:
Report by Alexander Serbul, 1C-Bitrix, and Anton Tuzlukov from Qsoft:
I recommend listening to it.
Developing your own solution from the user .
Found more .
Iiii… actually, everything.
I warn you, we did not check the quality of the solutions on the links above 🙂
By the way, when preparing our decision, I talked with Alexander Serbul, then his report was not yet available, so my slides contain the item “Bitrix does not use Kubernetes”.
But on the other hand, there are already a lot of ready-made Docker images for Bitrix to work in Docker:
Is this enough to create a complete solution for Bitrix in Kubernetes?
No. There are a large number of problems that need to be solved.
What are the problems with Bitrix in Kubernetes?
The first is that pre-built images from Dockerhub are not suitable for Kubernetes.
If we want to build a microservice architecture (and in Kubernetes we usually do), the Kubernetes application needs to be divided into containers and ensure that each container performs one small function (and does it well). Why only one? In short, the simpler, the more reliable.
If it's more authentic, check out this article and video, please:
Docker images in Dockerhub are mostly built on an all-in-one basis, so we still had to build our bike and even create images from scratch.
The second - the site code is edited from the admin panel
We created a new section on the site - the code was updated (a directory with the name of the new section was added).
We changed the properties of the component from the admin panel - the code has changed.
Kubernetes "by default" does not know how to work with this, containers must be immutable (Stateless).
Cause: Each container (pod) in the cluster handles only part of the traffic. If you change the code in only one container (pod), then the code will be different in different pods, the site will work differently, different versions of the site will be shown to different users. You can't live like that.
Third - you need to solve the issue with the deployment
If we have a monolith and one "classic" server, everything is quite simple: we deploy a new code base, migrate the database, switch traffic to the new version of the code. Switching happens instantly.
If we have a site in Kubernetes, cut into microservices, there are a lot of containers with code - oh. You need to collect containers with a new version of the code, roll them out instead of the old ones, perform the database migration correctly, and ideally do it invisibly to visitors. Fortunately, Kubernetes helps us with this, supporting a whole cloud of different types of deployment.
Fourth - you need to solve the issue of storing statics
If your site is “only” 10 gigabytes and you deploy it entirely in containers, you will end up with 10 gigabyte containers that will take forever to deploy.
It is necessary to store the most "heavy" parts of the site outside the containers, and the question arises how to do it correctly
What is not in our solution
The entire Bitrix code is not cut into microfunctions / microservices (so that registration is separate, the online store module is separate, etc.). We store the entire code base in each container entirely.
We also don’t store the base in Kubernetes (I still implemented solutions with the base in Kubernetes for developer environments, but not for production).
Site administrators will still notice that the site is running on Kubernetes. The "system check" function does not work correctly, to edit the site code from the admin panel, you must first click the "I want to edit the code" button.
We have identified the problems, we have decided on the need to implement microservices, the goal is clear - to get a working system for running applications on Bitrix in Kubernetes, while retaining both the capabilities of Bitrix and the advantages of Kubernetes. We start implementation.
Architecture
Many "worker" pods with a web server (workers).
One pod with cron-tasks (required only one).
One upgrade pod for editing the site code from the admin panel (also only one is required).
We solve questions:
- Where to store sessions?
- Where to store the cache?
- Where to store statics, not to place gigabytes of statics in a heap of containers?
- How will the database work?
docker image
We start by building a Docker image.
The ideal option is that we have one universal image, based on it we get both worker-pods, and pods with crontasks, and upgrade pods.
.
It includes nginx, apache/php-fpm (can be selected at build time), msmtp for sending mail, and cron.
When building the image, the full codebase of the site is copied to the /app directory (with the exception of those parts that we will move to a separate shared storage).
Microservices, services
worker pods:
- nginx container + apache/php-fpm + msmtp container
- msmtp could not be transferred to a separate microservice, Bitrix starts to resent that it cannot send mail directly
- Each container has a complete code base.
- Prohibition on changing code in containers.
cron under:
- container with apache, php, cron
- complete codebase included
- ban on changing code in containers
upgrade under:
- container with nginx + container apache/php-fpm + msmtp
- there is no prohibition on changing the code in containers
session storage
Bitrix cache storage
Another important thing: we store passwords for connecting to everything, from a database to mail, in kubernetes secrets. We get a bonus, passwords are visible only to those to whom we give access to secrets, and not to everyone who has access to the project's codebase.
Storage for statics
You can use anything: ceph, nfs (but nfs is not recommended for production), network storage from cloud providers, etc.
The storage will need to be connected in containers to the /upload/ directory of the site and other directories with statics.
Data Bank
For simplicity, we recommend moving the base outside of Kubernetes. The base in Kubernetes is a separate difficult task, it will make the scheme an order of magnitude more complicated.
Session storage
We use memcached 🙂
It handles session storage well, is clustered, natively supported as session.save_path in php. Such a system has been worked out many times in the classic monolithic architecture, when we built clusters with a large number of web servers. For deployment we use helm.
$ helm install stable/memcached --name sessionphp.ini - here in the image settings are set for storing sessions in memcached
We used Environment Variables to pass host data with memcached .
This allows you to use the same code in dev, stage, test, prod environments (memcached hostnames will be different in them, so we need to pass a unique hostname for sessions to each environment).
Bitrix cache storage
We need a fault-tolerant storage that all pods can write to and read from.
We also use memcached.
This solution is recommended by Bitrix itself.
$ helm install stable/memcached --name cachebitrix/.settings_extra.php - here in Bitrix it is set where we store the cache
We also use Environment Variables.
Krontasky
There are different approaches to running crontasks in Kubernetes.
- separate deployment with a pod for running crontasks
- cronjob to execute crontasks (if it is a web app - with wget , or kubectl exec inside one of the worker pods, etc.)
- etc.
You can argue about the most correct, but in this case we chose the option “separate deployment with pods for crontasks”
How it is done:
- add cron tasks via ConfigMap or via config/addcron file
- in one instance, we launch a container identical to the worker-pod + allow the execution of cron-tasks in it
- the same code base is used, thanks to the unification, the assembly of the container is simple
What good we get:
- we have working crontasks in an environment identical to the environment of the developers (docker)
- crontasks do not need to be "rewritten" for Kubernetes, they work in the same form and in the same codebase as before
- cron-tasks can be added by all team members with commit rights to the production branch, not just admins
Southbridge K8SDeploy module and code editing from admin panel
Did we talk about upgrade under?
And how to send traffic there?
Hooray, we wrote a php module for this 🙂 This is a small classic module for Bitrix. It is not yet in the public domain, but we plan to open it.
The module is installed as a regular module in Bitrix:
And it looks like this:
It allows you to set a cookie that identifies the site administrator and allows Kubernetes to send traffic to the upgrade pod.
When the changes are completed, you need to click git push, the code changes will be sent to git, then the system will build an image with a new version of the code and “roll” it across the cluster, replacing the old pods.
Yes, it’s a bit of a crutch, but at the same time we keep the microservice architecture and do not take away from Bitrix users their favorite opportunity to correct the code from the admin panel. In the end, this is an option, you can solve the task of editing the code in a different way.
Helm chart
To build applications in Kubernetes, we usually use the Helm package manager.
For our Bitrix solution in Kubernetes, Sergey Bondarev, our lead system administrator, wrote a special Helm chart.
It builds worker, ugrade, cron pods, sets up ingresses, services, passes variables from Kubernetes secrets to pods.
We store the code in Gitlab, and we also run the Helm build from Gitlab.
Briefly, it looks like this
$ helm upgrade --install project .helm --set image=registrygitlab.local/k8s/bitrix -f .helm/values.yaml --wait --timeout 300 --debug --tiller-namespace=productionHelm also allows you to do a "seamless" rollback, if suddenly something went wrong during the deployment. It’s nice when you’re not in a panic “fix the FTP code because the prod has crashed”, but Kubernetes does it automatically, and without downtime.
Deploy
Yes, we are Gitlab & Gitlab CI fans, we use it 🙂
When you commit in Gitlab to the project repository, Gitlab starts a pipeline that deploys a new version of the environment.
Steps:
- build (build a new Docker image)
- test (testing)
- clean up (delete the test environment)
- push (send it to the Docker registry)
- deploy (we deploy the application to Kubernetes via Helm).
Hurray, done, implement!
Well, or ask questions, if any.
So what have we done
From a technical point of view:
- dockerized Bitrix;
- "cut" Bitrix into containers, each of which performs a minimum of functions;
- have achieved a stateless state of containers;
- solved the problem with updating Bitrix in Kubernetes;
- all Bitrix functions continued to work (almost all);
- deployed to Kubernetes and rolled back between versions.
From a business point of view:
- fault tolerance;
- Kubernetes tools (easy integration with Gitlab CI, seamless deployment, etc);
- passwords in secrets (visible only to those who are directly granted access to passwords);
- it is convenient to create additional environments (for development, tests, etc.) within a single infrastructure.
Source: habr.com