In Chelyabinsk, Sysadminka system administrator meetups are taking place, and at the last one, I presented our solution for running applications on 1C-Bitrix in Kubernetes.
Bitrix, Kubernetes, Ceph — a great mix?
I'll explain how we assembled a working solution from all this.
Let's go!
The meetup took place on April 18 in Chelyabinsk. You can read about our meetups on and watch on .
If you want to come and present or attend as a listener — you are welcome, write to vadim.isakanov@gmail.com and on Telegram t.me/vadimisakanov.
My presentation
The solution "Bitrix in Kubernetes, version Southbridge 1.0"
I will talk about our solution in a "for beginners in Kubernetes" format, as it was done at the meetup. But I assume you are at least familiar with the terms Bitrix, Docker, Kubernetes, and Ceph at the level of Wikipedia articles.
What is available about Bitrix in Kubernetes?
There is very little information on the internet about running applications on Bitrix in Kubernetes.
I only found the following materials:
Presentation by Alexander Serbul, 1C-Bitrix, and Anton Tuzlukov from Qsoft:
I recommend listening to it.
Development of a custom solution from the user .
I also found .
And... that's all.
I warn you, we haven't verified the quality of the solutions linked above 🙂
By the way, during the preparation of our solution, I spoke with Alexander Serbul, and at that time his presentation was not available yet, so my slides include a point saying "Bitrix does not use Kubernetes."
But there are already many ready-made Docker images for running Bitrix in Docker:
Is this enough to create a full-fledged solution for Bitrix in Kubernetes?
No. There are many problems that need to be solved.
What are the problems with Bitrix in Kubernetes?
The first — ready-made images from Dockerhub are not suitable for Kubernetes.
If we want to build a microservices architecture (and usually we do in Kubernetes), the application in Kubernetes needs to be split into containers, and we must ensure that each container performs one small function (and does it well). Why only one? In short — the simpler, the more reliable.
If you want more details — please check this article and video:
Docker images in Dockerhub are mainly built on the principle of "everything in one," so we had to create our bicycle and even build images from scratch.
The second — the site code is adjusted from the admin panel.
A new section has been created on the site — the code has been updated (a directory with the name of the new section has been added).
The properties of the component were changed from the admin panel — the code has been modified.
Kubernetes does not support this by default; containers must be immutable (Stateless).
The reason: each container (pod) in the cluster processes only part of the traffic. If the code is changed in just one container (pod), different pods will have different code, the site will function inconsistently, and different users will see different versions of the site. This is not sustainable.
Third — we need to address the deployment issue.
If we have a monolith and a single 'classic' server, it's straightforward: we deploy a new codebase, perform a database migration, and switch the traffic to the new version of the code. The switch happens instantaneously.
If our site is on Kubernetes, split into microservices, and has many containers with code — then it gets complicated. We need to build containers with the new version of the code, deploy them instead of the old ones, perform the database migration correctly, and ideally do this unnoticed by visitors. Fortunately, Kubernetes helps us with this, supporting a whole array of different types of deployment.
Fourth — we need to address the static storage issue.
If your site weighs 'only' 10 gigabytes and you deploy it entirely in containers, you will end up with containers weighing 10 gigabytes that take an eternity to deploy.
We need to store the 'heaviest' parts of the site outside of containers, and the question arises of how to do this properly.
What is not in our solution.
The entire Bitrix code is not broken down into microservices/microfunctions (so that registration is separate, the online store module is separate, etc.). We store the entire codebase in each container.
We also do not store the database in Kubernetes (I have implemented solutions with a database in Kubernetes for development environments, but not for production).
Site administrators will still notice that the site operates in Kubernetes. The 'system check' function does not work correctly, and to edit the site's code from the admin panel, you first have to click the 'I want to edit the code' button.
We have resolved the issues, determined the necessity of implementing microservices, and our objective is clear — to create a functioning system for running Bitrix applications on Kubernetes while preserving both Bitrix capabilities and Kubernetes advantages. We are beginning the implementation.
Architecture
Several 'worker' pods with a web server.
One pod with cron jobs (definitely only one).
One upgrade pod for editing the site code from the admin panel (again, definitely only one).
We are addressing the questions:
- Where to store sessions?
- Where to store cache?
- Where to store static files? We can't place gigabytes of static files in a bunch of containers.
- How will the database work?
Docker image
We begin with building the Docker image.
The ideal option is to have one universal image from which we derive both worker pods, pods with cron jobs, and upgrade pods.
.
It includes nginx, apache/php-fpm (you can choose during the build), msmtp for sending mail, and cron.
During the image build, the complete code base of the site is copied into the /app directory (with the exception of those parts that we will place in a separate shared storage).
Microservices, services
worker pods:
- Container with nginx + container apache/php-fpm + msmtp
- We couldn't separate msmtp into a distinct microservice; Bitrix starts to complain that it cannot send mail directly.
- Each container has the full code base.
- No changes to the code are allowed in the containers.
cron pod:
- container with apache, php, cron
- includes the complete code base
- no changes to the code are allowed in the containers
upgrade pod:
- container with nginx + container apache/php-fpm + msmtp
- there are no restrictions on changing the code in the containers
session storage
Bitrix cache storage
It's also important: the passwords for accessing everything from the database to email are stored in Kubernetes secrets. We get the bonus that the passwords are visible only to those we grant access to the secrets, not to everyone who has access to the project’s code base.
Static file storage
You can use anything: ceph, nfs (but we don't recommend nfs for production), network storage from 'cloud' providers, etc.
The storage will need to be connected in the containers to the /upload/ directory of the site and other directories with static files.
Database
For simplicity, we recommend placing the database outside Kubernetes. The database in Kubernetes is a complex challenge that will make the architecture significantly more complicated.
Session storage
We use memcached 🙂
It handles session storage well, clusters, and is natively supported as session.save_path in PHP. This system has been proven many times in the classic monolithic architecture, when we built clusters with a large number of web servers. For deployment, we use helm.
$ helm install stable/memcached --name sessionphp.ini — this image specifies configurations for storing sessions in memcached
We used Environment Variables to pass information about the hosts with memcached .
This allows the same code to be used in dev, stage, test, and prod environments (the host names for memcached will differ in each, so we need to pass unique host names for sessions to each environment).
Bitrix cache storage
We need a resilient storage where all pods can write and read from.
We also use memcached.
This solution is recommended by Bitrix itself.
$ helm install stable/memcached --name cachebitrix/.settings_extra.php — this is where Bitrix specifies where our cache is stored
We also use Environment Variables.
Cron jobs
There are different approaches to executing cron jobs in Kubernetes.
- a separate deployment with a pod for executing cron jobs
- cron job for executing cron jobs (if it's a web app — with wget , or kubectl exec into one of the worker pods, etc.)
- etc.
One can argue about the most correct method, but in this case we chose the option of a separate deployment with pods for cron jobs
How it's done:
- we add cron jobs through ConfigMap or via the config/addcron file
- we run a single instance of a container identical to the worker pod + allow cron jobs to run in it
- the same codebase is used, enabling simple container builds due to standardization
What are the positives:
- we have working cron jobs in an environment identical to that of developers (docker)
- cron jobs do not need to be 'rewritten' for Kubernetes, they run as is in the same codebase as before
- all team members with commit rights to the production branch can add cron jobs, not just admins
The Southbridge K8SDeploy module and code editing from the admin panel
Did we mention the pod upgrade?
How do we direct traffic there?
Hooray, we wrote a module for this in PHP 🙂 This is a small classic module for Bitrix. It is not yet publicly available, but we plan to release it.
The module is installed like a regular module in Bitrix:
And it looks like this:
It allows you to set a cookie that identifies the website administrator and enables Kubernetes to route traffic to the upgrade pod.
Once the changes are complete, you need to hit git push; the code changes will be sent to git, after which the system will build an image with the new version of the code and 'roll it out' across the cluster, replacing the old pods.
Yes, it's somewhat of a workaround, but this way we maintain the microservice architecture without taking away the favorite ability for Bitrix users to edit code from the admin panel. After all, it's an option; the task of editing code can be approached differently.
Helm chart
For building applications in Kubernetes, we typically use the Helm package manager.
For our Bitrix solution in Kubernetes, Sergey Bondarev, our lead system administrator, wrote a special Helm chart.
It builds worker, upgrade, cron pods, configures ingress, services, and transfers variables from Kubernetes secrets to the pods.
We store the code in Gitlab, and we also trigger the Helm build from Gitlab.
In brief, it looks like this
$ helm upgrade --install project .helm --set image=registrygitlab.local/k8s/bitrix -f .helm/values.yaml --wait --timeout 300 --debug --tiller-namespace=productionHelm also allows for a 'seamless' rollback if something goes wrong during deployment. It's nice when you are not panicking, 'fixing the code via FTP because production is down', but Kubernetes does it automatically without any downtime.
Deploy
Yes, we are fans of Gitlab & Gitlab CI, we use it 🙂
When you commit to Gitlab in the project repository, Gitlab triggers a pipeline that deploys the new version of the environment.
Stages:
- build (creating a new Docker image)
- test (testing)
- clean up (removing the test environment)
- push (sending it to the Docker registry)
- deploy (deploying the application in Kubernetes via Helm).
Hooray, it's ready, let's implement it!
Or we can ask questions if there are any.
So, what did we do
From a technical perspective:
- we dockerized Bitrix;
- we 'sliced' Bitrix into containers, each performing minimal functions;
- we achieved a stateless state for the containers;
- we resolved the issue of updating Bitrix in Kubernetes;
- all Bitrix functions continued to work (almost all);
- we perfected deployment in Kubernetes and rollback between versions.
From a business perspective:
- fault tolerance;
- Kubernetes tools (easy integration with Gitlab CI, seamless deployment, etc);
- passwords in secrets (visible only to those who have direct access to the passwords);
- It is convenient to create additional environments (for development, testing, etc.) within a unified infrastructure.
Source: habr.com
