Kubernetes is a great tool for running Docker containers in a clustered production environment. However, there are tasks that Kubernetes cannot solve. With frequent production deployments, we need a fully automated Blue/Green deployment to avoid downtime in this process, which also needs to process external HTTP requests and perform SSL offloading. This requires integration with a load balancer such as ha-proxy. Another challenge is semi-automated scaling of the Kubernetes cluster itself when running in the cloud, such as partially downscaling the cluster at night.
While Kubernetes doesn't provide these features out of the box, it does provide an API that can be used to accomplish these tasks. Tools for automated Blue/Green deployment and scaling of a Kubernetes cluster were developed as part of the Cloud RTI project, which was created on the basis of open-source.
This article, video transcript, explains how to set up Kubernetes along with other open source components to get a production-ready environment that accepts code from a git commit with no downtime in production.
So, after you have gained access to your applications from the outside world, you can start fully setting up automation, that is, bringing it to the stage where you can perform a git commit and make sure that this git commit ends in production. Naturally, when implementing these steps, when implementing a deployment, we do not want to face downtime. So, any automation in Kubernetes starts with an API.
Kubernetes is not a tool that can be used productively out of the box. Of course, you can do this, use kubectl and so on, but still the API is the most interesting and useful thing about this platform. By using the API as a set of functions, you can access almost anything you want to do in Kubernetes. kubectl itself also uses a REST API.
It's REST, so you can use any languages and tools to work with this API, but custom libraries will make your life much easier. My team wrote 2 such libraries: one for Java/OSGi and one for Go. The second is not often used, but in any case, these useful things are at your disposal. They are a partially licensed open-source project. There are many such libraries for various languages, so you can choose the most suitable ones.
So, before you start automating deployment, you need to make sure that this process will not be subject to any downtime. For example, our team does production deployments in the middle of the day when people are making the most of their apps, so it's important to avoid delays in the process. In order to avoid downtime, 2 methods are used: blue / green deployment or rolling update. In the latter case, if you have 5 application replicas running, they are upgraded sequentially one after the other. This method works great, but it won't work if you have different versions of the application running at the same time during the deployment process. In such a case, you can update the user interface while the backend is working with the old version and the application will be terminated. Therefore, from the point of view of programming, work in such conditions is rather difficult.
This is one of the reasons why we prefer to use blue/green deployment to automate the deployment of our applications. With this method, you must ensure that only one version of the application is active at any given time.
The blue/green deployment mechanism is as follows. We receive traffic for our applications through ha-proxy, which directs it to running application replicas of the same version.
When a new deployment is made, we use a Deployer that is provided with the new components and deploys the new version. Deploying a new version of the application means that a new set of replicas is “lifted up”, after which these replicas of the new version are launched in a separate, new pod. However, ha-proxy doesn't know anything about them and doesn't send any workload to them yet.
Therefore, first of all, it is necessary to perform a health check of new versions of health checking to make sure that the replicas are ready to serve the load.
All deployment components must support some form of health check. This can be a very simple HTTP call check when you get a 200 status code, or a deeper check where you check the connection of the replicas to the database and other services, the stability of the dynamic environment links, whether everything starts and works properly. This process can be quite complicated.
After the system makes sure that all updated replicas are working, Deployer will update the configuration and pass the correct confd, which will reconfigure ha-proxy.
Only after that, the traffic will be directed to the pod with replicas of the new version, and the old pod will disappear.
This mechanism is not a feature of Kubernetes. The blue/green deployment concept has been around for quite some time and has always used a load balancer. First, you direct all traffic to the old version of the application, and after the update, you completely transfer it to the new version. This principle is used not only in Kubernetes.
Now I will introduce you to a new deployment component - Deployer, which performs health checks, reconfigures proxies, and so on. This is a concept that does not belong to the outside world and exists inside Kubernetes. I'll show you how you can create your own Deployer concept using open-source tools.
So the first thing Deployer does is create an RC replication controller using the Kubernetes API. This API creates pods and services for further deployment, that is, it creates a completely new cluster for our applications. As soon as the RC is convinced that the replicas have started, it will perform a Health check on them. To do this, Deployer uses the GET /health command. It launches the appropriate test components and checks all the elements that ensure the operation of the cluster.
After all pods have reported their "health", Deployer creates a new configuration element - etcd distributed storage, which is used internally by Kubernetes, including for storing load balancer configuration. We write data to etcd, and a little confd tool monitors etcd for new data.
If it detects any changes to the original configuration, it generates a new settings file and passes it to ha-proxy. In this case, ha-proxy restarts without losing any connections and directs the load to the new services that provide the new version of our applications.
As you can see, despite the abundance of components, there is nothing complicated here. You just need to pay more attention to the API and etcd. I want to tell you about the open-source deployer that we use ourselves - this is Amdatu Kubernetes Deployer.
It is a tool for orchestrating Kubernetes deployments with the following features:
- Blue/Green deployment;
- setting up an external load balancer;
- managing deployment descriptors;
- managing the actual deployment;
- health checks health checks during deployment;
- injecting environment variables into pods.
This Deployer is built on top of the Kubernetes API and provides a REST API for managing handles and deployments, as well as a Websocket API for stream logging during deployment.
It puts the load balancer configuration data in etcd, so you don't have to use ha-proxy with out-of-the-box support, but can easily use your own load balancer configuration file. Amdatu Deployer is written in Go, like Kubernetes itself, and licensed by Apache.
Before I started using this version of the deployer, I used the following deployment descriptor, which specifies the options I need.
One of the important parameters of this code is the inclusion of the "useHealthCheck" flag. We need to specify that a health check should be performed during the deployment process. This setting can be disabled when the deployment uses third-party containers that do not need to be validated. This descriptor also specifies the number of replicas and the frontend URL that ha-proxy needs. At the end is a "podspec" pod specification flag that calls Kubernetes for information on port settings, image, etc. This is a fairly simple JSON descriptor.
Another tool that is part of the Amdatu open-source project is Deploymentctl. It has a UI for configuring the deployment, stores the history of the deployment, and contains webhooks for callbacks from third-party users and developers. You may not need to use the UI since Amdatu Deployer itself is a REST API, but this interface can make it much easier for you to deploy without involving any API. Deploymentctl is written in OSGi/Vertx using Angular 2.
Now I will demonstrate the above on the screen using a pre-recorded recording so you don't have to wait. We will be deploying a simple Go application. Don't worry if you haven't used Go before, it's a very simple application so you should be able to understand everything.
Here we are creating an HTTP server that only responds to /health, so this application only checks the health check and nothing else. If the validation passes, the JSON structure shown below is used. It contains the version of the application that will be deployed by the deployer, the message that you see at the top of the file, and the boolean data type whether our application is operational or not.
With the last line, I cheated a bit, because I put a fixed boolean value at the top of the file, which will later help me deploy even an “unhealthy” application. We'll deal with this later.
So let's get started. First, we check for any running pods with the command ~ kubectl get pods , and by the absence of a frontend URL response, we verify that no deployments are currently in progress.
Next on the screen you see the Deploymentctl interface I mentioned, which sets the deployment parameters: namespace, application name, deployment version, number of replicas, frontend URL, container name, image, resource limits, port number for health check, etc. . Resource limits are very important, as they allow you to use the maximum possible amount of hardware. You can also view the Deployment log here.
If you now run the ~ kubectl get pods command, you can see that the system “freezes” for 20 seconds, during which the ha-proxy is reconfigured. After that, the pod starts up, and our replica can be seen in the deployment log.
I cut out the 20 second wait from the video, and now you can see on the screen that the first version of the application has been deployed. All this was done only with the help of the UI.
Now let's try the second version. To do this, I change the application's message from "Hello, Kubernetes!" to “Hello, Deployer!”, the system creates this image and places it in the Docker registry, after which we simply click on the “Deploy” button again in the Deploymentctl window. This automatically starts the deployment log in the same way as it did when deploying the first version of the application.
The ~ kubectl get pods command shows that 2 versions of the application are currently running, but the frontend shows that we still have version 1 running.
The load balancer waits for the health check to be performed before redirecting traffic to the new version. After 20 seconds, we switch to curl and see that now we have version 2 of the application deployed, and the first one has been removed.
It was the deployment of a "healthy" - healthy - application. Let's see what happens if I change the value of the Healthy parameter from true to false for a new version of the application, i.e. I try to deploy an unhealthy application that failed the health check. This can happen if some configuration errors were made in the application at the development stage, and it went to production in this form.
As you can see, the deployment goes through all the above steps, and ~ kubectl get pods shows that both pods are running. But unlike the previous deployment, the log shows the timeout status. That is, because the health check failed, the new version of the application cannot be deployed. As a result, you see that the system reverted to using the old version of the application, and the new version was simply deleted.
The good thing about this is that even if you have a huge amount of concurrent requests coming into your application, they won't even notice any downtime during the implementation of the deployment procedure. If you test this application with the Gatling framework, which sends it the maximum possible number of requests, then none of these requests will be dropped. This means our users won't even notice real-time version updates. If it fails, work will continue on the old version, if successful, users will switch to the new version.
There is only one thing that can lead to failure - if the health check succeeds and the application fails as soon as the workload arrives on it, that is, the collapse will occur only after the deployment is completed. In this case, you will have to manually roll back to the old version. So, we looked at how to use Kubernetes with open-source tools designed for it. The deployment process will be much easier if you embed these tools in your Build/Deploy pipelines. At the same time, you can use both the user interface to start the deployment, and completely automate this process, using, for example, commit to master.
Our Build Server will create a Docker image, insert it into Docker Hub or whatever registry you use. Docker Hub supports webhook, so we can run remote deployment via Deployer as shown above. In this way, you can fully automate the deployment of the application to potential production.
Let's move on to the next topic - scaling a Kubernetes cluster. Note that the kubectl command is a scale command. With more help, you can easily increase the number of replicas in our existing cluster. However, in practice, we usually want to increase the number of nodes, not pods.
At the same time, during working hours, you may need to increase, and at night, to reduce the cost of Amazon services, you may need to reduce the number of running instances of the application. This does not mean that it will be sufficient to scale only the number of pods, because even if one of the nodes is idle, you still have to pay Amazon for it. That is, along with scaling pods, you also need to scale the number of machines used.
This can be tricky because whether we use Amazon or another cloud service, Kubernetes knows nothing about the number of machines used. It lacks a tool that allows you to scale the system at the node level.
So we have to take care of both nodes and pods. We can easily scale new node launches using the AWS API and Scaling group machines to adjust the number of Kubernetes worker nodes. You can also use cloud-init or a similar script to register nodes in a Kubernetes cluster.
The new machine starts in the Scaling group, initiates itself as a node, registers itself in the master's registry and starts working. After that, you can increase the number of replicas to use on the resulting nodes. Downscaling requires more effort to make sure that such a step does not lead to the destruction of already running applications after shutting down "unnecessary" machines. To prevent such a scenario, you need to bring the nodes to the "unschedulable" status. This means that the default scheduler will ignore these nodes when scheduling DaemonSet pods. The scheduler won't delete anything from these servers, but it won't start any new containers there either. The next step is to evict the drain node, i.e. transfer the running pods from it to another machine, or other nodes that have sufficient capacity for this. After making sure that there are no more containers on these nodes, they can be removed from Kubernetes. After that, for Kubernetes, they simply cease to exist. Next, you need to use the AWS API to disable unnecessary nodes, or machines.
You can use Amdatu Scalerd, another open-source scaling tool similar to the AWS API. It provides a CLI to add or remove nodes in a cluster. Its interesting feature is the ability to customize the scheduler using the following json file.
The code shown halves the capacity of the cluster during the night. It is configured with both the number of replicas available and the desired Amazon cluster capacity. Using this scheduler will automatically reduce the number of nodes at night and increase them in the morning, saving you the cost of using nodes in a cloud service like Amazon. This feature is not built into Kubernetes, but using Scalerd will allow you to scale the platform however you want.
I want to draw your attention to the fact that many people say to me: "All this is good, but what about my database, which is usually in a static state?" How can you run something like this in a dynamic environment like Kubernetes? In my opinion, you shouldn't be doing this, you shouldn't be trying to run a data store on Kubernetes. It is technically possible and there are guides on the internet about this, but it will seriously complicate your life.
Yes, Kubernetes has the concept of persistent stores, and you can try to run data stores like Mongo or MySQL, but that's a lot of work. This is due to the fact that data stores do not fully support interaction with a dynamic environment. Most databases require a lot of tuning, including manually configuring the cluster, don't like autoscaling, and stuff like that.
Therefore, you should not complicate your life by trying to run a data warehouse in Kubernetes. Organize them in the traditional way using familiar services and just let Kubernetes use them.
At the end of the topic, I want to introduce you to the Cloud RTI platform based on Kubernetes, which my team is working on. It provides centralized logging, application and cluster monitoring, and many other useful features that will come in handy. It uses various open-source tools such as Grafana to display monitoring.
There was a question, why use a ha-proxy load balancer with Kubernetes. Good question, because there are currently 2 levels of load balancing. Kubernetes services are still on virtual IP addresses. You can't use them for external host machine ports, because if Amazon reboots their cloud host, the address will change. That's why we put ha-proxy in front of the services - to create a more static structure for seamless interaction of traffic with Kubernetes.
Another good question is how can you take care of changing the database schema in a blue/green deployment? The fact is that regardless of the use of Kubernetes, changing the database schema is a difficult task. You need to ensure that the old and new schemas are compatible, after which you can update the database and then update the applications themselves. You can hot swap databases and then upgrade applications. I know people who have booted a brand new database cluster with a new schema, this is an option if you have a schemeless database like Mongo, but it's not an easy task anyway. If there are no more questions, thank you for your attention!
Some ads 🙂
Thank you for staying with us. Do you like our articles? Want to see more interesting content? Support us by placing an order or recommending to friends, , a unique analogue of entry-level servers, which was invented by us for you: (available with RAID1 and RAID10, up to 24 cores and up to 40GB DDR4).
Dell R730xd 2 times cheaper in Equinix Tier IV data center in Amsterdam? Only here in the Netherlands! Dell R420 - 2x E5-2430 2.2Ghz 6C 128GB DDR3 2x960GB SSD 1Gbps 100TB - from $99! Read about
Source: habr.com