Operator in Kubernetes for managing database clusters. Vladislav Klimenko (Altinity, 2019)

The report is devoted to the practical issues of developing an operator in Kubernetes, designing its architecture and basic principles of operation.

In the first part of the report, we will consider:

• what is an operator in Kubernetes and why is it needed;
• how exactly the operator simplifies the management of complex systems;
• what the operator can and what the operator cannot.

Next, we turn to a discussion of the internal structure of the operator. Consider the architecture and operation of the operator step by step. Let's analyze in detail:

• interaction between the operator and Kubernetes;
• what functions the operator takes on and what delegates to Kubernetes.

Consider managing shards and database replicas in Kubernetes.
Next, we will discuss data storage issues:

• how to work with Persistent Storage from an operator's point of view;
• pitfalls of using Local Storage.

In the final part of the report, we will consider practical examples of the application clickhouse operator with Amazon or Google Cloud Service. The report is based on the example of the development and operating experience of the operator for ClickHouse.

Video:

My name is Vladislav Klimenko. Today I wanted to talk about our experience in developing and operating an operator, and this is a specialized operator for managing database clusters. For example ClickHouse-operator to manage the ClickHouse cluster.

Why do we have the opportunity to talk about the operator and ClickHouse?

• We support and develop ClickHouse.
• At the moment, we are trying to slowly make our contribution to the development of ClickHouse. And we are the second after Yandex in terms of the amount of changes made in ClickHouse.
• We are trying to make additional projects for the ClickHouse ecosystem.

I would like to talk about one of these projects. This is about the ClickHouse-operator for Kubernetes.

In my report, I would like to touch on two topics:

• The first topic is how our ClickHouse database operator works in Kubernetes.
• The second topic is how any operator works, i.e. how it interacts with Kubernetes.

However, these two questions will intersect throughout my report.

Who would be interested in hearing what I'm trying to say?

• The most interesting will be those who exploit the operators.
• Or for those who want to make their own in order to understand how it works inside, how the operator interacts with Kubernetes, and what pitfalls may appear.

In order to best understand what we are going to discuss today, it would be nice to know how Kubernetes works and have a basic background in cloud computing.

What is ClickHouse? This is a column database with specifics in online processing of analytical queries. And it's completely open source.

And we only need to know two things. You need to know that this is a database, so what I will tell you will be applicable to almost any database. And the fact that the ClickHouse DBMS scales very well gives the scalability almost linear. And therefore, the state of the cluster is a natural state for ClickHouse. And we are most interested in discussing how to serve a ClickHouse cluster in Kubernetes.

Why is he needed there? Why can't we continue to operate it ourselves? And the answers are partly technical and partly organizational.

• In practice, more and more often we encounter such a situation when in large companies almost all components are already in Kubernetes. Remain databases outside.
• And more and more often the question is asked: “Can it be placed inside?”. Therefore, large companies are trying to produce the maximum unification of management in order to quickly be able to manage their data warehouses.
• And this especially helps if you need the maximum opportunity to repeat the same thing in a new place, that is, maximum portability.

How easy or difficult is it? This, of course, can be done by hand. But this is not so easy, because we add up the complexity of managing Kubernetes itself, but at the same time the specifics of ClickHouse are imposed. And it turns out such an aggregation.

And all together, this gives a fairly large set of technologies, which is already becoming quite difficult to manage, because Kubernetes brings its everyday issues to operation, and ClickHouse brings its issues to everyday operation. Especially if we have several ClickHouses, and we need to constantly do something with them.

ClickHouse with a dynamic configuration has a fairly large number of issues that create a constant load on DevOps:

• When we want to change something in ClickHouse, for example, add a replica, a shard, then we need to manage the configuration.
• Then change the data scheme, because ClickHouse has a specific sharding method. There it is necessary to lay out the data scheme, lay out configurations.
• You need to set up monitoring.
• Collection of logs for new shards, for new replicas.
• Take care of recovery.
• And restart.

These are such routine works that I would very much like to facilitate in operation.

Kubernetes itself helps a lot in operation, but on basic system things.

Kubernetes is great at facilitating and automating things like:

• Recovery.
• Restart.
• Storage management.

That's good, that's the right direction, but he's completely out of touch with how to operate a database cluster.

I want more, I want the entire database to work for us in Kubernetes.

I would like to get something like a big one magic red button that you press and you have a cluster deployed and maintained throughout the entire life cycle with everyday tasks that need to be solved. ClickHouse cluster in Kubernetes.

And we tried to make a solution that would help facilitate the work. This is the ClickHouse-operator for Kubernetes from Altinity.

An operator is a program whose main task is to manage other programs, that is, it is a manager.

And it contains patterns of behavior. You can call it codified knowledge about the subject area.

And his main task is to make life easier for DevOps and reduce micromanagement so that he (DevOps) already thinks in high-level terms, that is, so that he (DevOps) does not micromanage, so that he does not manually configure all the details.

And just the operator is a robot assistant who struggles with microtasks and helps DevOps.

Why is an operator needed? He excels in two areas:

• When a ClickHouse specialist does not have enough experience, but it is already necessary to operate ClickHouse, the operator facilitates operation and allows you to operate a ClickHouse cluster with a rather complex configuration, while not going into too much detail about how it all works inside. You just give him high-level tasks, and it works.
• And the second task in which it shows itself most well is when it is necessary to automate a large number of typical tasks. Removes microtasks from sysadmins.

This is most needed either by those who are just starting their journey, or by those who need to do a lot of automation.

What is the difference between the operator-based approach and other systems? There is also Helm. It also helps to install ClickHouse, you can draw helm charts, which will even install a whole ClickHouse cluster. What then is the difference between the operator and from the same, for example, Helm?

The main fundamental difference is that Helm is all about package management, and the operator goes a step further. This is the support of the entire life cycle. This is not only installation, these are everyday tasks that include scaling, sharding, i.e. everything that needs to be done during the life cycle (if necessary, removal too) - this is all decided by the operator. It tries to automate and serve the entire software life cycle. This is its fundamental difference from other solutions that are presented.

That was the introductory part, let's move on.

How do we build our operator? We are trying to approach the issue in order to manage the ClickHouse cluster as a single resource.

Here we have the input data on the left side of the picture. This is YAML with a cluster specification, which is classically passed through kubectl to Kubernetes. There, our operator picks it up, does his magic. And as a result, we get such a scheme. This is the implementation of ClickHouse in Kubernetes.

And then we will slowly look at how the operator works, what typical tasks can be solved. We will consider only typical tasks, because we have limited time. And it will not be told about everything that the operator can decide.

Let's start from practice. Our project is completely open source, so you can see how it works on GitHub. And you can proceed from the considerations, if you just want to start, then you can start with the Quick Start Guide.

If you want to understand in detail, then we try to maintain the documentation in a more or less decent form.

Let's start with a practical problem. The first task we all want to start with is to run the first example somehow. How to launch ClickHouse with the help of an operator, without even knowing how it works? We are writing a manifesto, because all communication with k8s is communication through manifests.

Here is such a complex manifesto. What we have highlighted in red is what we need to focus on. We ask the operator to create a cluster named demo.

For now, these are basic examples. Storage has not yet been described, but we will return to storage a little later. For now, we will observe the development of the cluster in dynamics.

We have created this manifesto. We feed it to our operator. He worked, he did magic.

We look at the console. Three components are of interest - these are Pod, two Service-a, StatefulSet.

The operator has worked, and we can see what exactly he created.

He creates something like this. We have a StatefulSet, Pod, ConfigMap for each replica, ConfigMap for the entire cluster. Necessarily services as entry points to the cluster.

Services are the central Load Balancer Service and it is possible for each replica, for each shard.

Here is our base cluster looks something like this. He is from a single node.

Let's go further, we will complicate. You need to shard the cluster.

Our tasks are growing, dynamics are beginning. We want to add a shard. We follow the development. We change our specification. We indicate that we want two shards.

This is the same file that we dynamically develop with the growth of the system. There is no storage, storage will be discussed further, this is a separate issue.

We feed the YAML operator and see what happens.

The operator thought and made the following entities. We already have two Pods, three Services and, suddenly, 2 StatefulSets. Why 2 StatefulSets?

It was like this on the diagram - this is our initial state, when we had one pod.

It became like this. So far, everything is simple, it has been duplicated.

And why did the StatefulSet become two? Here we need to digress and discuss the question of how Pods are managed in Kubernetes.

There is such an object called StatefulSet, which allows you to make a set of Pods from a template. The key factor here is Template. And you can run many Pods in one StatefulSet according to one template. And the key phrase here is “one template many Pods”.

And there was a great temptation to make the entire cluster, packing it into one StatefulSet. It will work, there is no problem in it. But there is one caveat. If we want to assemble a heterogeneous cluster, i.e. from several versions of ClickHouse, then our questions begin. Yes, StatefulSet can do a rolling update, but there you can roll out a new version, explain that you need to try no more than so many nodes at the same time.

But if we extrapolate the task and say that we want to make a completely heterogeneous cluster and do not want to change from the old version to a new one using a rolling update, but simply want to create a heterogeneous cluster both in terms of different versions of ClickHouse and in terms of different storage. We want, for example, to make some replicas on separate disks, on slow ones, in general, to completely build a heterogeneous cluster. And due to the fact that StatefulSet makes a standardized solution out of one template, so there is no way to do this.

After some thought, it was decided that we do like this. We have each replica in its own StatefulSet. There are some drawbacks to this solution, but in practice it's all completely encapsulates the operator. And there are a lot of benefits. We can build a completely such cluster as we want, for example, an absolutely heterogeneous one. Therefore, in a cluster in which we have two shards with one replica, we will have 2 StatefulSets and 2 Pods precisely because we chose this approach due to the above reasons for the ability to build a heterogeneous cluster.

Let's return to practical tasks. In our cluster, we need to configure users, i.e. you need to do some configuration of ClickHouse in Kubernetes. The operator provides all the possibilities for this.

We can write what we want directly in YAML. All configuration options are directly mapped from this YAML into ClickHouse configs, which are then deployed throughout the cluster.

You can also write like this. This is for an example. The password can be encrypted. Absolutely all ClickHouse configuration options are supported. Here is just an example.

The cluster configuration is distributed as ConfigMap. In practice, the ConfigMap update does not happen instantly, so if there is a large cluster, then the process of pushing the configuration takes some time. But all this is very convenient to use.

We complicate the task. The cluster is developing. We want to replicate data. That is, we already have two shards, one replica each, users are configured. We are growing and want to replicate.

What do we need for replication?

We need ZooKeeper. In ClickHouse, replication is built using ZooKeeper. ZooKeeper is needed so that different ClickHouse replicas have a consensus on which data blocks are on which ClickHouse.

ZooKeeper can be used by anyone. If an enterprise has an external ZooKeeper, then it can be used. If not, then you can install from our repository. There is an installer that makes this whole thing easier.

And the scheme of interaction of the whole system turns out like this. We have Kubernetes as a platform. It executes the ClickHouse statement. ZooKeeper I pictured here. And the operator interacts with both ClickHouse and ZooKeeper. That is, an interaction is obtained.

And all this is necessary for ClickHouse to successfully replicate data to k8s.

Let's now look at the task itself, how the manifest for replication will look like.

We add two sections to our manifest. The first is where to get ZooKeeper, which can be either inside Kubernetes or external. This is just a description. And we order replicas. Those. we want two replicas. In total, we should have 4 pods at the output. We remember about storage, it will return a little further. Storage is a separate song.

It was like this.

It becomes like this. Replicas are added. The 4th did not fit, we believe that there may be many of them. And ZooKeeper is added on the side. The patterns are getting more complex.

And it's time to add the next task. We will add Persistent Storage.

For Persistent Storage, we have a variety of options.

If we are running in a cloud provider, for example, using Amazon, Google, then there is a great temptation to use cloud storage. It's very convenient, it's good.

And there is a second option. This is for local storage, when we have local disks on each node. This option is much more difficult to implement, but at the same time it is more productive.

Let's see what we have regarding cloud storage.

There are merits. It's very easy to configure. We simply order from a cloud provider that please give us storage of such and such a capacity, such and such a class. Classes are painted by providers independently.

And there is a drawback. For some, this is an uncritical shortcoming. Of course, there will be some performance overlays. It is very convenient to use, reliable, but there are some potential drawdowns in performance.

And since ClickHouse focuses on performance, you can even say that it squeezes out everything that is possible, so many clients try to squeeze out the maximum performance.

And to get the most out of it, we need local storage.

Kubernetes provides three abstractions for using local storage in Kubernetes. This:

• EmptyDir
• HostPath.
• Location

Consider how they differ, how they are similar.

Firstly, in all three approaches, we have storage - these are local disks that are located on the same physical k8s node. But they have some differences.

Let's start with the simplest, i.e. emptyDir. What is it in practice? It is we who ask the containerization system (most often Docker) from our specification to provide us with access to a folder on a local disk.

In practice, the docker creates a temporary folder somewhere in its own paths, calls it a long hash. And provides an interface to access it.

How will it perform in terms of performance? This will run at the speed of the local disk, i.e. this is full access to your screw.

But this case has its drawback. Persistent in this case is rather dubious. At the first movement of the docker with containers, Persistent is lost. If Kubernetes wants to move this Pod to another disk for some reason, then the data will be lost.

This approach is good for tests, because it already shows normal speed, but this option is not suitable for something serious.

Therefore, there is a second approach. This is hostPath. If you look at the previous slide and this one, you can see only one difference. Our folder left the docker directly to the Kubernetes node. It's a little faster here. We directly write the path on the local file system where we would like to store our data.

This method has advantages. This is already a real Persistent, and a classic one. On our disk, data will be written to some address.

There are also disadvantages. This is the complexity of management. Our Kubernetes may want to move the Pod to another physical node. This is where DevOps comes into play. It must correctly explain to the whole system that you can only move these pods to such nodes on which you have something mounted along these paths, and no more than one node at a time. It's difficult enough.

Especially for these purposes, we have made templates in our operator in order to hide all this complexity. And you could just say: "I want to have one instance of ClickHouse per physical node and along such and such a path."

But this need is not only for us, so the gentlemen from Kubernetes itself also understand that people want to have access to physical disks, so they provide a third level.

It's called local. There is practically no difference from the previous slide. Only earlier it was necessary to carry out by hand that we cannot transfer these pods from node to node, because they must be attached along such and such a path to the local physical disk, and now all this knowledge is encapsulated in Kubernetes itself. And it turns out much easier to configure.

Let's return to our practical task. Let's get back to the YAML template. Here we have a real storage. We are back to this. We set the classic VolumeClaim template as in k8s. And we describe what kind of storage we want.

After that, k8s will request storage. Allocate it to us in the StatefulSet. And in the end, it will turn out at the disposal of ClickHouse.

We had such a scheme. Our Persistent Storage was red, which seemed to hint that it should be done.

And it turns green. Now the ClickHouse on k8s cluster scheme is fully finalized. We have shards, replicas, ZooKeeper, we have real Persistent, which is implemented in one way or another. The scheme is already fully functional.

We continue to live on. Our cluster is growing. And Aleksey is trying and releasing a new version of ClickHouse.

A practical task arises - to test the new version of ClickHouse on our cluster. And, of course, I don’t want to roll it all out, I want to put a new version somewhere in the far corner in one replica, or maybe not one new version, but two at once, because they come out often.

What can we say about this?

Here we have just such an opportunity. These are pod templates. You can paint, our operator completely allows you to build a heterogeneous cluster. Those. configure, starting from all replicas in a bunch, ending with each personal replica, which version we want ClickHouse, which version we want storage. We can fully configure the cluster in such a configuration as we need.

Let's go deeper inside a bit. Before that, we talked about how the ClickHouse-operator works in relation to the specifics of ClickHouse.

Now I would like to say a few words about how any operator works in general, as well as how it interacts with K8s.

Consider interaction with K8s to begin with. What happens when we do kubectl apply? Through the API, our objects appear in etcd.

For example, the basic Kubernetes objects: pod, StatefulSet, service, and so on through the list.

However, nothing physical is happening yet. These objects must be materialized in a cluster.

This is where the controller comes in. The controller is a special k8s component that can materialize these descriptions. He knows how and what to do physically. He knows how to run containers, what needs to be configured there in order for the server to work.

And it materializes our objects in K8s.

But we want to operate not only with pods, StatefulSets, we want to create a ClickHouseInstallation, that is, an object of the ClickHouse type, in order to operate with it as a whole. So far, there is no such possibility.

But K8s has another nice thing. We want us to have such a complex entity somewhere, in which our cluster would be assembled from pods and StatefulSet.

And what should be done for this? First, Custom Resource Definition enters the scene. What it is? This is a description for K8s that you will have another data type that we want to add to the pod, StatefulSet, a custom resource that will be complex inside. This is a description of the data structure.

We also send it there via kubectl apply. Kubernetes happily took it.

And now in our storage, the object in etcd has the opportunity to write a custom resource called ClickHouseInstallation.

But for now, nothing else will happen. That is, if we now create a YAML file that we considered with a description of the shard, replicas and say “kubectl apply”, then Kubernetes will accept it, put it in etcd and say: “Great, but I don’t know what to do with it. I don't know how to maintain ClickHouseInstallation."

Accordingly, we need someone to help Kubernetes serve the new data type. On the left, we have a stock Kubernetes controller that works with stock data types. And on the right, we should have a custom controller that can work with custom data types.

And in another way it is called an operator. I specifically took it out here for Kubernetes, because it can also be executed outside of K8s. Most often, of course, all statements are executed in Kubernetes, but nothing prevents it from standing outside, so here it is specially brought out.

And already, in turn, the custom controller, also known as the operator, interacts with Kubernetes through the API. He already knows how to interact with the API. And he already knows how to materialize a complex scheme that we want to make from a custom resource. This is exactly what the operator does.

How does the operator work? Let's take a look at the right side to see how he does it. We will find out how the operator materializes all this and how further interaction with K8s takes place.

The operator is the program. She is event oriented. The operator subscribes to events using the Kubernetes API. The Kubernetes API has entry points where you can subscribe to events. And if something changes in K8s, then Kubernetes sends events to everyone, i.e. who subscribed to this API point will receive notifications.

The operator subscribes to events, and must do some kind of reaction. Its task is to respond to emerging events.

Events are generated by some updates. Our YAML file arrives with a description of ClickHouseInstallation. He went to etcd via kubectl apply. An event worked there, as a result, this event came to the ClickHouse-operator. The operator received this description. And he has to do something. If an update came to the ClickHouseInstallation object, then you need to update the cluster. And the task of the operator is to update the cluster.

What is he doing? First, we need to draw up an action plan for what we will do with this update. Updates can be very small, ie. small in YAML execution, but can lead to very large changes on the cluster. Therefore, the operator creates a plan, and then he adheres to it.

He begins, according to this plan, to boil this structure inside in order to materialize pods, services, i.e. to do what its main task is. It's like building a ClickHouse cluster in Kubernetes.

Now let's touch on such an interesting thing. This is a division of responsibility between Kubernetes and the operator, i.e. what Kubernetes does, what the operator does, and how they interact with each other.

Kubernetes is responsible for system things, i.e. for a basic set of objects that can be interpreted as a system-scope. Kubernetes knows how to start pods, how to restart containers, how to do mount volumes, how to work with ConfigMap, i.e. anything that can be called a system.

Operators operate in subject areas. Each operator is made for its subject area. We made for ClickHouse.

And the operator interacts precisely in terms of the subject area, such as adding a replica, making a scheme, setting up monitoring. There is such a division.

Let's look at a practical example of how this separation of concerns occurs when we do an add replica action.

The task comes to the operator - to add a replica. What is the operator doing? The operator will calculate that it is necessary to make a new StatefulSet, in which it is necessary to describe such and such templates, volume claim.

He prepared it all and passes it on to K8s. He says that he needs ConfigMap, StatefulSet, Volume. Kubernetes is working. He materializes the basic units with which he operates.

And then ClickHouse-operator comes into play again. He already has a physical pod on which you can already do something. And ClickHouse-operator again works in terms of the subject area. Those. Specifically, ClickHouse, in order to include a replica in a cluster, you must, firstly, configure the data scheme that exists in this cluster. And, secondly, this remark must be included in the monitoring so that it can be clearly traced. The operator already sets it up.

And only after that ClickHouse itself comes into play, i.e. another higher level entity. It's already a database. It has its own instance, the next configured replica, which is ready to join the cluster.

It turns out the chain of execution and separation of responsibility when adding a replica is long enough.

We continue our practical tasks. If the cluster already exists, then you can migrate the configuration.

We made it so that it is possible to pass through into the existing xml, which ClickHouse understands.

You can fine-tune ClickHouse. Just zoned deployment is what I talked about when explaining hostPath, local storage. This is how to do zoned deployment correctly.

The next practical task is monitoring.

If our cluster changes, then we need to periodically configure monitoring.

Let's take a look at the diagram. We have already considered the green arrows here. Now let's look at the red arrows. This is how we want to monitor our cluster. How metrics from the ClickHouse cluster get into Prometheus, and then into Grafana.

What's the problem with monitoring? Why is this presented as some kind of achievement? The difficulty lies in the dynamics. When we have one cluster and it is static, then you can set up monitoring once and not bother anymore.

But if we have a lot of clusters, or something is constantly changing, then the process is dynamic. And constantly reconfiguring monitoring is a waste of resources and time; even just lazy. This needs to be automated. The difficulty is in the dynamics of the process. And the operator automates this very well.

How did our cluster develop? In the beginning he was like this.

Then he was like this.

In the end, he became like this.

And monitoring is automatically done by the operator. Single point of entry.

And we just look at the exit in the Grafana dashboard, how the life of our cluster boils inside.

By the way, Grafana dashboard is also distributed with our operator right in the source code. You can connect and use. This screenshot was given to me by our DevOps.

Where would we like to go next? This:

• Develop test automation. The main task is automated testing of new versions.
• We also really want to automate the integration with ZooKeeper. And plans to integrate with ZooKeeper-operator. Those. an operator has been written for ZooKeeper, and it is logical that the two operators begin to integrate to build a more convenient solution.
• We want to do more complex life checks.
• I highlighted in green that we have Templates inheritance on the way - DONE, i.e. with the next release of the operator, we will already have template inheritance. This is a powerful tool that allows you to build complex configurations from pieces.
• And we want to automate complex tasks. The main one is Re-sharding.

Let's do some intermediate results.

What do we get as a result? And is it worth it or not? Do I even need to try to drag the database into Kubernetes and apply the operator in general and the Alitnity operator in particular.

At the output we get:

• Dramatically simplify and automate configuration, deployment, and maintenance.
• Immediately built-in monitoring.
• And ready-to-use codified templates for complex situations. Already the action of the type to add a replica does not need to be done by hand. This is done by the operator.

Only the last question remains. We already have a database in Kubernetes, virtualization. What about the performance of such a solution, especially since ClickHouse is optimized for performance?

The answer is everything is fine! I will not describe in detail, this is the topic of a separate report.

But there is such a project as TSBS. What is its main task? This is a database performance test. This is an attempt to compare warm with warm, soft with soft.

How does he work? One set of data is generated. Then this data set on the same test set is run on different databases. And each database solves one problem the way it can. And then you can compare the results.

It already supports a large bunch of databases. I have identified three main ones. This:

• timescaledb.
• InfluxDB.
• clickhouse.

A comparison was also made with another similar solution. Comparison with RedShift. The comparison was made on Amazon. ClickHouse is also well ahead of everyone in this matter.

What conclusions can be drawn from what I have said?

• DB in Kubernetes is possible. Probably, you can do anything, but in general it looks like you can. ClickHouse in Kubernetes is definitely possible with the help of our operator.
• The operator helps to automate processes and really simplifies life.
• Performance is normal.
• And, it seems to us that it can and should be used.

Open source - join us!

As I said, the operator is a completely open source product, so it would be very good if the maximum number of people used it. Join now! We are waiting for you all!

Thank you all!

Questions

Thanks for the report! My name is Anton. I'm from SEMrush. I'm wondering what's up with the logging. We hear about monitoring, but nothing about logging, if we talk about the whole cluster. For example, we have a cluster on hardware. And we use centralized logging, we collect it in a common heap by standard means. And then from there we get data that is interesting to us.

Good question, i.e. logging in the todo list. Our operator does not automate this yet. It is still developing, the project is still quite young. We understand the need for logging. This is also a very important topic. And it is probably no less important than monitoring. But first on the list for implementation was monitoring. There will be logging. Naturally, we are trying to automate all aspects of the life of the cluster. Therefore, the answer is that at the moment the operator, unfortunately, does not know how to do this, but it is in the plans, we will do it. If you want to join, then pull request, please.

Hello! Thanks for the report! I have a standard question related to Persistent Volumes. When we create a configuration with this operator, how does the operator determine on which node we have some disk or folder? We must first explain to him that, please, place our ClickHouse exactly on these nodes that have a disk?

As far as I understand, this question is a continuation of local storage, especially the hostPath part of it. It's like explaining to the whole system that it is necessary for the pod to be launched exactly on such and such a node, on which we have a physical connected disk, which is mounted on such and such a path. This is a whole section that I touched very superficially, because the answer there is quite large.

Briefly, it looks like this. Of course, we need to make provisioning of these volumes. At the moment, there is no dynamic provision in local storage, so DevOps must cut the disks themselves, here are these volumes. And they must explain Kubernetes provisioning, that you will have Persistent volumes of such and such a class, which is located on such and such nodes. Then it will be necessary to explain to Kubernetes that pods that require such and such a local storage class need to be scheduled according to labels only to such and such nodes. For these purposes, the operator has the ability to assign some kind of label and one per host instance. And it turns out that pods will be routed by Kubernetes to run only on nodes that meet the requirements, labels, in simple terms. Administrators assign labels, do provisioning of disks by hand. And then it scales.

And just the third option local helps to make it a little easier. As I have already emphasized, this is a painstaking work of tuning, which ultimately helps to get maximum performance.

I have a second question related to this. Kubernetes was conceived in such a way that it does not matter to us whether we lose a node or not. What should we do in this case if we have lost the node where we have a shard?

Yes, Kubernetes was originally positioned that our relationship with our pods is like cattle, but here each disk becomes something like a pet. There is such a problem that we cannot just throw them away. And the development of Kubernetes is going in the direction that it is impossible to completely treat it philosophically, as a completely discarded resource.

Now a practical question. What to do if you lost the node on which the disk was? Here the problem is solved at a higher level. In the case of ClickHouse, we have replicas that work at a higher level, i.e. at the ClickHouse level.

What is the disposition? DevOps is responsible for ensuring that data is not lost. It must properly set up replication and must ensure that replication is running. In the replica at the ClickHouse level, the data must be duplicated. This is not the task that the operator solves. And not the task that Kubernetes itself solves. This is at the ClickHouse level.

What to do if your iron node has fallen off? And it turns out that it will be necessary to put the second one, properly move the disk on it, apply labels. And after that, it will satisfy the requirements that Kubernetes on it can run a pod instance. Kubernetes will launch it. Your number of pods is not enough to the specified one. It will go through the cycle that I showed. And at the highest level, ClickHouse will understand that we have a replica entered, it is still empty and we need to start transferring data to it. Those. this process is still poorly automated.

Thanks for the report! When all sorts of nasty things happen, the operator crashes and restarts, and at that moment events arrive, do you somehow process this?

What happens if the operator crashes and restarts, yes?

Yes. And at that moment events came.

The task of what to do in this case is partially divided between the operator and Kubernetes. Kubernetes has the ability to replay an event that has occurred. He replays. And the task of the operator is to make sure that when the event log was replayed on it, these events are idempotent. And so that the re-occurrence of the same event does not break our system for us. And our operator copes with this task.

Hello! Thanks for the report! Dmitry Zavialov, company Smedov. Is it planned to add customization options with haproxy to the operator? Some other balancer is interesting besides the standard one, so that it is smart and understands that ClickHouse is real there.

Are you talking about Ingress?

Yes, replace Ingress with haproxy. In haproxy, you can specify the cluster topology where it has replicas.

So far, we haven't thought about it. If you need it and can explain why it is needed, then it will be possible to implement it, especially if you want to participate. We'll be happy to consider the option. The short answer is no, we currently do not have such functionality. Thanks for the tip, we'll take a look at this. And if you also explain the use case and why it is necessary in practice, for example, create issues on GitHub, then it will be great.

Has already.

Fine. We are open to any suggestions. And haproxy is put on the todo list. The todo list is growing, not shrinking yet. But this is good, it means that the product is in demand.

Source: habr.com