🥇Migration of Cassandra to Kubernetes: features and solutions

We regularly encounter the Apache Cassandra database and the need to operate it within a Kubernetes-based infrastructure. In this material, we will share our vision of the necessary steps, criteria and existing solutions (including an overview of operators) for migrating Cassandra to K8s.

“Whoever can rule a woman can also rule the state”

Who is Cassandra? It is a distributed storage system designed to manage large volumes of data while ensuring high availability without a single point of failure. The project hardly needs a long introduction, so I will give only the main features of Cassandra that will be relevant in the context of a specific article:

Cassandra is written in Java.
The Cassandra topology includes several levels:
- Node - one deployed Cassandra instance;
- Rack is a group of Cassandra instances, united by some characteristic, located in the same data center;
- Datacenter - a collection of all groups of Cassandra instances located in one data center;
- Cluster is a collection of all data centers.
Cassandra uses an IP address to identify a node.
To speed up writing and reading operations, Cassandra stores some of the data in RAM.

Now - to the actual potential move to Kubernetes.

Check-list for transfer

Speaking about the migration of Cassandra to Kubernetes, we hope that with the move it will become more convenient to manage. What will be required for this, what will help with this?

1. Data storage

As has already been clarified, Cassanda stores part of the data in RAM - in Memtable. But there is another part of the data that is saved to disk - in the form SSTable. An entity is added to this data Commit Log — records of all transactions, which are also saved to disk.

Write transaction diagram in Cassandra

In Kubernetes, we can use PersistentVolume to store data. Thanks to proven mechanisms, working with data in Kubernetes is becoming easier every year.

We will allocate our own PersistentVolume to each Cassandra pod

It is important to note that Cassandra itself implies data replication, offering built-in mechanisms for this. Therefore, if you are building a Cassandra cluster from a large number of nodes, then there is no need to use distributed systems like Ceph or GlusterFS for data storage. In this case, it would be logical to store data on the host disk using local persistent disks or mounting hostPath.

Another question is if you want to create a separate environment for developers for each feature branch. In this case, the correct approach would be to raise one Cassandra node and store the data in a distributed storage, i.e. the mentioned Ceph and GlusterFS will be your options. Then the developer will be sure that he will not lose test data even if one of the Kuberntes cluster nodes is lost.

2. Monitoring

The virtually uncontested choice for implementing monitoring in Kubernetes is Prometheus (we talked about this in detail in related report). How is Cassandra doing with metrics exporters for Prometheus? And, what’s even more important, with matching dashboards for Grafana?

An example of the appearance of graphs in Grafana for Cassandra

There are only two exporters: jmx_exporter и cassandra_exporter.

We chose the first one for ourselves because:

JMX Exporter is growing and developing, while Cassandra Exporter has not been able to get enough community support. Cassandra Exporter still does not support most versions of Cassandra.
You can run it as a javaagent by adding a flag -javaagent:<plugin-dir-name>/cassandra-exporter.jar=--listen=:9180.
There is one for him adequate dashboad, which is incompatible with Cassandra Exporter.

3. Selecting Kubernetes primitives

According to the above structure of the Cassandra cluster, let's try to translate everything that is described there into Kubernetes terminology:

Cassandra Node → Pod
Cassandra Rack → StatefulSet
Cassandra Datacenter → pool from StatefulSets
Cassandra Cluster → ???

It turns out that some additional entity is missing to manage the entire Cassandra cluster at once. But if something doesn't exist, we can create it! Kubernetes has a mechanism for defining its own resources for this purpose - Custom Resource Definitions.

Declaring additional resources for logs and alerts

But Custom Resource itself does not mean anything: after all, it requires controller. You may need to seek help kubernetes-operator...

4. Identification of pods

In the paragraph above, we agreed that one Cassandra node will equal one pod in Kubernetes. But the IP addresses of the pods will be different each time. And the identification of a node in Cassandra is based on the IP address... It turns out that after each removal of a pod, the Cassandra cluster will add a new node.

There is a way out, and not just one:

We can keep records by host identifiers (UUIDs that uniquely identify Cassandra instances) or by IP addresses and store it all in some structures/tables. The method has two main disadvantages:
- The risk of a race condition occurring if two nodes fall at once. After the rise, Cassandra nodes will simultaneously request an IP address from the table and compete for the same resource.
- If a Cassandra node has lost its data, we will no longer be able to identify it.
The second solution seems like a small hack, but nevertheless: we can create a Service with ClusterIP for each Cassandra node. Problems with this implementation:
- If there are a lot of nodes in a Cassandra cluster, we will have to create a lot of Services.
- The ClusterIP feature is implemented via iptables. This can become a problem if the Cassandra cluster has many (1000... or even 100?) nodes. Although balancing based on IPVS can solve this problem.
The third solution is to use a network of nodes for Cassandra nodes instead of a dedicated network of pods by enabling the setting hostNetwork: true. This method imposes certain limitations:
- To replace units. It is necessary that the new node must have the same IP address as the previous one (in clouds like AWS, GCP this is almost impossible to do);
- Using a network of cluster nodes, we begin to compete for network resources. Therefore, placing more than one pod with Cassandra on one cluster node will be problematic.

5. Backups

We want to save a full version of a single Cassandra node's data on a schedule. Kubernetes provides a convenient feature using CronJob, but here Cassandra herself puts a spoke in our wheels.

Let me remind you that Cassandra stores some of the data in memory. To make a full backup, you need data from memory (Memtables) move to disk (SSTables). At this point, the Cassandra node stops accepting connections, completely shutting down from the cluster.

After this, the backup is removed (snapshot) and the scheme is saved (keyspace). And then it turns out that just a backup doesn’t give us anything: we need to save the data identifiers for which the Cassandra node was responsible - these are special tokens.

Distribution of tokens to identify what data Cassandra nodes are responsible for

An example script for taking a Cassandra backup from Google in Kubernetes can be found at this link. The only point that the script does not take into account is resetting data to the node before taking the snapshot. That is, the backup is performed not for the current state, but for a state a little earlier. But this helps not to take the node out of operation, which seems very logical.

set -eu

if [[ -z "$1" ]]; then
  info "Please provide a keyspace"
  exit 1
fi

KEYSPACE="$1"

result=$(nodetool snapshot "${KEYSPACE}")

if [[ $? -ne 0 ]]; then
  echo "Error while making snapshot"
  exit 1
fi

timestamp=$(echo "$result" | awk '/Snapshot directory: / { print $3 }')

mkdir -p /tmp/backup

for path in $(find "/var/lib/cassandra/data/${KEYSPACE}" -name $timestamp); do
  table=$(echo "${path}" | awk -F "[/-]" '{print $7}')
  mkdir /tmp/backup/$table
  mv $path /tmp/backup/$table
done


tar -zcf /tmp/backup.tar.gz -C /tmp/backup .

nodetool clearsnapshot "${KEYSPACE}"

An example of a bash script for taking a backup from one Cassandra node

Ready solutions for Cassandra in Kubernetes

What is currently used to deploy Cassandra in Kubernetes and which of these best suits the given requirements?

1. Solutions based on StatefulSet or Helm charts

Using the basic StatefulSets functions to run a Cassandra cluster is a good option. Using the Helm chart and Go templates, you can provide the user with a flexible interface for deploying Cassandra.

This usually works fine... until something unexpected happens, such as a node failure. Standard Kubernetes tools simply cannot take into account all the features described above. Additionally, this approach is very limited in how much it can be extended for more complex uses: node replacement, backup, recovery, monitoring, etc.

Representatives:

Both charts are equally good, but are subject to the problems described above.

2. Solutions based on Kubernetes Operator

Such options are more interesting because they provide ample opportunities for managing the cluster. For designing a Cassandra operator, like any other database, a good pattern looks like Sidecar <-> Controller <-> CRD:

Node management scheme in a well-designed Cassandra operator

Let's look at existing operators.

1. Cassandra-operator from instaclustr

GitHub
Readiness: Alpha
License: Apache 2.0
Implemented in: Java

This is indeed a very promising and actively developing project from a company that offers managed Cassandra deployments. It, as described above, uses a sidecar container that accepts commands via HTTP. Written in Java, it sometimes lacks the more advanced functionality of the client-go library. Also, the operator does not support different Racks for one Datacenter.

But the operator has such advantages as support for monitoring, high-level cluster management using CRD, and even documentation for making backups.

2. Navigator from Jetstack

GitHub
Readiness: Alpha
License: Apache 2.0
Implemented in: Golang

A statement designed to deploy DB-as-a-Service. Currently supports two databases: Elasticsearch and Cassandra. It has such interesting solutions as database access control via RBAC (for this it has its own separate navigator-apiserver). An interesting project that would be worth taking a closer look at, but the last commit was made a year and a half ago, which clearly reduces its potential.

3. Cassandra-operator by vgkowski

GitHub
Readiness: Alpha
License: Apache 2.0
Implemented in: Golang

They didn’t consider it “seriously”, since the last commit to the repository was more than a year ago. Operator development is abandoned: the latest version of Kubernetes reported as supported is 1.9.

4. Cassandra-operator by Rook

GitHub
Readiness: Alpha
License: Apache 2.0
Implemented in: Golang

An operator whose development is not progressing as quickly as we would like. It has a well-thought-out CRD structure for cluster management, solves the problem of identifying nodes using Service with ClusterIP (the same “hack”)... but that’s all for now. There is currently no monitoring or backups out of the box (by the way, we are for monitoring took it ourselves). An interesting point is that you can also deploy ScyllaDB using this operator.

NB: We used this operator with minor modifications in one of our projects. No problems were noticed in the operator’s work during the entire period of operation (~4 months of operation).

5. CassKop from Orange

GitHub
Readiness: Alpha
License: Apache 2.0
Implemented in: Golang

The youngest operator on the list: the first commit was made on May 23, 2019. Already now it has in its arsenal a large number of features from our list, more details of which can be found in the project repository. The operator is built on the basis of the popular operator-sdk. Supports monitoring out of the box. The main difference from other operators is the use CassKop plugin, implemented in Python and used for communication between Cassandra nodes.

Conclusions

The number of approaches and possible options for porting Cassandra to Kubernetes speaks for itself: the topic is in demand.

At this stage, you can try any of the above at your own peril and risk: none of the developers guarantees 100% operation of their solution in a production environment. But already, many products look promising to try using in development benches.

I think in the future this woman on the ship will come in handy!

Migrating Cassandra to Kubernetes: features and solutions