{"id":33075,"date":"2019-10-31T21:50:35","date_gmt":"2019-10-31T18:50:35","guid":{"rendered":"https:\/\/prohoster.info\/blog\/besprostojnaya-migratsiya-rabbitmq-v-kubernetes\/"},"modified":"2019-10-31T21:50:35","modified_gmt":"2019-10-31T18:50:35","slug":"besprostojnaya-migratsiya-rabbitmq-v-kubernetes","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/besprostojnaya-migratsiya-rabbitmq-v-kubernetes","title":{"rendered":"Challenging migration of RabbitMQ to Kubernetes","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

\"Challenging
\n
\nRabbitMQ is a message broker written in Erlang, enabling the setup of a fault-tolerant cluster with full data replication across multiple nodes, where each node can handle read and write requests. Having numerous Kubernetes clusters in production, we manage a large number of RabbitMQ installations and faced the need to migrate data from one cluster to another without downtime.<\/a><\/noindex><\/p>\n

This operation was necessary for us in at least two cases:<\/p>\n

    \n
  1. Transferring data from a RabbitMQ cluster that is not in Kubernetes to a new 'Kubernetesized' cluster (i.e., functioning in K8s pods).<\/li>\n
  2. Migrating RabbitMQ within Kubernetes from one namespace to another (for instance, if boundaries are defined by namespaces, for transferring infrastructure from one boundary to another).<\/li>\n<\/ol>\n

    \nThe recipe proposed in this article is aimed at situations (but not limited to them) where there is an old RabbitMQ cluster (for example, consisting of 3 nodes) that is either already in K8s or on some old servers. An application running in Kubernetes (either already there or potentially) interacts with it:<\/p>\n

    \"Challenging
    \n
    \n\u2026 and we need to migrate it to a new production environment in Kubernetes.<\/p>\n

    First, a general approach to the migration will be described, followed by technical details on its implementation.<\/p>\n

    Migration Algorithm<\/h2>\n

    \nThe first, preliminary stage before any actions is to check that the old RabbitMQ installation has high availability mode enabled (HA<\/a><\/noindex>). The reason is obvious\u2014we don\u2019t want to lose any data. To perform this check, one can access the RabbitMQ admin panel and verify in the Admin \u2192 Policies tab that the value is set to ha-mode: all<\/code>:<\/p>\n

    \"Challenging
    \n
    \nThe next step is to set up a new RabbitMQ cluster in Kubernetes pods (in our case, for example, consisting of 3 nodes, though the number can vary).<\/p>\n

    After that, we merge the old and new RabbitMQ clusters, forming a single cluster (comprising 6 nodes):<\/p>\n

    \"Challenging
    \n
    \nThe process of synchronizing data between the old and new RabbitMQ clusters is initiated. Once all data is synchronized across all nodes in the cluster, we can switch the application to use the new cluster:<\/p>\n

    \"Challenging
    \n
    \nAfter these operations, it is sufficient to remove the old nodes from the RabbitMQ cluster, and the migration can be considered complete:<\/p>\n

    \"Challenging
    \n
    \nWe have repeatedly applied this scheme in our production. However, for our convenience, we implemented it within a specialized system that disseminates standard RMQ configurations across multiple Kubernetes clusters. (for those curious: this is about     addon-operator<\/a><\/noindex>, which we we recently discussed<\/a><\/noindex>)<\/i>. Below you will find specific instructions that anyone can apply to their installations to try the proposed solution in action.<\/p>\n

    Let's try it in practice<\/h2>\n

    <\/p>\n

    Requirements<\/h3>\n

    \nThe prerequisites are very simple:<\/p>\n

      \n
    1. A Kubernetes cluster (minikube will also work);<\/li>\n
    2. A RabbitMQ cluster (it can be deployed on bare metal or created as a regular cluster in Kubernetes using the official Helm chart).<\/li>\n<\/ol>\n

      \nFor the example described below, I deployed RMQ in Kubernetes and named it rmq-old<\/code>.<\/p>\n

      Preparing the environment<\/h3>\n

      \n1. Download the Helm chart and make some edits:<\/p>\n

      helm fetch --untar stable\/rabbitmq-ha<\/code><\/pre>\n
      \nFor convenience, we set the password, ErlangCookie<\/code> and create a policy ha-all<\/code>, so that by default, queues are synchronized between all nodes of the RMQ cluster:<\/p>\n
      rabbitmqPassword: guest\nrabbitmqErlangCookie: mae9joopaol7aiVu3eechei2waiGa2we\ndefinitions:\npolicies: |-\n  {\n    \"name\": \"ha-all\",\n    \"pattern\": \".*\",\n    \"vhost\": \"\\\/\",<\/code><\/pre>\n
      \n2. Install the chart:<\/p>\n
      helm install . --name rmq-old --namespace rmq-old<\/code><\/pre>\n
      \n3. Access the RabbitMQ admin panel, create a new queue, and add some messages. These will be necessary to ensure that all data is preserved after migration and nothing is lost:<\/p>\n
      \"Challenging
      \n
      \nThe test environment is ready: we have the 'old' RabbitMQ containing the data to be migrated.<\/p>\n
      Migrating the RabbitMQ cluster<\/h3>\n
      \n1. First, let's deploy a new RabbitMQ in a friend<\/b> the namespace with the same<\/b> ErlangCookie<\/code> and a password for the user. To do this, we will perform the operations described above, changing the final command for installing RMQ to the following:<\/p>\n
      helm install . --name rmq-new --namespace rmq-new<\/code><\/pre>\n
      \n2. Now, we need to merge the new cluster with the old one. For this, we enter each of the pods of the new<\/b> RabbitMQ and execute the commands:<\/p>\n
      export OLD_RMQ=rabbit@rmq-old-rabbitmq-ha-0.rmq-old-rabbitmq-ha-discovery.rmq-old.svc.cluster.local && \n  rabbitmqctl stop_app && \n  rabbitmqctl join_cluster $OLD_RMQ && \n  rabbitmqctl start_app<\/code><\/pre>\n
      \nIn the variable OLD_RMQ<\/code> is the address of one of the nodes of the old<\/b> RMQ cluster.<\/p>\n
      These commands will stop the current node of the new<\/b> of the RMQ cluster, join it to the old cluster, and then restart it.<\/p>\n
      3. The RMQ cluster with 6 nodes is ready:<\/p>\n
      \"Challenging
      \n
      \nYou need to wait for messages to synchronize between all nodes. It\u2019s easy to guess that the message synchronization time depends on the hardware capabilities of the cluster and the number of messages. In the scenario described, there are only 10 messages, so the data synchronized instantly, but with a sufficiently large number of messages, synchronization can take hours.<\/p>\n
      So, the synchronization status:<\/p>\n
      \"Challenging
      \n
      \nHere +5<\/code> means that the messages are already present still<\/b> on 5 nodes (in addition to the one specified in the field Node<\/code>). Thus, the synchronization was successful.<\/p>\n
      4. All that remains is to switch the RMQ address in the application to the new cluster (the specific actions here depend on the technology stack you are using and other specifics of your application), after which you can say goodbye to the old one.<\/p>\n
      For the last operation (i.e., already after<\/b> switching the application to the new cluster), we go to each node of the old<\/b> of the cluster and execute the commands:<\/p>\n
      rabbitmqctl stop_app\nrabbitmqctl reset<\/code><\/pre>\n
      \nThe cluster has 'forgotten' about the old nodes: you can delete the old RMQ, and the migration will be complete.<\/p>\n
      Note<\/b>: If you are using RMQ with certificates, then fundamentally nothing changes\u2014the migration process will proceed exactly the same way.<\/i><\/p>\n
      Conclusions<\/h2>\n
      \nThe scheme described is suitable for almost all cases where we need to move RabbitMQ or simply migrate to a new cluster.<\/p>\n
      In our case, difficulties arose only once when RMQ was accessed from multiple locations, and we couldn\u2019t change the RMQ address everywhere. At that point, we launched a new RMQ in the same namespace with the same labels so that it would fall under the already existing services and Ingresses, and when starting the pod manually, we manipulated the labels, removing them at first so that requests would not go to the empty RMQ, and adding them back after message synchronization.<\/p>\n
      We applied the same strategy when upgrading RabbitMQ to a new version with altered configuration\u2014it worked like a charm.<\/p>\n
      P.S.<\/h2>\n
      \nAs a logical continuation of this material, we are preparing articles about MongoDB (migration from a physical server to Kubernetes) and MySQL (how we set up this DBMS inside Kubernetes). They will be published in the coming months.<\/p>\n
      P.P.S.<\/h2>\n
      \nAlso read in our blog:<\/p>\n