🥇Ansible: Migrating 120 VM configurations from CoreOS to CentOS over 18 months

This is a transcript of a presentation at DevopsConf 2019-10-01 and SPbLUG 2019-09-25.

This is the story of a project that used a custom configuration management system and why the transition to Ansible took 18 months.

Day № -XXX: Before the beginning

Initially, the infrastructure consisted of many standalone hosts managed by Hyper-V. Creating a virtual machine required several actions: placing disks in the right location, setting up DNS, reserving DHCP, and storing the VM configuration in a git repository. This process was partially automated, but for example, VMs were manually distributed among hosts. However, developers could adjust the VM configuration in git and apply it by rebooting the VM.

Custom Configuration Management Solution

The original idea, I suspect, was envisioned as IaC: numerous stateless VMs that would reset their state upon reboot. What did configuration management of VMs look like? Visually, it seems simple:

A static MAC was assigned to VMs.
An ISO with CoreOS and a boot disk were attached to the VM.
CoreOS runs a customization script downloaded from a web server based on its IP.
The script fetches the VM configuration via SCP based on the IP address.
A series of systemd unit files and a series of bash scripts are executed.

This solution had many obvious problems:

The ISO in CoreOS was deprecated.
Many complex automated actions and magic during migration/creation of VMs.
Challenges with updates and when software of a specific version is needed. Even more fun with kernel modules.
VMs weren't truly stateless, meaning some VMs had disks mounted with user data.
Someone constantly messed up the systemd unit dependencies, causing CoreOS to hang on reboot. It was problematic to catch this with the available tools in CoreOS.
Secret management.
CM was virtually nonexistent. There was bash and YML configs for CoreOS.

To apply the VM configuration, it had to be rebooted, but it might fail to reboot. It seemed like an obvious problem, but there were no persistent disks to save logs. Alright, let’s try to add kernel boot options to send logs. But no, it’s all so complicated.

Day №0: Acknowledgment of the problem

It was an ordinary development infrastructure: Jenkins, testing environments, monitoring, registry. CoreOS was intended for hosting k8s clusters, so the issue was how CoreOS was used. The first step was choosing the stack. We decided on:

CentOS as the base distribution, as it is the closest distribution to production environments.
Ansible for managing configurations, as there was extensive expertise on it.
Jenkins as a framework for automating existing processes, as it was already actively used for development processes.
Hyper-V as a virtualization platform. There are several reasons beyond this explanation, but briefly – we cannot use clouds, we must use our own hardware.

Day #30: Documenting existing agreements — Agreements as Code

Once the stack was understood, preparations for the move began. Documenting existing agreements in code (Agreements as Code!). Transition manual labor -> mechanization -> automation.

1. Configure VMs

Ansible handles this task excellently. With minimal effort, you can take control of VM configurations:

Create a git repository.
Store the list of VMs in inventory, configurations in playbooks and roles.
Set up a special Jenkins slave from which Ansible can be run.
Create a job, configure Jenkins.

The first process is ready. Agreements documented.

2. Create new VM

Here, it was not very convenient. From Linux, it was not easy to create VMs on Hyper-V. One of the attempts to automate this process was:

Ansible connects to the Windows host via WinRM.
Ansible runs a PowerShell script.
The PowerShell script creates a new VM.
Using Hyper-V/ScVMM, when creating a VM in the guest OS, the hostname is configured.
The VM sends its hostname when renewing the DHCP lease.
The standard integration of DDNS & DHCP on the Domain Controller side configures the DNS record.
VMs can be added to the inventory and configured with Ansible.

3. Create VM template

Here, we didn’t reinvent anything — we took Packer.

We store the Packer config and kickstart in the git repository.
Set up a special Jenkins slave with Hyper-V and Packer.
Create a job, configure Jenkins.

How this combination works:

Packer creates an empty VM, attaches the ISO.
The VM boots up, Packer prompts in the bootloader to use our kickstart file from diskette or HTTP.
Anaconda starts with our config, performing the initial OS setup.
Packer waits for the VM to become available.
Packer runs Ansible in local mode inside the VM.
Ansible operates using exactly the same roles as in step #1.
Packer exports the VM template.

Day #75: Refactoring agreements without breaking = Test ansible + Testkitchen.

It may not be enough to fix agreements in code. If you want to change something in the underlying process, you might break something. Therefore, testing of the infrastructure itself comes into play. To synchronize knowledge within the team, we started testing Ansible roles. I won’t delve deeper since there are articles describing the events at that time. Test me if you can or do YML programmers dream of testing ansible?(spoiler this was not the final version and later everything became more complicated How to start testing Ansible, refactor a project in a year and not lose your mind.).

Day #130: Do we really need CentOS+ansible? What about openshift?

It should be understood that the process of introducing infrastructure was not the only one, and there were side subprojects. For example, a request came to deploy our application in openshift, which resulted in research lasting more than a week. We are launching the application in Openshift and comparing the existing tools. What slowed down the migration process. It turned out that openshift does not meet all our needs; we need real hardware or at least the ability to tinker with the kernel.

Day #170: Openshift is not suitable, shall we risk with Windows Azure Pack?

Hyper-V is not very friendly, SCVMM doesn’t make it much better. But there is this thing called Windows Azure Pack, which is an add-on to SCVMM and mimics Azure. However, in reality, the product appears abandoned: documentation has broken links and is quite sparse. But as part of the research into options for simplifying our cloud, we looked into it as well.

Day #250: Windows Azure Pack is not great. We will stay with SCVMM.

Windows Azure Pack looked promising, but it was decided not to introduce WAP with all its complexities into the system for unnecessary features, so we stayed with SCVMM.

Day #360: We eat the elephant piece by piece.

Only after a year was there a platform ready for migration, and the process began. For this, a S.M.A.R.T. task was set. We listed all the VMs and began to deal with the configuration one by one, documenting it in Ansible and covering it with tests.

Day #450: What kind of system did we end up with?

The process itself is not interesting. It's routine, and it's worth noting that most configurations were relatively simple or isomorphic, following the Pareto principle: 80% of configurations took 20% of the time. Similarly, 80% of the time was spent on preparing for the move, and only 20% was spent on the move itself.