Ansible + auto git pull in a cloud virtual machine cluster

Good day

We have several cloud clusters, each containing a large number of virtual machines. All of this is hosted on Hetzner. Each cluster has one master machine, from which a snapshot is taken and automatically distributed to all virtual machines within the cluster.

This scheme does not allow us to use gitlab-runners effectively, as it causes a lot of problems with the presence of many identically registered runners, which led us to find a workaround and write this article/manual.

This may not be a best practice, but this solution seems to be the most convenient and straightforward.

Please see the tutorial below.

Required packages on the master machine:

• python
• git
• file with ssh keys

The general principle of implementing automatic git pull on all virtual machines is that a machine is needed where Ansible will be installed. From this machine, Ansible will send git pull commands and restart the updated service. We created a separate virtual machine outside the clusters for this purpose and installed:

• python
• ansible
• gitlab-runner

From organizational issues — you need to register the gitlab-runner, generate ssh-keygen, and add the public ssh key of this machine to .ssh/authorized_keys on the master machine, and open port 22 on the master machine for Ansible.

Now let's configure Ansible

Since our goal is to automate everything possible. In the file /etc/ansible/ansible.cfg we will uncomment the line host_key_checking = False, so that Ansible does not ask for confirmation for new machines.

Next, it is necessary to automatically generate an inventory file for Ansible, from which it will fetch the IPs of the machines where git pull needs to be executed.

We generate this file using the Hetzner API, but you can also get the list of hosts from your AWS, Azure, database (you must have some API to output your running machines, right?).

For Ansible, the structure of the inventory file is very important; it should look like this:

[group]
ip-address
ip-address

[group2]
ip-address
ip-address

To generate such a file, we will create a simple script (let's call it vm_list):

#!/bin/bash
echo [group] > /etc/ansible/cloud_ip &&
"ваш CLI запрос на получение IP запущенных машин в кластере"  >> /etc/ansible/cloud_ip
echo " " >> /etc/ansible/cloud_ip
echo [group2] > /etc/ansible/cloud_ip &&
"ваш CLI запрос на получение IP запущенных машин в другом кластере"  >> /etc/ansible/cloud_ip

Now is the time to check that Ansible works and interacts with the IP address fetching:

/etc/ansible/./vm_list && ansible -i /etc/ansible/cloud_ip -m shell -a 'hostname' group

In the output, we should receive the hostnames of the machines where the command was executed.
A few words about syntax:

• /etc/ansible/./vm_list — генерируем список машин
• -i — absolute path to the inventory file
• -m — tells Ansible to use the shell module
• -a — argument. Here you can write any command
• group — the name of your cluster. If you want to apply it to all clusters, change group to all

Moving on — let's try to perform a git pull on our virtual machines:

/etc/ansible/./vm_list && ansible -i /etc/ansible/cloud_ip -m shell -a 'cd /path/to/project && git pull' group 

If we see already up to date or output from the repository, it means everything is working.

Now for what this was all intended for

We'll teach our script to run automatically when there is a commit to the master branch in GitLab

First, let's make our script prettier and put it into an executable file (let's call it exec_pull) —

#!/bin/bash

/etc/ansible/./get_vms && ansible -i /etc/ansible/cloud_ip -m shell -a "$@"

Let's go to our GitLab and create a file in the project .gitlab-ci.yml
Inside we place the following:

variables:
  GIT_STRATEGY: none
  VM_GROUP: group

stages:
  - pull
  - restart

run_exec_pull:
  stage: pull
  script:
  
   - /etc/ansible/exec_pull 'cd /path/to/project/'$CI_PROJECT_NAME' && git pull' $VM_GROUP
  
  only:
  - master

run_service_restart:
  stage: restart
  script:
 
   - /etc/ansible/exec_pull 'your_app_stop && your_app_start' $VM_GROUP
   
  only:
  - master 

Everything is ready. Now —

• let's make a commit
• let's hope that everything works

When transferring .yml to other projects, it is enough to change only the service name for restart and the name of the cluster on which the Ansible commands will be executed.

Source: habr.com