🥇Speeding Up Ansible

It's no secret that with default settings, Ansible may not perform very quickly. In this article, I'll point out several reasons for this and suggest a useful minimum of configurations that could realistically increase the speed of your project.

We discuss Ansible 2.9.x here and later, which was installed in a freshly created virtualenv using your preferred method.

After installation, create a file named 'ansible.cfg' next to your playbook — this placement will allow you to carry those settings along with the project, plus they will be loaded automatically.

Pipelining

Some might have heard about the need to use pipelining—that is, instead of copying modules to the target system's file system, sending a Base64-wrapped zip archive directly to the stdin of the Python interpreter—while others may not have been aware, but the fact remains: this setting is still undervalued. Unfortunately, some popular Linux distributions previously configured sudo poorly by default — to the extent that this command required a tty (terminal), hence this very useful setting was left disabled in Ansible by default.

pipelining = True

Fact Gathering

Did you know that with default settings, Ansible initiates fact gathering for each play across all hosts involved? If you didn't know, now you do. To prevent this from occurring, you need to enable either the explicit fact gathering mode or the smart mode. In smart mode, facts will only be gathered from hosts that haven't been encountered in previous plays.
UPD. When copying, you will have to choose one of these settings.

gathering = smart|explicit

Reusing SSH Connections

If you’ve ever run Ansible in debug output mode (the 'v' option, repeated from one to nine times), you may have noticed that SSH connections are constantly being established and broken. Well, there are a couple of nuances here too.

You can avoid the re-establishment of SSH connections at two levels: both in the SSH client itself and when transferring files to the managed host from the controller.
To reuse an open SSH connection, simply pass the necessary keys to the SSH client. It will then do the following: upon the first establishment of the SSH connection, it will additionally create what is known as a control socket; for subsequent connections, it will check for the existence of this socket and, if successful, reuse the existing SSH connection. To make this all meaningful, let's set a timeout for the connection when inactive. More details can be found in the SSH documentation, and in the context of Ansible, we simply use the ‘pass-through’ of the necessary SSH client options.

ssh_args = "-o ControlMaster=auto -o ControlPersist=15m"

To reuse an already opened SSH connection when transferring files to the managed host, it is sufficient to specify one more unknown setting: ssh_transfer_method. The documentation on this is rather sparse and can be misleading, as this option indeed works! However, reading it source code allows understanding what exactly will happen: a dd command will be executed on the managed host, directly interacting with the necessary file.

transfer_method = piped

By the way, this setting also exists in the ‘develop’ branch and has not gone anywhere.

Fear the knife, but fear the fork

Another useful setting is forks. It determines the number of worker processes that will connect to hosts and execute tasks simultaneously. Due to the peculiarities of Python as a programming language, processes, not threads, are used because Ansible still supports Python 2.7 — no asyncio here, we don’t need to stir up any asynchronous nonsense! By default, Ansible launches five workers, but if requested correctly, it can start more:

forks = 20

Just a heads-up, though; there may be some challenges related to the available memory on the controlling machine. In other words, while you can set forks=100500, who said it would actually work?

Let's wrap it all up

In the end, the required settings for ansible.cfg (ini format) may look like this:

[defaults]
gathering = smart|explicit
forks = 20
[ssh_connection]
pipelining = True
ssh_args = -o ControlMaster=auto -o ControlPersist=15m
transfer_method = piped

And if you want to tuck everything into a proper YAML inventory for a healthy person, it might look something like this:

---
all:
  vars:
    ansible_ssh_pipelining: true
    ansible_ssh_transfer_method: piped
    ansible_ssh_args: -o ControlMaster=auto -o ControlPersist=15m

Unfortunately, with the settings "gathering = smart/explicit" and "forks = 20", this won't work: their YaML equivalents do not exist. Either set them in ansible.cfg or pass them via the environment variables ANSIBLE_GATHERING and ANSIBLE_FORKS.

About Mitogen
— Where's the information about Mitogen? — you might ask, dear reader. In this article, it's nowhere. But if you're really ready to read its code and figure out why your playbook crashes with Mitogen while it works fine with vanilla Ansible, or why the same playbook that previously worked well has started to behave strangely after an update — well, Mitogen could potentially be your tool. Use it, investigate, write articles — I'll read them with interest.

Why don't I personally use Mitogen? Because it works only when the tasks are really simple and everything is fine. However, as soon as you veer slightly to the left or right — that's it: a handful of vague exceptions come flying at you, and to complete the picture, all that's missing is the clichéd phrase "thank you all, you're free to go". In short, I simply don't want to spend time figuring out the causes of yet another "underground thump."

Some of these settings were discovered while reading source code the connection plugin aptly named "ssh.py". I'm sharing the results of my reading in the hope that this inspires someone else to look at the source code, read it, check the implementation, compare it with the documentation — after all, this will bring you positive results sooner or later. Good luck!

Only registered users can participate in the survey. Please log in, please.