Building Your Nginx with a Few Commands

Hello!
My name is Sergey, I work as an infrastructure engineer on the API platform team at tinkoff.ru.

In this article, I will discuss the challenges our team encountered when preparing load balancers based on Nginx for various projects. I will also talk about the tool that helped us overcome most of them.

Nginx is a multifunctional and actively evolving proxy server. It features a vast number of modules, here's just a partial list. Each project imposes specific requirements on the functionality of the load balancer and the version of Nginx (such as the presence of http/2 and grpc proxying), as well as its module composition.

We want to see a fresh version with the required set of modules running on a specific Linux distribution. In our case, these are deb- and rpm-based systems. The container option is not considered in this article.

We want to quickly adjust the functionality of our load balancers. And here the question arises — how can we achieve this with minimal resource expenditure? It would also be better to establish a process where we can specify a finite number of input parameters and get an output artifact in the form of a deb/rpm package for the desired OS.

As a result, we can formulate a number of problems:

• There are not always packages with the latest version of Nginx.
• There are no packages with the required modules.
• Compiling and building the package manually takes a lot of time and is simply tiring.
• There is no description of how a particular instance of Nginx is compiled.

To solve these problems, a tool is needed that would take a specification in a human-readable format and compile an Nginx package with the required functionality based on it.

Not finding a suitable option for us in the vastness of GitHub, we decided to create our own tool — nginx-builder.

Specifications

In our tool, we wanted to create a specification description in the form of code, which could then be stored in a Git repository. For this purpose, we chose a familiar format for such things — yaml. Here's an example of a specification:

nginx_version: 1.14.1
output_package: deb
modules:
  - module:
      name: nginx-auth-ldap
      git_url: https://github.com/kvspb/nginx-auth-ldap.git
      git_branch: master
      dependencies:
        - libldap2-dev
  - module:
      name: ngx_http_substitutions_filter_module
      git_url: https://github.com/yaoweibin/ngx_http_substitutions_filter_module.git
  - module:
      name: headers-more-nginx-module
      web_url: https://github.com/openresty/headers-more-nginx-module/archive/v0.261.zip
  - module:
      name: nginx-module-vts
      git_url: https://github.com/vozlt/nginx-module-vts.git
      git_tag: v0.1.18
  - module:
      name: ngx_devel_kit
      git_url: https://github.com/simplresty/ngx_devel_kit.git
      git_tag: v0.3.0
  - module:
      name: ngx_cache_purge
      git_url: https://github.com/FRiCKLE/ngx_cache_purge.git
  - module:
      name: ngx_http_dyups_module
      git_url: https://github.com/yzprofile/ngx_http_dyups_module.git
  - module:
      name: nginx-brotli
      git_url: https://github.com/eustas/ngx_brotli.git
      git_tag: v0.1.2
  - module:
      name: nginx_upstream_check_module
      git_url: https://github.com/yaoweibin/nginx_upstream_check_module.git
  - module:
      name: njs
      git_url: https://github.com/nginx/njs.git
      git_tag: 0.2.5
      config_folder_path: nginx

Here we specify that we want to see a deb package with version Nginx 1.14.2 along with the required set of modules. The modules section is optional. For each module, we can specify:

• Name.
• Address where it can be obtained:
  • Git repository. A branch or tag can also be specified.
  • Web link to the archive.
  • Local link to the archive.

Some modules require the installation of additional dependencies; for example, nginx-auth-ldap requires the installed libldap2-dev. The necessary dependencies can also be specified in the module description.

Environment

In our tool, we can quickly obtain the environment with installed utilities for compiling, building the package, and other auxiliary software. Here, a Docker container with everything needed is ideal (the repository already has a couple of examples of Docker files for Ubuntu and CentOS).

After the specification is drafted and the environment is prepared, we run our builder, first installing its dependencies:

pip3 install -r requirements.txt
./main.py build -f [config_file].yaml -r [revision_number]

The revision number here is optional and serves for versioning the builds. It is recorded in the package metadata, making it easy to update on servers.
You can monitor the logs to observe what is happening. Here's an example of the main points:

builder - INFO - Parse yaml file: example.config.yaml
builder - INFO - Download scripts for build deb package
builder - INFO - Downloading nginx src...
builder - INFO - --> http://nginx.org/download/nginx-1.14.1.tar.gz
builder - INFO - Downloading 3d-party modules...
builder - INFO - Module nginx-auth-ldap will download by branch
builder - INFO - -- Done: nginx-auth-ldap
builder - INFO - -- Done: ngx_http_substitutions_filter_module
builder - INFO - Module headers-more-nginx-module will downloading
builder - INFO - Module nginx-module-vts will download by tag
builder - INFO - -- Done: nginx-module-vts
builder - INFO - Module ngx_devel_kit will download by tag
builder - INFO - -- Done: ngx_devel_kit
builder - INFO - -- Done: ngx_cache_purge
builder - INFO - -- Done: ngx_http_dyups_module
builder - INFO - Downloading dependencies
builder - INFO - Building .deb package
builder - INFO - Running 'dh_make'...
builder - INFO - Running 'dpkg-buildpackage'...
dpkg-deb: building package 'nginx' in '../nginx_1.14.1-1_amd64.deb'.

With just a couple of commands, we create the environment and the required Nginx build, and the package appears in the directory from where the script was executed.

Integration

We can also integrate our tool into CI/CD processes. This can be facilitated by any of the many existing CI systems today, such as Teamcity or Gitlab CI.

As a result, with every specification change in the Git repository, the artifact build is automatically triggered. The revision number is linked to the build run counter.
Spending a little more time, we can configure the artifact to be sent to a local package repository, Nexus, Artifactory, and so on.

An additional benefit is that the configuration yaml file can be connected in Ansible or another automation configuration system, allowing us to obtain the version number and package type that we want to deploy.

What's Next

The project is not yet complete. Here’s what we are currently working on:

• We are expanding the configuration capabilities while keeping it as simple as possible. We don't want to define a thousand parameters if only two are needed, and the rest defaults. This includes compilation flags (which can currently be changed in the internal configuration file src/config.py), installation paths, and the user for execution.
• Adding options for automatic package delivery to various artifact repositories.
• Execution of custom commands upon module loading (for example, for using github.com/nginx-modules/nginx_upstream_check_module a specific version patch needs to be applied first)
• Adding test execution:
  • The package installs correctly.
  • Nginx has the required version and is built with the necessary flags and modules.
  • Required paths, accounts, and so on are being created.

But you can start using this tool right now and also suggest improvements — github.com/TinkoffCreditSystems/Nginx-builder welcome!

Source: habr.com