🥇A simple rpm repository using Inotify and webdav

In this post, we'll look at an rpm artifact repository using a simple inotify + createrepo script. Artifacts are uploaded via webdav using apache httpd. Why apache httpd will be written towards the end of the post.

So, the solution must meet the following requirements for organizing only RPM storage:

Free
Availability of the package in the repository a few seconds after uploading to the artifact repository.
Easy to install and maintain
Ability to make high availability

Why not SonaType Nexus or Pulp:
Storage in SonaType Nexus or Pulp many types of artifacts leads to the fact that SonaType Nexus or Pulp become a single point of failure.
High availability in SonaType Nexus is paid.
Pulp Seems like an over-engineered solution to me.
Artifacts in SonaType Nexus stored in blob. In the event of a sudden power outage, you will not be able to restore the blob if you do not have a backup. We had this error: ERROR [ForkJoinPool.commonPool-worker-2] *SYSTEM [com.orientechnologies.orient.core.storage](http://com.orientechnologies.orient.core.storage/).fs.OFileClassic - $ANSI{green {db=security}} Error during data read for file 'privilege_5.pcl' 1-th attempt [java.io](http://java.io/).IOException: Bad address. Blob was never recovered.

Source

→ The source code is located here

The main script looks like this:

#!/bin/bash

source /etc/inotify-createrepo.conf
LOGFILE=/var/log/inotify-createrepo.log

function monitoring() {
    inotifywait -e close_write,delete -msrq --exclude ".repodata|.olddata|repodata" "${REPO}" | while read events 
    do
      echo $events >> $LOGFILE
      touch /tmp/need_create
    done
}

function run_createrepo() {
  while true; do
    if [ -f /tmp/need_create ];
    then
      rm -f /tmp/need_create
      echo "start createrepo $(date --rfc-3339=seconds)"
      /usr/bin/createrepo --update "${REPO}"
      echo "finish createrepo $(date --rfc-3339=seconds)"
    fi
    sleep 1
  done
}

echo "Start filesystem monitoring: Directory is $REPO, monitor logfile is $LOGFILE"
monitoring >> $LOGFILE &
run_createrepo >> $LOGFILE &

Installation

inotify-createrepo only works on CentOS 7 or higher. Couldn't get it to work on CentOS 6.

yum -y install yum-plugin-copr
yum copr enable antonpatsev/inotify-createrepo
yum -y install inotify-createrepo
systemctl start inotify-createrepo

Configuration

By default inotify-createrepo monitors a directory /var/www/repos/rpm-repo/.

You can change this directory in the file /etc/inotify-createrepo.conf.

Using

When adding any file to a directory /var/www/repos/rpm-repo/ inotifywait will create a file /tmp/need_create. The run_createrepo function runs in an infinite loop and monitors the file /tmp/need_create. If the file exists, then run createrepo --update.

An entry will appear in the file:

/var/www/repos/rpm-repo/ CREATE nginx-1.16.1-1.el7.ngx.x86_64.rpm
start createrepo 2020-03-02 09:46:21+03:00
Spawning worker 0 with 1 pkgs
Spawning worker 1 with 0 pkgs
Spawning worker 2 with 0 pkgs
Spawning worker 3 with 0 pkgs
Workers Finished
Saving Primary metadata
Saving file lists metadata
Saving other metadata
Generating sqlite DBs
Sqlite DBs complete
finish createrepo 2020-03-02 09:46:22+03:00

Ability to make high availability

To make high availability out of an existing solution, I think you can use 2 servers, Keepalived for HA and Lsyncd for artifact synchronization. Lsyncd - a daemon that monitors changes in the local directory, aggregates them, and after a certain time, rsync starts to synchronize them. Details and settings are described in the post "Fast synchronization of a billion files".

WebDav

There are several ways to upload files: SSH, NFS, WebDav. WebDav seems to be a modern and simple option.

For WebDav, we will use Apache httpd. Why Apache httpd in 2020 and not nginx?

I would like to use automated tools for building Nginx + modules (for example, Webdav).

There is a project for building Nginx + modules - nginx-builder. If you use nginx + wevdav to upload files, then you need a module nginx-dav-ext-module. When trying to build and use Nginx with nginx-dav-ext-module by means of nginx-builder we will get an error Used by http_dav_module instead of nginx-dav-ext-module. The same bug was closed in the summer nginx: [emerg] unknown directive dav_methods.

I made a pull request Add check git_url for embedded, refactored --with-{}_module и if module == "http_dav_module" append --with. But they were not accepted.

config webdav.conf

DavLockDB /var/www/html/DavLock
<VirtualHost localhost:80>
    ServerAdmin webmaster@localhost
    DocumentRoot /var/www/html
    ErrorLog /var/log/httpd/error.log
    CustomLog /var/log/httpd/access.log combined

    Alias /rpm /var/www/repos/rpm-repo
    <Directory /var/www/repos/rpm-repo>
        DAV On
        Options Indexes FollowSymlinks SymLinksifOwnerMatch IncludesNOEXEC
        IndexOptions NameWidth=* DescriptionWidth=*
        AllowOverride none
        Require all granted
    </Directory>
</VirtualHost>

I think you will do the rest of the Apache httpd configuration yourself.

Nginx in front of Apache httpd

Unlike Apache, Nginx uses an event-based request processing model, which means that only one HTTP server process is required for any number of clients. You can use nginx and reduce server load.

nginx-front.conf config. I think you will do the rest of the nginx configuration yourself.

upstream nginx_front {
    server localhost:80;
}

server {
    listen 443 ssl;
    server_name ваш-виртуальных-хост;
    access_log /var/log/nginx/nginx-front-access.log main;
    error_log /var/log/nginx/nginx-front.conf-error.log warn;

    location / {
        proxy_pass http://nginx_front;
    }
}

Downloading files via WebDav

Downloading rpm is very easy.

curl -T ./nginx-1.16.1-1.el7.ngx.x86_64.rpm https://ваш-виртуальный-хост/rpm/

Source: habr.com

Simple rpm repository using Inotify and webdav