🥇Simple RPM repository using Inotify and webdav

In this post, we will discuss RPM artifact storage using a simple script with inotify + createrepo. Artifacts are uploaded via WebDAV using Apache HTTPD. More details on Apache HTTPD will be provided later in the post.

So, the solution must meet the following requirements for setting up an RPM repository:

Free
The package should be available in the repository within seconds after being uploaded to the artifact storage.
Easy to install and maintain
Ability to achieve high availability
Why not SonaType Nexus or Pulp:
Storing multiple types of artifacts leads to SonaType Nexus or Pulp becoming a single point of failure. SonaType Nexus or Pulp High availability is
paid. SonaType Nexus It seems to me a complicated solution.
Pulp Artifacts in
are stored in a blob. In the event of a sudden power outage, you will not be able to recover the blob unless you have a backup. We had such an error: SonaType Nexus ERROR [ForkJoinPool.commonPool-worker-2] *SYSTEM [com.orientechnologies.orient.core.storage](http://com.orientechnologies.orient.core.storage/).fs.OFileClassic - $ANSI{green {db=security}} Error during data read for file 'privilege_5.pcl' 1st attempt [java.io](http://java.io/).IOException: Bad address . The blob was never recovered.→ The source code is located

Source Code

The main script looks like this: here

Inotify-createrepo works only on CentOS 7 or higher. It could not be made to work on CentOS 6.

#!/bin/bash

source /etc/inotify-createrepo.conf
LOGFILE=/var/log/inotify-createrepo.log

function monitoring() {
    inotifywait -e close_write,delete -msrq --exclude ".repodata|.olddata|repodata" "${REPO}" | while read events 
    do
      echo $events >> $LOGFILE
      touch /tmp/need_create
    done
}

function run_createrepo() {
  while true; do
    if [ -f /tmp/need_create ];
    then
      rm -f /tmp/need_create
      echo "start createrepo $(date --rfc-3339=seconds)"
      /usr/bin/createrepo --update "${REPO}"
      echo "finish createrepo $(date --rfc-3339=seconds)"
    fi
    sleep 1
  done
}

echo "Start filesystem monitoring: Directory is $REPO, monitor logfile is $LOGFILE"
monitoring >> $LOGFILE &
run_createrepo >> $LOGFILE &

Installation

yum -y install yum-plugin-copr yum copr enable antonpatsev/inotify-createrepo yum -y install inotify-createrepo systemctl start inotify-createrepo

By default, inotify-createrepo monitors the directory

Configuration

You can change this directory in the file /var/www/repos/rpm-repo/.

When any file is added to the directory /etc/inotify-createrepo.conf.

Using

inotifywait will create a file /var/www/repos/rpm-repo/ . The function run_createrepo runs in an infinite loop and monitors the file /tmp/need_create. If the file exists, then it executes /tmp/need_createcreaterepo --update A record will be added to the file:.

To achieve high availability from the existing setup, I think we can use 2 servers, Keepalived for HA and Lsyncd for artifact synchronization.

/var/www/repos/rpm-repo/ CREATE nginx-1.16.1-1.el7.ngx.x86_64.rpm
start createrepo 2020-03-02 09:46:21+03:00
Spawning worker 0 with 1 pkgs
Spawning worker 1 with 0 pkgs
Spawning worker 2 with 0 pkgs
Spawning worker 3 with 0 pkgs
Workers Finished
Saving Primary metadata
Saving file lists metadata
Saving other metadata
Generating sqlite DBs
Sqlite DBs complete
finish createrepo 2020-03-02 09:46:22+03:00

Ability to achieve high availability

Lsyncd is a daemon that watches for changes in a local directory, aggregates them, and after a certain period, starts rsync to synchronize them. Details and setup are described in the post " Fast synchronization of a billion filesWebDAV".

Files can be uploaded in several ways: SSH, NFS, WebDAV. WebDAV seems to be a modern and straightforward option.

For WebDAV, we will use Apache HTTPD. Why Apache HTTPD in 2020 and not Nginx?

We will use Apache httpd for WebDav. Why Apache httpd in 2020 instead of nginx?

I want to use automated tools for building Nginx + modules (for example, Webdav).

There is a project for building Nginx + modules — Nginx-builder. If using nginx + wevdav for file uploads, we need the module nginx-dav-ext-module. When attempting to build and use Nginx with nginx-dav-ext-module with the help of Nginx-builder we will encounter an error Used by http_dav_module instead of nginx-dav-ext-module. The same error was closed last summer nginx: [emerg] unknown directive dav_methods.

I made a Pull request Add check git_url for embedded, refactored —with-{}_module and if module == "http_dav_module" append —with. But they were not accepted.

webdav.conf configuration

DavLockDB /var/www/html/DavLock

    ServerAdmin webmaster@localhost
    DocumentRoot /var/www/html
    ErrorLog /var/log/httpd/error.log
    CustomLog /var/log/httpd/access.log combined

    Alias /rpm /var/www/repos/rpm-repo
    
        DAV On
        Options Indexes FollowSymlinks SymLinksifOwnerMatch IncludesNOEXEC
        IndexOptions NameWidth=* DescriptionWidth=*
        AllowOverride none
        Require all granted

I believe you can handle the rest of the Apache httpd configuration yourself.

Nginx before Apache httpd

Unlike Apache, Nginx uses an event-driven model for request processing, allowing a single HTTP server process to handle any number of clients. You can use nginx to reduce the load on the server.

nginx-front.conf configuration. I think you will manage the rest of the nginx configuration yourself.

upstream nginx_front {
    server localhost:80;
}

server {
    listen 443 ssl;
    server_name your-virtual-host;
    access_log /var/log/nginx/nginx-front-access.log main;
    error_log /var/log/nginx/nginx-front.conf-error.log warn;

    location / {
        proxy_pass http://nginx_front;
    }
}

File uploads via WebDav

Uploading RPM is very simple.

curl -T ./nginx-1.16.1-1.el7.ngx.x86_64.rpm https://your-virtual-host/rpm/

Source: habr.com

A simple RPM repository using Inotify and WebDAV