🥇How to make Ovirt and Let's Encrypt friends

Walking along the path of improving the infrastructure, I decided to finish off an ancient and painful question - without unnecessary gestures, provide an opportunity for colleagues (developers, testers, admins, etc) to independently manage their virtual machines in ovirt. There are several components in ovirt that need to be configured to solve my problem: the web interface itself, the noVNC console, and uploading disk images.

I did not find the “Make Hurt” buttons, so I show which knobs I twisted to solve this problem. Full instructions under the cut:

DISCLAIMER:

Before starting, I would like to note that for some reason unknown to me, infrastructure domains are created in private zones lan, local, and so on.

What prevents me from using the organization's domain in the public zone is unknown to me. For example, instead of the Alex-GLuck-Awesome-Company.local domain, you can safely use the domain for the Alex-GLuck-Awesome-Company.com website.

If you are afraid that you will not be able to keep track of the domains in your organization, and this will break something, then for a modest 100 rubles a year you can take a separate domain for the aglac.com infrastructure.

Why is it more profitable to use domains in public zones:

1. Your organization has services that are publicly accessible: vpn, file sharing (seafile, nextcloud), and others. Setting up traffic encryption on such services is usually a bit of a slapdash effort, and we won't be defending against MitM attacks because it's difficult (not really).

Or inside the office you have one address of the service, and from the Internet another, and these connections must be maintained, which is what our limited resources of specialists are spent on. Well, employees have to remember different addresses, which is inconvenient.

2. You can use free CAs to encrypt your internal services.

Own PKI is a service that needs to be maintained, 100 rubles a year for the opportunity to use PKI from free certificate authorities more than pays for the time of employees who could spend it on other tasks.

3. When using your own certificate authority, you will put spokes in the wheels of your remote employees and colleagues who want to work with BYOD (bring their laptops, phone, tablets) and you cannot control their devices. They bring poppies, Linux, androids, IOS, Windows - there is no point in maintaining such a zoo.

Of course, there are exceptions to everything, and banks with other harsh enterprises that have well-established security policies will never be able to improve the service for their employees.

For them, there are paid certificate authorities that, for a certain amount, can sign their CA certificate (google “root signing service”).

There are other reasons why it is more profitable to use a public domain (the most important thing is that it belongs to you), but this article is not about that.

The bottom line is...

ATTENTION! If you add the CA certificate from Let's Encrypt to ovirt's trusted list, it may affect the security of your systems!

The first thing to pay attention to is that it is bad practice to expose ovirt interfaces on the Internet. there is no practical sense in this, and it creates additional security threats.

Therefore, it is necessary to obtain a certificate on some of our bastion hosts, and then transfer the certificate and key to our host with ovirt-engine.

We add the external address of our bastion host to the dns with our name ovirta ovirtengine.example.com, I will leave the installation of certbot and nginx behind the scenes (how to do this has already been described on Habré).

Setting up nginx version >=1.15.7

/etc/nginx/conf.d/default.conf

server {
    server_name _;
    listen 80 default_server;
    location /robots.txt { alias /usr/share/nginx/html/robots.txt; }
    location /.well-known {
        root /usr/share/nginx/html;
    }
    location / {
        return 444;
    }
}

server {
    server_name _;
    listen 443 ssl http2 default_server;
    location /robots.txt { alias /usr/share/nginx/html/robots.txt; }
    location /.well-known {
        root /usr/share/nginx/html;
    }

    ssl_certificate /etc/nginx/ssl/$ssl_server_name/fullchain.pem; 
    ssl_certificate_key /etc/nginx/ssl/$ssl_server_name/privkey.pem;

    ssl_protocols TLSv1.2;
    ssl_prefer_server_ciphers on;

    ssl_dhparam /etc/nginx/ssl/dhparam.pem;
    ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA';
    ssl_session_timeout 1d;
    ssl_session_cache shared:SSL:50m;

    # позволяем серверу прикреплять OCSP-ответы, тем самым уменьшая время загрузки страниц у пользователей
    ssl_stapling on;
    ssl_stapling_verify on;
    add_header Strict-Transport-Security max-age=15768000;

    location / {
        return 444;
    }
}

Then we get our certificate and key:

certbot certonly --nginx -d ovirtengine.example.com

We archive our certificate and key:

tar Phczf /tmp/ovirtengine.example.com.tgz /etc/letsencrypt/live/ovirtengine.example.com

We download the archive from the bastion host, upload it to our ovirt-engine:

scp bastion-host:/tmp/ovirtengine.example.com.tgz /tmp/
scp /tmp/ovirtengine.example.com.tgz ovirtengine.example.com:/

Let's go to the goal

Next, we unpack our archive and create symlinks to make it easier to understand the file location system:

tar Pxzf /ovirtengine.example.com.tgz && rm -f ovirtengine.example.com.tgz
mkdir -p /etc/letsencrypt/live
ln -f -s /etc/letsencrypt/live /etc/pki/letsencrypt

We configure the built-in pki in ovirt so that the java (openjdk) certificate store is used to check certificates:

cat << EOF > /etc/ovirt-engine/engine.conf.d/99-setup-pki.conf 
ENGINE_HTTPS_PKI_TRUST_STORE="/etc/pki/java/cacerts"
ENGINE_HTTPS_PKI_TRUST_STORE_PASSWORD=""
EOF

Convert the CA from let's encrypt to der format and add an ovirta to the java trust store (this is a container that contains a list of certificates, such a system is used in java):

openssl x509 -outform der -in /etc/pki/letsencrypt/ovirtengine.example.com/chain.pem -out /tmp/ovirtengine.example.com.chain.der
keytool -import -alias "Let's Encrypt Authority X3" -file /tmp/ovirtengine.example.com.chain.der -keystore /etc/pki/ovirt-engine/.truststore -storepass $(grep '^ENGINE_PKI_TRUST_STORE_PASSWORD' /etc/ovirt-engine/engine.conf.d/10-setup-pki.conf | cut -f 2 -d '"')
rm -f /tmp/ovirtengine.example.com.chain.der

We edit the SSL settings for apache, add a parameter to support symlinks and remove the parameter for the CA that checks certificates (by default, the system set of trusted CAs will be used for verification):

sed -r -i 's|^(SSLCACertificateFile.*)|#1|g' /etc/httpd/conf.d/ssl.conf
sed -r -i '0,/(^#?SSLCACertificateFile.*)/ s//1nOptions FollowSymlinks/' /etc/httpd/conf.d/ssl.conf

After that, just in case, we backup the original files generated via ovirt's PKI automatically and replace them with symlinks for files from Let's Encrypt:

ln -f -s /etc/pki/letsencrypt/ovirtengine.example.com/fullchain.pem /etc/pki/ovirt-engine/apache-chain.pem
services=( 'apache' 'imageio-proxy' 'websocket-proxy' )
for i in "${services[@]}"; do
cp /etc/pki/ovirt-engine/certs/$i.cer{,."$( date +%F )".bak}
cp /etc/pki/ovirt-engine/keys/$i.key.nopass{,."$( date +%F )".bak}
ln -f -s /etc/pki/letsencrypt/ovirtengine.example.com/privkey.pem /etc/pki/ovirt-engine/keys/$i.key.nopass
ln -f -s /etc/pki/letsencrypt/ovirtengine.example.com/cert.pem /etc/pki/ovirt-engine/certs/{apache,imageio-proxy,websocket-proxy}.cer
done

Restore SElinux contexts on files and restart our services (httpd, ovirt-engine, ovirt-imageio-proxy, ovirt-websocket-proxy):

restorecon -Rv /etc/pki
systemctl restart httpd ovirt-engine ovirt-imageio-proxy ovirt-websocket-proxy

httpd — web server Apache
ovirt-engine - ovirt web interface
ovirt-imageio-proxy - daemon for downloading disk images
ovirt-websocket-proxy - service for the noVNC console

All of the above was tested on Ovirt version 4.2.

Auto-renewal of certificates on ovirt

According to good security practices, there should be no connection between the bastion host and the ovirt, and the certificate is issued only for 3 months. This is where a controversial point appears about how the renewal of certificates is implemented for me.

I have an ansible playbook that runs on foreman every day at 5am on a schedule. This playbook goes to ovirt, checks the validity of the certificate, and if less than 5 days are left before the expiration, it goes to the bastion host and starts the certificate renewal.

After updating the certificate, it archives the folder with files, downloads the forman to the host and unzips it to the ovirta host. Then it restores SElinux contexts on files and restarts our services.

Source: habr.com

How to make friends with Ovirt and Let's Encrypt

DISCLAIMER:

The bottom line is...

Let's go to the goal

Auto-renewal of certificates on ovirt