In my quest to improve infrastructure, I decided to tackle an old and painful issue — to enable colleagues (developers, testers, admins, etc.) to independently manage their virtual machines in Ovirt without extra hassle. Ovirt has several components that need to be configured to address my question: the web interface, noVNC console, and image uploads.
I couldn't find a 'Make It Awesome' button, so I'm showing what steps I took to solve this task. A complete guide is below:
DISCLAIMER:
Before we begin, I would like to point out that for some unknown reason, infrastructure domains are created in private zones like lan, local, and so on.
What prevents the use of the organization’s domain in the public zone is unclear to me. For example, instead of the domain Alex-GLuck-Awesome-Company.local, you can confidently use the domain for the company’s website Alex-GLuck-Awesome-Company.com.
If you are worried that you won't be able to keep track of the domains in your organization and that something might break, for just 100 rubles a year, you can get a separate domain for your infrastructure aglac.com.
Why it is more advantageous to use domains in public zones:
1. You have services within your organization that go into the public space: VPN, file sharing (seafile, nextcloud), and others. Configuring traffic encryption on such services usually looks like a haphazard effort, and we won't be protected against MitM because it seems difficult (in reality, it is not).
Or you have one service address inside the office and a different one from the internet, and these connections need to be maintained, which drains our limited specialist resources. Plus, employees have to remember different addresses, which is inconvenient.
2. You can use free certification centers to encrypt your internal services.
A private PKI is a service that needs support; 100 rubles a year for using PKI from free certification centers more than pays off the time of employees who could spend that time on other tasks.
3. Using your own certification authority will create obstacles for your remote employees and colleagues who want to work with BYOD (bring your own devices, such as laptops, phones, tablets) and you can't manage their devices. They bring Macs, Linux, Androids, iOS, Windows—there's no point in supporting such a menagerie.
Of course, there are exceptions, and banks with other large enterprises that have established security policies will never be able to improve services for their employees.
There are paid certification centers that, for a certain fee, can sign their CA certificate (Google "root signing service").
There are also other reasons why using a public domain is more advantageous (the most important is that it belongs to you), but this article is not about that.
The point is…
WARNING! If you add the CA certificate from Let’s Encrypt to the trusted list for ovirt, it may affect the security of your systems!
First and foremost, it should be noted that exposing ovirt interfaces to the internet is a bad practice, as it serves no practical purpose and creates additional security threats.
Therefore, you need to obtain the certificate on one of our bastion hosts, after which the certificate and key must be transferred to our host with ovirt-engine.
Add the external address of our bastion host in DNS with our ovirt name. ovirtengine.example.com, I will leave the installation of certbot and nginx out of this (how to do it has already been described on Habr).
Configure Nginx version >=1.15.7
/etc/nginx/conf.d/default.conf
server {
server_name _;
listen 80 default_server;
location /robots.txt { alias /usr/share/nginx/html/robots.txt; }
location /.well-known {
root /usr/share/nginx/html;
}
location / {
return 444;
}
}
server {
server_name _;
listen 443 ssl http2 default_server;
location /robots.txt { alias /usr/share/nginx/html/robots.txt; }
location /.well-known {
root /usr/share/nginx/html;
}
ssl_certificate /etc/nginx/ssl/$ssl_server_name/fullchain.pem;
ssl_certificate_key /etc/nginx/ssl/$ssl_server_name/privkey.pem;
ssl_protocols TLSv1.2;
ssl_prefer_server_ciphers on;
ssl_dhparam /etc/nginx/ssl/dhparam.pem;
ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA';
ssl_session_timeout 1d;
ssl_session_cache shared:SSL:50m;
# Allows the server to attach OCSP responses, thereby reducing page load times for users
ssl_stapling on;
ssl_stapling_verify on;
add_header Strict-Transport-Security max-age=15768000;
location / {
return 444;
}
}
Then we obtain our certificate and key:
certbot certonly --nginx -d ovirtengine.example.com
We archive our certificate and key:
tar Phczf /tmp/ovirtengine.example.com.tgz /etc/letsencrypt/live/ovirtengine.example.com
We download the archive from the bastion host and upload it to our oVirt engine:
scp bastion-host:/tmp/ovirtengine.example.com.tgz /tmp/
scp /tmp/ovirtengine.example.com.tgz ovirtengine.example.com:/
We move to the target
Next, we unpack our archive and create symlinks for easier understanding of the file location system:
tar Pxzf /ovirtengine.example.com.tgz && rm -f ovirtengine.example.com.tgz
mkdir -p /etc/letsencrypt/live
ln -f -s /etc/letsencrypt/live /etc/pki/letsencrypt
We configure the built-in PKI in oVirt to use the Java certificate store for certificate validation (this system is used in Java):
cat < /etc/ovirt-engine/engine.conf.d/99-setup-pki.conf
ENGINE_HTTPS_PKI_TRUST_STORE="/etc/pki/java/cacerts"
ENGINE_HTTPS_PKI_TRUST_STORE_PASSWORD=""
EOF
We convert the CA from Let's Encrypt to DER format and add it to the Java trust store in oVirt (this is a container that holds the list of certificates, used in Java):
openssl x509 -outform der -in /etc/pki/letsencrypt/ovirtengine.example.com/chain.pem -out /tmp/ovirtengine.example.com.chain.der
keytool -import -alias "Let's Encrypt Authority X3" -file /tmp/ovirtengine.example.com.chain.der -keystore /etc/pki/ovirt-engine/.truststore -storepass $(grep '^ENGINE_PKI_TRUST_STORE_PASSWORD' /etc/ovirt-engine/engine.conf.d/10-setup-pki.conf | cut -f 2 -d '"')
rm -f /tmp/ovirtengine.example.com.chain.der
Editing SSL settings for Apache, adding a parameter to support symlinks and removing the CA parameter used to verify certificates (by default, it will use the system's set of trusted CAs for verification):
sed -r -i 's|^(SSLCACertificateFile.*)|#1|g' /etc/httpd/conf.d/ssl.conf
sed -r -i '0,/^(#?SSLCACertificateFile.*)/ s//1nOptions FollowSymlinks/' /etc/httpd/conf.d/ssl.conf
After that, we back up the original files generated by PKI ovirt for safety and replace the symlinks with files from Let’s Encrypt:
ln -f -s /etc/pki/letsencrypt/ovirtengine.example.com/fullchain.pem /etc/pki/ovirt-engine/apache-chain.pem
services=( 'apache' 'imageio-proxy' 'websocket-proxy' )
for i in "${services[@]}"; do
cp /etc/pki/ovirt-engine/certs/$i.cer{,."$( date +%F )".bak}
cp /etc/pki/ovirt-engine/keys/$i.key.nopass{,."$( date +%F )".bak}
ln -f -s /etc/pki/letsencrypt/ovirtengine.example.com/privkey.pem /etc/pki/ovirt-engine/keys/$i.key.nopass
ln -f -s /etc/pki/letsencrypt/ovirtengine.example.com/cert.pem /etc/pki/ovirt-engine/certs/{apache,imageio-proxy,websocket-proxy}.cer
done
Restoring SELinux contexts on the files and restarting our services (httpd, ovirt-engine, ovirt-imageio-proxy, ovirt-websocket-proxy):
restorecon -Rv /etc/pki
systemctl restart httpd ovirt-engine ovirt-imageio-proxy ovirt-websocket-proxy
httpd — web server. apache
ovirt-engine — web interface for ovirt
ovirt-imageio-proxy — daemon for loading disk images
ovirt-websocket-proxy — service for working with the noVNC console
All of the above has been verified on ovirt version 4.2.
Automatic certificate renewal on ovirt
According to good security practices, there should be no connection between the bastion host and ovirt, and the certificate is issued for only 3 months. This raises a controversial point regarding how certificate renewal is implemented in my setup.
I have an Ansible playbook that runs on Foreman daily at 5 AM as scheduled. This playbook checks the certificate's expiration on ovirt, and if there are less than 5 days left, it goes to the bastion host and initiates the certificate renewal.
After renewing the certificate, it archives the folder with the files, downloads it to the Foreman host, and unzips it on the ovirt host. After that, it restores the SELinux contexts on the files and restarts our services.
Source: habr.com
