🥇Bare-Metal Provisioning on your own, or Automatic Server Setup from Scratch

Hello, I'm Denis, and one of my areas of expertise is developing infrastructure solutions at X5. Today, I would like to share how to deploy an automatic server preparation system using widely available tools. In my opinion, this is an interesting, simple, and flexible solution.

Preparation means: turning a new out-of-the-box server into a fully configured server with a Linux operating system or with the ESXi hypervisor (Windows is not discussed in this article). servers Windows is not discussed in this article.

Terms:

Servers – servers that need to be configured.
Install server – the main server that facilitates the entire preparation process over the network.

Why is automation needed?

Let's say there is a task: to prepare servers in bulk from scratch, with a peak of 30 per day. These are servers from various manufacturers and models, they can have different operating systems installed, and there may or may not be a hypervisor.

What operations are included in the setup process (without automation):

connect a keyboard, mouse, and monitor to the server;
configure the BIOS, RAID, IPMI;
update component firmware;
deploy a filesystem image (or install a hypervisor and copy virtual machines);

Note. As an option, OS deployment can be done via an installation with an answer file. However, this will not be discussed in the article. Although below you will see that adding this functionality is not complicated.

configure the OS parameters (hostname, IP, etc.).

With this approach, the same settings are applied sequentially on each server. The efficiency of such work is very low.

The essence of automation is to eliminate human involvement in the server preparation process as much as possible.

Thanks to automation, downtime between operations is reduced, allowing for the preparation of several servers simultaneously. It also significantly decreases the likelihood of errors due to human factors.

How does automatic server configuration take place?

Let's break down each stage in detail.

You have a Linux server that you use as a PXE install server. It has DHCP and TFTP services installed and configured.

So, we boot the server (that needs to be configured) via PXE. Let's recall how this works:

The server is set to boot from the network.
The server loads the PXE-ROM of the network card and contacts the installation server via DHCP to obtain a network address.
The DHCP installation server issues an address as well as instructions for further loading via PXE.
The server downloads the network loader from the installation server via PXE; further loading occurs according to the PXE configuration file.
Loading occurs based on the received parameters (kernel, initramfs, mount points, squashfs image, etc.).

Note: This article describes PXE booting via BIOS mode. Currently, manufacturers are actively implementing UEFI boot mode. For PXE, the difference will be in the configuration of the DHCP server and the presence of an additional loader.

Let's consider an example of PXE server configuration (pxelinux menu).

File pxelinux.cfg/default:

default menu.c32
prompt 0
timeout 100
menu title X5 PXE Boot Menu
LABEL InstallServer Menu
	MENU LABEL InstallServer
	KERNEL menu.c32
	APPEND pxelinux.cfg/installserver
LABEL VMware Menu
	MENU LABEL VMware ESXi Install
	KERNEL menu.c32
	APPEND pxelinux.cfg/vmware
LABEL toolkit // default menu
	MENU LABEL Linux Scripting Toolkits
	MENU default
	KERNEL menu.c32
	APPEND pxelinux.cfg/toolkit // switch to next menu

File pxelinux.cfg/toolkit:

prompt 0
timeout 100
menu title X5 PXE Boot Menu
label mainmenu
    menu label ^Return to Main Menu
    kernel menu.c32
    append pxelinux.cfg/default
label x5toolkit-auto // default — automatic mode
        menu label x5 toolkit autoinstall
        menu default
        kernel toolkit/tkcustom-kernel
        append initrd=toolkit/tk-initramfs.gz quiet net.ifnames=0 biosdevname=0 nfs_toolkit_ip=192.168.200.1 nfs_toolkit_path=tftpboot/toolkit nfs_toolkit_script=scripts/mount.sh script_cmd=master-install.sh CMDIS2="…"
label x5toolkit-shell // for debugging - console
        menu label x5 toolkit shell
        kernel toolkit/tkcustom-kernel
        append initrd=toolkit/tkcustom-initramfs.gz quiet net.ifnames=0 biosdevname=0 nfs_toolkit_ip=192.168.200.1 nfs_toolkit_path=tftpboot/toolkit nfs_toolkit_script=scripts/mount.sh script_cmd=/bin/bash CMDIS2="…"

At this stage, the kernel and initramfs serve as an intermediate Linux image, which will be used for the main preparation and configuration of the server.

As you can see, the bootloader passes many parameters to the kernel. Some of these parameters are used by the kernel itself, while others can be used for our purposes. This will be discussed later, but for now, just remember that all passed parameters will be available in the intermediate Linux image through /proc/cmdline.

Where can we obtain the kernel and initramfs?
You can choose any Linux distribution as a basis. What to pay attention to when selecting:

the boot image should be universal (availability of drivers, ability to install additional utilities);
It will most likely be necessary to customize initramfs.

How is this done in our solution for X5? We have chosen CentOS 7 as the basis. We will perform the following trick: prepare the future image structure, pack it into an archive, and create initramfs, within which our filesystem archive will be stored. When booting the image, the archive will be unpacked into the created tmpfs partition. This way, we will have a minimal yet fully functional live Linux image with all the necessary utilities, consisting of just two files: vmkernel and initramfs.

#создаем директории: 

mkdir -p /tftpboot/toolkit/CustomTK/rootfs /tftpboot/toolkit/CustomTK/initramfs/bin

#подготавливаем структуру:

yum groups -y install "Minimal Install" --installroot=/tftpboot/toolkit/CustomTK/rootfs/
yum -y install nfs-utils mariadb ntpdate mtools syslinux mdadm tbb libgomp efibootmgr dosfstools net-tools pciutils openssl make ipmitool OpenIPMI-modalias rng-tools --installroot=/tftpboot/toolkit/CustomTK/rootfs/
yum -y remove biosdevname --installroot=/tftpboot/toolkit/CustomTK/rootfs/

# подготавливаем initramfs:

wget https://busybox.net/downloads/binaries/1.31.0-defconfig-multiarch-musl/busybox-x86_64 -O /tftpboot/toolkit/CustomTK/initramfs/bin/busybox
chmod a+x /tftpboot/toolkit/CustomTK/initramfs/bin/busybox
cp /tftpboot/toolkit/CustomTK/rootfs/boot/vmlinuz-3.10.0-957.el7.x86_64 /tftpboot/toolkit/tkcustom-kernel

# создаем /tftpboot/toolkit/CustomTK/initramfs/init (ниже содержание скрипта):

#!/bin/busybox sh
/bin/busybox --install /bin
mkdir -p /dev /proc /sys /var/run /newroot
mount -t proc proc /proc
mount -o mode=0755 -t devtmpfs devtmpfs /dev
mkdir -p /dev/pts /dev/shm /dev/mapper /dev/vc
mount -t devpts -o gid=5,mode=620 devpts /dev/pts
mount -t sysfs sysfs /sys
mount -t tmpfs -o size=4000m tmpfs /newroot
echo -n "Extracting rootfs... "
xz -d -c -f rootfs.tar.xz | tar -x -f - -C /newroot
echo "done"
mkdir -p /newroot/dev /newroot/proc /newroot/sys
mount --move /sys  /newroot/sys
mount --move /proc /newroot/proc
mount --move /dev  /newroot/dev
exec switch_root /newroot /sbin/init

# упаковываем rootfs и initramfs:

cd /tftpboot/toolkit/CustomTK/rootfs
tar cJf /tftpboot/toolkit/CustomTK/initramfs/rootfs.tar.xz --exclude ./proc --exclude ./sys --exclude ./dev .
cd /tftpboot/toolkit/CustomTK/initramfs
find . -print0 | cpio --null -ov --format=newc | gzip -9 > /tftpboot/toolkit/tkcustom-initramfs-new.gz

So, we have specified the kernel and initramfs that should be loaded. As a result, at this stage, loading the intermediate Linux image via PXE will give us the OS console.

Great, but now we need to transfer control to our automation.

This can be done like this.

Let's assume that after loading the image we plan to transfer control to the script mount.sh.
We will include the script mount.sh in the autostart. To do this, it will be necessary to modify initramfs:

unpack initramfs (if using the above version of initramfs, this is not required)
include in the autostart code that will analyze the parameters passed through /proc/cmdline and transfer control further;
pack initramfs.

Note: In the case of the X5 toolkit, control is passed to the script. /opt/x5/toolkit/bin/hook.sh с помощью override.conf в getty tty1 (ExecStart=…)

So, an image is booted, in which the script mount.sh starts in the autostart. Then, during its execution, the mount.sh script analyzes the passed parameters (script_cmd=) and launches the required program/script.

label toolkit-auto
kernel …
append … nfs_toolkit_script=scripts/mount.sh script_cmd=master-install.sh

label toolkit-shell
kernel …
append … nfs_toolkit_script=scripts/mount.sh script_cmd=/bin/bash

Here on the left is the PXE menu, and on the right is the control transfer diagram.

We have figured out control transfer. Depending on the choice in the PXE menu, either the auto-configuration script or the console for debugging starts.

In the case of automatic configuration, the necessary directories from the install server are mounted, which contain:

scripts;
saved BIOS/UEFI templates for various servers;
firmware;
utilities for servers;
logs.

Next, the mount.sh script transfers control to the master-install.sh script from the scripts directory.

The script tree (the order of their execution) looks something like this:

master-install
sharefunctions (common functions)
info (information output)
models (installation parameters setup based on the server model)
prepare_utils (installation of necessary utilities)
fwupdate (firmware update)
diag (basic diagnostics)
biosconf (BIOS/UEFI configuration)
clockfix (setting time on the motherboard)
srmconf (configuring remote interface interface)
raidconf (configuring logical volumes)

one of:

preinstall (handing over control to the OS or hypervisor installer, e.g., ESXi)
merged-install (directly starting the unpacking of the image)

Now we know:

how to boot the server via PXE;
how to hand over control to our own script.

Let's continue. The following questions have become relevant:

How to identify the server we are preparing?
Which utilities and how to configure the server?
How to get settings for a specific server?

How to identify the server we are preparing?

It's simple – DMI:

dmidecode –s system-product-name
dmidecode –s system-manufacturer
dmidecode –s system-serial-number

Here is everything you need: vendor, model, serial number. If you are not sure that this information is available on all servers, you can identify them by MAC address. Or use both methods simultaneously if vendors are different and some models simply do not have serial number information.

Based on the received information, network folders are mounted from the install server and everything necessary (utilities, firmware, and more) is loaded.

Which utilities and how to configure the server?

I will provide utilities for Linux for some manufacturers. All utilities are available on the vendors' official websites.

With firmware, I think everything is clear. They are usually provided as packaged executable files. The executable file controls the firmware update process and reports the return code.

BIOS and IPMI are usually configured via templates. If necessary, the template can be edited just before booting.

Some vendor RAID utilities can also be configured via templates. If not, a configuration script will need to be written.

The RAID setup order is usually as follows:

Request the current configuration.
If logical arrays already exist – delete them.
Check what physical disks are present and how many there are.
Create a new logical array. Interrupt the process in case of an error.

How to get settings for a specific server?

Assuming all server settings will be stored on the install server. In this case, to answer our question, we first need to decide how to transfer settings to the install server.

Initially, it is quite possible to rely on text files. (In the future, a text file can be used as a backup method for transferring settings).

You can 'share' a text file on the install server and add its mounting to the mount.sh script.

The lines will appear like this:

These lines will be sent to a file by the engineer from their work machine. Then, during server setup, parameters for the specific server will be read from the file.

However, in the long run, it's better to use a database for storing settings, states, and installation logs of the servers.

Of course, one database alone won’t suffice, and a client-side component will need to be created to transmit the settings to the database. Implementing this is more complex compared to a text file, but in reality, it’s not as difficult as it seems. A minimal version of the client, which simply transmits data to the database, can definitely be developed independently. Moreover, enhancing the client program later on can be done freely (reports, printing labels, sending notifications, and anything else that comes to mind).

By making a specific request to the database and specifying the server's serial number, we will get the necessary parameters for configuring the server.

Additionally, we won't need to devise locks for simultaneous access, as is the case with a text file.

We can log the configuration at all stages into the database and control the installation process through events and stage flags.

Now we know how to:

boot the server via PXE;
pass control to our script;
identify the server that needs to be prepared by its serial number;
configure the server with the appropriate utilities;
pass settings to the install-server's database using the client component.

We figured out how:

the installing server gets the necessary settings from the database;
the entire preparation progress is recorded in the database (logs, events, stage flags).

What about different types of installable software? How to install a hypervisor, copy a VM, and set all this up?

In the case of deploying a file system image (Linux) to hardware, it’s quite straightforward:

After configuring all the server components, we deploy the image.
We install the grub bootloader.
We create a chroot and configure everything necessary.

How to hand over control to the OS installer (using ESXi as an example).

We organize the transfer of control from our script to the hypervisor installer via a response file (kickstart):
We delete the current partitions on the disk.
We create a partition of size 500MB.
We mark it as bootable.
We format it to FAT32.
We copy the installation files for ESXi to the root of it.
We install syslinux.
We copy syslinux.cfg to /syslinux/

default esxi
prompt 1
timeout 50
label esxi
kernel mboot.c32
append -c boot.cfg

We copy mboot.c32 to /syslinux.
In boot.cfg, it should be kernelopt=ks=ftp:///ks_esxi.cfg
Reboot the server.

After rebooting the server from its hard drive, the ESXi installer will load. All necessary installer files will be loaded into memory, and the installation of ESXi will begin according to the specified response file.

Here are a few lines from the response file ks_esxi.cfg:

%firstboot --interpreter=busybox
…
# Get the serial number

SYSSN=$(esxcli hardware platform get | grep Serial | awk -F " " '{print $3}')

# Get the IP address

IPADDRT=$(esxcli network ip interface ipv4 get | grep vmk0 | awk -F " " '{print $2}')
LAST_OCTET=$(echo $IPADDRT | awk -F'.' '{print $4}')

# Connect to the NFS install server

esxcli storage nfs add -H is -s \/srv\/nfs_share -v nfsshare1

# Copy temporary SSH settings for use with the SSH client

mv \/etc\/ssh \/etc\/ssh.tmp
cp -R \/vmfs\/volumes\/nfsshare1\/ssh \/etc\/\nchmod go-r \/etc\/ssh\/ssh_host_rsa_key

# Copy ovftool for deploying VMs now, plus it may be useful later

cp -R \/vmfs\/volumes\/nfsshare1\/ovftool \/vmfs\/volumes\/datastore1\/

# Deploy the VMs

\/vmfs\/volumes\/datastore1\/ovftool\/tools\/ovftool --acceptAllEulas --noSSLVerify --datastore=datastore1 --name=VM1 \/vmfs\/volumes\/nfsshare1\/VM_T\/VM1.ova vi:\/\/root:esxi_password@127.0.0.1
\/vmfs\/volumes\/datastore1\/ovftool\/tools\/ovftool --acceptAllEulas --noSSLVerify --datastore=datastore1 --name=VM2 \/vmfs\/volumes\/nfsshare1\/VM_T\/VM2.ova vi:\/\/root:esxi_password@127.0.0.1

# Get the line with our server's settings

ssh root@is "mysql -h'192.168.0.1' -D'servers' -u'user' -p'secretpassword' -e "SELECT ... WHERE servers.serial='$SYSSN'"" | grep -v ^$ | sed 's\/NULL\/\/g' > \/tmp\/servers
...
# Generate the network configuration script

echo '#!\/bin\/sh' > \/vmfs\/volumes\/datastore1\/netconf.sh
echo "esxcli network ip interface ipv4 set -i=vmk0 -t=static --ipv4=$IPADDR --netmask=$S_SUB || exit 1" >> \/vmfs\/volumes\/datastore1\/netconf.sh
echo "esxcli network ip route ipv4 add -g=$S_GW -n=default || exit 1" >> \/vmfs\/volumes\/datastore1\/netconf.sh
chmod a+x \/vmfs\/volumes\/datastore1\/netconf.sh

# Set the guestinfo.esxihost.id parameter with the serial number

echo "guestinfo.esxihost.id = "$SYSSN"" >> \/vmfs\/volumes\/datastore1\/VM1\/VM1.vmx
echo "guestinfo.esxihost.id = "$SYSSN"" >> \/vmfs\/volumes\/datastore1\/VM2\/VM2.vmx
...
# Update the information in the database

SYSNAME=$(esxcli hardware platform get | grep Product | sed 's\/Product Name:\/\/')
UUID=$(vim-cmd hostsvc\/hostsummary | grep uuid | sed 's\/ \/\/g;s\/$//;s\/uuid=\

At this stage, the hypervisor is installed and configured, and the virtual machines have been copied.

How to configure the virtual machines now?

We were a bit clever: during the installation, we set the parameter guestinfo.esxihost.id = "$SYSSN" in the VM1.vmx file, specifying the physical server's serial number.

Now, after starting, the virtual machine (with the VMware Tools package installed) can access this parameter:

ESXI_SN=$(vmtoolsd --cmd "info-get guestinfo.esxihost.id")

In other words, the VM will be able to identify itself (it knows the serial number of the physical host), make a request to the installation server database, and retrieve the parameters that need to be configured. This is all formalized in a script that should be automatically executed upon the startup of the guest OS VM (but only once: RunOnce).

Now we know how to:

boot the server via PXE;
pass control to our script;
identify the server that needs to be prepared by its serial number;
configure the server using the appropriate utilities;
transfer settings to the installation server database using the client-side;
configure various types of software, including deploying the ESXi hypervisor and setting up virtual machines (and all automatically).

We figured out how:

the installing server gets the necessary settings from the database;
the entire preparation progress is recorded in the database (logs, events, stage flags).

Conclusion:

I believe the uniqueness of this solution lies in its flexibility, simplicity, capabilities, and versatility.

Please write in the comments what you think.

Source: habr.com