Troubleshooting OS boot problems on non-KVM servers is a tricky business. We create ourselves a KVM-over-IP through a recovery image and a virtual machine.
In case of problems with the operating system , the administrator downloads the recovery image and performs the necessary work. This method works great when the cause of the failure is known, and the recovery image and the operating system installed on the server are from the same family. If the cause of the failure is not yet known, it is necessary to observe the progress of the operating system boot.
Remote KVM
You can access the server console using built-in tools such as IPMI or Intel® vPro™, or external devices called IP-KVM. There are situations in which all of these technologies are not available. However, this is not the end. If the server can be remotely rebooted into a recovery image based on a Linux-based operating system, then KVM-over-IP can be quickly set up.
The recovery image is a complete operating system that resides in RAM. Thus, we can run any software, including virtual machines (VMs). That is, you can run a VM inside which the server operating system will run. Access to the VM console can be organized, for example, via VNC.
To run the server operating system inside a VM, you must specify the server disks as VM disks. In operating systems of the Linux family, physical disks are represented by block devices of the form / Dev / sd, which you can work with like regular files.
Some hypervisors, such as QEMU and VirtualBox, allow you to store VM data in a "raw" form, that is, only storage data without hypervisor metadata. Thus, the VM can be started using the server's physical disks.
This method requires resources to run the recovery image and the VM inside it. However, with four or more gigabytes of RAM, this will not be a problem.
Preparing the Environment
You can use a lightweight and simple program as a virtual machine , which is most often not part of the recovery image and therefore must be installed separately. The recovery image we offer to customers is based on , which uses the package manager pacman.
First you need to make sure that the recovery image is using the latest software versions. You can check and update all OS components with the following command:
pacman -Suy
After the update, you need to install QEMU. The command to install via pacman will look like this:
pacman -S qemu
Let's check that qemu installed correctly:
root@sel-rescue ~ # qemu-system-x86_64 --version
QEMU emulator version 4.0.0
Copyright (c) 2003-2019 Fabrice Bellard and the QEMU Project developers
If so, then the recovery image is ready to go.
Starting the virtual machine
First, you need to determine the amount of resources allocated to the VM and find out the paths to the physical disks. In our case, we will allocate two cores and two gigabytes of RAM to the virtual machine, and the disks are located along the path / Dev / sda и / Dev / sdb. Let's start the VM:
qemu-system-x86_64
-m 2048M
-net nic -net user
-enable-kvm
-cpu host,nx
-M pc
-smp 2
-vga std
-drive file=/dev/sda,format=raw,index=0,media=disk
-drive file=/dev/sdb,format=raw,index=1,media=disk
-vnc :0,password
-monitor stdio
A little more about what each of the parameters means:
- -m 2048M - allocate 2 GB of RAM to the VM;
- -net nic -net user - add a simple network connection through a hypervisor using NAT (Network Address Translation);
- -enable-kvm - enable full virtualization of KVM (Kernel Virtual Machine);
- -cpu host - we tell the virtual processor to get all the functionality of the server processor;
- -M PC — type of PC equipment;
- -smp2 - the virtual processor must be dual-core;
- -vga std - select a standard video card that does not support large screen resolutions;
- -drive file=/dev/sda,format=raw,index=0,media=disk
- file=/dev/sdX — path to the block device representing the server disk;
- format=raw - we mark that in the specified file all the data is in a "raw" form, that is, as on a disk;
- index = 0 - disk number, should increase for each next disk by one;
- media=disk — the virtual machine must recognize this storage as a disk;
- -vnc :0, password - we start the default VNC server at 0.0.0.0:5900, we use the password as authorization;
- -monitor stdio - the administrator will communicate with qemu through standard input / output streams.
If everything is in order, then the QEMU monitor will start:
QEMU 4.0.0 monitor - type 'help' for more information
(qemu)
We indicated that authorization occurs with a password, but did not specify the password itself. This can be done by sending the change vnc password command to the QEMU monitor. Important note: the password cannot be more than eight characters.
(qemu) change vnc password
Password: ******
After that, we can connect with any VNC client, for example, Remmina, at the IP address of our server with the password we specified.
Now we not only see possible errors at the loading stage, but we can also deal with them.
Upon completion of the work, you must shut down the virtual machine. This can be done both inside the OS by giving a shutdown signal, or by giving the command system_powerdown in QEMU monitor. This will be equivalent to pressing the shutdown button once: the operating system inside the virtual machine will gracefully shut down.
Operating system installation
The virtual machine has full access to the server disks and therefore can be used to install the operating system manually. The only limitation is the amount of RAM: not always an ISO image can be placed in RAM. Let's allocate four gigabytes in RAM to store the image in / Mnt:
mount -t tmpfs -o size=4G tmpfs /mnt
We will also download the installation image of the FreeBSD 12.0 operating system:
wget -P /mnt ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/12.0/FreeBSD-12.0-RELEASE-amd64-bootonly.iso
Now you can start the VM:
qemu-system-x86_64
-m 2048M
-net nic -net user
-enable-kvm
-cpu host,nx
-M pc
-smp 2
-vga std
-drive file=/dev/sda,format=raw,index=0,media=disk
-drive file=/dev/sdb,format=raw,index=1,media=disk
-vnc :0,password
-monitor stdio
-cdrom /mnt/FreeBSD-12.0-RELEASE-amd64-bootonly.iso
-boot d
Flag -boot d sets to boot from the CD drive. We connect with a VNC client and see the FreeBSD bootloader.
Since obtaining an address via DHCP was used to access the Internet, it may be necessary to boot into a newly installed system after configuration and correct the network settings. In some cases, it may be necessary to install network adapter drivers, since the network card installed in the server and the one emulated in the VM are different.
Conclusion
This method of organizing remote access to the server console consumes part of the server resources, however, it does not impose any special requirements on the server hardware, and therefore can be performed in almost any conditions. The use of such a solution makes it much easier to diagnose software problems and restore the health of a remote server.
Source: habr.com