kmorph has been introduced for quick recovery after a Linux kernel crash or update

The developers of the Multikernel project, which allows multiple independent instances of the kernel to run on a single physical computer without hypervisors and virtualization, have introduced the kmorph toolkit, which implements tools for rapid recovery from kernel crashes in Linux or for replacing the kernel in Live mode without a full reboot. The project is led by Cong Wang, who maintains the Traffic Control (TC) subsystem in the kernel. The rapid restart method is based on preloading a backup kernel into memory, which replaces the primary working kernel in the event of a failure or when it is necessary to switch to a new kernel version.

The proposed approach allows for operation without a complete reboot and almost instantly resumes functionality, minimizing downtime. Currently, in the event of a kernel crash, a full reboot of large systems can take several minutes due to lengthy hardware initialization processes, loading the system environment, and saving the kernel memory dump via the kdump mechanism. servers Essentially, with kdump, two kernel reboots occur — initially, a separate kernel instance is loaded through kexec to save the memory dump, after which the primary kernel is loaded and system services are started.

When using kmorph in conjunction with a specific CPU kernel, a backup Linux kernel and minimal system environment are preloaded into memory, which monitors the state of the primary kernel. In the event of a hang or a panic state of the primary kernel, the backup environment takes over, saves the memory dump of the failing kernel, and switches control of the CPU cores, memory, devices, network configuration, and filesystem.

Switching is coordinated using two programs: kmorph — which runs in the active environment, prepares the backup environment, and synchronizes the network state; kmorphd — which runs in the backup environment and monitors the status of the active environment. Besides switching in case of a failure, kmorph can also be used to update the kernel without traditional rebooting. To speed up recovery in the backup environment, along with the kernel and a minimal set of system components, standard services like nginx can be launched in standby mode. In this case, when the backup environment is activated, the nginx service will already be loaded into memory and will immediately begin processing requests after switching to the new network interface environment, activating the network configuration, and changing the root filesystem.

In the conducted tests, a server with nginx using kmorph resumes operation approximately 2 seconds after a kernel crash, but in the future, there are plans to achieve instant switching unnoticed by connected clients. Currently, kmorph supports x86_64 and ARM64 architectures and work is underway to support RISC-V. Integration of kmorph into Linux distributions is done similarly to kdump support, using standard tools like dracut, initramfs-tools, kernel-install, and the systemd service.

The kernel already includes Kexec HandOver (KHO) and Live Update Orchestrator (LUO) mechanisms, allowing the launch of a new kernel from the old one without losing system state and without stopping device operations. However, they cannot be used in case of a kernel failure, as they rely on saving the state of the running kernel. KHO and LUO are also slower than kmorph in switching speeds because kmorph has the new kernel already in memory, while KHO and LUO require a kernel reboot through kexec.

Source: opennet.ru