Virginia Polytechnic Institute for discussion among Linux kernel developers regarding a set of patches implementing a distributed thread execution system (Distributed Thread Execution), which allows applications to run on multiple computers with distribution and transparent migration of threads between hosts. Using Popcorn, applications can be started on one host and then moved to another host seamlessly without stopping their operation. In multithreaded programs, individual threads can be migrated to other hosts.
Unlike the , which allows saving a process's state and resuming execution on another system, Popcorn provides seamless and dynamic migration between hosts during application execution, requiring no action from the user and ensuring consistency of virtual memory across all hosts where threads are concurrently executed.
The Popcorn software stack includes to the Linux kernel and with tests demonstrating how to use Popcorn system calls for thread migration in distributed executing applications. At the kernel level, extensions to the virtual memory subsystem are proposed, implementing distributed shared memory, which allows processes on different hosts to access a common and consistent virtual address space. Consistency of virtual memory pages is ensured through a protocol that replicates memory pages on the host when they are accessed for reading and invalidates memory pages upon writing.
Interaction between hosts is performed using a kernel-level message handler that transmits messages via TCP sockets. It is noted that TCP/IP is used to simplify debugging and testing during development. Developers understand that from a security and performance perspective, TCP/IP is not the best way to transfer kernel structure content and memory pages between hosts. All hosts running distributed applications must have the same level of trust. After stabilizing the main algorithms, a more efficient transport method will be applied.
Popcorn has been developing since 2014 as a research project to explore the capabilities of creating distributed applications, where streams can operate on different nodes in computing systems, where cores based on various instruction set architectures (Xeon/Xeon-Phi, ARM/x86, CPU/GPU/FPGA) can be combined. The proposed set of patches for Linux kernel developers only supports execution on hosts with x86 CPU, but there is a more functional version called Popcorn Linux, which allows applications to run on hosts with different CPU architectures (x86 and ARM). To use Popcorn in heterogeneous environments, a special based on LLVM must be used. When conducting distributed execution on hosts with a single architecture, rebuilding with a separate compiler is not required.
Additionally, it is worth noting a project similar to with the implementation of an initial prototype API for launching child processes on other computers in the cluster (like fork(), but transfers the forked process to another computer).
The code is written in Rust and currently allows cloning only the simplest processes that do not use system resources such as files. When the telefork call is made, the memory and related process structures are cloned to another host where the server handler (telepad) runs. Using ptrace, the memory state of the process is serialized and, along with the process state and registers, is transferred to another host. The API also allows saving the process state to a file and restoring it through that.
Source: opennet.ru
