Virginia Tech for discussion by developers of the Linux kernel, a set of patches with the implementation of a system of distributed execution of threads (Distributed Thread Execution), which allows you to organize the execution of applications on several computers with the distribution and transparent migration of threads between hosts. With Popcorn, applications can be launched on one host and then moved to another host without interruption. In multithreaded programs, individual threads can be migrated to other hosts.
Unlike the project , which allows you to save process state and resume execution on a different system, Popcorn provides seamless and dynamic migration between hosts while an application is running, requiring no user action and ensuring virtual memory consistency across all hosts running concurrent threads.
Popcorn software stack form to the Linux kernel and with tests demonstrating how Popcorn system calls can be used to migrate threads in distributed applications. At the kernel level, extensions to the virtual memory subsystem are proposed with the implementation of distributed shared memory, which allows processes on different hosts to access a common and consistent virtual address space. Virtual memory page consistency is enforced by a protocol that replicates memory pages to the host when they are read and invalidates memory pages when written.
Interaction between hosts is carried out using a kernel-level message handler transmitted via a TCP socket. It is noted that TCP / IP is used to simplify debugging and testing during development. Developers understand that, from a security and performance point of view, TCP/IP is not the best way to transfer the contents of kernel structures and memory pages between hosts. All hosts running distributed applications must have the same level of trust. After the stabilization of the main algorithms, a more efficient mode of transport will be applied.
Popcorn has been developing since 2014 as a research project to explore the possibilities of creating distributed applications whose threads can run on different nodes in computing systems that can combine cores based on different instruction set architectures (Xeon/Xeon-Phi, ARM/x86, CPU/GPU/FPGA). The set of patches proposed by the Linux kernel developers only supports execution on hosts with an x86 CPU, but there is also a more functional Popcorn Linux variant that allows applications to run on hosts with different CPU architectures (x86 and ARM). To use Popcorn in heterogeneous environments, you need to use a special based on LLVM. For distributed execution on hosts with the same architecture, rebuilding by a separate compiler is not required.
In addition, it can be noted something like a project with the implementation of the initial API prototype for starting child processes on other computers in the cluster (like fork(), but transfers the forked process to another computer).
The code is written in Rust and so far only allows you to clone the most basic processes that do not use system resources such as files. When a telefork call is made, the memory and process-related structures are cloned to another host running a server handler (telepad). With ptrace, the reflection of a process's memory is serialized and transferred, along with the state of the process and registers, to another host. The API also allows you to save the state of the process to a file and restore through it.
Source: opennet.ru