The release of the Cosmopolitan 2.0 project has been published, which develops the standard C library and a universal executable file format that can be used to distribute software for different operating systems without the need for interpreters or virtual machines. The output generated through compilation in GCC and Clang is linked into a statically bindable universal executable file, which is suitable for running on any distribution of Linux, macOS, Windows, FreeBSD, OpenBSD, NetBSD, and can even be invoked from BIOS. The project's code is distributed under the ISC license (a simplified version of MIT/BSD).
The container for creating universal executable files is based on combining OS-specific segments and headers (PE, ELF, MACHO, OPENBSD) into a single file, merging several different formats used in Unix, Windows, and macOS. To ensure that a single executable file can run in both Windows and Unix systems, a trick is used that encodes Windows PE files as a shell script, taking advantage of the fact that the Thompson Shell does not use the script marker '#!'. For creating programs that include multiple files (linking all resources into one file), the generation of an executable file in the form of a specially formatted ZIP archive is supported. The proposed format schema (example application hello.com):
MZqFpD=’ BIOS BOOT SECTOR’ exec 7 $(command -v $0) printf ‘\177ELF…LINKER-ENCODED-FREEBSD-HEADER’ > &7 exec "$0" "$@" exec qemu-x86_64 "$0" "$@" exit 1 REAL MODE… ELF SEGMENTS… OPENBSD NOTE… MACHO HEADERS… CODE AND DATA… ZIP DIRECTORY…
At the beginning of the file, the label 'MZqFpD' is specified, which is interpreted as the header of the Windows PE format. This sequence is also decoded into the instruction 'pop %r10 ; jno 0x4a ; jo 0x4a', while the string '\177ELF' corresponds to the instruction 'jg 0x47', which are used to redirect to the entry point. In Unix systems, shell code is executed, which uses the exec command to pass the executable code through an unnamed pipe. A limitation of the proposed method is that it can only run in Unix-like OSes when using shells that support compatibility mode with the Thompson Shell.
The qemu-x86_64 call is intended for additional portability and allows execution of code compiled for the x86_64 architecture on platforms other than x86, such as Raspberry Pi boards and Apple devices equipped with ARM processors. The project can also be used to create standalone applications that run without an operating system (bare metal). In such applications, a bootloader is attached to the executable file, and the program acts as the bootable operating system.
The standard C library libc, developed by the project, offers 2024 functions (compared to around 1400 in the first release). In terms of performance, Cosmopolitan runs as fast as glibc and significantly outperforms Musl and Newlib, while being an order of magnitude smaller in code size compared to glibc and approximately comparable to Musl and Newlib. To optimize frequently called functions such as memcpy and strlen, a 'trickle-down performance' technique is additionally used, where a macro wrapper informs the compiler about the CPU registers involved during code execution, allowing resource savings by preserving only the modified registers.
Among the changes in the new release:
- The scheme for accessing internal resources within a zip file has been changed (ordinary paths /zip/... are now used instead of the zip:.. prefix when opening files). Similarly, for disk access in Windows, the option to use paths like '/c/...' has been provided instead of 'C:/...'.
- A new loader, APE (Actually Portable Executable), has been proposed, defining a format for universal executable files. The new loader uses mmap to map the program into memory and no longer modifies contents on the fly. If necessary, the universal executable can be converted into standard executables bound to specific platforms.
- On the Linux platform, the ability to use the binfmt_misc kernel module to run APE programs has been implemented. It is noted that using binfmt_misc is the fastest method for starting.
- For Linux, an implementation of the pledge() and unveil() system call functionalities has been proposed, developed by the OpenBSD project. An API is provided for utilizing these calls in programs written in C, C++, Python, and Redbean, as well as a pledge.com utility for isolating arbitrary processes.
- The Landlock Make utility is used for building — a version of GNU Make with stricter dependency checking and the use of the Landlock system call to isolate the program from the rest of the system and enhance caching efficiency. The option for building with standard GNU Make has been retained.
- Functions for multithreading — _spawn() and _join() — have been implemented, providing universal wrappers over APIs specific to different operating systems. Work is also underway to implement support for POSIX Threads.
- The ability to use the _Thread_local keyword has been introduced for using separate thread-local storage (TLS, Thread-Local Storage) for each thread. By default, the C runtime initializes TLS for the main thread, leading to an increase in the minimum executable file size from 12 to 16 KB.
- Executable files have been enhanced with support for the ‘—ftrace’ and ‘—strace’ parameters to output information about all function calls and system calls to stderr.
- Support for the closefrom() system call has been added, which is supported in Linux 5.9+, FreeBSD 8+, and OpenBSD.
- On the Linux platform, the performance of clock_gettime and gettimeofday calls has been increased up to 10 times due to the use of the vDSO (virtual dynamic shared object) mechanism, which allows transferring the system call handler to user space and avoiding context switches.
- Mathematical functions for working with complex numbers have been moved from the Musl library. The performance of many mathematical functions has been accelerated.
- A nointernet() function has been proposed to disable network capabilities.
- New functions for efficient string appending have been added: appendd, appendf, appendr, appends, appendw, appendz, kappendf, kvappendf, and vappendf.
- A secure variant of the kprintf() function family has been added, designed for use under elevated privileges.
- The performance of implementations has been significantly increased SSL, SHA, curve25519, and RSA.
Source: opennet.ru
