Odyssey roadmap: what else do we want from the connection pooler. Andrey Borodin (2019)

After two months of development, Linus Torvalds presented announced the release of the kernel Linux 5.9. Among the most notable changes are: limiting the import of symbols from proprietary modules into GPL modules, speeding up context switch operations using the FSGSBASE CPU instruction, support for kernel image compression using Zstd, a reworked priority scheduling for threads in the kernel, support for the PRP (Parallel Redundancy Protocol), bandwidth-aware scheduling in the deadline scheduler, preemptive memory page packing, the capability flag CAP_CHECKPOINT_RESTOR, the close_range() system call, improved performance of dm-crypt, removal of code for 32-bit guest systems of Xen PV, a new slab memory management mechanism, the 'rescue' option in Btrfs, and support for inline encryption in ext4 and F2FS.

The new version includes 16,074 fixes from 2,011 developers,
the patch size is 62 MB (the changes affected 14,548 files, added 782,155 lines of code, and removed 314,792 lines). About 45% of all changes in 5.9
are related to device drivers, approximately 15% of the changes pertain
to the update of architecture-specific code, 13%
is related to the network stack, 3% to file systems, and 3% to internal
kernel subsystems.

Key innovations:

• Memory and system services
  • The protection against the use of GPL wrappers to bind proprietary drivers to kernel components exported only for GPL-licensed modules has been tightened. The TAINT_PROPRIETARY_MODULE flag is now inherited in all modules that import symbols from modules with this flag. If a GPL module attempts to import symbols from a non-GPL module, that GPL module will inherit the TAINT_PROPRIETARY_MODULE label and will not be able to access kernel components available only to GPL-licensed modules, even if the module previously imported symbols from the 'gplonly' category. The reverse block (exporting only EXPORT_SYMBOL_GPL in modules that imported EXPORT_SYMBOL_GPL), which could disrupt the operation of proprietary drivers, is not implemented (only the proprietary module flag is inherited, but not GPL binding). support for the kcompactd mechanism for
  • Added preemptive memory page packing. preventive memory page packaging in the background, allowing an increase in the number of large memory pages available to the core. Preliminary estimates suggest that background packing with minimal overhead can reduce delays in allocating large memory pages (huge-page) by 70-80 times compared to the previously used on-demand packing mechanism. To set limits on the external fragmentation that kcompactd will ensure, the sysctl vm.compaction_proactiveness has been added.
  • Added support for kernel image compression using the algorithm Zstandard (zstd) algorithm has been added.
  • Support for the processor instruction FSGSBASE, allowing reading and modifying the contents of the FS/GS registers from user space. In the kernel, FSGSBASE is used to accelerate context-switching operations by eliminating unnecessary MSR write operations for GSBASE, and in user space, it allows avoiding unnecessary system calls to modify FS/GS.
  • Added the "allow_writes" parameter, which prohibits changes to CPU MSR registers from user space and limits access to the contents of these registers to read operations, as modifying MSR can lead to issues. By default, writing is not yet prohibited, and changes to MSR are logged, but it is planned to switch the default access mode to read-only in the future.
  • In the asynchronous input/output interface io_uring full support for asynchronous buffered read operations has been added, which do not require kernel threads. Write support is expected in the next release.
  • In the deadline I/O scheduler implemented bandwidth-aware scheduling, allowing correct decisions to be made on asymmetric systems, such as ARM-based systems with DynamIQ and big.LITTLE architecture, combining powerful and less powerful energy-efficient CPU cores on the same chip. In particular, the new mode allows avoiding mismatches during scheduling when a slow CPU core lacks sufficient resources to complete a task on time.
  • The energy consumption model in the kernel (Energy Model framework) now describes not only CPU power consumption behavior but also encompasses peripheral devices.
  • The system call close_range() has been implemented, allowing a process to close a range of open file descriptors at once.
  • From the implementation of the text console and the fbcon driver, the code has been removed, providing the ability for programmatic backward text scrolling (CONFIG_VGACON_SOFT_SCROLLBACK) beyond the volume of video memory in text mode VGA.
  • Reworked the algorithm for priority assignment for threads within the kernel. The new version ensures better consistency across all kernel subsystems when assigning priorities to real-time tasks.
  • Added sysctl sched_uclamp_util_min_rt_default to manage CPU frequency forcing parameters for real-time tasks (for example, it allows dynamic changes in behavior when executing real-time tasks to save energy when switching to battery power or in mobile systems).
  • Preparation has been conducted for the implementation of support for Transparent Huge Pages technology in the page cache.
  • New flags FAN_REPORT_NAME and FAN_REPORT_DIR_FID have been implemented in the fanotify mechanism to transmit information about the parent name and unique identifier FID when events occur for creation, deletion, or movement of directory elements and objects not related to directories.
  • For cgroups, is implemented a new slab memory controller, which is notable for transferring slab accounting from the memory page level to the kernel object level, enabling shared use of slab pages across different cgroups instead of allocating separate slab caches for each cgroup. The proposed approach improves the efficiency of slab usage, reduces the size of memory used for slab by 30-45%, significantly decreases the overall memory consumption by the kernel, and lowers memory fragmentation.
  • In the sound subsystem, ALSA, and the USB stack,in accordance with recently accepted the guidelines for using inclusive terminology in the Linux kernel, a cleanup of politically incorrect terms has been conducted. The code has been cleared of the use of the words 'slave', 'master', 'blacklist', and 'whitelist'.
• Virtualization and Security
  • When building the kernel using the Clang compiler, a new feature has emerged the configuration option (CONFIG_INIT_STACK_ALL_ZERO) allows for automatic initialization to zero for all variables stored on the stack (when building, '-ftrivial-auto-var-init=zero' is specified).
  • In the seccomp subsystem, when using the process control mode in user space, has added of Substitution in the monitored process of file descriptors to fully emulate system calls that lead to the creation of file descriptors. This functionality is in demand in isolated container systems and sandbox implementations for Chrome.
  • Support for restricting system calls using the seccomp subsystem has been added for xtensa and csky architectures. Additional support for the auditing mechanism has been implemented for xtensa.
  • Added A new capability flag CAP_CHECKPOINT_RESTORE has been introduced, allowing access to features related to freezing and restoring process states without granting additional privileges.
  • GCC 11 has implemented all the necessary features for
    the KCSAN (Kernel Concurrency Sanitizer) debugging tool, designed for dynamically detecting race conditions within the kernel. Thus, KCSAN can now be used with kernels built with GCC.
  • For AMD Zen and newer CPU models has added support for the P2PDMA technology has been added, allowing DMA for direct data transfer between the memory of two devices connected to the PCI bus.
  • In dm-crypt, a mode has been added that reduces latency by performing cryptographic data processing without using work queues. This mode is also necessary for proper operation with zoned block devices (devices with zones that must be written sequentially, updating the entire group of blocks). Efforts have been made to increase throughput and reduce latency in dm-crypt.
  • Code for supporting 32-bit guest systems running in paravirtualization mode under the Xen hypervisor has been removed. Users of such systems should switch to using 64-bit kernels in guest environments or use full (HVM) or hybrid (PVH) virtualization modes instead of paravirtualization (PV).
• Disk subsystem, input/output, and file systems
  • In the Btrfs file system, within the 'rescue' mount option has been standardized to unify access to all other recovery options. The 'alloc_start' and 'subvolrootid' options have been removed, and the 'inode_cache' option has been deprecated. Performance optimizations have been made, significantly speeding up fsync() operations. Added the ability to use alternative checksum types other than CRC32c.
  • Added the possibility of using inline encryption in ext4 and F2FS file systems, with a mounting option "inlinecrypt". The inline encryption mode allows utilizing the encryption mechanisms built into the storage controller, which perform transparent encryption and decryption of input/output.
  • In XFS ensured inode flushing in fully asynchronous mode, not blocking processes during memory cleanup operations. A long-standing issue with quotas has been resolved, which incorrectly tracked warnings about exceeding soft limits and inode count restrictions. Unified support for DAX for ext4 and xfs has been implemented.
  • In Ext4 within lazy loading of block allocation bitmaps. Combined with the limitation on scanning uninitialized groups, the optimization has reduced the mounting time of very large partitions.
  • In F2FS implemented ioctl F2FS_IOC_SEC_TRIM_FILE, allowing the use of TRIM/discard commands for physically zeroing out specified data in a file, such as removing access keys without leaving residual data on the storage.
    In F2FS, there is also implemented a new garbage collection mode GC_URGENT_LOW, working more aggressively by skipping certain idle state checks before starting the garbage collector.
  • In bcache, the bucket_size for extents has been increased from 16 to 32 bits in preparation for enabling support for zoned device caching.
  • In the SCSI subsystem, the capability of using inline encryption based on built-in hardware encryption provided by UFS controllers has been added (Universal Flash Storage).
  • A new kernel command line parameter "debugfs" has been added, allowing control over the availability of the eponymous pseudo-file system.
  • In the NFSv4.2 client, support for extended file attributes (xattr) has been ensured.
  • In dm-dust implemented an interface for outputting the list of all detected bad blocks on the disk at once ("dmsetup message dust1 0 listbadblocks").
  • For md/raid5, a parameter /sys/block/md1/md/stripe_size has been added for configuring the STRIPE block size.
  • For NVMe storage devices has added support for zoning commands for the storage device (ZNS, NVM Express Zoned Namespace), enabling the division of storage space into zones that comprise groups of blocks for better management of data placement on the storage.
• Network subsystem
  • In Netfilter has added the ability to drop packets before route checking (the REJECT statement can now be used not only in INPUT, FORWARD, and OUTPUT chains but also at the PREROUTING stage for icmp and tcp).
  • In nftables has added the ability to audit events related to configuration changes.
  • In nftables in the netlink API has added support for anonymous chains, the names of which are dynamically assigned by the kernel. When the rule associated with an anonymous chain is removed, the chain itself is automatically deleted.
  • Support for iterators has been added to BPF for traversing, filtering, and modifying elements of associative arrays (maps) without copying data to user space. Iterators can be used for TCP and UDP sockets, allowing BPF programs to iterate over lists of open sockets and extract the necessary information from them.
  • A new type of BPF program, BPF_PROG_TYPE_SK_LOOKUP, has been added, which runs when the kernel looks for a suitable listening socket for an incoming connection. Using such a BPF program, one can create handlers that decide which socket to associate with the connection, unrestricted by the bounds of the bind() system call. For instance, one can bind a single socket to a range of addresses or ports. Additionally, support for the SO_KEEPALIVE flag has been added in bpf_setsockopt(), and support for BPF_CGROUP_INET_SOCK_RELEASE handlers has been implemented, which are called when a socket is released.
  • Support for the protocol PRP (Parallel Redundancy Protocol), which allows for transparent switching to a backup channel in case of failure of any network components based on Ethernet.
  • In the mac80211 stack has added support for four-way WPA/WPA2-PSK channel negotiation in access point mode.
  • The ability to switch the qdisc (queuing discipline) scheduler to use the FQ-PIE (Flow Queue PIE) algorithm by default has been added, aimed at reducing the negative impact of intermediate packet buffering on border network devices (bufferbloat) in cable modem networks.
  • New capabilities have been added to MPTCP (MultiPath TCP), a TCP protocol extension for managing TCP connections with packet delivery over multiple routes through different network interfaces associated with different IP addresses. Support for syn cookie, DATA_FIN, automatic buffer tuning, socket diagnostics, and the use of the REUSEADDR, REUSEPORT, and V6ONLY flags in setsockopt has been added.
  • For virtual routing tables VRF (Virtual Routing and Forwarding), which allow the implementation of multiple routing domains on a single system, a 'strict' mode has been implemented. In this mode, a virtual table can only be associated with a routing table that is not used in other virtual tables.
  • In the wireless driver ath11k has added support for the 6GHz frequency and spectral scanning.
• Hardware
  • Support code for the UniCore architecture, developed at the microprocessor center of Peking University and included in the Linux kernel in 2011, has been removed. This architecture has been unsupported since 2014 and lacks support in GCC.
  • Support for the RISC-V architecture has been implemented kcov (debugfs interface for analyzing kernel code coverage), kmemleak (memory leak detection system), stack protection, jump labels, and tickless operations (signal-independent timer multitasking).
  • Support for queues for spinlock has been implemented for the PowerPC architecture, significantly improving performance in lock conflict situations.
  • For ARM and ARM64 architectures, the schedutil (cpufreq governor) mechanism is enabled by default, which uses information directly from the scheduler to make decisions about frequency changes and can immediately interact with cpufreq drivers to dynamically adjust the frequency, swiftly tuning CPU parameters to current load conditions.
  • In the DRM driver i915 for Intel graphics cards, support for chips based on the Rocket Lake microarchitecture has been included, and initial support for discrete cards Intel Xe DG1.
  • has been added in the amdgpu driver with initial support for AMD GPU Navi 21 (Navy Flounder) and Navi 22 (Sienna Cichlid). Support for UVD/VCE video encoding and decoding acceleration engines for Southern Islands GPUs (Radeon HD 7000) has been added.
    A property has been added for rotating the display by 90, 180, or 270 degrees.

    Interestingly, the driver for AMD GPUs is the largest driver in the kernel — it accounts for about 2.71 million lines of code, which is approximately 10% of the total kernel size (27.81 million lines). Of this, 1.79 million lines are automatically generated header files with data for GPU registers, while the C code constitutes 366 thousand lines (for comparison, the Intel i915 driver includes 209 thousand lines, and Nouveau — 149 thousand).

  • In the Nouveau driver has added support for frame-by-frame integrity checking using CRC (Cyclic Redundancy Checks) in the display engines of NVIDIA GPUs. The implementation is based on documentation provided by NVIDIA.
  • Drivers for LCD panels have been added: Frida FRD350H54004, KOE TX26D202VM0BWA, CDTech S070PWS19HP-FC21, CDTech S070SWV29HG-DC44, Tianma TM070JVHG33, and Xingbangda XBD599.
  • In the ALSA sound subsystem, support for Intel Silent Stream (a mode for maintaining constant power for external HDMI devices to eliminate latency when playback starts) and a new device for controlling the backlighting of the microphone activation and mute buttons has been implemented, as well as support for new hardware, including the controller Loongson 7A1000.
  • Support for ARM boards, devices, and platforms has been added: Pine64 PinePhone v1.2, Lenovo IdeaPad Duet 10.1, ASUS Google Nexus 7, Acer Iconia Tab A500, Qualcomm Snapdragon SDM630 (used in Sony Xperia 10, 10 Plus, XA2, XA2 Plus, and XA2 Ultra), Jetson Xavier NX, Amlogic WeTek Core2, Aspeed EthanolX, five new boards based on NXP i.MX6, MikroTik RouterBoard 3011, Xiaomi Libra, Microsoft Lumia 950, Sony Xperia Z5, MStar, Microchip Sparx5, Intel Keem Bay, Amazon Alpine v3, Renesas RZ/G2H.

At the same time, the Latin American Free Software Foundation has formed
variant fully free kernel 5.9 — Linux-libre 5.9-gnu, cleaned of firmware and driver elements containing non-free components or code segments whose scope is limited by the manufacturer. In this release, blob loading has been disabled in the drivers for WiFi rtw8821c and MediaTek mt8183 SoC. The code for cleaning blobs in the drivers and subsystems Habanalabs, Wilc1000, amdgpu, mt7615, i915 CSR, Mellanox mlxsw (Spectrum3), r8169 (rtl8125b-2), and x86 touchscreen has been updated.

Source: opennet.ru