A detailed guide has been published on tuning the Linux environment to achieve maximum HTTP request processing performance. The proposed methods allowed the performance of the JSON handler based on the libreactor library in the Amazon EC2 (4 vCPU) environment to increase from 224,000 API requests per second at default settings of Amazon Linux 2 with kernel 4.14 to 1.2 million requests per second after optimization (an increase of 436%), and also reduced latency in request processing by 79%. The proposed methods are not specific to libreactor and work with other HTTP servers, including nginx, Actix, Netty, and Node.js (libreactor was used in tests as this solution showed the best performance).
Main optimizations:
- Optimization of libreactor code. The R18 version from the Techempower benchmark was used as a base, which was modified by removing code that limited the number of CPU cores utilized (optimization led to a speedup of 25-27%), compiling in GCC with options "-O3" (increase of 5-10%) and "-march-native" (5-10%), replacing read/write calls with recv/send (5-10%), and reducing overhead when using pthreads (2-3%). The overall performance increase after code optimization was 55%, and throughput increased from 224k req/s to 347k req/s.
- Disabling protections against vulnerabilities caused by speculative execution. Using boot parameters "nospectre_v1 nospectre_v2 pti=off mds=off tsx_async_abort=off" allowed performance to increase by 28%, and throughput rose from 347k req/s to 446k req/s. The individual contribution from the parameter "nospectre_v1" (protection against Spectre v1 + SWAPGS) was 1-2%, "nospectre_v2" (protection against Spectre v2) contributed 15-20%, "pti=off" (Spectre v3/Meltdown) provided a 6% increase, and "mds=off tsx_async_abort=off" (MDS/Zombieload and TSX Asynchronous Abort) also accounted for 6%. Settings for protection against L1TF/Foreshadow attacks (l1tf=flush), iTLB multihit, Speculative Store Bypass, and SRBDS were left unchanged as they did not affect performance by not intersecting with the tested configuration (e.g., specific to KVM, nested virtualization and other CPU models).
- Disabling audit and syscall blocking mechanisms using the command "auditctl -a never,task" and specifying the option "—security-opt seccomp=unconfined" when starting the Docker container. The overall performance increase was 11%, and throughput rose from 446k req/s to 495k req/s.
- Disabling iptables/netfilter by unloading related kernel modules. The idea to disable the firewall, which was unused in a specific server solution, was prompted by profiling results indicating that 18% of the time was spent on the nf_hook_slow function. It is noted that nftables operates more efficiently than iptables, but iptables continues to be used in Amazon Linux. After disabling iptables, performance increased by 22%, and throughput rose from 495k req/s to 603k req/s.
- Reducing handler migration between different CPU cores to enhance CPU cache utilization. Optimization was performed both at the level of process binding of libreactor to CPU cores (CPU Pinning) and through the binding of kernel network handlers (Receive Side Scaling). For example, irqbalance was disabled, and queue bindings were explicitly set to the CPU in /proc/irq/$IRQ/smp_affinity_list. A custom BPF handler was employed to ensure the same CPU core processed the libreactor process and the incoming packet network queue, connected via the SO_ATTACH_REUSEPORT_CBPF flag during socket creation. The settings for binding outgoing packet queues to CPU were modified in /sys/class/net/eth0/queues/tx-<n>/xps_cpus. Overall performance improvement was 38%, with throughput rising from 603k req/s to 834k req/s.
- Optimizing interrupt handling and using polling. Enabling adaptive-rx mode in the ENA driver and manipulating sysctl net.core.busy_read allowed for a 28% performance boost (throughput increased from 834k req/s to 1.06M req/s, and latencies decreased from 361μs to 292μs).
- Disabling system services that cause unnecessary locks in the network stack. Disabling dhclient and setting an IP address manually led to an increase in performance by 6%, with throughput rising from 1.06M req/s to 1.12M req/s. The reason for dhclient's impact on performance was analyzed through traffic analysis using a raw socket.
- Combating Spin Lock. Switching the network stack to 'noqueue' mode via sysctl 'net.core.default_qdisc=noqueue' and 'tc qdisc replace dev eth0 root mq' resulted in a performance gain of 2%, with throughput increasing from 1.12M req/s to 1.15M req/s.
- Final minor optimizations, such as disabling GRO (Generic Receive Offload) with the command 'ethtool -K eth0 gro off' and switching the congestion control algorithm from cubic to reno using sysctl 'net.ipv4.tcp_congestion_control=reno'. The overall performance gain was 4%. The throughput increased from 1.15M req/s to 1.2M req/s.
In addition to the successful optimizations, the article also discusses methods that did not lead to the expected performance growth. For instance, the following were ineffective:
- Running libreactor separately did not differ in performance from running it in a container. Replacing writev with send, increasing maxevents in epoll_wait, and experimenting with GCC versions and flags had no impact (the effect was noticeable only for flags '-O3' and '-march-native').
- Updating the Linux kernel to versions 4.19 and 5.4, using SCHED_FIFO and SCHED_RR schedulers, manipulating sysctl kernel.sched_min_granularity_ns, kernel.sched_wakeup_granularity_ns, transparent_hugepages=never, skew_tick=1, and clocksource=tsc did not affect performance.
- In the ENA driver, enabling Offload modes (segmentation, scatter-gather, rx/tx checksum), compiling with the flag '-O3', and applying parameters ena.rx_queue_size and ena.force_large_llq_header did not have an impact.
- Changes in the networking stack did not lead to performance improvement:
- Disabling IPv6: ipv6.disable=1
- Disabling VLAN: modprobe -rv 8021q
- Disabling packet source verification
- net.ipv4.conf.all.rp_filter=0
- net.ipv4.conf.eth0.rp_filter=0
- net.ipv4.conf.all.accept_local=1 (negative effect)
- net.ipv4.tcp_sack=0
- net.ipv4.tcp_dsack=0
- net.ipv4.tcp_mem/tcp_wmem/tcp_rmem
- net.core.netdev_budget
- net.core.dev_weight
- net.core.netdev_max_backlog
- net.ipv4.tcp_slow_start_after_idle=0
- net.ipv4.tcp_moderate_rcvbuf=0
- net.ipv4.tcp_timestamps=0
- net.ipv4.tcp_low_latency=1
- SO_PRIORITY
- A detailed guide has been published on tuning the Linux environment for maximum HTTP request processing performance.
Source: opennet.ru
