![\"Linux](\"\/wp-content\/uploads\/2021\/05\/908fb8d92e84a3e7557af7f6c622298e.png\")
<\/center> <\/p>\nMain optimizations: <\/p>\n
- \n
- Optimization of the libreactor code. The R18 variant from the Techempower set was used as a basis, improved by eliminating code that limited the number of CPU cores used (this optimization allowed for a speed increase of 25-27%), compiling in GCC with the options '-O3' (a gain of 5-10%) and '-march-native' (5-10%), replacing read\/write calls with recv\/send (5-10%), and reducing overhead when using pthreads (2-3%). The overall performance increase after code optimization amounted to 55%, and throughput increased from 224k req\/s to 347k req\/s.\n
- Disabling protections against vulnerabilities caused by speculative instruction execution. Using the kernel boot parameters 'nospectre_v1 nospectre_v2 pti=off mds=off tsx_async_abort=off' allowed for a performance increase of 28%, and throughput rose from 347k req\/s to 446k req\/s. Individually, the increase from the 'nospectre_v1' parameter (protection against Spectre v1 + SWAPGS) was 1-2%, 'nospectre_v2' (protection against Spectre v2) was 15-20%, 'pti=off' (Spectre v3\/Meltdown) was 6%, and 'mds=off tsx_async_abort=off' (MDS\/Zombieload and TSX Asynchronous Abort) was 6%. Settings for protection against L1TF\/Foreshadow attacks (l1tf=flush), iTLB multihit, Speculative Store Bypass, and SRBDS were left unchanged, as they did not affect performance, since they did not overlap with the tested configuration (for example, being specific to... KVM<\/a>, nested virtualization and other CPU models).\n
- Disabling audit mechanisms and blocking system calls using the command 'auditctl -a never,task' and specifying the option '--security-opt seccomp=unconfined' when starting the docker container. The overall performance increase was 11%, and throughput increased from 446k req\/s to 495k req\/s.\n
- Disabling iptables\/netfilter by unloading related kernel modules. The idea to disable the firewall, which was unused in a specific server solution, was prompted by profiling results indicating that 18% of the time was spent on the nf_hook_slow function. It is noted that nftables operates more efficiently than iptables, but iptables continues to be used in Amazon Linux. After disabling iptables, performance increased by 22%, and throughput rose from 495k req\/s to 603k req\/s.\n
- Reducing handler migration between different CPU cores to enhance CPU cache utilization. Optimization was performed both at the level of process binding of libreactor to CPU cores (CPU Pinning) and through the binding of kernel network handlers (Receive Side Scaling). For example, irqbalance was disabled, and queue bindings were explicitly set to the CPU in \/proc\/irq\/$IRQ\/smp_affinity_list. A custom BPF handler was employed to ensure the same CPU core processed the libreactor process and the incoming packet network queue, connected via the SO_ATTACH_REUSEPORT_CBPF flag during socket creation. The settings for binding outgoing packet queues to CPU were modified in \/sys\/class\/net\/eth0\/queues\/tx-<n>\/xps_cpus. Overall performance improvement was 38%, with throughput rising from 603k req\/s to 834k req\/s.\n
- Optimization of interrupt handling and use of polling. Enabling adaptive-rx mode in the ENA driver and manipulating sysctl net.core.busy_read allowed for a performance increase of 28% (throughput rose from 834k req\/s to 1.06M req\/s, and latency decreased from 361\u00b5s to 292\u00b5s).\n
- Disabling system services that cause unnecessary locks in the network stack. Disabling dhclient and setting an IP address<\/a> manually led to an increase in performance by 6%, with throughput rising from 1.06M req\/s to 1.12M req\/s. The reason for dhclient's impact on performance was analyzed through traffic analysis using a raw socket.\n
- Combatting Spin Lock. Switching the network stack to 'noqueue' mode via sysctl 'net.core.default_qdisc=noqueue' and 'tc qdisc replace dev eth0 root mq' resulted in a performance increase of 2%, and throughput rose from 1.12M req\/s to 1.15M req\/s.\n
- Final minor optimizations, such as disabling GRO (Generic Receive Offload) with the command «ethtool -K eth0 gro off» and replacing the congestion control algorithm from cubic to reno using sysctl «net.ipv4.tcp_congestion_control=reno». The overall performance gain was 4%. Throughput increased from 1.15M req\/s to 1.2M req\/s. <\/ul>\n
In addition to the successful optimizations, the article also discusses methods that did not lead to the expected performance growth. For instance, the following were ineffective: <\/p>\n
- \n
- The standalone launch of libreactor showed no difference in performance compared to running in a container. The replacement of writev with send, increasing maxevents in epoll_wait, and experiments with different versions and GCC flags (the effect was noticeable only for the flags «-O3» and «-march-native»).\n
- Updating the Linux kernel to versions 4.19 and 5.4, using SCHED_FIFO and SCHED_RR schedulers, manipulating sysctl kernel.sched_min_granularity_ns, kernel.sched_wakeup_granularity_ns, transparent_hugepages=never, skew_tick=1, and clocksource=tsc did not affect performance.\n
- The inclusion of Offload modes (segmentation, scatter-gather, rx\/tx checksum) had no impact on the ENA driver, nor did building with the «-O3» flag and applying the parameters ena.rx_queue_size and ena.force_large_llq_header.\n
- Changes in the networking stack did not lead to performance improvement:\n