Haɓaka Linux don ɗaukar buƙatun JSON miliyan 1.2 a sakan daya

An buga cikakken jagora akan daidaita yanayin Linux don cimma iyakar aiki don sarrafa buƙatun HTTP. Hanyoyin da aka tsara sun ba da damar haɓaka aikin na'ura na JSON bisa ga ɗakin karatu na masu kyauta a cikin yanayin Amazon EC2 (4 vCPU) daga buƙatun API 224 dubu a sakan daya tare da daidaitattun saitunan Amazon Linux 2 tare da kernel 4.14 zuwa 1.2 miliyan buƙatun kowane ɗayan. na biyu bayan ingantawa (ƙararar 436%), kuma ya haifar da raguwar jinkiri a cikin buƙatun sarrafawa da kashi 79%. Hanyoyin da aka tsara ba su da ƙayyadaddun masu kyauta da aiki lokacin amfani da wasu sabar http, ciki har da nginx, Actix, Netty da Node.js (an yi amfani da libreactor a cikin gwaje-gwaje saboda maganin da aka dogara da shi ya nuna kyakkyawan aiki).

Abubuwan ingantawa na asali:

• Ƙaddamar da lambar kyauta. An yi amfani da zaɓi na R18 daga kayan aikin Techempower a matsayin tushe, wanda aka inganta ta hanyar cire lambar don iyakance adadin adadin CPU da aka yi amfani da shi (ingantawa ya ba da damar saurin aiki ta hanyar 25-27%), haɗuwa a cikin GCC tare da zaɓuɓɓukan "-O3". (ƙararar 5-10%) da "-march-native" (5-10%), maye gurbin karantawa/rubutu kira tare da recv/aika (5-10%) da rage sama lokacin amfani da pthreads (2-3%) . Haɓaka aikin gabaɗaya bayan haɓaka lambar ya kasance 55%, kuma kayan aikin ya ƙaru daga 224k req/s zuwa 347k req/s.
• Kashe kariya daga rashin lahani na kisa. Yin amfani da sigogi "nospectre_v1 nospectre_v2 pti = kashe mds = kashe tsx_async_abort = kashe" lokacin loda kernel da aka ba da izinin haɓaka aiki da 28%, kuma kayan aiki ya ƙaru daga 347k req/s zuwa 446k req/s. Na dabam, haɓaka daga siga "nospectre_v1" (kariya daga Specter v1 + SWAPGS) ya kasance 1-2%, "nospectre_v2" (kariya daga Specter v2) - 15-20%, "pti = kashe" (Specter v3 / Meltdown) - 6 %, "mds = kashe tsx_async_abort = kashe" (MDS/Zombieload da TSX Asynchronous Abort) - 6%. Saituna don kariya daga L1TF / Foreshadow (l1tf = flush), iTLB multihit, Speculative Store Bypass da hare-haren SRBDS ba a canza su ba, wanda bai shafi aikin ba tun da ba su shiga tsakani da tsarin da aka gwada ba (misali, musamman ga KVM, gida). Virtualization da sauran samfuran CPU).
• Kashe hanyoyin tantancewa da tsarin kiran tsarin kira ta amfani da umarnin "auditctl -a never,ask" da kuma ƙayyade zaɓin "-security-opt seccomp=unconfined" lokacin fara kwandon docker. Yawan karuwar aikin gabaɗaya shine 11%, kuma kayan aikin ya ƙaru daga 446k req/s zuwa 495k req/s.
• Kashe iptables/netfilter ta sauke kayan aikin kwaya mai alaƙa. Tunanin kashe Tacewar zaɓi, wanda ba a yi amfani da shi a cikin takamaiman bayani na uwar garken ba, ya samo asali ne ta hanyar bayyana sakamakon, yin hukunci wanda aikin nf_hook_slow ya ɗauki 18% na lokacin aiwatarwa. An lura cewa nftables yana aiki da inganci fiye da iptables, amma Amazon Linux ya ci gaba da amfani da iptables. Bayan kashe iptables, haɓakar aikin shine 22%, kuma kayan aikin ya ƙaru daga 495k req/s zuwa 603k req/s.
• Rage ƙaura na masu sarrafawa tsakanin nau'ikan nau'ikan CPU daban-daban don haɓaka ingantaccen amfani da cache mai sarrafawa. An gudanar da ingantawa duka biyu a matakin ɗaure hanyoyin samar da albarkatu zuwa CPU cores (CPU Pinning) da kuma ta hanyar masu sarrafa hanyar sadarwar kwaya (Karɓa Side Scaling). Misali, an kashe irqbalance kuma an saita layin layi ga CPU a fili a /proc/irq/$IRQ/smp_affinity_list. Don amfani da ainihin CPU iri ɗaya don aiwatar da tsarin ma'auni da layin hanyar sadarwa na fakiti masu shigowa, ana amfani da mai sarrafa BPF na al'ada, an haɗa shi ta saita tutar SO_ATTACH_REUSEPORT_CBPF lokacin ƙirƙirar soket. Don ɗaure layi na fakiti masu fita zuwa CPU, an canza saitunan /sys/class/net/eth0/queues/tx- /xps_cpus. Haɓaka aikin gabaɗaya shine 38%, kuma kayan aikin ya ƙaru daga 603k req/s zuwa 834k req/s.
• Haɓaka sarrafa katsewa da amfani da jefa ƙuri'a. Bayar da yanayin daidaitawa-rx a cikin direban ENA da sarrafa sysctl net.core.busy_read ya karu da 28% (sarin ya karu daga 834k req/s zuwa 1.06M req/s, kuma latency ya ragu daga 361μs zuwa 292μs).
• Kashe ayyukan tsarin da ke haifar da toshewar da ba dole ba a cikin tarin cibiyar sadarwa. Kashe dhclient da saita adireshin IP da hannu ya haifar da haɓaka aikin 6% da haɓaka kayan aiki daga 1.06M req/s zuwa 1.12M req/s. Dalilin dhclient yana rinjayar aiki yana cikin nazarin zirga-zirga ta amfani da danyen soket.
• Yaƙi Spin Lock. Canja wurin tari na cibiyar sadarwa zuwa yanayin "noqueue" ta hanyar sysctl "net.core.default_qdisc=noqueue" da "tc qdisc maye gurbin dev eth0 tushen mq" ya haifar da karuwar aikin 2%, kuma kayan aiki ya karu daga 1.12M req/s zuwa 1.15M req/s.
• Ƙarshen ingantawa na ƙarshe, kamar kashe GRO (Generic Receive Offload) tare da umarnin "ethtool -K eth0 gro off" da maye gurbin algorithm mai sarrafa cunkoso tare da reno ta amfani da sysctl "net.ipv4.tcp_congestion_control=reno". Yawan karuwar yawan aiki ya kasance 4%. Abubuwan da aka samar sun karu daga 1.15M req/s zuwa 1.2M req/s.

Baya ga ingantawa da suka yi aiki, labarin ya kuma tattauna hanyoyin da ba su haifar da karuwar aikin da ake tsammani ba. Misali, abubuwan da suka biyo baya sun zama marasa tasiri:

• Gudun liberactor daban bai bambanta ba a cikin aikin da gudanar da shi a cikin akwati. Maye gurbin rubutav tare da aikawa, haɓaka maxevents a epoll_wait, da gwaji tare da sigogin GCC da tutoci ba su da wani tasiri (tasirin ya kasance sananne ne kawai ga tutocin “-O3” da “-march-native”).
• Haɓaka kwaya ta Linux zuwa nau'ikan 4.19 da 5.4, ta amfani da masu tsara tsarin SCHED_FIFO da SCHED_RR, sarrafa sysctl kernel.sched_min_granularity_ns, kernel.sched_wakeup_granularity_ns, transparent_hugepages=skew=skew=skew=skew=skew=skew.
• A cikin direban ENA, kunna yanayin Offload (bangare, watsawa-gather, rx/tx checksum), gini tare da tutar “-O3”, da kuma amfani da ma'aunin ena.rx_queue_size da ena.force_large_llq_header ba su da wani tasiri.
• Canje-canje a cikin tarin cibiyar sadarwa bai inganta aiki ba:
  • Kashe IPv6: ipv6.disable=1
  • Kashe VLAN: modprobe -rv 8021q
  • Kashe binciken tushen fakitin
    • net.ipv4.conf.all.rp_filter=0
    • net.ipv4.conf.eth0.rp_filter=0
    • net.ipv4.conf.all.accept_local=1 (tasiri mara kyau)
  • net.ipv4.tcp_sack = 0
  • net.ipv4.tcp_dsack=0
  • net.ipv4.tcp_mem/tcp_wmem/tcp_rmem
  • net.core.netdev_budget
  • net.core.dev_weight
  • net.core.netdev_max_backlog
  • net.ipv4.tcp_slow_start_after_idle=0
  • net.ipv4.tcp_moderate_rcvbuf=0
  • net.ipv4.tcp_timestamps=0
  • net.ipv4.tcp_low_latency = 1
  • SO_PRIORITY
  • TCP_NODELAY

  source: budenet.ru