Linux optimization yekubata 1.2 miriyoni JSON zvikumbiro pasekondi

Nhungamiro yakadzama yakaburitswa pakugadzirisa nharaunda yeLinux kuti iwane yakanyanya kuita yekugadzirisa zvikumbiro zveHTTP. Nzira dzakarongwa dzakaita kuti zvikwanisike kuwedzera kushanda kweJSON processor zvichibva paraibhurari yemahara muAmazon EC2 nharaunda (4 vCPU) kubva ku224 zviuru zvikumbiro zveAPI pasekondi neyakajairwa marongero eAmazon Linux 2 ine kernel 4.14 kusvika 1.2 miriyoni zvikumbiro pa chechipiri mushure mekugadzirisa (kuwedzera kwe436%), uye zvakakonzerawo kuderedzwa kwekunonoka mukugadzirisa zvikumbiro ne79%. Nzira dzakarongwa hadzisi dzakananga kune libreactor uye kushanda kana uchishandisa mamwe ma-http maseva, kusanganisira nginx, Actix, Netty uye Node.js (libreactor yakashandiswa mukuedzwa nekuti mhinduro yakavakirwa pairi yakaratidza kuita kuri nani).

Basic optimizations:

• Kugadzirisa libreactor kodhi. Iyo R18 sarudzo kubva kuTechempower kit yakashandiswa sehwaro, iyo yakagadziridzwa nekubvisa kodhi kudzikamisa nhamba yeCPU cores yakashandiswa (optimization inobvumira kukurumidza kukurumidza basa ne25-27%), kuungana muGCC ne "-O3" sarudzo. (kuwedzera kwe5-10%) uye "-march-native" (5-10%), kutsiva kuverenga/kunyora mafoni nerecv/send (5-10%) uye kuderedza pamusoro kana uchishandisa pthreads (2-3%). . Kuwedzera kwekuita kwese mushure mekugadzirisa kodhi yaive 55%, uye kubuda kwakawedzera kubva pa224k req/s kusvika 347k req/s.
• Dzivirira dziviriro kubva panjodzi yekufungidzira. Kushandisa parameters "nospectre_v1 nospectre_v2 pti=off mds=off tsx_async_abort=off" pakuisa kernel inobvumirwa kuwedzera kuita ne28%, uye kubuda kwakawedzera kubva pa347k req/s kusvika 446k req/s. Kusiyana, kuwedzera kubva paparameter "nospectre_v1" (kudzivirirwa kubva kuSpecter v1 + SWAPGS) yaiva 1-2%, "nospectre_v2" (kudzivirirwa kubva kuSpecter v2) - 15-20%, "pti = off" (Specter v3 / Meltdown) - 6 %, "mds=off tsx_async_abort=off" (MDS/Zombieload uye TSX Asynchronous Abort) - 6%. Izvo zvigadziriso zvekudzivirira kubva kuL1TF/Foreshadow (l1tf=flush), iTLB multihit, Speculative Store Bypass uye SRBDS kurwiswa zvakasiiwa zvisina kuchinjwa, izvo zvisina kukanganisa kuita sezvo vasina kupindirana neyakaedzwa kumisikidzwa (semuenzaniso, yakanangana neKVM, yakagara. virtualization uye mamwe maCPU modhi).
• Kudzima maodhita uye system yekuvharisa kufona nzira uchishandisa iyo "auditctl -a never,task" kuraira uye kutsanangura iyo "-security-opt seccomp=unconfined" sarudzo paunotanga docker mudziyo. Kuwedzera kwekuita kwese kwaive 11%, uye kubuda kwakawedzera kubva pa446k req/s kuenda ku495k req/s.
• Kudzima iptables/netfilter nekuburitsa yakabatana kernel modules. Pfungwa yekudzima firewall, iyo isina kushandiswa mune chaiyo sevha mhinduro, yakakurudzirwa neprofiling mhinduro, kutonga iyo nf_hook_slow basa yakatora 18% yenguva yekuita. Zvinocherechedzwa kuti nftables inoshanda zvakanyanya kupfuura iptables, asi Amazon Linux inoramba ichishandisa iptables. Mushure mekudzima iptables, kuwedzera kwekuita kwaive 22%, uye kubuda kwakawedzera kubva ku495k req / s kusvika 603k req / s.
• Yakaderedzwa kutama kwevanobata pakati peakasiyana CPU cores kuvandudza kushanda kwe processor cache kushandiswa. Optimization yakaitwa zvese padanho rekusunga libreactor maitiro kuCPU cores (CPU Pinning) uye kuburikidza nepinning kernel network vanobata (Receive Side Scaling). Semuenzaniso, irqbalance yakaremara uye mutsara wehukama kuCPU wakaiswa zvakajeka mukati /proc/irq/$IRQ/smp_affinity_list. Kuti ushandise iyo yakafanana CPU musimboti kugadzirisa iyo libreactor process uye network mutsara wemapaketi anouya, tsika yeBPF inobata inoshandiswa, yakabatana nekuisa iyo SO_ATTACH_REUSEPORT_CBPF mureza paunenge uchigadzira socket. Kusunga mitsetse yemapaketi anobuda kuCPU, marongero /sys/class/net/eth0/queues/tx- akachinjwa. /xps_cpus. Kuwedzera kwekuita kwese kwaive 38%, uye kubuda kwakawedzera kubva pa603k req/s kuenda ku834k req/s.
• Optimization yekukanganisa kubata uye kushandiswa kwekuvhota. Kugonesa iyo adaptive-rx modhi mumutyairi weENA uye kugadzirisa sysctl net.core.busy_read yakawedzera kuita ne28% (kuburikidza kwakawedzera kubva pa834k req/s kuenda ku1.06M req/s, uye latency yakadzikira kubva pa361μs kusvika 292μs).
• Kudzima masevhisi ehurongwa izvo zvinotungamira kuvharika kusingakoshi mune network stack. Kudzima dhclient uye nemaoko kuseta IP kero kwakakonzera kuwedzera kwe6% kuita uye kubuda kwakawedzera kubva 1.06M req/s kusvika 1.12M req/s. Chikonzero icho dhclient chinokanganisa mashandiro ari mukuongorora traffic uchishandisa socket mbishi.
• Kurwisa Spin Lock. Kuchinjisa network stack kuita "noqueue" modhi kuburikidza nesysctl "net.core.default_qdisc=noqueue" uye "tc qdisc tsiva dev eth0 midzi mq" zvakakonzera kuwedzera kwe2% kuita, uye kubuda kwakawedzera kubva 1.12M req/s kusvika 1.15M. req/s.
• Magadzirirwo madiki ekupedzisira, akadai sekudzima GRO (Generic Receive Offload) nemurairo "ethtool -K eth0 gro off" uye kutsiva cubic congestion control algorithm nereno uchishandisa sysctl "net.ipv4.tcp_congestion_control=reno". Kuwedzera kwehuwandu hwekuwedzera kwaive ne4%. Kupinza kwakawedzera kubva pa1.15M req/s kusvika ku1.2M req/s.

Pamusoro pekugadzirisa kwakashanda, chinyorwa chinokurukurawo nzira dzisina kutungamira kune inotarisirwa kuwedzera kwekuita. Semuenzaniso, zvinotevera zvakaratidza kuti hazvina kushanda:

• Kumhanya libreactor zvakasiyana hakuna kusiyana mukuita kubva pakuimhanyisa mumudziyo. Kutsiva writev nekutumira, kuwedzera maxevents mu epoll_wait, uye kuyedza neGCC shanduro nemireza hakuna mhedzisiro (mhedzisiro yaionekwa chete kune "-O3" uye "-march-yekuzvarwa" mireza).
• Kusimudzira Linux kernel kuti ive shanduro 4.19 uye 5.4, uchishandisa SCHED_FIFO uye SCHED_RR kuronga, manipulating sysctl kernel.sched_min_granularity_ns, kernel.sched_wakeup_granularity_ns, transparent_hugepages=never, skew did not affect=tscw
• Mutyairi weENA, kugonesa Offload modes (segmentation, scatter-gather, rx/tx checksum), kuvaka ne “-O3” mureza, uye kushandisa ena.rx_queue_size uye ena.force_large_llq_header parameters hazvina zvazvinoita.
• Shanduko mune network stack haina kunatsiridza mashandiro:
  • Dzima IPv6: ipv6.disable=1
  • Dzima VLAN: modprobe -rv 8021q
  • Dzima kutariswa kwepasuru
    • net.ipv4.conf.all.rp_filter=0
    • net.ipv4.conf.eth0.rp_filter=0
    • net.ipv4.conf.all.accept_local=1 (negative effect)
  • net.ipv4.tcp_sack = 0
  • net.ipv4.tcp_dsack=0
  • net.ipv4.tcp_mem/tcp_wmem/tcp_rmem
  • net.core.netdev_budget
  • net.core.dev_weight
  • net.core.netdev_max_backlog
  • net.ipv4.tcp_slow_start_after_idle=0
  • net.ipv4.tcp_moderate_rcvbuf=0
  • net.ipv4.tcp_timestamps=0
  • net.ipv4.tcp_low_latency = 1
  • SO_PRIORITY
  • TCP_NODELAY

  Source: opennet.ru