ื ืคึผืึธืจ ืคืื ืืึธืจ ืฆืืจืืง Kubernetes
ืืกืขื ืฉืึทืื, ืึทืคึผืืึทืงืืืฉืึทื ื ืืขืจืคืึทืจืื ื ืึท ืคึผืึธื ืขื ืืจืึทืค - ื ืขืฅ ืืืืืึทื ืกื ืคืื ืึทืจืืืฃ ืฆื 100ms ืึธืืขืจ ืืขืจ, ืจืืืึทืืืื ื ืืื ืืืืืึทืืฅ ืึธืืขืจ ืจืืืจืืื. ืกืขืจืืืืกืขืก ืืขื ืขื ืืขืจืืื ืฆื ืงืขื ืขื ืฆื ืจืืกืคึผืึทื ื ืฆื ืจืืงืืืขืก ืคืื ืคืึทืกืืขืจ ืืื 100ms. ืืืขืจ ืืึธืก ืืื ืืืืืขืืืขื ืืืื ืื ืงืฉืจ ืืื ื ืขืื ืึทืืื ืคืื ืฆืืื. ืกืขืคึผืขืจืึทืืื, ืืืจ ืืืืขืจืงื ืืืืขืจ ืฉื ืขื MySQL ืคึฟืจืืื ืืืึธืก ืืึธื ืืึธืื ืืขื ืืืขื ืืืืืกืขืงืึทื ืื, ืืื MySQL ืืื ืืึทื ืฅ ืืื ืืืืืกืขืงืึทื ืื, ืึธืืขืจ ืคึฟืื ืืขืจ ืคึผืขืจืกืคึผืขืงืืืื ืคืื ืื ืจืืงืืืขืกืืื ื ืึทืคึผืืึทืงืืืฉืึทื, ืื ืขื ืืคืขืจ ืืขื ืืืขื 100ms ืึธืืขืจ ืืขืจ.
ืขืก ืืื ืืืืื ืงืืึธืจ ืืขืืืืจื ืึทื ืืขืจ ืคึผืจืึธืืืขื ืืื ืืขืืืขื ืืืืื ืืืขื ืงืึทื ืขืงืืื ื ืฆื ืึท ืงืืืขืจื ืขืืขืก ื ืึธืืข, ืึทืคึฟืืื ืืืื ืื ืจืืคื ืืขืงืืืขื ืคึฟืื ืึทืจืืืก ืงืืืขืจื ืขืืขืก. ืื ืืืืึทืกื ืืืขื ืฆื ืจืขืคึผืจืึธืืืฆืืจื ืืขื ืคึผืจืึธืืืขื ืืื ืืื ืึท ืคึผืจืึธืืข
ืืืืืึทื ืืืืื ื ืืื ืืืืืง ืงืึทืืคึผืืขืงืกืืื ืืื ืื ืงืืื ืืืืื ื ืฆื ืืืจืืคืึทื
ืืืจื ืจืขืคึผืจืึธืืืฆืืจื ืื ืืขืืืข ืืืึทืฉืคึผืื, ืืืจ ืืขืืืืื ืฆื ืฉืืึธื ืื ืคืึธืงืืก ืคืื ืื ืคึผืจืึธืืืขื ืืื ืืึทืืืึทืืืงื ืืื ืืืืืง ืืืึทืขืจืก ืคืื ืงืึทืืคึผืืขืงืกืืื. ืืืืืขืก, ืขืก ืืขื ืขื ืืขืืืขื ืฆื ืคืืืข ืขืืขืืขื ืื ืืื ืื ืืืืคื ืฆืืืืฉื ืืืขืืขืืึท ืืื ืื ืงืืืขืจื ืขืืขืก ืคึผืึธืืก. ืฆื ืืืขื ืืืคืืฆืืจื ืึท ืืืคึผืขืจ ื ืขืฅ ืคึผืจืึธืืืขื, ืืืจ ืืึทืจืคึฟื ืฆื ืืืกืฉืืืกื ืขืืืขืืข ืคืื โโืืื.
ืืขืจ ืงืืืขื ื (ืืืขืืขืืึท) ืงืจืืืืฅ ืึท ืืงืคึผ ืงืฉืจ ืืื ืงืืื ื ืึธืืข ืืื ืืขื ืงื ืืื. Kubernetes ืึทืคึผืขืจืืืฅ ืืื ืึท ืึธืืืืขืจืืื ื ืขืฅ (ืืืืฃ ืื ืืืืืกืืื ื ืืึทืื ืฆืขื ืืขืจ ื ืขืฅ) ืืืึธืก ื ืืฆื
ื ืืฆื tcpdump
ืืื ืื Vegeta ืคึผืจืืืืจื ืขืก ืืื ืึท ืคืึทืจืืึทืืื ืืขืฉืึทืก ืื TCP ืืึทื ืืฉืืืง (ืฆืืืืฉื SYN ืืื SYN-ACK). ืฆื ืืึทืืืึทืืืงื ืืขื ืืื ืืืืืง ืงืึทืืคึผืืขืงืกืืื, ืืืจ ืงืขื ืขื ื ืืฆื hping3
ืคึฟืึทืจ ืคึผืฉืื "ืคึผืื ืืก" ืืื SYN ืคึผืึทืงืืฅ. ืืืจ ืงืึธื ืืจืึธืืืจื ืืืื ืขืก ืืื ืึท ืคืึทืจืืึทืืื ืืื ืื ืขื ืืคืขืจ ืคึผืึทืงืึทื, ืืื ืืขืืึธืื ืืึทืฉืืขืืืง ืื ืงืฉืจ. ืืืจ ืงืขื ืขื ืคืืืืขืจ ืื ืืึทืื ืืืืื ืฆื ืึทืจืืึทื ื ืขืืขื ืคึผืึทืงืืฅ ืืจืขืกืขืจ ืืื 100ms ืืื ืืึทืงืืืขื ืึท ืืจืื ืืขืจ ืืืขื ืฆื ืจืขืคึผืจืึธืืืฆืืจื ืืขื ืคึผืจืึธืืืขื ืืื ืื ืคืื ื ืขืฅ ืฉืืืืข 7 ืคึผืจืืืืจื ืืื Vegeta. ืืึธ ืืขื ืขื ืื Kubernetes ื ืึธืืข "ืคึผืื ืืก" ื ืืฆื TCP SYN / SYN-ACK ืืืืฃ ืื ืกืขืจืืืืก "ื ืึธืืข ืคึผืึธืจื" (30927) ืืื 10ms ืื ืืขืจืืืึทืื, ืคืืืืขืจื ืืืจื ืกืืึธืืึทืกื ืจืขืกืคึผืึธื ืกืขืก:
theojulienne@shell ~ $ sudo hping3 172.16.47.27 -S -p 30927 -i u10000 | egrep --line-buffered 'rtt=[0-9]{3}.'
len=46 ip=172.16.47.27 ttl=59 DF id=0 sport=30927 flags=SA seq=1485 win=29200 rtt=127.1 ms
len=46 ip=172.16.47.27 ttl=59 DF id=0 sport=30927 flags=SA seq=1486 win=29200 rtt=117.0 ms
len=46 ip=172.16.47.27 ttl=59 DF id=0 sport=30927 flags=SA seq=1487 win=29200 rtt=106.2 ms
len=46 ip=172.16.47.27 ttl=59 DF id=0 sport=30927 flags=SA seq=1488 win=29200 rtt=104.1 ms
len=46 ip=172.16.47.27 ttl=59 DF id=0 sport=30927 flags=SA seq=5024 win=29200 rtt=109.2 ms
len=46 ip=172.16.47.27 ttl=59 DF id=0 sport=30927 flags=SA seq=5231 win=29200 rtt=109.2 ms
ืงืขื ืขื ืืื ืืึทืื ืื ืขืจืฉืืขืจ ืึธืืกืขืจืืืึทืฆืืข. ืืืื ืืฉืคืื ืืืื ืื ืกืืงืืืึทื ืก ื ืืืขืจื ืืื ืืืืืื ืื, ืขืก ืืื ืงืืึธืจ ืึทื ืืึธืก ืืขื ืขื ื ืืฉื ืืืื-ืฆืืึทื ืงืึทื ืืืฉืขืกืืฉืึทื ื. ืื ืคืึทืจืืึทืืื ืึธืคื ืึทืงืืืืืึทืืืืฅ ืืื ืืื ืืืืขื ืืฉืึทืืืึทืื ืคึผืจืึทืกืขืกื.
ืืืืึทืืขืจ, ืืืจ ืืืืื ืฆื ืืขืคึฟืื ืขื ืืืืก ืืืึธืก ืงืึทืืคึผืึธืื ืึทื ืฅ ืงืขื ืืืื ืื ืืืึทืืืื ืืื ืื ืคึผืึทืกืืจืื ื ืคืื ืงืึทื ืืืฉืขืกืืฉืึทื. ืืคึฟืฉืจ ืืึธืก ืืขื ืขื ืขืืืขืืข ืคืื โโโโืื ืืื ืืขืจืืขืจ ืคืื Iptables ืึผืืืื ืืื NAT? ืึธืืขืจ ืืขื ืขื ืขืก ืงืืื ืคืจืืืืขืืขื ืืื IPIP ืืึทื ืึทืืื ื ืืืืฃ ืื ื ืขืฅ? ืืืื ืืืขื ืฆื ืคึผืจืืืืจื ืืึธืก ืืื ืฆื ืคึผืจืืืืจื ืืขืืขืจ ืฉืจืื ืคืื ืื ืกืืกืืขื ืืืจื ืืืืืึทื ืืืืื ื ืขืก. ืืืึธืก ืืึทืคึผืึทื ื ืืืื ืืืจ ืืึทืืืึทืืืงื NAT ืืื Firewall ืืึธืืืง, ืืื ืืึธืื ืืืืื ืื IPIP ืืืื:
ืฆืื ืืืืง, ืืื ืืงืก ืืืื ืขืก ืืจืื ื ืฆื ืึทืงืกืขืก ืื IP ืึธืืืืขืจืืื ืฉืืืืข ืืืืึทื ืืืื ืื ืืึทืฉืื ืืื ืืืืฃ ืืขืจ ืืขืืืืงืขืจ ื ืขืฅ:
theojulienne@kube-node-client ~ $ sudo hping3 10.125.20.64 -S -i u10000 | egrep --line-buffered 'rtt=[0-9]{3}.'
len=40 ip=10.125.20.64 ttl=64 DF id=0 sport=0 flags=RA seq=7346 win=0 rtt=127.3 ms
len=40 ip=10.125.20.64 ttl=64 DF id=0 sport=0 flags=RA seq=7347 win=0 rtt=117.3 ms
len=40 ip=10.125.20.64 ttl=64 DF id=0 sport=0 flags=RA seq=7348 win=0 rtt=107.2 ms
ืืืื ืืฉืคืื ืืืื ืื ืจืขืืืืืึทืื, ืื ืคึผืจืึธืืืขื ื ืึธื ืืืืืื! ืืึธืก ืืงืกืงืืืื iptables ืืื NAT. ืึทืืื ืื ืคึผืจืึธืืืขื ืืื TCP? ืืึธืืืจ ืืขื ืืื ืึท ืจืขืืืืขืจ ICMP ืคึผืื ื ืืืื:
theojulienne@kube-node-client ~ $ sudo hping3 10.125.20.64 --icmp -i u10000 | egrep --line-buffered 'rtt=[0-9]{3}.'
len=28 ip=10.125.20.64 ttl=64 id=42594 icmp_seq=104 rtt=110.0 ms
len=28 ip=10.125.20.64 ttl=64 id=49448 icmp_seq=4022 rtt=141.3 ms
len=28 ip=10.125.20.64 ttl=64 id=49449 icmp_seq=4023 rtt=131.3 ms
len=28 ip=10.125.20.64 ttl=64 id=49450 icmp_seq=4024 rtt=121.2 ms
len=28 ip=10.125.20.64 ttl=64 id=49451 icmp_seq=4025 rtt=111.2 ms
len=28 ip=10.125.20.64 ttl=64 id=49452 icmp_seq=4026 rtt=101.1 ms
len=28 ip=10.125.20.64 ttl=64 id=50023 icmp_seq=4343 rtt=126.8 ms
len=28 ip=10.125.20.64 ttl=64 id=50024 icmp_seq=4344 rtt=116.8 ms
len=28 ip=10.125.20.64 ttl=64 id=50025 icmp_seq=4345 rtt=106.8 ms
len=28 ip=10.125.20.64 ttl=64 id=59727 icmp_seq=9836 rtt=106.1 ms
ืื ืจืขืืืืืึทืื ืืืืึทืื ืึทื ืื ืคึผืจืึธืืืขื ืืื ื ืืฉื ืึทืืืขืง. ืืึธืืขืจ ืืึธืก ืืื ืึทื IPIP ืืื ืขื? ืืื ืก ืคืึทืจืคึผืึธืฉืขืืขืจื ืื ืคึผืจืึธืืข ืืืืึทืืขืจ:
ืืขื ืขื ืึทืืข ืคึผืึทืงืืฅ ืืขืฉืืงื ืฆืืืืฉื ืื ืฆืืืื ืืื ืืช?
theojulienne@kube-node-client ~ $ sudo hping3 172.16.47.27 --icmp -i u10000 | egrep --line-buffered 'rtt=[0-9]{3}.'
len=46 ip=172.16.47.27 ttl=61 id=41127 icmp_seq=12564 rtt=140.9 ms
len=46 ip=172.16.47.27 ttl=61 id=41128 icmp_seq=12565 rtt=130.9 ms
len=46 ip=172.16.47.27 ttl=61 id=41129 icmp_seq=12566 rtt=120.8 ms
len=46 ip=172.16.47.27 ttl=61 id=41130 icmp_seq=12567 rtt=110.8 ms
len=46 ip=172.16.47.27 ttl=61 id=41131 icmp_seq=12568 rtt=100.7 ms
len=46 ip=172.16.47.27 ttl=61 id=9062 icmp_seq=31443 rtt=134.2 ms
len=46 ip=172.16.47.27 ttl=61 id=9063 icmp_seq=31444 rtt=124.2 ms
len=46 ip=172.16.47.27 ttl=61 id=9064 icmp_seq=31445 rtt=114.2 ms
len=46 ip=172.16.47.27 ttl=61 id=9065 icmp_seq=31446 rtt=104.2 ms
ืืืจ ืืึธืื ืกืืืคึผืืึทืคืืื ืื ืกืืืืึทืฆืืข ืฆื ืฆืืืื Kubernetes ื ืึธืืื ืฉืืงื ืืขืืขืจ ืื ืืขืจืข ืงืืื ืคึผืึทืงืึทื, ืืคืืื ืึทื ICMP ืคึผืื ื. ืืื ื ืึธื ืืขื ืืืืืึทื ืกื ืืืื ืืขืจ ืฆืื ืืึทืืขืืึธืก ืืื "ืฉืืขืื" (ืขืืืขืืข ืขืจืืขืจ ืืื ืื ืืขืจืข).
ืืืฆื ืื ืืขืฆืืข ืงืฉืื: ืืืึธืก ืืื ืื ืคืึทืจืืึทืืื ืืืืื ืืืืฃ ืงืืืข-ื ืึธืืข ืกืขืจืืืขืจืก? ืืื ืืื ืขืก ืคึผืึทืกืืจื ืืืขื ืงืืืข-ื ืึธืืข ืืื ืืขืจ ืกืขื ืืขืจ ืึธืืขืจ ืืขืจ ืืคื ืขืืขืจ? ืฆืื ืืืืง, ืืึธืก ืืื ืืืื ืืึทื ืฅ ืืจืื ื ืฆื ืจืขืืขื ืขื ืืืจื ืฉืืงื ืึท ืคึผืึทืงืึทื ืคืื ืึท ืืึทืืขืืึธืก ืึทืจืืืก ืคืื Kubernetes, ืึธืืขืจ ืืื ืืขืจ ืืขืืืืงืขืจ "ืืืงืื ื ืฉืืขืื" ืืึทืงืืืขืจ. ืืื ืืืจ ืงืขื ืขื ืืขื, ืื ืคึผืจืึธืืืขื ืืื ื ืืฉื ืคืึทืจืฉืืืื ืื:
theojulienne@shell ~ $ sudo hping3 172.16.47.27 -p 9876 -S -i u10000 | egrep --line-buffered 'rtt=[0-9]{3}.'
len=46 ip=172.16.47.27 ttl=61 DF id=0 sport=9876 flags=RA seq=312 win=0 rtt=108.5 ms
len=46 ip=172.16.47.27 ttl=61 DF id=0 sport=9876 flags=RA seq=5903 win=0 rtt=119.4 ms
len=46 ip=172.16.47.27 ttl=61 DF id=0 sport=9876 flags=RA seq=6227 win=0 rtt=139.9 ms
len=46 ip=172.16.47.27 ttl=61 DF id=0 sport=9876 flags=RA seq=7929 win=0 rtt=131.2 ms
ืืขืจื ืึธื, ืืืจ ืืืขืื ืืืืคื ืื ืืขืืืข ืจืืงืืืขืก ืคืื ืื ืคืจืืขืจืืืงืข ืืงืืจ ืงืืืข-ื ืึธืืข ืฆื ืื ืคืื ืืจืืืกื ืืืง ืืึทืืขืืึธืก (ืืืึธืก ืืงืกืงืืืื ืื ืืงืืจ ืืึทืืขืืึธืก ืืื ื ืื ืคึผืื ื ืืืื ืึท ืจืงืก ืืื ืืงืก ืงืึธืืคึผืึธื ืขื ื):
theojulienne@kube-node-client ~ $ sudo hping3 172.16.33.44 -p 9876 -S -i u10000 | egrep --line-buffered 'rtt=[0-9]{3}.'
^C
--- 172.16.33.44 hping statistic ---
22352 packets transmitted, 22350 packets received, 1% packet loss
round-trip min/avg/max = 0.2/7.6/1010.6 ms
ืืืจื ืื ืืขืจืืืื ืืืืืึทื ืกื ืคึผืึทืงืึทื ืงืึทืคึผืืฉืขืจื, ืืืจ ืืืงืืืขื ืขืืืขืืข ื ืึธื ืืื ืคึฟืึธืจืืึทืฆืืข. ืกืคึผืึทืกืืคืืงืื, ืึทื ืืขืจ ืึธืคึผืฉืืงืขืจ (ืื ืึธ) ืืขื ืืขื ืืืืืึทืื, ืึธืืขืจ ืืขืจ ืืึทืงืืืขืจ (ืฉืคึผืืฅ) ืงืขื ื ืืฉื - ืืขื ืื ืืขืืืึท ืืืึทื (ืืื ืกืขืงืื ืืขืก):
ืืื ืึทืืืฉืึทื, ืืืื ืืืจ ืงืืง ืืืึท ืื ืืืืืง ืืื ืื ืกืืจ ืคืื ืืงืคึผ ืืื ICMP ืคึผืึทืงืืฅ (ืืืจื ืกืืงืืืึทื ืก ื ืืืขืจื) ืืืืฃ ืื ืืึทืงืืืขืจ ืืืึทื, ICMP ืคึผืึทืงืืฅ ืฉืืขื ืืืง ืึธื ืงืืืขื ืืื ืืขืจ ืืขืืืืงืขืจ ืกืืงืืืึทื ืก ืืื ืืืึธืก ืืื ืืขื ืขื ืืขืฉืืงื, ืึธืืขืจ ืืื ืคืึทืจืฉืืืขื ืข ืืืืืื ื. ืืื ืืขืจ ืืขืืืืงืขืจ ืฆืืื, TCP ืคึผืึทืงืืฅ ืืื ืื ืืขืจืืืึทืื, ืืื ืขืืืขืืข ืคืื โโืืื ืกืืึทืง. ืืื ืืึทืืื ืืขืจ, ืืืื ืืืจ ืื ืืขืจืืืื ืื ืคึผืึธืจืฅ ืคืื SYN ืคึผืึทืงืืฅ, ืืื ืืขื ืขื ืืื ืกืืจ ืืืืฃ ืื ืกืขื ืืขืจ ืก ืืืึทื, ืึธืืขืจ ื ืืฉื ืืืืฃ ืื ืืคื ืขืืขืจ ืก ืืืึทื.
ืขืก ืืื ืึท ืกืึทืืึทื ืืืืืง ืืื ืืื
ืื ืื ืืขืจ ื ืืึทืข ืึธืืกืขืจืืืึทืฆืืข: ืืขืฉืึทืก ืืขื ืคึผืขืจืืึธื ืืืจ ืืขื ICMP ืืืืืื ืืืืฃ ืึทืืข ืงืึธืืื ืืงืึทืฆืืข ืฆืืืืฉื ืฆืืืื ืืื ืืช, ืึธืืขืจ TCP ืืื ื ืืฉื. ืืึธืก ืืขืจืฆืืืื ืืื ืื ืึทื ืื ืกืืื ืืื ืืกืชึผืื ืฉืืึทืืืช ืฆื RX ืจืื ืืึทืฉืื ื: ืื ืงืึทื ืืืฉืขืกืืฉืึทื ืืื ืึผืืขื ืืืืขืจ ืืื ืื ืคึผืจืึทืกืขืกืื ื ืคืื RX ืคึผืึทืงืืฅ, ื ืืฉื ืืื ืื ืฉืืงื ืคืื ืจืขืกืคึผืึธื ืกืขืก.
ืืึธืก ืืืืืึทื ืืืฅ ืฉืืงื ืคึผืึทืงืืฅ ืคืื ืืขืจ ืจืฉืืื ืคืื ืืขืืืขื ืกืืืืช. ืืืจ ืืืฆื ืืืืกื ืึทื ืื ืคึผืึทืงืึทื ืคึผืจืึทืกืขืกืื ื ืคึผืจืึธืืืขื ืืื ืืืืฃ ืื ืืึทืงืืืขื ืืืึทื ืืืืฃ ืขืืืขืืข ืงืืืข-ื ืึธืืข ืกืขืจืืืขืจืก.
ืคืืจืฉืืื ื ืคืื ืคึผืึทืงืึทื ืคึผืจืึทืกืขืกืื ื ืืื ืื ืืื ืืงืก ืงืขืจื
ืฆื ืคึฟืึทืจืฉืืืื ืืืึธืก ืื ืคึผืจืึธืืืขื ืึทืงืขืจื ืืื ืื ืืคื ืขืืขืจ ืืืืฃ ืขืืืขืืข ืงืืืข-ื ืึธืืข ืกืขืจืืืขืจืก, ืืึธืื ืืื ืื ืงืืง ืืื ืื ืืื ืืงืก ืงืขืจื ืคึผืจืึทืกืขืกืึทื ืคึผืึทืงืืฅ.
ืฆืืจืืงืงืืืขื ืฆื ืื ืกืืืคึผืืึทืกื ืืจืืืืฆืืื ืขืื ืืืคึผืืึทืืขื ืืืืฉืึทื, ืื ื ืขืฅ ืงืึธืจื ื ืขืื ืื ืคึผืึทืงืึทื ืืื ืกืขื ืื
ืืขืจ ืงืึธื ืืขืงืกื ืกืืืืืฉืื ื ืืื ืคึผืึทืืขืืขื: ืืืืืึทื ืกื ืงืขื ื ืืฉื ืืืื ืืืืขืจืงื ืืืืฃ 10 ืืืคึผืก ื ืขืฅ ืงืึทืจืืก ืืื ืื '90 ืก, ืึธืืขืจ ืืืืฃ ืืึธืืขืจื 10 ื ืงืึทืจืืก ืืื ืึท ืืึทืงืกืืืื ืืจืืคึผืื ืคืื 15 ืืืืืึธื ืคึผืึทืงืืฅ ืคึผืขืจ ืกืขืงืื ืืข, ืืขืืขืจ ืืึทืจืฅ ืคืื ืึท ืงืืืื ืึทืื-ืืึทืจืฅ ืกืขืจืืืขืจ ืงืขื ืขื ืืืื ืื ืืขืจืึทืคึผืืื ืืืืืึทื ื. ืืื ืคึผืขืจ ืกืขืงืื ืืข.
ืืื ืกืืจ ื ืืฉื ืฆื ืงืขืกืืืืขืจ ืฉืขืคึผื ืื ืืขืจืึทืคึผืฅ, ืคืืืข ืืึธืจื ืฆืืจืืง ืืื ืืงืก ืฆืืืขืืขืื
ืืึธืก ืืื ืคืื ืคืึทืกืืขืจ, ืึธืืขืจ ื ืึท ืึทื ืืขืจืฉ ืคึผืจืึธืืืขื. ืืืื ืขืก ืืขื ืขื ืฆื ืคืืืข ืคึผืึทืงืืฅ, ืึทืืข ืื ืฆืืื ืืื ืคืืจืืจืืื ืคึผืจืึทืกืขืกืื ื ืคึผืึทืงืืฅ ืคึฟืื ืื ื ืขืฅ ืงืึธืจื, ืืื ืืึทื ืืฆืขืจ ืคึผืืึทืฅ ืคึผืจืึทืกืขืกืึทื ืืึธื ื ืื ืืึธืื ืฆืืื ืฆื ืึทืงืืฉืึทืืืึทืื ืืืืืืง ืื ืงืืื (ืืืืขื ืขื ืคึฟืื TCP ืงืึทื ืขืงืฉืึทื ื, ืืื"ื ื). ืืืืขื ืืฉืึทืืืึทืื ืื ืงืืื ืคึผืืึธืืืืจื ืืื ืืื ืืืจ ืึธื ืืืืื ืืจืึทืคึผืื ื ืคึผืึทืงืืฅ. ืืื ืึทื ืคึผืจืืืื ืฆื ืืขืคึฟืื ืขื ืึท ืืืึธื, ืืขืจ ืงืขืจื ืฉืืขืื ืึท ืืืืืฉืขื ืคึฟืึทืจ ืื ืืึทืงืกืืืื ื ืืืขืจ ืคืื ืคึผืึทืงืืฅ ืคึผืจืึทืกืขืกื ืืื ืื ืกืึธืคืืืจืง ืงืึธื ืืขืงืกื. ืึทืืึธื ืืขื ืืืืืฉืขื ืืื ืืงืกืืืื, ืึท ืืึทืืื ืืขืจ ืคืึธืืขื ืืื ืึทืืืืืงืึทื ื ksoftirqd
(ืืืจ ืืืขื ืืขื ืืืื ืขืจ ืคืื ืืื ืืื ps
ืคึผืขืจ ืืึทืจืฅ) ืืืึธืก ืืึทื ืืึทืื ืื ืกืึธืคืืืจืงืก ืึทืจืืืก ืคืื ืืขืจ ื ืึธืจืืึทื ืกืืกืงืึทื / ืืืขืจืจืืึทืกื ืืจื. ืืขืจ ืคืึธืืขื ืืื ืกืงืขืืืฉืืื ื ืืฆื ืื ื ืึธืจืืึทื ืคึผืจืึธืฆืขืก ืกืงืขืืืฉืืืขืจ, ืืืึธืก ืคืจืืืื ืฆื ืึทืืึทืงืืื ืจืขืกืืจืกื ืคืขืจืื.
ื ืึธื ืืขืืขืจื ื ืืื ืืขืจ ืงืขืจื ืคึผืจืึทืกืขืกืึทื ืคึผืึทืงืืฅ, ืืืจ ืงืขื ืขื ืืขื ืึทื ืขืก ืืื ืึท ืืืืขืจ ืืึทืฉืืึธืขืก ืคืื ืงืึทื ืืืฉืขืกืืฉืึทื. ืืืื ืกืึธืคืืืจืง ืจืืคื ืืขื ืขื ืืืงืืืขื ืืืืื ืืงืขืจ ืึธืคื, ืคึผืึทืงืืฅ ืืืขื ืืึธืื ืฆื ืืืึทืจืื ืคึฟืึทืจ ืขืืืขืืข ืืึธื ืฆื ืืืื ืคึผืจืึทืกืขืกื ืืื ืื RX ืจืื ืืืืฃ ืื ื ืขืฅ ืงืึธืจื. ืืึธืก ืงืขื ืืืื ืจืขืื ืฆื ืขืืืขืืข ืึทืจืืขื ืืืึทืงืื ื ืื ืคึผืจืึทืกืขืกืขืจ ืืึทืจืฅ, ืึธืืขืจ ืขืคึผืขืก ืึทื ืืขืจืฉ ืืื ืคึผืจืืืืขื ืืื ืื ืืึทืจืฅ ืคืื ืคืืืกื ืืืง ืืืืืืืืืจื.
ื ืขืจืึธืืื ื ืื ืคึผืจืึทืกืขืกืื ื ืึทืจืึธืคึผ ืฆื ืื ืืึทืจืฅ ืึธืืขืจ ืืืคึฟื
Softirq ืืืืืื ืืขื ืขื ืืืืื ืึท ืืจืขืคื ืคึฟืึทืจ ืืืฆื. ืืืขืจ ืขืก ืืืื ืืื ืขื, ืืื ืืืจ ืืืืกื ืึทื ืืืจ ืืขื ืขืคึผืขืก ืืืืขืจ ืขื ืืขื. ืืขืจ ืืืืึทืืขืจ ืฉืจืื ืืื ืฆื ืืึทืฉืืขืืืงื ืืขื ืืขืึธืจืืข. ืืื ืืืื ืขืก ืืื ืืืฉืืขืืืงื, ืืขืคึฟืื ืขื ืื ืกืืื ืคึฟืึทืจ ืื ืืืืืื.
ืืึธืืืจ ืฆืืจืืงืงืืืขื ืฆื ืืื ืืืขืจ ืคึผืึทืืขืืขื ืคึผืึทืงืึทืฅ:
len=46 ip=172.16.53.32 ttl=61 id=29573 icmp_seq=1953 rtt=99.3 ms
len=46 ip=172.16.53.32 ttl=61 id=29574 icmp_seq=1954 rtt=89.3 ms
len=46 ip=172.16.53.32 ttl=61 id=29575 icmp_seq=1955 rtt=79.2 ms
len=46 ip=172.16.53.32 ttl=61 id=29576 icmp_seq=1956 rtt=69.1 ms
len=46 ip=172.16.53.32 ttl=61 id=29577 icmp_seq=1957 rtt=59.1 ms
len=46 ip=172.16.53.32 ttl=61 id=29790 icmp_seq=2070 rtt=75.7 ms
len=46 ip=172.16.53.32 ttl=61 id=29791 icmp_seq=2071 rtt=65.6 ms
len=46 ip=172.16.53.32 ttl=61 id=29792 icmp_seq=2072 rtt=55.5 ms
ืืื ืืืกืงืึทืกื ืคืจืืขืจ, ืื ICMP ืคึผืึทืงืืฅ ืืขื ืขื ืืึทืฉื ืืื ืึท ืืืื RX NIC ืจืื ืืื ืคึผืจืึทืกืขืกื ืืืจื ืึท ืืืื ืงืคึผื ืืึทืจืฅ. ืืืื ืืืจ ืืืืื ืฆื ืคึฟืึทืจืฉืืืื ืืื ืืื ืืงืก ืึทืจืืขื, ืขืก ืืื ื ืืฆืืง ืฆื ืืืืกื ืืื (ืืืืฃ ืืืึธืก ืงืคึผื ืืึทืจืฅ) ืืื ืืื (softirq, ksoftirqd) ืื ืคึผืึทืงืึทืืืฉืึทื ืืขื ืขื ืคึผืจืึทืกืขืกื ืืื ืกืืจ ืฆื ืฉืคึผืืจ ืืขื ืคึผืจืึธืฆืขืก.
ืืืฆื ืขืก ืืื ืฆืืื ืฆื ื ืืฆื ืืืฉืืจืื ืืืึธืก ืืึธืื ืืืจ ืฆื ืืึธื ืืืึธืจ ืื ืืื ืืงืก ืงืขืจื ืืื ืคืึทืงืืืฉ ืฆืืื. ืืึธ ืืืจ ืืขืืืืื ื
ืืขืจ ืคึผืืึทื ืืึธ ืืื ืคึผืฉืื: ืืืจ ืืืืกื ืึทื ืืขืจ ืงืขืจื ืคึผืจืึทืกืขืกืึทื ืื ICMP ืคึผืื ืืก, ืึทืืื ืืืจ ืฉืืขืื ืึท ืงืจืืง ืืืืฃ ืื ืงืขืจื ืคืื ืงืฆืืข hping3
ืืขืืขืจ.
ืงืึธืืขืงืก icmp_echo
ืงืึทื ืืืืื struct sk_buff *skb
: ืืึธืก ืืื ืึท ืคึผืึทืงืึทื ืืื ืึท "ืขืงืึธื ืืขืื". ืืืจ ืงืขื ืขื ืฉืคึผืืจ ืขืก, ืฆืืขื ืื ืกืืงืืืึทื ืก echo.sequence
(ืืืึธืก ืงืึทืืคึผืขืจืื ืืื icmp_seq
ืืืจื hping3 ะฒััะต
), ืืื ืฉืืงื ืขืก ืฆื ืืึทื ืืฆืขืจ ืคึผืืึทืฅ. ืขืก ืืื ืืืื ืืึทืงืืืขื ืฆื ืืึทืคึผื ืืขื ืงืจืึทื ื ืคึผืจืึธืฆืขืก ื ืึธืืขื / ืฉืืึทื. ืืื ืื ืืขื ืขื ืื ืจืขืืืืืึทืื ืืืึธืก ืืืจ ืืขื ืืืืื ืืฉืขืช ืืขืจ ืงืขืจื ืคึผืจืึทืกืขืกืึทื ืคึผืึทืงืืฅ:
TGID PID ืคึผืจืึธืฆืขืก ื ืึธืืขื ICMP_SEQ 0 0 ืกืืืึทืคึผืขืจ/11 770 0 ืกืืืึทืคึผืขืจ/0 11 771 ืกืืืึทืคึผืขืจ/0 0 11 ืกืืืึทืคึผืขืจ/772 0 0 ืกืืืึทืคึผืขืจ / 11 773 0 ืคึผืจืึธืืขืืืขืืก 0 11 774 ืกืืืึทืคึผืขืจ / 20041 20086 775 ืกืืืึทืคึผืขืจ/0 0 11 ืกืืืึทืคึผืขืจ/776 0 0 ืกืคึผืึธืืงืก ืืึทืจืืื-ืก 11
ืขืก ืืึธื ืืืื ืื ืืขืืืืื ืืึธ ืึทื ืืื ืืขื ืงืึธื ืืขืงืกื softirq
ืคึผืจืึทืกืขืกืึทื ืืืึธืก ืืขืืืื ืกืืกืืขื ืงืึทืืืก ืืืขื ืืขืจืฉืืึทื ืขื ืืื "ืคึผืจืึทืกืขืกืึทื" ืืืขื ืืื ืคืึทืงื ืขืก ืืื ืืขืจ ืงืขืจื ืืืึธืก ืืขืฉืึธืืขื ืคึผืจืึทืกืขืกืึทื ืคึผืึทืงืืฅ ืืื ืืขื ืงืึธื ืืขืงืกื ืคืื ืื ืงืขืจื.
ืืื ืืขื ืืขืฆืืึทื ืืืจ ืงืขื ืขื ืคืึทืจืืื ืื ืกืคึผืขืฆืืคืืฉ ืคึผืจืึทืกืขืกืึทื ืืื ืกืคึผืขืฆืืคืืฉ ืคึผืึทืงืึทืืืฉืึทื ืืืึธืก ืืืืึทืื ืึท ืคืึทืจืืึทืืื ืคืื hping3
. ืืื ืก ืืึทืื ืขืก ืคึผืฉืื grep
ืืืืฃ ืืขื ืืึทืคึผื ืคึฟืึทืจ ืืืืขืจ ืืืึทืืืขืก icmp_seq
. ืคึผืึทืงืืฅ ืืืึธืก ืจืืืื ืืื ืฆื ืื ืืืืื icmp_seq ืืืึทืืืขืก ืืขื ืขื ืคืืึทืื ืืื ืืืืขืจ RTT ืืืึธืก ืืืจ ืืืืขืจืงื ืืืืื (ืืื ืงืืึทืืขืจื ืืขื ืขื ืื ืืขืจืืืึทืจื RTT ืืืึทืืืขืก ืคึฟืึทืจ ืคึผืึทืงืืฅ ืืืึธืก ืืืจ ืคืืืืขืจื ืจืขืื ืฆื RTT ืืืึทืืืขืก ืืืืื ืืงืขืจ ืืื 50ms):
TGID PID ืคึผืจืึธืฆืขืก ื ืึธืืขื ICMP_SEQ ** RTT -- 10137 10436 ืงืึทืืืืืกืึธืจ 1951 10137 10436 ืงืึทืืืืืกืึธืจ 1952 76 76 ืงืกืึธืคืืจืงื/11 1953 ** 99ืืก 76 76 ืงืกืึธืคืืจืงื/11 1954 ** 89ืืก 76 76 ืงืกืึธืคืืจืงื/11 1955 ** 79ืืก 76 76 ืงืกืึธืคืืจืงื/11 1956 ** 69ืืก 76 76 ืงืกืึธืคืืจืงื/11 1957 ** 59ืืก 76 76 ksoftirqd/11 1958 ** (49ms) 76 76 ืงืกืึธืคืืจืงื/11 1959 ** (39 ืืื) 76 76 ืงืกืึธืคืืจืงื / 11 1960 ** (29 ืืื) 76 76 ืงืกืึธืคืืจืงื / 11 1961 ** (19 ืืื) 76 76 ืงืกืึธืคืืจืงื/11 1962 ** (9 ืืื) -- 10137 10436 ืงืึทืืืืืกืึธืจ 2068 10137 10436 ืงืึทืืืืืกืึธืจ 2069 76 76 ืงืกืึธืคืืจืงื/11 2070 ** 75ืืก 76 76 ืงืกืึธืคืืจืงื/11 2071 ** 65ืืก 76 76 ืงืกืึธืคืืจืงื/11 2072 ** 55ืืก 76 76 ืงืกืึธืคืืจืงื/11 2073 ** (45 ืืื) 76 76 ืงืกืึธืคืืจืงื/11 2074 ** (35 ืืื) 76 76 ืงืกืึธืคืืจืงื/11 2075 ** (25 ืืื) 76 76 ืงืกืึธืคืืจืงื/11 2076 ** (15 ืืื) 76 76 ืงืกืึธืคืืจืงื/11 2077 ** (5ืืก)
ืื ืจืขืืืืืึทืื ืืึธืื ืืื ืื ืขืืืขืืข ืืืื. ืคืืจืกืืื, ืึทืืข ืื ืคึผืึทืงืึทืืืฉืึทื ืืขื ืขื ืคึผืจืึทืกืขืกื ืืืจื ืืขื ืงืึธื ืืขืงืกื ksoftirqd/11
. ืืขื ืืืื ืึทื ืคึฟืึทืจ ืืขื ืืึทืืื ืืขืจ ืคึผืึธืจ ืคืื ืืืฉืื ืขื, ICMP ืคึผืึทืงืืฅ ืืขื ืขื ืืึทืฉื ืฆื ืืึทืจืฅ 11 ืืื ืื ืจืืกืืืืื ื ืกืืฃ. ืืืจ ืืืื ืืขื ืึทื ืืืขื ืขืก ืืื ืึท ืงืืขื, ืขืก ืืขื ืขื ืคึผืึทืงืืฅ ืืืึธืก ืืขื ืขื ืคึผืจืึทืกืขืกื ืืื ืืขื ืงืึธื ืืขืงืกื ืคืื ืื ืกืืกืืขื ืจืืคื cadvisor
... ืืขืจื ืึธื ksoftirqd
ื ืขืื ืืืืขืจ ืื ืึทืจืืขื ืืื ืคึผืจืึทืกืขืกืึทื ืื ืึทืงืืืืืึทืืืืืื ืจืื: ืคึผืื ืงื ืื ื ืืืขืจ ืคืื ืคึผืึทืงืืฅ ืืืึธืก ืืื ืึทืงืืืืืึทืืืืืื ื ืึธื cadvisor
.
ืืขืจ ืคืึทืงื ืึทื ืืื ืืืืืขืจ ืขืก ืฉืืขื ืืืง ืึทืจืืขื cadvisor
, ืืืคึผืืืื ืืืื ืื ืืืึทืืืืืึทื ื ืืื ืืขื ืคึผืจืึธืืืขื. ืืจืึธื ืืงืึทืืื, ืืขืจ ืฆืื
ืืื ืืื ืื ืืขืจืข ืึทืกืคึผืขืงืฅ ืคืื ืงืึทื ืืืื ืขืจื, ืืึธืก ืืขื ืขื ืึทืืข ืืขืืกื ืึทืืืึทื ืกืืจืืข ืืืฉืืจืื ืืื ืงืขื ืขื ืืืื ืืขืจืืืึทืจื ืฆื ืืขืจืคืึทืจืื ื ืคืึธืจืฉืืขืืื ื ืืฉืื ืืื ืืขืจ ืขืืืขืืข ืึทื ืคืึธืจืกืื ืฆืืฉืืื ืื.
ืืืึธืก ืืื ืงืึทืืืืืกืึธืจ ืึทื ืกืืึธืื ืึทืจืึธืคึผ ืื ืคึผืึทืงืึทื ืจืื?
ืืืจ ืืืฆื ืืึธืื ืึท ืฉืืื ืืื ืคืืจืฉืืื ื ืคืื ืืื ืืขืจ ืงืจืึทื ืึทืงืขืจื, ืืืึธืก ืคึผืจืึธืฆืขืก ืืื ืงืึธืืื ื ืขืก ืืื ืืืืฃ ืืืึธืก ืงืคึผื. ืืืจ ืืขื ืึทื ืจืขืื ืฆื ืฉืืืขืจ ืืืึทืงืื ื, ืื ืืื ืืงืก ืงืขืจื ืืื ื ืืฉื ืืึธืื ืฆืืื ืฆื ืคึผืืึทื ksoftirqd
. ืืื ืืืจ ืืขื ืึทื ืคึผืึทืงืืฅ ืืขื ืขื ืคึผืจืึทืกืขืกื ืืื ืงืึธื ืืขืงืกื cadvisor
. ืขืก ืืื ืืึทืืืฉืืงืึทื ืฆื ืืืขืจื ืขืืขื ืึทื cadvisor
ืืึธื ืืฉืื ืึท ืคึผืึทืืขืืขื ืกืืกืงืึทืื, ื ืึธื ืืืึธืก ืึทืืข ืคึผืึทืงืืฅ ืึทืงืืืืืึทืืืืืื ืืื ืืขืจ ืฆืืื ืืขื ืขื ืคึผืจืึทืกืขืกื:
ืืึธืก ืืื ืึท ืืขืึธืจืืข, ืึธืืขืจ ืืื ืฆื ืคึผืจืืืืจื ืขืก? ืืืึธืก ืืืจ ืงืขื ืขื ืืึธื ืืื ืฉืคึผืืจ ืื ืงืคึผื ืืึทืจืฅ ืืืืขืจ ืืขื ืคึผืจืึธืฆืขืก, ืืขืคึฟืื ืขื ืื ืคืื ื ืืื ืื ื ืืืขืจ ืคืื ืคึผืึทืงืืฅ ืืืื ืืืืขืจ ืืืืืฉืขื ืืื ksoftirqd ืืื ืืขืจืืคื, ืืื ืงืืง ืึท ืืืกื ืืขืจ ืฆืืจืืง ืฆื ืืขื ืืืึธืก ืคึผืื ืงื ืืื ืคืืืกื ืืืง ืืืืฃ ืื ืงืคึผื ืืึทืจืฅ ืคึผืื ืงื ืืืืืขืจ ืืขื ืคืื ื . ืขืก ืืื ืืื ืจืขื ืืืขื -ืจืืืื ื ืื ืงืคึผื ืืขืืขืจ ืืืกื ืืืืืกืขืงืึทื ืื. ืขืก ืืืขื ืงืืงื ืขืคึผืขืก ืืื ืืึธืก:
ืงืึทื ืืืื ืืึทื ืืื, ืึทืืข ืืขื ืงืขื ืขื ืืืื ืืขืืื ืืื ืืืืืกืืื ื ืืืฉืืจืื. ืืืฉื, ksoftirqd
:
# record 999 times a second, or every 1ms with some offset so not to align exactly with timers
sudo perf record -C 11 -g -F 999
# take that recording and make a simpler stack trace.
sudo perf script 2>/dev/null | ./FlameGraph/stackcollapse-perf-ordered.pl | grep ksoftir -B 100
ืืึธ ืืขื ืขื ืื ืจืขืืืืืึทืื:
(ัะพัะฝะธ ัะปะตะดะพะฒ, ะบะพัะพััะต ะฒัะณะปัะดัั ะฟะพั
ะพะถะธะผะธ)
cadvisor;[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];entry_SYSCALL_64_after_swapgs;do_syscall_64;sys_read;vfs_read;seq_read;memcg_stat_show;mem_cgroup_nr_lru_pages;mem_cgroup_node_nr_lru_pages cadvisor;[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];entry_SYSCALL_64_after_swapgs;do_syscall_64;sys_read;vfs_read;seq_read;memcg_stat_show;mem_cgroup_nr_lru_pages;mem_cgroup_node_nr_lru_pages cadvisor;[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];entry_SYSCALL_64_after_swapgs;do_syscall_64;sys_read;vfs_read;seq_read;memcg_stat_show;mem_cgroup_iter cadvisor;[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];entry_SYSCALL_64_after_swapgs;do_syscall_64;sys_read;vfs_read;seq_read;memcg_stat_show;mem_cgroup_nr_lru_pages;mem_cgroup_node_nr_lru_pages cadvisor;[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];entry_SYSCALL_64_after_swapgs;do_syscall_64;sys_read;vfs_read;seq_read;memcg_stat_show;mem_cgroup_nr_lru_pages;mem_cgroup_node_nr_lru_pages ksoftirqd/11;ret_from_fork;kthread;kthread;smpboot_thread_fn;smpboot_thread_fn;run_ksoftirqd;__do_softirq;net_rx_action;ixgbe_poll;ixgbe_clean_rx_irq;napi_gro_receive;netif_receive_skb_internal;inet_gro_receive;bond_handle_frame;__netif_receive_skb_core;ip_rcv_finish;ip_rcv;ip_forward_finish;ip_forward;ip_finish_output;nf_iterate;ip_output;ip_finish_output2;__dev_queue_xmit;dev_hard_start_xmit;ipip_tunnel_xmit;ip_tunnel_xmit;iptunnel_xmit;ip_local_out;dst_output;__ip_local_out;nf_hook_slow;nf_iterate;nf_conntrack_in;generic_packet;ipt_do_table;set_match_v4;ip_set_test;hash_net4_kadt;ixgbe_xmit_frame_ring;swiotlb_dma_mapping_error;hash_net4_test ksoftirqd/11;ret_from_fork;kthread;kthread;smpboot_thread_fn;smpboot_thread_fn;run_ksoftirqd;__do_softirq;net_rx_action;gro_cell_poll;napi_gro_receive;netif_receive_skb_internal;inet_gro_receive;__netif_receive_skb_core;ip_rcv_finish;ip_rcv;ip_forward_finish;ip_forward;ip_finish_output;nf_iterate;ip_output;ip_finish_output2;__dev_queue_xmit;dev_hard_start_xmit;dev_queue_xmit_nit;packet_rcv;tpacket_rcv;sch_direct_xmit;validate_xmit_skb_list;validate_xmit_skb;netif_skb_features;ixgbe_xmit_frame_ring;swiotlb_dma_mapping_error;__dev_queue_xmit;dev_hard_start_xmit;__bpf_prog_run;__bpf_prog_run
ืขืก ืืขื ืขื ืึท ืคึผืืึทืฅ ืคืื ืืื ืื ืืึธ, ืึธืืขืจ ืื ืืืืคึผื ืืึทื ืืื ืึทื ืืืจ ืืขืคึฟืื ืขื ืื "ืงืึทืืืืืกืึธืจ ืืืืืขืจ ืงืกืึธืคืืจืงื" ืืืกืืขืจ ืืืึธืก ืืืจ ืืขืืขื ืคืจืืขืจ ืืื ืื ICMP ืืจืืืกืขืจ. ืืืืก ืืืื ื ืขืก?
ืืขืืขืจ ืฉืืจื ืืื ืึท ืงืคึผื ืฉืคึผืืจ ืืื ืึท ืกืคึผืขืฆืืคืืฉ ืคืื ื ืืื ืฆืืื. ืืขืืขืจ ืจืืคื ืึทืจืึธืคึผ ืื ืึธื ืืืืื ืืืืฃ ืึท ืฉืืจื ืืื ืืคืืขืฉืืืื ืืืจื ืึท ืกืขืืืงืึธืืึธื. ืืื ืื ืืืื ืคืื ืื ืฉืืจืืช ืืืจ ืืขื ืื ืกืืกืงืึทื ืืขืจืืคื: read(): .... ;do_syscall_64;sys_read; ...
. ืึทืืื ืงืึทืืืืืกืึธืจ ืกืคึผืขื ืื ืึท ืคึผืืึทืฅ ืคืื ืฆืืื ืืืืฃ ืื ืกืืกืืขื ืจืืคื read()
ืฉืืึทืืืช ืฆื ืคืึทื ืืงืฉืึทื ื mem_cgroup_*
(ืฉืคึผืืฅ ืคืื ืจืืคื ืึธื ืืืืื / ืกืืฃ ืคืื ืฉืืจื).
ืขืก ืืื ืืืืึทืงืืืขื ืฆื ืืขื ืืื ืึท ืจืืคื ืฉืคึผืืจ ืืืึธืก ืคึผืื ืงื ืืื ืืืืขื ืขื, ืึทืืื ืืึธืื ืืื ืื ืืืืคื strace
ืืื ืืึธืืืจ ืืขื ืืืึธืก ืงืึทืืืืืกืึธืจ ืืื ืืื ืืขืคึฟืื ืขื ืกืืกืืขื ืจืืคื ืืขืจ ืืื 100ms:
theojulienne@kube-node-bad ~ $ sudo strace -p 10137 -T -ff 2>&1 | egrep '<0.[1-9]'
[pid 10436] <... futex resumed> ) = 0 <0.156784>
[pid 10432] <... futex resumed> ) = 0 <0.258285>
[pid 10137] <... futex resumed> ) = 0 <0.678382>
[pid 10384] <... futex resumed> ) = 0 <0.762328>
[pid 10436] <... read resumed> "cache 154234880nrss 507904nrss_h"..., 4096) = 658 <0.179438>
[pid 10384] <... futex resumed> ) = 0 <0.104614>
[pid 10436] <... futex resumed> ) = 0 <0.175936>
[pid 10436] <... read resumed> "cache 0nrss 0nrss_huge 0nmapped_"..., 4096) = 577 <0.228091>
[pid 10427] <... read resumed> "cache 0nrss 0nrss_huge 0nmapped_"..., 4096) = 577 <0.207334>
[pid 10411] <... epoll_ctl resumed> ) = 0 <0.118113>
[pid 10382] <... pselect6 resumed> ) = 0 (Timeout) <0.117717>
[pid 10436] <... read resumed> "cache 154234880nrss 507904nrss_h"..., 4096) = 660 <0.159891>
[pid 10417] <... futex resumed> ) = 0 <0.917495>
[pid 10436] <... futex resumed> ) = 0 <0.208172>
[pid 10417] <... futex resumed> ) = 0 <0.190763>
[pid 10417] <... read resumed> "cache 0nrss 0nrss_huge 0nmapped_"..., 4096) = 576 <0.154442>
ืืื ืืืจ ืงืขื ืืขืจืืืึทืจืื, ืืืจ ืืขื ืคึผืึทืืขืืขื ืจืืคื ืืึธ read()
. ืคืื ืื ืืื ืืึทืื ืคืื ืืืืขื ืขื ืึทืคึผืขืจืืืฉืึทื ื ืืื ืงืึธื ืืขืงืกื mem_cgroup
ืขืก ืืื ืงืืึธืจ ืึทื ืื ืืฉืึทืืึทื ืืืฉืื read()
ืึธืคึผืฉืืงื ืฆื ืืขืจ ืืขืงืข memory.stat
, ืืืึธืก ืืืืืื ืืึผืจืื ืืึทื ืืฅ ืืื ืงืืจืืคึผ ืืืืึทืฅ (ืืึธืงืงืขืจ ืก ืืืื ืืคืืขืืื ืืขืจืืงืืื ืืขืื ืึธืืึธืืืข). ืื ืงืึทืืืืืกืึธืจ ืืขืฆืืึทื ืงืืืขื ืงืืขื ืืขื ืืขืงืข ืฆื ืืึทืงืืืขื ืืื ืคึฟืึธืจืืึทืฆืืข ืืึทื ืืฅ ืืื ืคึฟืึธืจืืึทืฆืืข ืคึฟืึทืจ ืงืึทื ืืืื ืขืจื. ืืึธืืืจ ืงืึธื ืืจืึธืืืจื ืฆื ืืขืจ ืงืขืจื ืึธืืขืจ ืงืึทืืืืืกืึธืจ ืืื ืขืคึผืขืก ืืืืืขืจืืื:
theojulienne@kube-node-bad ~ $ time cat /sys/fs/cgroup/memory/memory.stat >/dev/null
real 0m0.153s
user 0m0.000s
sys 0m0.152s
theojulienne@kube-node-bad ~ $
ืืืฆื ืืืจ ืงืขื ืขื ืจืขืคึผืจืึธืืืฆืืจื ืืขื ืืฉืืง ืืื ืคึฟืึทืจืฉืืืื ืึทื ืื ืืื ืืงืก ืงืขืจื ืืื ืคืืืกืื ื ืึท ืคึผืึทืืึทืืึทืืืฉื.
ืคืืจืืืืก ืืื ืื ืืืืขื ืขื ืึธืคึผืขืจืึทืฆืืข ืึทืืื ืคึผืึทืืขืืขื?
ืืื ืืขื ืืื ืข, ืขืก ืืื ืคืื ืืจืื ืืขืจ ืฆื ืืขืคึฟืื ืขื ืึทืจืืืงืืขื ืคืื ืื ืืขืจืข ื ืืฆืขืจืก ืืืขืื ืขื ืืขื ืคึผืจืึธืืืขืืก. ืืื ืขืก ืคืืจืงืขืจื ืืืืก, ืืื ืื ืงืึทืืืืืกืึธืจ ืืจืึทืงืขืจ ืืขื ืืฉืืง ืืื ืืขืืืืื ืืื
ืื ืคึผืจืึธืืืขื ืืื ืึทื cgroups ื ืขืืขื ืืื ืืฉืืื ืืึผืจืื ืืึทื ืืฅ ืืื ืื ื ืึทืืขืกืคึผืึทืกืข (container). ืืืขื ืึทืืข ืคึผืจืึทืกืขืกืึทื ืืื ืืขื cgroup ืึทืจืืืกืืึทื ื, Docker ืจืืืืกืื ืื ืืึผืจืื cgroup. ืึธืืขืจ, "ืืืงืึธืจื" ืืื ื ืืฉื ืืืืื ืคึผืจืึธืฆืขืก ืืึผืจืื. ืืึธืืฉ ืืขืจ ืคึผืจืึธืฆืขืก ืืึผืจืื ืืื ืืื ื ืื ืืขืจ ืืขืืืืื ื, ืขืก ืืืืก ืึทื ืืขืจ ืงืขืจื ื ืึธื ืึทืกืืื ื ืงืึทืฉื ืืื ืืึทืื, ืึทืืึท ืืื ืืขื ืืจืืขืก ืืื ืื ืึธืืขืก (ืืืขืืืืืึทืืขืจ ืืื ืืขืงืข ืืขืืึทืืึทืืึท), ืืืึธืก ืืขื ืขื ืงืึทืฉื ืืื ืื ืืึผืจืื ืงืืจืืคึผ. ืคืื ืื ืคึผืจืึธืืืขื ืืึทืฉืจืืึทืืื ื:
zombie cgroups: cgroups ืืืึธืก ืืึธืื ืงืืื ืคึผืจืึทืกืขืกืึทื ืืื ืืึธืื ืฉืืื ืืืืกืืขืืขืงื, ืึธืืขืจ ื ืึธื ืืึธืื ืืืงืึธืจื ืึทืืึทืงืืืืื (ืืื ืืืื ืคืึทื, ืคึฟืื ืื ืืขืืจื ืงืึทืฉ, ืึธืืขืจ ืขืก ืงืขื ืขื ืืืื ืืืื ืึทืืึทืงืืืืื ืคึฟืื ืื ืืืึทื ืงืึทืฉ ืึธืืขืจ tmpfs).
ืื ืงืขืจื ืขื ืก ืืฉืขืง ืคืื ืึทืืข ืืืขืืขืจ ืืื ืื ืงืึทืฉ ืืืขื ืคืจืื ืึท cgroup ืงืขื ืขื ืืืื ืืืืขืจ ืคึผืึทืืขืืขื, ืึทืืื ืืขืจ ืคืืื ืคึผืจืึธืฆืขืก ืืื ืืืืกืืขืจืืืืืื: ืืืึทืจืื ืืื ืื ืืืขืืขืจ ืืขื ืขื ืืขืืขืื ืืืืืขืจ, ืืื ืืขืกืึธืฃ ืืืกืืขืงื ืื cgroup ืืืขื ืืขืจ ืืึผืจืื ืืื ืึทืงืฉืื ืืืจืฃ. ืืื ืืขื ืคืื ื, cgroup ืืื ื ืึธื ืืขื ืืืขื ืืื ืืฉืืื ืืืขื ืงืึทืืขืงืืื ื ืกืืึทืืืกืืืง.
ืคึฟืื ืึท ืคืึธืจืฉืืขืืื ื ืกืืึทื ืืคึผืืื ื, ืืื ืืงืจืื ืืืงืึธืจื ืคึฟืึทืจ ืคืึธืจืฉืืขืืื ื: ืคืึทืจืืืืขืจื ืื ืขืจืฉื ืงืืื ืึทืคึผ ืืืจื ืืึธืื ืขืืืขืืข ืงืึทืฉื ืืึผืจืื ืืื ืืขืจ. ืืึธืก ืืื ืืื. ืืืขื ืืขืจ ืงืขืจื ื ืืฆื ืื ืืขืฆืืข ืคืื โโืื ืงืึทืฉื ืืึผืจืื, ืื cgroup ืืื ืืืืขื ืืฉืึทืืืึทืื ืงืืืจื, ืึทืืื ืขืก ืงืขื ื ืืฉื ืืืื ืืขืจืืคื ืึท "ืจืื ืขื". ืฆืื ืืึทืืืืขืจื, ืื ืกืคึผืขืฆืืคืืฉ ืืืคึผืืึทืืขื ืืืืฉืึทื ืคืื ืื ืืืื ืืขืงืึทื ืืืึทื memory.stat
ืืื ืืขื ืงืขืจื ืืืขืจืกืืข (4.9), ืงืึทืืืืื ื ืืื ืื ืจืืืืง ืกืืืข ืคืื โโโโืืืงืึธืจื ืืืืฃ ืืื ืืืขืจ ืกืขืจืืืขืจืก, ืขืก ื ืขืื ืคืื ืืขืจ ืฆืืื ืฆื ืืืงืขืจื ืื ืืขืฆืืข ืงืึทืฉื ืืึทืื ืืื ืืืกืืขืงื ืงืืจืืคึผ ืืึทืืืื.
ืขืก ืืืจื ืก ืืืืก ืึทื ืขืืืขืืข ืคืื โโืืื ืืืขืจ ื ืึธืืื ืืึธืื ืึทืืื ืคืืืข ืงืืจืืคึผืข ืืึทืืืื ืึทื ืื ืืืืขื ืขื ืืื ืืืืืึทื ืกื ืืงืกืื ืึท ืจืืข.
ืื ืืืึธืจืงืึทืจืึธืื ื ืคึฟืึทืจ ืื ืงืึทืืืืืกืึธืจ ืึทืจืืืกืืขืื ืืื ืฆื ืืืืื ืคืจืื ืืขืืจืืขืก / ืื ืึธืืขืก ืงืึทืฉืื ืืืืขืจ ืื ืกืืกืืขื, ืืืึธืก ืืืืื ืืืืืึทื ืืืฅ ืืืืขื ืขื ืืืืืึทื ืกื ืืื ื ืขืฅ ืืืืืึทื ืกื ืืืืฃ ืืขืจ ืืึทืืขืืึธืก, ืืื ื ืงืืึธืจ ืื ืงืึทืฉ ืืืจื ืก ืืืืฃ ืื ืงืึทืฉื ืงืืจืืคึผ ืืึทืืื ืืืขืืขืจ ืืื ืืื ืืขื ืขื ืืืื ืืืคืจืืื. ืืึธืก ืืื ื ืืฉื ืึท ืืืืืื ื, ืึธืืขืจ ืขืก ืงืึทื ืคืขืจืื ืื ืืจืื ื ืคืื ืืขื ืคึผืจืึธืืืขื.
ืขืก ืคืืจืงืขืจื ืืืืก ืึทื ืืื ื ืืึทืขืจ ืงืขืจื ืืืขืจืกืืขืก (4.19+) ืจืืคื ืคืึธืจืฉืืขืืื ื ืืื ืืืคึผืจืืืื memory.stat
, ืึทืืื ืกืืืืืฉืื ื ืฆื ืืขื ืงืขืจื ืคืึทืจืคืขืกืืืงื ืื ืคึผืจืึธืืืขื. ืืื ืืขืจ ืืขืืืืงืขืจ ืฆืืื, ืืืจ ืืึธืื ืืืฉืืจืื ืฆื ืืขืืขืงื ืคึผืจืึธืืืขืืึทืืืง ื ืึธืืื ืืื ืงืืืขืจื ืขืืขืก ืงืืึทืกืืขืจื, ืืจืืืคืึทืื ืืืกืืขืงื ืืื ืืื ืจืขืืึธืึธื ืืื. ืืืจ ืงืึทื ืึทืืข ืื ืงืืึทืกืืขืจื, ืืขืคืื ืขื ื ืึธืืื ืืื ืืืื ืืขื ืื ืืืืืึทื ืกื ืืื ืจืขืืึธืึธื ืืื. ืืึธืก ืืึธื ืืื ืื ืฆืืื ืฆื ืืขืจืืืึทื ืืืงื ืื ืึทืก ืืืืฃ ืื ืจืืขื ืกืขืจืืืขืจืก.
ืกืืืืื ื ืึทืจืืืฃ
ืืืืึทื ืืขื ืืฉืืง ืคืืจืฉืืืคื RX NIC ืจืื ืคึผืจืึทืกืขืกืื ื ืคึฟืึทืจ ืืื ืืขืจืืขืจ ืคืื ืืืืืกืขืงืึทื ืื, ืขืก ืกืืืืึทืืืืื ืืึทืกืื ืืขืคึฟืืจื ืืืืืข ืืืื ืืืืืึทื ืกื ืืืืฃ ืงืืจืฅ ืงืึทื ืขืงืฉืึทื ื ืืื ืืืื ืคึฟืึทืจืืื ืืื ื ืืืืืึทื ืกื, ืึทืืึท ืืื ืฆืืืืฉื MySQL ืจืืงืืืขืก ืืื ืขื ืืคืขืจ ืคึผืึทืงืืฅ.
ืคืืจืฉืืื ื ืืื ืืืื ืืืื ืื ื ืื ืคืึธืจืฉืืขืืื ื ืคืื ืื ืืขืจืกื ืคืื ืืึทืืขื ืืึทื ืกืืกืืขืืขื, ืึทืืึท ืืื Kubernetes, ืืื ืงืจืืืืฉ ืคึฟืึทืจ ืื ืจืืืืืึทืืืืึทืื ืืื ืืืืงืืึทื ืคืื ืึทืืข ืกืขืจืืืืกืขืก ืืืืืจื ืืืืฃ ืืื. ืืขืืขืจ ืกืืกืืขื ืืืจ ืืืืคื ืืขื ืขืคืืฅ ืคืื ืงืืืขืจื ืขืืขืก ืคืึธืจืฉืืขืืื ื ืืืคึผืจืืืืืึทื ืฅ.
ืืงืืจ: www.habr.com