ืืื ืืงืก ืืื ืึท ืืจืืืก ื ืืืขืจ ืคืื ืืืฉืืจืื ืคึฟืึทืจ ืืืืึทืืื ื ืื ืงืขืจื ืืื ืึทืคึผืืึทืงืืืฉืึทื ื. ืจืืึฟ ืคืื ืืื ืืึธืื ืึท ื ืขืืึทืืืื ืคึผืจืึทื ืืืืฃ ืึทืคึผืืึทืงืืืฉืึทื ืคืึธืจืฉืืขืืื ื ืืื ืงืขื ืขื ื ืื ืืืื ืืขืืืืื ื ืืื ืคึผืจืึธืืืงืฆืืข.
ื ืคึผืึธืจ ืคืื ืืึธืจื ืฆืืจืืง ืขืก ืืื ืืขืืืขื - eBPF. ืขืก ืืืื ืขืก ืืขืืืขื ืฆื ืฉืคึผืืจ ืื ืงืขืจื ืืื ืืึทื ืืฆืขืจ ืึทืคึผืืึทืงืืืฉืึทื ื ืืื ื ืืืขืจืืง ืึธืืืืขืจืืขื ืืื ืึธื ืื ื ืืื ืฆื ืจืืืืื ืืืืื ืืื ืืึธืื ืืจืื-ืคึผืึทืจืืื ืืึทืืืฉืืื ืืื ืื ืงืขืจื.
ืขืก ืืขื ืขื ืฉืืื ืคืืืข ืึทืคึผืืึทืงืืืฉืึทื ืืืืืืึทืืื ืืืึธืก ื ืืฆื eBPF, ืืื ืืื ืืขื ืึทืจืืืงื ืืืจ ืืืขืื ืงืืงื ืืื ืืื ืฆื ืฉืจืืึทืื ืืืื ืืืืืขื ืข ืคึผืจืึธืคืืืื ื ืืืืืืึทืืื ืืืืืจื ืืืืฃ ืืขืจ ืืืืืืึธืืขืง . ืืขืจ ืึทืจืืืงื ืืื ืืืืืจื ืืืืฃ ืคืึทืงืืืฉ ืืขืฉืขืขื ืืฉื. ืืืจ ืืืขืื ืืืื ืคืื ืคึผืจืึธืืืขื ืฆื ืคืึทืจืจืืืื ืฆื ืืืืึทืื ืืื ืืืืืกืืื ื ืืืืืืึทืืื ืงืขื ืขื ืืืื ืืขืืืืื ื ืืื ืกืคึผืขืฆืืคืืฉ ืกืืืืึทืืืึธื ืก.
ืกืขืฃ ืืื ืคึผืึทืืขืืขื
ื ื ืืึทืข ืืึทืืขืืึธืก ืืื ืฆืืืขืืขืื ืฆื ืื Ceph ืงื ืืื. ื ืึธื ืืืืืจืืืืื ื ืขืืืขืืข ืคืื โโืื ืืึทืื ืฆื ืขืก, ืืืจ ืืืืขืจืงื ืึทื ืื ืืืืงืืึทื ืคืื ืคึผืจืึทืกืขืกืื ื ืฉืจืืึทืื ืจืืงืืืขืก ืืื ืคืื ื ืืืขืจืืงืขืจ ืืื ืืืืฃ ืื ืืขืจืข ืกืขืจืืืขืจืก.
ื ืื ืขื ืืขื ืื ืืขืจืข ืคึผืืึทืืคืึธืจืืก, ืืขืจ ืืึทืืขืืึธืก ืืขืืืืื ื bcache ืืื ืื ื ืืึทืข ืืื ืืงืก 4.15 ืงืขืจื. ืืึธืก ืืื ืืขืืืขื ืืขืจ ืขืจืฉืืขืจ ืืึธื ืึท ืืึทืืขืืึธืก ืคืื ืืขื ืงืึทื ืคืืืืขืจืืืฉืึทื ืืื ืืขื ืืฆื ืืึธ. ืืื ืืื ืืขื ืืึธืืขื ื ืขืก ืืื ืืขืืืขื ืงืืึธืจ ืึทื ืืขืจ ืืืึธืจืฆื ืคืื ืื ืคึผืจืึธืืืขื ืงืขื ืืขืึธืจืขืืืฉ ืืืื ืึทืืฅ.
ืื ืืืขืกืืึทืืืืืื ื ืื ืืึทืืขืืึธืก
ืืึธืืืจ ืึธื ืืืืื ืืื ืงืืงื ืืื ืืืึธืก ืืึทืคึผืึทื ื ืืื ืื ืกืขืฃ-ืึธืกื ืคึผืจืึธืฆืขืก. ืคึฟืึทืจ ืืขื ืืืจ ืืืขืื ื ืืฆื ะธ (ืืขืจ ืืืขืื ืืืึธืก ืืืจ ืงืขื ืขื ืืืืขื ืขื ):
ืื ืืืื ืืขืจืฆืืืื ืืื ืื ืึทื ืื ืคึฟืื ืงืฆืืข fdatasync() ืคืืจืืจืืื ืึท ืคึผืืึทืฅ ืคืื ืฆืืื ืฆื ืฉืืงื ืึท ืืงืฉื ืฆื ืคืึทื ืืงืฉืึทื ื generic_make_request(). ืืขื ืืืื ืึทื ืจืืึฟ ืืกืชึผืื ืื ืืจืื ื ืคืื ืื ืคึผืจืึธืืืขืืก ืืื ืขืจืืขืฅ ืึทืจืืืก ืื ืึธืกื ืืืืืึทื ืืื. ืืึธืก ืงืขื ืืืื ืื ืงืขืจื ืึธืืขืจ ืืืกืงืก. ืื ืืึธืกืืึทื ืจืขืืืืืึทื ืืขืืืืื ืึท ืืืื ืืืืืึทื ืกื ืืื ืคึผืจืึทืกืขืกืื ื ืจืืงืืืขืก ืืืจื bcache ืืืกืงืก.
ืืืขื ืงืึธื ืืจืึธืืืจื ืืขื ืืึทืืขืืึธืก, ืืืจ ืืขืคึฟืื ืขื ืึทื ืื systemd-udevd ืืืืืึทื ืงืึทื ืกืืื ืึท ืืจืืืก ืกืืืข ืคืื โโืงืคึผื ืฆืืื - ืืืขืื 20% ืืืืฃ ืขืืืขืืข ืงืึธืจืขืก. ืืึธืก ืืื ืืึธืื ืข ื ืึทืืืจ, ืึทืืื ืืืจ ืืึทืจืคึฟื ืฆื ืืขืคึฟืื ืขื ืืืืก ืืืึธืก. ืืื ื Systemd-udevd ืึทืจืืขื ืืื ืืืขืืืขื ื, ืืืจ ืืึทืฉืืึธืกื ืฆื ืงืืงื ืืื ืืื ืืืขืืืึทืื ืืึธื ืืืึธืจ. ืขืก ืืืจื ืก ืืืืก ืึทื ืึท ืืจืืืก ื ืืืขืจ ืคืื ืขื ืืขืจืื ืืขื ืืขืฉืขืขื ืืฉื ืืขื ืขื ืืืฉืขื ืขืจืืืืึทื ืคึฟืึทืจ ืืขืืขืจ ืืืึธืง ืืืื ืืื ืื ืกืืกืืขื. ืืึธืก ืืื ืืึทื ืฅ ืืืืขืืืืื ืืืขื, ืึทืืื ืืืจ ืืืขืื ืืึธืื ืฆื ืงืืงื ืืื ืืืึธืก ืืืฉืขื ืขืจืืืฅ ืึทืืข ืื ืืขืฉืขืขื ืืฉื.
ื ืืฆื ืื BCC Toolkit
ืืื ืืืจ ืืึธืื ืฉืืื ืืขืคืื ืขื, ืืขืจ ืงืขืจื (ืืื ืื ืกืขืฃ ืืืืืึทื ืืื ืื ืกืืกืืขื ืจืืคื) ืกืคึผืขื ืื ืึท ืคึผืืึทืฅ ืคืื ืฆืืื ืืื generic_make_request(). ืืื ืก ืคึผืจืืืืจื ืฆื ืืขืกืื ืื ืืืืงืืึทื ืคืื ืืขื ืคึฟืื ืงืฆืืข. ืืื ืขืก ืืื ืฉืืื ืึท ืืืื ืืขืจืืขื ื ืืฆื - ืคืื ืงืฆืืึธื ืืจื. ืืืจ ืืืขืื ืฉืคึผืืจ ืื ืืืืืึทื ืืืจื ืืืื PID ืืื ืึท 1 ืจืืข ืืขืืึทืืขื ืฆืืืืฉื ืึทืืืคึผืืฅ ืืื ืจืขืืืืืึทื ืื ืจืขืืืืืึทื ืืื ืืืืืกืขืงืึทื ืื.
ืืขืจ ืฉืืจืื ืืืืฉืึทืืืึทืื ืึทืจืืขื ืืขืฉืืืื ื. ืื ืขืก ืืื ืืื ืคืึธืจื ืื ืืงืฉื ืฆื ืื ืืืื ืืจืืืืืขืจ ืจืื.
ืืงืึทืืฉืข ืืื ืึท ืงืึธืืคึผืืขืงืก ืืืื ืืืึธืก ืึทืงืฉืื ืืืฉืืืื ืคืื ืืจืื ืืืกืงืก:
- ืืึทืงืื ื ืืืื (ืงืึทืฉื ืืืกืง), ืืื ืืขื ืคืึทื ืขืก ืืื ืึท ืคึผืึทืืขืืขื ืืื;
- ืงืึทืืฉืื ื ืืืื (ืงืึทืืฉืื ื ืืืกืง), ืืึธ ืืึธืก ืืื ืืืื ืฆืขืืืืืื ื ืคืื ืื NVMe ืืืื;
- ืื bcache ืืืืจืืืึทื ืืืื ืืื ืืืึธืก ืื ืึทืคึผืืึทืงืืืฉืึทื ืืืืคื.
ืืืจ ืืืืกื ืึทื ืื ืืจืึทื ืกืืืกืืข ืคืื โโืื ืืขืื ืืื ืคึผืึทืืขืืขื, ืึธืืขืจ ืคึฟืึทืจ ืืืึธืก ืคืื ืื ืืขืืืืกืขืก? ืืืจ ืืืขืื ืืึทื ืืืขื ืืื ืืขื ืึท ืืืกื ืฉืคึผืขืืขืจ.
ืืืจ ืืืฆื ืืืืกื ืึทื ืืืขืืืขื ื ืืขื ืขื ืืกืชึผืื ืฆื ืคืึทืจืฉืึทืคื ืคึผืจืึธืืืขืืก. ืืขืคึฟืื ืขื ืืืึธืก ืคึผืื ืงื ื ืืืืขืจ ืืืจ ืืื ื ืืฉื ืึทืืื ืืจืื ื. ืืื ืก ืืืขืจื ืขืืขื ืึทื ืืึธืก ืืื ืึท ืกืึธืจื ืคืื ืืืืืืืืืจื ืืืึธืก ืืื ืคึผืืจืืึทืืืงืื ืืึธื ืืฉื. ืืื ืก ืืขื ืืืึธืก ืืื ืคืื ืืืืืืืืืจื ืืืืคื ืืืืฃ ืื ืกืืกืืขื ื ืืฆื ืึท ืฉืจืืคื execsnoop ืคืื ืื ืืขืืืข . ืืื ืก ืืืืคื ืขืก ืืื ืฉืืงื ืื ืจืขืืืืืึทื ืฆื ืึท ืืขืงืข.
ืืืฉื ืืื ืืึธืก:
/usr/share/bcc/tools/execsnoop | tee ./execdump
ืืืจ ืืืขืื ื ืืฉื ืืืืึทืื ืื ืคืื ืคึผืจืึธืืืงืฆืืข ืคืื โโืขืงืกืขืงืกื ืึธืึธืคึผ ืืึธ, ืึธืืขืจ ืืืื ืฉืืจื ืคืื ืืื ืืขืจืขืก ืฆื ืืื ืื ืืขืงืืงื ืืื ืืึธืก:
sh 1764905 5802 0 sudo arcconf getconfig 1 AD | grep Temperature | awk -F '[:/]' '{print $2}' | sed 's/^ ([0-9]*) C.*/1/'
ืื ืืจืื ืืืึทื ืืื ืื PPID (ืคืึธืืขืจ PID) ืคืื ืืขื ืคึผืจืึธืฆืขืก. ืืขืจ ืคึผืจืึธืฆืขืก ืืื PID 5802 ืืื ืืขืืืขื ืืืื ืขืจ ืคืื ืื ืคึฟืขืืขื ืคืื ืืื ืืืขืจ ืืึธื ืืืึธืจืื ื ืกืืกืืขื. ืืืขื ืงืึธื ืืจืึธืืืจืื ื ืื ืงืึทื ืคืืืืขืจืืืฉืึทื ืคืื ืื ืืึธื ืืืึธืจืื ื ืกืืกืืขื, ืขืจืึธืื ืืึทืก ืคึผืึทืจืึทืืขืืขืจืก ืืขื ืขื ืืขืคึฟืื ืขื. ืื ืืขืืคึผืขืจืึทืืืจ ืคืื ืื HBA ืึทืืึทืคึผืืขืจ ืืื ืืขื ืืืขื ืืขืืขืจ 30 ืกืขืงืื ืืขืก, ืืืึธืก ืืื ืคืื ืืขืจ ืึธืคื ืืื ื ืืืืืง. ื ืึธื ืืฉืึทื ืืื ื ืื ืืฉืขืง ืืขืืึทืืขื ืฆื ืึท ืืขืจ ืืืื ืขืจ, ืืืจ ืืขืคึฟืื ืขื ืึทื ืื ืืืืืึทื ืกื ืคืื ืื ืืขืื ืคึผืจืึทืกืขืกืื ื ืืืืฃ ืืขื ืืึทืืขืืึธืก ื ืื ืืขืจ ืืขืฉืืื ืขื ืืืืก ืงืึทืืคึผืขืจื ืืื ืื ืืขืจืข ืืื ืืช.
ืึธืืขืจ ืขืก ืืื ื ืึธื ืืืงืืึธืจ ืืืึธืก ืื bcache ืืืื ืืื ืืขืืืขื ืึทืืื ืคึผืึทืืขืืขื. ืืืจ ืืึธืื ืฆืืืขืืจืืื ืึท ืคึผืจืึธืืข ืคึผืืึทืืคืึธืจืืข ืืื ืึทื ืืืืขื ืืงืึทื ืงืึทื ืคืืืืขืจืืืฉืึทื ืืื ืืขืคืจืืืืื ืฆื ืจืขืคึผืจืึธืืืฆืืจื ืืขื ืคึผืจืึธืืืขื ืืืจื ืืืืคื ืคืืึธ ืืืืฃ bcache, ืคึผืืจืืึทืืืงืื ืืืืคื udevadm ืฆืื ืื ืฆื ืืืฉืขื ืขืจืืื ืืืขืืืขื ื.
ืฉืจืืืื ืืงืง-ืืืืืจื ืืืฉืืจืื
ืืึธืืืจ ืคึผืจืืืืจื ืฆื ืฉืจืืึทืื ืึท ืคึผืฉืื ื ืืฆื ืฆื ืฉืคึผืืจ ืืื ืืืืึทืื ืื ืกืืึธืืึทืกื ืงืึทืืืก generic_make_request(). ืืืจ ืืขื ืขื ืืืื ืืื ืืขืจืขืกืืจื ืืื ืื ื ืึธืืขื ืคืื ืื ืคืึธืจ ืคึฟืึทืจ ืืืึธืก ืื ืคึฟืื ืงืฆืืข ืืื ืืขืจืืคื.
ืืขืจ ืคึผืืึทื ืืื ืคึผืฉืื:
- ืจืขืืืกืืจืืจื kprobe ืืืืฃ generic_make_request():
- ืืืจ ืจืึทืืขืืืขื ืื ืืืกืง ื ืึธืืขื ืืื ืืึผืจืื, ืฆืืืจืืืืขื ืืืจื ืื ืคึฟืื ืงืฆืืข ืึทืจืืืืขื ื;
- ืืืจ ืจืึทืืขืืืขื ืื ืฆืืื ืฉืืขืืคึผื.
- ืจืขืืืกืืจืืจื kretprobe ืคึฟืึทืจ ืฆืืจืืงืงืขืจ ืคืื generic_make_request():
- ืืืจ ืืึทืงืืืขื ืื ืงืจืึทื ื ืฆืืื ืฉืืขืืคึผื;
- ืืืจ ืงืืงื ืคึฟืึทืจ ืื ืืขืจืืืขืืืขื ืืืืขืกืืึทืืคึผ ืืื ืคืึทืจืืืืึทืื ืขืก ืืื ืื ืงืจืึทื ื;
- ืืืื ืืขืจ ืจืขืืืืืึทื ืืื ืืจืขืกืขืจ ืืื ืื ืกืคึผืขืกืืคืืขื ืืืื ืขืจ, ืืืจ ืืขืคึฟืื ืขื ืื ืืขืจืืืขืืืขื ืืืกืง ื ืึธืืขื ืืื ืืืืึทืื ืขืก ืืืืฃ ืื ืืืึธืงืืึทื.
ืงืคึผืจืึธืืขืก ะธ kretprobes ื ืืฆื ืึท ืืจืืืงืคึผืืื ื ืืขืงืึทื ืืืึทื ืฆื ืืืืฉื ืคืื ืงืฆืืข ืงืึธื ืืืืฃ ืื ืคืืืขื. ืืืจ ืงืขื ืขื ืืืืขื ืขื ะธ ืึทืจืืืงื ืืืืฃ ืืขื ืืขืืข. ืืืื ืืืจ ืงืืง ืืื ืื ืงืึธื ืคืื ืคืึทืจืฉืืื ืืืืืืึทืืื ืืื , ืืขืืึธืื ืืืจ ืงืขื ืขื ืืขื ืึทื ืืื ืืึธืื ืึทื ืืืืขื ืืงืึทื ืกืืจืืงืืืจ. ืึทืืื ืืื ืืขื ืึทืจืืืงื ืืืจ ืืืขืื ืืึธืคึผืงืขื ืคึผืึทืจืกืื ื ืฉืจืืคื ืึทืจืืืืขื ืื ืืื ืืึทื ืืืืฃ ืฆื ืื BPF ืคึผืจืึธืืจืึทื ืืื.
ืืขืจ eBPF ืืขืงืกื ืืื ืื ืคึผืืืืึธื ืฉืจืืคื ืงืืงื ืืื ืืึธืก:
bpf_text = โโโ # Here will be the bpf program code โโโ
ืฆื ืืืขืงืกื ืืึทืื ืฆืืืืฉื ืคืึทื ืืงืฉืึทื ื, eBPF ืืืืื ื ืืฆื . ืืืจ ืืืขืื ืืึธื ืืึธืก ืืขืืืข. ืืืจ ืืืขืื ื ืืฆื ืืขื ืคึผืจืึธืฆืขืก PID ืืื ืืขืจ ืฉืืืกื, ืืื ืืขืคืื ืืจื ืื ืกืืจืืงืืืจ ืืื ืื ืืืขืจื:
struct data_t {
u64 pid;
u64 ts;
char comm[TASK_COMM_LEN];
u64 lat;
char disk[DISK_NAME_LEN];
};
BPF_HASH(p, u64, struct data_t);
BPF_PERF_OUTPUT(events);
ืืึธ ืืืจ ืจืขืืืกืืจืืจื ืึท ืืึทืฉ ืืืฉ ืืขืจืืคื p, ืืื ืฉืืืกื ืืืคึผ ืืงืกื ืืืงืก ืืื ืึท ืืืขืจื ืคืื ืืืคึผ ืกืืจืืงืืืจ ืืึทืืึท_ื. ืืขืจ ืืืฉ ืืืขื ืืืื ืืืจืขืืืืื ืืื ืืขื ืงืึธื ืืขืงืกื ืคืื ืืื ืืืขืจ BPF ืคึผืจืึธืืจืึทื. ืื BPF_PERF_OUTPUT ืืึทืงืจืึธื ืจืขืืืฉืืกืืขืจื ืื ืื ืืขืจ ืืืฉ ืืขืจืืคื ืืขืฉืขืขื ืืฉื, ืืืึธืก ืืื ืืขื ืืฆื ืคึฟืึทืจ ืืื ืืึทื ืืฆืขืจ ืคึผืืึทืฅ.
ืืืขื ืืขืกืื ืืืืืื ืฆืืืืฉื ืจืืคื ืึท ืคึฟืื ืงืฆืืข ืืื ืฆืืจืืงืงืืืขื ืคืื ืืื, ืึธืืขืจ ืฆืืืืฉื ืงืึทืืืก ืฆื ืคืึทืจืฉืืืขื ืข ืคืึทื ืืงืฉืึทื ื, ืืืจ ืืึทืจืคึฟื ืฆื ื ืขืืขื ืืื ืืฉืืื ืึทื ืื ืืืงืืืขื ืืึทืื ืืืื ืืขืืขืจื ืฆื ืืขืจ ืืขืืืืงืขืจ ืงืึธื ืืขืงืกื. ืืื ืื ืืขืจืข ืืืขืจืืขืจ, ืืืจ ืืึทืจืคึฟื ืฆื ืืขืืขื ืงืขื ืืืขืื ืื ืืขืืืขื ืคึผืึทืจืึทืืขื ืงืึทืืขืจ ืคืื ืคืึทื ืืงืฉืึทื ื. ืืืจ ืืึธืื ืื ืคืืืืงืืื ืฆื ืืขืกืื ืื ืืืืืึทื ืกื ืฆืืืืฉื ืจืืคื ืึท ืคึฟืื ืงืฆืืข ืืื ืืขื ืงืึธื ืืขืงืกื ืคืื ืืืื ืคึผืจืึธืฆืขืก ืืื ืฆืืจืืงืงืืืขื ืคืื ืืขื ืคึฟืื ืงืฆืืข ืืื ืืขื ืงืึธื ืืขืงืกื ืคืื ืื ืื ืืขืจ ืคึผืจืึธืฆืขืก, ืึธืืขืจ ืืึธืก ืืื ืืกืชึผืื ืึทืจืืืกืืขืืืึธืจืคื. ื ืืื ืืืึทืฉืคึผืื ืืึธ ืืืึธืื ืืืื , ืืื ืื ืืึทืฉ ืืืฉ ืฉืืืกื ืืื ืืึทืฉืืืื ืฆื ืึท ืืืึทืื ืฆื ืกืืจืืงืืืจ ืืขืื, ืืืึธืก ืจืืคืืขืงืก ืืืื ืืืกืง ืืขืื.
ืืืืึทืืขืจ, ืืืจ ืืึทืจืคึฟื ืฆื ืฉืจืืึทืื ืื ืงืึธื ืืืึธืก ืืืขื ืืืืคื ืืืขื ืื ืคึฟืื ืงืฆืืข ืืื ืืขืจ ืืขืจื ืขื ืืื ืืขืจืืคื:
void start(struct pt_regs *ctx, struct bio *bio) {
u64 pid = bpf_get_current_pid_tgid();
struct data_t data = {};
u64 ts = bpf_ktime_get_ns();
data.pid = pid;
data.ts = ts;
bpf_probe_read_str(&data.disk, sizeof(data.disk), (void*)bio->bi_disk->disk_name);
p.update(&pid, &data);
}
ืืึธ ืืขืจ ืขืจืฉืืขืจ ืึทืจืืืืขื ื ืคืื ืื ืืขืจืืคื ืคึฟืื ืงืฆืืข ืืืขื ืืืื ืกืึทืืกืืึทืืืืึทื ืืื ืื ืจืืข ืึทืจืืืืขื ื . ื ืึธื ืืขื, ืืืจ ืืึทืงืืืขื ืื PID ืคืื ืืขื ืคึผืจืึธืฆืขืก ืืื ืืขื ืงืึธื ืืขืงืกื ืคืื ืืืึธืก ืืืจ ืึทืจืืขืื, ืืื ืื ืงืจืึทื ื ืฆืืื ืกืืึทืืคึผ ืืื ื ืึทื ืึธืกืขืงืึธื ืืก. ืืืจ ืฉืจืืึทืื ืขืก ืึทืืข ืึทืจืึธืคึผ ืืื ืึท ืคืจืืฉ ืืืืกืืขืงืืืื ืกืืจืืงืืืจ ืืึทืืึท_ื ืืึทืื. ืืืจ ืืึทืงืืืขื ืื ืืืกืง ื ืึธืืขื ืคืื ืื ืกืืจืืงืืืจ ืืืึธ, ืฐื ืก ืฐืขืจ ื ืืืจืืืขืคืืจ ื ืืฒ ื ืจืืคื generic_make_request(), ืืื ืจืึทืืขืืืขื ืขืก ืืื ืืขืจ ืืขืืืืงืขืจ ืกืืจืืงืืืจ ืืึทืืข. ืื ืืขืฆืืข ืฉืจืื ืืื ืฆื ืืืืื ืึท ืคึผืึธืืืฆืืข ืฆื ืื ืืึทืฉ ืืืฉ ืืืึธืก ืืื ืืขืืืขื ืืขืจืืื ื ืคืจืืขืจ.
ืื ืคืืืืขื ืืข ืคึฟืื ืงืฆืืข ืืืขื ืืืื ืืขืจืืคึฟื ืืืืฃ ืฆืืจืืงืงืืืขื ืคืื generic_make_request():
void stop(struct pt_regs *ctx) {
u64 pid = bpf_get_current_pid_tgid();
u64 ts = bpf_ktime_get_ns();
struct data_t* data = p.lookup(&pid);
if (data != 0 && data->ts > 0) {
bpf_get_current_comm(&data->comm, sizeof(data->comm));
data->lat = (ts - data->ts)/1000;
if (data->lat > MIN_US) {
FACTOR
data->pid >>= 32;
events.perf_submit(ctx, data, sizeof(struct data_t));
}
p.delete(&pid);
}
}
ืื ืคึฟืื ืงืฆืืข ืืื ืขื ืืขื ืฆื ืื ืคืจืืขืจืืืงืข: ืืืจ ืืขืคึฟืื ืขื ืื PID ืคืื ืืขื ืคึผืจืึธืฆืขืก ืืื ืื ืืืืขืกืืึทืืคึผ, ืึธืืขืจ ืืึธื ื ืื ืึทืืึทืงืืื ืืืงืึธืจื ืคึฟืึทืจ ืื ื ืืึทืข ืืึทืื ืกืืจืืงืืืจ. ืึทื ืฉืืึธื, ืืืจ ืืืื ืื ืืึทืฉ ืืืฉ ืคึฟืึทืจ ืึท ืฉืืื ืืืืืกืืื ื ืกืืจืืงืืืจ ืืื ืื ืฉืืืกื == ืงืจืึทื ื PID. ืืืื ืื ืกืืจืืงืืืจ ืืื ืืขืคึฟืื ืขื, ืืืจ ืืขืคึฟืื ืขื ืื ื ืึธืืขื ืคืื ืื ืคืืืกื ืืืง ืคึผืจืึธืฆืขืก ืืื ืืืืื ืขืก ืฆื ืขืก.
ืื ืืืื ืขืจื ืืืขืจืจืืง ืืืจ ื ืืฆื ืืึธ ืืื ืืืจืฃ ืฆื ืืึทืงืืืขื ืื ืคืึธืืขื GID. ืืขื ืข. PID ืคืื ืื ืืืืคึผื ืคึผืจืึธืฆืขืก ืืืึธืก ืกืืึทืจืืขื ืืขื ืคืึธืืขื ืืื ืืขื ืงืึธื ืืขืงืกื ืคืื ืืืึธืก ืืืจ ืึทืจืืขืื. ืื ืคึฟืื ืงืฆืืข ืืืึธืก ืืืจ ืจืืคื ืงืขืจื ืืืืืข ืื GID ืคืื ืื ืคืึธืืขื ืืื ืืืื PID ืืื ืึท ืืืื 64-ืืืกื ืืืขืจื.
ืืืขื ืึทืจืืืกืคืืจื ืฆื ืื ืืืึธืงืืึทื, ืืืจ ืืขื ืขื ื ืืฉื ืืื ืืขืจืขืกืืจื ืืื ืืขื ืคืึธืืขื, ืึธืืขืจ ืืืจ ืืขื ืขื ืืื ืืขืจืขืกืืจื ืืื ืื ืืืืคึผื ืคึผืจืึธืฆืขืก. ื ืึธื ืงืึทืืคึผืขืจืื ื ืื ืจืืืึทืืืื ื ืคืึทืจืืึทืืื ืืื ืึท ืืขืืขืื ืฉืืืขื, ืืืจ ืคืึธืจื ืืื ืืืขืจ ืกืืจืืงืืืจ ืืึทืืข ืืื ืืึทื ืืฆืขืจ ืคึผืืึทืฅ ืืืจื ืืืฉ ืืขืฉืขืขื ืืฉื, ื ืึธื ืืืึธืก ืืืจ ืืืกืืขืงื ืื ืคึผืึธืืืฆืืข ืคืื p.
ืืื ืื ืคึผืืืืึธื ืฉืจืืคื ืืืึธืก ืืืขื ืืึธืื ืืขื ืงืึธื, ืืืจ ืืึทืจืคึฟื ืฆื ืคืึทืจืืืึทืื MIN_US ืืื FACTOR ืืื ืื ืคืึทืจืืึทืืื ืืจืขืฉืืึธืืืื ืืื ืฆืืื ืื ืืฅ, ืืืึธืก ืืืจ ืืืขืื ืคืึธืจื ืืืจื ืื ืึทืจืืืืขื ืื:
bpf_text = bpf_text.replace('MIN_US',str(min_usec))
if args.milliseconds:
bpf_text = bpf_text.replace('FACTOR','data->lat /= 1000;')
label = "msec"
else:
bpf_text = bpf_text.replace('FACTOR','')
label = "usec"
ืืืฆื ืืืจ ืืึทืจืคึฟื ืฆื ืฆืืืจืืืื ืื BPF ืคึผืจืึธืืจืึทื ืืืจื ืืื ืจืขืืืกืืจืืจื ืกืึทืืคึผืึทืื:
b = BPF(text=bpf_text)
b.attach_kprobe(event="generic_make_request",fn_name="start")
b.attach_kretprobe(event="generic_make_request",fn_name="stop")
ืืืจ ืืืขืื ืืืื ืืึธืื ืฆื ืืึทืฉืืืกื ืกืืจืืงืืืจ ืืึทืืึท_ื ืืื ืืื ืืืขืจ ืฉืจืืคื, ืึทื ืืขืจืฉ ืืืจ ืงืขื ืขื ื ืืฉื ืืืืขื ืขื ืขืคึผืขืก:
TASK_COMM_LEN = 16 # linux/sched.h
DISK_NAME_LEN = 32 # linux/genhd.h
class Data(ct.Structure):
_fields_ = [("pid", ct.c_ulonglong),
("ts", ct.c_ulonglong),
("comm", ct.c_char * TASK_COMM_LEN),
("lat", ct.c_ulonglong),
("disk",ct.c_char * DISK_NAME_LEN)]
ืื ืืขืฆืืข ืฉืจืื ืืื ืฆื ืึทืจืืืกืคืืจื ืืึทืื ืฆื ืื ืืืึธืงืืึทื:
def print_event(cpu, data, size):
global start
event = ct.cast(data, ct.POINTER(Data)).contents
if start == 0:
start = event.ts
time_s = (float(event.ts - start)) / 1000000000
print("%-18.9f %-16s %-6d %-1s %s %s" % (time_s, event.comm, event.pid, event.lat, label, event.disk))
b["events"].open_perf_buffer(print_event)
# format output
start = 0
while 1:
try:
b.perf_buffer_poll()
except KeyboardInterrupt:
exit()
ืืขืจ ืฉืจืืคื ืืื ืืื ืื ืืืฆื ืืื . ืืึธืืืจ ืคึผืจืืืืจื ืฆื ืืืืคื ืขืก ืืืืฃ ืึท ืคึผืจืึธืืข ืคึผืืึทืืคืึธืจืืข ืืื ืคืืึธ ืืื ืคืืืกื ืืืง, ืฉืจืืืื ืฆื bcache, ืืื ืจืืคื udevadm ืืึธื ืืืึธืจ:
ืฆืื ืกืืฃ! ืืืฆื ืืืจ ืืขื ืึทื ืืืึธืก ืืขืงืืงื ืืื ืึท ืกืืึธืืื ื ืืงืึทืืฉืข ืืืื ืืื ืคืืงืืืฉ ืึท ืกืืึธืืื ื ืจืืคื generic_make_request() ืคึฟืึทืจ ืึท ืงืึทืฉื ืืืกืง.
ืืจืึธืื ืืื ืื ืงืขืจื ืขื
ืืืึธืก ืคึผืื ืงื ืกืืึธืืื ื ืึทืจืึธืคึผ ืืขืฉืึทืก ืืขืื ืืจืึทื ืกืืืกืืข? ืืืจ ืืขื ืึทื ืื ืคืึทืจืืึทืืื ืึทืงืขืจื ืืคืืื ืืืืืขืจ ืื ืึธื ืืืื ืคืื ืืขืื ืึทืงืึทืื ืืื ื, ื.ื. ืึทืงืึทืื ืืื ื ืคืื ืึท ืกืคึผืขืฆืืคืืฉ ืืขืื ืคึฟืึทืจ ืืืืึทืืขืจ ืจืขืืืืืึทื ืคืื ืกืืึทืืืกืืืง ืืืืฃ ืขืก (/proc/diskstats ืึธืืขืจ iostat) ืืื ื ืืฉื ื ืึธื ืื ืืขืืืืื. ืืขื ืงืขื ืขื ืืืื ืืืืื ืืืขืจืึทืคืืื ืืืจื ืคืืืกื ืืืง ืืึธืกืืึทื ืืฉืขืช ืจืืคึผืจืึทืืืกืื ื ืื ืคึผืจืึธืืืขื, ืึธืืขืจ , ืืืึธืก ืืื ืืืืืจื ืืืืฃ ืื ืึธื ืืืื ืืื ืกืืฃ ืคืื ืืขืื ืึทืงืึทืื ืืื ื. ืงืืื ืคืื ืื ืืืืืืึทืืื ืืืขื ืืืืึทืื ืคึผืจืึธืืืขืืก ืคึฟืึทืจ ืจืืงืืืขืก ืฆื ืื ืงืึทืฉื ืืืกืง.
ืืืื ืืืจ ืงืืงื ืืื ืื ืคึฟืื ืงืฆืืข generic_make_request(), ืืขืืึธืื ืืืจ ืืืขืื ืืขื ืึทื ืืืืืขืจ ืื ืืขืื ืึทืงืึทืื ืืื ื ืืืืื, ืฆืืืื ืืขืจ ืคืึทื ืืงืฉืึทื ื ืืขื ืขื ืืขืจืืคื. ืขืจืฉืืขืจ - generic_make_request_checks(), ืคึผืขืจืคืึธืจืื ืืฉืขืงืก ืืืืฃ ืื ืืึทืืืฉืืืึทืืึทืื ืคืื ืื ืืขืื ืืืขืื ืื ืืืกืง ืกืขืืืื ืืก. ืกืขืงืื ืืข - , ืืืึธืก ืืื ืึท ืืฉืืงืึทืืืข ืึทืจืืืกืจืืคื :
ret = wait_event_interruptible(q->mq_freeze_wq,
(atomic_read(&q->mq_freeze_depth) == 0 &&
(preempt || !blk_queue_preempt_only(q))) ||
blk_queue_dying(q));
ืืื ืขืก, ืืขืจ ืงืขืจื ืืืืืฅ ืคึฟืึทืจ ืื ืจืื ืฆื ืึทื ืคืจืืื. ืืื ืก ืืขืกืื ืื ืคืึทืจืืึทืืื blk_queue_enter():
~# /usr/share/bcc/tools/funclatency blk_queue_enter -i 1 -m
Tracing 1 functions for "blk_queue_enter"... Hit Ctrl-C to end.
msecs : count distribution
0 -> 1 : 341 |****************************************|
msecs : count distribution
0 -> 1 : 316 |****************************************|
msecs : count distribution
0 -> 1 : 255 |****************************************|
2 -> 3 : 0 | |
4 -> 7 : 0 | |
8 -> 15 : 1 | |
ืขืก ืงืืงื ืืื ืืืจ ืืขื ืขื ื ืึธืขื ื ืฆื ืึท ืืืืืื ื. ืื ืคืึทื ืืงืฉืึทื ื ืืขื ืืฆื ืฆื ืคืจืืจื / ืื ืคืจืขืขืืข ืึท ืจืื ืืขื ืขื ะธ . ืืื ืืขื ืขื ืืขื ืืฆื ืืืขื ืขืก ืืื ื ืืืืืง ืฆื ืืืืฉื ืื ืืขืื ืจืื ืกืขืืืื ืืก, ืืืึธืก ืืขื ืขื ืคึผืึทืืขื ืืฉืึทืื ืืขืคืขืจืืขื ืคึฟืึทืจ ืจืืงืืืขืก ืืื ืืขื ืจืื. ืืืขื ืจืืคื blk_mq_freeze_queue() ืคึฟืื ืงืฆืืข ืืขืจ ืืึธืืืึทื ืง ืืื ืื ืงืจืึทืืึทื ืืื q->mq_freeze_depth. ื ืึธื ืืขื, ืืขืจ ืงืขืจื ืืืืจืื ืคึฟืึทืจ ืื ืจืื ืฆื ืืืืืืง ืืื .
ืื ืฆืืื ืขืก ื ืขืื ืฆื ืืืกืืขืงื ืืขื ืจืื ืืื ืขืงืืืืืืึทืืขื ื ืฆื ืืืกืง ืืืืืึทื ืกื, ืืืืึทื ืืขืจ ืงืขืจื ืืืืืฅ ืคึฟืึทืจ ืึทืืข ืงืื ืึทืคึผืขืจืืืฉืึทื ื ืฆื ืคืึทืจืขื ืืืงื. ืึทืืึธื ืื ืจืื ืืื ืืืืืืง, ืื ืกืขืืืื ืืก ืขื ืืขืจืื ืืขื ืืขื ืขื ืืขืืืขื ืื. ื ืึธื ืืืึธืก ืขืก ืืื ืืขืจืืคื , ืืขืงืจืขืืขื ืืื ื ืื ืืึธืืืึทื ืง freeze_depth.
ืืืฆื ืืืจ ืืืืกื ืืขื ืื ืฆื ืคืึทืจืจืืืื ืื ืกืืืืึทืฆืืข. ืื udevadm ืฆืื ืื ืืึทืคึฟืขื ื ืื ืกืขืืืื ืืก ืคึฟืึทืจ ืื ืืืึธืง ืืืื ืฆื ืืืื ืืขืืืขื ืื. ืื ืกืขืืืื ืืก ืืขื ืขื ืืืกืงืจืืืื ืืื ืื ืืืขืื ืึผืืืื. ืืืจ ืงืขื ืขื ืืขืคึฟืื ืขื ืืืึธืก ืกืขืืืื ืืก ืคืจืืื ืื ืจืื ืืืจื ืืจืืื ื ืฆื ืืืืฉื ืืื ืืืจื sysfs ืึธืืขืจ ืืืจื ืงืืงื ืืื ืื ืงืขืจื ืืงืืจ ืงืึธื. ืืืจ ืงืขื ืขื ืืืื ืคึผืจืืืืจื ืื BCC ื ืืฆื , ืืืึธืก ืืืขื ืจืขืืืืืึทื ืงืขืจื ืืื ืืึทื ืืฆืขืจ-ืกืคึผืึทืกืข ืกืืึทืง ืืจืึทืกืขืก ืคึฟืึทืจ ืืขืืขืจ ืจืืคื ืฆื ืื ืืืึธืงืืึทื blk_freeze_queue, ืคึฟืึทืจ ืืืึทืฉืคึผืื:
~# /usr/share/bcc/tools/trace blk_freeze_queue -K -U
PID TID COMM FUNC
3809642 3809642 systemd-udevd blk_freeze_queue
blk_freeze_queue+0x1 [kernel]
elevator_switch+0x29 [kernel]
elv_iosched_store+0x197 [kernel]
queue_attr_store+0x5c [kernel]
sysfs_kf_write+0x3c [kernel]
kernfs_fop_write+0x125 [kernel]
__vfs_write+0x1b [kernel]
vfs_write+0xb8 [kernel]
sys_write+0x55 [kernel]
do_syscall_64+0x73 [kernel]
entry_SYSCALL_64_after_hwframe+0x3d [kernel]
__write_nocancel+0x7 [libc-2.23.so]
[unknown]
3809631 3809631 systemd-udevd blk_freeze_queue
blk_freeze_queue+0x1 [kernel]
queue_requests_store+0xb6 [kernel]
queue_attr_store+0x5c [kernel]
sysfs_kf_write+0x3c [kernel]
kernfs_fop_write+0x125 [kernel]
__vfs_write+0x1b [kernel]
vfs_write+0xb8 [kernel]
sys_write+0x55 [kernel]
do_syscall_64+0x73 [kernel]
entry_SYSCALL_64_after_hwframe+0x3d [kernel]
__write_nocancel+0x7 [libc-2.23.so]
[unknown]
ืืืขืื ืึผืืืื ืืืืฉื ืืึทื ืฅ ืจืึทืจืขืื ืืื ืืืืฉืึทืืืึทืื ืืึธืก ืืึทืคึผืึทื ื ืืื ืึท ืงืึทื ืืจืึธืืื ืฉืืืืืขืจ. ืึทืืื ืืืจ ืืขื ืึทื ืืคืืื ืึทืคึผืืืืื ื ืื ืฉืืื ืืึทืฉืืืื ืืืึทืืืขืก ื ืึท ืคืึทืจืืจืขืกืขืจื ืืื ืื ืคืึทืจืืึทืืื ืืื ืืจืึทื ืกืคืขืจืื ื ืื ืืงืฉื ืคืื ืื ืึทืคึผืืึทืงืืืฉืึทื ืฆื ืื ืืืกืง. ืคืื ืงืืจืก, ืืืฉืขื ืขืจืืืืื ื ืืืขืื ืืขืฉืขืขื ืืฉื ืืืขื ืขืก ืืขื ืขื ืงืืื ืขื ืืขืจืื ืืขื ืืื ืื ืืืกืง ืงืึทื ืคืืืืขืจืืืฉืึทื (ืืืฉื, ืื ืืืื ืืื ื ืืฉื ืืึธืื ืืขื / ืืืกืงืึทื ืขืงืืื) ืืื ื ืืฉื ืึท ืืื ืคืืจ. ืึธืืขืจ, ืืืจ ืงืขื ืขื ืืขืืคึฟื ืื ืงืขืจื ื ืืฉื ืืึธื ืืื ืืืืืง ืึทืจืืขื ืืื ืคืจืืจื ืื ืืขืื ืจืื ืืืื ืขืก ืืื ื ืื ื ืืืืืง. ืจืืืืืง ืื ืกืืืืึทืฆืืข.
ืกืึธืฃ
eBPF ืืื ืึท ืืืืขืจ ืคืืขืงืกืึทืืึทื ืืื ืฉืืึทืจืง ืืขืฆืืึทื. ืืื ืืขื ืึทืจืืืงื ืืืจ ืืขืงืืงื ืืืืฃ ืืืื ืคึผืจืึทืงืืืฉ ืคืึทื ืืื ืืขืืึทื ืกืืจืืืืื ืึท ืงืืืื ืืืื ืคืื ืืืึธืก ืงืขื ืขื ืืืื ืืขืืื. ืืืื ืืืจ ืืขื ื ืืื ืืขืจืขืกืืจื ืืื ืืขืืืขืืึธืคึผืื ื BCC ืืืืืืึทืืื, ืขืก ืืื ืืืขืจื ืึท ืงืืง ืืื , ืืืึธืก ืืืฉืจืืืื ืื ืืึทืกืืงืก ืืขืืื ื.
ืขืก ืืขื ืขื ืื ืืขืจืข ืืฉืืงืึทืืืข ืืืืึทืืื ื ืืื ืคึผืจืึธืคืืืื ื ืืืฉืืจืื ืืืืืจื ืืืืฃ eBPF. ืืืื ืขืจ ืคืื ืืื - , ืืืึธืก ืึทืืึทืื ืืืจ ืฆื ืฉืจืืึทืื ืฉืืึทืจืง ืืืื-ืืืื ืขืจื ืืื ืงืืืื ืืืืื ืืื ืึท ืืืืขืืืืคึผืขืจื ืฉืคึผืจืึทื. ืื ืื ืืขืจ - , ืึทืืึทืื ืืืจ ืฆื ืืึทืืืขื ื ืืืขืจืืง-ืืืจืื, ืืืื-ืืึทืืืึธืืข ืืขืืจืืงืก ืืืืึทื ืืื ืืืื ืคึผืจืึธืืขืืืขืืก ืกืขืจืืืขืจ, ืืื ืื ืคืืืืงืืื ืฆื ืฉืคึผืขืืขืจ ืืึทืงืืืขื ืฉืืื ืืืืืฉืืืึทืืึทืืืืฉืึทื ื ืืื ืืคืืื ืึทืืขืจืฅ.
ืืงืืจ: www.habr.com