ื ืืืืฅ ืืืืขืจืืขืฆืื ื: ืืึธืก ืืื ืึทื ืืืืขืจืืขืฆืื ื ืคืื ืึท ืขืคื ืืืขื ืคึผืึธืกืืืึธืจืืขื ืคืื ืื ืคืืจืืข 'ืก ืื ืืฉืขื ืืขืจืืข ืืืึธื . ืขืก ืืืฉืจืืืื ืึท ืคึผืจืึธืืืขื ืืื ืงืึทื ืืจืึทืงืง ืืื ืึท Kubernetes ืงื ืืื, ืืืึธืก ืืขืคืืจื ืฆื ืืืื ืืึทืื ืืืื ืคืื ืขืืืขืืข ืคึผืจืึธืืืงืฆืืข ืืึทืืื ืื ืืก.
ืืขืจ ืึทืจืืืงื ืงืขื ืืืื ื ืืฆืืง ืคึฟืึทืจ ืืขื ืข ืืืึธืก ืืืืื ืฆื ืืขืจื ืขื ืึท ืืืกื ืืขืจ ืืืขืื ืคึผืึธืกืืืึธืจืืขืืก ืึธืืขืจ ืคืึทืจืืืึทืื ืขืืืขืืข ืคึผืึธืืขื ืฆืืขื ืื ืก ืคึผืจืึธืืืขืืก ืืื ืืขืจ ืฆืืงืื ืคึฟื.
ืืึธืก ืืื ื ืืฉื ืื ืก
ืขืก ืงืขื ื ืืฉื ืืืื DNS
ืขืก ืืื ืืขืืืขื ืื ืก
ื ืืืกื ืืืขืื ืคึผืึธืกืืืึธืจืืขืืก ืืื ืคึผืจืึทืกืขืกืึทื ืืื ืคึผืจืขืคึผืื
ื ืคึผืึธืกืืืึธืจืืขื ืืืฉืจืืืื ืึท ืืึทืืคืึทื ืืงืฉืึทื ืึธืืขืจ ืขืืืขืืข ืืขืฉืขืขื ืืฉ ืืื ืคึผืจืึธืืืงืฆืืข. ืื ืคึผืึธืกืืืึธืจืืขื ืืืื ืึท ืืืืืืืื ืคืื ืืขืฉืขืขื ืืฉื, ืืึทื ืืฆืขืจ ืคึผืจืึทื, ืืืึธืจืฆื ืืจืื ื, ืึทืงืฉืึทื ื ืืขื ืืืขื ืืื ืืขืงืฆืืขืก ืืขืืขืจื ื.
ืืื ืืืืื ืฉืจืืคื ืืืืื ืื ืืื ืคึผืืฆืึท, ืฆืืืืฉื ืื ืืขืื ืืฉ ืืึทื ืฉืึทืคึฟื, ืืืจ ืืืืื ืคืึทืจืฉืืื ืืื ืคึฟืึธืจืืึทืฆืืข. ืืืื ืขืจ ืคืื ืื ืืขืจืกื ืืืืืืืง ืืืืื ืคืื ืึทืืึท ืืืืื ืื ืืื ืคึผืึธืกื-ืืึธืจืืขืืก, ืืืึธืก ืืขื ืขื ืืขืจืกื ืึธืคื ืืืืืืื ืืืจื ืึท ืคึผืจืขืืขื ืืืจืื ื ืืื ืกืืืืื ืืื ืึท ืืขืจ ืืืฃ ืึทื ืึทืืืกืืก ืคืื ืืขื ืืื ืฆืืืขื ื. ืืึธืืฉ ืืืจ ืงืืึทืคึผื ื ืืฉื ื ืึธื ืคึผืึธืกื-ืืึธืจืืขืืก, ืืืจ ืคึผืจืืืืจื ืฆื ืึทื ืืืืืงืืขื ืึท ืงืืืืืจ ืคืื "ืงืืื ืฉืืื" (). ืืืจ ืืืืืื ืึทื ืฉืจืืืื ืืื ืคึผืจืืืขื ืืื ื ืคึผืึธืกืืืึธืจืืขืืก ืงืขื ืขื ืืขืืคึฟื ืืื ืื (ืืื ืื ืืขืจืข) ืคืึทืจืืืึทืื ืขื ืืขื ืื ืกืึทืืึทื ืฅ ืืื ืืขืจ ืฆืืงืื ืคึฟื, ืืืึธืก ืืื ืืืึธืก ืืืจ ืืืืื ืืื.
ืืขื ืืฉื ืื ืืืึทืืืื ืืื ืึทื ืืื ืฆืืืขื ื ืืึธื ืคืืื ืึทื ืืื ืงืขื ืขื ืจืขืื ืืื ืืขืืึทื ืึธื ืืืจื ืคืื ืฉืืจืึธืฃ ืึธืืขืจ ื ืงืื. ืงืืื ืฉืืื! ืฉืจืืืื ืึท ืคึผืึธืกืืืึธืจืืขื ืืื ื ืืฉื ืึท ืฉืืจืึธืฃ, ืึธืืขืจ ืึท ืืขืจื ืขื ืืขืืขืื ืืืื ืคึฟืึทืจ ืื ืืื ืฆืข ืคืืจืืข.
ืคึผืจืึธืืืขืืก ืืื DNS ืืื Kubernetes. ืคึผืึธืกืืืึธืจืืขื
Date: 28.02.2020
ืืืืจืื: Amet U., Andrey S., Igor K., Alexey P.
ืกืืืืืก: ืืขืขื ืืืงื
Briefly: ืืืื DNS ืึทื ืึทืืืืืืึทืืืืืื (26 ืืื) ืคึฟืึทืจ ืขืืืขืืข ืืึทืืื ืื ืืก ืืื ืื Kubernetes ืงื ืืื
ืืฉืคึผืขื: 15000 ืืขืฉืขืขื ืืฉื ืคืึทืจืคืึทืื ืคึฟืึทืจ ืกืขืจืืืืกืขืก ื, ื ืืื ืกื
ืฉืืจืฉ ืกืืื: Kube-proxy ืืื ื ืืฉื ืืืืืืช ืฆื ืจืืืืืง ืืึทืืืึทืืืงื ืึทื ืึทืื ืคึผืึธืืืฆืืข ืคืื โโืื ืงืึทื ืืจืึทืงืง ืืืฉ, ืึทืืื ืขืืืขืืข ืกืขืจืืืืกืขืก ืืขื ืขื ื ืึธื ืืจืืื ื ืฆื ืคืึทืจืืื ืื ืฆื ื ืื-ืขืืืืกืืึทื ื ืคึผืึธืืก
E0228 20:13:53.795782 1 proxier.go:610] Failed to delete kube-system/kube-dns:dns endpoint connections, error: error deleting conntrack entries for UDP peer {100.64.0.10, 100.110.33.231}, error: conntrack command returned: ...ืืจืืืขืจ: ืจืขืื ืฆื ืืขืจ ื ืืืขืจืืง ืืึทืกืข ืืื ืื Kubernetes ืงื ืืื, CoreDNS-autoscaler ืจืืืืกื ืื ื ืืืขืจ ืคืื ืคึผืึธืืก ืืื ืื ืืืคึผืืืืืึทื ื ืคืื ืืจืื ืฆื ืฆืืืื
ืืืืืื ื: ืืขืจ ืืืืึทืืขืจ ืืืคึผืืืืืึทื ื ืคืื ืื ืึทืคึผืืึทืงืืืฉืึทื ืื ืืฉืืืืืื ืื ืฉืึทืคืื ื ืคืื ื ืืึท ื ืึธืืื, CoreDNS-autoscaler ืฆืืืขืืขืื ืืขืจ ืคึผืึธืืก ืฆื ืืื ืขื ืืขื ืงื ืืื, ืืืึธืก ืคึผืจืึทืืืึธืืงื ืึท ืจืืจืืื ืคืื ืื ืงืึธื ืืจืึทืงืง ืืืฉ
ืืขืืขืงืฉืึทื: ืคึผืจืึธืืขืืืขืืก ืืึธื ืืืึธืจืื ื ืืืืขืงืืึทื ืึท ืืจืืืก ื ืืืขืจ ืคืื 5xx ืขืจืจืึธืจืก ืคึฟืึทืจ ืกืขืจืืืืกืขืก ื, ื ืืื C ืืื ืื ืืฉืืืืืื ืึท ืจืืฃ ืฆื ืื ืขื ืืืฉืึทื ืืจื ืืืืฃ ืคืืืื
5xx ืขืจืจืึธืจืก ืืื ืงืืืึทื ืึท
ืืขืืืงืืื
ืงืึทืืฃ
ืืืคึผ
ืคืึทืจืึทื ืืืืึธืจืืืขื
ืึทืจืืขื
ืืืกืืืืึทื ืึทืืืึธืกืงืึทืืขืจ ืคึฟืึทืจ CoreDNS
ืคืึทืจืืืื
ืึทืืขื ืื.
ืืขืืืึธืคึผืก-695
ืฉืืขืื ืึทืจืืืฃ ืึท ืงืึทืืฉืื ื ืื ืก ืกืขืจืืืขืจ
ืคืึทืจืืื ืขืจื
ืืึทืงืก V.
ืืขืืืึธืคึผืก-665
ืืึทืฉืืขืืืงื ืงืึทื ืืจืึทืงืง ืืึธื ืืืึธืจืื ื
ืคืึทืจืืืื
ืึทืืขื ืื.
ืืขืืืึธืคึผืก-674
ืืขืงืฆืืขืก ืืขืืขืจื ื
ืืืึธืก ืืื ืืื:
- ืื ืืึธื ืืืึธืจืื ื ืืขืืจืืขื ืืขืืื ื. ืืขืจ ืขื ืืคืขืจ ืืื ืืขืืืขื ืฉื ืขื ืืื ืึธืจืืึทื ืืืืจื
- ืืืจ ืืึธืื ืงืืื ืืืืึทืฅ ืืืืฃ ืื ื ืึธืืื
ืืืึธืก ืืื ืคืึทืืฉ:
- ื ืึธื ืืืืืึทืงืึทื ื ืคืึทืงืืืฉ ืืืึธืจืฆื ืืจืื ื, ืขื ืืขื ืฆื ืืื ืงืึธื ืืจืึทืงื
- ืื ืึทืงืฉืึทื ื ืจืืืืืง ืืืืื ืื ืงืึทื ืกืึทืงืืืขื ืกืึทื, ื ืืฉื ืื ืืืึธืจืฆื ืืจืื ื (ืืฉืืง)
- ืืืจ ืืขืืืืืกื ืึทื ืืืืขืจ ืึธืืขืจ ืฉืคึผืขืืขืจ ืืืจ ืงืขื ืืึธืื ืคึผืจืึธืืืขืืก ืืื ืื ืก, ืึธืืขืจ ืืืจ ืืึธืื ื ืืฉื ืคึผืจืืืึธืจืึทืืืื ืื ืืึทืกืงืก
ืืื ืืืจ ืืึธืื ืืึทืืืืืง:
- ืืขืจ ืืืืึทืืขืจ ืืืคึผืืืืืึทื ื ืืื ืืจืืืขืจื ืืืจื CoreDNS-autoscaler, ืืืึธืก ืึธืืืืขืจืจืืืืื ืื ืงืึธื ืืจืึทืงืง ืืืฉ
- ืืขืจ ืืฉืืง ืึทืคืขืงืืึทื ืืืืื ืขืืืขืืข ืกืขืจืืืืกืขืก
ืืืืืืืื (EET)
ืฆืืึทื
ืงืึทืืฃ
22:13
CoreDNS-autoscaler ืจืืืืกื ืื ื ืืืขืจ ืคืื ืคึผืึธืืก ืคืื ืืจืื ืฆื ืฆืืืื
22:18
ืขื ืืืฉืึทื ืืจื ืืืืฃ ืคืืืื ืื ืืขืืืืื ืฆื ืืึทืงืืืขื ืจืืคื ืคืื ืื ืืึธื ืืืึธืจืื ื ืกืืกืืขื
22:21
ืื ืคืืืื ืขื ืืืฉืึทื ืืจื ืื ืืขืืืืื ืฆื ืืขืคึฟืื ืขื ืื ืืจืื ื ืคืื ืื ืขืจืจืึธืจืก.
22:39
ืขื ืืืฉืึทื ืืจื ืืืืฃ ืคืืืื ืื ืืขืืืืื ืจืึธืืืื ื ืฆืืจืืง ืืืื ืขืจ ืคืื ืื ืืขืฆืืข ืืึทืืื ืื ืืก ืฆื ืื ืคืจืืขืจืืืงืข ืืืขืจืกืืข
22:40
5xx ืขืจืจืึธืจืก ืคืืจืฉืืืคื ืืื, ืื ืกืืืืึทืฆืืข ืืื ืกืืืืืึทืืืืื
- ืฆืืื ืฆื ืืขืจืงืขื ืขื: 4 min
- ืฆืืื ืืืืืขืจ ืงืึทืืฃ: 21 min
- ืฆืืื ืฆื ืคืึทืจืจืืืื: 1 min
ื ืึธื ืืื ืคึฟืึธืจืืึทืฆืืข
- CoreDNS ืืึธืืก:
I0228 20:13:53.507780 1 event.go:221] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"kube-system", Name:"coredns", UID:"2493eb55-3dc0-11ea-b3a2-02bb48f8c230", APIVersion:"apps/v1", ResourceVersion:"132690686", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled down replica set coredns-6cbb6646c9 to 2 - ืืื ืงืก ืฆื ืงืืืึทื ืึท (ืฉื ืืึทืื), ืืจืึทืคืึทื ืึท (ืฉื ืืึทืื)
ืฆื ืืื ืึทืืืื ืงืคึผื ืืึทื ืืฅ, ืื ืืื ืืงืก ืงืขืจื ื ืืฆื ืขืคึผืขืก ืืขืจืืคื ืงืึธื ืืจืึทืงืง. ืืื ืงืืจืฅ, ืืึธืก ืืื ืึท ื ืืฆื ืืืึธืก ืึผืืื ืึท ืจืฉืืื ืคืื NAT ืจืขืงืึธืจืืก ืืืึธืก ืืขื ืขื ืกืืึธืจื ืืื ืึท ืกืคึผืขืฆืืขื ืืืฉ. ืืืขื ืืขืจ ืืืืึทืืขืจ ืคึผืึทืงืึทื ืงืืื ืคืื ืืขืจ ืืขืืืืงืขืจ ืคึผืึธื ืฆื ืืขืจ ืืขืืืืงืขืจ ืคึผืึธื ืืื ืคืจืืขืจ, ืื ืืขืฆื IP ืึทืืจืขืก ืืืขื ื ืืฉื ืืืื ืจืืงืึทืืงืืึทืืืืืื, ืึธืืขืจ ืืืขื ืืืื ืืขื ืืืขื ืคืื ืื ืงืึทื ืืจืึทืงืง ืืืฉ.
ืืื ืงืึทื ืืจืึทืงืง ืึทืจืืขื
ืจืขืืืืืึทืื ืคืื
ืืึธืก ืืื ืืขืืืขื ืึท ืืืืฉืคึผืื ืคืื ืืืื ืขืจ ืคืื ืืื ืืืขืจ ืคึผืึธืกืืืึธืจืืขืืก ืืื ืขืืืขืืข ื ืืฆืืง ืืื ืงืก. ืกืคึผืขืฆืืขื ืืื ืืขื ืึทืจืืืงื, ืืืจ ืืืืื ืืื ืคึฟืึธืจืืึทืฆืืข ืืืึธืก ืงืขื ืืืื ื ืืฆืืง ืคึฟืึทืจ ืื ืืขืจืข ืงืึธืืคึผืึทื ืืขืก. ืืขืจืคึฟืึทืจ ืืึธืื ืืืจ ื ืืฉื ืงืืื ืืืจื ืฆื ืืึทืื ืืจืฒึทืื ืืื ืืขืจืคึฟืึทืจ ืืึทืื ืืืจ ืืืื ืขืจ ืคึฟืื ืืื ืืืขืจืข ืคึผืึธืกืืืึธืจืืขืืก ืคึผืืืืืงืื. ืืึธ ืืขื ืขื ืขืืืขืืข ืืขืจ ืืฉืืงืึทืืืข ืฆืืืืจ ืคึผืึธืกืืืึธืจืืขืืก:
- ืืื ืืึทื:
- Dropbox:
- Spotify:
- ืคืืืข ืื ืืขืจืข ืคืื ืืื ืจืืคึผืึทืืึทืืึธืจื
- ืืืื ืฆืืืืจ ืคึผืึธืกืืืึธืจืืขื ืืื SRE ืกืคืจ
ืืงืืจ: www.habr.com