ืงืึทืœื™ืงืึธ ืคึฟืึทืจ ื ืขื˜ื•ื•ืึธืจืงื™ื ื’ ืื™ืŸ Kubernetes: ื”ืงื“ืžื” ืื•ืŸ ืึท ื‘ื™ืกืœ ื“ืขืจืคืึทืจื•ื ื’

ืงืึทืœื™ืงืึธ ืคึฟืึทืจ ื ืขื˜ื•ื•ืึธืจืงื™ื ื’ ืื™ืŸ Kubernetes: ื”ืงื“ืžื” ืื•ืŸ ืึท ื‘ื™ืกืœ ื“ืขืจืคืึทืจื•ื ื’

ื“ืขืจ ืฆื™ืœ ืคื•ืŸ ื“ืขื ืึทืจื˜ื™ืงืœ ืื™ื– ืฆื• ื‘ืึทืงืขื ืขืŸ ื“ื™ ืœื™ื™ืขื ืขืจ ืฆื• ื“ื™ ื‘ืึทืกื™ืงืก ืคื•ืŸ ื ืขื˜ื•ื•ืึธืจืงื™ื ื’ ืื•ืŸ ืึธื ืคื™ืจื•ื ื’ ื ืขืฅ ืคึผืึทืœืึทืกื™ื– ืื™ืŸ Kubernetes, ื•ื•ื™ ื’ืขื–ื•ื ื˜ ื•ื•ื™ ื“ื™ ื“ืจื™ื˜-ืคึผืึทืจื˜ื™ื™ Calico ืคึผืœื•ื’ื™ืŸ ื•ื•ืึธืก ื™ืงืกื˜ืขื ื“ื– ื ืึธืจืžืึทืœ ืงื™ื™ืคึผืึทื‘ื™ืœืึทื˜ื™ื–. ืฆื•ื–ืืžืขืŸ ื“ื™ ื•ื•ืขื’, ื“ื™ ื™ื– ืคื•ืŸ ืงืึทื ืคื™ื’ื™ืขืจื™ื™ืฉืึทืŸ ืื•ืŸ ืขื˜ืœืขื›ืข ืคึฟืขื™ึดืงื™ื™ื˜ืŸ ื•ื•ืขื˜ ื–ื™ื™ืŸ ื“ืขืžืึทื ืกื˜ืจื™ื™ื˜ื™ื“ ืžื™ื˜ ืคืึทืงื˜ื™ืฉ ื‘ื™ื™ืฉืคื™ืœืŸ ืคื•ืŸ ืื•ื ื“ื–ืขืจ ืึทืคึผืขืจื™ื™ื˜ื™ื ื’ ื“ืขืจืคืึทืจื•ื ื’.

ื ืฉื ืขืœ ื”ืงื“ืžื” ืฆื• Kubernetes ื ืขื˜ื•ื•ืึธืจืงื™ื ื’ ืึทืคึผืคึผืœื™ืึทื ืกืข

ื ืงื•ื‘ืขืจื ืขื˜ืขืก ืงื ื•ื™ืœ ืงืขื ืขืŸ ื ื™ื˜ ื–ื™ื™ืŸ ื™ืžืึทื“ื–ืฉืึทื ื“ ืึธืŸ ืึท ื ืขืฅ. ืžื™ืจ ื”ืึธื‘ืŸ ืฉื•ื™ืŸ ืืจื•ื™ืก ืžืึทื˜ืขืจื™ืึทืœืก ืื•ื™ืฃ ื–ื™ื™ืขืจ ื‘ืึทืกื™ืงืก: "ืึทืŸ ืื™ืœื•ืกื˜ืจื™ืจื˜ืข ืคื™ืจืขืจ ืฆื• ื ืขื˜ื•ื•ืึธืจืงื™ื ื’ ืื™ืŸ Kubernetes"ืื•ืŸ"ืึท ื”ืงื“ืžื” ืฆื• Kubernetes ื ืขื˜ื•ื•ืึธืจืง ืคึผืึทืœืึทืกื™ื– ืคึฟืึทืจ ื–ื™ื›ืขืจื”ื™ื™ื˜ ืคึผืจืึธืคืขืกืกื™ืึธื ืึทืœืก'.

ืื™ืŸ ื“ืขื ืงืึธื ื˜ืขืงืกื˜ ืคื•ืŸ ื“ืขื ืึทืจื˜ื™ืงืœ, ืขืก ืื™ื– ื•ื•ื™ื›ื˜ื™ืง ืฆื• ื˜ืึธืŸ ืึทื– K8s ื–ื™ืš ืื™ื– ื ื™ืฉื˜ ืคืึทืจืึทื ื˜ื•ื•ืึธืจื˜ืœืขืš ืคึฟืึทืจ ื ืขืฅ ืงืึทื ืขืงื˜ื™ื•ื•ื™ื˜ื™ ืฆื•ื•ื™ืฉืŸ ืงืึทื ื˜ื™ื™ื ืขืจื– ืื•ืŸ ื ืึธื•ื“ื–: ืคึฟืึทืจ ื“ืขื, ืคืึทืจืฉื™ื“ืŸ CNI ืคึผืœื•ื’ื™ื ืก (Container Networking Interface). ืžืขืจ ื•ื•ืขื’ืŸ ื“ืขื ื‘ืึทื’ืจื™ืฃ ืžื™ืจ ื–ื™ื™ ื”ืึธื‘ืŸ ืžื™ืจ ืื•ื™ืš ื’ืขื–ืึธื’ื˜.

ืคึฟืึทืจ ื‘ื™ื™ึทืฉืคึผื™ืœ, ื“ื™ ืžืขืจืกื˜ ืคึผืจืึธืกื˜ ืคื•ืŸ ื“ื™ ืคึผืœื•ื’ื™ื ืก ืื™ื– ืคืœืึทื ืึทืœ - ื’ื™ื˜ ืคื•ืœ ื ืขืฅ ืงืึทื ืขืงื˜ื™ื•ื•ื™ื˜ื™ ืฆื•ื•ื™ืฉืŸ ืึทืœืข ืงื ื•ื™ืœ ื ืึธื•ื“ื– ื“ื•ืจืš ืจื™ื™ื–ื™ื ื’ ื‘ืจื™ืงืŸ ืื•ื™ืฃ ื™ืขื“ืขืจ ื ืึธื“ืข, ืึทืกื™ื™ื ื™ื ื’ ืึท ืกื•ื‘ื ืขื˜ ืฆื• ืขืก. ืึธื‘ืขืจ, ื’ืึทื ืฅ ืื•ืŸ ืึทื ืจืขื’ื™ืึทืœื™ื™ื˜ื™ื“ ืึทืงืกืขืกืึทื‘ื™ืœื™ื˜ื™ ืื™ื– ื ื™ื˜ ืฉื˜ืขื ื“ื™ืง ื•ื•ื•ื™ืœื˜ื•ื™ืง. ืฆื• ืฆื•ืฉื˜ืขืœืŸ ืึท ืžื™ืŸ ืคื•ืŸ ืžื™ื ื™ืžืึทืœ ืืคื’ืขื–ื•ื ื“ืขืจื˜ืงื™ื™ื˜ ืื™ืŸ ื“ืขื ืงื ื•ื™ืœ, ืขืก ืื™ื– ื ื™ื™ื˜ื™ืง ืฆื• ื™ื ืขืจื•ื•ื™ืŸ ืื™ืŸ ื“ื™ ืงืึทื ืคื™ื’ื™ืขืจื™ื™ืฉืึทืŸ ืคื•ืŸ ื“ื™ ืคื™ื™ืจื•ื•ืึทืœ. ืื™ืŸ ืึทืœื’ืขืžื™ื™ืŸ, ืขืก ืื™ื– ื’ืขืฉื˜ืขืœื˜ ืื•ื ื˜ืขืจ ื“ื™ ืงืึธื ื˜ืจืึธืœ ืคื•ืŸ ื“ืขืจ ื–ืขืœื‘ื™ืงืขืจ CNI, ื•ื•ืึธืก ืื™ื– ื•ื•ืึธืก ืงื™ื™ืŸ ื™ื ื˜ืขืจื•ื•ืขื ื˜ืฉืึทื ื– ืคื•ืŸ ื“ืจื™ื˜-ืคึผืึทืจื˜ื™ื™ ืื™ืŸ ื™ืคึผื˜ืึทื‘ืœืขืก ืงืขื ืขืŸ ื–ื™ื™ืŸ ื™ื ื˜ืขืจืคึผืจืึทื˜ืึทื“ ืคืึทืœืฉ ืึธื“ืขืจ ืื™ื’ื ืึธืจื™ืจื˜ ื’ืึธืจ.

ืื•ืŸ "ืื•ื™ืก ืคื•ืŸ ื“ื™ ืงืขืกื˜ืœ" ืคึฟืึทืจ ืึธืจื’ืึทื ื™ื™ื–ื™ื ื’ ื ืขืฅ ืคึผืึธืœื™ื˜ื™ืง ืคืึทืจื•ื•ืึทืœื˜ื•ื ื’ ืื™ืŸ ืึท Kubernetes ืงื ื•ื™ืœ ืื™ื– ืฆื•ื’ืขืฉื˜ืขืœื˜ ื ืขื˜ื•ื•ืึธืจืง ืคึผืึธืœื™ื˜ื™ืง ืึทืคึผื™. ื“ื™ ืžื™ื˜ืœ, ืคื•ื ืื ื“ืขืจื’ืขื˜ื™ื™ืœื˜ ืื™ื‘ืขืจ ืื•ื™ืกื’ืขืงืœื™ื‘ืŸ ื ืึธืžืขืŸ ืกืคึผื™ื™ืกืึทื–, ืงืขืŸ ืึทื ื˜ื”ืึทืœื˜ืŸ ื›ึผืœืœื™ื ืฆื• ื“ื™ืคืขืจืขื ืฉื™ื™ื™ื˜ ืึทืงืกืขืก ืคื•ืŸ ืื™ื™ืŸ ืึทืคึผืœืึทืงื™ื™ืฉืึทืŸ ืฆื• ืื ื“ืขืจืŸ. ืขืก ืื•ื™ืš ืึทืœืึทื•ื– ืื™ืจ ืฆื• ืงืึทื ืคื™ื’ื™ืขืจ ืึทืงืกืขืกืึทื‘ื™ืœื™ื˜ื™ ืฆื•ื•ื™ืฉืŸ ืกืคึผืขืฆื™ืคื™ืฉ ืคึผืึธื“ืก, ื™ื ื•ื•ื™ื™ืจืึทื ืžืึทื ืฅ (ื ืึทืžืขืกืคึผืึทืกืขืก) ืึธื“ืขืจ ื‘ืœืึทืงืก ืคื•ืŸ IP ืึทื“ืจืขืกืขืก:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: test-network-policy
  namespace: default
spec:
  podSelector:
    matchLabels:
      role: db
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - ipBlock:
        cidr: 172.17.0.0/16
        except:
        - 172.17.1.0/24
    - namespaceSelector:
        matchLabels:
          project: myproject
    - podSelector:
        matchLabels:
          role: frontend
    ports:
    - protocol: TCP
      port: 6379
  egress:
  - to:
    - ipBlock:
        cidr: 10.0.0.0/24
    ports:
    - protocol: TCP
      port: 5978

ื“ืึธืก ืื™ื– ื ื™ืฉื˜ ื“ื™ ืžืขืจืกื˜ ืคึผืจื™ืžื™ื˜ื™ื•ื• ื‘ื™ื™ึทืฉืคึผื™ืœ ืคื•ืŸ ื‘ืึทืึทืžื˜ืขืจ ื“ืึทืงื™ื•ืžืขื ื˜ื™ื™ืฉืึทืŸ ืงืขืŸ ืึทืžืึธืœ ืื•ืŸ ืคึฟืึทืจ ืึทืœืข ื“ื™ืกืงืขืจื™ื“ื–ืฉ ื“ื™ ืคืึทืจืœืึทื ื’ ืฆื• ืคึฟืึทืจืฉื˜ื™ื™ืŸ ื“ื™ ืœืึธื’ื™ืง ืคื•ืŸ ื•ื•ื™ ื ืขืฅ ืคึผืึทืœืึทืกื™ื– ืึทืจื‘ืขื˜. ืึธื‘ืขืจ, ืžื™ืจ ื•ื•ืขืœืŸ ื ืึธืš ืคึผืจื•ื‘ื™ืจืŸ ืฆื• ืคึฟืึทืจืฉื˜ื™ื™ืŸ ื“ื™ ื’ืจื•ื ื˜ ืคึผืจื™ื ืกืึทืคึผืึทืœื– ืื•ืŸ ืžืขื˜ื”ืึธื“ืก ืคื•ืŸ ืคึผืจืึทืกืขืกื™ื ื’ ืคืึทืจืงืขืจ ืคืœืึธื•ื– ื ื™ืฆืŸ ื ืขืฅ ืคึผืึทืœืึทืกื™ื– ...

ืขืก ืื™ื– ืœืึทื“ื–ืฉื™ืงืึทืœ ืึทื– ืขืก ื–ืขื ืขืŸ 2 ื˜ื™ื™ืคึผืก ืคื•ืŸ ืคืึทืจืงืขืจ: ืึทืจื™ื™ึทืŸ ื“ื™ ืคึผืึธื“ (ื™ื ื’ืจืขืกืก) ืื•ืŸ ืึทื•ื˜ื’ืึธื•ื™ื ื’ ืคื•ืŸ ืขืก (ืขื’ืจืขืกืก).

ืงืึทืœื™ืงืึธ ืคึฟืึทืจ ื ืขื˜ื•ื•ืึธืจืงื™ื ื’ ืื™ืŸ Kubernetes: ื”ืงื“ืžื” ืื•ืŸ ืึท ื‘ื™ืกืœ ื“ืขืจืคืึทืจื•ื ื’

ืึทืงื˜ื•ืึทืœืœื™, ืคึผืึธืœื™ื˜ื™ืง ืื™ื– ืฆืขื˜ื™ื™ืœื˜ ืื™ืŸ ื“ื™ 2 ืงืึทื˜ืขื’ืึธืจื™ืขืก ื‘ืื–ื™ืจื˜ ืื•ื™ืฃ ื“ืขืจ ืจื™ื›ื˜ื•ื ื’ ืคื•ืŸ ื‘ืึทื•ื•ืขื’ื•ื ื’.

ื“ืขืจ ื•ื•ื™ื™ึทื˜ืขืจ ืคืืจืœืื ื’ื˜ ืึทื˜ืจื™ื‘ื™ื•ื˜ ืื™ื– ืึท ืกืขืœืขืงื˜ืึธืจ; ื“ืขืจ, ืื•ื™ืฃ ื•ื•ืขืžืขืŸ ื“ืขืจ ื›ืœืœ ื’ื™ื™ื˜. ื“ืึธืก ืงืขืŸ ื–ื™ื™ืŸ ืึท ืคึผืึธื“ (ืึธื“ืขืจ ืึท ื’ืจื•ืคึผืข ืคื•ืŸ โ€‹โ€‹ืคึผืึธื“ืก) ืึธื“ืขืจ ืึท ืกื•ื•ื™ื•ื•ืข (ื“"ื” ืึท ื ืึทืžืขืกืคึผืึทืกืข). ืึท ื•ื•ื™ื›ื˜ื™ืง ื“ืขื˜ืึทืœ: ื‘ื™ื™ื“ืข ื˜ื™ื™ืคึผืก ืคื•ืŸ ื“ื™ ืึทื‘ื“ื–ืฉืขืงืฅ ืžื•ื–ืŸ ืึทื ื˜ื”ืึทืœื˜ืŸ ืึท ืคื™ืจืžืข (ืคื™ืจืžืข ืื™ืŸ Kubernetes ื˜ืขืจืžื™ื ืึธืœืึธื’ื™ืข) - ื“ืึธืก ื–ืขื ืขืŸ ื“ื™ ื•ื•ืึธืก ืคึผืึทืœืึทื˜ื™ืฉืึทื ื– ืึทืจื‘ืขื˜ืŸ ืžื™ื˜.

ืื™ืŸ ืึทื“ื™ืฉืึทืŸ ืฆื• ืึท ืขื ื“ืœืขืš ื ื•ืžืขืจ ืคื•ืŸ ืกืขืœืขืงื˜ืึธืจืก ืคึฟืึทืจืื™ื™ื ื™ืงื˜ืข ื“ื•ืจืš ืขื˜ืœืขื›ืข ืกืึธืจื˜ ืคื•ืŸ ืคื™ืจืžืข, ืขืก ืื™ื– ืžืขื’ืœืขืš ืฆื• ืฉืจื™ื™ึทื‘ืŸ ื›ึผืœืœื™ื ื•ื•ื™ "ืœืึธื–ืŸ / ืœื™ื™ืงืขื ืขืŸ ืึทืœืฅ / ืึทืœืขืžืขืŸ" ืื™ืŸ ืคืึทืจืฉื™ื“ืขื ืข ื•ื•ืขืจื™ื™ื™ืฉืึทื ื–. ืคึฟืึทืจ ื“ืขื ืฆื•ื•ืขืง, ืงืึทื ืกื˜ืจืึทืงืฉืึทื ื– ืคื•ืŸ ื“ื™ ืคืึธืจืขื ื–ืขื ืขืŸ ื’ืขื ื™ืฆื˜:

  podSelector: {}
  ingress: []
  policyTypes:
  - Ingress

- ืื™ืŸ ื“ืขื ื‘ื™ื™ึทืฉืคึผื™ืœ, ืึทืœืข ืคึผืึธื“ืก ืื™ืŸ ื“ื™ ืกื•ื•ื™ื•ื•ืข ื–ืขื ืขืŸ ืืคื’ืขืฉื˜ืขืœื˜ ืคื•ืŸ ื™ื ืงืึทืžื™ื ื’ ืคืึทืจืงืขืจ. ื“ื™ ืคืึทืจืงืขืจื˜ ื ืึทื˜ื•ืจ ืงืขื ืขืŸ ื–ื™ื™ืŸ ืึทื˜ืฉื™ื•ื•ื“ ืžื™ื˜ ื“ื™ ืคืืœื’ืขื ื“ืข ืงืึทื ืกื˜ืจืึทืงืฉืึทืŸ:

  podSelector: {}
  ingress:
  - {}
  policyTypes:
  - Ingress

ืกื™ืžื™ืœืึทืจืœื™ ืคึฟืึทืจ ืึทื•ื˜ื’ืึธื•ื™ื ื’:

  podSelector: {}
  policyTypes:
  - Egress

- ืฆื• ืงืขืจ ืขืก ืึทื•ื•ืขืง. ืื•ืŸ ื“ืึธ ืก ื•ื•ืึธืก ืฆื• ืึทืจื™ื™ึทื ื ืขืžืขืŸ:

  podSelector: {}
  egress:
  - {}
  policyTypes:
  - Egress

ืฆื•ืจื™ืง ืฆื• ื“ื™ ื‘ืจื™ืจื” ืคื•ืŸ ืึท CNI ืคึผืœื•ื’ื™ืŸ ืคึฟืึทืจ ืึท ืงื ื•ื™ืœ, ืขืก ืื™ื– ื›ื“ืื™ ืฆื• ื‘ืืžืขืจืงืŸ ืึทื– ื ื™ื˜ ื™ืขื“ืขืจ ื ืขืฅ ืคึผืœื•ื’ื™ืŸ ืฉื˜ื™ืฆื˜ NetworkPolicy. ืคึฟืึทืจ ื‘ื™ื™ึทืฉืคึผื™ืœ, ื“ื™ ืฉื•ื™ืŸ ื“ืขืจืžืื ื˜ ืคืœืึทื ืึทืœ ื˜ื•ื˜ ื ื™ืฉื˜ ื•ื•ื™ืกืŸ ื•ื•ื™ ืฆื• ืงืึทื ืคื™ื’ื™ืขืจ ื ืขืฅ ืคึผืึทืœืึทืกื™ื–, โ€‹โ€‹ื•ื•ืึธืก ืขืก ืื™ื– ื’ืขื–ืื’ื˜ ื’ืœื™ื™ึทืš ืื™ืŸ ื“ืขืจ ื‘ืึทืึทืžื˜ืขืจ ืจื™ืคึผืึทื–ืึทื˜ืึธืจื™. ืึทืŸ ืึธืœื˜ืขืจื ืึทื˜ื™ื•ื• ืื™ื– ืื•ื™ืš ื“ืขืจืžืื ื˜ ื“ืึธืจื˜ - ืึทืŸ ืขืคึฟืŸ ืžืงื•ืจ ืคึผืจื•ื™ืขืงื˜ ืงืึทืœื™ืงืึธ, ื•ื•ืึธืก ื‘ืื˜ื™ื™ื˜ื™ืง ื™ืงืกืคึผืึทื ื“ื– ื“ื™ ื ืึธืจืžืึทืœ ื’ืึทื ื’ ืคื•ืŸ Kubernetes APIs ืื™ืŸ ื˜ืขืจืžื™ื ืขืŸ ืคื•ืŸ ื ืขืฅ ืคึผืึทืœืึทืกื™ื–.

ืงืึทืœื™ืงืึธ ืคึฟืึทืจ ื ืขื˜ื•ื•ืึธืจืงื™ื ื’ ืื™ืŸ Kubernetes: ื”ืงื“ืžื” ืื•ืŸ ืึท ื‘ื™ืกืœ ื“ืขืจืคืึทืจื•ื ื’

ื‘ืึทืงื•ืžืขืŸ ืฆื• ื•ื•ื™ืกืŸ ืงืึทืœื™ืงืึธ: ื˜ืขืึธืจื™ืข

ื“ื™ ืงืึทืœื™ืงืึธ ืคึผืœื•ื’ื™ืŸ ืงืขื ืขืŸ ื–ื™ื™ืŸ ื’ืขื•ื•ื™ื™ื ื˜ ืื™ืŸ ื™ื ืึทื’ืจื™ื™ืฉืึทืŸ ืžื™ื˜ ืคืœืึทื ืึทืœ (ืกื•ื‘ืคึผืจืึธื“ื–ืฉืขืงื˜ ืงืึทื ืึทืœ) ืึธื“ืขืจ ื™ื ื“ื™ืคึผืขื ื“ืึทื ื˜ืœื™, ืงืึทื•ื•ืขืจื™ื ื’ ื‘ื™ื™ื“ืข ื ืขืฅ ืงืึทื ืขืงื˜ื™ื•ื•ื™ื˜ื™ ืื•ืŸ ืึทื•ื•ื™ื™ืœืึทื‘ื™ืœื™ื˜ื™ ืคืึทืจื•ื•ืึทืœื˜ื•ื ื’ ืงื™ื™ืคึผืึทื‘ื™ืœืึทื˜ื™ื–.

ื•ื•ืึธืก ืึทืคึผืขืจื˜ื•ื ืึทื˜ื™ื– ื’ื™ื˜ ื ื™ืฆืŸ ื“ื™ K8s "ื‘ืึธืงืกืขื“" ืœื™ื™ื–ื•ื ื’ ืื•ืŸ ื“ื™ ืึทืคึผื™ ืฉื˜ืขืœืŸ ืคื•ืŸ Calico?

ื“ืึธ ืก ื•ื•ืึธืก ืื™ื– ื’ืขื‘ื•ื™ื˜ ืื™ืŸ ื ืขื˜ื•ื•ืึธืจืง ืคึผืึธืœื™ื˜ื™ืง:

  • ืคึผืึทืœืึทื˜ื™ืฉืึทื ื– ื–ืขื ืขืŸ ื‘ืื’ืจืขื ืขืฆื˜ ื“ื•ืจืš ื“ื™ ืกื•ื•ื™ื•ื•ืข;
  • ืคึผืึทืœืึทืกื™ื– ื–ืขื ืขืŸ ื’ืขื•ื•ืขื ื“ื˜ ืฆื• ืคึผืึธื“ืก ืื ื’ืขืฆื™ื™ื›ื ื˜ ืžื™ื˜ ืœืึทื‘ืขืœืก;
  • ื›ึผืœืœื™ื ืงืขื ืขืŸ ื–ื™ื™ืŸ ื’ืขื•ื•ืขื ื“ื˜ ืฆื• ืคึผืึธื“ืก, ื™ื ื•ื•ื™ื™ืจืึทื ืžืึทื ืฅ ืึธื“ืขืจ ืกื•ื‘ื ืขืฅ;
  • ื›ึผืœืœื™ื ืงืขื ืขืŸ ืึทื ื˜ื”ืึทืœื˜ืŸ ืคึผืจืึธื˜ืึธืงืึธืœืก, ื’ืขื”ื™ื™ืกืŸ ืึธื“ืขืจ ืกื™ืžื‘ืึธืœื™ืฉ ืคึผืึธืจื˜ ืกืคึผืขืกืึทืคืึทืงื™ื™ืฉืึทื ื–.

ื“ืึธ ืก ื•ื•ื™ ืงืึทืœื™ืงืึธ ื™ืงืกื˜ืขื ื“ื– ื“ื™ ืคืึทื ื’ืงืฉืึทื ื–:

  • ืคึผืึทืœืึทืกื™ื– ืงืขื ืขืŸ ื–ื™ื™ืŸ ื’ืขื•ื•ืขื ื“ื˜ ืฆื• ืงื™ื™ืŸ ื›ื™ื™ืคืขืฅ: ืคึผืึธื“, ืงืึทื ื˜ื™ื™ื ืขืจ, ื•ื•ื™ืจื˜ื•ืึทืœ ืžืึทืฉื™ืŸ ืึธื“ืขืจ ืฆื•ื‘ื™ื ื“;
  • ื›ึผืœืœื™ื ืงืขื ืขืŸ ืึทื ื˜ื”ืึทืœื˜ืŸ ืึท ืกืคึผืขืฆื™ืคื™ืฉ ืงืึทืžืฃ (ืคืึทืจื•ื•ืขืจ, ื“ืขืจืœื•ื™ื‘ืขื ื™ืฉ, ืœืึธื’ื™ื ื’);
  • ื“ืขืจ ืฆื™ืœ ืึธื“ืขืจ ืžืงื•ืจ ืคื•ืŸ ื›ึผืœืœื™ื ืงืขื ืขืŸ ื–ื™ื™ืŸ ืึท ืคึผืึธืจื˜, ืึท ืงื™ื™ื˜ ืคื•ืŸ ืคึผืึธืจืฅ, ืคึผืจืึธื˜ืึธืงืึธืœืก, ื”ื˜ื˜ืคึผ ืึธื“ืขืจ ICMP ืึทื˜ืจื™ื‘ื™ื•ืฅ, IP ืึธื“ืขืจ ืกื•ื‘ื ืขื˜ (4 ืึธื“ืขืจ 6 ื“ื•ืจ), ืงื™ื™ืŸ ืกืขืœืขืงื˜ืึธืจืก (ื ืึธื“ืขืก, ืžื—ื ื•ืช, ื™ื ื•ื•ื™ื™ืจืึทื ืžืึทื ืฅ);
  • ืึทื“ื“ื™ื˜ื™ืึธื ืึทืœืœื™, ืื™ืจ ืงืขื ืขืŸ ืจืขื’ื•ืœื™ืจืŸ ื“ื™ ื“ื•ืจื›ืคืึธืจ ืคื•ืŸ ืคืึทืจืงืขืจ ื ื™ืฆืŸ DAT ืกืขื˜ื˜ื™ื ื’ืก ืื•ืŸ ืคืึทืจืงืขืจ ืคืึธืจื•ื•ืขืจื“ื™ื ื’ ืคึผืึทืœืึทืกื™ื–.

ื“ืขืจ ืขืจืฉื˜ืขืจ ืงืึทืžื™ืฅ ืื•ื™ืฃ GitHub ืื™ืŸ ื“ื™ ืงืึทืœื™ืงืึธ ืจื™ืคึผืึทื–ืึทื˜ืึธืจื™ ื“ืึทื˜ืขืก ืฆื•ืจื™ืง ืฆื• ื™ื•ืœื™ 2016, ืื•ืŸ ืึท ื™ืึธืจ ืฉืคึผืขื˜ืขืจ ื“ื™ ืคึผืจื•ื™ืขืงื˜ ื’ืขื ื•ืžืขืŸ ืึท ืœื™ื“ื™ื ื’ ืฉื˜ืขืœืข ืื™ืŸ ืึธืจื’ืึทื ื™ื™ื–ื™ื ื’ ืงื•ื‘ืขืจื ืขื˜ืขืก ื ืขืฅ ืงืึทื ืขืงื˜ื™ื•ื•ื™ื˜ื™ - ื“ืึธืก ืื™ื– ืขื•ื•ื™ื“ืึทื ืกื˜, ืœืžืฉืœ, ื“ื•ืจืš ื“ื™ ื™ื‘ืขืจื‘ืœื™ืง ืจืขื–ื•ืœื˜ืึทื˜ืŸ, ื’ืขืคื™ืจื˜ ื“ื•ืจืš ื“ื™ ื ื™ื• ืกื˜ืึทืง:

ืงืึทืœื™ืงืึธ ืคึฟืึทืจ ื ืขื˜ื•ื•ืึธืจืงื™ื ื’ ืื™ืŸ Kubernetes: ื”ืงื“ืžื” ืื•ืŸ ืึท ื‘ื™ืกืœ ื“ืขืจืคืึทืจื•ื ื’

ืคื™ืœืข ื’ืจื•ื™ืก ื’ืขืจืื˜ืŸ ืกืึทืœื•ืฉืึทื ื– ืžื™ื˜ K8s, ืึทื–ืึท ื•ื•ื™ ืึทืžืึทื–ืึธืŸ ืขืงืก, Azure AKS, ื’ื•ื’ืœ GKE ืื•ืŸ ืื ื“ืขืจืข ืื ื’ืขื”ื•ื™ื‘ืŸ ืฆื• ืจืขืงืึธืžืขื ื“ื™ืจืŸ ืขืก ืคึฟืึทืจ ื ื•ืฆืŸ.

ื•ื•ื™ ืคึฟืึทืจ ืคืึธืจืฉื˜ืขืœื•ื ื’, ืึทืœืฅ ืื™ื– ื’ืจื•ื™ืก ื“ืึธ. ืื™ืŸ ื˜ืขืกื˜ื™ื ื’ ื–ื™ื™ืขืจ ืคึผืจืึธื“ื•ืงื˜, ื“ื™ ืงืึทืœื™ืงืึธ ืึทื ื˜ื•ื•ื™ืงืœื•ื ื’ ืžืึทื ืฉืึทืคึฟื˜ ื“ืขืžืึทื ืกื˜ืจื™ื™ื˜ื™ื“ ืึทืกื˜ืจืึทื ืึทืžื™ืงืึทืœ ืคืึธืจืฉื˜ืขืœื•ื ื’, ืคืœื™ืกื ื“ื™ืง ืžืขืจ ื•ื•ื™ 50000 ืงืึทื ื˜ื™ื™ื ืขืจื– ืื•ื™ืฃ 500 ื’ืฉืžื™ื•ืช ื ืึธื•ื“ื– ืžื™ื˜ ืึท ืฉืึทืคื•ื ื’ ืงื•ืจืก ืคื•ืŸ 20 ืงืึทื ื˜ื™ื™ื ืขืจื– ืคึผืขืจ ืกืขืงื•ื ื“ืข. ืงื™ื™ืŸ ืคืจืื‘ืœืขืžืขืŸ ื–ืขื ืขืŸ ื™ื™ื“ืขื ืึทืคื™ื™ื“ ืžื™ื˜ ืกืงื™ื™ืœื™ื ื’. ืื–ืขืœื›ืข ืจืขื–ื•ืœื˜ืื˜ืŸ ื–ืขื ืขืŸ ืžื•ื“ื™ืข ื’ืขื•ื•ืขืŸ ืฉื•ื™ืŸ ื‘ื™ื™ ื“ื™ ืžืขืœื“ืŸ ืคื•ืŸ ื“ืขืจ ืขืจืฉื˜ืขืจ ื•ื•ืขืจืกื™ืข. ืคืจื™ื™ึท ืฉื˜ื•ื“ื™ื•ื ืคืึธื•ืงื™ืกื™ื ื’ ืื•ื™ืฃ ื˜ืจื•ืคึผื•ื˜ ืื•ืŸ ืžื™ื˜ืœ ืงืึทื ืกืึทืžืฉืึทืŸ ืื•ื™ืš ื‘ืึทืฉื˜ืขื˜ื™ืงืŸ ื“ื™ ืคืึธืจืฉื˜ืขืœื•ื ื’ ืคื•ืŸ Calico ืื™ื– ืงื™ืžืึทื˜ ื•ื•ื™ ื’ื•ื˜ ื•ื•ื™ ืคืœืึทื ืึทืœ. ืœืขืžืึธืฉืœ:

ืงืึทืœื™ืงืึธ ืคึฟืึทืจ ื ืขื˜ื•ื•ืึธืจืงื™ื ื’ ืื™ืŸ Kubernetes: ื”ืงื“ืžื” ืื•ืŸ ืึท ื‘ื™ืกืœ ื“ืขืจืคืึทืจื•ื ื’

ื“ื™ ืคึผืจื•ื™ืขืงื˜ ืื™ื– ื“ืขื•ื•ืขืœืึธืคึผื™ื ื’ ื–ื™ื™ืขืจ ื’ืขืฉื•ื•ื™ื ื“, ืขืก ืฉื˜ื™ืฆื˜ ืึทืจื‘ืขื˜ ืื™ืŸ ืคืึธืœืงืก ืกืึทืœื•ืฉืึทื ื– ื’ืขืจืื˜ืŸ K8s, OpenShift, OpenStack, ืขืก ืื™ื– ืžืขื’ืœืขืš ืฆื• ื ื•ืฆืŸ Calico ื•ื•ืขืŸ ื“ื™ืคึผืœื•ื™ื™ื ื’ ืึท ืงื ื•ื™ืœ ื ื™ืฆืŸ kops, ืขืก ื–ืขื ืขืŸ ืจืขืคืขืจืขื ืฆืŸ ืฆื• ื“ื™ ืงืึทื ืกื˜ืจืึทืงืฉืึทืŸ ืคื•ืŸ ืกืขืจื•ื•ื™ืก ืžืขืฉ ื ืขื˜ื•ื•ืึธืจืงืก (ื“ืึธ ืื™ื– ืึท ื‘ื™ื™ึทืฉืคึผื™ืœ ื’ืขื•ื•ื™ื™ื ื˜ ืื™ืŸ ืงืึทื ื“ื–ืฉืึทื ื’ืงืฉืึทืŸ ืžื™ื˜ Istio).

ืคึผืจืึทืงื˜ื™ืกื™ื– ืžื™ื˜ ืงืึทืœื™ืงืึธ

ืื™ืŸ ืึทืœื’ืขืžื™ื™ืŸ ืคืึทืœ ืคื•ืŸ ื ื™ืฆืŸ ื•ื•ืึทื ื™ืœ ืงื•ื‘ืขืจื ืขื˜ืขืก, ื™ื ืกื˜ืึธืœื™ื ื’ CNI ืงื•ืžื˜ ืึทืจืึธืคึผ ืฆื• ื ื•ืฆืŸ ื“ื™ ื˜ืขืงืข calico.yaml, ื“ืึทื•ื ืœืึธื•ื“ื™ื“ ืคื•ืŸ ื“ืขืจ ื‘ืึทืึทืžื˜ืขืจ ื•ื•ืขื‘ื–ื™ื™ึทื˜ืœ, ื“ื•ืจืš ื ื•ืฆืŸ kubectl apply -f.

ื•ื•ื™ ืึท ื”ืขืจืฉืŸ, ื“ื™ ืงืจืึทื ื˜ ื•ื•ืขืจืกื™ืข ืคื•ืŸ โ€‹โ€‹ื“ื™ ืคึผืœื•ื’ื™ืŸ ืื™ื– ืงืึทืžืคึผืึทื˜ืึทื‘ืึทืœ ืžื™ื˜ ื“ื™ ืœืขืฆื˜ืข 2-3 ื•ื•ืขืจืกื™ืขืก ืคื•ืŸ Kubernetes: ืึธืคึผืขืจืึทืฆื™ืข ืื™ืŸ ืขืœื˜ืขืจืข ื•ื•ืขืจืกื™ืขืก ืื™ื– ื ื™ืฉื˜ ื˜ืขืกื˜ืขื“ ืื•ืŸ ืื™ื– ื ื™ืฉื˜ ื’ืขืจืึทื ื˜ื™ื“. ืœื•ื™ื˜ ื“ื™ ื“ืขื•ื•ืขืœืึธืคึผืขืจืก, Calico ืœื•ื™ืคื˜ ืื•ื™ืฃ ืœื™ื ื•ืงืก ืงืขืจื ืึทืœื– ื”ืขื›ืขืจ 3.10 ืžื™ื˜ CentOS 7, Ubuntu 16 ืึธื“ืขืจ Debian 8, ืื•ื™ืฃ ืฉืคึผื™ืฅ ืคื•ืŸ ื™ืคึผื˜ืึทื‘ืœืขืก ืึธื“ืขืจ IPVS.

ืืคื’ืขื–ื•ื ื“ืขืจื˜ืงื™ื™ื˜ ืื™ืŸ ื“ืขืจ ืกื‘ื™ื‘ื”

ืคึฟืึทืจ ืึท ื’ืขื ืขืจืึทืœ ืคืืจืฉื˜ืื ื“, ืœืึธืžื™ืจ ืงื•ืงืŸ ืื™ืŸ ืึท ืคึผืฉื•ื˜ ืคืึทืœ ืฆื• ืคึฟืึทืจืฉื˜ื™ื™ืŸ ื•ื•ื™ ื ืขืฅ ืคึผืึทืœืึทืกื™ื– ืื™ืŸ ื“ื™ ืงืึทืœื™ืงืึธ ื ืึธื•ื˜ื™ื™ืฉืึทืŸ ื–ืขื ืขืŸ ืึทื ื“ืขืจืฉ ืคื•ืŸ ื ืึธืจืžืึทืœ ืึธื ืขืก ืื•ืŸ ื•ื•ื™ ื“ืขืจ ืฆื•ื’ืึทื ื’ ืฆื• ืฉืึทืคึฟืŸ ื›ึผืœืœื™ื ืกื™ืžืคึผืœืึทืคื™ื™ื– ื–ื™ื™ืขืจ ืจื™ื“ืึทื‘ื™ืœื™ื˜ื™ ืื•ืŸ ืงืึทื ืคื™ื’ื™ืขืจื™ื™ืฉืึทืŸ ื‘ื™ื™ื’ื™ืงื™ื™ื˜:

ืงืึทืœื™ืงืึธ ืคึฟืึทืจ ื ืขื˜ื•ื•ืึธืจืงื™ื ื’ ืื™ืŸ Kubernetes: ื”ืงื“ืžื” ืื•ืŸ ืึท ื‘ื™ืกืœ ื“ืขืจืคืึทืจื•ื ื’

ืขืก ื–ืขื ืขืŸ 2 ื•ื•ืขื‘ ืึทืคึผืœืึทืงื™ื™ืฉืึทื ื– ื“ื™ืคึผืœื•ื™ื“ ืื™ืŸ ื“ืขื ืงื ื•ื™ืœ: ืื™ืŸ Node.js ืื•ืŸ PHP, ืื™ื™ื ืขืจ ืคื•ืŸ ื•ื•ืึธืก ื ื™ืฆื˜ Redis. ืฆื• ืคืึทืจืฉืคึผืึทืจืŸ ืึทืงืกืขืก ืฆื• Redis ืคึฟื•ืŸ PHP, ื‘ืฉืขืช ืื™ืจ ื”ืึทืœื˜ืŸ ืงืึทื ืขืงื˜ื™ื•ื•ื™ื˜ื™ ืžื™ื˜ Node.js, ื ืึธืจ ืฆื•ืœื™ื™ื’ืŸ ื“ื™ ืคืืœื’ืขื ื“ืข ืคึผืึธืœื™ื˜ื™ืง:

kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: allow-redis-nodejs
spec:
  podSelector:
    matchLabels:
      service: redis
  ingress:
  - from:
    - podSelector:
        matchLabels:
          service: nodejs
    ports:
    - protocol: TCP
      port: 6379

ื™ืกืขื ืฉืึทืœื™, ืžื™ืจ ืขืจืœื•ื™ื‘ื˜ ื™ื ืงืึทืžื™ื ื’ ืคืึทืจืงืขืจ ืฆื• ื“ื™ Redis ืคึผืึธืจื˜ ืคึฟื•ืŸ Node.js. ืื•ืŸ ื–ื™ื™ ื”ืื‘ืŸ ืงืœืืจ ื ื™ืฉื˜ ืคืืจื‘ืื˜ืŸ ืขืคืขืก ืื ื“ืขืจืฉ. ื•ื•ื™ ื‘ืึทืœื“ ื•ื•ื™ NetworkPolicy ืื™ื– ืืจื•ื™ืก, ืึทืœืข ืกืขืœืขืงื˜ืึธืจืก ื“ืขืจืžืื ื˜ ืื™ืŸ ืขืก ืึธื ื”ื™ื™ื‘ืŸ ืฆื• ื–ื™ื™ืŸ ืืคื’ืขื–ื•ื ื“ืขืจื˜, ืกื™ื™ึทื“ืŸ ืึทื ื“ืขืจืฉ ืกืคึผืขืกื™ืคื™ืขื“. ืึธื‘ืขืจ, ื“ื™ ืืคื’ืขื–ื•ื ื“ืขืจื˜ืงื™ื™ื˜ ื›ึผืœืœื™ื ื˜ืึธืŸ ื ื™ื˜ ืฆื•ืœื™ื™ื’ืŸ ืฆื• ืื ื“ืขืจืข ืึทื‘ื“ื–ืฉืขืงืฅ ื•ื•ืึธืก ื–ืขื ืขืŸ ื ื™ืฉื˜ ื‘ืื“ืขืงื˜ ื“ื•ืจืš ื“ื™ ืกืขืœืขืงื˜ืึธืจ.

ื“ื™ ื‘ื™ื™ืฉืคึผื™ืœ ื ื™ืฆื˜ apiVersion ืงื•ื‘ืขืจื ืขื˜ืขืก ืื•ื™ืก ืคื•ืŸ ื“ื™ ืงืขืกื˜ืœ, ืึธื‘ืขืจ ื’ืึธืจื ื™ืฉื˜ ืคึผืจื™ื•ื•ืขื ืฅ ืื™ืจ ืคื•ืŸ ื ื™ืฆืŸ ืขืก ืžื™ื˜ืœ ืคื•ืŸ ื“ื™ ื–ืขืœื‘ืข ื ืึธืžืขืŸ ืคื•ืŸ ื“ื™ ืงืึทืœื™ืงืึธ ืขืงืกืคึผืจืขืก. ื“ืขืจ ืกื™ื ื˜ืึทืงืก ื“ืึธืจื˜ ืื™ื– ืžืขืจ ื“ื™ื˜ื™ื™ืœื“, ืึทื–ื•ื™ ืื™ืจ ื•ื•ืขื˜ ื“ืึทืจืคึฟืŸ ืฆื• ืจื™ืจื™ื™ื˜ ื“ื™ ื”ืขืจืฉืŸ ืคึฟืึทืจ ื“ื™ ืื•ื™ื‘ืŸ ืคืึทืœ ืื™ืŸ ื“ื™ ืคืืœื’ืขื ื“ืข ืคืึธืจืขื:

apiVersion: crd.projectcalico.org/v1
kind: NetworkPolicy
metadata:
  name: allow-redis-nodejs
spec:
  selector: service == 'redis'
  ingress:
  - action: Allow
    protocol: TCP
    source:
      selector: service == 'nodejs'
    destination:
      ports:
      - 6379

ื“ื™ ืื•ื™ื‘ืŸ-ื“ืขืจืžืื ื˜ ืงืึทื ืกื˜ืจืึทืงืฉืึทื ื– ืคึฟืึทืจ ืึทืœืึทื•ื™ื ื’ ืึธื“ืขืจ ืคืืจืœื™ื™ืงืขื ืขืŸ ืึทืœืข ืคืึทืจืงืขืจ ื“ื•ืจืš ื“ื™ ืจืขื’ื•ืœืขืจ NetworkPolicy API ืึทื ื˜ื”ืึทืœื˜ืŸ ืงืึทื ืกื˜ืจืึทืงืฉืึทื ื– ืžื™ื˜ ืงืœืึทืžืขืจืŸ ื•ื•ืึธืก ื–ืขื ืขืŸ ืฉื•ื•ืขืจ ืฆื• ืคึฟืึทืจืฉื˜ื™ื™ืŸ ืื•ืŸ ื’ืขื“ืขื ืงืขืŸ. ืื™ืŸ ื“ื™ ืคืึทืœ ืคื•ืŸ Calico, ืฆื• ื˜ื•ื™ืฉืŸ ื“ื™ ืœืึธื’ื™ืง ืคื•ืŸ ืึท ืคื™ื™ืจื•ื•ืึทืœ ื”ืขืจืฉืŸ ืฆื• ื“ื™ ืคืึทืจืงืขืจื˜, ื ืึธืจ ื˜ื•ื™ืฉืŸ action: Allow ืื•ื™ืฃ action: Deny.

ืืคื’ืขื–ื•ื ื“ืขืจื˜ืงื™ื™ื˜ ื“ื•ืจืš ืกื•ื•ื™ื•ื•ืข

ืื™ืฆื˜ ื™ืžืึทื“ื–ืฉืึทืŸ ืึท ืกื™ื˜ื•ืึทืฆื™ืข ื•ื•ื• ืึท ืึทืคึผืœืึทืงื™ื™ืฉืึทืŸ ื“ื–ืฉืขื ืขืจื™ื™ืฅ ื’ืขืฉืขืคื˜ ืžืขื˜ืจื™ืงืก ืคึฟืึทืจ ื–ืึทืžืœื•ื ื’ ืื™ืŸ ืคึผืจืึธืžืขื˜ื”ืขื•ืก ืื•ืŸ ื•ื•ื™ื™ึทื˜ืขืจ ืึทื ืึทืœื™ืกื™ืก ืžื™ื˜ Grafana. ื“ื™ ื•ืคึผืœืึธืึทื“ ืงืขืŸ ืึทื ื˜ื”ืึทืœื˜ืŸ ืฉืคึผื™ืจืขื•ื•ื“ื™ืง ื“ืึทื˜ืŸ, ื•ื•ืึธืก ืื™ื– ื•ื•ื™ื“ืขืจ ืขืคื ื˜ืœืขืš ืงืขื ื˜ื™ืง ื“ื•ืจืš ืคืขืœื™ืงื™ื™ึทื˜. ื–ืืœ ืก ื‘ืึทื”ืึทืœื˜ืŸ ื“ื™ ื“ืึทื˜ืŸ ืคื•ืŸ ืคึผืจื™ื™ื ื’ ืื•ื™ื’ืŸ:

ืงืึทืœื™ืงืึธ ืคึฟืึทืจ ื ืขื˜ื•ื•ืึธืจืงื™ื ื’ ืื™ืŸ Kubernetes: ื”ืงื“ืžื” ืื•ืŸ ืึท ื‘ื™ืกืœ ื“ืขืจืคืึทืจื•ื ื’

ืคึผืจืึธืžืขื˜ื”ืขื•ืก, ื•ื•ื™ ืึท ื”ืขืจืฉืŸ, ืื™ื– ื’ืขืฉื˜ืขืœื˜ ืื™ืŸ ืึท ื‘ืึทื–ื•ื ื“ืขืจ ื“ื™ื ืกื˜ ืกื•ื•ื™ื•ื•ืข - ืื™ืŸ ื“ืขื ื‘ื™ื™ึทืฉืคึผื™ืœ, ืขืก ื•ื•ืขื˜ ื–ื™ื™ืŸ ืึท ื ืึทืžืขืกืคึผืึทืกืข ื•ื•ื™ ื“ืึธืก:

apiVersion: v1
kind: Namespace
metadata:
  labels:
    module: prometheus
  name: kube-prometheus

ืคืขืœื“ metadata.labels ื“ืึธืก ื”ืึธื˜ ื–ื™ืš ืึทืจื•ื™ืกื’ืขื•ื•ื™ื–ืŸ ื ื™ืฉื˜ ืงื™ื™ืŸ ืฆื•ืคืึทืœ. ื•ื•ื™ ื“ืขืจืžืื ื˜ ืื•ื™ื‘ืŸ, namespaceSelector (ื•ื•ื™ ืื•ื™ืš podSelector) ืึทืคึผืขืจื™ื™ืฅ ืžื™ื˜ ืœืึทื‘ืขืœืก. ื“ืขืจื™ื‘ืขืจ, ืฆื• ืœืึธื–ืŸ ืžืขื˜ืจื™ืงืก ืฆื• ื–ื™ื™ืŸ ื’ืขื ื•ืžืขืŸ ืคื•ืŸ ืึทืœืข ืคึผืึธื“ืก ืื•ื™ืฃ ืึท ืกืคึผืขืฆื™ืคื™ืฉ ืคึผืึธืจื˜, ืื™ืจ ื•ื•ืขื˜ ื”ืึธื‘ืŸ ืฆื• ืœื™ื™ื’ืŸ ืขื˜ืœืขื›ืข ืกืึธืจื˜ ืคื•ืŸ ืคื™ืจืžืข (ืึธื“ืขืจ ื ืขืžืขืŸ ืคื•ืŸ ื™ื’ื–ื™ืกื˜ื™ื ื’ ืึธื ืขืก) ืื•ืŸ ืฆื•ืœื™ื™ื’ืŸ ืึท ืงืึทื ืคื™ื’ื™ืขืจื™ื™ืฉืึทืŸ ื•ื•ื™:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-metrics-prom
spec:
  podSelector: {}
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          module: prometheus
    ports:
    - protocol: TCP
      port: 9100

ืื•ืŸ ืื•ื™ื‘ ืื™ืจ ื ื•ืฆืŸ ืงืึทืœื™ืงืึธ ืคึผืึทืœืึทืกื™ื–, โ€‹โ€‹ื“ื™ ืกื™ื ื˜ืึทืงืก ื•ื•ืขื˜ ื–ื™ื™ืŸ ื•ื•ื™ ื“ืึธืก:

apiVersion: crd.projectcalico.org/v1
kind: NetworkPolicy
metadata:
  name: allow-metrics-prom
spec:
  ingress:
  - action: Allow
    protocol: TCP
    source:
      namespaceSelector: module == 'prometheus'
    destination:
      ports:
      - 9100

ืื™ืŸ ืึทืœื’ืขืžื™ื™ืŸ, ื“ื•ืจืš ืึทื“ื™ื ื’ ื“ื™ ื˜ื™ื™ืคึผืก ืคื•ืŸ ืคึผืึทืœืึทืกื™ื– ืคึฟืึทืจ ืกืคึผืขืฆื™ืคื™ืฉ ื‘ืื“ืขืจืคืขื ื™ืฉืŸ, ืื™ืจ ืงืขื ืขืŸ ื‘ืึทืฉื™ืฆืŸ ืงืขื’ืŸ ื‘ื™ื™ื–ืข ืึธื“ืขืจ ืึทืงืกืึทื“ืขื ื˜ืึทืœ ื™ื ื˜ืขืจืคื™ืจืึทื ืก ืื™ืŸ ื“ื™ ืึธืคึผืขืจืึทืฆื™ืข ืคื•ืŸ โ€‹โ€‹ืึทืคึผืœืึทืงื™ื™ืฉืึทื ื– ืื™ืŸ ื“ืขื ืงื ื•ื™ืœ.

ื“ืขืจ ื‘ืขืกื˜ืขืจ ืคื™ืจ, ืœื•ื™ื˜ ื“ื™ ืงืจื™ื™ื™ื˜ืขืจื– ืคื•ืŸ ืงืึทืœื™ืงืึธ, ืื™ื– ื“ื™ "ืคืึทืจืฉืคึผืึทืจืŸ ืึทืœืฅ ืื•ืŸ ื‘ืคื™ืจื•ืฉ ืขืคึฟืขื ืขืŸ ื•ื•ืึธืก ืื™ืจ ื“ืึทืจืคึฟืŸ" ืฆื•ื’ืึทื ื’, ื“ืึทืงื™ื•ืžืขื ื˜ืึทื“ ืื™ืŸ ื‘ืึทืึทืžื˜ืขืจ ื“ืึทืงื™ื•ืžืขื ื˜ื™ื™ืฉืึทืŸ (ืื ื“ืขืจืข ื ืึธื›ื’ื™ื™ืŸ ืึท ืขื ืœืขืš ืฆื•ื’ืึทื ื’ - ืกืคึผืขืฆื™ืขืœ ืื™ืŸ ืฉื•ื™ืŸ ื“ืขืจืžืื ื˜ ืึทืจื˜ื™ืงืœ).

ื ื™ืฆืŸ ื ืึธืš ืงืึทืœื™ืงืึธ ืึธื‘ื“ื–ืฉืขืงืฅ

ืœืึธื–ืŸ ืžื™ืจ ื“ืขืจืžืึธื ืขืŸ ืื™ืจ ืึทื– ื“ื•ืจืš ื“ื™ ืขืงืกื˜ืขื ื“ืขื“ ื’ืึทื ื’ ืคื•ืŸ ืงืึทืœื™ืงืึธ ืึทืคึผื™ืก ืื™ืจ ืงืขื ืขืŸ ืจืขื’ื•ืœื™ืจืŸ ื“ื™ ืึทื•ื•ื™ื™ืœืึทื‘ื™ืœืึทื˜ื™ ืคื•ืŸ ื ืึธื•ื“ื–, ื ื™ื˜ ืœื™ืžื™ื˜ืขื“ ืฆื• ืคึผืึธื“ืก. ืื™ืŸ ื“ื™ ืคืืœื’ืขื ื“ืข ื‘ื™ื™ึทืฉืคึผื™ืœ ื ื™ืฆืŸ GlobalNetworkPolicy ื“ื™ ืคื™ื™ื™ืงื™ื™ื˜ ืฆื• ืคืึธืจืŸ ICMP ืจื™ืงื•ื•ืขืก ืื™ืŸ ื“ืขื ืงื ื•ื™ืœ ืื™ื– ืคึฟืึทืจืžืึทื›ื˜ (ืœืžืฉืœ, ืคึผื™ื ื’ืก ืคื•ืŸ ืึท ืคึผืึธื“ ืฆื• ืึท ื ืึธื“ืข, ืฆื•ื•ื™ืฉืŸ ืคึผืึธื“ืก ืึธื“ืขืจ ืคึฟื•ืŸ ืึท ื ืึธื“ืข ืฆื• ืึทืŸ IP ืคึผืึธื“):

apiVersion: crd.projectcalico.org/v1
kind: GlobalNetworkPolicy
metadata:
  name: block-icmp
spec:
  order: 200
  selector: all()
  types:
  - Ingress
  - Egress
  ingress:
  - action: Deny
    protocol: ICMP
  egress:
  - action: Deny
    protocol: ICMP

ืื™ืŸ ื“ื™ ืื•ื™ื‘ืŸ ืคืึทืœ, ืขืก ืื™ื– ื ืึธืš ืžืขื’ืœืขืš ืคึฟืึทืจ ืงื ื•ื™ืœ ื ืึธื•ื“ื– ืฆื• "ื“ืขืจื’ืจื™ื™ื›ืŸ ื–ื™ืš" ืฆื• ื™ืขื“ืขืจ ืื ื“ืขืจืขืจ ื“ื•ืจืš ICMP. ืื•ืŸ ื“ืขื ืขื ื™ืŸ ืื™ื– ืจื™ื–ืึทืœื•ื•ื“ ื“ื•ืจืš ืžื™ื˜ืœ GlobalNetworkPolicy, ื’ืขื•ื•ืขื ื“ื˜ ืฆื• ืึทืŸ ืขื ื˜ื™ื˜ื™ HostEndpoint:

apiVersion: crd.projectcalico.org/v1
kind: GlobalNetworkPolicy
metadata:
  name: deny-icmp-kube-02
spec:
  selector: "role == 'k8s-node'"
  order: 0
  ingress:
  - action: Allow
    protocol: ICMP
  egress:
  - action: Allow
    protocol: ICMP
---
apiVersion: crd.projectcalico.org/v1
kind: HostEndpoint
metadata:
  name: kube-02-eth0
  labels:
    role: k8s-node
spec:
  interfaceName: eth0
  node: kube-02
  expectedIPs: ["192.168.2.2"]

ื“ื™ VPN ืคืึทืœ

ืฆื•ื ืกื•ืฃ, ืื™ืš ื•ื•ืขืœ ื’ืขื‘ืŸ ืึท ื–ื™ื™ืขืจ ืคืึทืงื˜ื™ืฉ ื‘ื™ื™ึทืฉืคึผื™ืœ ืคื•ืŸ ื ื™ืฆืŸ ืงืึทืœื™ืงืึธ ืคืึทื ื’ืงืฉืึทื ื– ืคึฟืึทืจ ื“ื™ ืคืึทืœ ืคื•ืŸ ื ืึธืขื ื˜-ืงื ื•ื™ืœ ื™ื ื˜ืขืจืึทืงืฉืึทืŸ, ื•ื•ืขืŸ ืึท ื ืึธืจืžืึทืœ ื’ืึทื ื’ ืคื•ืŸ ืคึผืึทืœืึทืกื™ื– ืื™ื– ื ื™ืฉื˜ ื’ืขื ื•ื’. ืงืœื™ื™ืึทื ืฅ ื ื•ืฆืŸ ืึท VPN ื˜ื•ื ืขืœ ืฆื• ืึทืงืกืขืก ื“ื™ ื•ื•ืขื‘ ืึทืคึผืœืึทืงื™ื™ืฉืึทืŸ, ืื•ืŸ ื“ืขื ืึทืงืกืขืก ืื™ื– ื˜ื™ื™ื˜ืœื™ ืงืึทื ื˜ืจืึธื•ืœื“ ืื•ืŸ ืœื™ืžื™ื˜ืขื“ ืฆื• ืึท ืกืคึผืขืฆื™ืคื™ืฉ ืจืฉื™ืžื” ืคื•ืŸ ืกืขืจื•ื•ื™ืกืขืก ืขืจืœื•ื™ื‘ื˜ ืคึฟืึทืจ ื ื•ืฆืŸ:

ืงืึทืœื™ืงืึธ ืคึฟืึทืจ ื ืขื˜ื•ื•ืึธืจืงื™ื ื’ ืื™ืŸ Kubernetes: ื”ืงื“ืžื” ืื•ืŸ ืึท ื‘ื™ืกืœ ื“ืขืจืคืึทืจื•ื ื’

ืงืœื™ื™ืึทื ืฅ ืคืึทืจื‘ื™ื ื“ืŸ ืฆื• ื“ื™ ื•ื•ืคึผืŸ ื“ื•ืจืš ื ืึธืจืžืึทืœ UDP ืคึผืึธืจื˜ 1194 ืื•ืŸ, ื•ื•ืขืŸ ืงืึธื ื ืขืงื˜ืขื“, ื‘ืึทืงื•ืžืขืŸ ืจื•ืฅ ืฆื• ื“ื™ ืงื ื•ื™ืœ ืกื•ื‘ื ืขืฅ ืคื•ืŸ ืคึผืึธื“ืก ืื•ืŸ ื‘ืึทื“ื™ื ื•ื ื’ืก. ื’ืึทื ืฅ ืกื•ื‘ื ืขืฅ ื–ืขื ืขืŸ ืคึผื•ืฉื˜ ืึทื–ื•ื™ ื ื™ืฉื˜ ืฆื• ืคืึทืจืœื™ืจืŸ ืกืขืจื•ื•ื™ืกืขืก ื‘ืขืฉืึทืก ืจื™ืกื˜ืึทืจื˜ ืื•ืŸ ืึทื“ืจืขืก ืขื ื“ืขืจื•ื ื’ืขืŸ.

ื“ืขืจ ืคึผืึธืจื˜ ืื™ืŸ ื“ื™ ืงืึทื ืคื™ื’ื™ืขืจื™ื™ืฉืึทืŸ ืื™ื– ื ืึธืจืžืึทืœ, ื•ื•ืึธืก ื™ืžืคึผืึธื•ื–ืึทื– ืขื˜ืœืขื›ืข ื ื•ืึทื ืกื™ื– ืื•ื™ืฃ ื“ืขื ืคึผืจืึธืฆืขืก ืคื•ืŸ ืงืึทื ืคื™ื’ื™ืขืจ ื“ื™ ืึทืคึผืœืึทืงื™ื™ืฉืึทืŸ ืื•ืŸ ื˜ืจืึทื ืกืคืขืจื™ื ื’ ืขืก ืฆื• ื“ื™ Kubernetes ืงื ื•ื™ืœ. ืฆื•ื ื‘ื™ื™ืฉืคึผื™ืœ, ืื™ืŸ ื“ืขืจ ื–ืขืœื‘ื™ืงืขืจ AWS LoadBalancer ืคึฟืึทืจ UDP ืื™ื– ื’ืขื•ื•ืขืŸ ืžืžืฉ ืื™ืŸ ื“ื™ ืกื•ืฃ ืคื•ืŸ ืœืขืฆื˜ืข ื™ืึธืจ ืื™ืŸ ืึท ืœื™ืžื™ื˜ืขื“ ืจืฉื™ืžื” ืคื•ืŸ ืžืงื•ืžื•ืช, ืื•ืŸ NodePort ืงืขื ืขืŸ ื ื™ื˜ ื–ื™ื™ืŸ ื’ืขื•ื•ื™ื™ื ื˜ ืจืขื›ื˜ ืฆื• ื–ื™ื™ืŸ ืคืึธืจื•ื•ืขืจื“ื™ื ื’ ืื•ื™ืฃ ืึทืœืข ืงื ื•ื™ืœ ื ืึธื•ื“ื– ืื•ืŸ ืขืก ืื™ื– ืื•ืžืžืขื’ืœืขืš ืฆื• ื•ื•ืึธื’ ื“ื™ ื ื•ืžืขืจ ืคื•ืŸ ืกืขืจื•ื•ืขืจ ื™ื ืกื˜ืึทื ืกื™ื– ืคึฟืึทืจ. ืฉื•ืœื“ ื˜ืึธืœืขืจืึทื ืฅ ืฆื•ื•ืขืงืŸ. ืคึผืœื•ืก, ืื™ืจ ื•ื•ืขื˜ ื”ืึธื‘ืŸ ืฆื• ื˜ื•ื™ืฉืŸ ื“ื™ ืคืขืœื™ืงื™ื™ึทื˜ ืงื™ื™ื˜ ืคื•ืŸ ืคึผืึธืจืฅ ...

ื•ื•ื™ ืึท ืจืขื–ื•ืœื˜ืึทื˜ ืคื•ืŸ ื–ื•ื›ืŸ ื“ื•ืจืš ืžืขื’ืœืขืš ืกืึทืœื•ืฉืึทื ื–, ื“ื™ ืคืืœื’ืขื ื“ืข ืื™ื– ืื•ื™ืกื“ืขืจื•ื•ื™ื™ืœื˜:

  1. ืคึผืึธื“ืก ืžื™ื˜ VPN ื–ืขื ืขืŸ ืกืงืขื“ื–ืฉื•ืœื“ ืคึผืขืจ ื ืึธื“ืข ืื™ืŸ hostNetwork, ื“ืึธืก ืื™ื–, ืฆื• ื“ื™ ืคืึทืงื˜ื™ืฉ IP.
  2. ื“ื™ ื“ื™ื ืกื˜ ืื™ื– ืึทืจื™ื™ึทื ื’ืขืฉื™ืงื˜ ืึทืจื•ื™ืก ื“ื•ืจืš ClusterIP. ื ืคึผืึธืจื˜ ืื™ื– ืคื™ื–ื™ืงืœื™ ืื™ื ืกื˜ืึทืœื™ืจืŸ ืื•ื™ืฃ ื“ื™ ื ืึธื“ืข, ื•ื•ืึธืก ืื™ื– ืฆื•ื˜ืจื™ื˜ืœืขืš ืคึฟื•ืŸ ื“ื™ ืึทืจื•ื™ืก ืžื™ื˜ ืžื™ื ืขืจื•ื•ืขืจื˜ื™ืง ืจืขื–ืขืจื•ื•ื™ื™ืฉืึทื ื– (ืงืึทื ื“ื™ืฉืึทื ืึทืœ ื‘ื™ื™ึทื–ื™ื™ึทืŸ ืคื•ืŸ ืึท ืคืึทืงื˜ื™ืฉ IP ืึทื“ืจืขืก).
  3. ื“ื™ื˜ืขืจืžืึทื ื™ื ื’ ื“ื™ ื ืึธื“ืข ืื•ื™ืฃ ื•ื•ืึธืก ื“ื™ ืคึผืึธื“ ืจื•ื™ื– ืื™ื– ื•ื•ื™ื™ึทื˜ืขืจ ืคื•ืŸ ื“ื™ ืคืึทืจื ืขื ืคื•ืŸ ืื•ื ื“ื–ืขืจ ื’ืขืฉื™ื›ื˜ืข. ืื™ืš ื•ื•ืขื˜ ื ืึธืจ ื–ืึธื’ืŸ ืึทื– ืื™ืจ ืงืขื ืขืŸ ืคืึทืจื‘ื™ื ื“ืŸ ื“ื™ ืกืขืจื•ื•ื™ืก ืฆื• ืึท ื ืึธื“ืข ืึธื“ืขืจ ืฉืจื™ื™ึทื‘ืŸ ืึท ืงืœื™ื™ืŸ ืกื™ื™ื“ืงืึทืจ ื“ื™ื ืกื˜ ื•ื•ืึธืก ื•ื•ืขื˜ ืžืึธื ื™ื˜ืึธืจ ื“ื™ ืงืจืึทื ื˜ IP ืึทื“ืจืขืก ืคื•ืŸ ื“ื™ ื•ื•ืคึผืŸ ื“ื™ื ืกื˜ ืื•ืŸ ืจืขื“ืึทื’ื™ืจืŸ ื“ื™ ื“ื ืก ืจืขืงืึธืจื“ืก ืจืขื’ื™ืกื˜ืจื™ืจื˜ ืžื™ื˜ ืงืœื™ื™ืึทื ืฅ - ื•ื•ืขืจ ืกืข ื”ืื˜ ื’ืขื ื•ื’ ืคืึทื ื˜ืึทื–ื™ืข.

ืคึฟื•ืŸ ืึท ืจื•ื˜ื™ื ื’ ืคึผืขืจืกืคึผืขืงื˜ื™ื•ื•, ืžื™ืจ ืงืขื ืขืŸ ื™ื•ื ื™ืงืœื™ ื™ื“ืขื ื˜ื™ืคื™ืฆื™ืจืŸ ืึท ื•ื•ืคึผืŸ ืงืœื™ืขื ื˜ ื“ื•ืจืš ื–ื™ื™ืŸ IP ืึทื“ืจืขืก ืืจื•ื™ืก ื“ื•ืจืš ื“ื™ ื•ื•ืคึผืŸ ืกืขืจื•ื•ืขืจ. ื•ื ื˜ืขืจ ืื™ื– ืึท ืคึผืจื™ืžื™ื˜ื™ื•ื• ื‘ื™ื™ืฉืคึผื™ืœ ืคื•ืŸ ื‘ืึทื’ืจืขื ืขืฆืŸ ืึทื–ืึท ืึท ืงืœื™ืขื ื˜ ืก ืึทืงืกืขืก ืฆื• ืกืขืจื•ื•ื™ืกืขืก, ื™ืœืึทืกื˜ืจื™ื™ื˜ื™ื“ ืื•ื™ืฃ ื“ื™ ืื•ื™ื‘ืŸ-ื“ืขืจืžืื ื˜ Redis:

apiVersion: crd.projectcalico.org/v1
kind: HostEndpoint
metadata:
  name: vpnclient-eth0
  labels:
    role: vpnclient
    environment: production
spec:
  interfaceName: "*"
  node: kube-02
  expectedIPs: ["172.176.176.2"]
---
apiVersion: crd.projectcalico.org/v1
kind: GlobalNetworkPolicy
metadata:
  name: vpn-rules
spec:
  selector: "role == 'vpnclient'"
  order: 0
  applyOnForward: true
  preDNAT: true
  ingress:
  - action: Deny
    protocol: TCP
    destination:
      ports: [6379]
  - action: Allow
    protocol: UDP
    destination:
      ports: [53, 67]

ื“ืึธ, ืงืึทื ืขืงื˜ื™ื ื’ ืฆื• ืคึผืึธืจื˜ 6379 ืื™ื– ืฉื˜ืจืขื ื’ ืคึผืจืึธื•ื›ื™ื‘ืึทื˜ืึทื“, ืึธื‘ืขืจ ืื™ืŸ ื“ืขืจ ื–ืขืœื‘ื™ืงืขืจ ืฆื™ื™ึทื˜ ื“ื™ ืึธืคึผืขืจืึทืฆื™ืข ืคื•ืŸ โ€‹โ€‹ื“ื™ ื“ื ืก ื“ื™ื ืกื˜ ืื™ื– ืืคื’ืขื”ื™ื˜, ื“ื™ ืคืึทื ื’ืงืฉืึทื ื™ื ื’ ืคื•ืŸ ื•ื•ืึธืก ืกืึทืคืขืจื– ื’ืึทื ืฅ ืึธืคื˜ ื•ื•ืขืŸ ืฆื™ื™ื›ืขื ื•ื ื’ ื›ึผืœืœื™ื. ื•ื•ื™ื™ึทืœ, ื•ื•ื™ ืคืจื™ืขืจ ื“ืขืจืžืื ื˜, ื•ื•ืขืŸ ืึท ืกืขืœืขืงื˜ืึธืจ ืื•ื™ืก, ื“ื™ ืคืขืœื™ืงื™ื™ึทื˜ ืึธืคึผืœื™ื™ืงืขื ื•ื ื’ ืคึผืึธืœื™ื˜ื™ืง ืื™ื– ื’ืขื•ื•ืขื ื“ื˜ ืฆื• ืขืก ืกื™ื™ึทื“ืŸ ืึทื ื“ืขืจืฉ ืกืคึผืขืกื™ืคื™ืขื“.

ืจืขื–ื•ืœื˜ืึทื˜ืŸ ืคื•ืŸ

ืื–ื•ื™, ืžื™ื˜ ื“ื™ ืึทื•ื•ืึทื ืกื™ืจื˜ืข ืึทืคึผื™ ืคื•ืŸ Calico, ืื™ืจ ืงืขื ืขืŸ ืคืœืขืงืกืึทื‘ืึทืœ ืงืึทื ืคื™ื’ื™ืขืจ ืื•ืŸ ื“ื™ื ืึทืžื™ืงืึทืœืœื™ ื˜ื•ื™ืฉืŸ ืจื•ื˜ื™ื ื’ ืื™ืŸ ืื•ืŸ ืึทืจื•ื ื“ืขื ืงื ื•ื™ืœ. ืื™ืŸ ืึทืœื’ืขืžื™ื™ืŸ, ื“ื™ ื ื•ืฆืŸ ืงืขืŸ ืงื•ืงืŸ ื•ื•ื™ ืฆื• ื“ืจื™ื™ืขืŸ ืคื™ื™ื’ืึทืœื– ืžื™ื˜ ืึท ืงืึทื ืึธืŸ, ืื•ืŸ ื™ืžืคึผืœืึทืžืขื ื™ื ื’ ืึท L3 ื ืขืฅ ืžื™ื˜ BGP ืื•ืŸ IP-IP ื˜ืึทื ืึทืœื– ืงื•ืงื˜ ืžืึทื ืกื˜ืจืึทืก ืื™ืŸ ืึท ืคึผืฉื•ื˜ ืงื•ื‘ืขืจื ืขื˜ืขืก ื™ื ืกื˜ืึทืœื™ืจื•ื ื’ ืื•ื™ืฃ ืึท ืคืœืึทืš ื ืขืฅ ... ืึธื‘ืขืจ, ืึทื ื“ืขืจืฉ ื“ื™ ื’ืขืฆื™ื™ึทื’ ืงื•ืงื˜ ื’ืึทื ืฅ ื•ื•ื™ื™ืึทื‘ืึทืœ ืื•ืŸ ื ื•ืฆื™ืง .

ื™ื–ืึธืœื™ืจืŸ ืึท ืงื ื•ื™ืœ ืฆื• ื˜ืจืขืคืŸ ื–ื™ื›ืขืจื”ื™ื™ื˜ ืจืขืงื•ื•ื™ืจืขืžืขื ืฅ ืงืขืŸ ื ื™ืฉื˜ ืฉื˜ืขื ื“ื™ืง ื–ื™ื™ืŸ ืคื™ื–ืึทื‘ืึทืœ, ืื•ืŸ ื“ืึธืก ืื™ื– ื•ื•ื• ืงืึทืœื™ืงืึธ (ืึธื“ืขืจ ืึท ืขื ืœืขืš ืœื™ื™ื–ื•ื ื’) ืงื•ืžื˜ ืฆื• ืจืึทื˜ืขื•ื•ืขืŸ. ื“ื™ ื‘ื™ื™ืฉืคื™ืœืŸ ื’ืขื’ืขื‘ืŸ ืื™ืŸ ื“ืขื ืึทืจื˜ื™ืงืœ (ืžื™ื˜ ืžื™ื ืขืจื•ื•ืขืจื˜ื™ืง ืžืึธื“ื™ืคื™ืงืึทื˜ื™ืึธื ืก) ื–ืขื ืขืŸ ื’ืขื ื™ืฆื˜ ืื™ืŸ ืขื˜ืœืขื›ืข ื™ื ืกื˜ืึทืœื™ื™ืฉืึทื ื– ืคื•ืŸ ืื•ื ื“ื–ืขืจ ืงืœื™ื™ืึทื ืฅ ืื™ืŸ AWS.

ืคึผืก

ืœื™ื™ืขื ืขืŸ ืื•ื™ืš ืื•ื™ืฃ ืื•ื ื“ื–ืขืจ ื‘ืœืึธื’:

ืžืงื•ืจ: www.habr.com

ืœื™ื™ื’ืŸ ืึท ื‘ืึทืžืขืจืงื•ื ื’