Как Ρ€Π°Π±ΠΎΡ‚Π°Π΅Ρ‚ kubectl exec?

ΠŸΡ€ΠΈΠΌ. ΠΏΠ΅Ρ€Π΅Π².: Π°Π²Ρ‚ΠΎΡ€ ΡΡ‚Π°Ρ‚ΡŒΠΈ β€” Erkan Erol, ΠΈΠ½ΠΆΠ΅Π½Π΅Ρ€ ΠΈΠ· SAP β€” дСлится своим ΠΈΠ·ΡƒΡ‡Π΅Π½ΠΈΠ΅ΠΌ ΠΌΠ΅Ρ…Π°Π½ΠΈΠ·ΠΌΠΎΠ² функционирования ΠΊΠΎΠΌΠ°Π½Π΄Ρ‹ kubectl exec, ΡΡ‚ΠΎΠ»ΡŒ ΠΏΡ€ΠΈΠ²Ρ‹Ρ‡Π½ΠΎΠΉ для всСх, ΠΊΡ‚ΠΎ Ρ€Π°Π±ΠΎΡ‚Π°Π΅Ρ‚ с Kubernetes. Π’Π΅ΡΡŒ Π°Π»Π³ΠΎΡ€ΠΈΡ‚ΠΌ ΠΎΠ½ сопровоТдаСт листингами исходного ΠΊΠΎΠ΄Π° Kubernetes (ΠΈ связанных ΠΏΡ€ΠΎΠ΅ΠΊΡ‚ΠΎΠ²), ΠΊΠΎΡ‚ΠΎΡ€Ρ‹Π΅ ΠΏΠΎΠ·Π²ΠΎΠ»ΡΡŽΡ‚ Ρ€Π°Π·ΠΎΠ±Ρ€Π°Ρ‚ΡŒΡΡ Π² Ρ‚Π΅ΠΌΠ΅ Π½Π°ΡΡ‚ΠΎΠ»ΡŒΠΊΠΎ Π³Π»ΡƒΠ±ΠΎΠΊΠΎ, насколько это трСбуСтся.

Как Ρ€Π°Π±ΠΎΡ‚Π°Π΅Ρ‚ kubectl exec?

Π’ ΠΎΠ΄Π½Ρƒ ΠΈΠ· пятниц ΠΊΠΎ ΠΌΠ½Π΅ подошСл ΠΊΠΎΠ»Π»Π΅Π³Π° ΠΈ поинтСрСсовался, ΠΊΠ°ΠΊ Π²Ρ‹ΠΏΠΎΠ»Π½ΠΈΡ‚ΡŒ ΠΊΠΎΠΌΠ°Π½Π΄Ρƒ Π² pod’Π΅ с ΠΏΠΎΠΌΠΎΡ‰ΡŒΡŽ client-go. Π― Π½Π΅ смог Π΅ΠΌΡƒ ΠΎΡ‚Π²Π΅Ρ‚ΠΈΡ‚ΡŒ ΠΈ Π²Π½Π΅Π·Π°ΠΏΠ½ΠΎ осознал, Ρ‡Ρ‚ΠΎ Π½ΠΈΡ‡Π΅Π³ΠΎ Π½Π΅ знаю ΠΎ ΠΌΠ΅Ρ…Π°Π½ΠΈΠ·ΠΌΠ΅ Ρ€Π°Π±ΠΎΡ‚Ρ‹ kubectl exec. Π”Π°, Ρƒ мСня Π±Ρ‹Π»ΠΈ ΠΎΠΏΡ€Π΅Π΄Π΅Π»Π΅Π½Π½Ρ‹Π΅ прСдставлСния ΠΎ Π΅Π³ΠΎ устройствС, ΠΎΠ΄Π½Π°ΠΊΠΎ я Π½Π΅ Π±Ρ‹Π» ΡƒΠ²Π΅Ρ€Π΅Π½ Π½Π° 100% Π² ΠΈΡ… ΠΏΡ€Π°Π²ΠΈΠ»ΡŒΠ½ΠΎΡΡ‚ΠΈ ΠΈ ΠΏΠΎΡ‚ΠΎΠΌΡƒ Ρ€Π΅ΡˆΠΈΠ» Π·Π°Π½ΡΡ‚ΡŒΡΡ этим вопросом. ΠŸΡ€ΠΎΡˆΡ‚ΡƒΠ΄ΠΈΡ€ΠΎΠ²Π°Π² Π±Π»ΠΎΠ³ΠΈ, Π΄ΠΎΠΊΡƒΠΌΠ΅Π½Ρ‚Π°Ρ†ΠΈΡŽ ΠΈ исходный ΠΊΠΎΠ΄, ΡƒΠ·Π½Π°Π» ΠΌΠ½ΠΎΠ³ΠΎ Π½ΠΎΠ²ΠΎΠ³ΠΎ, ΠΈ Π² этой ΡΡ‚Π°Ρ‚ΡŒΠ΅ Ρ…ΠΎΡ‡Ρƒ ΠΏΠΎΠ΄Π΅Π»ΠΈΡ‚ΡŒΡΡ своими открытиями ΠΈ ΠΏΠΎΠ½ΠΈΠΌΠ°Π½ΠΈΠ΅ΠΌ. Если Ρ‡Ρ‚ΠΎ-Ρ‚ΠΎ Π½Π΅ Ρ‚Π°ΠΊ, поТалуйста, ΡΠ²ΡΠΆΠΈΡ‚Π΅ΡΡŒ со ΠΌΠ½ΠΎΠΉ Π² Twitter.

ΠŸΠΎΠ΄Π³ΠΎΡ‚ΠΎΠ²ΠΊΠ°

Π§Ρ‚ΠΎΠ±Ρ‹ ΡΠΎΠ·Π΄Π°Ρ‚ΡŒ кластСр Π½Π° MacBook’Π΅, я склонировал ecomm-integration-ballerina/kubernetes-cluster. Π—Π°Ρ‚Π΅ΠΌ ΠΏΠΎΠΏΡ€Π°Π²ΠΈΠ» IP-адрСса ΡƒΠ·Π»ΠΎΠ² Π² ΠΊΠΎΠ½Ρ„ΠΈΠ³Π΅ kubelet’а, ΠΏΠΎΡΠΊΠΎΠ»ΡŒΠΊΡƒ настройки ΠΏΠΎ ΡƒΠΌΠΎΠ»Ρ‡Π°Π½ΠΈΡŽ Π½Π΅ позволяли Π²Ρ‹ΠΏΠΎΠ»Π½ΡΡ‚ΡŒ kubectl exec. ΠŸΠΎΠ΄Ρ€ΠΎΠ±Π½Π΅Π΅ ΠΎΠ± основной ΠΏΡ€ΠΈΡ‡ΠΈΠ½Π΅ Ρ‚ΠΎΠΌΡƒ ΠΌΠΎΠΆΠ½ΠΎ ΠΏΡ€ΠΎΡ‡ΠΈΡ‚Π°Ρ‚ΡŒ здСсь.

  • Π›ΡŽΠ±Π°Ρ машина = ΠΌΠΎΠΉ MacBook
  • IP master-ΡƒΠ·Π»Π° = 192.168.205.10
  • IP worker-ΡƒΠ·Π»Π° = 192.168.205.11
  • ΠΏΠΎΡ€Ρ‚ API-сСрвСра = 6443

ΠšΠΎΠΌΠΏΠΎΠ½Π΅Π½Ρ‚Ρ‹

Как Ρ€Π°Π±ΠΎΡ‚Π°Π΅Ρ‚ kubectl exec?

  • kubectl exec process: ΠΊΠΎΠ³Π΄Π° ΠΌΡ‹ выполняСм Β«kubectl exec …», запускаСтся процСсс. Π”Π΅Π»Π°Ρ‚ΡŒ это ΠΌΠΎΠΆΠ½ΠΎ Π½Π° любой машинС с доступом ΠΊ API-сСрвСру K8s. ΠŸΡ€ΠΈΠΌ. ΠΏΠ΅Ρ€Π΅Π².: Π”Π°Π»Π΅Π΅ Π² ΠΊΠΎΠ½ΡΠΎΠ»ΡŒΠ½Ρ‹Ρ… листингах Π°Π²Ρ‚ΠΎΡ€ ΠΈΡΠΏΠΎΠ»ΡŒΠ·ΡƒΠ΅Ρ‚ ΠΊΠΎΠΌΠΌΠ΅Π½Ρ‚Π°Ρ€ΠΈΠΉ Β«any machineΒ», подразумСвая, Ρ‡Ρ‚ΠΎ ΠΏΠΎΡΠ»Π΅Π΄ΡƒΡŽΡ‰ΠΈΠ΅ ΠΊΠΎΠΌΠ°Π½Π΄Ρ‹ ΠΌΠΎΠΆΠ½ΠΎ Π²Ρ‹ΠΏΠΎΠ»Π½ΡΡ‚ΡŒ Π½Π° Π»ΡŽΠ±Ρ‹Ρ… Ρ‚Π°ΠΊΠΈΡ… ΠΌΠ°ΡˆΠΈΠ½Π°Ρ… с доступом ΠΊ Kubernetes.
  • api server: ΠΊΠΎΠΌΠΏΠΎΠ½Π΅Π½Ρ‚ Π½Π° мастСр-ΡƒΠ·Π»Π΅, ΠΏΡ€Π΅Π΄ΠΎΡΡ‚Π°Π²Π»ΡΡŽΡ‰ΠΈΠΉ доступ ΠΊ API Kubernetes. Π­Ρ‚ΠΎ Ρ„Ρ€ΠΎΠ½Ρ‚Π΅Π½Π΄ для control plane Π² Kubernetes.
  • kubelet: Π°Π³Π΅Π½Ρ‚, ΠΊΠΎΡ‚ΠΎΡ€Ρ‹ΠΉ Ρ€Π°Π±ΠΎΡ‚Π°Π΅Ρ‚ Π½Π° ΠΊΠ°ΠΆΠ΄ΠΎΠΌ ΡƒΠ·Π»Π΅ Π² кластСрС. Он обСспСчиваСт Ρ€Π°Π±ΠΎΡ‚Ρƒ ΠΊΠΎΠ½Ρ‚Π΅ΠΉΠ½Π΅Ρ€ΠΎΠ² Π² pod’Π΅.
  • container runtime (исполняСмая срСда ΠΊΠΎΠ½Ρ‚Π΅ΠΉΠ½Π΅Ρ€Π°): ΠΏΡ€ΠΎΠ³Ρ€Π°ΠΌΠΌΠ½ΠΎΠ΅ обСспСчСниС, ΠΎΡ‚Π²Π΅Ρ‡Π°ΡŽΡ‰Π΅Π΅ Π·Π° Ρ€Π°Π±ΠΎΡ‚Ρƒ ΠΊΠΎΠ½Ρ‚Π΅ΠΉΠ½Π΅Ρ€ΠΎΠ². ΠŸΡ€ΠΈΠΌΠ΅Ρ€Ρ‹: Docker, CRI-O, containerd…
  • kernel: ядро ОБ Π½Π° Ρ€Π°Π±ΠΎΡ‡Π΅ΠΌ ΡƒΠ·Π»Π΅; ΠΎΡ‚Π²Π΅Ρ‡Π°Π΅Ρ‚ Π·Π° ΡƒΠΏΡ€Π°Π²Π»Π΅Π½ΠΈΠ΅ процСссами.
  • target (Ρ†Π΅Π»Π΅Π²ΠΎΠΉ) container: ΠΊΠΎΠ½Ρ‚Π΅ΠΉΠ½Π΅Ρ€, ΡΠ²Π»ΡΡŽΡ‰ΠΈΠΉΡΡ Ρ‡Π°ΡΡ‚ΡŒΡŽ pod’Π° ΠΈ Ρ„ΡƒΠ½ΠΊΡ†ΠΈΠΎΠ½ΠΈΡ€ΡƒΡŽΡ‰ΠΈΠΉ Π½Π° ΠΎΠ΄Π½ΠΎΠΌ ΠΈΠ· Ρ€Π°Π±ΠΎΡ‡ΠΈΡ… ΡƒΠ·Π»ΠΎΠ².

Π§Ρ‚ΠΎ я ΠΎΠ±Π½Π°Ρ€ΡƒΠΆΠΈΠ»

1. ΠΠΊΡ‚ΠΈΠ²Π½ΠΎΡΡ‚ΡŒ Π½Π° сторонС ΠΊΠ»ΠΈΠ΅Π½Ρ‚Π°

Π‘ΠΎΠ·Π΄Π°Π΅ΠΌ pod Π² пространствС ΠΈΠΌΠ΅Π½ default:

// any machine
$ kubectl run exec-test-nginx --image=nginx

Π—Π°Ρ‚Π΅ΠΌ выполняСм ΠΊΠΎΠΌΠ°Π½Π΄Ρƒ exec ΠΈ ΠΆΠ΄Π΅ΠΌ 5000 сСкунд для Π΄Π°Π»ΡŒΠ½Π΅ΠΉΡˆΠΈΡ… наблюдСний:

// any machine
$ kubectl exec -it exec-test-nginx-6558988d5-fgxgg -- sh
# sleep 5000

ΠŸΠΎΡΠ²Π»ΡΠ΅Ρ‚ΡΡ процСсс kubectl (с pid=8507 Π² нашСм случаС):

// any machine
$ ps -ef |grep kubectl
501  8507  8409   0  7:19PM ttys000    0:00.13 kubectl exec -it exec-test-nginx-6558988d5-fgxgg -- sh

Если ΠΏΡ€ΠΎΠ²Π΅Ρ€ΠΈΡ‚ΡŒ ΡΠ΅Ρ‚Π΅Π²ΡƒΡŽ Π°ΠΊΡ‚ΠΈΠ²Π½ΠΎΡΡ‚ΡŒ процСсса, ΠΌΡ‹ ΠΎΠ±Π½Π°Ρ€ΡƒΠΆΠΈΠΌ, Ρ‡Ρ‚ΠΎ Ρƒ Π½Π΅Π³ΠΎ Π΅ΡΡ‚ΡŒ ΠΏΠΎΠ΄ΠΊΠ»ΡŽΡ‡Π΅Π½ΠΈΡ ΠΊ api-server’Ρƒ (192.168.205.10.6443):

// any machine
$ netstat -atnv |grep 8507
tcp4       0      0  192.168.205.1.51673    192.168.205.10.6443    ESTABLISHED 131072 131768   8507      0 0x0102 0x00000020
tcp4       0      0  192.168.205.1.51672    192.168.205.10.6443    ESTABLISHED 131072 131768   8507      0 0x0102 0x00000028

Π”Π°Π²Π°ΠΉΡ‚Π΅ посмотрим Π½Π° ΠΊΠΎΠ΄. Kubectl создаСт POST-запрос с субрСсурсом exec ΠΈ посылаСт REST-запрос:

              req := restClient.Post().
                        Resource("pods").
                        Name(pod.Name).
                        Namespace(pod.Namespace).
                        SubResource("exec")
                req.VersionedParams(&corev1.PodExecOptions{
                        Container: containerName,
                        Command:   p.Command,
                        Stdin:     p.Stdin,
                        Stdout:    p.Out != nil,
                        Stderr:    p.ErrOut != nil,
                        TTY:       t.Raw,
                }, scheme.ParameterCodec)

                return p.Executor.Execute("POST", req.URL(), p.Config, p.In, p.Out, p.ErrOut, t.Raw, sizeQueue)

(kubectl/pkg/cmd/exec/exec.go)

Как Ρ€Π°Π±ΠΎΡ‚Π°Π΅Ρ‚ kubectl exec?

2. ΠΠΊΡ‚ΠΈΠ²Π½ΠΎΡΡ‚ΡŒ Π½Π° сторонС мастСр-ΡƒΠ·Π»Π°

ΠœΡ‹ Ρ‚Π°ΠΊΠΆΠ΅ ΠΌΠΎΠΆΠ΅ΠΌ Π½Π°Π±Π»ΡŽΠ΄Π°Ρ‚ΡŒ запрос Π½Π° сторонС api-server’Π°:

handler.go:143] kube-apiserver: POST "/api/v1/namespaces/default/pods/exec-test-nginx-6558988d5-fgxgg/exec" satisfied by gorestful with webservice /api/v1
upgradeaware.go:261] Connecting to backend proxy (intercepting redirects) https://192.168.205.11:10250/exec/default/exec-test-nginx-6558988d5-fgxgg/exec-test-nginx?command=sh&input=1&output=1&tty=1
Headers: map[Connection:[Upgrade] Content-Length:[0] Upgrade:[SPDY/3.1] User-Agent:[kubectl/v1.12.10 (darwin/amd64) kubernetes/e3c1340] X-Forwarded-For:[192.168.205.1] X-Stream-Protocol-Version:[v4.channel.k8s.io v3.channel.k8s.io v2.channel.k8s.io channel.k8s.io]]

ΠžΠ±Ρ€Π°Ρ‚ΠΈΡ‚Π΅ Π²Π½ΠΈΠΌΠ°Π½ΠΈΠ΅, Ρ‡Ρ‚ΠΎ HTTP-запрос Π²ΠΊΠ»ΡŽΡ‡Π°Π΅Ρ‚ запрос Π½Π° ΠΈΠ·ΠΌΠ΅Π½Π΅Π½ΠΈΠ΅ ΠΏΡ€ΠΎΡ‚ΠΎΠΊΠΎΠ»Π°. SPDY позволяСт ΠΌΡƒΠ»ΡŒΡ‚ΠΈΠΏΠ»Π΅ΠΊΡΠΈΡ€ΠΎΠ²Π°Ρ‚ΡŒ ΠΎΡ‚Π΄Π΅Π»ΡŒΠ½Ρ‹Π΅ Β«ΠΏΠΎΡ‚ΠΎΠΊΠΈΒ» stdin/stdout/stderr/spdy-error Ρ‡Π΅Ρ€Π΅Π· Π΅Π΄ΠΈΠ½ΠΎΠ΅ TCP-соСдинСниС.

API-сСрвСр ΠΏΠΎΠ»ΡƒΡ‡Π°Π΅Ρ‚ запрос ΠΈ ΠΏΡ€Π΅ΠΎΠ±Ρ€Π°Π·ΡƒΠ΅Ρ‚ Π΅Π³ΠΎ Π² PodExecOptions:

// PodExecOptions is the query options to a Pod's remote exec call
type PodExecOptions struct {
        metav1.TypeMeta

        // Stdin if true indicates that stdin is to be redirected for the exec call
        Stdin bool

        // Stdout if true indicates that stdout is to be redirected for the exec call
        Stdout bool

        // Stderr if true indicates that stderr is to be redirected for the exec call
        Stderr bool

        // TTY if true indicates that a tty will be allocated for the exec call
        TTY bool

        // Container in which to execute the command.
        Container string

        // Command is the remote command to execute; argv array; not executed within a shell.
        Command []string
}

(pkg/apis/core/types.go)

Π§Ρ‚ΠΎΠ±Ρ‹ Π²Ρ‹ΠΏΠΎΠ»Π½ΠΈΡ‚ΡŒ Ρ‚Ρ€Π΅Π±ΡƒΠ΅ΠΌΡ‹Π΅ дСйствия, api-server Π΄ΠΎΠ»ΠΆΠ΅Π½ Π·Π½Π°Ρ‚ΡŒ, с ΠΊΠ°ΠΊΠΈΠΌ pod’ΠΎΠΌ Π΅ΠΌΡƒ Π½Π΅ΠΎΠ±Ρ…ΠΎΠ΄ΠΈΠΌΠΎ ΡΠ²ΡΠ·Π°Ρ‚ΡŒΡΡ:

// ExecLocation returns the exec URL for a pod container. If opts.Container is blank
// and only one container is present in the pod, that container is used.
func ExecLocation(
        getter ResourceGetter,
        connInfo client.ConnectionInfoGetter,
        ctx context.Context,
        name string,
        opts *api.PodExecOptions,
) (*url.URL, http.RoundTripper, error) {
        return streamLocation(getter, connInfo, ctx, name, opts, opts.Container, "exec")
}

(pkg/registry/core/pod/strategy.go)

ΠšΠΎΠ½Π΅Ρ‡Π½ΠΎ, Π΄Π°Π½Π½Ρ‹Π΅ ΠΎΠ± endpoint’Π΅ бСрутся ΠΈΠ· ΠΈΠ½Ρ„ΠΎΡ€ΠΌΠ°Ρ†ΠΈΠΈ ΠΎΠ± ΡƒΠ·Π»Π΅:

        nodeName := types.NodeName(pod.Spec.NodeName)
        if len(nodeName) == 0 {
                // If pod has not been assigned a host, return an empty location
                return nil, nil, errors.NewBadRequest(fmt.Sprintf("pod %s does not have a host assigned", name))
        }
        nodeInfo, err := connInfo.GetConnectionInfo(ctx, nodeName)

(pkg/registry/core/pod/strategy.go)

Π£Ρ€Π°! Π£ kubelet’Π° Ρ‚Π΅ΠΏΠ΅Ρ€ΡŒ Π΅ΡΡ‚ΡŒ ΠΏΠΎΡ€Ρ‚ (node.Status.DaemonEndpoints.KubeletEndpoint.Port), ΠΊ ΠΊΠΎΡ‚ΠΎΡ€ΠΎΠΌΡƒ ΠΌΠΎΠΆΠ΅Ρ‚ ΠΏΠΎΠ΄ΠΊΠ»ΡŽΡ‡ΠΈΡ‚ΡŒΡΡ API-сСрвСр:

// GetConnectionInfo retrieves connection info from the status of a Node API object.
func (k *NodeConnectionInfoGetter) GetConnectionInfo(ctx context.Context, nodeName types.NodeName) (*ConnectionInfo, error) {
        node, err := k.nodes.Get(ctx, string(nodeName), metav1.GetOptions{})
        if err != nil {
                return nil, err
        }

        // Find a kubelet-reported address, using preferred address type
        host, err := nodeutil.GetPreferredNodeAddress(node, k.preferredAddressTypes)
        if err != nil {
                return nil, err
        }

        // Use the kubelet-reported port, if present
        port := int(node.Status.DaemonEndpoints.KubeletEndpoint.Port)
        if port <= 0 {
                port = k.defaultPort
        }

        return &ConnectionInfo{
                Scheme:    k.scheme,
                Hostname:  host,
                Port:      strconv.Itoa(port),
                Transport: k.transport,
        }, nil
}

(pkg/kubelet/client/kubelet_client.go)

Из Π΄ΠΎΠΊΡƒΠΌΠ΅Π½Ρ‚Π°Ρ†ΠΈΠΈ Master-Node Communication > Master to Cluster > apiserver to kubelet:

Π­Ρ‚ΠΈ ΠΏΠΎΠ΄ΠΊΠ»ΡŽΡ‡Π΅Π½ΠΈΡ Π·Π°ΠΌΡ‹ΠΊΠ°ΡŽΡ‚ΡΡ Π½Π° HTTPS endpoint’Π΅ kubelet’Π°. По ΡƒΠΌΠΎΠ»Ρ‡Π°Π½ΠΈΡŽ, apiserver Π½Π΅ провСряСт сСртификат kubelet’Π°, Ρ‡Ρ‚ΠΎ Π΄Π΅Π»Π°Π΅Ρ‚ соСдинСниС уязвимым ΠΊ Β«Π°Ρ‚Π°ΠΊΠ°ΠΌ посрСдника» (MITM) ΠΈ нСбСзопасным для Ρ€Π°Π±ΠΎΡ‚Ρ‹ Π² Π½Π΅Π½Π°Π΄Π΅ΠΆΠ½Ρ‹Ρ… ΠΈ/ΠΈΠ»ΠΈ ΠΏΡƒΠ±Π»ΠΈΡ‡Π½Ρ‹Ρ… сСтях.

Π’Π΅ΠΏΠ΅Ρ€ΡŒ API-сСрвСр Π·Π½Π°Π΅Ρ‚ endpoint ΠΈ устанавливаСт соСдинСниС:

// Connect returns a handler for the pod exec proxy
func (r *ExecREST) Connect(ctx context.Context, name string, opts runtime.Object, responder rest.Responder) (http.Handler, error) {
        execOpts, ok := opts.(*api.PodExecOptions)
        if !ok {
                return nil, fmt.Errorf("invalid options object: %#v", opts)
        }
        location, transport, err := pod.ExecLocation(r.Store, r.KubeletConn, ctx, name, execOpts)
        if err != nil {
                return nil, err
        }
        return newThrottledUpgradeAwareProxyHandler(location, transport, false, true, true, responder), nil
}

(pkg/registry/core/pod/rest/subresources.go)

Π”Π°Π²Π°ΠΉΡ‚Π΅ посмотрим, Ρ‡Ρ‚ΠΎ происходит Π½Π° мастСр-ΡƒΠ·Π»Π΅.

Π‘Π½Π°Ρ‡Π°Π»Π° ΡƒΠ·Π½Π°Π΅ΠΌ IP Ρ€Π°Π±ΠΎΡ‡Π΅Π³ΠΎ ΡƒΠ·Π»Π°. Π’ нашСм случаС это 192.168.205.11:

// any machine
$ kubectl get nodes k8s-node-1 -o wide
NAME         STATUS   ROLES    AGE   VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
k8s-node-1   Ready    <none>   9h    v1.15.3   192.168.205.11   <none>        Ubuntu 16.04.6 LTS   4.4.0-159-generic   docker://17.3.3

Π—Π°Ρ‚Π΅ΠΌ установим ΠΏΠΎΡ€Ρ‚ kubelet’Π° (10250 Π² нашСм случаС):

// any machine
$ kubectl get nodes k8s-node-1 -o jsonpath='{.status.daemonEndpoints.kubeletEndpoint}'
map[Port:10250]

Π’Π΅ΠΏΠ΅Ρ€ΡŒ ΠΏΠΎΡ€Π° ΠΏΡ€ΠΎΠ²Π΅Ρ€ΠΈΡ‚ΡŒ ΡΠ΅Ρ‚ΡŒ. Π•ΡΡ‚ΡŒ Π»ΠΈ ΠΏΠΎΠ΄ΠΊΠ»ΡŽΡ‡Π΅Π½ΠΈΠ΅ ΠΊ Ρ€Π°Π±ΠΎΡ‡Π΅ΠΌΡƒ ΡƒΠ·Π»Ρƒ (192.168.205.11)? Оно Π΅ΡΡ‚ΡŒ! Если Β«ΡƒΠ±ΠΈΡ‚ΡŒΒ» процСсс exec, ΠΎΠ½ΠΎ исчСзнСт, поэтому я знаю, Ρ‡Ρ‚ΠΎ ΠΏΠΎΠ΄ΠΊΠ»ΡŽΡ‡Π΅Π½ΠΈΠ΅ установлСно api-server’ΠΎΠΌ ΠΊΠ°ΠΊ слСдствиС Π²Ρ‹ΠΏΠΎΠ»Π½Π΅Π½Π½ΠΎΠΉ exec-ΠΊΠΎΠΌΠ°Π½Π΄Ρ‹.

// master node
$ netstat -atn |grep 192.168.205.11
tcp        0      0 192.168.205.10:37870    192.168.205.11:10250    ESTABLISHED
…

Как Ρ€Π°Π±ΠΎΡ‚Π°Π΅Ρ‚ kubectl exec?

Π‘ΠΎΠ΅Π΄ΠΈΠ½Π΅Π½ΠΈΠ΅ ΠΌΠ΅ΠΆΠ΄Ρƒ kubectl’ΠΎΠΌ ΠΈ api-server’ΠΎΠΌ ΠΏΠΎ-ΠΏΡ€Π΅ΠΆΠ½Π΅ΠΌΡƒ ΠΎΡ‚ΠΊΡ€Ρ‹Ρ‚ΠΎ. ΠšΡ€ΠΎΠΌΠ΅ Ρ‚ΠΎΠ³ΠΎ, Π΅ΡΡ‚ΡŒ Π΅Ρ‰Π΅ ΠΎΠ΄Π½ΠΎ соСдинСниС, ΡΠ²ΡΠ·Ρ‹Π²Π°ΡŽΡ‰Π΅Π΅ api-server ΠΈ kubelet.

3. ΠΠΊΡ‚ΠΈΠ²Π½ΠΎΡΡ‚ΡŒ Π½Π° Ρ€Π°Π±ΠΎΡ‡Π΅ΠΌ ΡƒΠ·Π»Π΅

Π’Π΅ΠΏΠ΅Ρ€ΡŒ Π΄Π°Π²Π°ΠΉΡ‚Π΅ ΠΏΠΎΠ΄ΠΊΠ»ΡŽΡ‡ΠΈΠΌΡΡ ΠΊ worker-ΡƒΠ·Π»Ρƒ ΠΈ посмотрим, Ρ‡Ρ‚ΠΎ происходит Π½Π° Π½Π΅ΠΌ.

ΠŸΡ€Π΅ΠΆΠ΄Π΅ всСго ΠΌΡ‹ Π²ΠΈΠ΄ΠΈΠΌ, Ρ‡Ρ‚ΠΎ соСдинСниС с Π½ΠΈΠΌ Ρ‚Π°ΠΊΠΆΠ΅ установлСно (вторая строка); 192.168.205.10 β€” это IP master-ΡƒΠ·Π»Π°:

 // worker node
  $ netstat -atn |grep 10250
  tcp6       0      0 :::10250                :::*                    LISTEN
  tcp6       0      0 192.168.205.11:10250    192.168.205.10:37870    ESTABLISHED

А ΠΊΠ°ΠΊ насчСт нашСй ΠΊΠΎΠΌΠ°Π½Π΄Ρ‹ sleep? Π£Ρ€Π°, ΠΎΠ½Π° Ρ‚ΠΎΠΆΠ΅ присутствуСт!

 // worker node
  $ ps -afx
  ...
  31463 ?        Sl     0:00      _ docker-containerd-shim 7d974065bbb3107074ce31c51f5ef40aea8dcd535ae11a7b8f2dd180b8ed583a /var/run/docker/libcontainerd/7d974065bbb3107074ce31c51
  31478 pts/0    Ss     0:00          _ sh
  31485 pts/0    S+     0:00              _ sleep 5000
  …

Но постойтС: ΠΊΠ°ΠΊ kubelet ΠΏΡ€ΠΎΠ²Π΅Ρ€Π½ΡƒΠ» это? Π’ kubelet Π΅ΡΡ‚ΡŒ Π΄Π΅ΠΌΠΎΠ½, ΠΊΠΎΡ‚ΠΎΡ€Ρ‹ΠΉ ΠΎΡ‚ΠΊΡ€Ρ‹Π²Π°Π΅Ρ‚ доступ ΠΊ API Ρ‡Π΅Ρ€Π΅Π· ΠΏΠΎΡ€Ρ‚ для запросов api-server’Π°:

// Server is the library interface to serve the stream requests.
type Server interface {
        http.Handler

        // Get the serving URL for the requests.
        // Requests must not be nil. Responses may be nil iff an error is returned.
        GetExec(*runtimeapi.ExecRequest) (*runtimeapi.ExecResponse, error)
        GetAttach(req *runtimeapi.AttachRequest) (*runtimeapi.AttachResponse, error)
        GetPortForward(*runtimeapi.PortForwardRequest) (*runtimeapi.PortForwardResponse, error)

        // Start the server.
        // addr is the address to serve on (address:port) stayUp indicates whether the server should
        // listen until Stop() is called, or automatically stop after all expected connections are
        // closed. Calling Get{Exec,Attach,PortForward} increments the expected connection count.
        // Function does not return until the server is stopped.
        Start(stayUp bool) error
        // Stop the server, and terminate any open connections.
        Stop() error
}

(pkg/kubelet/server/streaming/server.go)

Kubelet вычисляСт ΠΎΡ‚Π²Π΅Ρ‚Π½Ρ‹ΠΉ endpoint для exec-запросов:

func (s *server) GetExec(req *runtimeapi.ExecRequest) (*runtimeapi.ExecResponse, error) {
        if err := validateExecRequest(req); err != nil {
                return nil, err
        }
        token, err := s.cache.Insert(req)
        if err != nil {
                return nil, err
        }
        return &runtimeapi.ExecResponse{
                Url: s.buildURL("exec", token),
        }, nil
}

(pkg/kubelet/server/streaming/server.go)

НС ΠΏΠ΅Ρ€Π΅ΠΏΡƒΡ‚Π°ΠΉΡ‚Π΅. Он Π²ΠΎΠ·Π²Ρ€Π°Ρ‰Π°Π΅Ρ‚ Π½Π΅ Ρ€Π΅Π·ΡƒΠ»ΡŒΡ‚Π°Ρ‚ ΠΊΠΎΠΌΠ°Π½Π΄Ρ‹, Π° endpoint для связи:

type ExecResponse struct {
        // Fully qualified URL of the exec streaming server.
        Url                  string   `protobuf:"bytes,1,opt,name=url,proto3" json:"url,omitempty"`
        XXX_NoUnkeyedLiteral struct{} `json:"-"`
        XXX_sizecache        int32    `json:"-"`
}

(cri-api/pkg/apis/runtime/v1alpha2/api.pb.go)

Kubelet Ρ€Π΅Π°Π»ΠΈΠ·ΡƒΠ΅Ρ‚ интСрфСйс RuntimeServiceClient, ΡΠ²Π»ΡΡŽΡ‰ΠΈΠΉΡΡ Ρ‡Π°ΡΡ‚ΡŒΡŽ Container Runtime Interface (ΠΏΠΎΠ΄Ρ€ΠΎΠ±Π½Π΅Π΅ ΠΎ Π½Ρ‘ΠΌ ΠΌΡ‹ писали, Π½Π°ΠΏΡ€ΠΈΠΌΠ΅Ρ€, здСсь β€” ΠΏΡ€ΠΈΠΌ. ΠΏΠ΅Ρ€Π΅Π².):

Π”Π»ΠΈΠ½Π½Ρ‹ΠΉ листинг ΠΈΠ· cri-api Π² kubernetes/kubernetes

// For semantics around ctx use and closing/ending streaming RPCs, please refer to https://godoc.org/google.golang.org/grpc#ClientConn.NewStream.
type RuntimeServiceClient interface {
        // Version returns the runtime name, runtime version, and runtime API version.
        Version(ctx context.Context, in *VersionRequest, opts ...grpc.CallOption) (*VersionResponse, error)
        // RunPodSandbox creates and starts a pod-level sandbox. Runtimes must ensure
        // the sandbox is in the ready state on success.
        RunPodSandbox(ctx context.Context, in *RunPodSandboxRequest, opts ...grpc.CallOption) (*RunPodSandboxResponse, error)
        // StopPodSandbox stops any running process that is part of the sandbox and
        // reclaims network resources (e.g., IP addresses) allocated to the sandbox.
        // If there are any running containers in the sandbox, they must be forcibly
        // terminated.
        // This call is idempotent, and must not return an error if all relevant
        // resources have already been reclaimed. kubelet will call StopPodSandbox
        // at least once before calling RemovePodSandbox. It will also attempt to
        // reclaim resources eagerly, as soon as a sandbox is not needed. Hence,
        // multiple StopPodSandbox calls are expected.
        StopPodSandbox(ctx context.Context, in *StopPodSandboxRequest, opts ...grpc.CallOption) (*StopPodSandboxResponse, error)
        // RemovePodSandbox removes the sandbox. If there are any running containers
        // in the sandbox, they must be forcibly terminated and removed.
        // This call is idempotent, and must not return an error if the sandbox has
        // already been removed.
        RemovePodSandbox(ctx context.Context, in *RemovePodSandboxRequest, opts ...grpc.CallOption) (*RemovePodSandboxResponse, error)
        // PodSandboxStatus returns the status of the PodSandbox. If the PodSandbox is not
        // present, returns an error.
        PodSandboxStatus(ctx context.Context, in *PodSandboxStatusRequest, opts ...grpc.CallOption) (*PodSandboxStatusResponse, error)
        // ListPodSandbox returns a list of PodSandboxes.
        ListPodSandbox(ctx context.Context, in *ListPodSandboxRequest, opts ...grpc.CallOption) (*ListPodSandboxResponse, error)
        // CreateContainer creates a new container in specified PodSandbox
        CreateContainer(ctx context.Context, in *CreateContainerRequest, opts ...grpc.CallOption) (*CreateContainerResponse, error)
        // StartContainer starts the container.
        StartContainer(ctx context.Context, in *StartContainerRequest, opts ...grpc.CallOption) (*StartContainerResponse, error)
        // StopContainer stops a running container with a grace period (i.e., timeout).
        // This call is idempotent, and must not return an error if the container has
        // already been stopped.
        // TODO: what must the runtime do after the grace period is reached?
        StopContainer(ctx context.Context, in *StopContainerRequest, opts ...grpc.CallOption) (*StopContainerResponse, error)
        // RemoveContainer removes the container. If the container is running, the
        // container must be forcibly removed.
        // This call is idempotent, and must not return an error if the container has
        // already been removed.
        RemoveContainer(ctx context.Context, in *RemoveContainerRequest, opts ...grpc.CallOption) (*RemoveContainerResponse, error)
        // ListContainers lists all containers by filters.
        ListContainers(ctx context.Context, in *ListContainersRequest, opts ...grpc.CallOption) (*ListContainersResponse, error)
        // ContainerStatus returns status of the container. If the container is not
        // present, returns an error.
        ContainerStatus(ctx context.Context, in *ContainerStatusRequest, opts ...grpc.CallOption) (*ContainerStatusResponse, error)
        // UpdateContainerResources updates ContainerConfig of the container.
        UpdateContainerResources(ctx context.Context, in *UpdateContainerResourcesRequest, opts ...grpc.CallOption) (*UpdateContainerResourcesResponse, error)
        // ReopenContainerLog asks runtime to reopen the stdout/stderr log file
        // for the container. This is often called after the log file has been
        // rotated. If the container is not running, container runtime can choose
        // to either create a new log file and return nil, or return an error.
        // Once it returns error, new container log file MUST NOT be created.
        ReopenContainerLog(ctx context.Context, in *ReopenContainerLogRequest, opts ...grpc.CallOption) (*ReopenContainerLogResponse, error)
        // ExecSync runs a command in a container synchronously.
        ExecSync(ctx context.Context, in *ExecSyncRequest, opts ...grpc.CallOption) (*ExecSyncResponse, error)
        // Exec prepares a streaming endpoint to execute a command in the container.
        Exec(ctx context.Context, in *ExecRequest, opts ...grpc.CallOption) (*ExecResponse, error)
        // Attach prepares a streaming endpoint to attach to a running container.
        Attach(ctx context.Context, in *AttachRequest, opts ...grpc.CallOption) (*AttachResponse, error)
        // PortForward prepares a streaming endpoint to forward ports from a PodSandbox.
        PortForward(ctx context.Context, in *PortForwardRequest, opts ...grpc.CallOption) (*PortForwardResponse, error)
        // ContainerStats returns stats of the container. If the container does not
        // exist, the call returns an error.
        ContainerStats(ctx context.Context, in *ContainerStatsRequest, opts ...grpc.CallOption) (*ContainerStatsResponse, error)
        // ListContainerStats returns stats of all running containers.
        ListContainerStats(ctx context.Context, in *ListContainerStatsRequest, opts ...grpc.CallOption) (*ListContainerStatsResponse, error)
        // UpdateRuntimeConfig updates the runtime configuration based on the given request.
        UpdateRuntimeConfig(ctx context.Context, in *UpdateRuntimeConfigRequest, opts ...grpc.CallOption) (*UpdateRuntimeConfigResponse, error)
        // Status returns the status of the runtime.
        Status(ctx context.Context, in *StatusRequest, opts ...grpc.CallOption) (*StatusResponse, error)
}

(cri-api/pkg/apis/runtime/v1alpha2/api.pb.go)
Он просто ΠΈΡΠΏΠΎΠ»ΡŒΠ·ΡƒΠ΅Ρ‚ gRPC для Π²Ρ‹Π·ΠΎΠ²Π° ΠΌΠ΅Ρ‚ΠΎΠ΄Π° Ρ‡Π΅Ρ€Π΅Π· Container Runtime Interface:

type runtimeServiceClient struct {
        cc *grpc.ClientConn
}

(cri-api/pkg/apis/runtime/v1alpha2/api.pb.go)

func (c *runtimeServiceClient) Exec(ctx context.Context, in *ExecRequest, opts ...grpc.CallOption) (*ExecResponse, error) {
        out := new(ExecResponse)
        err := c.cc.Invoke(ctx, "/runtime.v1alpha2.RuntimeService/Exec", in, out, opts...)
        if err != nil {
                return nil, err
        }
        return out, nil
}

(cri-api/pkg/apis/runtime/v1alpha2/api.pb.go)

Container Runtime ΠΎΡ‚Π²Π΅Ρ‡Π°Π΅Ρ‚ Π·Π° Ρ€Π΅Π°Π»ΠΈΠ·Π°Ρ†ΠΈΡŽ RuntimeServiceServer:

Π”Π»ΠΈΠ½Π½Ρ‹ΠΉ листинг ΠΈΠ· cri-api Π² kubernetes/kubernetes

// RuntimeServiceServer is the server API for RuntimeService service.
type RuntimeServiceServer interface {
        // Version returns the runtime name, runtime version, and runtime API version.
        Version(context.Context, *VersionRequest) (*VersionResponse, error)
        // RunPodSandbox creates and starts a pod-level sandbox. Runtimes must ensure
        // the sandbox is in the ready state on success.
        RunPodSandbox(context.Context, *RunPodSandboxRequest) (*RunPodSandboxResponse, error)
        // StopPodSandbox stops any running process that is part of the sandbox and
        // reclaims network resources (e.g., IP addresses) allocated to the sandbox.
        // If there are any running containers in the sandbox, they must be forcibly
        // terminated.
        // This call is idempotent, and must not return an error if all relevant
        // resources have already been reclaimed. kubelet will call StopPodSandbox
        // at least once before calling RemovePodSandbox. It will also attempt to
        // reclaim resources eagerly, as soon as a sandbox is not needed. Hence,
        // multiple StopPodSandbox calls are expected.
        StopPodSandbox(context.Context, *StopPodSandboxRequest) (*StopPodSandboxResponse, error)
        // RemovePodSandbox removes the sandbox. If there are any running containers
        // in the sandbox, they must be forcibly terminated and removed.
        // This call is idempotent, and must not return an error if the sandbox has
        // already been removed.
        RemovePodSandbox(context.Context, *RemovePodSandboxRequest) (*RemovePodSandboxResponse, error)
        // PodSandboxStatus returns the status of the PodSandbox. If the PodSandbox is not
        // present, returns an error.
        PodSandboxStatus(context.Context, *PodSandboxStatusRequest) (*PodSandboxStatusResponse, error)
        // ListPodSandbox returns a list of PodSandboxes.
        ListPodSandbox(context.Context, *ListPodSandboxRequest) (*ListPodSandboxResponse, error)
        // CreateContainer creates a new container in specified PodSandbox
        CreateContainer(context.Context, *CreateContainerRequest) (*CreateContainerResponse, error)
        // StartContainer starts the container.
        StartContainer(context.Context, *StartContainerRequest) (*StartContainerResponse, error)
        // StopContainer stops a running container with a grace period (i.e., timeout).
        // This call is idempotent, and must not return an error if the container has
        // already been stopped.
        // TODO: what must the runtime do after the grace period is reached?
        StopContainer(context.Context, *StopContainerRequest) (*StopContainerResponse, error)
        // RemoveContainer removes the container. If the container is running, the
        // container must be forcibly removed.
        // This call is idempotent, and must not return an error if the container has
        // already been removed.
        RemoveContainer(context.Context, *RemoveContainerRequest) (*RemoveContainerResponse, error)
        // ListContainers lists all containers by filters.
        ListContainers(context.Context, *ListContainersRequest) (*ListContainersResponse, error)
        // ContainerStatus returns status of the container. If the container is not
        // present, returns an error.
        ContainerStatus(context.Context, *ContainerStatusRequest) (*ContainerStatusResponse, error)
        // UpdateContainerResources updates ContainerConfig of the container.
        UpdateContainerResources(context.Context, *UpdateContainerResourcesRequest) (*UpdateContainerResourcesResponse, error)
        // ReopenContainerLog asks runtime to reopen the stdout/stderr log file
        // for the container. This is often called after the log file has been
        // rotated. If the container is not running, container runtime can choose
        // to either create a new log file and return nil, or return an error.
        // Once it returns error, new container log file MUST NOT be created.
        ReopenContainerLog(context.Context, *ReopenContainerLogRequest) (*ReopenContainerLogResponse, error)
        // ExecSync runs a command in a container synchronously.
        ExecSync(context.Context, *ExecSyncRequest) (*ExecSyncResponse, error)
        // Exec prepares a streaming endpoint to execute a command in the container.
        Exec(context.Context, *ExecRequest) (*ExecResponse, error)
        // Attach prepares a streaming endpoint to attach to a running container.
        Attach(context.Context, *AttachRequest) (*AttachResponse, error)
        // PortForward prepares a streaming endpoint to forward ports from a PodSandbox.
        PortForward(context.Context, *PortForwardRequest) (*PortForwardResponse, error)
        // ContainerStats returns stats of the container. If the container does not
        // exist, the call returns an error.
        ContainerStats(context.Context, *ContainerStatsRequest) (*ContainerStatsResponse, error)
        // ListContainerStats returns stats of all running containers.
        ListContainerStats(context.Context, *ListContainerStatsRequest) (*ListContainerStatsResponse, error)
        // UpdateRuntimeConfig updates the runtime configuration based on the given request.
        UpdateRuntimeConfig(context.Context, *UpdateRuntimeConfigRequest) (*UpdateRuntimeConfigResponse, error)
        // Status returns the status of the runtime.
        Status(context.Context, *StatusRequest) (*StatusResponse, error)
}

(cri-api/pkg/apis/runtime/v1alpha2/api.pb.go)
Как Ρ€Π°Π±ΠΎΡ‚Π°Π΅Ρ‚ kubectl exec?

Если это Ρ‚Π°ΠΊ, ΠΌΡ‹ Π΄ΠΎΠ»ΠΆΠ½Ρ‹ Π²ΠΈΠ΄Π΅Ρ‚ΡŒ соСдинСниС ΠΌΠ΅ΠΆΠ΄Ρƒ kubelet’ΠΎΠΌ ΠΈ исполняСмой срСдой ΠΊΠΎΠ½Ρ‚Π΅ΠΉΠ½Π΅Ρ€Π°, ΠΏΡ€Π°Π²ΠΈΠ»ΡŒΠ½ΠΎ? Π”Π°Π²Π°ΠΉΡ‚Π΅ ΠΏΡ€ΠΎΠ²Π΅Ρ€ΠΈΠΌ.

Π’Ρ‹ΠΏΠΎΠ»Π½ΠΈΡ‚Π΅ эту ΠΊΠΎΠΌΠ°Π½Π΄Ρƒ Π΄ΠΎ ΠΈ послС exec-ΠΊΠΎΠΌΠ°Π½Π΄Ρ‹ ΠΈ посмотритС Π½Π° отличия. Π’ ΠΌΠΎΠ΅ΠΌ случаС Ρ€Π°Π·Π½ΠΈΡ†Π° Ρ‚Π°ΠΊΠΎΠ²Π°:

// worker node
$ ss -a -p |grep kubelet
...
u_str  ESTAB      0      0       * 157937                * 157387                users:(("kubelet",pid=5714,fd=33))
...

Π₯ΠΌ-ΠΌ-м… НовоС соСдинСниС Ρ‡Π΅Ρ€Π΅Π· unix-сокСты ΠΌΠ΅ΠΆΠ΄Ρƒ kubelet’ΠΎΠΌ (pid=5714) ΠΈ Ρ‡Π΅ΠΌ-Ρ‚ΠΎ нСизвСстным. Π§Ρ‚ΠΎ ΠΆΠ΅ это ΠΌΠΎΠΆΠ΅Ρ‚ Π±Ρ‹Ρ‚ΡŒ? ΠŸΡ€Π°Π²ΠΈΠ»ΡŒΠ½ΠΎ, это Docker (pid=1186)!

// worker node
$ ss -a -p |grep 157387
...
u_str  ESTAB      0      0       * 157937                * 157387                users:(("kubelet",pid=5714,fd=33))
u_str  ESTAB      0      0      /var/run/docker.sock 157387                * 157937                users:(("dockerd",pid=1186,fd=14))
...

Как Π²Ρ‹ ΠΏΠΎΠΌΠ½ΠΈΡ‚Π΅, это процСсс docker-Π΄Π΅ΠΌΠΎΠ½Π° (pid=1186), ΠΊΠΎΡ‚ΠΎΡ€Ρ‹ΠΉ выполняСт Π½Π°ΡˆΡƒ ΠΊΠΎΠΌΠ°Π½Π΄Ρƒ:

// worker node
$ ps -afx
...
 1186 ?        Ssl    0:55 /usr/bin/dockerd -H fd://
17784 ?        Sl     0:00      _ docker-containerd-shim 53a0a08547b2f95986402d7f3b3e78702516244df049ba6c5aa012e81264aa3c /var/run/docker/libcontainerd/53a0a08547b2f95986402d7f3
17801 pts/2    Ss     0:00          _ sh
17827 pts/2    S+     0:00              _ sleep 5000
...

4. ΠΠΊΡ‚ΠΈΠ²Π½ΠΎΡΡ‚ΡŒ Π² исполняСмой срСдС ΠΊΠΎΠ½Ρ‚Π΅ΠΉΠ½Π΅Ρ€Π°

Π”Π°Π²Π°ΠΉΡ‚Π΅ ΠΈΠ·ΡƒΡ‡ΠΈΠΌ исходный ΠΊΠΎΠ΄ CRI-O, Ρ‡Ρ‚ΠΎΠ±Ρ‹ ΠΏΠΎΠ½ΡΡ‚ΡŒ, Ρ‡Ρ‚ΠΎ происходит. Π’ Docker’Π΅ Π»ΠΎΠ³ΠΈΠΊΠ° аналогичная.

Π˜ΠΌΠ΅Π΅Ρ‚ΡΡ сСрвСр, ΠΎΡ‚Π²Π΅Ρ‡Π°ΡŽΡ‰ΠΈΠΉ Π·Π° Ρ€Π΅Π°Π»ΠΈΠ·Π°Ρ†ΠΈΡŽ RuntimeServiceServer:

// Server implements the RuntimeService and ImageService
type Server struct {
        config          libconfig.Config
        seccompProfile  *seccomp.Seccomp
        stream          StreamService
        netPlugin       ocicni.CNIPlugin
        hostportManager hostport.HostPortManager

        appArmorProfile string
        hostIP          string
        bindAddress     string

        *lib.ContainerServer
        monitorsChan      chan struct{}
        defaultIDMappings *idtools.IDMappings
        systemContext     *types.SystemContext // Never nil

        updateLock sync.RWMutex

        seccompEnabled  bool
        appArmorEnabled bool
}

(cri-o/server/server.go)

// Exec prepares a streaming endpoint to execute a command in the container.
func (s *Server) Exec(ctx context.Context, req *pb.ExecRequest) (resp *pb.ExecResponse, err error) {
        const operation = "exec"
        defer func() {
                recordOperation(operation, time.Now())
                recordError(operation, err)
        }()

        resp, err = s.getExec(req)
        if err != nil {
                return nil, fmt.Errorf("unable to prepare exec endpoint: %v", err)
        }

        return resp, nil
}

(cri-o/erver/container_exec.go)

Π’ ΠΊΠΎΠ½Ρ†Π΅ Ρ†Π΅ΠΏΠΎΡ‡ΠΊΠΈ исполняСмая срСда ΠΊΠΎΠ½Ρ‚Π΅ΠΉΠ½Π΅Ρ€Π° выполняСт ΠΊΠΎΠΌΠ°Π½Π΄Ρƒ Π½Π° Ρ€Π°Π±ΠΎΡ‡Π΅ΠΌ ΡƒΠ·Π»Π΅:

// ExecContainer prepares a streaming endpoint to execute a command in the container.
func (r *runtimeOCI) ExecContainer(c *Container, cmd []string, stdin io.Reader, stdout, stderr io.WriteCloser, tty bool, resize <-chan remotecommand.TerminalSize) error {
        processFile, err := prepareProcessExec(c, cmd, tty)
        if err != nil {
                return err
        }
        defer os.RemoveAll(processFile.Name())

        args := []string{rootFlag, r.root, "exec"}
        args = append(args, "--process", processFile.Name(), c.ID())
        execCmd := exec.Command(r.path, args...)
        if v, found := os.LookupEnv("XDG_RUNTIME_DIR"); found {
                execCmd.Env = append(execCmd.Env, fmt.Sprintf("XDG_RUNTIME_DIR=%s", v))
        }
        var cmdErr, copyError error
        if tty {
                cmdErr = ttyCmd(execCmd, stdin, stdout, resize)
        } else {
                if stdin != nil {
                        // Use an os.Pipe here as it returns true *os.File objects.
                        // This way, if you run 'kubectl exec <pod> -i bash' (no tty) and type 'exit',
                        // the call below to execCmd.Run() can unblock because its Stdin is the read half
                        // of the pipe.
                        r, w, err := os.Pipe()
                        if err != nil {
                                return err
                        }
                        go func() { _, copyError = pools.Copy(w, stdin) }()

                        execCmd.Stdin = r
                }
                if stdout != nil {
                        execCmd.Stdout = stdout
                }
                if stderr != nil {
                        execCmd.Stderr = stderr
                }

                cmdErr = execCmd.Run()
        }

        if copyError != nil {
                return copyError
        }
        if exitErr, ok := cmdErr.(*exec.ExitError); ok {
                return &utilexec.ExitErrorWrapper{ExitError: exitErr}
        }
        return cmdErr
}

(cri-o/internal/oci/runtime_oci.go)

Как Ρ€Π°Π±ΠΎΡ‚Π°Π΅Ρ‚ kubectl exec?

НаконСц, ядро выполняСт ΠΊΠΎΠΌΠ°Π½Π΄Ρ‹:

Как Ρ€Π°Π±ΠΎΡ‚Π°Π΅Ρ‚ kubectl exec?

Напоминания

  • API Server Ρ‚Π°ΠΊΠΆΠ΅ ΠΌΠΎΠΆΠ΅Ρ‚ ΠΈΠ½ΠΈΡ†ΠΈΠ°Π»ΠΈΠ·ΠΈΡ€ΠΎΠ²Π°Ρ‚ΡŒ соСдинСниС с kubelet’ΠΎΠΌ.
  • Π‘Π»Π΅Π΄ΡƒΡŽΡ‰ΠΈΠ΅ соСдинСния ΡΠΎΡ…Ρ€Π°Π½ΡΡŽΡ‚ΡΡ Π΄ΠΎ окончания ΠΈΠ½Ρ‚Π΅Ρ€Π°ΠΊΡ‚ΠΈΠ²Π½ΠΎΠ³ΠΎ exec-сСанса:
    • ΠΌΠ΅ΠΆΠ΄Ρƒ kubectl ΠΈ api-server’ΠΎΠΌ;
    • ΠΌΠ΅ΠΆΠ΄Ρƒ api-server’ΠΎΠΌ ΠΈ kubectl;
    • ΠΌΠ΅ΠΆΠ΄Ρƒ kubelet’ΠΎΠΌ ΠΈ исполняСмой срСдой ΠΊΠΎΠ½Ρ‚Π΅ΠΉΠ½Π΅Ρ€Π°.
  • Kubectl ΠΈΠ»ΠΈ api-server Π½Π΅ ΠΌΠΎΠ³ΡƒΡ‚ Π½ΠΈΡ‡Π΅Π³ΠΎ Π·Π°ΠΏΡƒΡΠΊΠ°Ρ‚ΡŒ Π½Π° Ρ€Π°Π±ΠΎΡ‡ΠΈΡ… ΡƒΠ·Π»Π°Ρ…. Kubelet ΠΌΠΎΠΆΠ΅Ρ‚ Π·Π°ΠΏΡƒΡΠΊΠ°Ρ‚ΡŒ, Π½ΠΎ для этих дСйствий ΠΎΠ½ Ρ‚Π°ΠΊΠΆΠ΅ взаимодСйствуСт с исполняСмой срСдой ΠΊΠΎΠ½Ρ‚Π΅ΠΉΠ½Π΅Ρ€Π°.

РСсурсы

P.S. ΠΎΡ‚ ΠΏΠ΅Ρ€Π΅Π²ΠΎΠ΄Ρ‡ΠΈΠΊΠ°

Π§ΠΈΡ‚Π°ΠΉΡ‚Π΅ Ρ‚Π°ΠΊΠΆΠ΅ Π² нашСм Π±Π»ΠΎΠ³Π΅:

Π˜ΡΡ‚ΠΎΡ‡Π½ΠΈΠΊ: habr.com