Kumbuka. tafsiri.: mwandishi wa makala - Erkan Erol, mhandisi kutoka SAP - anashiriki utafiti wake wa taratibu za utendaji wa timu kubectl exec, inajulikana sana kwa kila mtu anayefanya kazi na Kubernetes. Anaambatana na algoriti nzima na uorodheshaji wa msimbo wa chanzo wa Kubernetes (na miradi inayohusiana), ambayo hukuruhusu kuelewa mada kwa undani inavyohitajika.
Ijumaa moja, mwenzangu alinijia na kuniuliza jinsi ya kutekeleza amri kwenye ganda kwa kutumia mteja-kwenda. Sikuweza kumjibu na ghafla nikagundua kuwa sikujua chochote kuhusu utaratibu wa operesheni kubectl exec. Ndiyo, nilikuwa na mawazo fulani kuhusu muundo wake, lakini sikuwa na uhakika wa 100% wa usahihi wao na kwa hiyo niliamua kukabiliana na suala hili. Baada ya kusoma blogi, hati na msimbo wa chanzo, nilijifunza mambo mengi mapya, na katika makala hii nataka kushiriki uvumbuzi na ufahamu wangu. Ikiwa kuna kitu kibaya, tafadhali wasiliana nami kwa Twitter.
Mafunzo ya
Ili kuunda nguzo kwenye MacBook, nilitengeneza ecomm-integration-ballerina/kubernetes-cluster. Kisha nikarekebisha anwani za IP za nodi kwenye usanidi wa kubelet, kwani mipangilio ya chaguo-msingi haikuruhusu kubectl exec. Unaweza kusoma zaidi juu ya sababu kuu ya hii hapa.
Gari lolote = MacBook yangu
Mwalimu node IP = 192.168.205.10
Nodi ya mfanyakazi IP = 192.168.205.11
Bandari ya seva ya API = 6443
Vipengele
kubectl kutekeleza mchakato: Tunapotekeleza βkubectl exec...β mchakato unaanza. Hii inaweza kufanywa kwenye mashine yoyote iliyo na ufikiaji wa seva ya API ya K8s. Kumbuka transl.: Zaidi katika orodha za dashibodi, mwandishi anatumia maoni "mashine yoyote", akimaanisha kuwa amri zinazofuata zinaweza kutekelezwa kwenye mashine zozote kama hizo zenye ufikiaji wa Kubernetes.
seva ya api: Sehemu kwenye nodi kuu ambayo hutoa ufikiaji wa Kubernetes API. Hii ndio sehemu ya mbele ya ndege ya udhibiti huko Kubernetes.
mchemraba: Wakala anayeendesha kila nodi kwenye nguzo. Inahakikisha uendeshaji wa vyombo kwenye ganda.
wakati wa kukimbia kwa chombo (muda wa kukimbia wa kontena): Programu inayohusika na kuendesha vyombo. Mifano: Docker, CRI-O, iliyobeba...
punje: OS kernel kwenye nodi ya mfanyakazi; inawajibika kwa usimamizi wa mchakato.
lengo (lengo) chombo: chombo ambacho ni sehemu ya ganda na inaendeshwa kwenye nodi moja ya mfanyakazi.
Nilichogundua
1. Shughuli ya upande wa mteja
Unda ganda katika nafasi ya majina default:
// any machine
$ kubectl run exec-test-nginx --image=nginx
Kisha tunatoa amri ya kutekeleza na subiri sekunde 5000 kwa uchunguzi zaidi:
// any machine
$ kubectl exec -it exec-test-nginx-6558988d5-fgxgg -- sh
# sleep 5000
Mchakato wa kubectl unaonekana (na pid=8507 kwa upande wetu):
Tunaweza pia kutazama ombi kwa upande wa seva ya api:
handler.go:143] kube-apiserver: POST "/api/v1/namespaces/default/pods/exec-test-nginx-6558988d5-fgxgg/exec" satisfied by gorestful with webservice /api/v1
upgradeaware.go:261] Connecting to backend proxy (intercepting redirects) https://192.168.205.11:10250/exec/default/exec-test-nginx-6558988d5-fgxgg/exec-test-nginx?command=sh&input=1&output=1&tty=1
Headers: map[Connection:[Upgrade] Content-Length:[0] Upgrade:[SPDY/3.1] User-Agent:[kubectl/v1.12.10 (darwin/amd64) kubernetes/e3c1340] X-Forwarded-For:[192.168.205.1] X-Stream-Protocol-Version:[v4.channel.k8s.io v3.channel.k8s.io v2.channel.k8s.io channel.k8s.io]]
Kumbuka kuwa ombi la HTTP linajumuisha ombi la kubadilisha itifaki. SPDY hukuruhusu kuzidisha "mikondo" ya mtu binafsi ya stdin/stdout/stderr/spdy-error juu ya muunganisho mmoja wa TCP.
Seva ya API hupokea ombi na kulibadilisha kuwa PodExecOptions:
// PodExecOptions is the query options to a Pod's remote exec call
type PodExecOptions struct {
metav1.TypeMeta
// Stdin if true indicates that stdin is to be redirected for the exec call
Stdin bool
// Stdout if true indicates that stdout is to be redirected for the exec call
Stdout bool
// Stderr if true indicates that stderr is to be redirected for the exec call
Stderr bool
// TTY if true indicates that a tty will be allocated for the exec call
TTY bool
// Container in which to execute the command.
Container string
// Command is the remote command to execute; argv array; not executed within a shell.
Command []string
}
Ili kutekeleza vitendo vinavyohitajika, seva ya api lazima ijue ni ganda gani inahitaji kuwasiliana:
// ExecLocation returns the exec URL for a pod container. If opts.Container is blank
// and only one container is present in the pod, that container is used.
func ExecLocation(
getter ResourceGetter,
connInfo client.ConnectionInfoGetter,
ctx context.Context,
name string,
opts *api.PodExecOptions,
) (*url.URL, http.RoundTripper, error) {
return streamLocation(getter, connInfo, ctx, name, opts, opts.Container, "exec")
}
Kwa kweli, data kuhusu mwisho inachukuliwa kutoka kwa habari kuhusu nodi:
nodeName := types.NodeName(pod.Spec.NodeName)
if len(nodeName) == 0 {
// If pod has not been assigned a host, return an empty location
return nil, nil, errors.NewBadRequest(fmt.Sprintf("pod %s does not have a host assigned", name))
}
nodeInfo, err := connInfo.GetConnectionInfo(ctx, nodeName)
Hooray! kubelet sasa ina bandari (node.Status.DaemonEndpoints.KubeletEndpoint.Port), ambayo seva ya API inaweza kuunganishwa:
// GetConnectionInfo retrieves connection info from the status of a Node API object.
func (k *NodeConnectionInfoGetter) GetConnectionInfo(ctx context.Context, nodeName types.NodeName) (*ConnectionInfo, error) {
node, err := k.nodes.Get(ctx, string(nodeName), metav1.GetOptions{})
if err != nil {
return nil, err
}
// Find a kubelet-reported address, using preferred address type
host, err := nodeutil.GetPreferredNodeAddress(node, k.preferredAddressTypes)
if err != nil {
return nil, err
}
// Use the kubelet-reported port, if present
port := int(node.Status.DaemonEndpoints.KubeletEndpoint.Port)
if port <= 0 {
port = k.defaultPort
}
return &ConnectionInfo{
Scheme: k.scheme,
Hostname: host,
Port: strconv.Itoa(port),
Transport: k.transport,
}, nil
}
Miunganisho hii inafanywa hadi mwisho wa HTTPS ya kubelet. Kwa chaguo-msingi, apiserver haithibitishi cheti cha kubelet, ambacho hufanya muunganisho kuwa hatarini kwa mashambulizi ya mtu wa kati (MITM) na isiyo salama kwa kufanya kazi katika mitandao isiyoaminika na/au ya umma.
Sasa seva ya API inajua mwisho na inaanzisha unganisho:
// Connect returns a handler for the pod exec proxy
func (r *ExecREST) Connect(ctx context.Context, name string, opts runtime.Object, responder rest.Responder) (http.Handler, error) {
execOpts, ok := opts.(*api.PodExecOptions)
if !ok {
return nil, fmt.Errorf("invalid options object: %#v", opts)
}
location, transport, err := pod.ExecLocation(r.Store, r.KubeletConn, ctx, name, execOpts)
if err != nil {
return nil, err
}
return newThrottledUpgradeAwareProxyHandler(location, transport, false, true, true, responder), nil
}
Kwanza, tunapata IP ya nodi ya mfanyakazi. Kwa upande wetu ni 192.168.205.11:
// any machine
$ kubectl get nodes k8s-node-1 -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
k8s-node-1 Ready <none> 9h v1.15.3 192.168.205.11 <none> Ubuntu 16.04.6 LTS 4.4.0-159-generic docker://17.3.3
Kisha weka bandari ya kubelet (10250 kwa upande wetu):
// any machine
$ kubectl get nodes k8s-node-1 -o jsonpath='{.status.daemonEndpoints.kubeletEndpoint}'
map[Port:10250]
Sasa ni wakati wa kuangalia mtandao. Je, kuna uhusiano na nodi ya mfanyakazi (192.168.205.11)? Ni! Ukiua mchakato exec, itatoweka, kwa hivyo najua kuwa unganisho ulianzishwa na seva ya api kama matokeo ya amri ya utekelezaji iliyotekelezwa.
Lakini subiri: kubelet alivutaje hii? Kubelet ina daemon ambayo hutoa ufikiaji wa API kupitia bandari kwa maombi ya seva ya api:
// Server is the library interface to serve the stream requests.
type Server interface {
http.Handler
// Get the serving URL for the requests.
// Requests must not be nil. Responses may be nil iff an error is returned.
GetExec(*runtimeapi.ExecRequest) (*runtimeapi.ExecResponse, error)
GetAttach(req *runtimeapi.AttachRequest) (*runtimeapi.AttachResponse, error)
GetPortForward(*runtimeapi.PortForwardRequest) (*runtimeapi.PortForwardResponse, error)
// Start the server.
// addr is the address to serve on (address:port) stayUp indicates whether the server should
// listen until Stop() is called, or automatically stop after all expected connections are
// closed. Calling Get{Exec,Attach,PortForward} increments the expected connection count.
// Function does not return until the server is stopped.
Start(stayUp bool) error
// Stop the server, and terminate any open connections.
Stop() error
}
Kubelet hutumia kiolesura RuntimeServiceClient, ambayo ni sehemu ya Kiolesura cha Muda wa Kontena (tuliandika zaidi juu yake, kwa mfano, hapa - takriban. tafsiri.):
Orodha ndefu kutoka kwa cri-api katika kubernetes/kubernetes
// For semantics around ctx use and closing/ending streaming RPCs, please refer to https://godoc.org/google.golang.org/grpc#ClientConn.NewStream.
type RuntimeServiceClient interface {
// Version returns the runtime name, runtime version, and runtime API version.
Version(ctx context.Context, in *VersionRequest, opts ...grpc.CallOption) (*VersionResponse, error)
// RunPodSandbox creates and starts a pod-level sandbox. Runtimes must ensure
// the sandbox is in the ready state on success.
RunPodSandbox(ctx context.Context, in *RunPodSandboxRequest, opts ...grpc.CallOption) (*RunPodSandboxResponse, error)
// StopPodSandbox stops any running process that is part of the sandbox and
// reclaims network resources (e.g., IP addresses) allocated to the sandbox.
// If there are any running containers in the sandbox, they must be forcibly
// terminated.
// This call is idempotent, and must not return an error if all relevant
// resources have already been reclaimed. kubelet will call StopPodSandbox
// at least once before calling RemovePodSandbox. It will also attempt to
// reclaim resources eagerly, as soon as a sandbox is not needed. Hence,
// multiple StopPodSandbox calls are expected.
StopPodSandbox(ctx context.Context, in *StopPodSandboxRequest, opts ...grpc.CallOption) (*StopPodSandboxResponse, error)
// RemovePodSandbox removes the sandbox. If there are any running containers
// in the sandbox, they must be forcibly terminated and removed.
// This call is idempotent, and must not return an error if the sandbox has
// already been removed.
RemovePodSandbox(ctx context.Context, in *RemovePodSandboxRequest, opts ...grpc.CallOption) (*RemovePodSandboxResponse, error)
// PodSandboxStatus returns the status of the PodSandbox. If the PodSandbox is not
// present, returns an error.
PodSandboxStatus(ctx context.Context, in *PodSandboxStatusRequest, opts ...grpc.CallOption) (*PodSandboxStatusResponse, error)
// ListPodSandbox returns a list of PodSandboxes.
ListPodSandbox(ctx context.Context, in *ListPodSandboxRequest, opts ...grpc.CallOption) (*ListPodSandboxResponse, error)
// CreateContainer creates a new container in specified PodSandbox
CreateContainer(ctx context.Context, in *CreateContainerRequest, opts ...grpc.CallOption) (*CreateContainerResponse, error)
// StartContainer starts the container.
StartContainer(ctx context.Context, in *StartContainerRequest, opts ...grpc.CallOption) (*StartContainerResponse, error)
// StopContainer stops a running container with a grace period (i.e., timeout).
// This call is idempotent, and must not return an error if the container has
// already been stopped.
// TODO: what must the runtime do after the grace period is reached?
StopContainer(ctx context.Context, in *StopContainerRequest, opts ...grpc.CallOption) (*StopContainerResponse, error)
// RemoveContainer removes the container. If the container is running, the
// container must be forcibly removed.
// This call is idempotent, and must not return an error if the container has
// already been removed.
RemoveContainer(ctx context.Context, in *RemoveContainerRequest, opts ...grpc.CallOption) (*RemoveContainerResponse, error)
// ListContainers lists all containers by filters.
ListContainers(ctx context.Context, in *ListContainersRequest, opts ...grpc.CallOption) (*ListContainersResponse, error)
// ContainerStatus returns status of the container. If the container is not
// present, returns an error.
ContainerStatus(ctx context.Context, in *ContainerStatusRequest, opts ...grpc.CallOption) (*ContainerStatusResponse, error)
// UpdateContainerResources updates ContainerConfig of the container.
UpdateContainerResources(ctx context.Context, in *UpdateContainerResourcesRequest, opts ...grpc.CallOption) (*UpdateContainerResourcesResponse, error)
// ReopenContainerLog asks runtime to reopen the stdout/stderr log file
// for the container. This is often called after the log file has been
// rotated. If the container is not running, container runtime can choose
// to either create a new log file and return nil, or return an error.
// Once it returns error, new container log file MUST NOT be created.
ReopenContainerLog(ctx context.Context, in *ReopenContainerLogRequest, opts ...grpc.CallOption) (*ReopenContainerLogResponse, error)
// ExecSync runs a command in a container synchronously.
ExecSync(ctx context.Context, in *ExecSyncRequest, opts ...grpc.CallOption) (*ExecSyncResponse, error)
// Exec prepares a streaming endpoint to execute a command in the container.
Exec(ctx context.Context, in *ExecRequest, opts ...grpc.CallOption) (*ExecResponse, error)
// Attach prepares a streaming endpoint to attach to a running container.
Attach(ctx context.Context, in *AttachRequest, opts ...grpc.CallOption) (*AttachResponse, error)
// PortForward prepares a streaming endpoint to forward ports from a PodSandbox.
PortForward(ctx context.Context, in *PortForwardRequest, opts ...grpc.CallOption) (*PortForwardResponse, error)
// ContainerStats returns stats of the container. If the container does not
// exist, the call returns an error.
ContainerStats(ctx context.Context, in *ContainerStatsRequest, opts ...grpc.CallOption) (*ContainerStatsResponse, error)
// ListContainerStats returns stats of all running containers.
ListContainerStats(ctx context.Context, in *ListContainerStatsRequest, opts ...grpc.CallOption) (*ListContainerStatsResponse, error)
// UpdateRuntimeConfig updates the runtime configuration based on the given request.
UpdateRuntimeConfig(ctx context.Context, in *UpdateRuntimeConfigRequest, opts ...grpc.CallOption) (*UpdateRuntimeConfigResponse, error)
// Status returns the status of the runtime.
Status(ctx context.Context, in *StatusRequest, opts ...grpc.CallOption) (*StatusResponse, error)
}
Muda wa Kutumika kwa Kontena unawajibika kwa utekelezaji RuntimeServiceServer:
Orodha ndefu kutoka kwa cri-api katika kubernetes/kubernetes
// RuntimeServiceServer is the server API for RuntimeService service.
type RuntimeServiceServer interface {
// Version returns the runtime name, runtime version, and runtime API version.
Version(context.Context, *VersionRequest) (*VersionResponse, error)
// RunPodSandbox creates and starts a pod-level sandbox. Runtimes must ensure
// the sandbox is in the ready state on success.
RunPodSandbox(context.Context, *RunPodSandboxRequest) (*RunPodSandboxResponse, error)
// StopPodSandbox stops any running process that is part of the sandbox and
// reclaims network resources (e.g., IP addresses) allocated to the sandbox.
// If there are any running containers in the sandbox, they must be forcibly
// terminated.
// This call is idempotent, and must not return an error if all relevant
// resources have already been reclaimed. kubelet will call StopPodSandbox
// at least once before calling RemovePodSandbox. It will also attempt to
// reclaim resources eagerly, as soon as a sandbox is not needed. Hence,
// multiple StopPodSandbox calls are expected.
StopPodSandbox(context.Context, *StopPodSandboxRequest) (*StopPodSandboxResponse, error)
// RemovePodSandbox removes the sandbox. If there are any running containers
// in the sandbox, they must be forcibly terminated and removed.
// This call is idempotent, and must not return an error if the sandbox has
// already been removed.
RemovePodSandbox(context.Context, *RemovePodSandboxRequest) (*RemovePodSandboxResponse, error)
// PodSandboxStatus returns the status of the PodSandbox. If the PodSandbox is not
// present, returns an error.
PodSandboxStatus(context.Context, *PodSandboxStatusRequest) (*PodSandboxStatusResponse, error)
// ListPodSandbox returns a list of PodSandboxes.
ListPodSandbox(context.Context, *ListPodSandboxRequest) (*ListPodSandboxResponse, error)
// CreateContainer creates a new container in specified PodSandbox
CreateContainer(context.Context, *CreateContainerRequest) (*CreateContainerResponse, error)
// StartContainer starts the container.
StartContainer(context.Context, *StartContainerRequest) (*StartContainerResponse, error)
// StopContainer stops a running container with a grace period (i.e., timeout).
// This call is idempotent, and must not return an error if the container has
// already been stopped.
// TODO: what must the runtime do after the grace period is reached?
StopContainer(context.Context, *StopContainerRequest) (*StopContainerResponse, error)
// RemoveContainer removes the container. If the container is running, the
// container must be forcibly removed.
// This call is idempotent, and must not return an error if the container has
// already been removed.
RemoveContainer(context.Context, *RemoveContainerRequest) (*RemoveContainerResponse, error)
// ListContainers lists all containers by filters.
ListContainers(context.Context, *ListContainersRequest) (*ListContainersResponse, error)
// ContainerStatus returns status of the container. If the container is not
// present, returns an error.
ContainerStatus(context.Context, *ContainerStatusRequest) (*ContainerStatusResponse, error)
// UpdateContainerResources updates ContainerConfig of the container.
UpdateContainerResources(context.Context, *UpdateContainerResourcesRequest) (*UpdateContainerResourcesResponse, error)
// ReopenContainerLog asks runtime to reopen the stdout/stderr log file
// for the container. This is often called after the log file has been
// rotated. If the container is not running, container runtime can choose
// to either create a new log file and return nil, or return an error.
// Once it returns error, new container log file MUST NOT be created.
ReopenContainerLog(context.Context, *ReopenContainerLogRequest) (*ReopenContainerLogResponse, error)
// ExecSync runs a command in a container synchronously.
ExecSync(context.Context, *ExecSyncRequest) (*ExecSyncResponse, error)
// Exec prepares a streaming endpoint to execute a command in the container.
Exec(context.Context, *ExecRequest) (*ExecResponse, error)
// Attach prepares a streaming endpoint to attach to a running container.
Attach(context.Context, *AttachRequest) (*AttachResponse, error)
// PortForward prepares a streaming endpoint to forward ports from a PodSandbox.
PortForward(context.Context, *PortForwardRequest) (*PortForwardResponse, error)
// ContainerStats returns stats of the container. If the container does not
// exist, the call returns an error.
ContainerStats(context.Context, *ContainerStatsRequest) (*ContainerStatsResponse, error)
// ListContainerStats returns stats of all running containers.
ListContainerStats(context.Context, *ListContainerStatsRequest) (*ListContainerStatsResponse, error)
// UpdateRuntimeConfig updates the runtime configuration based on the given request.
UpdateRuntimeConfig(context.Context, *UpdateRuntimeConfigRequest) (*UpdateRuntimeConfigResponse, error)
// Status returns the status of the runtime.
Status(context.Context, *StatusRequest) (*StatusResponse, error)
}
Ikiwa ndivyo, tunapaswa kuona muunganisho kati ya kubelet na wakati wa kukimbia wa kontena, sivyo? Hebu tuangalie.
Tumia amri hii kabla na baada ya exec amri na uangalie tofauti. Katika kesi yangu tofauti ni:
// worker node
$ ss -a -p |grep kubelet
...
u_str ESTAB 0 0 * 157937 * 157387 users:(("kubelet",pid=5714,fd=33))
...
Hmmm... Muunganisho mpya kupitia soketi unix kati ya kubelet (pid=5714) na kitu kisichojulikana. Inaweza kuwa nini? Hiyo ni kweli, ni Docker (pid=1186)!
Seva ya API pia inaweza kuanzisha muunganisho kwenye kubelet.
Viunganisho vifuatavyo vinaendelea hadi kipindi cha utekelezaji shirikishi kitakapomalizika:
kati ya kubectl na api-server;
kati ya api-server na kubectl;
kati ya kubelet na wakati wa kukimbia kwa kontena.
Kubectl au api-server haiwezi kuendesha chochote kwenye nodi za wafanyikazi. Kubelet inaweza kukimbia, lakini pia inaingiliana na wakati wa kukimbia wa kontena kufanya vitu hivyo.