Tonight another release of Kubernetes — . As is tradition for our blog, we will discuss the key changes in this new version of this wonderful Open Source product.
The information used to prepare this material was taken from , and the relevant issues, pull requests, Kubernetes Enhancement Proposals (KEP).
Let's start with an important introduction from SIG cluster-lifecycle: dynamic fault-tolerant clusters Kubernetes (or more precisely, self-hosted HA deployments) can now using familiar (in the context of single-node clusters) commands kubeadm (init and join). In short, to do this:
- the certificates used by the cluster are transferred to secrets;
- to enable the use of etcd within the K8s cluster (i.e., eliminating the external dependency that has existed so far), ;
- documents recommended settings for an external load balancer, ensuring a fault-tolerant configuration (in the future, there are plans to eliminate this dependency as well, but not at this stage).
Architecture of the HA cluster Kubernetes, created with kubeadm
Details about the implementation can be found in . This feature was truly long-awaited: the alpha version was expected back in K8s 1.9 but has only now appeared.
API
The command Apply And overall declarative object management from kubectl to the apiserver. The developers briefly explain their decision by stating that kubectl apply — it’s a fundamental part of working with configurations in Kubernetes, but it's "full of bugs and difficult to fix," hence this functionality needs to be brought to a normal state and moved to the control plane. Simple and clear examples of existing problems today:
Details on the implementation — in . The current readiness — alpha version (promotion to beta is scheduled for the next Kubernetes release).
In the alpha version, it has become available to use the OpenAPI v3 scheme for creating and publishing OpenAPI documentation for CustomResources (CR), used for validating (on the server side) user-defined K8s resources (CustomResourceDefinition, CRD). Publishing OpenAPI for CRD allows clients (for example, kubectl) to perform validation on their side (within kubectl create and kubectl apply) and provide schema documentation (kubectl explain). Details — in .
The previously existing logs with the flag O_APPEND (not O_TRUNC) to avoid log loss in certain situations and for convenience when truncating logs with external utilities for rotation.
In the context of the Kubernetes API, it is also worth noting that PodSandbox and PodSandboxStatus field runtime_handler to account for information about RuntimeClass in the pod (read more about it in the section on , where this class was introduced as an alpha version), and in Admission Webhooks the ability to determine which versions AdmissionReview they support. Finally, Admission Webhook rules now the scope of their application by namespaces and cluster boundaries.
Storage
, which had beta status since the release of , stable (GA): this feature gate is no longer disabled and will be removed in Kubernetes 1.17.
the use of environment variables from the so-called for directory names mounted as , has developed — in the form of a new field subPathExpr, which now defines the desired directory name. Initially, the feature appeared in Kubernetes 1.11, but remained in alpha status for 1.14.
As with the previous Kubernetes release, many significant changes have been presented for the actively developing CSI (Container Storage Interface):
CSI
has become available (in alpha status) for resizing CSI volumes.To use it, you will need to enable the feature gate called ExpandCSIVolumes, as well as support for this operation in the specific CSI driver.
Another alpha feature for CSI is the direct referencing (i.e., without using PV/PVC) of CSI volumes within pod specifications. This removes the limitation of using CSI solely as remote data storage, opening the door for them to the world of To use it (), it is necessary to enable the CSIInlineVolume feature gate.
Progress has also been observed in the 'internals' of Kubernetes related to CSI, which are not immediately noticeable to end users (system administrators)… At present, developers have to maintain two versions of each storage plugin: one — 'the old way', within the K8s codebase (in-tree), and the second — within the new CSI (read more about it, for example, in )This creates understandable inconveniences that must be addressed as CSI stabilizes in its own right. Simply declaring the internal (in-tree) plugin APIs as deprecated is not feasible due to .
All of this has led to the alpha versions reaching of the internal plugin code, implemented as in-tree, into CSI plugins, which will reduce developers' concerns to supporting a single version of their plugins, while maintaining compatibility with older APIs, allowing them to be declared deprecated by the usual scenario. It is expected that by the next release of Kubernetes (1.15), all cloud provider plugins will be migrated, the implementation will attain beta status, and will be activated in K8s installations by default. For details, see . The consequence of this migration was also of volume constraints defined by specific cloud providers (AWS, Azure, GCE, Cinder).
Additionally, support for block devices with CSI (CSIBlockVolume) to beta status.
Nodes / Kubelet
An alpha version in Kubelet, intended for delivering metrics on core resources. Generally speaking, while Kubelet used to receive container usage statistics from cAdvisor, now this data comes from the container runtime via CRI (Container Runtime Interface), but compatibility with older Docker versions is maintained. Previously collected statistics in Kubelet were provided through REST API, and now an endpoint located at /metrics/resource/v1alpha1is used for this purpose. The long-term strategy of the developers is to minimize the set of metrics provided by Kubelet. By the way, these metrics not 'core metrics', but 'resource metrics', describing them as 'first-class resources, such as CPU and memory'.
A rather interesting nuance: despite the clear performance advantage of the gRPC endpoint compared to various use cases of the Prometheus format (the results of one of the benchmarks can be seen below), the authors preferred the text format of Prometheus due to the clear leadership of this monitoring system in the community.
"gRPC is not compatible with major monitoring pipelines. The Endpoint will only be useful for delivering metrics to the Metrics Server or monitoring components that integrate directly with it. When caching is used in the Metrics Server, the performance of the Prometheus text format is quite good for us to prefer Prometheus over gRPC, considering the widespread use of Prometheus in the community. When the OpenMetrics format becomes more stable, we can achieve gRPC-like performance with the proto-based format."
One of the comparative performance tests of using gRPC and Prometheus formats in the new Kubelet endpoint for metrics. More graphs and other details can be found in .
Among other changes:
- Kubelet now (once) containers in unknown state before restart and delete operations.
- When using now have the same information as regular containers. the same information as a regular container.
- Kubelet
usageNanoCoresfrom the CRI statistics provider, and for nodes and containers in Windows network statistics. - Information about the operating system and architecture is now recorded in labels
kubernetes.io/osandkubernetes.io/archof Node objects (moved from beta to GA). - The ability to specify a specific system user group for containers in a pod (
RunAsGroup) appeared in ) to beta version (enabled by default). - du and find, used in cAdvisor, with Go implementations.
CLI
In cli-runtime and kubectl the -k flag for integration with (by the way, its development is now carried out in a separate repository), i.e., for processing additional YAML files from specialized kustomization directories (details on their use can be found in ):
Example of simple usage of the file (possibly more complex applications of kustomize within )
Additionally:
- a new command
kubectl create cronjob, the name of which speaks for itself. - In
kubectl logscan now be Support for system-versioned tables has been expanded, which not only store the current snapshot of data but also retain information about all previously made changes.-f(--followfor streaming logs) and-l(--selectorfor label query). - kubectl copy files selected by wildcard.
- In the command
kubectl waitflag--allto select all resources in the specified resource type namespace.
Others
The following features have attained stable (GA) status:
- , used in the pod specification to define additional conditions considered in the pod's readiness;
- Support for large pages (feature gate named );
- ;
- PriorityClass API, .
Other changes introduced in Kubernetes 1.14:
- The default RBAC policy no longer grants access to the API
discoveryandaccess-reviewto users without authentication (unauthenticated). - Official CoreDNS support only for Linux, so when using kubeadm for its (CoreDNS) deployment in the cluster, nodes must run exclusively on Linux (this restriction uses nodeSelectors).
- The default CoreDNS configuration now plugin instead of proxy. Additionally, in CoreDNS a readinessProbe has been added to prevent load balancing to the appropriate (not ready for service) pods.
- In kubeadm, during the phases
initorupload-certs, to upload the certificates required for connecting a new control-plane to the kubeadm-certs secret (the flag used is--experimental-upload-certs). - For Windows installations, an alpha version has been released gMSA (Group Managed Service Account) — special accounts in Active Directory that can be used by containers.
- For GCE Updates in the used/dependent software: Go 1.12.1, CSI 1.1, CoreDNS 1.3.1, support for Docker 18.09 in kubeadm, and the minimum supported Docker API version became 1.26.
- Kubernetes 1.12: summary of key features
P.S.
Also read in our blog:
- «»;
- «»;
- «»;
- «».
Source: habr.com
