🥇Kubernetes 1.16: overview of key innovations

Today, Wednesday, there will be the next release of Kubernetes - 1.16. According to the tradition for our blog, for the tenth time now, we are discussing the most significant changes in the new version.

The information used to prepare this material was taken from Kubernetes enhancements tracking table, CHANGELOG-1.16 and the corresponding issues, pull requests, as well as Kubernetes Enhancement Proposals (KEP). So, let's get started!..

Nodes

A truly large number of notable innovations (in alpha status) is presented on the node side of K8s clusters (Kubelet).

First of all, the so-called «ephemeral containers» (Ephemeral Containers), designed to simplify debugging processes in pods.The new mechanism allows you to launch special containers that start in the namespace of existing pods and live for a short time. Their purpose is to interact with other pods and containers to solve problems and debug. For this capability, a new command has been implemented, kubectl debug, similar in essence to kubectl exec: but instead of launching a process in the container (as with exec), it launches a container in the pod. For example, such a command will connect a new container to the pod:

kubectl debug -c debug-shell --image=debian target-pod -- bash

Details about ephemeral containers (and examples of their usage) can be found in the corresponding KEP. The current implementation (in K8s 1.16) is in alpha version, and among the criteria for its transition to beta is "testing the Ephemeral Containers API for at least 2 releases [Kubernetes]."

NB: In essence and even in name, the feature resembles the already existing plugin kubectl-debug, which we have already written about. It is assumed that with the emergence of ephemeral containers, the development of a separate external plugin will cease.

Another innovation is PodOverhead , designed to provide a mechanism for calculating overhead for pods, which can vary significantly depending on the runtime environment used. As an example, the authors of this KEP cite Kata Containers, which require the launch of a guest kernel, kata agent, init systems, etc. When the overhead becomes so significant, it cannot be ignored, and thus - a way to account for it is needed for further quota management, scheduling, etc. For its implementation in PodSpec , a field has been added Overhead *ResourceList (corresponding to the data in RuntimeClass, if used).

Another notable innovation is the Node Topology Manager. (Node Topology Manager), aimed at standardizing the approach to fine-tuning resource distribution across various components in Kubernetes. This initiative is driven by the increasing demand from diverse modern systems (in telecommunications, machine learning, financial services, etc.) for high-performance parallel computing and minimizing delays during operation executions, leveraging advanced CPU capabilities and hardware acceleration. Such optimizations in Kubernetes have been achieved through disparate components (CPU manager, Device manager, CNI), and now a unified internal interface will be introduced to standardize the approach and simplify the integration of new similar topology-aware components on the Kubelet side. More details can be found in corresponding KEP.

Topology Manager Component Diagram

The next feature is container checks during their startup (startup probe). As is known, it's often challenging to obtain the current status for containers that take a long time to start: they are either 'killed' before truly starting to function or they end up in a deadlock for a long time. The new check (enabled via a feature gate called StartupProbeEnabled) defers — rather than cancels — the execution of any other checks until the pod has completed its startup. For this reason, the feature was initially dubbed pod-startup liveness-probe holdoff. For pods that take a long time to start, health checks can be conducted at relatively short intervals.

Additionally, an improvement for RuntimeClass is introduced directly in beta status, adding support for 'heterogeneous clusters'. With RuntimeClass Scheduling , it is no longer necessary for each node to support every RuntimeClass: for pods, one can select a RuntimeClass without considering the cluster's topology. Previously, to ensure that pods landed on nodes with support for all their required needs, corresponding rules had to be assigned to NodeSelector and tolerations. In KEP discusses use cases and, of course, implementation details.

Network

Two significant networking features that first appeared (in alpha) in Kubernetes 1.16 are:

Support dual-stack networking — IPv4/IPv6 — and its corresponding "understanding" at the level of pods, nodes, and services. It includes IPv4-to-IPv4 and IPv6-to-IPv6 interactions among pods, from pods to external services, reference implementations (within the Bridge CNI, PTP CNI, and Host-Local IPAM plugins), as well as backward compatibility with Kubernetes clusters operating solely on IPv4 or IPv6. Implementation details are in KEP.
An example output of IP addresses of two types (IPv4 and IPv6) in the list of pods:
```
kube-master# kubectl get pods -o wide
NAME               READY     STATUS    RESTARTS   AGE       IP                          NODE
nginx-controller   1/1       Running   0          20m       fd00:db8:1::2,192.168.1.3   kube-minion-1
kube-master#
```
New API for Endpoint — EndpointSlice API. It addresses the performance/scalability issues of the existing Endpoint API affecting various components in the control plane (apiserver, etcd, endpoints-controller, kube-proxy). The new API will be added to the Discovery API group and will support tens of thousands of backend endpoints on each service in a cluster consisting of thousands of nodes. For this purpose, each Service maps to N objects EndpointSlice, each of which by default has no more than 100 endpoints (this value is configurable). The EndpointSlice API will also provide capabilities for its future development: supporting multiple IP addresses for each pod, new states for endpoints (not only Ready and NotReady), dynamic subsetting for endpoints.

The introduced finalizer, named service.kubernetes.io/load-balancer-cleanup , attached to each service of type LoadBalancer. Upon the deletion of such a service, it prevents the actual deletion of the resource until the "cleanup" of all relevant load balancer resources is completed.

API Machinery

A true "milestone stabilization" has been recorded in the area of the Kubernetes API server and its interaction. Much of this has happened thanks to the transition to stable status of not requiring special handling CustomResourceDefinitions (CRD), which had beta status since the distant Kubernetes 1.7 (that is, June 2017!). The same stabilization came to related features:

"subresources" for CustomResources; with /status and /scale versions for CRD, based on an external webhook;
conversion recently introduced (in K8s 1.15) defaulting values
and automatic field removal applying OpenAPI v3 schema for creating and publishing OpenAPI documentation, used for server-side validation of CRD resources. (defaulting) and automatic field removal (pruning) versions for CRD, based on an external webhook;
of using the OpenAPI v3 schema to create and publish OpenAPI documentation for validating CRD resources on the server side.

Another mechanism that has long become familiar for Kubernetes administrators: admission webhook — had also long been in beta status (since K8s 1.9) and is now declared stable.

Two other features have reached beta status: server-side apply and watch bookmarks.

The only significant innovation in the alpha version became the abandonment from SelfLink — a special URI representing the specified object and part of ObjectMeta and ListMeta (i.e., part of any object in Kubernetes). Why is it being abandoned? The motivation "simply put" sounds like the absence of any real (insurmountable) reasons for this field to still exist. More formal reasons include optimizing performance (by removing an unnecessary field) and simplifying the operation of the generic-apiserver, which has to handle such a field in a special way (this is the only field set just before object serialization). The actual "deprecation" (within the beta version) SelfLink will happen by Kubernetes version 1.20, and final deprecation will be in 1.21.

Data Storage

The main work in the storage area, as in previous releases, is observed in the area of CSI support. The main changes here include:

for the first time (in the alpha version) a new feature has emerged support for CSI plugins for Windows worker nodes: the current way to work with storage here will replace in-tree plugins in the Kubernetes core and FlexVolume plugins from Microsoft based on Powershell;

The implementation scheme for CSI plugins in Kubernetes for Windows
of CSI volume resizing, introduced back in K8s 1.12, has matured to beta status;
an equivalent "upgrade" (from alpha to beta version) was achieved with the use of CSI for creating local ephemeral volumes (CSI Inline Volume Support).

The volume cloning feature that appeared in the previous version of Kubernetes (using existing PVCs as DataSource for creating new PVCs) has also now reached beta status. Two notable changes in scheduling (both in alpha version):

Scheduler

EvenPodsSpreading

— the ability to use pods instead of logical application units (like Deployment and ReplicaSet) for "fair distribution" of workloads and adjustment of this distribution (as a hard requirement or as a soft condition, i.e., a priority). This feature will expand the existing capabilities for scheduling pods, currently limited by the options PodAffinity PodAntiAffinity PodAffinity and PodAntiAffinity, providing administrators with more nuanced control over this matter, which translates to better high availability and optimized resource consumption. Details can be found in KEP.
Using BestFit Policy downward API support (simultaneously with this in RequestedToCapacityRatio Priority Function when planning pods, allowing applying bin packing ("container packing") for both primary resources (CPU, memory) and extended resources (like GPU). For more information, see KEP.

Pod scheduling: before using the best fit policy (directly via the default scheduler) and with its use (through the scheduler extender)

Additionally, it has been presented the ability to create custom plugins for the scheduler outside the main Kubernetes development tree (out-of-tree).

Other changes

Also notable in the Kubernetes 1.16 release is the initiative for bringing existing metrics into full compliance, more specifically — in alignment with official guidelines for K8s instrumentation. They largely rely on the relevant Prometheus documentation. Inconsistencies arose for various reasons (for instance, some metrics were simply created before the current instructions were available), and developers concluded that it was time to standardize everything, "in line with the rest of the Prometheus ecosystem." The current implementation of this initiative is in alpha status, which will progressively increase in subsequent Kubernetes versions to beta (1.17) and stable (1.18).

Additionally, the following changes can be noted:

Enhancement of Windows support with for the introduction of of the Kubeadm utility for this OS (alpha version), with the ability RunAsUserName for Windows containers (alpha version), improving Group Managed Service Account (gMSA) support to beta version, support for mount/attach for vSphere volumes.
Redesigned data compression mechanism in API responses. Previously, an HTTP filter was used for this purpose, which imposed a number of restrictions that prevented it from being enabled by default. Now, "transparent request compression" works: clients sending Accept-Encoding: gzip in the header receive a GZIP-compressed response if its size exceeds 128 KB. Clients in Go automatically support compression (send the necessary header), so they will immediately notice a reduction in traffic. (Other languages may require slight modifications.)
It is now possible to HPA scaling from/to zero pods based on external metricsIf scaling is based on objects/external metrics, you can automatically scale down to 0 replicas when workloads are idle to save resources. This feature should be particularly useful in cases where workers request GPU resources, and the number of different types of idle workers exceeds the number of available GPUs.
New client — k8s.io/client-go/metadata.Client — for 'generic' access to objects. It is designed to easily retrieve metadata (i.e., subdivisions metadata) from cluster resources and perform operations such as garbage collection and quota enforcement.
Gathering Kubernetes can now be without deprecated ('in-tree') cloud providers (alpha version).
In the kubeadm utility added an experimental (alpha version) feature to apply kustomize patches during init, join and upgrade. For more information on using the flag --experimental-kustomize, see KEP.
A new endpoint for the apiserver — readyz, — allows exporting information about its readiness. The API server also has a flag --maximum-startup-sequence-duration, allowing you to control its restarts.
Two features for Azure have been declared stable: support for availability zones (Availability Zones) and cross resource group (RG). In addition, Azure has added:
- support for authentication AAD and ADFS;
- annotation service.beta.kubernetes.io/azure-pip-name to specify the public IP of the load balancer;
- of settings LoadBalancerName and LoadBalancerResourceGroup.
AWS has introduced disabling the tracker blocking for EBS on Windows and optimized API calls for EC2 DescribeInstances.
Kubeadm now automatically migrates CoreDNS configuration when upgrading CoreDNS version.
Binaries etcd in the corresponding Docker image were made are world-executable, allowing this image to be run without root privileges. Additionally, the migration image for etcd ceased supports etcd2 version.
In Cluster Autoscaler 1.16.0 switched to using distroless as the base image, improved performance, and added new cloud providers (DigitalOcean, Magnum, Packet).
Updates in the used/dependent software: Go 1.12.9, etcd 3.3.15, CoreDNS 1.6.2.

Kubernetes 1.15: an overview of major innovations