🥇Kubernetes 1.17: Overview of Key Innovations

Yesterday, December 9, the latest release of Kubernetes — 1.17. In line with our blog's tradition, we discuss the most significant changes in the new version.

Kubernetes 1.17: Overview of Key Innovations

The information used to prepare this material is taken from the official announcement, Kubernetes enhancements tracking table, CHANGELOG-1.17 and corresponding issues, pull requests, as well as Kubernetes Enhancement Proposals (KEP). So, what's new?..

Topology-aware Routing

For a long time, the Kubernetes community has been waiting for this feature — Topology-aware service routing. If KEP has its origin in October 2018, and the official enhancement — two years ago, while regular issues (like this one) — are even older by several years…

The general idea is to provide the capability to implement ‘local’ routing for services within Kubernetes. ‘Locality’ in this case means ‘the same topological level’ (topology level), which can be:

the same node for services,
the same server rack,
the same region,
the same cloud provider,
…

Examples of using this feature include:

traffic savings in cloud installations with multiple availability zones (multi-AZ) — see the new illustration showing traffic from one region but different AZ in AWS;
lower latency in performance/better throughput;
a sharded service with local information about the node in each shard;
placing fluentd (or similar) on a single node with applications whose logs are collected;
…

This routing, which is ‘aware’ of topology, is also referred to as a type of network affinity — similar to node affinity, pod affinity/anti-affinity or the recently introduced Topology-Aware Volume Scheduling and Volume Provisioning. The current level of implementationServiceTopology in Kubernetes is in alpha version. For details on how this feature works and how you can already use it, read

this article from one of the authors. Dual Stack IPv4/IPv6 Support

Significant progress

has been recorded in another network feature: the simultaneous support of two IP stacks, which was first introduced in K8s 1.16 . In particular, the new release brought the following changes:in kube-proxy

the capability to work simultaneously in both modes (IPv4 and IPv6) is implemented; within Pod.Status.PodIPs
downward API support (simultaneously with this in Pod.Status.PodIPs a new feature has emerged support for the downward API (simultaneously with this in /etc/hosts now requires to add an IPv6 address for the host);
dual stack support in KIND (Kubernetes IN Docker) and kubeadm;
updated e2e tests.

Kubernetes 1.17: Overview of Key Innovations
Illustration the use of the dual stack IPV4/IPv6 in KIND

Progress on CSI

Declared stable support for topologies for CSI-based storage, first introduced in K8s 1.12.

The initiative for migrating volume plugins to CSI — CSI Migration — has reached beta status. This feature is crucial for transitioning existing storage plugins (in-tree) to a modern interface (CSI, out-of-tree) seamlessly for Kubernetes end users. Cluster administrators will simply need to enable CSI Migration, after which existing stateful resources and workloads will continue to "just work"… but using current CSI drivers instead of the deprecated ones in the Kubernetes core.

Currently, beta migration is ready for AWS EBS drivers (kubernetes.io/aws-ebs) and GCE PD (kubernetes.io/gce-pd). Forecasts for other storages are as follows:

Kubernetes 1.17: Overview of Key Innovations

We discussed how "traditional" storage support in K8s came to CSI in from one of the authors.. The migration of CSI to beta status is detailed in a separate publication on the project blog.

Additionally, another significant functionality in the context of CSI reached beta status (i.e., enabled by default) in the Kubernetes 1.17 release, stemming from its (alpha implementation) in K8s 1.12 — snapshot creation and restoration from them. Among the changes made in Kubernetes Volume Snapshot on the way to beta release:

splitting the CSI external-snapshotter sidecar into two controllers,
added deletion secret as an annotation to the volume snapshot content, new finalizer
to prevent deletion of the snapshot API object when remaining bindings exist. At the time of release 1.17, the feature is supported by three CSI drivers: GCE Persistent Disk CSI Driver, Portworx CSI Driver, and NetApp Trident CSI Driver. More details on its implementation and usage can be found in this publication

on the blog. Cloud Provider Labels Labels that are automatically

assigned to created nodes and volumes depending on the cloud provider used

, have been available in Kubernetes as a beta version for a long time — since the K8s 1.2 release (April 2016!). Given their widespread use for so long, the developers decided, that it was time to declare the feature stable (GA). have resolved, that it is time to declare the feature stable (GA).

Therefore, they were all renamed accordingly (by topologies):

beta.kubernetes.io/instance-type → node.kubernetes.io/instance-type
failure-domain.beta.kubernetes.io/zone → topology.kubernetes.io/zone
failure-domain.beta.kubernetes.io/region → topology.kubernetes.io/region

… but they are still available under their old names (for backward compatibility). However, all administrators are advised to switch to the current labels. The relevant documentation has been updated.

Structured output kubeadm

was first presented in alpha format structured output for the kubeadm utility. Supported formats: JSON, YAML, Go template.

The motivation for implementing this feature (according to KEP) is as follows:

Although Kubernetes can be deployed manually, the de facto (if not de jure) standard for this operation is the use of kubeadm. Popular system management tools like Terraform rely on kubeadm for deploying Kubernetes. Planned improvements in Cluster API include a composable package for bootstrapping Kubernetes with kubeadm and cloud-init.
Without structured output, even seemingly harmless changes can break Terraform, Cluster API, and other software relying on kubeadm's output.

The upcoming support includes (in the form of structured output) for the following kubeadm commands:

alpha certs
config images list
init
token create
token list
upgrade plan
version

Illustration of the JSON response for the command kubeadm init -o json:

{
  "node0": "192.168.20.51:443",
  "caCrt": "sha256:1f40ff4bd1b854fb4a5cf5d2f38267a5ce5f89e34d34b0f62bf335d74eef91a3",
  "token": {
    "id":          "5ndzuu.ngie1sxkgielfpb1",
    "ttl":         "23h",
    "expires":     "2019-05-08T18:58:07Z",
    "usages":      [
      "authentication",
      "signing"
    ],
    "description": "The default bootstrap token generated by 'kubeadm init'.",
    "extraGroups": [
      "system:bootstrappers:kubeadm:default-node-token"
    ]
  },
  "raw": "Rm9yIHRoZSBhY3R1YWwgb3V0cHV0IG9mIHRoZSAia3ViZWFkbSBpbml0IiBjb21tYW5kLCBwbGVhc2Ugc2VlIGh0dHBzOi8vZ2lzdC5naXRodWIuY29tL2FrdXR6LzdhNjg2ZGU1N2JmNDMzZjkyZjcxYjZmYjc3ZDRkOWJhI2ZpbGUta3ViZWFkbS1pbml0LW91dHB1dC1sb2c="
}

Stabilization of other innovations

Overall, the release of Kubernetes 1.17 took place under the slogan “Stability”. This was facilitated by the fact that many features in it (their total number — 14) received GA status. Among them:

Node ‘tainting’ based on specific conditions (TaintNodesByCondition), introduced in K8s 1.8;
Watch Bookmarks — a new type of event labeled to indicate that all objects up to a certain version (resourceVersion) have already been processed by the watch;
default values (defaulting) for Custom Resources;
shared between containers in the pod process namespaces;
ScheduleDaemonSetPods — scheduling pods in DaemonSet using kube-scheduler (instead of the DaemonSet controller);
dynamic limits on the number of volumes based on node type;
support for environment variables for directory names mounted as subPath;
moving Kubelet heartbeats to a specialized Lease API;
‘finalizer protection’ (Finalizer Protection) for load balancers (checking corresponding Service resources before deleting LoadBalancer resources);
kube-apiserver optimization in performance when dealing with multiple watches observing identical sets of objects — achieved by avoiding the re-serialization of the same objects for each watcher.

Other changes

The complete list of innovations in Kubernetes 1.17 is certainly not limited to those mentioned above. Here are some other ones (for a more complete list, see CHANGELOG):

the feature that 'matured' to beta version presented in the previous release RunAsUserName for Windows;
an analogous change was introduced to the EndpointSlice API (also from K8s 1.16), however, this solution for improving the performance/scalability of the Endpoint API is not activated by default;
critical pods for the functioning of the cluster can now be created not only in the namespaces kube-system (more details can be found in the documentation on Limit Priority Class consumption);
a new option for kubelet — --reserved-cpus — allows explicitly defining a list of CPUs reserved for the system;
for kubectl logs introduced a new flag --prefix, which adds the pod and source container name to each log line;
downward API support (simultaneously with this in label.Selector added RequiresExactMatch;
all containers in kube-dns now run with lower privileges;
hyperkube has been separated into its own GitHub repository and will no longer be included in Kubernetes releases;
significantly improved kube-proxy performance for non-UDP ports. Changes in dependencies: