Some applications also need to store data, but they are quite relaxed about the fact that data will not be saved after a restart.
For example, caching services are limited by RAM but can also move infrequently used data to storage that operates slower than RAM, with little impact on overall performance. Other applications need to know that some input in files can be read-only, such as configurations or secret keys.
Kubernetes already has several types , but their functionality is limited to what is implemented in K8s.
Ephemeral allowed expanding Kubernetes with CSI drivers to support lightweight local volumes. In this way, it is possible to apply : configurations, secrets, identity data, variables, and so on. CSI drivers need to be modified to support this Kubernetes feature, as it is expected that standard drivers will not work—however, such volumes are expected to be usable on any node chosen for the pod.
This could become a problem for volumes with significant node resource consumption or for storage available only on certain nodes. Therefore, Kubernetes 1.19 introduces two new volume features for alpha testing, conceptually similar to EmptyDir volumes:
general-purpose ephemeral volumes;
CSI storage capacity tracking.
Advantages of the new approach include:
storage can be local or network-attached;
volumes can have a specified size that cannot be exceeded by the application;
works with any CSI drivers that support the provision of persistent volumes and (for capacity tracking support) implement the call
GetCapacity;volumes can have some initial data depending on the driver and parameters;
all standard volume operations (such as state snapshot creation, resizing, etc.) are supported;
volumes can be used with any application controller accepting a module or volume specification;
The Kubernetes scheduler selects suitable nodes by itself, so there's no need to provide and configure scheduler extensions or modify webhooks.
Use Cases
Thus, ephemeral general-purpose volumes are suitable for the following use cases:
Persistent memory as a replacement for RAM for memcached
Latest memcached releases using persistent memory (e.g., Intel Optane, translator's note) instead of conventional RAM. When deploying memcached via an application controller, ephemeral general-purpose volumes can request the allocation of a volume of a specified size from PMEM using the CSI driver, for example, .
Local LVM storage as workspace
Applications dealing with data sizes that exceed the size of RAM can request local storage with sizes or performance metrics that cannot be provided by ordinary EmptyDir volumes from Kubernetes. For this purpose, .
Read-only access for data volumes
Volume allocation can lead to the creation of a filled volume when:
restoring ;
creating ;
work .
These volumes can be mounted in read-only mode.
How it works
Ephemeral general-purpose volumes
A key feature of ephemeral general-purpose volumes is the new volume source, EphemeralVolumeSource, containing all fields for creating a volume request (historically known as a persistent volume request, PVC). A new controller in kube-controller-manager watches for pods creating such a volume source, and then creates PVCs for those pods. For the CSI driver, this request looks the same as the others, so no special support is needed here.
As long as such PVCs exist, they can be used just like any other volume requests. In particular, they can serve as a data source when cloning a volume or creating a snapshot from the volume. The PVC object also contains the current state of the volume.
The names of automatically created PVCs are predefined: they are a combination of the pod name and the volume name, separated by a hyphen. The predefined names simplify interaction with PVCs, as one does not need to search for it if the pod name and the volume name are known. The downside is that the name may already be in use, which Kubernetes detects, blocking the pod launch as a result.
To ensure that the volume is deleted along with the pod, the controller makes the pod the owner of the volume request. When the pod is deleted, the standard garbage collection mechanism runs, which removes both the request and the volume.
Requests are matched to a storage driver via the standard storage class mechanism. Although immediate and delayed binding classes are supported, it is sensible to use delayed binding for ephemeral volumes. WaitForFirstConsumer) so that the scheduler can take into account both node usage and storage availability when choosing a node. A new feature arises here. WaitForFirstConsumerStorage capacity tracking
Typically, the scheduler does not have data about where the CSI driver will create the volume. The scheduler also has no way to contact the driver directly to request this information. Therefore, the scheduler polls the nodes until it finds one where the volumes might be available (delayed binding), or completely leaves the choice of location to the driver (immediate binding).
New
CSIStorageCapacity , currently in alpha, allows storing necessary data in etcd, making it available to the scheduler. Unlike the support for ephemeral volumes of general purpose, when deploying the driver, storage capacity tracking must be enabled:external-provisioner must publish capacity information obtained from the driver via the standard If the scheduler needs to choose a node for a pod with an unbound volume using delayed binding, and the driver activated this feature during deployment by setting the flag GetCapacity.
CSIDriver.storageCapacity , then nodes with insufficient storage capacity will be automatically discarded. This works for both ephemeral general-purpose and persistent volumes, but not for ephemeral CSI volumes, as their parameters cannot be read by Kubernetes., the nodes with insufficient storage capacity will be automatically disregarded. This applies to both ephemeral general-purpose volumes and persistent volumes, but not to ephemeral CSI volumes, as their parameters cannot be read by Kubernetes.
As usual, volumes with immediate binding are created before scheduling pods, and their placement is chosen by the storage driver, so when configuring must publish capacity information obtained from the driver via the standard by default, storage classes with immediate binding are skipped since this data will not be used anyway.
Since the Kubernetes scheduler has to work with potentially outdated information, there are no guarantees that capacity will be available at any time when the volume is created, but nonetheless, the chances of it being created without retries increase.
N.B. You can get more detailed information and safely "practice on cats in a sandbox", and in case of complete confusion, you can receive qualified technical support help during the intensives — which will take place from September 28 to 30, and for more advanced specialists from October 14 to 16.
Security
, currently in alpha, allows storing necessary data in etcd, making it available to the scheduler. Unlike the support for ephemeral volumes of general purpose, when deploying the driver, storage capacity tracking must be enabled:
CSIStorageCapacity objects exist within namespaces, and when rolling out each CSI driver in its namespace, it is advisable to restrict RBAC permissions for CSIStorageCapacity in that namespace, as it is obvious where the data comes from. In any case, Kubernetes does not check this, and usually, drivers are deployed in the same namespace, so ultimately it is expected that drivers will work and not publish incorrect data (and here I got carried away, translator's note inspired by a silly joke)
Ephemeral general-purpose volumes
If users have permissions to create a pod (directly or indirectly) — they will also be able to create ephemeral general-purpose volumes even if they do not have permissions to create a volume request. This is because RBAC permission checks are applied to the controller that creates the PVC, not to the user. This is a fundamental change that needs to be added , before enabling this feature in clusters, if untrusted users should not have rights to create volumes.
Example
A separate in PMEM-CSI contains all the necessary changes to run a Kubernetes 1.19 cluster inside QEMU virtual machines with all features at the alpha stage. The driver code has not changed, only the deployment.
On a suitable machine (Linux, a regular user can use , see details) these commands will bring up the cluster and install the PMEM-CSI driver:
git clone --branch=kubernetes-1-19-blog-post https://github.com/intel/pmem-csi.git
cd pmem-csi
export TEST_KUBERNETES_VERSION=1.19 TEST_FEATURE_GATES=CSIStorageCapacity=true,GenericEphemeralVolume=true TEST_PMEM_REGISTRY=intel
make start && echo && test/setup-deployment.sh
Once everything works, the output will contain usage instructions:
The test cluster is ready. Log in with [...]pmem-csi/_work/pmem-govm/ssh.0, run
kubectl once logged in. Alternatively, use kubectl directly with the
following env variable:
KUBECONFIG=[...]pmem-csi/_work/pmem-govm/kube.config
secret/pmem-csi-registry-secrets created
secret/pmem-csi-node-secrets created
serviceaccount/pmem-csi-controller created
...
To try out the pmem-csi driver ephemeral volumes:
cat deploy/kubernetes-1.19/pmem-app-ephemeral.yaml |
[...]pmem-csi/_work/pmem-govm/ssh.0 kubectl create -f -
CSIStorageCapacity objects are not meant for human reading, so some processing is necessary. Using template filters in Golang will show storage classes; in this example, the name, topology, and capacity will be displayed:
$ kubectl get
-o go-template='{{range .items}}{{if eq .storageClassName "pmem-csi-sc-late-binding"}}{{.metadata.name}} {{.nodeTopology.matchLabels}} {{.capacity}}
{{end}}{{end}}'
csistoragecapacities
csisc-2js6n map[pmem-csi.intel.com/node:pmem-csi-pmem-govm-worker2] 30716Mi
csisc-sqdnt map[pmem-csi.intel.com/node:pmem-csi-pmem-govm-worker1] 30716Mi
csisc-ws4bv map[pmem-csi.intel.com/node:pmem-csi-pmem-govm-worker3] 30716Mi
A separate object contains the following content:
$ kubectl describe csistoragecapacities/csisc-6cw8j
Name: csisc-sqdnt
Namespace: default
Labels:
Annotations:
API Version: storage.k8s.io/v1alpha1
Capacity: 30716Mi
Kind: CSIStorageCapacity
Metadata:
Creation Timestamp: 2020-08-11T15:41:03Z
Generate Name: csisc-
Managed Fields:
...
Owner References:
API Version: apps/v1
Controller: true
Kind: StatefulSet
Name: pmem-csi-controller
UID: 590237f9-1eb4-4208-b37b-5f7eab4597d1
Resource Version: 2994
Self Link: /apis/storage.k8s.io/v1alpha1/namespaces/default/csistoragecapacities/csisc-sqdnt
UID: da36215b-3b9d-404a-a4c7-3f1c3502ab13
Node Topology:
Match Labels:
pmem-csi.intel.com/node: pmem-csi-pmem-govm-worker1
Storage Class Name: pmem-csi-sc-late-binding
Events:
Let's try to create a demo application with a single ephemeral general-purpose volume. The content of the file pmem-app-ephemeral.yaml:
# This example Pod definition demonstrates
# how to use generic ephemeral inline volumes
# with a PMEM-CSI storage class.
kind: Pod
apiVersion: v1
metadata:
name: my-csi-app-inline-volume
spec:
containers:
- name: my-frontend
image: intel/pmem-csi-driver-test:v0.7.14
command: [ "sleep", "100000" ]
volumeMounts:
- mountPath: "/data"
name: my-csi-volume
volumes:
- name: my-csi-volume
ephemeral:
volumeClaimTemplate:
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 4Gi
storageClassName: pmem-csi-sc-late-binding
After creation, as shown in the instructions above, we ended up with an additional pod and PVC:
$ kubectl get pods/my-csi-app-inline-volume -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
my-csi-app-inline-volume 1/1 Running 0 6m58s 10.36.0.2 pmem-csi-pmem-govm-worker1
$ kubectl get pvc/my-csi-app-inline-volume-my-csi-volume
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
my-csi-app-inline-volume-my-csi-volume Bound pvc-c11eb7ab-a4fa-46fe-b515-b366be908823 4Gi RWO pmem-csi-sc-late-binding 9m21s
The owner of the PVC is the pod:
$ kubectl get -o yaml pvc/my-csi-app-inline-volume-my-csi-volume
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
annotations:
pv.kubernetes.io/bind-completed: "yes"
pv.kubernetes.io/bound-by-controller: "yes"
volume.beta.kubernetes.io/storage-provisioner: pmem-csi.intel.com
volume.kubernetes.io/selected-node: pmem-csi-pmem-govm-worker1
creationTimestamp: "2020-08-11T15:44:57Z"
finalizers:
- kubernetes.io/pvc-protection
managedFields:
...
name: my-csi-app-inline-volume-my-csi-volume
namespace: default
ownerReferences:
- apiVersion: v1
blockOwnerDeletion: true
controller: true
kind: Pod
name: my-csi-app-inline-volume
uid: 75c925bf-ca8e-441a-ac67-f190b7a2265f
...
The information has been updated as expected for pmem-csi-pmem-govm-worker1:
csisc-2js6n map[pmem-csi.intel.com/node:pmem-csi-pmem-govm-worker2] 30716Mi
csisc-sqdnt map[pmem-csi.intel.com/node:pmem-csi-pmem-govm-worker1] 26620Mi
csisc-ws4bv map[pmem-csi.intel.com/node:pmem-csi-pmem-govm-worker3] 30716Mi
If another application requires more than 26620Mi, the scheduler will not take it into account pmem-csi-pmem-govm-worker1 in any case.
What's next?
Both features are still under development. Several requests were opened during alpha testing. Links with suggestions for improvements are being documented, detailing the work needed to move to the beta stage, as well as which alternatives have already been reviewed and rejected:
Source: habr.com
