Hello, Habr!
We remind you that we have released another extremely interesting and useful about Kubernetes patterns. It all started with "" by Brendan Burns, and, by the way, our work in this segment is . Today, we invite you to read an article from the MinIO blog, which briefly outlines the trends and specifics of data storage patterns in Kubernetes.
Kubernetes has fundamentally changed traditional patterns of application development and deployment. Now, teams can take just a few days to develop, test, and deploy applications across different environments, all within Kubernetes clusters. Previously, working with legacy technologies typically took weeks, if not months.
This acceleration has been made possible by the abstraction provided by Kubernetes — that is, because Kubernetes itself handles the interaction with the low-level details of physical or virtual machines, allowing users to specify parameters such as the required CPU, required memory capacity, and the number of container instances. With a vast community dedicated to supporting Kubernetes and its expanding applications, it clearly leads the way among all container orchestration platforms.
As Kubernetes usage expands, so does the confusion regarding the data storage patterns employed..
With widespread competition for a piece of the Kubernetes pie (that is, for data storage), when the topic of data storage arises, the signal gets lost in a lot of noise.
Kubernetes embodies the modern model of application development and deployment, as well as their management. This modern model decouples data storage from computing. To fully understand this decoupling in the context of Kubernetes, it is also necessary to understand stateful and stateless applications, and how data storage fits into this. This is where the REST API approach used by S3 has clear advantages over the POSIX/CSI approach characteristic of other solutions.
In this article, we will discuss data storage patterns in Kubernetes and address the debate regarding stateful and stateless applications to fully understand the differences between them and why they are important. Following this discussion, we will explore applications and the data storage patterns used in them in light of best practices for working with containers and Kubernetes.
Stateless Containers
Containers are inherently lightweight and ephemeral. They can be easily stopped, deleted, or deployed on another node in a matter of seconds. In a large container orchestration system, such operations happen continuously, and users hardly notice these changes. However, these movements are only possible if the container has no dependencies on the node it is located on. Such containers are described as operating statelessly.
Stateful Containers
If a container stores data on locally attached devices (or on a block device), the data storage it relies on must be moved to a new node along with the container itself in case of a failure. This is crucial because otherwise, the application running in the container will not function correctly, as it needs to access data stored on local drives. Such containers are described as operating stably.
From a purely technical perspective, stateful containers can also be moved to other nodes. This is typically facilitated by distributed file systems or networked block storage devices attached to all nodes running the containers. Thus, containers access volumes for persistent data storage, and information is stored on disks spread throughout the network. I will refer to this method as thestateful container approachand will continue to call it that for the sake of consistency throughout the rest of the article.
In a typical stateful container approach, all application pods are attached to a single distributed file system—resulting in a kind of shared storage where all application data resides. While some variations are possible, this is a high-level approach.
Now let's explore why a stateful container approach in a cloud-oriented world is considered an anti-pattern.
Cloud-oriented application design
Traditionally, applications used databases for structured information storage and local disks or distributed file systems to store all unstructured or semi-structured data. As the volumes of unstructured data grew, developers realized that POSIX was too "chatty", associated with significant overhead, and ultimately hindered application performance when scaling to truly large sizes.
This mainly contributed to the emergence of a new data storage standard, namely cloud-oriented storage systems that predominantly operate based on REST APIs, freeing applications from the burdensome maintenance of local data storage. In this case, the application effectively operates in a stateless mode (as the state resides in remote storage). Modern applications are built from the ground up with this factor in mind. Typically, any modern application that processes data of any kind (logs, metadata, blobs, etc.) is constructed according to a cloud-oriented paradigm where the state is shifted to a dedicated software system for its storage.
The stateful container approach forces this entire paradigm to revert back to where it all started!
When using POSIX interfaces for data storage, applications operate in the same manner as if they were saving state. As a result, they deviate from the most important tenets of cloud-oriented design, namely the ability to vary application thread sizes based on incoming load, move to a new node as soon as the current node fails, and so forth.
Upon closer examination of this situation, we find that when choosing a data storage solution, we repeatedly encounter the dilemma of 'POSIX versus REST API,' BUT with additional complications related to the distributed nature of Kubernetes environments. Specifically,
- POSIX is verbose: the semantics of POSIX require associating metadata and file descriptors with each operation to maintain the state of the operation. This leads to significant overhead with no real value. Object storage APIs, such as S3 API, eliminate these requirements, allowing the application to execute and then 'forget' about the call. The response from the storage system indicates whether the action was completed successfully or not. In case of failure, the application can retry.
- Network constraints: In a distributed system, it is assumed that multiple applications may attempt to write data to the same attached storage. Therefore, not only will applications compete with each other for bandwidth (to send data to the storage), but the storage system itself will compete for that bandwidth by distributing data across physical disks. Due to the verbosity of POSIX, the number of network calls increases significantly. On the other hand, the S3 API provides a clear distinction between network calls made from the client to the server and those that occur within the server.
- Security: The POSIX security model relies on active human involvement: administrators configure specific access levels for each user or group. This paradigm is difficult to adapt to a cloud-oriented world. Modern applications depend on security models tied to APIs, where access rights are defined as a set of policies, service accounts are allocated, temporary credentials are provided, etc.
- Manageability: Stateful containers incur certain management costs. This involves synchronizing concurrent access to data, ensuring data consistency, and requires careful consideration of which data access patterns to use. It necessitates the installation, monitoring, and configuration of additional programs, not to mention the extra efforts spent on development.
Container Storage Interface
While the Container Storage Interface (CSI) has greatly facilitated the distribution of the Kubernetes volume layer, partially handing it over to third-party storage vendors, it has also inadvertently contributed to the belief that the stateful container approach is the recommended method for data storage in Kubernetes.
CSI was designed as a standard for providing arbitrary block and file storage systems for legacy applications when working with Kubernetes. As illustrated in this article, the only situation where a stateful container approach (and CSI in its current form) makes sense is when the application itself is a legacy system that cannot support object storage API integration.
It is important to understand that by using CSI in its current form, i.e., mounting volumes for modern applications, we will encounter similar problems to those faced in systems where data storage is organized in a POSIX style.
A higher quality approach
In this case, it's important to understand that most applications are not fundamentally designed to operate with state preservation or without it. This behavior depends on the overall architecture of the system and the specific choices made during design. Let's discuss applications that maintain state a little bit.
In general, all application data can be categorized into several broad types:
- Log data
- Timestamp data
- Transaction data
- Metadata
- Container images
- Blob data (large binary objects)
All these data types are well supported on modern data storage platforms, and there are several cloud-oriented platforms designed to deliver data in each of these specific formats. For instance, transaction data and metadata can reside in a modern cloud-oriented database like CockroachDB, YugaByte, etc. Container images or blob data can be stored in a docker registry based on MinIO. Timestamp data can be stored in a time-series database like InfluxDB, etc. We won’t delve into the details of each data type and the corresponding applications here, but the general idea is to avoid persistent data storage based on local disk mounting.
Additionally, it often proves effective to provide a level of temporary caching that serves as a kind of storage for temporary files for applications, but applications should not depend on this level as a source of truth.
Storage for stateful applications
While it is generally beneficial to keep applications stateless, those applications that are designed to store data—such as databases, object stores, and key-value stores—must maintain state. Let's discuss why these applications are deployed on Kubernetes. We'll take MinIO as an example, but similar principles apply to any other large cloud-oriented data storage systems.
Cloud-native applications are designed to maximize the effective use of the flexibility inherent in containers. This means that no assumptions are made about the environment in which they will be deployed. For example, MinIO uses an internal erasure coding mechanism that provides sufficient resilience to keep the system operational even in the event of half the disks failing. MinIO also manages data integrity and security by using its own hashing and server-side encryption.
For such cloud-native applications, local persistent volumes (PV) are the most convenient backup storage option. A local PV allows for the storage of raw data, while applications running on top of these PVs collect the information needed to scale data and manage growing data demands.
This approach is much simpler and scales significantly better compared to CSI-based PVs, which introduce their own layers of data management and redundancy; the issue is that these layers typically conflict with stateful applications.
A confident move towards decoupling data from computing
In this article, we discussed how applications are shifting towards stateless operations, or in other words, data storage is being separated from the computations performed on it. Finally, let's consider a few real-world examples of this trend.
, the famous data analysis platform, has traditionally been used in a stateful manner and deployed on the HDFS file system. However, as Spark transitions to a cloud-native world, this platform is increasingly being utilized in a stateless fashion using `s3a`. Spark employs s3a to pass state onto other systems, while the Spark containers themselves operate entirely in a stateless manner. Other major enterprise players in big data analytics, in particular, , , are also moving towards separating data storage from the computations performed on it.
Similar patterns can also be seen on other major analytics platforms, including Presto, TensorFlow, and Jupyter. By exporting your state to remote cloud storage systems, it becomes much easier to manage and scale your application. Additionally, this enhances the portability of your application across various environments.
Source: habr.com
