🥇Local Files When Migrating an Application to Kubernetes

When building a CI/CD process using Kubernetes, a common problem arises due to incompatibilities between the requirements of the new infrastructure and the application being migrated to it. In particular, during the application's build stage, it is important to obtain one an image that will be used in all the project's environments and clusters. This principle underlies proper container management, according to Google (this has been mentioned multiple times by our tech director). said However, one often encounters situations where a ready-made framework is utilized in website code, which imposes limitations on its further use. While this can be easily managed in a 'traditional environment,' in Kubernetes, such behavior can become problematic, especially when faced with it for the first time. Although a resourceful mind can propose infrastructure solutions that seem obvious and even quite good at first glance, it is important to remember that most situations can and should

be resolved architecturally. Let's examine popular workaround solutions for file storage that may lead to unpleasant consequences when operating a cluster and point out a more correct approach..

Static Storage

To illustrate, let's consider a web application that uses some static asset generator to obtain a set of images, styles, and more. For example, in the PHP framework Yii, there is a built-in asset manager that generates unique directory names. Consequently, the output is a set of paths for the site's static resources that do not intersect (this is done for several reasons—such as avoiding duplicates when the same resource is used by multiple components). Thus, out of the box, upon the first request to the web resource module, static assets are created and distributed (often as symlinks, but more on that later) with a unique root directory for this deployment:

webroot/assets/2072c2df/css/…

webroot/assets/2072c2df/images/…
webroot/assets/2072c2df/js/…
What risks does this pose in the context of a cluster?

A simple example

Let's take a fairly common case where nginx stands before PHP to serve static files and handle simple requests. The simplest way is

with two containers: Deployment с двумя контейнерами:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: site
spec:
  selector:
    matchLabels:
      component: backend
  template:
    metadata:
      labels:
        component: backend
    spec:
      volumes:
        - name: nginx-config
          configMap:
            name: nginx-configmap
      containers:
      - name: php
        image: own-image-with-php-backend:v1.0
        command: ["/usr/local/sbin/php-fpm","-F"]
        workingDir: /var/www
      - name: nginx
        image: nginx:1.16.0
        command: ["/usr/sbin/nginx", "-g", "daemon off;"]
        volumeMounts:
        - name: nginx-config
          mountPath: /etc/nginx/conf.d/default.conf
          subPath: nginx.conf

In simplified terms, the nginx configuration can be summarized as follows:

apiVersion: v1
kind: ConfigMap
metadata:
  name: "nginx-configmap"
data:
  nginx.conf: |
    server {
        listen 80;
        server_name _;
        charset utf-8;
        root  /var/www;

        access_log /dev/stdout;
        error_log /dev/stderr;

        location / {
            index index.php;
            try_files $uri $uri/ /index.php?$args;
        }

        location ~ .php$ {
            fastcgi_pass 127.0.0.1:9000;
            fastcgi_index index.php;
            include fastcgi_params;
        }
    }

When the website is first accessed in the PHP container, assets appear. However, with two containers within the same pod, nginx knows nothing about the static files that (according to the configuration) should be served specifically by it. As a result, for all requests to CSS and JS files, the client will see a 404 error. The simplest solution here would be to organize a shared directory between the containers. A basic option would be a shared emptyDir:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: site
spec:
  selector:
    matchLabels:
      component: backend
  template:
    metadata:
      labels:
        component: backend
    spec:
      volumes:
        - name: assets
          emptyDir: {}
        - name: nginx-config
          configMap:
            name: nginx-configmap
      containers:
      - name: php
        image: own-image-with-php-backend:v1.0
        command: ["/usr/local/sbin/php-fpm","-F"]
        workingDir: /var/www
        volumeMounts:
        - name: assets
          mountPath: /var/www/assets
      - name: nginx
        image: nginx:1.16.0
        command: ["/usr/sbin/nginx", "-g", "daemon off;"]
        volumeMounts:
        - name: assets
          mountPath: /var/www/assets
        - name: nginx-config
          mountPath: /etc/nginx/conf.d/default.conf
          subPath: nginx.conf

Now, the static files generated in the container are served correctly by nginx. However, I remind you that this is a primitive solution, meaning it is far from ideal and has its nuances and shortcomings, which are discussed below.

A more advanced storage option

Now imagine a situation where a user visits a website, loads a page with the existing styles in the container, and while they are reading the page, we redeploy the container. The asset directory becomes empty, and a request to PHP is required to trigger the generation of new ones. However, even after this, links to the old static content will be outdated, leading to static display errors.

Moreover, we are likely dealing with a project that experiences some load, which means a single copy of the application will not be sufficient:

Scale Deployment to two replicas.
Upon the first request to the site, assets were created in one replica.
At some point, the ingress decided (for load balancing purposes) to send the request to the second replica, where those assets are still missing. Or perhaps they are no longer there because we are using RollingUpdate and currently deploying.

In short, the result is again errors.

To avoid losing old assets, we can change emptyDir to hostPath, physically storing the static content on a cluster node. This approach is bad because we essentially have to tie ourselves to a specific cluster node with our application because, if we move to other nodes, the directory will not contain the necessary files. Alternatively, some background synchronization of the directory between nodes is required.

What are the solutions?

If the hardware and resources allow, we can use cephfs to create an accessible directory for static needs. Official Documentation It recommends SSD drives, at least threefold replication, and a stable 'thick' connection between cluster nodes.
A less demanding option would be to set up an NFS server. However, in that case, one must consider a possible increase in response time for processing requests by the web server, and the fault tolerance will leave much to be desired. The consequences of a failure can be catastrophic: losing the mount could doom the cluster to collapse under the load pressure soaring to the sky.

In addition, for all options for creating permanent storage, a background cleanup of outdated file sets accumulated over a certain period will be required. Caching nginx can be placed in front of PHP containers to store copies of assets for a limited time. This behavior is easily configurable using DaemonSet proxy_cache. proxy_cache with storage depth in days or gigabytes of disk space.

Combining this method with the distributed file systems mentioned above opens up vast possibilities, limited only by the budget and technical capability of those who will implement and maintain it. From experience, we can say that simpler systems tend to be more stable. Adding such layers complicates infrastructure maintenance significantly, increasing the time required for diagnostics and recovery in case of failures.

Recommendation

If the implementation of the proposed storage options seems unreasonable (complex, expensive...), it is worth looking at the situation from a different angle. Specifically, delve into the project's architecture and root out the problem in the code, tying it to some static data structure in the image, unambiguously defining the content or the "warming up" procedure and/or precompiling assets at the image build stage. This way, we achieve absolutely predictable behavior and an identical set of files for all environments and replicas of the launched application.

Returning to the specific example with the Yii framework and not delving into its structure (which is not the purpose of the article), it is enough to point out two popular approaches:

Change the image build process to place assets in a predictable location. This is what is proposed/implemented in extensions like yii2-static-assets.
Define specific hashes for asset directories, as discussed, for example, in this presentation. (starting from slide #35). Interestingly, the report's author ultimately (and not without reason!) recommends uploading assets to a central repository (like S3) after building them on the build server, placing a CDN in front of it.

Uploaded files

Another case that will definitely arise when migrating an application to a Kubernetes cluster is storing user files in the file system. For instance, we have a PHP application that accepts files via an upload form, does something with them during processing, and returns them.

The location where these files should be placed in Kubernetes reality needs to be shared across all replicas of the application. Depending on the complexity of the application and the need to organize the persistence of these files, such a location could be the aforementioned types of shared storage, but as we can see, they have their drawbacks.

Recommendation

One solution option is using S3-compatible storage (even if it's some variant of self-hosted like minio). Transitioning to work with S3 will require changes at the code level,, and how the content will be delivered at the frontend, we already reported.

User sessions

It's also worth noting the organization of user session storage. Often, this is also files on disk, which in the context of Kubernetes will lead to constant authentication requests from the user if their request goes to a different container.

Part of the problem can be solved by enabling stickySessions on ingress This feature is supported in all popular ingress controllers — see more in our review), to bind the user to a specific pod with the application:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: nginx-test
  annotations:
    nginx.ingress.kubernetes.io/affinity: "cookie"
    nginx.ingress.kubernetes.io/session-cookie-name: "route"
    nginx.ingress.kubernetes.io/session-cookie-expires: "172800"
    nginx.ingress.kubernetes.io/session-cookie-max-age: "172800"

spec:
  rules:
  - host: stickyingress.example.com
    http:
      paths:
      - backend:
          serviceName: http-svc
          servicePort: 80
        path: /

But this will not eliminate the problems during redeployments.

Recommendation

A better approach would be to shift the application to session storage in memcached, Redis, and similar solutions — in general, completely abandon file-based options.

Conclusion

The infrastructural solutions discussed in the text are only worthy of application in the format of temporary "workarounds". They may be relevant in the initial stages of migrating the application to Kubernetes, but they should not "take root."

The overall recommended path is to eliminate them in favor of architectural adjustments to the application in accordance with the already well-known 12-Factor App. However, making the application stateless inevitably means that changes in the code will be required, and it is important to find a balance between business capabilities/requirements and the prospects for implementing and maintaining the chosen path.