Note: translation.: In the Kubernetes community, a trend known as GitOps is gaining prominent popularity, which we personally witnessed, KubeCon Europe 2019. This term was relatively recently by the head of Weaveworks — Alexis Richardson — and refers to the application of familiar tools for developers (primarily Git, from which the name derives) to solve operational tasks. Specifically, it concerns managing Kubernetes through storing its configurations in Git and automatically deploying changes to the cluster. Matthias Jg discusses the two approaches to deployment in this article.
Last year (in fact, this formally occurred in August 2017 — note from the translator) a new approach to deploying applications in Kubernetes emerged. It's called GitOps, and it is fundamentally based on the idea that deployment versioning is maintained in a secure Git repository.
The main advantages of this approach are as follows:
- Versioning of deployments and change history. The state of the entire cluster is stored in a Git repository, and deployments are updated only via commits. Moreover, all changes can be tracked through the commit history.
- Rollbacks using familiar Git commands. Simple
git resetallows reverting changes in deployments; previous states are always accessible. - Ready access control. Typically, a Git system contains a lot of sensitive data, so most companies pay special attention to its protection. Consequently, this protection extends to operations with deployments.
- Policies for deployments. Most Git systems natively support policies for different branches — for instance, only pull requests can update the master branch, and a different team member must review and approve changes. Similar to access control, the same policies apply to deployment updates.
As you can see, the GitOps method has many advantages. Over the last year, two approaches have gained particular popularity. One is based on push, the other on pull. Before delving into them, let’s first examine what typical Kubernetes deployments look like.
Deployment Methods
In recent years, various methods and tools for deployments have been established in Kubernetes:
- Based on native Kubernetes templates/KustomizeThis is the simplest way to deploy applications in Kubernetes. The developer creates basic YAML files and applies them. To avoid repeatedly rewriting the same templates, Kustomize was developed (it turns Kubernetes templates into modules). Note: translation.: Kustomize has been integrated into kubectl since .
- Helm Charts. Helm Charts allow for the creation of sets of templates, init containers, sidecars, etc., which are used to deploy applications with more flexible configuration options than the template-based approach. At the core of this method are templated YAML files. Helm fills them with various parameters and then sends them to Tiller—a cluster component that deploys them in the cluster and allows for updates and rollbacks. Importantly, Helm essentially just inserts the necessary values into the templates and then applies them in the same way as in the traditional approach. (for more information on how it all works and how to use it, read our - translator's note). There is a wide variety of ready-made Helm charts covering a broad range of tasks.
- Alternative Tools. There are many alternative tools. What unites them is that they convert certain template files into understandable Kubernetes YAML files and then apply them.
In our work, we constantly use Helm charts for important tools (as much of it is pre-configured, which significantly simplifies life) and 'clean' Kubernetes YAML files for deploying our own applications.
Pull & Push
In one of my recent blog posts, I introduced a tool , which allows committing templates to a Git repository and updating the deployment after each commit or push of a container. My experience shows that this tool is one of the key players in promoting the pull approach, so I will reference it frequently. If you want to learn more about how to use it, here’s .
NB! All the benefits of using GitOps remain for both approaches.
Pull-based approach
The pull approach is based on the fact that all changes are applied from inside the cluster. Inside the cluster, there is an operator that regularly checks the associated Git repositories and Docker Registry. If there are any changes, the state of the cluster is updated from within. This process is generally considered quite safe, as no external client has administrator access to the cluster.
Pros:
- No external client has the rights to make changes to the cluster; all updates are applied from within.
- Some tools also allow for synchronizing updates of Helm charts and linking them to the cluster.
- Docker Registry can be scanned for new versions. If a new image appears, the Git repository and deployment are updated to the new version.
- Pull tools can be distributed across different namespaces with different Git repositories and access rights. This allows for a multitenant model. For example, Team A can use Namespace A, Team B can use Namespace B, and the infrastructure team can use a global space.
- Typically, tools are quite lightweight.
- In conjunction with tools like the operator , secrets can be stored in an encrypted form in the Git repository and retrieved from within the cluster.
- There is no connection to CD pipelines, as deployments occur within the cluster.
Cons:
- Managing deployment secrets from Helm charts is more complex than ordinary ones, as they first need to be generated in the form of sealed secrets, then decrypted by the internal operator, and only then do they become accessible to the pull tool. Then a release can be run in Helm with the values in the already deployed secrets. The simplest way is to create a secret with all Helm values used for the deployment, decrypt it, and commit it to Git.
- By using the pull approach, you find yourself tied to tools that operate on pulls. This limits the ability to customize the deployment process in the cluster. For example, working with Kustomize is complicated because it must be executed before the final templates are committed to Git. I'm not saying that separate tools can't be used, but they are harder to integrate into the deployment process.
Push-Based Approach
In the push approach, an external system (mainly CD pipelines) triggers deployments in the cluster after a commit to the Git repository or upon successful completion of the preceding CI pipeline. In this method, the system has access to the cluster.
Advantages:
- Security is determined by the Git repository and the build pipeline.
- Deploying Helm charts is easier, with support for Helm plugins.
- Managing secrets is simpler, as secrets can be applied in pipelines and stored in Git in encrypted form (depending on user preferences).
- No binding to a specific tool, as any types can be used.
- Container version updates can be initiated by the build pipeline.
Cons:
- Data for accessing the cluster resides within the build system.
- Updating container deployments is still easier with the pull process.
- Strong dependency on the CD system, as the pipelines we need may have originally been written for GitLab Runners, and then the team decides to switch to Azure DevOps or Jenkins... leading to the necessity of migrating a large number of build pipelines.
Conclusion: Push or Pull?
As it often happens, each approach has its pros and cons. Some tasks are easier to accomplish with one method and harder with another. Initially, I deployed manually, but after coming across several articles about Weave Flux, I decided to implement GitOps processes for all projects. For basic templates, it turned out to be simple, but then I began to face difficulties with Helm charts. At that time, Weave Flux only offered a nascent version of the Helm Chart Operator, but even now some tasks are more complex due to the need to manually create secrets and apply them. You could argue that the pull approach is much more secure since the cluster credentials are not accessible outside of it, which significantly enhances security and justifies the extra effort.
After some reflection, I came to an unexpected conclusion that this is not the case. When it comes to components that require maximum protection, secret storage and CI/CD systems, as well as Git repositories, would be on that list. The information within them is quite vulnerable and needs the highest level of protection. Furthermore, if someone gains access to your Git repository and can push code there, they can deploy whatever they wish (regardless of the chosen approach, whether pull or push) and infiltrate the cluster systems. Thus, the most critical components requiring protection are the Git repository and CI/CD systems, rather than cluster credentials. If you have well-defined policies and security measures for such systems, and cluster credentials are only extracted in pipeline secrets, the additional security of the pull approach may not be as valuable as initially thought.
So, if the pull approach is more labor-intensive and does not provide a security advantage, wouldn't it be more logical to use only the push approach? However, someone might argue that with the push approach, you are too tied to the CD system, and perhaps it’s better not to do this to facilitate migrations in the future.
In my opinion (as always), it is best to use what fits the specific case or to combine approaches. Personally, I use both approaches: Weave Flux for pull-based deployments, which mainly involve our own services, and the push approach with Helm and plugins, which simplifies the application of Helm charts to the cluster and allows for easy secret creation. I think there will never be a one-size-fits-all solution, as there are always many nuances that depend on the specific use case. That said, I strongly recommend GitOps—it greatly simplifies life and enhances security.
I hope my experience on this topic helps you decide which method is more suitable for your type of deployments, and I would be glad to hear your opinion.
P.S. Note from the translator
One downside of the pull model is the difficulty of committing rendered manifests to Git; however, there's no downside that the CD pipeline in the pull model lives separately from the deployment and essentially becomes a pipeline of the category Continuous Apply. Therefore, it will require even more effort to gather the status from all deployments and somehow provide access to logs/status, preferably linked to the CD system.
In this sense, the push model allows some guarantees for deployment, as the lifespan of the pipeline can be made equal to the lifespan of the deployment.
We have tested both models and arrived at the same conclusions as the author of the article:
- The pull model is suitable for organizing updates of system components across a large number of clusters (see ).
- The push model based on GitLab CI is well-suited for deploying applications using Helm charts. Furthermore, the deployment of applications within pipelines is tracked using the tool . By the way, in the context of this project, we constantly heard the term "GitOps" when discussing pressing issues faced by DevOps engineers at our booth at KubeCon Europe '19.
Our experience with data in the etcd Kubernetes cluster directly (without K8s API)
Also read in our blog:
- «»;
- «»;
- «»;
- «».
Only registered users can participate in the survey. , please.
Do you use GitOps?
Yes, pull approach
Yes, push
Yes, pull + push
Yes, something else
No
30 users voted. 10 users abstained.
Source: habr.com
