Note: translation.: After the recent publication on pull and push methods in GitOps, we noticed a growing interest in this model overall; however, there are very few publications on this topic in Russian (there are simply none on Habr). Therefore, we are pleased to present you with a translation of another article — although it is almost a year old! — from Weaveworks, whose head coined the term 'GitOps.' The text explains the essence of the approach and the key differences from what already exists.
A year ago, we published . At that time, we explained how Weaveworks launched a SaaS entirely based on Kubernetes and developed a set of prescriptive best practices for deployment, management, and monitoring in a cloud-native environment.
The article became popular. Other people began to talk about GitOps, publishing new tools for , , , , and so on. Our site featured articles and use cases for GitOps. However, some people still had questions. How does the model differ from traditional and continuous delivery ()? Is it mandatory to use Kubernetes?
Soon we realized that a new description was necessary, offering:
- A large number of examples and stories;
- A specific definition of GitOps;
- A comparison with traditional continuous delivery.
In this article, we attempted to cover all these topics. You will find an updated introduction to GitOps and a perspective on it from the viewpoint of developers and CI/CD. We primarily focus on Kubernetes, although the model can be generalized.
Meet GitOps
Imagine Alice. She runs Family Insurance, which offers health, auto, property, and travel insurance policies to people who are too busy to navigate the nuances of contracts on their own. Her business started as a side project when Alice worked at a bank as a data scientist. One day she realized that she could use advanced computer algorithms for more efficient data analysis and insurance package creation. Investors funded the project, and now her company generates over $20 million a year and is rapidly growing. Currently, it has 180 employees in various positions, including a technology team responsible for the development, maintenance of the website, database, and customer base analysis. The team of 60 is led by Bob, the company's Chief Technology Officer.
Bob's team deploys production systems in the cloud. Their main applications run on GKE, leveraging the advantages of Kubernetes on Google Cloud. Additionally, they utilize various tools for data management and analytics.
Family Insurance initially wasn't planning to use containers, but they became excited about Docker. Soon, the company's specialists discovered that GKE allows them to easily deploy clusters for testing new features. They added Jenkins for CI and Quay for container registry management, and wrote scripts for Jenkins that pushed new containers and configurations to GKE.
Some time has passed. Alice and Bob grew disappointed with the performance of their chosen approach and its impact on the business. Implementing containers did not enhance performance as much as the team had hoped. Sometimes deployments failed, and it was unclear whether the code changes were to blame. It was also difficult to track configuration changes. Often, they had to create a new cluster and move applications into it, as it was the easiest way to clean up the mess the system had become. Alice feared that the situation would worsen as the application evolved (additionally, a new project based on machine learning was looming). Bob had automated much of the work and did not understand why the pipeline was still unstable, poorly scalable, and periodically required manual intervention.
Then they learned about GitOps. This solution turned out to be exactly what they needed to move forward with confidence.
Alice and Bob had been hearing about Git-based workflows, DevOps, and infrastructure as code for several years. The uniqueness of GitOps lies in the way it brings a set of best practices—categorical and normative—for implementing these ideas in the context of Kubernetes. This topic , including in the .
Family Insurance decides to implement GitOps. Now the company has an automated operational model that is compatible with Kubernetes and combines speed with stability, as they:
- discovered that the team's performance doubled without anyone losing their mind;
- stopped maintaining scripts. Instead, they can now focus on new features and improve engineering methods—for example, implementing canary releases and enhancing testing;
- refined the deployment process—it rarely fails now;
- gained the ability to recover deployments after partial failures without manual intervention;
- acquired morehigher level of isolation, as if one controller is broken, the problem is confined to that specific context).confidence in delivery systems. Alice and Bob found that they could split the team into groups working on microservices in parallel;
- each group could make 30-50 changes to the project each day and try new techniques.
- easily attract new developers to the project, who can deploy updates to production via pull requests within a few hours;
- easily pass SOC2 audits (for compliance with service provider requirements for secure data management; read more, for example, - translator's note).
What happened?
GitOps is two things:
- An operational model for Kubernetes and cloud-native. It provides a set of best practices for deploying, managing, and monitoring containerized clusters and applications. An elegant definition in the form of from :
- The path to creating a developer-centric environment for managing applications. We apply the Git workflow to both operations and development. Note that this is not just about Git push, but about organizing a complete set of CI/CD and UI/UX tools.
A few words about Git
If you are unfamiliar with version control systems and Git-based workflows, we strongly recommend learning them. Initially, working with branches and pull requests may seem like black magic, but the benefits are worth the effort. Here’s to get you started.
How Kubernetes works
In our story, Alice and Bob turned to GitOps after working with Kubernetes for a while. Indeed, GitOps is closely linked to Kubernetes — it’s an operational model for infrastructure and applications based on Kubernetes.
What does Kubernetes offer users?
Here are some key capabilities:
- In the Kubernetes model, everything can be described in a declarative form.
- The Kubernetes API server takes this declaration as input and then continually attempts to bring the cluster to the state described in the declaration.
- Declarations are sufficient to describe and manage a wide variety of workloads — 'applications'.
- As a result, changes in the application and cluster occur due to:
- changes in container images;
- changes in the declarative specification;
- errors in the environment — for example, container crashes.
Kubernetes' excellent convergence capabilities
When an administrator makes changes to the configuration, the Kubernetes orchestrator will apply them to the cluster until its state approaches the new configurationThis model works for any Kubernetes resource and is extensible with Custom Resource Definitions (CRDs). Therefore, Kubernetes deployments have the following wonderful properties:
- Automation: Kubernetes updates provide a mechanism for automating the process of applying changes correctly and in a timely manner.
- Convergence: Kubernetes will continue to attempt updates until successful.
- Idempotence: Reapplying convergence leads to the same result.
- Determinism: Given sufficient resources, the state of the updated cluster depends only on the desired state.
How GitOps Works
We have learned enough about Kubernetes to explain the principles of GitOps.
Let's return to the Family Insurance teams related to microservices. What do they usually have to deal with? Look at the list below (if any items seem strange or unfamiliar — please hold off on criticism and stay with us). These are just examples of workflows based on Jenkins. There are many other processes when working with other tools.
The main point is that we see that each update ends with changes to configuration files and Git repositories. These changes in Git cause the "GitOps operator" to update the cluster:
1. Workflow: "Jenkins Build — master branch».
Task List:
- Jenkins pushes tagged images to Quay;
- Jenkins pushes config and Helm charts to the master storage bucket;
- A cloud function copies config and charts from the master storage bucket to the master Git repository;
- The GitOps operator updates the cluster.
2. Jenkins Build — release or hotfix branch:
- Jenkins pushes untagged images to Quay;
- Jenkins pushes config and Helm charts to the staging storage bucket;
- A cloud function copies config and charts from the staging storage bucket to the staging Git repository;
- The GitOps operator updates the cluster.
3. Jenkins Build — develop or feature branch:
- Jenkins pushes untagged images to Quay;
- Jenkins pushes config and Helm charts to the develop storage bucket;
- A cloud function copies config and charts from the develop storage bucket to the develop Git repository;
- The GitOps operator updates the cluster.
4. Adding a New Client:
- The manager or administrator (LCM/ops) invokes Gradle for the initial deployment and network load balancer (NLB) setup;
- LCM/ops commits the new config to prepare the deployment for updates;
- The GitOps operator updates the cluster.
Brief Description of GitOps
- Describe the desired state of the entire system using declarative specifications for each environment (in our story, Bob's team defines the entire system configuration in Git).
- The Git repository is the single source of truth regarding the desired state of the entire system.
- All changes to the desired state are made through commits to Git.
- All desired parameters of the cluster are also observable within the cluster itself. Thus, we can determine if they converge, converge) or diverge, diverge) from the desired and observed states.
- If the desired and observed states differ, then:
- There is a convergence mechanism that will eventually and automatically synchronize the target and observed states. Within the cluster, this is handled by Kubernetes.
- The process is initiated immediately with a notification of 'change committed.'
- After a configurable interval, a 'diff' notification may be sent if the states differ.
- Thus, all commits in Git trigger verifiable and idempotent updates in the cluster.
- Rollback is convergence to a previously desired state.
- Convergence is definitive. Its occurrence is indicated by:
- The absence of 'diff' notifications for a certain period.
- 'Converged' notification (e.g., webhook, Git writeback event).
What is divergence?
Let's reiterate: all desired properties of the cluster must be observable within the cluster itself..
Several examples of divergence:
- Change in the configuration file due to branch merges in Git.
- Change in the configuration file due to a commit in Git made by a GUI client.
- Multiple changes in the desired state due to a PR in Git followed by building a container image and making config changes.
- Change in the cluster state due to an error, resource conflict leading to 'bad behavior,' or simply an accidental deviation from the original state.
What constitutes a convergence mechanism?
Several examples include:
- For containers and clusters, the convergence mechanism is provided by Kubernetes.
- The same mechanism can be used for managing applications and constructs based on Kubernetes (e.g., Istio and Kubeflow).
- The mechanism for managing the operational interaction between Kubernetes, image repositories, and Git provides , which is part of .
- For basic machines, the convergence mechanism must be declarative and autonomous. From our experience, we can say that is closest to this definition, yet still requires human oversight. In this sense, GitOps expands the traditions of Infrastructure as Code.
GitOps combines Git with the excellent convergence mechanism of Kubernetes, offering a model for operations.
GitOps allows us to declare: only those systems that can be described and monitored are subject to automation and control.
GitOps is intended for the entire cloud native stack (e.g., Terraform, etc.)
GitOps is not just Kubernetes. We want the entire system to be managed declaratively and utilize convergence. By the entire system, we mean the collection of environments that work with Kubernetes — for example, 'dev cluster 1', 'production', etc. Each environment includes machines, clusters, applications, and interfaces for external services providing data, monitoring, etc.
Note how important Terraform is for the bootstrapping problem in this case. Kubernetes must be deployed somewhere, and using Terraform means we can apply the same GitOps workflows to create the control layer underlying Kubernetes and applications. This is a useful best practice.
Great attention is paid to applying GitOps concepts to layers above Kubernetes. Currently, there are GitOps-type solutions for Istio, Helm, Ksonnet, OpenFaaS, and Kubeflow, as well as for Pulumi, which create a layer for developing applications for cloud native.
Kubernetes CI/CD: comparing GitOps with other approaches
As mentioned, GitOps is two things:
- An operational model for Kubernetes and cloud native, as described above.
- A path to organizing a developer-centric environment for managing applications.
For many, GitOps is primarily a workflow based on Git pushes. We like it too. But that's not all: let's now look at CI/CD pipelines.
GitOps provides continuous deployment (CD) under Kubernetes
GitOps offers a continuous deployment mechanism, eliminating the need for separate 'deployment management systems'. Kubernetes does all the work for you.
- The application update requires an update in Git. This is a transactional update to the desired state. The 'deployment' is then performed within the cluster by Kubernetes based on the updated description.
- Due to the nature of Kubernetes, these updates are convergent. This ensures a mechanism for continuous deployment, where all updates are atomic.
- Note: offers a GitOps operator that integrates Git and Kubernetes, allowing for CD by reconciling the desired and current state of the cluster.
Without kubectl and scripts
It is advisable to avoid using kubectl for updating the cluster, especially scripts for batching kubectl commands. Instead, using a GitOps pipeline, a user can update their Kubernetes cluster via Git.
The benefits include:
- Correctness. A group of updates can be applied, converged, and finally validated, bringing us closer to the goal of atomic deployment. In contrast, using scripts provides no guarantees of convergence (more on this below).
- Security. Kelsey Hightower: 'Limit access to the Kubernetes cluster to automation tools and administrators responsible for debugging or maintaining its operation.' See also on security and compliance with specifications, as well as by stealing credentials from a carelessly written Jenkins script.
- User Experience. Kubectl exposes the mechanics of the Kubernetes object model, which is quite complex. Ideally, users should interact with the system at a higher level of abstraction. Here I will again refer to Kelsey and recommend checking out .
The difference between CI and CD
GitOps improves existing CI/CD models.
A modern CI server is an orchestration tool. In particular, it is a tool for orchestrating CI pipelines. These include build, test, merge to trunk, etc. CI servers automate the management of complex multi-step pipelines. A common temptation is to create a script for a set of Kubernetes updates and run it as a pipeline element for pushing changes to the cluster. Indeed, many specialists do this. However, it is not optimal, and here's why.
CI should be used to apply updates to the trunk, and the Kubernetes cluster should change itself based on these updates to manage CD "internally." We call this , as opposed to the CI push model. CD is a part of runtime orchestration..
Why CI servers should not perform CD through direct updates to Kubernetes
Do not use a CI server to orchestrate direct updates to Kubernetes as a set of CI tasks. This is an anti-pattern that we in our blog.
Let's return to Alice and Bob.
What problems did they face? Bob's CI server applies changes to the cluster, but if it fails during the process, Bob won't know what state the cluster is in (or should be) and how to fix it. The same is true in case of success.
Let's assume Bob's team built a new image and then patched their deployments to roll out the image (all from the CI pipeline).
If the image builds successfully, but the pipeline fails, the team will have to figure out:
- Did the update deploy?
- Are we running a new build? Will this cause unnecessary side effects — potentially getting two builds of the same unchanged image?
- Should we wait for the next update before triggering a build?
- What exactly went wrong? Which steps need to be repeated (and which of them are safe to repeat)?
Organizing a Git-based workflow does not guarantee that Bob's team won't face these issues. They can still make mistakes with pushing commits, tagging, or any other parameter; however, this approach is still much closer to an explicit all-or-nothing.
To summarize, here’s why CI servers should not engage in CD:
- Update scripts are not always deterministic; they are easy to make mistakes with.
- CI servers do not converge to a declarative model of the cluster.
- It's difficult to guarantee idempotence. Users need to understand the deep semantics of the system.
- It's harder to recover from a partial failure.
Note about Helm: if you want to use Helm, we recommend combining it with a GitOps operator, such as . This will help ensure convergence. Helm alone is neither deterministic nor atomic.
GitOps as the best way to implement Continuous Delivery for Kubernetes
Alice and Bob's team is implementing GitOps and finding it much easier to work with software products while maintaining high performance and stability. Let's conclude this article with illustrations showing what their new approach looks like. Note that we are mainly discussing applications and services, but GitOps can be used to manage the entire platform.
Operational model for Kubernetes
Look at the following diagram. It represents Git and the container image repository as common resources for two orchestrated life cycles:
- The Continuous Integration pipeline, which reads and writes files to Git and can update the container image repository.
- The Runtime GitOps pipeline, which combines deployment with management and observability. It reads and writes files to Git and can load container images.
What are the main takeaways?
- Separation of concerns: Note that both pipelines can exchange data only by updating Git or the image repository. In other words, there is a firewall between the CI and the runtime environment. We call it the 'immutability firewall' (immutability firewall), as all repository updates create new versions. For additional information on this topic, refer to slides 72-87. .
- Any CI and Git server can be used: GitOps works with any components. You can continue using your favorite CI and Git servers, image repositories, and test suites. Almost all other Continuous Delivery tools on the market require their own CI/Git server or image repository. This can become a limiting factor in cloud-native development. With GitOps, you can use familiar tools.
- Events as an integration tool: Once data in Git is updated, Weave Flux (or the Weave Cloud operator) notifies the runtime. Whenever Kubernetes receives a set of changes, Git is updated. This provides a simple integration model for organizing workflows for GitOps, as shown below.
Conclusion
GitOps provides significant guarantees for updates that any modern CI/CD tool requires:
- automation;
- convergence;
- idempotence;
- determinism.
This is important as it provides an operating model for developers in the cloud native field.
- Traditional tools for system management and monitoring are associated with operations teams working within a runbook framework. (a set of routine procedures and operations — ed. note), tied to a specific deployment.
- In managing cloud native systems, monitoring tools are the best way to assess deployment outcomes so that the development team can respond promptly.
Imagine numerous clusters spread across various clouds and a multitude of services with their own teams and deployment plans. GitOps offers a scale-invariant model for managing this abundance.
P.S. from the translator
Also read in our blog:
- «»;
- «»;
- «».
Only registered users can participate in the survey. , please.
Did you know about GitOps before these two translations on Habr?
Yes, I was aware of it.
Only superficially.
No
35 users voted. 10 users abstained.
Source: habr.com
