Today we will talk about the principles and models of GitOps, as well as how these models are implemented on the OpenShift platform. An interactive guide on this topic is available .
In short, GitOps is a set of best practices for using Git pull requests to manage infrastructure and application configurations. The Git repository is treated as the single source of system state within GitOps, and any changes to that state are fully traceable and auditable.
The idea of change tracking in GitOps is by no means new; this approach has long been and almost universally used when working with the source code of applications. GitOps simply implements the same functionality (reviews, pull requests, tags, etc.) in infrastructure and application configuration management, and provides the same benefits as source code management.
There is no academic definition or set of rules for GitOps, just a set of principles on which the practice is based:
- The declarative description of the system is stored in the Git repository (configs, monitoring, etc.).
- State changes are made via pull requests.
- The state of running systems is brought into line with the data in the repository using Git push requests.
Principles of GitOps
- System definitions are described as source code
Systems configuration is treated as code, so it can be stored and automatically versioned in a Git repository, which serves as the only source of truth. This approach makes it easy to rollout (rollout) and rollback (rollback) changes in systems.
- The desired state and configuration of systems is set and versioned in Git
By storing and versioning the desired state of systems in Git, we are able to easily roll and roll back changes to systems and applications. We can also use Git's security mechanisms to control ownership of the code and verify its authenticity.
- Configuration changes can be automatically applied via pull requests
Using Git pull requests, we can easily control how changes are applied to the configurations in the repository. For example, they can be given to other team members for verification or run through CI tests, etc.
And thus it is not necessary to distribute administrative powers to the right and to the left. To commit configuration changes, users only need the appropriate permissions on the Git repository where those configurations are stored.
- Fixing the problem of uncontrolled configuration drift
Once the desired state of the system is stored in the Git repository, all we have to do is find some software to ensure that the current state of the system matches the desired state. If this is not the case, then this software should - depending on the settings - either fix the discrepancy on its own, or notify us of a configuration drift.
GitOps Models for OpenShift
On-Cluster Resource Reconciler
According to this model, the cluster has a controller that is responsible for comparing the Kubernetes resources (YAML files) in the Git repository with the actual cluster resources. When discrepancies are identified, the controller sends out notifications and possibly takes action to correct the discrepancies. This GitOps model is used by Anthos Config Management and Weaveworks Flux.
External Resource Reconciler (Push)
This model can be viewed as a variation of the previous one, when we have one or more controllers responsible for synchronizing resources in Git repository - Kubernetes cluster pairs. The difference here is that each managed cluster does not have to have its own separate controller. Git-k8s cluster pairs are often defined as CRDs (custom resources definitions) that describe how the controller should perform synchronization. In this model, controllers compare the Git repository specified in the CRD with the Kubernetes cluster resources also specified in the CRD and act accordingly based on the results of the comparison. In particular, this GitOps model is used in ArgoCD.
GitOps on the OpenShift platform
Administration of a multi-cluster Kubernetes infrastructure
With the spread of Kubernetes and the growing popularity of multi-cloud strategies and edge computing (edge computing), the average number of OpenShift clusters per customer is also increasing.
For example, when using edge computing, clusters of one customer can be deployed in hundreds and even thousands. As a result, he is forced to manage multiple independent or consistent OpenShift clusters in the public cloud and on-premise.
At the same time, a lot of problems have to be solved, in particular:
- Control that the clusters are in an identical state (configs, monitoring, storage, etc.)
- Recreate (or restore) clusters from a known state.
- Create new clusters based on known state.
- Roll changes across multiple OpenShift clusters.
- Rollback changes on multiple OpenShift clusters.
- Associate templated configurations with different environments.
Application Configurations
During their life cycle, applications often go through a chain of clusters (dev, stage, etc.) before reaching the production cluster. In addition, due to availability and scalability requirements, customers often deploy applications across multiple on-premise clusters or across multiple public cloud platform regions.
In doing so, the following tasks have to be solved:
- Ensure the movement of applications (binaries, configs, etc.) between clusters (dev, stage, etc.).
- Roll changes in applications (binaries, configs, etc.) in several OpenShift clusters.
- Roll back changes to applications to a previous known state.
Scenarios for using OpenShift GitOps
1. Apply changes from a Git repository
The cluster administrator can store OpenShift cluster configurations in a Git repository and automatically apply them to effortlessly create new clusters and bring them into a state identical to the known state stored in the Git repository.
2. Synchronization with Secret Manager
The admin will also benefit from the ability to synchronize OpenShift secrets with appropriate software such as Vault to manage them using tools specially created for this.
3. Configuration drift control
The admin will only be in favor if OpenShift GitOps itself detects and warns about discrepancies between real configurations and those set in the repository so that you can quickly respond to drift.
4. Configuration Drift Notifications
Useful in the case when the administrator wants to quickly learn about cases of configuration drift in order to quickly take appropriate measures on their own.
5. Manual synchronization of configurations when drifting
Allows an admin to synchronize an OpenShift cluster with a Git repository in the event of a configuration drift to quickly revert the cluster to a previous known state.
6. Auto-sync configurations when drifting
The admin can also configure the OpenShift cluster to automatically synchronize with the repository when a drift is detected, so that the cluster configuration always matches the configs in Git.
7. Multiple clusters - one repository
The admin can store configurations of several different OpenShift clusters in one Git repository at once and selectively apply them as needed.
8. Hierarchy of cluster configurations (inheritance)
The admin can set a hierarchy of cluster configurations in the repository (stage, prod, app portfolio, etc. with inheritance). In other words, it can determine whether configurations should be applied to one or more clusters.
For example, if the admin sets the hierarchy in the Git repository to “Production clusters (prod) → System X clusters → System X production clusters”, then the following configs are combined for the production clusters of system X:
- Configs common to all production clusters.
- Configs for the X system cluster.
- Configs for the production cluster of the X system.
9. Templates and configuration overrides
The admin can override the set of inherited configs and their values, for example, to fine-tune the configuration for certain clusters to which they will apply.
10. Selective include's and exclude's for configurations, application configurations
The administrator can set the conditions for applying or not applying certain configurations to clusters with certain characteristics.
11. Template support
Developers will benefit from the ability to choose how application resources are defined (Helm Chart, pure Kubernetes yaml, etc.) to use the most appropriate format for each particular application.
GitOps tools on the OpenShift platform
ArgoCD
ArgoCD implements the External Resource Reconcile model and offers a centralized UI for one-to-many orchestration of relationships between clusters and Git repositories. The disadvantages of this program include the inability to manage applications when ArgoCD is not working.
Flux
Flux implements the On-Cluster Resource Reconcile model, and as a result, there is no centralized management of the definition repository, which is a weakness. On the other hand, precisely because of the lack of centralization, the ability to manage applications is preserved even if one cluster fails.
Installing ArgoCD on OpenShift
ArgoCD offers a great command line interface and web console, so we won't cover Flux and other alternatives here.
To deploy ArgoCD on the OpenShift 4 platform, follow these steps as a cluster administrator:
Deploying ArgoCD components on the OpenShift platform
# Create a new namespace for ArgoCD components
oc create namespace argocd
# Apply the ArgoCD Install Manifest
oc -n argocd apply -f https://raw.githubusercontent.com/argoproj/argo-cd/v1.2.2/manifests/install.yaml
# Get the ArgoCD Server password
ARGOCD_SERVER_PASSWORD=$(oc -n argocd get pod -l "app.kubernetes.io/name=argocd-server" -o jsonpath='{.items[*].metadata.name}')Modifying ArgoCD Server to be seen by OpenShift Route
# Patch ArgoCD Server so no TLS is configured on the server (--insecure)
PATCH='{"spec":{"template":{"spec":{"$setElementOrder/containers":[{"name":"argocd-server"}],"containers":[{"command":["argocd-server","--insecure","--staticassets","/shared/app"],"name":"argocd-server"}]}}}}'
oc -n argocd patch deployment argocd-server -p $PATCH
# Expose the ArgoCD Server using an Edge OpenShift Route so TLS is used for incoming connections
oc -n argocd create route edge argocd-server --service=argocd-server --port=http --insecure-policy=RedirectArgoCD Cli Tool Deployment
# Download the argocd binary, place it under /usr/local/bin and give it execution permissions
curl -L https://github.com/argoproj/argo-cd/releases/download/v1.2.2/argocd-linux-amd64 -o /usr/local/bin/argocd
chmod +x /usr/local/bin/argocdChanging ArgoCD Server admin password
# Get ArgoCD Server Route Hostname
ARGOCD_ROUTE=$(oc -n argocd get route argocd-server -o jsonpath='{.spec.host}')
# Login with the current admin password
argocd --insecure --grpc-web login ${ARGOCD_ROUTE}:443 --username admin --password ${ARGOCD_SERVER_PASSWORD}
# Update admin's password
argocd --insecure --grpc-web --server ${ARGOCD_ROUTE}:443 account update-password --current-password ${ARGOCD_SERVER_PASSWORD} --new-password
After completing these steps, ArgoCD Server can be operated through the ArgoCD WebUI web console or the ArgoCD Cli command line toolkit.
GitOps - it's never too late
“The train has left” - this is how they say about a situation when the opportunity to do something is missed. In the case of OpenShift, the desire to immediately start using this new cool platform often creates just such a situation with the management and maintenance of routes, deployments and other OpenShift objects. But is the chance always completely missed?
Continuing the series of articles on , today we'll show you how to convert a hand-crafted application and its resources into a process where everything is managed by the GitOps toolkit. To do this, we will first manually deploy the httpd application. The screenshot below shows how we create a namespace, a deployment, and a service, and then expose that service to create a route.
oc create -f https://raw.githubusercontent.com/openshift/federation-dev/master/labs/lab-4-assets/namespace.yaml
oc create -f https://raw.githubusercontent.com/openshift/federation-dev/master/labs/lab-4-assets/deployment.yaml
oc create -f https://raw.githubusercontent.com/openshift/federation-dev/master/labs/lab-4-assets/service.yaml
oc expose svc/httpd -n simple-appSo, we have a manually created application. Now it needs to be brought under GitOps control without loss of availability. Briefly, it does this:
- Create a Git repository for the code.
- We export our current objects and upload them to the Git repository.
- Choosing and deploying the GitOps toolkit.
- We add our repository to this toolkit.
- Define an application in our GitOps toolkit.
- We perform a trial run of the application using the GitOps toolkit.
- Synchronizing objects using the GitOps toolkit.
- We include cleaning (pruning) and auto-synchronization of objects.
As mentioned in the previous , GitOps has one and only one source of information about all the objects in the Kubernetes cluster(s) - the Git repository. Further, we proceed from the premise that you already have a Git repository in use in your organization. It can be public or private, but it must be available to Kubernetes clusters. This can be the same repository as for application code, or it can be a separate repository created specifically for deployments. It is recommended to have strict permissions in the repository, as secrets, routes, and other security-sensitive things will be stored there.
In our example, we will create a new public repository on GitHub. You can name it whatever you like, we use the name blogpost.
If the object YAML files were not stored locally or in Git, then you will have to use the oc or kubectl binaries. In the screenshot below, we are requesting YAML for our namespace, deployment, service, and route. Before that, we cloned the newly created repository and moved into it with the cd command.
oc get namespace simple-app -o yaml --export > namespace.yaml
oc get deployment httpd -o yaml -n simple-app --export > deployment.yaml
oc get service httpd -o yaml -n simple-app --export > service.yaml
oc get route httpd -o yaml -n simple-app --export > route.yamlNow let's fix the deployment.yaml file to remove the field that Argo CD can't sync.
sed -i '/sgeneration: .*/d' deployment.yamlIn addition, you need to change the route. We'll first define a multi-line variable, and then we'll replace ingress: null with the contents of that variable.
export ROUTE=" ingress:
- conditions:
- status: 'True'
type: Admitted"
sed -i "s/ ingress: null/$ROUTE/g" route.yamlSo, we figured out the files, it remains to save them to the Git repository. After that, this repository becomes the one and only source of information, and any manual changes to objects should be strictly prohibited.
git commit -am ‘initial commit of objects’
git push origin masterFurther, we proceed from the fact that you have already deployed ArgoCD (how to do this, see the previous ). Therefore, we will add the repository we created to Argo CD, which contains the application code from our example. Just make sure you point to the exact repository you created earlier.
argocd repo add https://github.com/cooktheryan/blogpostNow let's create an application. The application sets the values so that the GitOps toolkit understands which repository and paths to use, which OpenShift is needed to manage objects, and which specific branch of the repository is needed, and whether resources should be auto-synced.
argocd app create --project default
--name simple-app --repo https://github.com/cooktheryan/blogpost.git
--path . --dest-server https://kubernetes.default.svc
--dest-namespace simple-app --revision master --sync-policy none
After the application is defined in Argo CD, this toolkit starts checking already deployed objects against the definitions in the repository. In our example, auto-sync and cleanup are disabled, so items don't change yet. Please note that in the Argo CD interface, our application will have the status "Out of Sync" (Not synchronized), since there is no label that ArgoCD puts down.
This is why when we run synchronization a little later, the objects will not be redeployed.
Now let's do a test run to make sure there are no errors in our files.
argocd app sync simple-app --dry-runIf there are no errors, then you can proceed to synchronization.
argocd app sync simple-appAfter executing the argocd get command on our application, we should see that the application status has changed to Healthy (Healthy) or Synced (Synchronized). This will mean that all resources in the Git repository now match the resources already deployed.
argocd app get simple-app
Name: simple-app
Project: default
Server: https://kubernetes.default.svc
Namespace: simple-app
URL: https://argocd-server-route-argocd.apps.example.com/applications/simple-app
Repo: https://github.com/cooktheryan/blogpost.git
Target: master
Path: .
Sync Policy: <none>
Sync Status: Synced to master (60e1678)
Health Status: Healthy
... Now, you can turn on auto-sync and cleanup to ensure that nothing is manually created and that every time an object is created or updated to the repository, a deployment will take place.
argocd app set simple-app --sync-policy automated --auto-prune
So, we have successfully moved an application under GitOps control that did not originally use GitOps in any way.
Source: habr.com