ProHoster > Blog > Administration > Mastering the deployment task in GKE without plugins, SMS and registration. With one eye we look under Jenkins' jacket

Mastering the deployment task in GKE without plugins, SMS and registration. With one eye we look under Jenkins' jacket

It all started with the fact that the team leader of one of our development teams asked to put their new application out in test mode, which had been containerized the day before. I posted. After about 20 minutes, a request was received to update the application, because a very necessary thing was finished there. I renewed. A couple more hours later… well, you can already guess what happened next…

I must admit that I am rather lazy (did I admit this earlier? no?), and, given the fact that team leads have access to Jenkins, in which we have all CI / CD, I thought: yes, let him deploy as much as he pleases ! I remembered a joke: give a man a fish and he will eat for a day; call a man Sat and he will be Sat all his life. And went to make a job, which would be able to deploy into a container with the application of any successfully built version and pass any values ​​​​to it EPS (my grandfather, a philologist, an English teacher in the past, would now twist his finger at his temple and look at me very expressively after reading this sentence).

So, in a note, I will talk about how I learned:

  1. Dynamically update jobs in Jenkins from the job itself or from other jobs;
  2. Connect to the cloud console (Cloud shell) from the node with the Jenkins agent installed;
  3. Deploy workload to Google Kubernetes Engine.


In fact, of course, I'm somewhat cunning. It is assumed that you have at least part of the infrastructure in the Google cloud, and therefore you are its user and, of course, you have a GCP account. But this note is not about that.

This is my next cheat sheet. I want to write such notes only in one case: I had a task in front of me, I initially didn’t know how to solve it, the solution didn’t google in finished form, so I googled it in parts and eventually solved the problem. And so that in the future, when I forget how I did it, I don’t have to google everything piece by piece and compile it all together again, I write myself such cheat sheets.

Disclaimer: 1. The note was written β€œfor myself”, for the role best practice does not apply. I am happy to read the options β€œit was better to do so” in the comments.
2. If the applied part of the note is considered salt, then, like all my previous notes, this one is a low-salt solution.

Dynamic update of job settings in Jenkins

I foresee your question: what does a dynamic job update have to do with it? Entered the value of the string parameter with pens and go!

I answer: I’m really lazy, I don’t like it when they complain: Misha, the deployment is crashing, everything is gone! You start looking, and there is a typo in the value of some task launch parameter. Therefore, I prefer to do everything as fully as possible. If it is possible to deprive the user of the ability to enter data directly, giving instead a list of values ​​to choose from, then I organize the choice.

The plan is as follows: we create a job in Jenkins, in which, before starting, we could select a version from the list, specify values ​​for the parameters passed to the container via EPS, then it collects the container and pushes it to the Container Registry. Further from there, the container is launched in the cuber as workloads with the parameters specified in the job.

We will not consider the process of creating and configuring a job in Jenkins, this is offtopic. We will assume that the task is ready. To implement an updatable list with versions, we need two things: an already existing source list with a priori valid version numbers, and a type variable Choice parameter in the task. In our example, let the variable be named BUILD_VERSION, we will not dwell on it in detail. But let's take a closer look at the source list.

There are not so many options. Two immediately came to my mind:

  • Use the Remote access API that Jenkins offers to its users;
  • Request the contents of a remote repository folder (in our case, this is JFrog Artifactory, which is not important).

Jenkins Remote Access API

According to the established fine tradition, I prefer to avoid lengthy explanations.
I will only allow myself a free translation of a piece of the first paragraph the first page of the API documentation:

Jenkins provides an API for remote machine-readable access to its functionality. <…> Remote access is offered in a REST-like style. This means that there is no single entry point to all capabilities, but instead a URL like ".../api/", Where "..." means the object to which the API capabilities apply.

In other words, if the deployment task we are currently talking about is available at http://jenkins.mybuild.er/view/AweSomeApp/job/AweSomeApp_build, then the API whistles for this task are available at http://jenkins.mybuild.er/view/AweSomeApp/job/AweSomeApp_build/api/

Next, we have a choice in what form to receive the output. Let's focus on XML, since the API only allows you to use filtering in this case.

Let's just try to get a list of all job runs. We are only interested in the assembly name (displayName) and its result (result):

http://jenkins.mybuild.er/view/AweSomeApp/job/AweSomeApp_build/api/xml?tree=allBuilds[displayName,result]

Got it?

Now let's filter out only those launches that end up with the result SUCCESS. Using the Argument &exclude and as a parameter we pass to it the path to a value not equal to SUCCESS. Yes Yes. A double negative is an affirmation. We exclude everything that does not interest us:

http://jenkins.mybuild.er/view/AweSomeApp/job/AweSomeApp_build/api/xml?tree=allBuilds[displayName,result]&exclude=freeStyleProject/allBuild[result!='SUCCESS']

Screenshot of the success list
Mastering the deployment task in GKE without plugins, SMS and registration. With one eye we look under Jenkins' jacket

Well, just for pampering, we will make sure that the filter did not deceive us (filters never lie!) and display a list of β€œnot-successful” ones:

http://jenkins.mybuild.er/view/AweSomeApp/job/AweSomeApp_build/api/xml?tree=allBuilds[displayName,result]&exclude=freeStyleProject/allBuild[result='SUCCESS']

Screenshot of the list of non-successful
Mastering the deployment task in GKE without plugins, SMS and registration. With one eye we look under Jenkins' jacket

List of versions from a folder on a remote server

There is a second way to get a list of versions. I like it even more than calling the Jenkins API. Well, because if the application was successfully built, then it was packaged and put into the repository in the appropriate folder. Like, the repository is the default repository of working versions of applications. Type. Well, let's ask him what versions are in storage. We will curl, grep and awk the remote folder. If someone is interested in the liner, then it is under the spoiler.

One line command
Pay attention to two things: I pass connection details in the header and I don’t need all the versions from the folder directly, and I select only those that were created within a month. Edit the command according to your realities and needs:

curl -H "X-JFrog-Art-Api:VeryLongAPIKey" -s http://arts.myre.po/artifactory/awesomeapp/ | sed 's/a href=//' | grep "$(date +%b)-$(date +%Y)|$(date +%b --date='-1 month')-$(date +%Y)" | awk '{print $1}' | grep -oP '>K[^/]+' )

Job setup and job configuration file in Jenkins

With the source of the list of versions sorted out. Now let's turn the resulting list into a task. For me, the obvious solution was to add a step in the app build job. The step that would be executed if the result was "success".

Open the build job settings and scroll to the bottom. We click on the buttons: Add build step -> Conditional step (single). In the step settings, select the condition current build status, set the value SUCCESSThe action to take if successful Run shell command.

And now the most interesting. Jenkins stores job configurations in files. In XML format. Along the way http://ΠΏΡƒΡ‚ΡŒ-Π΄ΠΎ-задания/config.xml Accordingly, you can download the configuration file, edit it as needed and put it in the place where you got it from.

Remember, above we agreed that for the list of versions we will create a parameter BUILD_VERSION?

Let's download the configuration file and take a look inside it. Just to make sure that the parameter is in place and really the right kind.

Screenshot under the spoiler.

Your config.xml fragment should look the same. Except that the content of the choices element is currently missing
Mastering the deployment task in GKE without plugins, SMS and registration. With one eye we look under Jenkins' jacket

Convinced? That's all, we write a script that will be executed in case of a successful build.
The script will receive a list of versions, download the configuration file, write a list of versions into it in the place we need, and then put it back. Yes. All right. Write a list of versions in XML to the place where there is already a list of versions (it will be in the future, after the first run of the script). I know there are still die-hard fans of regular expressions in the world. I do not belong to them. Please install xmlstarler on the machine where the config will be edited. I don't think it's that much of a price to pay to avoid editing XML with sed.

Under the spoiler, I give the code that performs the entire sequence described above.

We write to the config a list of versions from a folder on a remote server

#!/bin/bash
############## Π‘ΠΊΠ°Ρ‡ΠΈΠ²Π°Π΅ΠΌ ΠΊΠΎΠ½Ρ„ΠΈΠ³
curl -X GET -u username:apiKey http://jenkins.mybuild.er/view/AweSomeApp/job/AweSomeApp_k8s/config.xml -o appConfig.xml

############## УдаляСм ΠΈ Π·Π°Π½ΠΎΠ²ΠΎ создаСм xml-элСмСнт для списка вСрсий
xmlstarlet ed --inplace -d '/project/properties/hudson.model.ParametersDefinitionProperty/parameterDefinitions/hudson.model.ChoiceParameterDefinition[name="BUILD_VERSION"]/choices[@class="java.util.Arrays$ArrayList"]/a[@class="string-array"]' appConfig.xml

xmlstarlet ed --inplace --subnode '/project/properties/hudson.model.ParametersDefinitionProperty/parameterDefinitions/hudson.model.ChoiceParameterDefinition[name="BUILD_VERSION"]/choices[@class="java.util.Arrays$ArrayList"]' --type elem -n a appConfig.xml

xmlstarlet ed --inplace --insert '/project/properties/hudson.model.ParametersDefinitionProperty/parameterDefinitions/hudson.model.ChoiceParameterDefinition[name="BUILD_VERSION"]/choices[@class="java.util.Arrays$ArrayList"]/a' --type attr -n class -v string-array appConfig.xml

############## Π§ΠΈΡ‚Π°Π΅ΠΌ Π² массив список вСрсий ΠΈΠ· рСпозитория
readarray -t vers < <( curl -H "X-JFrog-Art-Api:Api:VeryLongAPIKey" -s http://arts.myre.po/artifactory/awesomeapp/ | sed 's/a href=//' | grep "$(date +%b)-$(date +%Y)|$(date +%b --date='-1 month')-$(date +%Y)" | awk '{print $1}' | grep -oP '>K[^/]+' )

############## ПишСм массив элСмСнт Π·Π° элСмСнтом Π² ΠΊΠΎΠ½Ρ„ΠΈΠ³
printf '%sn' "${vers[@]}" | sort -r | 
                while IFS= read -r line
                do
                    xmlstarlet ed --inplace --subnode '/project/properties/hudson.model.ParametersDefinitionProperty/parameterDefinitions/hudson.model.ChoiceParameterDefinition[name="BUILD_VERSION"]/choices[@class="java.util.Arrays$ArrayList"]/a[@class="string-array"]' --type elem -n string -v "$line" appConfig.xml
                done

############## КладСм ΠΊΠΎΠ½Ρ„ΠΈΠ³ Π²Π·Π°Π΄
curl -X POST -u username:apiKey http://jenkins.mybuild.er/view/AweSomeApp/job/AweSomeApp_k8s/config.xml --data-binary @appConfig.xml

############## ΠŸΡ€ΠΈΠ²ΠΎΠ΄ΠΈΠΌ Ρ€Π°Π±ΠΎΡ‡Π΅Π΅ мСсто Π² порядок
rm -f appConfig.xml

If you liked the option of getting versions from Jenkins better and you are as lazy as me, then under the spoiler is the same code, but the list is from Jenkins:

We write a list of versions from Jenkins to the config
Just keep in mind the moment: my assembly name consists of a serial number and a version number, separated by a colon. Accordingly, awk cuts off the unnecessary part. For yourself, change this line to suit your needs.

#!/bin/bash
############## Π‘ΠΊΠ°Ρ‡ΠΈΠ²Π°Π΅ΠΌ ΠΊΠΎΠ½Ρ„ΠΈΠ³
curl -X GET -u username:apiKey http://jenkins.mybuild.er/view/AweSomeApp/job/AweSomeApp_k8s/config.xml -o appConfig.xml

############## УдаляСм ΠΈ Π·Π°Π½ΠΎΠ²ΠΎ создаСм xml-элСмСнт для списка вСрсий
xmlstarlet ed --inplace -d '/project/properties/hudson.model.ParametersDefinitionProperty/parameterDefinitions/hudson.model.ChoiceParameterDefinition[name="BUILD_VERSION"]/choices[@class="java.util.Arrays$ArrayList"]/a[@class="string-array"]' appConfig.xml

xmlstarlet ed --inplace --subnode '/project/properties/hudson.model.ParametersDefinitionProperty/parameterDefinitions/hudson.model.ChoiceParameterDefinition[name="BUILD_VERSION"]/choices[@class="java.util.Arrays$ArrayList"]' --type elem -n a appConfig.xml

xmlstarlet ed --inplace --insert '/project/properties/hudson.model.ParametersDefinitionProperty/parameterDefinitions/hudson.model.ChoiceParameterDefinition[name="BUILD_VERSION"]/choices[@class="java.util.Arrays$ArrayList"]/a' --type attr -n class -v string-array appConfig.xml

############## ПишСм Π² Ρ„Π°ΠΉΠ» список вСрсий ΠΈΠ· Jenkins
curl -g -X GET -u username:apiKey 'http://jenkins.mybuild.er/view/AweSomeApp/job/AweSomeApp_build/api/xml?tree=allBuilds[displayName,result]&exclude=freeStyleProject/allBuild[result!=%22SUCCESS%22]&pretty=true' -o builds.xml

############## Π§ΠΈΡ‚Π°Π΅ΠΌ Π² массив список вСрсий ΠΈΠ· XML
readarray vers < <(xmlstarlet sel -t -v "freeStyleProject/allBuild/displayName" builds.xml | awk -F":" '{print $2}')

############## ПишСм массив элСмСнт Π·Π° элСмСнтом Π² ΠΊΠΎΠ½Ρ„ΠΈΠ³
printf '%sn' "${vers[@]}" | sort -r | 
                while IFS= read -r line
                do
                    xmlstarlet ed --inplace --subnode '/project/properties/hudson.model.ParametersDefinitionProperty/parameterDefinitions/hudson.model.ChoiceParameterDefinition[name="BUILD_VERSION"]/choices[@class="java.util.Arrays$ArrayList"]/a[@class="string-array"]' --type elem -n string -v "$line" appConfig.xml
                done

############## КладСм ΠΊΠΎΠ½Ρ„ΠΈΠ³ Π²Π·Π°Π΄
curl -X POST -u username:apiKey http://jenkins.mybuild.er/view/AweSomeApp/job/AweSomeApp_k8s/config.xml --data-binary @appConfig.xml

############## ΠŸΡ€ΠΈΠ²ΠΎΠ΄ΠΈΠΌ Ρ€Π°Π±ΠΎΡ‡Π΅Π΅ мСсто Π² порядок
rm -f appConfig.xml

In theory, if you tested the code written based on the examples above, then in the deployment task you should already have a drop-down list with versions. It's like in the screenshot under the spoiler.

Correctly completed list of versions
Mastering the deployment task in GKE without plugins, SMS and registration. With one eye we look under Jenkins' jacket

If everything worked, then copy-paste the script in Run shell command and save your changes.

Connecting to the cloud shell

We have pickers in containers. We use Ansible as our application delivery and configuration manager. Accordingly, when it comes to building containers, three options come to mind: install Docker in Docker, install Docker on a machine with Ansible, or build containers in the cloud console. We agreed to keep silent about plugins for Jenkins in this article. Remember?

I decided: well, since the containers "out of the box" can be collected in the cloud console, then why fence the garden? Keep it clean, right? I want to collect containers with Jenkins in the cloud console, and then shoot them into the Kuber from there. Moreover, Google has very fat channels inside the infrastructure, which will favorably affect the deployment speed.

Two things are required to connect to the cloud console: gcloud and access rights to Google Cloud API for the instance of the VM with which this very connection will be made.

For those who plan to connect at all from a non-Google cloud
Google allows you to disable interactive authorization in its services. This will allow you to connect to the console even from the coffee machine, if it is under *nix's and it has a console itself.

If there is a need for me to cover this issue in more detail within the framework of this note, write in the comments. If I get enough votes, I'll write an update on this topic.

The easiest way to grant rights is through the web interface.

  1. Stop the VM instance from which you will later connect to the cloud console.
  2. Open Instance Details and click Change.
  3. At the very bottom of the page, select the instance access scope Full access to all Cloud APIs.

    Screenshot
    Mastering the deployment task in GKE without plugins, SMS and registration. With one eye we look under Jenkins' jacket

  4. Save your changes and start the instance.

Once the VM has finished booting, connect to it via SSH and make sure that the connection is successful. Use the command:

gcloud alpha cloud-shell ssh

A successful connection looks something like this
Mastering the deployment task in GKE without plugins, SMS and registration. With one eye we look under Jenkins' jacket

Deploy to GKE

Since we are striving in every possible way to completely switch to IaC (Infrastucture as a Code), we store docker files in the git. This is on the one hand. And the deployment in kubernetes is described by a yaml file that is used only by this task, which in itself is also like a code. It's on the other side. Basically, my plan is:

  1. We take the values ​​of variables BUILD_VERSION and, optionally, the values ​​of the variables that will be passed through EPS.
  2. Download dockerfile from git.
  3. We generate yaml for deployment.
  4. Upload both of these files via scp to the cloud console.
  5. We build a container there and push it to the Container registry
  6. We apply the load deployment file to Kuber.

Let's be more specific. Once talking about EPS, then suppose we need to pass the values ​​of two parameters: PARAM1 ΠΈ PARAM2. Add their deployment task, type - String Parameter.

Screenshot
Mastering the deployment task in GKE without plugins, SMS and registration. With one eye we look under Jenkins' jacket

We will generate yaml with a simple redirect threw out to file. It is assumed, of course, that in your dockerfile you have PARAM1 ΠΈ PARAM2that the name of the load will be awesome app, and the assembled container with the application of the specified version lies in container registry along the way gcr.io/awesomeapp/awesomeapp-$BUILD_VERSIONWhere $BUILD_VERSION just selected from the dropdown list.

Command listing

touch deploy.yaml
echo "apiVersion: apps/v1" >> deploy.yaml
echo "kind: Deployment" >> deploy.yaml
echo "metadata:" >> deploy.yaml
echo "  name: awesomeapp" >> deploy.yaml
echo "spec:" >> deploy.yaml
echo "  replicas: 1" >> deploy.yaml
echo "  selector:" >> deploy.yaml
echo "    matchLabels:" >> deploy.yaml
echo "      run: awesomeapp" >> deploy.yaml
echo "  template:" >> deploy.yaml
echo "    metadata:" >> deploy.yaml
echo "      labels:" >> deploy.yaml
echo "        run: awesomeapp" >> deploy.yaml
echo "    spec:" >> deploy.yaml
echo "      containers:" >> deploy.yaml
echo "      - name: awesomeapp" >> deploy.yaml
echo "        image: gcr.io/awesomeapp/awesomeapp-$BUILD_VERSION:latest" >> deploy.yaml
echo "        env:" >> deploy.yaml
echo "        - name: PARAM1" >> deploy.yaml
echo "          value: $PARAM1" >> deploy.yaml
echo "        - name: PARAM2" >> deploy.yaml
echo "          value: $PARAM2" >> deploy.yaml

Jenkins agent after connecting with gcloud alpha cloud-shell ssh interactive mode is not available, so we pass commands to the cloud console using the parameter --command.

We clean the home folder in the cloud console from the old dockerfile:

gcloud alpha cloud-shell ssh --command="rm -f Dockerfile"

We put the freshly downloaded dockerfile in the home folder of the cloud console using scp:

gcloud alpha cloud-shell scp localhost:./Dockerfile cloudshell:~

We collect, tag and push the container in the Container registry:

gcloud alpha cloud-shell ssh --command="docker build -t awesomeapp-$BUILD_VERSION ./ --build-arg BUILD_VERSION=$BUILD_VERSION --no-cache"
gcloud alpha cloud-shell ssh --command="docker tag awesomeapp-$BUILD_VERSION gcr.io/awesomeapp/awesomeapp-$BUILD_VERSION"
gcloud alpha cloud-shell ssh --command="docker push gcr.io/awesomeapp/awesomeapp-$BUILD_VERSION"

We do the same with the deployment file. Please note that the commands below use fictitious names of the cluster where the deployment takes place (awsm cluster) and project name (awesome-project) where the cluster is located.

gcloud alpha cloud-shell ssh --command="rm -f deploy.yaml"
gcloud alpha cloud-shell scp localhost:./deploy.yaml cloudshell:~
gcloud alpha cloud-shell ssh --command="gcloud container clusters get-credentials awsm-cluster --zone us-central1-c --project awesome-project && 
kubectl apply -f deploy.yaml"

We run the job, open the console output and hope to see a successful build of the container.

Screenshot
Mastering the deployment task in GKE without plugins, SMS and registration. With one eye we look under Jenkins' jacket

And then a successful deployment of the assembled container

Screenshot
Mastering the deployment task in GKE without plugins, SMS and registration. With one eye we look under Jenkins' jacket

I deliberately overlooked the setting income. For one simple reason: once you set it to workloads with the given name, it will remain operational, no matter how many deployments with this name are made. Well, in general, this is a bit beyond the scope of the story.

Instead of conclusions

All the above steps, probably, could not be done, but simply install some plugin for Jenkins, their muuulon. But for some reason I don't like plugins. Well, more precisely, I resort to them only out of desperation.

And I just like to pick up some new topic for me. The text above is also a way to share the findings that I made while solving the problem described at the very beginning. Share with those who, like, are not at all a dire wolf in devops. If my findings help at least someone, I will be happy.

Source: habr.com

Add a comment