I often have to build a pipeline for Java project builds. Sometimes it's open-source, sometimes not. Recently, I decided to try moving some of my repositories from Travis-CI and TeamCity to GitHub Actions, and here's what came out of it.
What Are We Going to Automate
First, we need a project to automate; let’s create a small application using Spring Boot / Java 11 / Maven. In this article, we won't focus on the application's logic at all; the infrastructure around the application is what matters, so a simple REST API controller will suffice.
You can check out the source code here: all stages of the pipeline construction are reflected in the pull requests of this project.
JIRA and Planning
It's worth mentioning that we usually use JIRA as our task tracker, so let’s create a separate board for this project and jot down the initial tasks:
Later, we will return to what interesting synergies JIRA and GitHub can offer.
Automating the Project Build
Our test project is built using Maven, so its build is pretty straightforward; all we need is mvn clean package.
To do this with GitHub Actions, we will need to create a file in the repository describing our workflow. This can be done with a regular yml file. I can't say I enjoy 'programming in yml', but it is what it is — we will create a file named build.yml in the .github/workflows directory to describe the actions for building the master branch:
name: Build
on:
pull_request:
branches:
- '*'
push:
branches:
- 'master'
jobs:
build:
runs-on: ubuntu-18.04
steps:
- uses: actions/checkout@v1
- name: set up JDK 11
uses: actions/setup-java@v1
with:
java-version: 1.11
- name: Maven Package
run: mvn -B clean package -DskipTests
on — this describes the event that will trigger our script.
on: pull_request / push — indicates that this workflow should run on every push to the master branch and on pull request creations.
Next is the description of jobs (jobs) and steps of execution (steps) for each job.
runs-on — here we can choose the target OS; surprisingly, you can even select Mac OS, but using it on private repositories is quite an expensive affair (compared to Linux).
uses allows reusing other actions; for example, by using the actions/setup-java action, we set up the environment for Java 11.
Using with We can specify the parameters with which we run the action, essentially the arguments that will be passed to the action.
Now we just need to run the Maven build of the project: run: mvn -B clean package flag -B indicates that we need non-interactive mode, so Maven doesn't suddenly want to ask us something
Great! Now, with every commit to master, the project build is triggered.
Let's automate the test execution.
Building is good, but in reality, the project can be built successfully and still not work. Therefore, the next step is to focus on automating test runs. It’s also quite convenient to view test pass results when reviewing a PR — you’ll know for sure that the tests pass and no one forgot to run their branch before merging.
We’ll run tests on pull request creation and when merging into master, and at the same time, we’ll add a report generation for code coverage.
name: Build
on:
pull_request:
branches:
- '*'
push:
branches:
- 'master'
jobs:
build:
runs-on: ubuntu-18.04
steps:
- uses: actions/checkout@v1
- name: set up JDK 11
uses: actions/setup-java@v1
with:
java-version: 1.11
- name: Maven Verify
run: mvn -B clean verify
- name: Test Coverage
uses: codecov/codecov-action@v1
with:
token: ${{ secrets.CODECOV_TOKEN }}
For test coverage, I use Codecov in conjunction with the Jacoco plugin. Codecov has its own action, but it needs a token to work with our pull request:
${{ secrets.CODECOV_TOKEN }} — we will encounter this structure many more times, secrets is a mechanism for storing secrets in GitHub, where we can specify passwords/tokens/hosts/URLs and other data that shouldn't be exposed in the codebase of the repository.
You can add a variable to secrets in the repository settings on GitHub:
You can obtain a token at after logging in via GitHub; to add a public project, you simply need to follow a link in the format: /[repo name]. Приватный репозиторий тоже можно добавить, для этого надо дать права codecov приложению в гитхабе.
Add the Jacoco plugin to the POM file:
org.jacoco
jacoco-maven-plugin
0.8.4
prepare-agent
report
test
report
org.apache.maven.plugins
maven-surefire-plugin
2.22.2
plain
**/*Test*.java
**/*IT*.java
Now, a codecov bot will enter each of our pull requests and add a coverage change graph:
Let's add a static analyzer
In most of my open-source projects, I use SonarCloud for static code analysis, which is pretty easy to connect to Travis CI. So, it’s logical to do the same when migrating to GitHub Actions. The Actions marketplace is a cool thing, but this time it let me down a bit because I habitually found the necessary action and added it to the workflow. It turned out that Sonar does not support working through an action for analyzing Maven or Gradle projects. Of course, it’s written in the documentation, but who reads that?!
Since we can't do it through an action, we will use the Maven plugin instead:
name: SonarCloud
on:
push:
branches:
- master
pull_request:
types: [opened, synchronize, reopened]
jobs:
sonarcloud:
runs-on: ubuntu-16.04
steps:
- uses: actions/checkout@v1
- name: Set up JDK
uses: actions/setup-java@v1
with:
java-version: 1.11
- name: Analyze with SonarCloud
# set environment variables:
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
# run sonar maven plugin:
run: mvn -B verify sonar:sonar -Dsonar.projectKey=antkorwin_github-actions -Dsonar.organization=antkorwin-github -Dsonar.host.url=https://sonarcloud.io -Dsonar.login=$SONAR_TOKEN -Dsonar.coverage.jacoco.xmlReportPaths=./target/site/jacoco/jacoco.xml
SONAR_TOKEN — can be obtained at and should be set in secrets. GITHUB_TOKEN — is a built-in token generated by GitHub, which allows sonarcloud[bot] to authenticate in Git, so it can leave comments in our pull requests.
Dsonar.projectKey — is the project name in Sonar, which can be found in the project settings.
Dsonar.organization — is the organization name from GitHub.
We create a pull request and wait for the sonarcloud[bot] to come with comments:
Release management
We have configured the build, run the tests, and now we can create a release. Let's see how GitHub Actions significantly simplifies release management.
At work, I have projects whose codebase is stored in Bitbucket (just like in that story 'during the day I write in Bitbucket, at night I commit to GitHub'). Unfortunately, Bitbucket lacks built-in tools for managing releases. This is a problem because for each release, I have to manually create a page in Confluence and put all the features included in the release there, sift through the depths of my mind, tasks in Jira, and commits in the repository. There are many chances to make mistakes; something might be forgotten, or I might write something that was already released last time, and sometimes it's just unclear how to categorize a pull request — is it a feature, a bug fix, a test adjustment, or something infrastructure-related.
How can GitHub Actions help us? There's an excellent action — Release Drafter, which allows you to set up a template for the release notes file to categorize pull requests and automatically group them in the release notes file:
Example template for configuring the report (.github/release-drafter.yml):
name-template: 'v$NEXT_PATCH_VERSION'
tag-template: 'v$NEXT_PATCH_VERSION'
categories:
- title: ' New Features'
labels:
- 'type:features'
# This category collects all PRs with the label type:features
- title: ' Bugs Fixes'
labels:
- 'type:fix'
# Similarly for the label type:fix etc.
- title: ' Documentation'
labels:
- 'type:documentation'
- title: ' Configuration'
labels:
- 'type:config'
change-template: '- $TITLE @$AUTHOR (#$NUMBER)'
template: |
## Changes
$CHANGES
Add a script to generate the draft release (.github/workflows/release-draft.yml):
name: "Create draft release"
on:
push:
branches:
- master
jobs:
update_draft_release:
runs-on: ubuntu-18.04
steps:
- uses: release-drafter/release-drafter@v5
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
From this moment on, all pull requests will automatically be gathered into the release notes — magic!
Here a question may arise: what if the developers forget to assign labels to the PR? Then it’s unclear to which category it belongs, and again I will have to figure it out manually, with each PR separately. To solve this problem, we can use another action — Label Verifier — which checks for the presence of tags on the pull request. If none of the required tags are present, the check will fail, and we will see a message about this in our pull request.
name: "Verify type labels"
on:
pull_request:
types: [opened, labeled, unlabeled, synchronize]
jobs:
triage:
runs-on: ubuntu-18.04
steps:
- uses: zwaldowski/match-label-action@v2
with:
allowed: 'type:fix, type:features, type:documentation, type:tests, type:config'
Now any pull request needs to be tagged with one of the tags: type:fix, type:features, type:documentation, type:tests, type:config.
Auto-annotating pull requests
Since we've touched on the topic of effective work with pull requests, it’s worth mentioning the action called labeler, which assigns labels in PRs based on the files that have been changed. For example, we can label any pull request with [build] if it has changes in the directory .github/workflow.
It's quite simple to connect it:
name: "Auto-assign themes to PR"
on:
- pull_request
jobs:
triage:
runs-on: ubuntu-18.04
steps:
- uses: actions/labeler@v2
with:
repo-token: ${{ secrets.GITHUB_TOKEN }}
We will also need a file that describes the correspondence between project directories and PR themes:
theme:build:
- ".github/**"
- "pom.xml"
- ".travis.yml"
- ".gitignore"
- "Dockerfile"
theme:code:
- "src/main/*"
theme:tests:
- "src/test/*"
theme:documentation:
- "docs/**"
theme:TRASH:
- ".idea/**"
- "target/**"
I couldn't get the action that automatically labels pull requests to work well with the action that checks for required labels, match-label refuses to recognize the labels assigned by the bot. It seems easier to write my own action that combines both stages. But even in this form, it’s quite convenient, as you just need to select a label from the list when creating a pull request.
It's time to deploy
I tried several deployment options through GitHub Actions (via ssh, via scp, and using docker-hub), and I can say that you will most likely find a way to upload a binary to the server, no matter how convoluted your pipeline may be.
I liked the idea of keeping all infrastructure in one place, so let's consider how to deploy to GitHub Packages (this is a repository for binary content, npm, jar, docker).
Script for building a docker image and publishing it to GitHub Packages:
name: Deploy docker image
on:
push:
branches:
- 'master'
jobs:
build_docker_image:
runs-on: ubuntu-18.04
steps:
# Build JAR:
- uses: actions/checkout@v1
- name: set up JDK 11
uses: actions/setup-java@v1
with:
java-version: 1.11
- name: Maven Package
run: mvn -B clean compile package -DskipTests
# Set global environment variables:
- name: set global env
id: global_env
run: |
echo "::set-output name=IMAGE_NAME::${GITHUB_REPOSITORY#*/}"
echo "::set-output name=DOCKERHUB_IMAGE_NAME::docker.pkg.github.com/${GITHUB_REPOSITORY}/${GITHUB_REPOSITORY#*/}"
# Build Docker image:
- name: Build and tag image
run: |
docker build -t "${{ steps.global_env.outputs.DOCKERHUB_IMAGE_NAME }}:latest" -t "${{ steps.global_env.outputs.DOCKERHUB_IMAGE_NAME }}:${GITHUB_SHA::8}" .
- name: Docker login
run: docker login docker.pkg.github.com -u $GITHUB_ACTOR -p ${{secrets.GITHUB_TOKEN}}
# Publish image to github package repository:
- name: Publish image
env:
IMAGE_NAME: $GITHUB_REPOSITORY
run: docker push "docker.pkg.github.com/$GITHUB_REPOSITORY/${{ steps.global_env.outputs.IMAGE_NAME }}"
First, we need to build the JAR file of our application, after which we determine the path to the GitHub docker registry and the name of our image. There are a few tricks here that we haven’t encountered yet:
- The construct echo «::set-output name=NAME::VALUE» allows you to set a variable value in the current step so that it can later be read in all other steps.
- You can obtain the value of a variable set in the previous step using that step's identifier: ${{ steps.global_env.outputs.DOCKERHUB_IMAGE_NAME }}
- The standard variable GITHUB_REPOSITORY contains the repository name and its owner ("owner/repo-name"). To extract everything from this string except for the repository name, we will use bash syntax: ${GITHUB_REPOSITORY#*/}
Next, we need to build the Docker image:
docker build -t "docker.pkg.github.com/antkorwin/github-actions/github-actions:latest"
Log in to the registry:
docker login docker.pkg.github.com -u $GITHUB_ACTOR -p ${{secrets.GITHUB_TOKEN}}
And publish the image to the GitHub Packages Repository:
docker push "docker.pkg.github.com/antkorwin/github-actions/github-actions"
To specify the image version, we use the first digits from the commit SHA hash — GITHUB_SHA also has nuances here; if you are building in response to events other than merging into master, such as pull request creation, the SHA may not match the hash seen in git history because the actions/checkout does a unique hash to avoid mutual blocking of actions in PR.
If everything went well, opening the packages section (https://github.com/antkorwin/github-actions/packages) in the repository, you will see the new Docker image:
You can also view the list of Docker image versions there.
All that remains is to configure our server to work with this registry and restart the service. I'll discuss how to do this with systemd another time.
Monitoring
Let's look at a simple way to perform a health check of our application using GitHub Actions. Our boot application has an actuator, so there's no need to write an API for checking its status; it's already done for the lazy ones. Just make a call to the host: SERVER-URL:PORT/actuator/health
$ curl -v 127.0.0.1:8080/actuator/health
> GET /actuator/health HTTP/1.1
> Host: 127.0.0.1:8080
> User-Agent: curl/7.61.1
> Accept: */*
< HTTP/1.1 200
< Content-Type: application/vnd.spring-boot.actuator.v3+json
< Transfer-Encoding: chunked
< Date: Thu, 04 Jun 2020 12:33:37 GMT
{"status":"UP"}
All we need to do is write a task to check the server on a cron schedule, and if it doesn't respond, we'll send a notification to Telegram.
First, let's figure out how to trigger a workflow with cron:
on:
schedule:
- cron: '*/5 * * * *'
It's quite simple, it’s hard to believe that you can create such events in GitHub that don't fit into webhooks at all. Details are in the documentation:
We'll check the server status manually using curl:
jobs:
ping:
runs-on: ubuntu-18.04
steps:
- name: curl actuator
id: ping
run: |
echo "::set-output name=status::$(curl ${{secrets.SERVER_HOST}}/api/actuator/health)"
- name: health check
run: |
if [[ ${{ steps.ping.outputs.status }} != *"UP"* ]]; then
echo "health check failed"
exit 1
fi
echo "It's OK"
First, we save the server's response to a variable, then in the next step, we check if the status is UP, and if not, we exit with an error. If you need to manually "fail" the action, then exit 1 — is the right weapon.
- name: send alert in telegram
if: ${{ failure() }}
uses: appleboy/telegram-action@master
with:
to: ${{ secrets.TELEGRAM_TO }}
token: ${{ secrets.TELEGRAM_TOKEN }}
message: |
Health check of the:
${{secrets.SERVER_HOST}}/api/actuator/health
failed with the result:
${{ steps.ping.outputs.status }}
We only send a message to Telegram if the action failed in the previous step. To send a message, we use appleboy/telegram-action; you can read the documentation on how to obtain the bot token and chat ID:
Don't forget to specify in the secrets on GitHub: the server URL and tokens for the Telegram bot.
Bonus track — JIRA for the lazy
I promised that we would come back to JIRA, and we did. I have seen hundreds of times during stand-ups when developers completed a feature, merged the branch, but forgot to pull the task into JIRA. Of course, if everything were done in one place, it would be simpler, but in reality, we write code in an IDE, merge branches in Bitbucket or GitHub, and then carry tasks over to JIRA, which requires opening new windows, sometimes logging in again, and so on. When you clearly remember what needs to be done next, there's no point in opening the board again. As a result, in the morning at the stand-up, we have to waste time updating the task board.
GitHub will help us with this routine task as well; initially, we can automatically pull tasks into the code_review column when we submit a pull request. All we need to do is adhere to the naming convention of branches:
[project_name]-[task_number]-title
for example, if the project key for 'GitHub Actions' is GA, then GA-8-jira-bot could be a branch for implementing task GA-8.
The integration with JIRA works through actions from Atlassian; they are not perfect. I must say that some of them did not work for me at all. But we will discuss only those that definitely work and are actively used.
To get started, you need to authenticate in JIRA using the action: atlassian/gajira-login
jobs:
build:
runs-on: ubuntu-latest
name: Jira Workflow
steps:
- name: Login
uses: atlassian/gajira-login@master
env:
JIRA_BASE_URL: ${{ secrets.JIRA_BASE_URL }}
JIRA_USER_EMAIL: ${{ secrets.JIRA_USER_EMAIL }}
JIRA_API_TOKEN: ${{ secrets.JIRA_API_TOKEN }}
For this, you need to get a token in JIRA. How to do this is described here:
We extract the issue ID from the branch name:
- name: Find Issue
id: find_issue
shell: bash
run: |
echo "::set-output name=ISSUE_ID::$(echo ${GITHUB_HEAD_REF} | egrep -o 'GA-[0-9]{1,4}')"
echo branch name: $GITHUB_HEAD_REF
echo extracted issue: ${GITHUB_HEAD_REF} | egrep -o 'GA-[0-9]{1,4}'
- name: Check Issue
shell: bash
run: |
if [[ "${{steps.find_issue.outputs.ISSUE_ID}}" == "" ]]; then
echo "Please name your branch according to the JIRA issue: [project_key]-[task_number]-branch_name"
exit 1
fi
echo successfully found JIRA issue: ${{steps.find_issue.outputs.ISSUE_ID}}
If you search in the GitHub marketplace, you can find an action for this task, but I had to write the same thing using grep based on the branch name because that action from Atlassian just didn't want to work in my project. Figuring out what was wrong took longer than doing the same thing manually.
You just need to move the task to the "Code review" column when creating a pull request:
- name: Transition issue
if: ${{ success() }}
uses: atlassian/gajira-transition@master
with:
issue: ${{ steps.find_issue.outputs.ISSUE_ID }}
transition: "Code review"
There is a special action for this on GitHub; all it requires is the issue ID obtained in the previous step and authorization in JIRA, which we completed earlier.
In the same way, tasks can be dragged when merging into master and during other events in the GitHub workflow. Ultimately, it all depends on your creativity and desire to automate routine processes.
Conclusions
If we look at a classic DEVOPS diagram, we have covered all stages except for operate. I believe if we try, we can find some action in the marketplace for integration with a help-desk system, so we can say that the pipeline is robust and based on its use we can draw conclusions.
Pros:
- The marketplace with ready-made actions for all occasions is really cool. In most of them, you can even check the source code to see how to solve a similar task or post a feature request directly to the author in the GitHub repository.
- Choosing the target platform for building: Linux, macOS, Windows is quite an interesting feature.
- GitHub Packages is great; it's convenient to have the entire infrastructure in one place, eliminating the need to surf through different windows, everything is within one or two clicks of the mouse and is perfectly integrated with GitHub Actions. The support for the Docker registry in the free version is also a good advantage.
- GitHub hides secrets in the build logs, so using it to store passwords and tokens isn't too scary. Throughout my experiments, I never managed to see a secret in plain view in the console.
- Free for Open Source projects.
Cons:
- YML, I just don't like it. When working with such a flow, my most common commit message is "fix yml format"; sometimes I forget to place a tab somewhere or write on the wrong line. Overall, sitting in front of the screen with a protractor and ruler isn't the most enjoyable task.
- DEBUG, debugging the flow with commits, running rebuilds, and outputting to console isn’t always convenient, but that’s more like a case of 'you’ve got too used to it,' getting accustomed to working with comfortable IDEs where you can debug anything.
- You can write your action in anything if you wrap it in Docker, but only JavaScript is natively supported. Of course, it's a matter of taste, but I would prefer something other than JS.
Just a reminder, the repository with all the scripts is here:
Next week, I will be speaking at the Heisenbug 2020 Piter conference. I will not only discuss how to avoid mistakes when preparing test data but also share my secrets for working with datasets in Java applications!
Source: habr.com
