Setting up GitLab CI to upload a java project to maven central
This article is intended for java developers who need to quickly publish their products to sonatype and/or maven central repositories using GitLab. In this article, I will talk about setting up gitlab-runner, gitlab-ci and maven-plugin to solve this problem.
Prerequisites:
Safe storage of mvn and GPG keys.
Secure execution of public CI tasks.
Uploading artifacts (release/snapshot) to public repositories.
Automatic check of release versions for publication in maven central.
A general solution for uploading artifacts to a repository for multiple projects.
A detailed description of the mechanism for publishing artifacts to Maven Central via the Sonatype OSS Repository Hosting Service is already described in this article user Googolplex, so I will refer to this article in the right places.
Pre-register at Sonatype JIRA and start a ticket to open the repository (for more details, read the section Create a Sonatype JIRA ticket). After opening the repository, the JIRA login/password pair (hereinafter referred to as the Sonatype account) will be used to upload artifacts to the Sonatype nexus.
If you are using the Linux console to generate a GPG key (gnupg/gnupg2), then you need to install rng-tools to generate entropy. Otherwise, key generation can take a very long time.
First of all, you need to create and configure a project in which the pipeline will be stored for the deployment of artifacts. I called my project simply and uncomplicated - deploy
After creating the repository, you need to restrict access to change the repository.
Go to the project -> Settings -> Repository -> Protected Branches. We delete all rules and add a single rule with Wildcard * with the right to push and merge only for users with the Maintainers role. This rule will work for all users of both this project and the group to which this project belongs.
If there are several maintainers, then the best solution would be to restrict access to the project in principle.
Go to the project -> Settings -> General -> Visibility, project features, permissions and set Project visibility to Private.
I have a project in public access, since I use my own GitLab Runner and only I have access to modify the repository. Well, actually it is not in my interests to show private information in public pipeline logs.
Tightening the rules for changing the repository
Go to the project -> Settings -> Repository -> Push Rules and set the flags Committer restriction, Check whether author is a GitLab user. I also recommend setting commit signing, and set the Reject unsigned commits flag.
Next, you need to configure a trigger to run tasks
Go to project -> Settings -> CI / CD -> Pipeline triggers and create a new trigger-token
This token can be immediately added to the general configuration of variables for a group of projects.
Go to the group -> Settings -> CI / CD -> Variables and add a variable DEPLOY_TOKEN with trigger-token in the value.
This section describes the configuration for running tasks on deploy using the native (Specific) and public (Shared) runner.
Specific Runner
I use my own runners, because first of all it's convenient, fast, cheap.
For runner I recommend Linux VDS with 1 CPU, 2 GB RAM, 20 GB HDD. Issue price ~ 3000₽ per year.
My runner
For the runner I took VDS 4 CPU, 4 GB RAM, 50 GB SSD. It cost ~11000₽ and never regretted it.
I have a total of 7 machines. 5 on aruba and 2 on ihor.
So, we have a runner. Now we will set it up.
We go to the machine via SSH and install java, git, maven, gnupg2.
Create a directory for the maven cache and assign group rights runner
You can skip this step if you don't plan to run multiple runners on the same machine.
Runtime platform arch=amd64 os=linux pid=17594 revision=3001a600 version=11.10.0
Running in system-mode.
Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/):
https://gitlab.com/
Please enter the gitlab-ci token for this runner:
REGISTRATION_TOKEN
Please enter the gitlab-ci description for this runner:
[ih1174328.vds.myihor.ru]: Deploy Runner
Please enter the gitlab-ci tags for this runner (comma separated):
deploy
Registering runner... succeeded runner=ZvKdjJhx
Please enter the executor: docker-ssh, parallels, virtualbox, docker-ssh+machine, kubernetes, docker, ssh, docker+machine, shell:
shell
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!
Check that the runner is registered. Go to gitlab.com -> deploy-project -> Settings -> CI/CD -> Runners -> Specific Runners -> Runners activated for this project
Screen
Add separate service /etc/systemd/system/gitlab-deployer.service
We generate a key by answering questions. I used my own name and email.
Be sure to specify the password for the key. Artifacts will be signed with this key.
gpg --gen-key
Checking
gpg --list-keys -a
/home/gitlab-deployer/.gnupg/pubring.gpg
----------------------------------------
pub 4096R/00000000 2019-04-19
uid Petruha Petrov <[email protected]>
sub 4096R/11111111 2019-04-19
Uploading our public key to the keyserver
gpg --keyserver keys.gnupg.net --send-key 00000000
gpg: sending key 00000000 to hkp server keys.gnupg.net
Create a maven directory Repository and link with the cache (make no mistake)
This step can be skipped if you do not plan to run several runners on the same machine.
Add the file .gitlab-ci.yml to the root of the deploy project
The script presents two mutually exclusive deployment tasks. Specific Runner or Shared Runner respectively.
.gitlab-ci.yml
stages:
- deploy
Specific Runner:
extends: .java_deploy_template
# Задача будет выполняться на вашем shell-раннере
tags:
- deploy
Shared Runner:
extends: .java_deploy_template
# Задача будет выполняться на публичном docker-раннере
tags:
- docker
# Образ из раздела GitLab Runner -> Shared Runner -> Docker
image: registry.gitlab.com/group/deploy-project:latest
before_script:
# Импортируем GPG ключ
- printf "${GPG_SECRET_KEY}" | gpg --batch --import
# Сохраняем maven конфигурацию
- printf "${SETTINGS_SECURITY_XML}" > ~/.m2/settings-security.xml
- printf "${SETTINGS_XML}" > ~/.m2/settings.xml
.java_deploy_template:
stage: deploy
# Задача сработает по триггеру, если передана переменная DEPLOY со значением java
only:
variables:
- $DEPLOY == "java"
variables:
# отключаем клонирование текущего проекта
GIT_STRATEGY: none
script:
# Предоставляем возможность хранения пароля в незашифрованном виде
- git config --global credential.helper store
# Сохраняем временные креды пользователя gitlab-ci-token
# Токен работает для всех публичных проектов gitlab.com и для проектов группы
- echo "https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.com" >> ~/.git-credentials
# Полностью чистим текущую директорию
- rm -rf .* *
# Клонируем проект который, будем деплоить в Sonatype Nexus
- git clone ${DEPLOY_CI_REPOSITORY_URL} .
# Переключаемся на нужный коммит
- git checkout ${DEPLOY_CI_COMMIT_SHA} -f
# Если хоть один pom.xml содержит параметр autoReleaseAfterClose валим сборку.
# В противном случае есть риск залить сырые артефакты в maven central
- >
for pom in $(find . -name pom.xml); do
if [[ $(grep -q autoReleaseAfterClose "$pom" && echo $?) == 0 ]]; then
echo "File $pom contains prohibited setting: <autoReleaseAfterClose>";
exit 1;
fi;
done
# Если параметр DEPLOY_CI_COMMIT_TAG пустой, то принудительно ставим SNAPSHOT-версию
- >
if [[ "${DEPLOY_CI_COMMIT_TAG}" != "" ]]; then
mvn versions:set -DnewVersion=${DEPLOY_CI_COMMIT_TAG}
else
VERSION=$(mvn -q -Dexec.executable=echo -Dexec.args='${project.version}' --non-recursive exec:exec)
if [[ "${VERSION}" == *-SNAPSHOT ]]; then
mvn versions:set -DnewVersion=${VERSION}
else
mvn versions:set -DnewVersion=${VERSION}-SNAPSHOT
fi
fi
# Запускаем задачу на сборку и деплой артефактов
- mvn clean deploy -DskipTests=true
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-javadoc-plugin</artifactId>
<executions>
<execution>
<goals>
<goal>jar</goal>
</goals>
<!-- Генерация javadoc должна быть после фазы генерации ресурсов -->
<phase>prepare-package</phase>
<configuration>
<!-- Очень помогает в публичных проектах -->
<failOnError>true</failOnError>
<failOnWarnings>true</failOnWarnings>
<!-- Убирает ошибку поиска документации в target директории -->
<detectOfflineLinks>false</detectOfflineLinks>
</configuration>
</execution>
</executions>
</plugin>
If you have a module that does not contain java (for example only resources)
Or you do not want to generate javadoc in principle, then to help maven-jar-plugin
If you have a multi-module project, and you do not need to upload a specific module to the repository, then you need to add to the pom.xml of this module nexus-staging-maven-plugin with flag skipNexusStagingDeployMojo
<repositories>
<repository>
<id>SonatypeNexus</id>
<url>https://oss.sonatype.org/content/groups/staging/</url>
<!-- Не надо указывать флаги snapshot/release для репозитория -->
</repository>
</repositories>
More pluses
A very rich list of targets for working with the nexus repository (mvn help:describe -Dplugin=org.sonatype.plugins:nexus-staging-maven-plugin).
Automatic release check for downloadability in maven central
When the tag is set, the corresponding task in the deploy project is automatically triggered to upload the release version to nexus (example).
The best part is that close release automatically triggers in nexus.
[INFO] Performing remote staging...
[INFO]
[INFO] * Remote staging into staging profile ID "9043b43f77dcc9"
[INFO] * Created staging repository with ID "orgtouchbit-1037".
[INFO] * Staging repository at https://oss.sonatype.org:443/service/local/staging/deployByRepositoryId/orgtouchbit-1037
[INFO] * Uploading locally staged artifacts to profile org.touchbit
[INFO] * Upload of locally staged artifacts finished.
[INFO] * Closing staging repository with ID "orgtouchbit-1037".
Waiting for operation to complete...
.........
[INFO] Remote staged 1 repositories, finished with success.
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Shields4J 1.0.0 .................................... SUCCESS [ 9.603 s]
[INFO] test-core .......................................... SUCCESS [ 3.419 s]
[INFO] Shields4J client ................................... SUCCESS [ 9.793 s]
[INFO] TestNG listener 1.0.0 .............................. SUCCESS [01:23 min]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:47 min
[INFO] Finished at: 2019-04-21T04:05:46+03:00
[INFO] ------------------------------------------------------------------------
And if something went wrong, then the task will fail
[INFO] Performing remote staging...
[INFO]
[INFO] * Remote staging into staging profile ID "9043b43f77dcc9"
[INFO] * Created staging repository with ID "orgtouchbit-1038".
[INFO] * Staging repository at https://oss.sonatype.org:443/service/local/staging/deployByRepositoryId/orgtouchbit-1038
[INFO] * Uploading locally staged artifacts to profile org.touchbit
[INFO] * Upload of locally staged artifacts finished.
[INFO] * Closing staging repository with ID "orgtouchbit-1038".
Waiting for operation to complete...
.......
[ERROR] Rule failure while trying to close staging repository with ID "orgtouchbit-1039".
[ERROR]
[ERROR] Nexus Staging Rules Failure Report
[ERROR] ==================================
[ERROR]
[ERROR] Repository "orgtouchbit-1039" failures
[ERROR] Rule "signature-staging" failures
[ERROR] * No public key: Key with id: (1f42b618d1cbe1b5) was not able to be located on <a href=http://keys.gnupg.net:11371/>http://keys.gnupg.net:11371/</a>. Upload your public key and try the operation again.
...
[ERROR] Cleaning up local stage directory after a Rule failure during close of staging repositories: [orgtouchbit-1039]
[ERROR] * Deleting context 9043b43f77dcc9.properties
[ERROR] Cleaning up remote stage repositories after a Rule failure during close of staging repositories: [orgtouchbit-1039]
[ERROR] * Dropping failed staging repository with ID "orgtouchbit-1039" (Rule failure during close of staging repositories: [orgtouchbit-1039]).
[ERROR] Remote staging finished with a failure: Staging rules failure!
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Shields4J 1.0.0 .................................... SUCCESS [ 4.073 s]
[INFO] test-core .......................................... SUCCESS [ 2.788 s]
[INFO] Shields4J client ................................... SUCCESS [ 3.962 s]
[INFO] TestNG listener 1.0.0 .............................. FAILURE [01:07 min]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
As a result, we are left with only one choice. Or delete this version or publish.
After the release, after some time, the artifacts will be in
offtop
It was a revelation to me that maven indexes other public repositories.
I had to upload robots.txt because it indexed my old repository.
A separate deploy project in which you can implement several CI tasks for uploading artifacts to public repositories for various development languages.
The deployment project is isolated from outside interference and can only be modified by users with the Owner and Maintainer roles.
Separate Specific Runner with "hot" cache to run deploy tasks only.
Publication of snapshot/release versions in a public repository.
Automatic check of the release version for readiness for publication in maven central.
Protection against automatic publication of "raw" versions in maven central.
Building and publishing snapshot versions "on click".
Single repository for getting snapshot/release versions.
General pipeline for building / testing / publishing a java project.
Setting up GitLab CI is not such a complicated topic as it seems at first glance. It is enough to set up CI on a turnkey basis a couple of times, and now, you are far from being an amateur in this matter. Moreover, GitLab documentation is very redundant. Don't be afraid to take the first step. The road appears under the steps of the walker (I don’t remember who said 🙂).
I will be glad to feedback.
In the next article, I'll show you how to set up GitLab CI to run integration test tasks competitively (running test services with docker-compose) if you only have one shell runner.