Hi all!
In this article, I want to share one of the ways to run JMeter performance tests in OpenShift using Jenkins as an automation. First, we will do all the necessary actions (create ImageStreams
, BuildConfig
, Job
etc.) in manual mode. After that, we will write the Jenkins Pipeline.
As a starting point, we should have:
- running OpenShift (v3.11) cluster
- Jenkins server with configured credentials to run on OpenShift
- file
apache-jmeter-5.2.tgz
The tests will be simple HTTP Request
on ya.ru
into one stream.
Creating a project in OpenShift
Let's start by creating a new environment. Let's create perftest
team environment:
$ oc new-project perftest --display-name="Performance Tests" --description="Performance Tests - JMeter"
We will automatically be transferred to the newly created environment perftest
, let's check that this is so:
$ oc project
Using project "perftest" on server "https://127.0.0.1:8443".
Creating a Storage
Test reports will be stored in common for the web server and jmeter-meter
'a place - /jmeter/reports
.
It is better to create storages now, because PODs will be tied to them jmeter-web
ΠΈ jmeter-master
.
More detailed information on storages can be found in the official documentation
Let's create yaml files for PV
ΠΈ PVC
.
pv.yaml
$ tee pv.yaml<<EOF
apiVersion: v1
kind: PersistentVolume
metadata:
name: jmeter-reports
spec:
capacity:
storage: 10Gi
accessModes:
- ReadWriteMany
glusterfs:
endpoints: glusterfs-cluster
path: /jmeter/reports
readOnly: false
persistentVolumeReclaimPolicy: Retain
EOF
pvc.yaml
$ tee pvc.yaml<<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: jmeter-reports
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 10Gi
EOF
Let's create PV
ΠΈ PVC
in an OpenShift environment:
$ oc create -f pv.yaml -n perftest
$ oc create -f pvc.yaml -n perftest
Checking the status for PVC
:
$ oc get pvc -n perftest
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
jmeter-reports Bound pvc-b0e5f152-db4b-11ea-a497-566f75280024 10Gi RWX glusterfs-storage 8m
Here's what it will look like in the GUI:
Creating a base JMeter image
Let's move on to creating ImageStream
ΠΈ BuildConfig
.
You can find all the necessary information in the documentation -
The image build strategy is Docker
from a local source.
Let's create a base image jmeter-base
, which will be the basis for jmeter-master
.
Dockerfile
FROM openjdk:8u212-jdk
ARG JMETER_VER="5.2"
ENV JMETER_HOME /jmeter/apache-jmeter-$JMETER_VER
ENV PATH $JMETER_HOME/bin:$PATH
RUN mkdir -p /jmeter/results
&& mkdir /jmeter/tests
WORKDIR /jmeter
COPY apache-jmeter-$JMETER_VER.tgz .
RUN tar -xzf $JMETER_HOME.tgz
&& rm $JMETER_HOME.tgz
&& ls -la
RUN sed -i s/#server.rmi.ssl.disable=false/server.rmi.ssl.disable=true/ $JMETER_HOME/bin/jmeter.properties
EXPOSE 60000
is.yaml
$ tee is.yaml<<EOF
apiVersion: v1
kind: ImageStream
metadata:
labels:
build: jmeter-base
name: jmeter-base
EOF
bc.yaml
$ tee bc.yaml<<EOF
apiVersion: v1
kind: BuildConfig
metadata:
name: jmeter-base
spec:
failedBuildsHistoryLimit: 5
nodeSelector: null
output:
to:
kind: ImageStreamTag
name: 'jmeter-base:latest'
postCommit: {}
resources: {}
runPolicy: Serial
source:
binary: {}
type: Binary
strategy:
dockerStrategy:
from:
kind: ImageStreamTag
name: 'openjdk:8u212-jdk'
type: Docker
successfulBuildsHistoryLimit: 5
EOF
Let's create objects IS
ΠΈ BC
:
$ oc create -f is.yaml -n perftest
$ oc create -f bc.yaml -n perftest
Now let's build the base image jmeter-base
:
$ oc start-build jmeter-base -n perftest --from-dir=. --follow
JMeter WEB
jmeter-web
this is the Apache web server. Its task is to provide a directory with test results for viewing.
Prepared Dockerfile
and config file httpd.conf
. For directive DocumentRoot
set value /jmeter/reports
, i.e. directory where test results are saved.
Dockerfile
$ tee Dockerfile<<EOF
FROM httpd:2.4
COPY httpd.conf /usr/local/apache2/conf/httpd.conf
RUN chmod -R 777 /usr/local/apache2/logs
EXPOSE 8080
CMD ["httpd", "-D", "FOREGROUND"]
EOF
is.yaml
$ tee is.yaml<<EOF
apiVersion: v1
kind: ImageStream
metadata:
generation: 1
labels:
build: jmeter-web
name: jmeter-web
EOF
bc.yaml
$ tee bc.yaml<<EOF
apiVersion: v1
kind: BuildConfig
metadata:
name: jmeter-web
spec:
failedBuildsHistoryLimit: 5
nodeSelector: null
output:
to:
kind: ImageStreamTag
name: 'jmeter-web:latest'
runPolicy: Serial
source:
binary: {}
type: Binary
strategy:
dockerStrategy:
from:
kind: ImageStreamTag
name: 'httpd:2.4'
type: Docker
successfulBuildsHistoryLimit: 5
EOF
Let's create ImageStream
ΠΈ BuildConfig
objects:
$ oc create -f is.yaml -n perftest
$ oc create -f bc.yaml -n perftest
Collecting an image from Dockerfile
:
$ oc start-build jmeter-web -n perftest --from-dir=. --follow
dc.yaml
$ tee dc.yaml<<EOF
apiVersion: apps.openshift.io/v1
kind: DeploymentConfig
metadata:
name: jmeter-web
spec:
replicas: 1
template:
metadata:
labels:
name: jmeter-web
spec:
containers:
- image: 172.30.1.1:5000/perftest/jmeter-web
name: jmeter-web
volumeMounts:
- mountPath: /jmeter/reports
name: jmeter-reports
ports:
- containerPort: 80
protocol: TCP
- containerPort: 8080
protocol: TCP
volumes:
- name: jmeter-reports
persistentVolumeClaim:
claimName: jmeter-reports
EOF
sc.yaml
$ tee sc.yaml<<EOF
apiVersion: v1
kind: Service
metadata:
labels:
app: jmeter-web
name: jmeter-web
spec:
ports:
- name: 8080-tcp
port: 8080
protocol: TCP
targetPort: 8080
selector:
deploymentconfig: jmeter-web
sessionAffinity: None
type: ClusterIP
EOF
Let's create objects Service
ΠΈ DeploymentConfig
:
$ oc create -f sc.yaml -n perftest
$ oc create -f dc.yaml -n perftest
jmeter-master
Let's deploy the Apache web server.
This is a Dockerfile jmeter-master
'a based on jmeter-base
, which will run the tests and store the results in storage.
Dockerfile
Dockerfile
for jmeter-master
, based on jmeter-base
.
FROM jmeter-base
ARG JMETER_VER="5.2"
ENV JMETER_HOME /jmeter/apache-jmeter-$JMETER_VER
ENV PATH $JMETER_HOME/bin:$PATH
WORKDIR /jmeter
COPY run.sh /jmeter/
COPY tests/*.jmx /jmeter/tests/
RUN chmod +x /jmeter/run.sh
ENTRYPOINT ["/bin/bash"]
CMD ["/jmeter/run.sh"]
run.sh
run.sh
this is a script that runs JMeter and saves the results to a directory files
.
Each time it is run, the script deletes the previous tests, so that you can only work with the latest data. But this is not a problem, because it can be changed to suit your needs.
#!/bin/bash
set -e
if [ -d "/jmeter/reports/files" ]
then
echo "Directory /jmeter/reports/files exist - OK"
else
echo "Creating /jmeter/reports/files directory"
mkdir /jmeter/reports/files
fi
if [ -d "/jmeter/reports/dashboards" ]
then
echo "Directory /jmeter/reports/dashboards exist"
else
echo "Creating /jmeter/reports/dashboards directory"
mkdir /jmeter/reports/dashboards
fi
echo "*** JMeter START Tests ***"
for item in $(ls -1 /jmeter/tests | grep jmx)
do
echo "*** Removing dashboard directory for $item"
rm -rdf /jmeter/reports/dashboards/${item}*
echo "*** Removing tests directory for $item"
rm -rdf /jmeter/reports/files/${item}*
echo "*** Testing a $item file ***"
jmeter -n -t /jmeter/tests/${item} -l /jmeter/reports/files/${item}-report.jtl -e -o /jmeter/reports/dashboards/${item}-dash
done
is.yaml
$ tee is.yaml<<EOF
apiVersion: image.openshift.io/v1
kind: ImageStream
metadata:
generation: 1
labels:
build: jmeter-master
name: jmeter-master
EOF
bc.yaml
$ tee bc.yaml<<EOF
apiVersion: build.openshift.io/v1
kind: BuildConfig
metadata:
name: jmeter-master
spec:
failedBuildsHistoryLimit: 5
nodeSelector: null
output:
to:
kind: ImageStreamTag
name: 'jmeter-master:latest'
runPolicy: Serial
source:
binary: {}
type: Binary
strategy:
dockerStrategy:
from:
kind: ImageStreamTag
name: 'jmeter-base:latest'
type: Docker
successfulBuildsHistoryLimit: 5
EOF
Let's create IS
ΠΈ BC
objects:
$ oc create -f is.yaml -n perftest
$ oc create -f bc.yaml -n perftest
We collect jmeter-master
image:
$ oc start-build jmeter-master -n perftest --from-dir=. --follow
Job
Job
's are used in OpenShift
'e to run one or more POD
's and ensure their successful completion after executing the command/script .
$ tee job.yaml<<EOF
apiVersion: batch/v1
kind: Job
metadata:
name: jmeter-master
labels:
jobName: jmeter-master
spec:
completions: 1
parallelism: 1
template:
metadata:
name: jmeter-master
labels:
jobName: jmeter-master
spec:
containers:
- name: jmeter-master
image: 172.30.1.1:5000/perftest/jmeter-master:latest
volumeMounts:
- mountPath: /jmeter/reports
name: jmeter-reports
imagePullPolicy: Always
volumes:
- name: jmeter-reports
persistentVolumeClaim:
claimName: jmeter-reports
restartPolicy: Never
terminationGracePeriodSeconds: 30
EOF
Create an object Job
:
$ oc create -f job.yaml -n perftest
Let's check the status of the job:
$ oc get jobs -n perftest
NAME DESIRED SUCCESSFUL AGE
jmeter-master 1 1 5m
To delete Job
let's use the command:
$ oc delete jobs/jmeter-master -n perftest --ignore-not-found=true
Jenkins Pipeline
Now automation. Let's talk about the steps again:
git clone
oc whoami -t
oc start-build ...
oc delete jobs/jmeter-master
oc create -f job.yaml -n perftest
Below is a pipeline that clones the repository, deletes and creates OpenShift Job
's.
#!groovy
pipeline {
agent any
stages {
stage('Start Notifications') {
steps {
echo "Sending Email Notification"
}
post {
always {
echo "STARTED - Performance Tests"
mail(to: '[email protected]', from: "[email protected]", subject: "START - Performance Tests",mimeType: "text/html", body: "<strong>START - Performance Tests</strong><br /><br />Project: Name of Project<br />Environment: PerfTest<br />Build number: ${env.BUILD_NUMBER}<br />Build URL: ${env.BUILD_URL}"
}
}
}
stage('Git checkout') {
steps {
...
}
}
stage('Perf Tests') {
steps {
script {
sh '''
OC_CMD1="oc login -u=username -p=PASS -n=perftest
--server=https://...:8443"
$OC_CMD1
OC_TOKEN=`oc whoami -t`
OC_CMD2="oc --token=$OC_TOKEN --server=https://...:8443
start-build jmeter-master -n=perftest --from-dir=./master
--follow=true"
OC_CMD3="oc --token=$OC_TOKEN --server=https://...:8443
delete jobs/jmeter-master -n=perftest --ignore-not-found=true"
OC_CMD4="oc--token=$OC_TOKEN --server=https://...:8443
create -f ./master/job.yaml -n=perftest"
$OC_CMD2
$OC_CMD3
$OC_CMD4
'''
}
}
}
post {
failure {
echo "FAILED - Performance Tests"
mail(to: '[email protected]', from: "[email protected]", subject: "FAILED - Performance Tests",mimeType: "text/html", body: "<strong>FAILED - Performance Tests</strong><br /><br />Project: Name of Project<br />Environment: PerfTest<br />Build number: ${env.BUILD_NUMBER}<br />Build URL: ${env.BUILD_URL}"
}
success {
echo "SUCCESSED - Performance Tests"
mail(to: '[email protected]', from: "[email protected]", subject: "SUCCESSED - Performance Tests",mimeType: "text/html", body: "<strong>SUCCESSED - Performance Tests</strong><br /><br />Project: Name of Project<br />Environment: PerfTest<br />Build number: ${env.BUILD_NUMBER}<br />Build URL: ${env.BUILD_URL}"
}
}
}
}
After the Pipeline has worked, we will receive an email notification '[email protected]
from [email protected]
.
By clicking on the link files
, which stores test reports:
File contents ya.HTTP.Request.jmx-report.jtk
:
timeStamp,elapsed,label,responseCode,responseMessage,threadName,dataType,success,failureMessage,bytes,sentBytes,grpThreads,allThreads,URL,Latency,IdleTime,Connect
1597311456443,569,Yandex - HTTP Request,200,Ok,Thread Group 1-1,text,true,,59449,220,1,1,https://ya.ru/,145,0,57
1597311456443,147,Yandex - HTTP Request-0,302,Found,Thread Group 1-1,,true,,478,110,1,1,http://ya.ru/,145,0,57
1597311456592,420,Yandex - HTTP Request-1,200,Ok,Thread Group 1-1,text,true,,58971,110,1,1,https://ya.ru/,370,0,259
Conclusion
This article demonstrated one of the options for running JMeter tests in an OpenShift environment. We did all the steps manually, after which we created the Jenkins Pipeline to automate the process of running tests.
Sources and Documentation
JMeter - Distributed Load Testing using Docker Running JMeter Tests in OpenShift from a Jenkins Pipeline open shiftPersistent storage Builds and Image Streams
Source: habr.com