Filed under: Automation, Continuous integration, DevOps, — Tags: Jenkins, Jenkins Configuration as Code, jcasc — Thomas Sundberg — 2019-12-23
Configuring Jenkins is not as easy as one might imagine or want. There are some 1500 plugins available. Selecting the proper plugins and then configuring them is a daunting task.
Jenkins configuration as code doesn't solve the selection problem. But it does solve the problem of having a repeatable setup of Jenkins. The setup is version controlled and thus traceable.
Here is an example where I use this this tool chain:
A working example is available at GitHub.
This is an example of a Jenkins installation where everything is version controlled. There is one command that has to be executed in order to create a fully working build environment.
Lots of tools are used. Getting a build environment up and running is mostly a configuration issue. You need to choose the proper tools and configure them properly. Finding a working combination is challenging.
If you read this you probably know that Jenkins is a continuous integration build server, also known as a CI server. It is a tool that builds your project.
Docker is a tool that can create virtual computers. It is configurable from text files and thus a good fit in a tutorial where repeatability is important. It's not really interesting that it works on my machine, it should work on your machine as well so you can learn how to use this tool chain.
Java is a portable programming language that is executed using a virtual machine. You can develop on a Mac or a Windows machine and run the result on a Linux box.
The Java program in this example is only here so that we have something to build. A build server without a project would be rather uninteresting.
The build in itself has created some issues that must be resolved. And that's the only reason why this text is written in the first place, to document a working example.
Maven is a Java build tool that is used to build Java programs. It simplifies dependency handling, running tests as well as packaging the result in a jar file.
Git is a version control system. The goal with this exercise is to create a working example where every change can be traced to the individual who did the change.
There are a lot of moving parts here. Lots of tools involved. How are they connected?
Docker will host a Linux system. Jenkins is installed on that Linux system.
Jobs are defined in Jenkins. Each job will do something with the Java project that solves some business issue.
The example job is a Maven build. It will download a copy of the project from a version control system, GitHub, execute Maven that in turn will use Java to compile, test and package the project.
There are five files that will be used in this example:
src/docker-compose.yml - a file that will help us get all the Docker stuff up and runningsrc/master/Dockerfile - the definition of the Docker image that will be created and testedsrc/master/plugins.txt - the plugins that will be pre-installed in Jenkinssrc/config/jenkins.yaml - the definition of Jenkinssrc/config/jobs.yaml - the jobs definedLet us start from the outside and work our way in. The first thing needed is a Linux host where Jenkins can be installed.
The Docker image can be built and started from a command line. It is, however, much easier, to do it from
a docker-compose file.
A working example looks like this:
version: '3.7'
services:
jenkins:
build:
context: ./master
ports:
- 80:8080
- 50000:50000
volumes:
- jenkins_home:/var/jenkins_home
- ./config:/var/jenkins_conf
environment:
- CASC_JENKINS_CONFIG=/var/jenkins_conf
volumes:
jenkins_home:
This will create a docker image from the Docker file we are about to create. It will expose Jenkins on port 80 and make sure that there is a volume in Docker where jobs can be stored. Our job history will be gone after a restart if we don't store the executions outside of the Docker container.
The main work of creating a Linux box to run Jenkins in is done in the Docker file. It defines which version of Jenkins that will be used and it pre-installs some plugins.
FROM jenkins/jenkins:2.209
LABEL maintainer="thomas@thinkcode.se"
COPY plugins.txt /usr/share/jenkins/ref/plugins.txt
ENV JENKINS_HOME /var/jenkins_home
ARG JAVA_OPTS
ENV JAVA_OPTS "-Djenkins.install.runSetupWizard=false ${JAVA_OPTS:-}"
RUN xargs /usr/local/bin/install-plugins.sh < /usr/share/jenkins/ref/plugins.txt
The Docker file refers to a file plugins.txt. It contains a list of plugins and versions that should
be installed before Jenkins is started.
This is where you start configuring Jenkins for real. Up until now it has only been a matter of preparing an environment to run in.
configuration-as-code:1.34
jdk-tool:1.4
adoptopenjdk:1.2
job-dsl:1.76
credentials:2.3.0
git:4.0.0
ssh-slaves:1.31.0
warnings:5.0.1
matrix-auth:2.5
workflow-aggregator:2.6
timestamper:1.10
github-oauth:0.33
pretested-integration:3.1.0
envinject:2.3.0
text-finder:1.12
email-ext:2.68
slack:2.35
parameterized-trigger:2.36
copyartifact:1.43
htmlpublisher:1.21
bouncycastle-api:2.17
command-launcher:1.4
This set of plugins should be optimized. But only you know how. This is a starting point. It probably contains stuff you don't care too much about.
Next step is to prepare Jenkins. I have a reference to ./config:/var/jenkins_conf in docker-compose that is
referred to from the environment variable CASC_JENKINS_CONFIG. This will allow the Jenkins as code
plugin, configuration-as-code:1.34 above, to find the config files.
This will setup Jenkins:
jenkins:
systemMessage: "Jenkins Configured using Code"
numExecutors: 1
mode: NORMAL
scmCheckoutRetryCount: 3
labelString: "master-label"
securityRealm:
local:
allowsSignup: false
users:
- id: admin
password: ${adminpw:-passw0rd}
authorizationStrategy:
globalMatrix:
grantedPermissions:
- "Overall/Read:anonymous"
- "Job/Read:anonymous"
- "View/Read:anonymous"
- "Overall/Administer:anonymous"
crumbIssuer: "standard"
remotingSecurity:
enabled: true
unclassified:
location:
adminAddress: "jenkins@example.com"
url: "http://jenkins.example.com/"
tool:
git:
installations:
- name: Default
home: "git"
jdk:
installations:
- name: "open-jdk8"
properties:
- installSource:
installers:
- adoptOpenJdkInstaller:
id: "jdk8u232-b09"
- name: "oracle-jdk8"
properties:
- installSource:
installers:
- jdkInstaller:
acceptLicense: true
id: "jdk-8u221-oth-JPR"
maven:
installations:
- name: "Maven 3"
properties:
- installSource:
installers:
- maven:
id: "3.5.4"
Understanding how to configure each part of Jenkins as above is not very easy. I used the the web interface and then I looked at the configuration that could be exported from Jenkins. This is how I understood how the installation of Open JDK should be done.
Coming up with this definition is perhaps natural if your core competence is to maintain Jenkins.
- name: "open-jdk8"
properties:
- installSource:
installers:
- adoptOpenJdkInstaller:
id: "jdk8u232-b09"
As a Jenkins user, I had to get some help with coming up with it.
Follow the link Manage Jenkins -> Configuration as Code -> View configuration and search for the configuration you want to make sure always will be done.
Oracle require an account for downloading and installing a JDK. The tool definition
for oracle-jdk8 above is an example.
When you start Jenkins, you need to enter your Oracle credentials for the download to work. The credentials can probably be stored somewhere else and read by the configuration, I just haven't bothered to learn how.
As a result, I included the definition in the example but the job defined below requires open jdk. Open jdk can be downloaded without any credentials. That's a better fit for a setup where no manual work should be needed.
The last thing to do is to create the jobs that should be available.
This is done in src/config/jobs.yaml. An example looks like this:
jobs:
- script: >
pipelineJob('maven-pipeline-version-controlled') {
definition {
triggers {
cron('H/5 * * * *')
}
cpsScm {
scm {
git {
remote { url 'https://github.com/tsundberg/maven-build-with-jenkins-pipeline.git' }
branch '*/master'
extensions {}
}
}
scriptPath 'src/ci/Jenkinsfile'
}
}
}
This is a pipeline job and it's definition is version controlled together with the project. It is refereed with the url https://github.com/tsundberg/maven-build-with-jenkins-pipeline.git.
It is worth noting that I have added a trigger in this job so that it will be triggered the first time.
triggers {
cron('H/5 * * * *')
}
This will trigger the job within five minutes after Jenkins has started. The actual job definition is stored together with the project. This means the poll frequency etc is defined there. If this trigger wasn't here, then you would have to trigger the job manually and that was something we wanted to avoid.
Everything is in place now. The only thing missing now is the command that build the Docker image, install Jenkins, and start serving Jenkins on http://localhost:80.
Run
docker-compose up
in the directory src and wait. If everything works properly, you will have a working Jenkins up and running within a few minutes.
I built the example many times and to make sure I had a clean state I killed my Docker images rather hard. I created a clean script with this content:
#!/usr/bin/env sh
docker kill $(docker ps -q)
docker rm $(docker ps -a -q)
docker rmi $(docker images -q)
docker volume prune -f
It will clean your Docker environment. If you have something valuable there, make sure you know which commands to run. I don't have anything I care about so I purged stuff rather violently
Setting up Jenkins configuration as code took some time. Mostly because it is still a rather young tool. The documentation and tutorials need refinement.
I foresee that it will enable me to take a good step towards continuous deployment when it is possible, and preferred, to make sure that all changes to the build environment are traced and version controlled. An auditor can follow the development of the source code as well as the build environment.
This example originated from Ewelina Wilkosz at Praqma and shared at GitHub
I would like to thank Malin Ekholm and Mika Kytöläinen for feedback.
RSS