Difference between revisions of "Debian builds with Jenkins"

From PDP/Grid Wiki
Jump to navigationJump to search
(prereq)
(source buils)
Line 12: Line 12:
 
** [https://wiki.jenkins-ci.org/display/JENKINS/Matrix+Project+Plugin Matrix Project] for multi configuration jobs
 
** [https://wiki.jenkins-ci.org/display/JENKINS/Matrix+Project+Plugin Matrix Project] for multi configuration jobs
 
** [https://wiki.jenkins-ci.org/display/JENKINS/Subversion+Plugin Subversion] and [https://wiki.jenkins-ci.org/display/JENKINS/Git+Plugin Git] support
 
** [https://wiki.jenkins-ci.org/display/JENKINS/Subversion+Plugin Subversion] and [https://wiki.jenkins-ci.org/display/JENKINS/Git+Plugin Git] support
 +
** [https://wiki.jenkins-ci.org/display/JENKINS/Workspace+Cleanup+Plugin Workspace Cleanup Plugin]
 
** [https://wiki.jenkins-ci.org/display/JENKINS/Configuration+Slicing+Plugin Configuration Slicing] for easier maintenance
 
** [https://wiki.jenkins-ci.org/display/JENKINS/Configuration+Slicing+Plugin Configuration Slicing] for easier maintenance
 
'''Note:''' The use of [https://wiki.jenkins-ci.org/display/JENKINS/Debian+Package+Builder+Plugin Debian Package Builder] has been discarded due to its limitation on not using a clean build environment  
 
'''Note:''' The use of [https://wiki.jenkins-ci.org/display/JENKINS/Debian+Package+Builder+Plugin Debian Package Builder] has been discarded due to its limitation on not using a clean build environment  
Line 23: Line 24:
 
== Package building jobs ==
 
== Package building jobs ==
  
=== Build source ===
+
The list of available jobs for debian packaging can be found in our [https://jenkins.nikhef.nl:8008 local jenkins instance], under the DEBIAN-BUILDS tab. We started out creating Jenkins jobs based on the recommendations of [http://jenkins-debian-glue.org jenkins-debian-glue], but soon started deviating from it as more and more customization was needed to fit specific use cases. For every package that needs to be build for debian we dedicate two separate Jenkins jobs as suggested in the setup guide by [http://jenkins-debian-glue.org jenkins-debian-glue]. The <package-name>.source will build the source package, and the <package-name>.binaries will build all binary packages for different architectures and distributions. These two can be executed independently from each other The creation of these jobs are outlined below.
  
=== Build binaries ===
+
=== Building source packages ===
 +
 
 +
The steps taken by a <package-name>.source job are:
 +
 
 +
# Restrict where this project can be run: must be a debian slave
 +
# Delete previous workspace
 +
# Source Code Management: svn or git checkout of the '''debian subdirectory''' containing the relevant files [https://www.debian.org/doc/manuals/maint-guide/dreq.en.html] into a directory called 'source'
 +
# Execute source building script
 +
 
 +
The first 3 steps of the job are straight forwards, while the last step is the one that does the actual work. The source building script differs for svn checkouts (local projects) and for git checkouts (adopted projects). For local projects coming from out local svn repository we execute the following build script:
 +
 
 +
ORIG_DIR=$(pwd)
 +
 +
cd source
 +
dch --distribution unstable --release ""
 +
 +
svn upgrade
 +
 +
if [ -f debian/orig-tar.sh ]; then
 +
    chmod +x debian/orig-tar.sh
 +
fi
 +
mkdir -p ../tarballs
 +
uscan --download-current-version --destdir ../tarballs
 +
cp ../*.tar.gz ../tarballs || true
 +
 +
svn-buildpackage -S --svn-builder dpkg-buildpackage -d --svn-move-to=${ORIG_DIR} --svn-dont-purge -uc -us --svn-ignore-new -rfakeroot
 +
 +
lintian -IiE --pedantic `find . -type f -name *.changes` || true
 +
 
 +
First, the changelog is modified to reflect a new build via the ''dhc'' command. After an ''svn upgrade'', the tarballs containing the sources are fetched with the use of ''uscan'' and/or the aid of the debian/orig-tar.sh script (thus it is important for it have an executable flag). In case the custom debian/orig-tar.sh downloads the tarballs into the parent directory, we make sure they are copied into the tarballs directory, where ''svn-buildpackage'' expects them to be. Finally, the source package is built with ''svn-buildpackage'', and lintian checks are executed on the results.
 +
 
 +
At the time of this writing our only encountered adopted projects are the argus c components, for which the '''debian subdirectory''' is checked out through github. These packages come with a predefined Makefile, so building them boils down to executing:
 +
 
 +
dch --distribution unstable --release ""
 +
make deb-src
 +
 
 +
=== Building binaries packages ===
  
 
== Problems and Solution ==
 
== Problems and Solution ==

Revision as of 16:02, 12 May 2015

Goals

The aim of this work was to ease and automate the way debian packages are created for the supported middleware security software. Just like the Koji Testbed for automated RPM packaging, a similar solution was proposed with the use of Jenkins. A Jenkins job can be configured for every component which can be run on a debian build node. This job can than be used to create packages for multiple distributions and architectures in a clean environment with the use of cowbuilder. Much of this work was built on top of the debian building procedure already outlined before.

Prerequisites

There are a couple of prerequisites assumed to be already in place:

Note: The use of Debian Package Builder has been discarded due to its limitation on not using a clean build environment

  • The latest stable debian image (jessie at the time) configured as Jenkins slave (debian build node)
  • Software installed on the debian build node
apt-get install dh-make autotools-dev dh-autoreconf build-essentials devscripts cdbs quilt \
                debhelper fakeroot linitan pbuilder cowbuilder svn_buildpackage maven_debian_helper

Package building jobs

The list of available jobs for debian packaging can be found in our local jenkins instance, under the DEBIAN-BUILDS tab. We started out creating Jenkins jobs based on the recommendations of jenkins-debian-glue, but soon started deviating from it as more and more customization was needed to fit specific use cases. For every package that needs to be build for debian we dedicate two separate Jenkins jobs as suggested in the setup guide by jenkins-debian-glue. The <package-name>.source will build the source package, and the <package-name>.binaries will build all binary packages for different architectures and distributions. These two can be executed independently from each other The creation of these jobs are outlined below.

Building source packages

The steps taken by a <package-name>.source job are:

  1. Restrict where this project can be run: must be a debian slave
  2. Delete previous workspace
  3. Source Code Management: svn or git checkout of the debian subdirectory containing the relevant files [1] into a directory called 'source'
  4. Execute source building script

The first 3 steps of the job are straight forwards, while the last step is the one that does the actual work. The source building script differs for svn checkouts (local projects) and for git checkouts (adopted projects). For local projects coming from out local svn repository we execute the following build script:

ORIG_DIR=$(pwd)

cd source
dch --distribution unstable --release ""

svn upgrade

if [ -f debian/orig-tar.sh ]; then
    chmod +x debian/orig-tar.sh
fi
mkdir -p ../tarballs
uscan --download-current-version --destdir ../tarballs
cp ../*.tar.gz ../tarballs || true

svn-buildpackage -S --svn-builder dpkg-buildpackage -d --svn-move-to=${ORIG_DIR} --svn-dont-purge -uc -us --svn-ignore-new -rfakeroot

lintian -IiE --pedantic `find . -type f -name *.changes` || true

First, the changelog is modified to reflect a new build via the dhc command. After an svn upgrade, the tarballs containing the sources are fetched with the use of uscan and/or the aid of the debian/orig-tar.sh script (thus it is important for it have an executable flag). In case the custom debian/orig-tar.sh downloads the tarballs into the parent directory, we make sure they are copied into the tarballs directory, where svn-buildpackage expects them to be. Finally, the source package is built with svn-buildpackage, and lintian checks are executed on the results.

At the time of this writing our only encountered adopted projects are the argus c components, for which the debian subdirectory is checked out through github. These packages come with a predefined Makefile, so building them boils down to executing:

dch --distribution unstable --release ""
make deb-src

Building binaries packages

Problems and Solution