Debian builds with Jenkins

From PDP/Grid Wiki
Revision as of 17:35, 12 May 2015 by Tamasb@nikhef.nl (talk | contribs) (external dep)
Jump to navigationJump to search

Goals

The aim of this work was to ease and automate the way debian packages are created for the supported middleware security software. Just like the Koji Testbed for automated RPM packaging, a similar solution was proposed with the use of Jenkins. A Jenkins job can be configured for every component which can be run on a debian build node. This job can than be used to create packages for multiple distributions and architectures in a clean environment with the use of cowbuilder. Much of this work was built on top of the debian building procedure already outlined before.

Prerequisites

There are a couple of prerequisites assumed to be already in place:

Note: The use of Debian Package Builder has been discarded due to its limitation on not using a clean build environment

  • The latest stable debian image (jessie at the time) configured as Jenkins slave (debian build node)
  • Software installed on the debian build node 
apt-get install dh-make autotools-dev dh-autoreconf build-essentials devscripts cdbs quilt \
                debhelper fakeroot linitan pbuilder cowbuilder svn_buildpackage maven_debian_helper

Package building jobs

The list of available jobs for debian packaging can be found in our local jenkins instance, under the DEBIAN-BUILDS tab. We started out creating Jenkins jobs based on the recommendations of jenkins-debian-glue, but soon started deviating from it as more and more customization was needed to fit specific use cases. For every package that needs to be build for debian we dedicate two separate Jenkins jobs as suggested in the setup guide by jenkins-debian-glue. The <package-name>.source will build the source package, and the <package-name>.binaries will build all binary packages for different architectures and distributions. These two can be executed independently from each other The creation of these jobs are outlined below.

Building source packages

The steps taken by a <package-name>.source job are:

  1. Restrict where this project can be run: must be a debian slave
  2. Delete previous workspace
  3. Source Code Management: svn or git checkout of the debian subdirectory containing the relevant files [1] into a directory called 'source'
  4. Execute source building script

The first 3 steps of the job are straight forwards, while the last step is the one that does the actual work. The source building script differs for svn checkouts (local projects) and for git checkouts (adopted projects). For local projects coming from out local svn repository we execute the following build script: 

ORIG_DIR=$(pwd)

cd source
dch --distribution unstable --release ""

svn upgrade

if [ -f debian/orig-tar.sh ]; then
    chmod +x debian/orig-tar.sh
fi
mkdir -p ../tarballs
uscan --download-current-version --destdir ../tarballs
cp ../*.tar.gz ../tarballs || true

svn-buildpackage -S --svn-builder dpkg-buildpackage -d --svn-move-to=${ORIG_DIR} --svn-dont-purge -uc -us --svn-ignore-new -rfakeroot

lintian -IiE --pedantic `find . -type f -name *.changes` || true

First, the changelog is modified to reflect a new build via the dhc command. After an svn upgrade, the tarballs containing the sources are fetched with the use of uscan and/or the aid of the debian/orig-tar.sh script (thus it is important for it have an executable flag). In case the custom debian/orig-tar.sh downloads the tarballs into the parent directory, we make sure they are copied into the tarballs directory, where svn-buildpackage expects them to be. Finally, the source package is built with svn-buildpackage, and lintian checks are executed on the results.

At the time of this writing our only encountered adopted projects are the argus c components, for which the debian subdirectory is checked out through github. These packages come with a predefined Makefile, so building them boils down to executing: 

dch --distribution unstable --release ""
make deb-src

So that you wouldn't have to execute every build separately you can chain them into a single job which calls every <package-name>.source job as a downstream project, using Parametrized Trigger Plugin. On our Jenkins instance this job is called build-add.source

Building binaries packages

The steps taken by a <package-name>.binaries job are:

  1. Define ${architecture}, ${distribution} and debian slave from the matrix configuration
  2. Delete previous workspace
  3. Source Code Management: svn or git checkout of the debian subdirectory containing the relevant files [2] into a directory called 'source'
  4. Execute binary building script
    1. add package suffix
    2. build source package
    3. build binary packages
    4. execute lintian checks

Every <package-name>.binaries job is a multi-configuration job with the following axis defined:

  • User-defined Axis: architecture=amd64 i386
  • User-defined Axis: distribution=jessie wheezy squeeze sid
  • Label expression: label_exp=debian8

The two User-defined axis will create 8 sub-jobs, one for each combinations of architecture and distribution, while the Label expression will restrict the jobs' execution to debian8 nodes (see Jenkins Setup on how to set up nodes with labels).

After clearing the workspace and checking out the debian subdirectory to following script is executed: 

###############################################################
#  Building source with modified changelog                    #
###############################################################
ORIG_DIR=$(pwd)

cd source
if [ "${distribution}" = "sid" ]; then
    dch --distribution unstable --release ""
else
    if [ "${distribution}" = "squeeze" ]; then
       bptag=bpo60+1
    elif [ "${distribution}" = "wheezy" ]; then
       bptag=bpo70+1
    elif [ "${distribution}" = "jessie" ]; then
       bptag=bpo80+1
    fi

    version=`dpkg-parsechangelog | sed -n 's/^Version: //p'`
    dch --force-distribution --distribution ${distribution}-backports -b -v ${version}~${bptag} "Rebuild for ${distribution}"
fi

svn upgrade

if [ -f debian/orig-tar.sh ]; then
    chmod +x debian/orig-tar.sh
fi
mkdir -p ../tarballs
uscan --download-current-version --destdir ../tarballs
cp ../*.tar.gz ../tarballs || true

svn-buildpackage -S --svn-builder dpkg-buildpackage -d --svn-move-to=${ORIG_DIR} --svn-dont-purge -uc -us --svn-ignore-new -rfakeroot

cd ../

###############################################################
#  Building binaries                                          #
###############################################################

USE_LOCAL_REPOSITORY=true

if [ -n ${USE_LOCAL_REPOSITORY} ]; then
    export release=${distribution}
    export REMOVE_FROM_RELEASE=true
else
     export REPOSITORY_EXTRA="deb http://software.nikhef.nl/dist/debian/ ${distribution} main"
     export REPOSITORY_EXTRA_KEYS='http://software.nikhef.nl/dist/debian/DEB-GPG-KEY-MWSEC.asc'
fi

/usr/bin/build-and-provide-package

###############################################################
#  Lintian reports                                            #
###############################################################


/usr/bin/lintian-junit-report `find . -type f -name *.dsc`
cat lintian.txt

The first part of the script build a source package with the relevant name suffix according to backporting conventions. This part is similar to the script executed in a <package-name>.source job, with a modified dch behaviour. Because of the different suffix appended to different distribution backports, it is necesarry to rebuild the source with the appropriate name. This means that we cannot rely on the output of <package-name>.source to be used in the <package-name>.binaries, as suggested by the jenkins-debian-glue guide.

The second part of the script is used to define where dependencies should be taken from, and executes the build-and-provide-package script. The build-and-provide-package script, provided by jenkins-debian-glue, will use an existing cowbuilder base (or create a new one) for every distribution-architecture pair to build the package. Once the packages are build, the script uploads them into the local repository.

As an extra step, lintian checks are executed at the end of the script.

Build dependency resolution

There are two types of dependencies that we come across when building the grid middlware: internal dependencies are referring to closely related packages from the same software stack which are maintained locally, while external dependencies are referring to packages not maintained locally.

Internal dependencies

When resolving internal dependencies there are two options to choose from: one can either use the official release repository [3], or use the unofficial local repository of the building node. The first option works well for builds having dependencies that have been build and packaged before. In case new releases are out for two interdependent packages, or we are building for a new distribution, the first option will fail, in which case we can resort to the local repository. The choice between the inclusion of the official repository or the local repository is made at build time by: 

USE_LOCAL_REPOSITORY=true

if [ -n ${USE_LOCAL_REPOSITORY} ]; then
    export release=${distribution}
    export REMOVE_FROM_RELEASE=true
else
     export REPOSITORY_EXTRA="deb http://software.nikhef.nl/dist/debian/ ${distribution} main"
     export REPOSITORY_EXTRA_KEYS='http://software.nikhef.nl/dist/debian/DEB-GPG-KEY-MWSEC.asc'
fi

The build-and-provide-package script recognizes the exported environmental variables. The ${release} variable is controlling the name of local repository where packages are uploaded, if unspecified it defaults to "<package-name>-distribution" (in which case every package ends up in a separate repository). By setting release=${distribution} we are making sure that every resulting package will get into the same local repository. The build-and-provide-package script will also implicitly include the specified release repository into the /etc/apt/sources.list.d of the cowbuilder environment. The ${REMOVE_FROM_RELEASE} variable ensures that packages are removed from the local repository before uploading the results of the new build, in order to avoid hash collisions on the same filename.

By using ${REPOSITORY_EXTRA} and ${REPOSITORY_EXTRA_KEYS}, build-and-provide-package will include these into the /etc/apt/sources.list.d of the cowbuilder environment.

In principle you can include both local and official repositories into your build process, but this could lead to confusions later on. The build process will resolve every dependency from the available repositories included.

External dependencies

When it comes to external dependencies, one can mostly rely on official debian repositories. Every once in a while, a specific package is missing from official debian repositories, which leaves us with the only choice of including external repositories in the build environment (such as [4], [5]). These external repositories are often unmaintained, or have outdated packages that are not desirable in the clean environment. To ensure that only the most necessary packages are installed into the build environment we can run a pbuilder hook. This way the external repository can be added briefly to install the necessary packages, and removed afterwards, leaving a clean repository list. If the external repository is not removed, the automatic dependency resolver might use it for further dependency resolution, which is undesirable.

The pbuilder hook script executed for external dependency resolution can be found at /usr/share/jenkins-debian-glue/pbuilder-hookdir/D30dependency-downloader. 

#!/bin/sh

set -x

if [ -n "${DEB_EXTRA_REPO}" -a -n "${DEB_EXTRA_REPO_KEY}" -a -n "${DEB_INSTALL_PACKAGES}" ]; then

    apt-get install -y wget

    echo "${DEB_EXTRA_REPO}" > /etc/apt/sources.list.d/extra-repo.list
    wget -O- "${DEB_EXTRA_REPO_KEY}" | apt-key add -

    apt-get update
    apt-get install -y ${DEB_INSTALL_PACKAGES}

    rm /etc/apt/sources.list.d/extra-repo.list 
    apt-get update

fi

Once the script is in place on your debian node, all you have to do is inject the required environmental variables into your jenkins job. As an example: 

 DEB_EXTRA_REPO=deb http://repo-deb.ige-project.eu/debian/ squeeze main
 DEB_EXTRA_REPO_KEY=http://repo-deb.ige-project.eu/DEB-GPG-KEY-IGE.asc
 DEB_INSTALL_PACKAGES=libglobus-gridmap-callout-error-dev libglobus-common0 libglobus-gsi-credential1

These variable will be passed into the build environment where to hook will pick them up. Make sure to configure your jenkins user on the debian node with sudo rights and include this in the sudoers file: 

Defaults:jenkins env_keep+="DEB_* DIST ARCH"

Problems and Solution

Retrieved from "https://wiki.nikhef.nl/grid/index.php?title=Debian_builds_with_Jenkins&oldid=9672"