SAC Debian packaging

The following procedures extend the SAC software procedures with respect to the creation and distribution of Debian and Ubuntu packages.

packaging a new release of an existing component

When a component has been updated and released, it should also be repackaged for Debian at some point. The following steps outline what needs to be done.

Update the Debian source package

Source packages are maintained in SVN. Check out the component and run uscan to fetch the upstream tarball.

cd $mwsec/packaging/debian/trunk/$component
svn update
uscan --report # optional
uscan --destdir ../tarballs

Update debian/changelog.

dch -v $version-1 -D UNRELEASED "New upstream release"

Review and test the packaging

Check the upstream changelog if there is anything that may affect the build. Changed paths, modified shared libraries, etc. Test a local build:

svn-buildpackage --svn-ignore-new -us -uc
debc ../build-area/${component}_${version}-1_${arch}.changes | less
lintian -IiE --pedantic ../build-area/${component}_${version}-1_${arch}.changes

When you're happy commit to svn.

svn commit

Prepare the source package for uploading

When the packaging is in good shape, it is time to prepare a Debian source package (.dsc). This will be used in later steps for building for different distributions.

svn-buildpackage -S

releasing packages for multiple distributions

The Debian packages can be uploaded to several places:

• Debian unstable (via Debian mentors)
• Ubuntu ppa (personal package archive)
• software.nikhef.nl/dist/debian

The normal lifecycle of a package is that it starts in Debian unstable. From there it will (eventually) reach Debian stable and Ubuntu. However, since this takes (much) time, we make our packages available for users of several distributions; this is called backporting. The backport targets are given in the table below.

The target distribution is set in debian/changelog. We have to repackage the source package to set the distribution and the release number, which is coding the backport.

The first Debian release of a package always has 1 as the release number, so upstream version 1.2.3 becomes 1.2.3-1. If the Debian packaging for the same upstream version is updated, the release number increments. The table below indicates what the release numbers should be for various backports. The use of the '~' is special: when comparing versions the tilde precedes any other symbol including the empty space, e.g. 1.2.3-1 is considered newer than 1.2.3-1~backport. This is a very useful trick to make sure that the distribution's own package of the same version is preferred over a backport. The number after the backport codename is another sequence number that increments with every backport-specific rebuild of the same upstream-version-and-debian-release.

distribution	version	code name	release tag	notes
Debian	unstable	sid	1	Sid (Debian unstable) is not a backport.
	6	squeeze	1~bpo60+1	This is the Debian style for backports.
Ubuntu	8.04 LTS	hardy	1~hardy1	Hardy is difficult, because the debhelper package is too old (see man dh).
	10.04 LTS	lucid	1~lucid1	The Ubuntu style for backporting is slightly simpler.
	10.10	maverick	1~maverick1
	11.04	natty	1~natty1
	11.10	oneiric	1~oneiric1

Preparing the source packages for backporting

The following script extracts the given UNRELEASED source packages (the output of the first step), updates the changelog, and repackages so they can be build by cowbuilder or cowpoke.

cat > prepare-backports.sh <<'EOF'
#!/bin/sh

# prepare backports for packages
# arguments: list of .dsc files

# action: for each dsc file and for each distribution:
# unpack the dsc file
# dch -v $version-release~$backportcode --distribution $dist "Rebuild for xxx"

# arguments: source package, dist (directory name), target distribution, release code
makesrc() {
    cd $2
    dpkg-source -x $1
    pkg=`basename $1`
    cd `echo $pkg | sed -e 's/\([^_]*\)_\([^-]*\).*/\1-\2/'`
    version=`dpkg-parsechangelog | sed -n 's/^Version: //p'`
    dch --distribution $3 -b -v ${version}~${4} "Rebuild for $2"
    dpkg-buildpackage -S
    cd ../..

}

ubuntu="hardy lucid maverick natty oneiric"
mkdir $ubuntu squeeze sid

while [ $# -ge 1 ]; do
    for dist in $ubuntu; do
	makesrc $1 $dist $dist-backports ${dist}1
    done
    # similar but different for debian
    makesrc $1 squeeze squeeze bpo60+1

    # treat sid specially
    cd sid
    dpkg-source -x $1
    pkg=`basename $1`
    cd `echo $pkg | sed -e 's/\([^_]*\)_\([^-]*\).*/\1-\2/'`
    dch --distribution unstable --release ""
    dpkg-buildpackage -S
    cd ../..

    shift
done
EOF

Running this script unpacks and repacks the sources multiple times in various subdirectories. After building with cowbuilder or cowpoke (cowbuilder on a remote machine) it is no longer required to keep them around.

Starting the remote build for a backport

The reference machine ref-debian6-64.testbed has been configured for cowbuilder for all distributions, in the architectures i386 and amd64. The cowpoke program uploads the sources and launches a remote build, downloads the results and signs the changes files after the build. (See the configuration section on how to set up cowpoke.)

( cd sid
  cowpoke --dist=sid --arch=amd64 ${component}_{version}-1.dsc
  cowpoke --dist=sid --arch=i386 --dpkg-opts=-B ${component}_{version}-1.dsc )
( cd squeeze
  cowpoke --dist=squeeze --arch=amd64 ${component}_{version}-1~bpo60+1.dsc
  cowpoke --dist=squeeze --arch=i386 --dpkg-opts=-B ${component}_{version}-1~bpo60+1.dsc
)

The other builds follow the same pattern. Note: the -B option only builds the architecture-specific packages. This prevents regenerating the source package and architecture-independent packages, which are already produced by the amd64 build. (The ordering is not specific, but if both builds produce the same packages we'll get in trouble later on when uploading to the repository. It refuses to accept packages that it already has, and on top of that the checksums will be different.)

Uploading the builds

The build results are found under ref-debian6-64.testbed:/var/cache/pbuilder/$dist-$arch/results. The dput command is used to perform the uploading (see the configuration section how to set up dput).

Ubuntu ppa

There is no ppa configured for mwsec at the moment.

Putting the results in the main repository

• ssh ref-debian6-64.testbed
• dput mwsec-squeeze /var/cache/pbuilder/squeeze-amd64/results/<component>-<version>-1~bpo60+1_amd64.changes
• dput mwsec-squeeze /var/cache/pbuilder/squeeze-i386/results/<component>-<version>-1~bpo60+1_i386.changes
• rsync -a -vP --delete /var/local/debian-repository/debian/ software.nikhef.nl:/project/srv/www/site/software/html/dist/debian/

Setting up a personal build environment

Before getting anything to work, it is necessary to make several preparations. Luckily, it's just a one-time endeavor.

GPG key

All Debian packaging tools use GPG signing by default. If you already have a gpg key, you can use that. If not, create one. Preferably using a key length of at least 2048 (although 4096 is common these days).

Environment variables

The Debian new maintainers guide suggests to put the following variables in ${HOME}/.bashrc:

DEBEMAIL=youremail@nikhef.nl
DEBFULLNAME="John D. Developer"

You must use values that exactly match your GPG key identity.

For using the debsign tool later on (for signing packages) put these settings in ${HOME}/.devscripts:

DEBSIGN_KEYID=8AAE244E

but fill in the key id of your own GPG key. (You can find it by typing gpg --list-secret-keys.)

Machine environment

Development should be done on either a Debian or Ubuntu machine. Root privileges are not required, but it is necessary that a certain number of developer's tools are installed (besides the build dependencies of the packages you work on, obviously). The New Maintainers' Guide suggests several packages.

apt-get install build-essentials dh-make devscripts autotools-dev cdbs debhelper fakeroot lintian pbuilder \
                cowbuilder quilt

Take care to install recent versions of at least lintian, devscripts and debhelper. Older distributions may not have the required versions, in which case you should consider installing a backport of the right versions.

Setting up pbuilder and cowbuilder is rather advanced, but very worthwhile to do clean-room installations.

Anatomy of a Debian source package

This material is provided to help understand how to package for Debian. The most important thing is to familiarize yourself with the available material for Debian maintainers and developers, but as this can be a bit overwhelming, these notes are somewhat tailored to our situation and may help to clarify.

older material

TODO move these to the relevant sections above

Debian packaging requires a debian/ subdirectory in the root of the upstream source directory. The structure of this directory is described in the Debian New Maintainer's Guide. For the SAC middleware, the debian packaging is maintained in SVN. Debian does not prescribe the exact method of creating packages, as is the case with RPM, and several helper libraries have emerged over time. The most notable are debhelper and the newer CDBS. Both should make it a snap to build packages with.

The following steps are required only once for every source packaged to be debianized. All of this could be done manually, but it is a lot of hassle and prone to errors, so better leave it to the automated tools. Should any package decide to radically change it's nature (hopefully this will never happen!) it may be required to repeat these steps.

Required System: Debian/Ubuntu. Required tools: dh_make.

• Unpack the upstream sources.
• Run dh_make -r cdbs (and answer some questions).
• Review the contents of the debian/ subdirectory.
• Create a subdirectory in SVN under mwsec/packaging/debian/trunk for the new package.
• add the debian directory to this subdirectory
• set the svn:MergeWithUpstream property.

(TODO: review and verify this part!)

Setting up pbuilder for building Debian packages

Debian's pbuilder is a system for building packages in a clean environment. It can use chroot or copy-on-write. In order to set it up so it can build for multiple distributions, some enhancements to the configuration need to be applied.

Because pbuilder needs to be run as root, users are allowed to run /usr/sbin/pbuilder through sudo.

The main installation is on ref-debian6-64.testbed.

FIXME: the new machine has things set up slightly differently than before. We're moving to cowbuilder + cowpoke...

To accommodate Ubuntu builds, the ubuntu-archive-keyring package was installed manually from the Ubuntu archives. Otherwise debootstrap will fail.

To build a package, it should suffice to go to the SVN directory mwsec/packaging/debian/trunk/component/ and run:

DIST=stable svn-buildpackage --svn-builder pdebuild --pbuilder cowbuilder

The cowbuilder will use the copy-on-write root in /var/cache/pbuilder/stable-amd64/base.cow. For other distributions just change DIST=stable, e.g. DIST=oneiric for the 11.10 Ubuntu release.

In order to build a i386 build after the amd64 build, select -B to do a binary build only. Passing arguments is a bit tricky:

ARCH=i386 DIST=stable svn-buildpackage --svn-builder pdebuild --pbuilder cowbuilder --debbuildopts "-us -uc -B"

Creating a repository of packages to be downloaded

Debian offers several tools for generating repositories; some are meant for full-size Debian archives and others for simple, private use. For the SAC middleware the reprepro utility offers enough features while at the same time it is easy to install and maintain.

Basically follow the guidelines on how to set up a repository. TODO: link the pbuilder output to this.

Put the build results in the 'incoming' directory. Use this in $HOME/.dput.cf

[mwsec]
method = local
incoming = /var/www/repos/apt/incoming
allow_unsigned_uploads = 1
run_dinstall = 0

And run dput:

dput mwsec /var/cache/pbuilder/stable-amd64/result/lcmaps_1.4.34-1_amd64.changes 

Process the 'incoming' directory (this should be automated sometime):

reprepro -b /var/www/repos/apt/debian processincoming squeeze lcmaps_1.4.34-1_amd64.changes

(First it may be required to set the distribution in the changes file to 'squeeze' with the changestool command).

Sync the local repository to software.nikhef.nl:

rsync -rvP /var/www/repos/apt/debian/ software.nikhef.nl:/project/srv/www/site/software/html/dist/debian/