Koji setup on the testbed

The Koji system is used by Fedora for builds of packages for Fedora Core and EPEL. It is open source and available as a standard package, and documentation for running a private installation is available.

Koji consists of a number of interrelated components:

Koji Hub, the central XMLRPC service.
A postgresql database as a backend for the hub
Koji Web, a user interface front-end for the hub (other interaction is by the command-line client)
Kojira, managing repository building and clean-up.
One or more builders, on separate machines.

It is possible to install most of these components on separate machines, but our simple setup has only three:

koji-hub.testbed, holding the hub, web, kojira and database
koji-builder.testbed and koji-boulder.testbed, for building i386 and x86_64 packages.
(temporarily, a third builder has been added: koji-monster.testbed as part of a performance evaluation. It's been taken out of the active pool)

The machines share access to a large NFS share on put.testbed:/mnt/put/koji for storing results and buildroots (the builders only have read-only access). Currently the machines are deployed on the M610 blades (blade13 and blade14) with the possibility for hot migration.

The installation mostly followed the very complete guidance on [1], choosing the SSL authentication option.

Using a squid proxy for external repositories

The builds use mock to install a base set of packages to fulfill the BuildRequires in the spec file. These packages come from CentOS and Fedora mirrors, and in order to reduce the load on the mirrors a local squid caching proxy is configured to store up to 100GB of popular packages. To use the proxy, set

yum_proxy = http://span.testbed:3128/

in /etc/kojid/kojid.conf on all builders.

memory and disk space requirement

The regeneration of repositories which is done by kojira between builds is a memory-hungry task. As builders are configured by default to accept a load of 2 tasks, they may receive two createrepo/mergerepo jobs and run out of memory. If such tasks fail with a message like:

  GenericError: failed to create repo: /usr/bin/createrepo -vd -o /var/lib/koji/tasks/257/257/repo -u http://koji-hub.testbed/kojifiles -i /mnt/koji/repos/dist-fc17-build/45/i386/pkglist -g /mnt/koji/repos/dist-fc17-build/45/groups/comps.xml --update --skip-stat /mnt/koji was killed by signal 9

check /var/log/messages for signs of the OOM killer springing into action.

The builders are configured with 4096MB memory at the moment which should be enough.

The disk space on the builders could easily run out, as there is no automatic clean-up of used build directories.

The builders have a tmpwatch script in /etc/cron.daily that erases old data (older than 4 hours)

/usr/bin/tmpwatch -caMf 4 /mnt/mock/

Another approach is being tried at the moment on koji-monster.testbed, which runs every hour and deletes any directory that does not have the buildroot.lock file:

for i in /mnt/mock/* ; do
	if [ ! -f $i/buildroot.lock ]; then
		rm -rf $i
	fi
done

SSL Authentication

The authentication system is picky when it comes to Common Names. For users, they have to be the same as user ids, and for servers (only koji-hub.testbed, really) it has to be the fully qualified host name.

A new CA was created for the purpose of handing out certificates (as reuse of existing certificates was hard because of the above restriction, but we may still find a way around that sometime). The CA installation is currently found on bleek.testbed:/srv/koji-pki.

For access to koji-web, the user needs to export the certificate file from there to a pkcs12 file and import it in the browser. For command-line koji use, put it in ~/.koji/clientcert.crt.

Adding tags, targets and repositories

See the main documentation on the Fedora wiki.

The builds are done against targets, which have source and target tags, which have buildroots populated from repositories. The initial buildroots are set up from virtual packages called 'build' and 'srpm-build' created in koji.

A dump of the history to see what steps are required

  99  koji add-tag dist-epel5
 100  koji add-tag --parent dist-epel5 --arches "i386 x86_64" dist-epel5-build
 102  koji add-external-repo -t dist-epel5-build dist-epel5-external-repo http://mirrors.nl.eu.kernel.org/fedora-epel/5/\$arch/
 103  koji add-external-repo -t dist-epel5-build dist-epel5-centos5-repo http://spiegel.nikhef.nl/mirror/centos/5/os/\$arch/
 104  koji add-target dist-epel5 dist-epel5-build

These groups are going to be passed to 'yum groupinstall' when the buildroot is created.

 106  koji add-group dist-epel5-build build
 107  koji add-group dist-epel5-build srpm-build

They need to be populated. Clone from a typical value for the Fedora Koji systems, e.g.

 109  koji -s http://koji.fedoraproject.org/kojihub/ list-groups dist-5E-epel-build

Use the output to produce a proper list of packages. Be careful to exclude the fedpkg package which pulls in way too much! It includes mock which causes strange failures with conflicting group ids within the buildroot.

 112  koji add-group-pkg dist-epel5-build build `cat epel5-build-pkgs `
 114  koji add-group-pkg dist-epel5-build srpm-build `cat epel5-srpm-build-pkgs `

Don't forget to add the package!

 140  koji add-pkg --owner okoeroo dist-epel5 lcmaps
 124  koji build dist-epel5 git://bleek.testbed/koji.git\?lcmaps/#a942e490ace1da950cedae2976bf59fa3b3f5038

The git repository must be added to the allowed repositories on the builder in kojid.conf.

Automated builds from tags in mwsec

Building from git (deprecated)

This step still needs to be automated with a commit hook. The git repository on git://bleek.testbed/koji.git has one subdirectory per component, and each directory contains:

a SPEC file
a Makefile with a 'sources' target, which will fetch (with curl) the sources for the package.

After pushing the updates to this repository, record the commit hash and start the koji build like so (replace the hash with yours):

koji build dist-epel5 'git://bleek.testbed/koji.git?lcmaps/#7a2b193fdd8a26875a16303da609952cccc4cba8'

The format is particular to koji; it is parsed as an URL and that is why the ? and # are used to mark certain elements.

Triggering a build in svn

the mwsec/packaging/fedora/tags directory is used for triggering builds in Koji. Each component has its own subdirectory, with the version number as a directory below that, so we have e.g.:

svn+ssh://svn@ndpfsvn.nikhef.nl/repos/mwsec/packaging/fedora/tags/lcmaps/1.5.5/

Within that directory, create a Makefile and a spec file. The spec file can be the same as the one from packaging/fedora/trunk for that component. The Makefile can be fairly generic, in most cases this should suffice:

dir = $(shell pwd)
version = $(notdir $(dir))
component = lcmaps-plugins-verify-proxy

sources:
	curl -O http://software.nikhef.nl/security/$(component)/$(component)-$(version).tar.gz
	sed -i -e 's/^Version: .*/Version: $(version)/' $(component).spec

Should a rebuild be necessary because of errors in Koji or in the spec file, update the Release field in the spec file as usual.

Scripts on the svn repository

The svn repository needs a post-commit hook to start koji builds automatically from svn. The svn account also needs a koji certificate with build rights. Network access to the koji-hub is by a DNAT proxy through bleek.

The following scripts are installed on sikkel.nikhef.nl:

/project/srv/svn/repos/mwsec/hooks
-rwxr-xr-x 1 svn apache 1026 Jul 18 14:05 koji-build-trigger
-rwxr-xr-x 1 svn apache 2717 Jul 19 14:27 post-commit

post-commit:

REPOS="$1"
REV="$2"

echo "Commit revision $REV to $REPOS" >&2

PATH=/bin:/usr/bin
export PATH

# We are looking for additions (A) or updates (U) to a specific path to trigger
# a koji build. The 'svnlook changed' output may look like:
## A   packaging/fedora/tags/lcmaps/
## A   packaging/fedora/tags/lcmaps/1.2.3/
## A   packaging/fedora/tags/lcmaps/1.2.3/Makefile
# And it's the middle one we trigger on.

svnlook changed -r $REV "$REPOS" | awk '$1 ~ /(A|U)/ && $2 ~ /packaging\/fedora\/tags\/[[:alnum:]-]+\/[[:alnum:].]+\/.*\.spec$/ { 
    split($2, pe, "/");
    print pe[4], pe[5];
}' | while read component tag ; do
	echo "component: $component; tag: $tag" >&2
    if [ -n "$component" -a -n "$tag" ]; then
	"$REPOS"/hooks/koji-build-trigger $component $tag $REV
    fi
done

koji-build-trigger:

#!/bin/sh

component="$1"
tag="$2"
revision="$3"

if [ -z "$component" ]; then
    echo "Empty component specified for koji build." >&2
    exit 1
fi

if [ -z "$tag" ]; then
    echo "Empty tag specified for koji build." >&2
    exit 1
fi

if [ -z "$revision" ]; then
    echo "Empty revision specified for koji build." >&2
    exit 1
fi

/usr/bin/koji -c /project/srv/svn/home/.koji/koji.conf build --nowait dist-epel5 svn+http://ndpfsvn.nikhef.nl/ro/mwsec\?packaging/fedora/tags/${component}/${tag}#$revision
/usr/bin/koji -c /project/srv/svn/home/.koji/koji.conf build --nowait dist-epel6 svn+http://ndpfsvn.nikhef.nl/ro/mwsec\?packaging/fedora/tags/${component}/${tag}#$revision
/usr/bin/koji -c /project/srv/svn/home/.koji/koji.conf build --nowait dist-fc16 svn+http://ndpfsvn.nikhef.nl/ro/mwsec\?packaging/fedora/tags/${component}/${tag}#$revision
/usr/bin/koji -c /project/srv/svn/home/.koji/koji.conf build --nowait dist-fc17 svn+http://ndpfsvn.nikhef.nl/ro/mwsec\?packaging/fedora/tags/${component}/${tag}#$revision

koji.conf:

[koji]

;configuration for koji cli tool

;url of XMLRPC server
server = https://bleek.nikhef.nl:8443/kojihub

;url of web interface
weburl = https://bleek.nikhef.nl:8443/koji

;url of package download site
topurl = https://bleek.nikhef.nl:8443/kojifiles/packages/

;path to the koji top directory
topdir = /mnt/koji

;configuration for Kerberos authentication

;the service name of the principal being used by the hub
;krbservice = host

;configuration for SSL authentication

;client certificate
cert = ~/.koji/clientcert.crt

;certificate of the CA that issued the client certificate
ca = ~/.koji/koji_ca_cert.crt

;certificate of the CA that issued the HTTP server certificate
serverca = ~/.koji/koji_ca_cert.crt

iptables rule on bleek:

iptables -t nat -A PREROUTING --proto tcp --dport 8443 --destination 194.171.96.17 -j DNAT --to-destination 10.198.8.7:443

Generating distribution repos from koji

To render the products of koji into something that can be installed by yum, we use the mash utility installed on koji-hub.testbed. The output of mash should be send to /mnt/koji/dists, and from there synced to software.nikhef.nl.

The mash configuration has one main config file and one file per distribution: /etc/mash/mash.conf

[defaults]
configdir = /etc/mash
buildhost = http://koji-hub.testbed/kojihub
repodir = file:///mnt/koji
use_sqlite = True
use_repoview = True

And e.g. /etc/mash/epel5.mash (the other ones are similar):

[epel5]
rpm_path = %(arch)s/os/Packages
repodata_path = %(arch)s/os/
source_path = source/SRPMS
debuginfo = True
multilib = True
multilib_method = devel
tag = dist-epel5
inherit = True
strict_keys = False
repoviewurl = http://koji-hub.testbed/repo-view/%(arch)s/os/
repoviewtitle = "MWSEC for EPEL5 - %(arch)s"
arches = i386 x86_64
delta = False
hash_packages = False

To run mash, simply use the following command:

for i in epel{5,6} fc{16,17} ; do mash -o /mnt/koji/dists $i ; done

and rsync to software.nikhef.nl:

rsync -a --delete /mnt/koji/dists/ software.nikhef.nl:/project/srv/www/site/software/html/experimental/mwsec/

Nightly builds from mwsec/trunk

The dedicated account kojinb for nightly builds is added on bleek.

At the moment nightly builds are run from a crontab of account dennisvd.

01 2 * * * ${HOME}/bin/koji-nightly

The koji-nightly script:

#!/bin/sh

PATH=/bin:/usr/bin

components="
lcas
lcmaps
ees
glexec
jobrepository
lcas-lcmaps-gt4-interface
lcas-plugins-basic
lcas-plugins-check-executable
lcas-plugins-voms
lcmaps-plugins-afs
lcmaps-plugins-basic
lcmaps-plugins-c-pep
lcmaps-plugins-gums
lcmaps-plugins-jobrep
lcmaps-plugins-scas-client
lcmaps-plugins-tracking-groupid
lcmaps-plugins-verify-proxy
lcmaps-plugins-voms
scas
"

dists="epel5 epel6 fc16 fc17"

rev=`svn info http://ndpfsvn.nikhef.nl/ro/mwsec/ | sed -n 's/^Revision: //p'`

for i in $components ; do
    for j in $dists ; do
	koji -c ${HOME}/.koji/nightly.conf build --nowait nightly-$j svn+http://ndpfsvn.nikhef.nl/ro/mwsec\?packaging/fedora/branches/nightly-builds/$i/#$rev
    done
done

Each component directory in the nightly-builds branch contains the same set of files: a Makefile:

component = $(notdir $(shell pwd))

sources:
	./fetch-sources $(component)

And a fetch-sources script:

#!/bin/sh

set -e

component=$1

svn co http://ndpfsvn.nikhef.nl/ro/mwsec/trunk/${component}
cd ${component}
revision=`svn info | sed -n 's/^Last Changed Rev: //p'`

if [ -z $revision ]; then
   echo "Could not retrieve revision from svn info" >&2
   echo "svn info:" >&2
   svn info >&2
   exit 1
fi

./bootstrap
mkdir build
cd build 
../configure
make dist
mv *.tar.gz ..
cd ..

version=`echo ${component}-*.tar.gz | sed -n "s/${component}-\\([0-9.]*\\)\\.tar\\.gz/\\1/p"`

if [ -z $version ]; then
   echo "Could not retrieve version from generated tarball" >&2
   echo "tarballs: " ${component}-*.tar.gz >&2
   exit 1
fi

mv ${component}-*.tar.gz ..
cd ..

sed -i -e "s/^Version: .*/Version: $version/" -e "s/^Release: .*/Release: 0.$revision%{?dist}/" ${component}.spec

The list of components misses just two packages: argus-pep-api-c, which is not under our active development, and xacml, which has some problems at this moment; gsoap-devel needs to be added to the build-srpm group (done) and as the nightly build does the bootstrapping as well, the autoconf for the epel5 tag is too old according to configure.in. Whether this is really the case remains to be seen.

Preventing repeat builds

Koji will refuse build artefacts that it already has. The nightly builds are versioned according to the svn revision of the latest update of the tree to build. That means that if no updates are done to the tree, there is no sense in repeating the build. The cron script should be enhanced to deal with this situation.

odd details

On fc17 libtoolize may fail with a strange error message:

libtoolize: can not copy `/usr/share/libtool/config/ltmain.sh' to `./'

which is due to 'tar' missing on the system. See bug https://bugzilla.redhat.com/show_bug.cgi?id=794675. The resolution is to include tar as a required package in the srpm-build group.

Koji Testbed

Contents