Koji setup on the testbed

The Koji system is used by Fedora for builds of packages for Fedora Core and EPEL. It is open source and available as a standard package, and documentation for running a private installation is available.

Koji consists of a number of interrelated components:

Koji Hub, the central XMLRPC service.
A postgresql database as a backend for the hub
Koji Web, a user interface front-end for the hub (other interaction is by the command-line client)
Kojira, managing repository building and clean-up.
One or more builders, on separate machines.

It is possible to install most of these components on separate machines, but our simple setup has only three:

koji-hub.testbed, holding the hub, web, kojira and database
koji-builder.testbed and koji-boulder.testbed, for building i386 and x86_64 packages.

The machines share access to a large NFS share on put.testbed:/mnt/put/koji for storing results and buildroots (the builders only have read-only access). Currently the machines are deployed on the M610 blades (blade13 and blade14) with the possibility for hot migration.

The installation mostly followed the very complete guidance on [1], choosing the SSL authentication option.

A note on memory consumption

The regeneration of repositories which is done by kojira between builds is a memory-hungry task. As builders are configured by default to accept a load of 2 tasks, they may receive two createrepo/mergerepo jobs and run out of memory. If such tasks fail with a message like:

  GenericError: failed to create repo: /usr/bin/createrepo -vd -o /var/lib/koji/tasks/257/257/repo -u http://koji-hub.testbed/kojifiles -i /mnt/koji/repos/dist-fc17-build/45/i386/pkglist -g /mnt/koji/repos/dist-fc17-build/45/groups/comps.xml --update --skip-stat /mnt/koji was killed by signal 9

check /var/log/messages for signs of the OOM killer springing into action.

The builders are configured with 4096MB memory at the moment which should be enough.

SSL Authentication

The authentication system is picky when it comes to Common Names. For users, they have to be the same as user ids, and for servers (only koji-hub.testbed, really) it has to be the fully qualified host name.

A new CA was created for the purpose of handing out certificates (as reuse of existing certificates was hard because of the above restriction, but we may still find a way around that sometime). The CA installation is currently found on bleek.testbed:/srv/koji-pki.

For access to koji-web, the user needs to export the certificate file from there to a pkcs12 file and import it in the browser. For command-line koji use, put it in ~/.koji/clientcert.crt.

Building from git

This step still needs to be automated with a commit hook. The git repository on git://bleek.testbed/koji.git has one subdirectory per component, and each directory contains:

a SPEC file
a Makefile with a 'sources' target, which will fetch (with curl) the sources for the package.

After pushing the updates to this repository, record the commit hash and start the koji build like so (replace the hash with yours):

koji build dist-epel5 'git://bleek.testbed/koji.git?lcmaps/#7a2b193fdd8a26875a16303da609952cccc4cba8'

The format is particular to koji; it is parsed as an URL and that is why the ? and # are used to mark certain elements.

Adding tags, targets and repositories

See the main documentation on the Fedora wiki.

The builds are done against targets, which have source and target tags, which have buildroots populated from repositories. The initial buildroots are set up from virtual packages called 'build' and 'srpm-build' created in koji.

A dump of the history to see what steps are required

  99  koji add-tag dist-epel5
 100  koji add-tag --parent dist-epel5 --arches "i386 x86_64" dist-epel5-build
 102  koji add-external-repo -t dist-epel5-build dist-epel5-external-repo http://mirrors.nl.eu.kernel.org/fedora-epel/5/\$arch/
 103  koji add-external-repo -t dist-epel5-build dist-epel5-centos5-repo http://spiegel.nikhef.nl/mirror/centos/5/os/\$arch/
 104  koji add-target dist-epel5 dist-epel5-build

These groups are going to be passed to 'yum groupinstall' when the buildroot is created.

 106  koji add-group dist-epel5-build build
 107  koji add-group dist-epel5-build srpm-build

They need to be populated

 109  koji -s http://koji.fedoraproject.org/koji list-groups dist-epel5-build
 110  koji -s https://koji.fedoraproject.org/koji list-groups dist-epel5-build

Clone from a typical value for the Fedora Koji systems. Be careful to exclude the fedpkg package which pulls in way too much! It includes mock which causes strange failures with conflicting group ids within the buildroot.

 112  koji add-group-pkg dist-epel5-build build `cat epel5-build-pkgs `
 114  koji add-group-pkg dist-epel5-build srpm-build `cat epel5-srpm-build-pkgs `

Don't forget to add the package!

 140  koji add-pkg --owner okoeroo dist-epel5 lcmaps
 124  koji build dist-epel5 git://bleek.testbed/koji.git\?lcmaps/#a942e490ace1da950cedae2976bf59fa3b3f5038

The git repository must be added to the allowed repositories on the builder in kojid.conf.

Automated builds from tags in mwsec

Triggering a build in svn

the mwsec/packaging/fedora/tags directory is used for triggering builds in Koji. Each component has its own subdirectory, with the version number as a directory below that, so we have e.g.:

svn+ssh://svn@ndpfsvn.nikhef.nl/repos/mwsec/packaging/fedora/tags/lcmaps/1.5.5/

Within that directory, create a Makefile and a spec file. The spec file can be the same as the one from packaging/fedora/trunk for that component. The Makefile can be fairly generic, in most cases this should suffice:

dir = $(shell pwd)
version = $(notdir $(dir))
component = lcmaps-plugins-verify-proxy

sources:
	curl -O http://software.nikhef.nl/security/$(component)/$(component)-$(version).tar.gz
	sed -i -e 's/^Version: .*/Version: $(version)/' $(component).spec

Should a rebuild be necessary because of errors in Koji or in the spec file, update the Release field in the spec file as usual.

Scripts on the svn repository

The svn repository needs a post-commit hook to start koji builds automatically from svn. The svn account also needs a koji certificate with build rights. Network access to the koji-hub is by a DNAT proxy through bleek.

The following scripts are installed on sikkel.nikhef.nl:

/project/srv/svn/repos/mwsec/hooks
-rwxr-xr-x 1 svn apache 1026 Jul 18 14:05 koji-build-trigger
-rwxr-xr-x 1 svn apache 2717 Jul 19 14:27 post-commit

post-commit:

REPOS="$1"
REV="$2"

echo "Commit revision $REV to $REPOS" >&2

PATH=/bin:/usr/bin
export PATH

# We are looking for additions (A) or updates (U) to a specific path to trigger
# a koji build. The 'svnlook changed' output may look like:
## A   packaging/fedora/tags/lcmaps/
## A   packaging/fedora/tags/lcmaps/1.2.3/
## A   packaging/fedora/tags/lcmaps/1.2.3/Makefile
# And it's the middle one we trigger on.

svnlook changed -r $REV "$REPOS" | awk '$1 ~ /(A|U)/ && $2 ~ /packaging\/fedora\/tags\/[[:alnum:]-]+\/[[:alnum:].]+\/.*\.spec$/ { 
    split($2, pe, "/");
    print pe[4], pe[5];
}' | while read component tag ; do
	echo "component: $component; tag: $tag" >&2
    if [ -n "$component" -a -n "$tag" ]; then
	"$REPOS"/hooks/koji-build-trigger $component $tag $REV
    fi
done

koji-build-trigger:

#!/bin/sh

component="$1"
tag="$2"
revision="$3"

if [ -z "$component" ]; then
    echo "Empty component specified for koji build." >&2
    exit 1
fi

if [ -z "$tag" ]; then
    echo "Empty tag specified for koji build." >&2
    exit 1
fi

if [ -z "$revision" ]; then
    echo "Empty revision specified for koji build." >&2
    exit 1
fi

/usr/bin/koji -c /project/srv/svn/home/.koji/koji.conf build --nowait dist-epel5 svn+http://ndpfsvn.nikhef.nl/ro/mwsec\?packaging/fedora/tags/${component}/${tag}#$revision
/usr/bin/koji -c /project/srv/svn/home/.koji/koji.conf build --nowait dist-epel6 svn+http://ndpfsvn.nikhef.nl/ro/mwsec\?packaging/fedora/tags/${component}/${tag}#$revision
/usr/bin/koji -c /project/srv/svn/home/.koji/koji.conf build --nowait dist-fc16 svn+http://ndpfsvn.nikhef.nl/ro/mwsec\?packaging/fedora/tags/${component}/${tag}#$revision
/usr/bin/koji -c /project/srv/svn/home/.koji/koji.conf build --nowait dist-fc17 svn+http://ndpfsvn.nikhef.nl/ro/mwsec\?packaging/fedora/tags/${component}/${tag}#$revision

koji.conf:

[koji]

;configuration for koji cli tool

;url of XMLRPC server
server = https://bleek.nikhef.nl:8443/kojihub

;url of web interface
weburl = https://bleek.nikhef.nl:8443/koji

;url of package download site
topurl = https://bleek.nikhef.nl:8443/kojifiles/packages/

;path to the koji top directory
topdir = /mnt/koji

;configuration for Kerberos authentication

;the service name of the principal being used by the hub
;krbservice = host

;configuration for SSL authentication

;client certificate
cert = ~/.koji/clientcert.crt

;certificate of the CA that issued the client certificate
ca = ~/.koji/koji_ca_cert.crt

;certificate of the CA that issued the HTTP server certificate
serverca = ~/.koji/koji_ca_cert.crt

iptables rule on bleek:

iptables -t nat -A PREROUTING --proto tcp --dport 8443 --destination 194.171.96.17 -j DNAT --to-destination 10.198.8.7:443

Nightly builds from mwsec/trunk

The dedicated account kojinb for nightly builds is added on bleek.

At the moment nightly builds are run from a crontab of account dennisvd.

01 2 * * * ${HOME}/bin/koji-nightly

The koji-nightly script:

#!/bin/sh

PATH=/bin:/usr/bin

components="
lcas
lcmaps
xacml
ees
glexec
jobrepository
lcas-lcmaps-gt4-interface
lcas-plugins-basic
lcas-plugins-check-executable
lcas-plugins-voms
lcmaps-plugins-afs
lcmaps-plugins-basic
lcmaps-plugins-c-pep
lcmaps-plugins-gums
lcmaps-plugins-jobrep
lcmaps-plugins-scas-client
lcmaps-plugins-tracking-groupid
lcmaps-plugins-verify-proxy
lcmaps-plugins-voms
scas
"

dists="epel5 epel6 fc16 fc17"

rev=`svn info http://ndpfsvn.nikhef.nl/ro/mwsec/ | sed -n 's/^Revision: //p'`

for i in $components ; do
    for j in $dists ; do
	koji -c ${HOME}/.koji/nightly.conf build --nowait nightly-$j svn+http://ndpfsvn.nikhef.nl/ro/mwsec\?packaging/fedora/branches/nightly-builds/$i/#$rev
    done
done

Preventing repeat builds

Koji will refuse build artefacts that it already has. The nightly builds are versioned according to the svn revision of the latest update of the tree to build. That means that if no updates are done to the tree, there is no sense in repeating the build. The cron script should be enhanced to deal with this situation.

odd details

On fc17 libtoolize may fail with a strange error message:

libtoolize: can not copy `/usr/share/libtool/config/ltmain.sh' to `./'

which is due to 'tar' missing on the system. See bug https://bugzilla.redhat.com/show_bug.cgi?id=794675. The resolution is to include tar as a required package in the srpm-build group.

Koji Testbed

Contents

Koji setup on the testbed

A note on memory consumption

SSL Authentication

Building from git

Adding tags, targets and repositories

Automated builds from tags in mwsec

Triggering a build in svn

Scripts on the svn repository

Nightly builds from mwsec/trunk

Preventing repeat builds

odd details

Navigation menu

Page actions

Page actions

Personal tools

Navigation

Search

Tools