Koji Testbed

Koji setup on the testbed

The Koji system is used by Fedora for builds of packages for Fedora Core and EPEL. It is open source and available as a standard package, and documentation for running a private installation is available.

Koji consists of a number of interrelated components:

• Koji Hub, the central XMLRPC service.
• A postgresql database as a backend for the hub
• Koji Web, a user interface front-end for the hub (other interaction is by the command-line client)
• Kojira, managing repository building and clean-up.
• One or more builders, on separate machines.

It is possible to install most of these components on separate machines, but our simple setup has only three:

• koji-hub.testbed, holding the hub, web, kojira and database
• koji-builder.testbed and koji-boulder.testbed, for building i386 and x86_64 packages.
• (temporarily, a third builder has been added: koji-monster.testbed as part of a performance evaluation. It's been taken out of the active pool)

The machines share access to a large NFS share on put.testbed:/mnt/put/koji for storing results and buildroots (the builders only have read-only access). Currently the machines are deployed on the M610 blades (blade13 and blade14) with the possibility for hot migration.

The installation mostly followed the very complete guidance on [1], choosing the SSL authentication option.

Using a squid proxy for external repositories

The builds use mock to install a base set of packages to fulfill the BuildRequires in the spec file. These packages come from CentOS and Fedora mirrors, and in order to reduce the load on the mirrors a local squid caching proxy is configured to store up to 100GB of popular packages. To use the proxy, set

yum_proxy = http://span.testbed:3128/

in /etc/kojid/kojid.conf on all builders.

memory and disk space requirement

The builders are configured with 4096MB memory at the moment which should be enough (but see the #Troubleshooting section below).

The disk space on the builders could easily run out, as there is no automatic clean-up of used build directories.

The builders have a tmpwatch script in /etc/cron.daily that erases old data (older than 4 hours)

/usr/bin/tmpwatch -caMf 4 /mnt/mock/

SSL Authentication

The authentication system is picky when it comes to Common Names. For users, they have to be the same as user ids, and for servers (only koji-hub.testbed, really) it has to be the fully qualified host name.

A new CA was created for the purpose of handing out certificates (as reuse of existing certificates was hard because of the above restriction, but we may still find a way around that sometime). The CA installation is currently found on bleek.testbed:/srv/koji-pki.

For access to koji-web, the user needs to export the certificate file from there to a pkcs12 file and import it in the browser. For command-line koji use, put it in ~/.koji/clientcert.crt.

Adding tags, targets and repositories

See the main documentation on the Fedora wiki.

The builds are done against targets, which have source and target tags, which have buildroots populated from repositories. The initial buildroots are set up from virtual packages called 'build' and 'srpm-build' created in koji.

These groups can be inherited from earlier releases by using the 'add-tag-inheritance' command.

koji add-tag-inheritance --maxdepth 1 --priority 2 dist-fc18-build dist-fc17-build

(I'm not too sure whath maxdepth and priority mean here; but Fedora's own koji lists the build group for FC18 as going all the way back to FC9.)

Steps to follow if you need to do this from scratch

A dump of the history to see what steps are required

  99  koji add-tag dist-epel5
 100  koji add-tag --parent dist-epel5 --arches "i386 x86_64" dist-epel5-build
 102  koji add-external-repo -t dist-epel5-build \
      dist-epel5-external-repo http://mirrors.nl.eu.kernel.org/fedora-epel/5/\$arch/
 103  koji add-external-repo -t dist-epel5-build dist-epel5-buildsys-macros-repo \
      http://buildsys.fedoraproject.org/buildgroups/rhel5/\$arch/
 104  koji add-external-repo -t dist-epel5-build dist-epel5-centos5-repo \
      http://spiegel.nikhef.nl/mirror/centos/5/os/\$arch/
 105  koji add-target dist-epel5 dist-epel5-build

NOTE: the buildsys external repo is ONLY needed on EPEL5 to provide the correct buildsys-macros package. The standard CentOS package causes the dist macro to expand to el5.centos. By providing this external Fedora repo and putting it before the centos5 repo, the correct buildsys-macros is installed. See also https://fedoraproject.org/wiki/Koji/KojiMisc#.25dist_tags. If needed (i.e. when the ordering turns out to be wrong) we can provide explicit priorities using a -p flag to add-external-repo. Check with

koji taginfo dist-epel5-build

These groups are going to be passed to 'yum groupinstall' when the buildroot is created.

 106  koji add-group dist-epel5-build build
 107  koji add-group dist-epel5-build srpm-build

They need to be populated. Clone from a typical value for the Fedora Koji systems, e.g.

 109  koji -s http://koji.fedoraproject.org/kojihub/ list-groups dist-5E-epel-build

Use the output to produce a proper list of packages. Be careful to exclude the fedpkg package which pulls in way too much! It includes mock which causes strange failures with conflicting group ids within the buildroot.

 112  koji add-group-pkg dist-epel5-build build `cat epel5-build-pkgs `
 114  koji add-group-pkg dist-epel5-build srpm-build `cat epel5-srpm-build-pkgs `

Don't forget to add the package!

 140  koji add-pkg --owner okoeroo dist-epel5 lcmaps
 124  koji build dist-epel5 git://bleek.testbed/koji.git\?lcmaps/#a942e490ace1da950cedae2976bf59fa3b3f5038

The git repository must be added to the allowed repositories on the builder in kojid.conf.

Automated builds from tags in mwsec

Building from git (deprecated)

This step still needs to be automated with a commit hook. The git repository on git://bleek.testbed/koji.git has one subdirectory per component, and each directory contains:

• a SPEC file
• a Makefile with a 'sources' target, which will fetch (with curl) the sources for the package.

After pushing the updates to this repository, record the commit hash and start the koji build like so (replace the hash with yours):

koji build dist-epel5 'git://bleek.testbed/koji.git?lcmaps/#7a2b193fdd8a26875a16303da609952cccc4cba8'

The format is particular to koji; it is parsed as an URL and that is why the ? and # are used to mark certain elements.

Triggering a build in svn

the mwsec/packaging/fedora/tags directory is used for triggering builds in Koji. Each component has its own subdirectory, with the version number as a directory below that, so we have e.g.:

svn+ssh://svn@ndpfsvn.nikhef.nl/repos/mwsec/packaging/fedora/tags/lcmaps/1.5.5/

Within that directory, create a Makefile and a spec file. The spec file can be the same as the one from packaging/fedora/trunk for that component. The Makefile can be fairly generic, in most cases this should suffice:

dir = $(shell pwd)
version = $(notdir $(dir))
component = lcmaps-plugins-verify-proxy

sources:
	curl -O http://software.nikhef.nl/security/$(component)/$(component)-$(version).tar.gz
	sed -i -e 's/^Version: .*/Version: $(version)/' $(component).spec

Should a rebuild be necessary because of errors in Koji or in the spec file, update the Release field in the spec file as usual.

Scripts on the svn repository

The svn repository needs a post-commit hook to start koji builds automatically from svn. The svn account also needs a koji certificate with build rights. Network access to the koji-hub is by a DNAT proxy through bleek.

The following scripts are installed on sikkel.nikhef.nl:

/project/srv/svn/repos/mwsec/hooks
-rwxr-xr-x 1 svn apache 1026 Jul 18 14:05 koji-build-trigger
-rwxr-xr-x 1 svn apache 2717 Jul 19 14:27 post-commit

• post-commit:

REPOS="$1"
REV="$2"

echo "Commit revision $REV to $REPOS" >&2

PATH=/bin:/usr/bin
export PATH

# We are looking for additions (A) or updates (U) to a specific path to trigger
# a koji build. The 'svnlook changed' output may look like:
## A   packaging/fedora/tags/lcmaps/
## A   packaging/fedora/tags/lcmaps/1.2.3/
## A   packaging/fedora/tags/lcmaps/1.2.3/Makefile
# And it's the middle one we trigger on.

svnlook changed -r $REV "$REPOS" | \
    awk '$1 ~ /(A|U)/ && $2 ~ /packaging\/fedora\/tags\/[[:alnum:]-]+\/[[:alnum:].]+\/.*\.spec$/ { 
    split($2, pe, "/");
    print pe[4], pe[5];
}' | while read component tag ; do
	echo "component: $component; tag: $tag" >&2
    if [ -n "$component" -a -n "$tag" ]; then
	"$REPOS"/hooks/koji-build-trigger $component $tag $REV
    fi
done

• koji-build-trigger:

#!/bin/sh

component="$1"
tag="$2"
revision="$3"

if [ -z "$component" ]; then
    echo "Empty component specified for koji build." >&2
    exit 1
fi

if [ -z "$tag" ]; then
    echo "Empty tag specified for koji build." >&2
    exit 1
fi

if [ -z "$revision" ]; then
    echo "Empty revision specified for koji build." >&2
    exit 1
fi

/usr/bin/koji -c /project/srv/svn/home/.koji/koji.conf build --nowait dist-epel5 \
    svn+http://ndpfsvn.nikhef.nl/ro/mwsec\?packaging/fedora/tags/${component}/${tag}#$revision
/usr/bin/koji -c /project/srv/svn/home/.koji/koji.conf build --nowait dist-epel6 \
    svn+http://ndpfsvn.nikhef.nl/ro/mwsec\?packaging/fedora/tags/${component}/${tag}#$revision
/usr/bin/koji -c /project/srv/svn/home/.koji/koji.conf build --nowait dist-fc16 \
    svn+http://ndpfsvn.nikhef.nl/ro/mwsec\?packaging/fedora/tags/${component}/${tag}#$revision
/usr/bin/koji -c /project/srv/svn/home/.koji/koji.conf build --nowait dist-fc17 \
    svn+http://ndpfsvn.nikhef.nl/ro/mwsec\?packaging/fedora/tags/${component}/${tag}#$revision

• koji.conf:

[koji]

;configuration for koji cli tool

;url of XMLRPC server
server = https://bleek.nikhef.nl:8443/kojihub

;url of web interface
weburl = https://bleek.nikhef.nl:8443/koji

;url of package download site
topurl = https://bleek.nikhef.nl:8443/kojifiles/packages/

;path to the koji top directory
topdir = /mnt/koji

;configuration for Kerberos authentication

;the service name of the principal being used by the hub
;krbservice = host

;configuration for SSL authentication

;client certificate
cert = ~/.koji/clientcert.crt

;certificate of the CA that issued the client certificate
ca = ~/.koji/koji_ca_cert.crt

;certificate of the CA that issued the HTTP server certificate
serverca = ~/.koji/koji_ca_cert.crt

• iptables rule on bleek:

iptables -t nat -A PREROUTING --proto tcp --dport 8443 --destination 194.171.96.17 -j DNAT --to-destination 10.198.8.7:443

Signing RPMs with sigul

Before the RPMs that Koji produces are distributed, they should be signed. The YUM configuration can then be configured to use GPG to verify the origin of the RPMs upon installing.

Generating distribution repos from koji

To render the products of koji into something that can be installed by yum, we use the mash utility installed on koji-hub.testbed. The repositories are made using mash on an NFS share from bleek.nikhef.nl:

/srv/project/mwsec  10.198.8.7(rw,async,no_root_squash,insecure)

bleek.testbed:/srv/project/mwsec        /mnt/mwsec      nfs defaults  0 0

The output of mash should be send to /mnt/mwsec/mash. We create symlinks to the different distributions in /mnt/mwsec/rpm to allow syncing to software.nikhef.nl, the reason is that mash cannot easily put testing RPMs in ./testing/epel5/ and release RPMs in ./epel5/ since the directory is taken directly from the section name in the mwsec.mash config file:

lrwxrwxrwx   rpm/epel5 -> ../mash/epel5-release
lrwxrwxrwx   rpm/epel6 -> ../mash/epel6-release
lrwxrwxrwx   rpm/fc16 -> ../mash/fc16-release
lrwxrwxrwx   rpm/fc17 -> ../mash/fc17-release
drwxrwsr-x   rpm/testing
lrwxrwxrwx   rpm/testing/epel5 -> ../../mash/epel5-testing
lrwxrwxrwx   rpm/testing/epel6 -> ../../mash/epel6-testing
lrwxrwxrwx   rpm/testing/fc16 -> ../../mash/fc16-testing
lrwxrwxrwx   rpm/testing/fc17 -> ../../mash/fc17-testing

For each distribution we create two additional tags -release and -testing:

koji add-tag --parent dist-epel5 dist-epel5-release
koji add-tag --parent dist-epel5 dist-epel5-testing

and likewise for epel6, fc16 and fc17. Succesful builds can be added to either testing or release:

koji tag-pkg dist-epel5-release glexec-0.9.7-1.el5

Or moved from testing to release

koji move-pkg dist-epel5-testing dist-epel5-release lcmaps-1.5.7-1.el5

The mash configuration has one main config file and multiple distribution files: /etc/mash/mash.conf

[defaults]
configdir = /etc/mash
buildhost = http://koji-hub.testbed/kojihub
repodir = file:///mnt/koji
use_sqlite = True
use_repoview = True

And /etc/mash/mwsec.mash containing the different distributions:

[epel5-release]
rpm_path = %(arch)s
repodata_path = %(arch)s/
source_path = SRPMS
debuginfo = True
# Do not create i386s in x86_64
multilib = False
multilib_method = devel
# Use old-style sha hash for EPEL5 only
hash = sha
tag = dist-epel5-release
# Next one is needed to prevent mixing of testing and release.
inherit = False
strict_keys = False
#keys = 22B3B81A, 1ACA3465, A82BA4B7, 069C8460, 97A1071F, E8E40FDE, 57BBCCBA, D22E77F2, 4EBFC273, 0B86274E, 6DF2196F, DF9B0AE9
# Note: repoviewurl used in latest-feed.xml for RPM location
repoviewurl = /dist/mwsec/rpm/epel5/%(arch)s
repoviewtitle = MWSEC for EPEL5 - %(arch)s
arches = i386 x86_64
delta = False
# Change distro_tags as fedora-release version gets bumped
#distro_tags = cpe:/o:fedoraproject:fedora:17 rawhide
hash_packages = False
# This forces to also create older version of same package
latest = False

The testing repository is the same except:

tag = dist-epel5-testing
repoviewurl = /dist/mwsec/rpm/testing/epel5/%(arch)s
repoviewtitle = MWSEC Testing for EPEL5 - %(arch)s

The epel6, fc16 and fc17 are the same except that they do not override the hash =

To run mash, simply use the following command, NOTE: it's best to have a umask 002 setting

for dist in epel{5,6} fc{16,17} ; do
    mash -o /mnt/mwsec/mash ${dist}-release
    mash -o /mnt/mwsec/mash ${dist}-testing
done

Note: Mash completely removes and recreates the output directories! For epel5 we have old builds which should afterwards be merged back in, we use a tarball with symlinks in /mnt/mwsec/old_releases.tgz. The relative symlinks point to the actual files in /mnt/mwsec/mock/. The directory /mnt/mwsec/mock/ is a read-only NFS share on bleek.nikhef.nl:

/srv/project/mock   10.198.0.0/16(ro,async,root_squash,insecure)

bleek.testbed:/srv/project/mock         /mnt/mock       nfs defaults  0 0

By having the mock and mwsec directories at the same level, we can use relative symlinks from one to the other both on bleek.nikhef.nl and koji-hub.testbed.

See SAC_software_procedures#Updating the mash repositories for a practical procedure.

Nightly builds from mwsec/trunk

The dedicated account kojinb for nightly builds is added on bleek.

At the moment nightly builds are run from a crontab of account dennisvd.

01 2 * * * ${HOME}/bin/koji-nightly

The koji-nightly script:

#!/bin/sh

PATH=/bin:/usr/bin

components="
lcas
lcmaps
ees
glexec
jobrepository
lcas-lcmaps-gt4-interface
lcas-plugins-basic
lcas-plugins-check-executable
lcas-plugins-voms
lcmaps-plugins-afs
lcmaps-plugins-basic
lcmaps-plugins-c-pep
lcmaps-plugins-gums
lcmaps-plugins-jobrep
lcmaps-plugins-scas-client
lcmaps-plugins-tracking-groupid
lcmaps-plugins-verify-proxy
lcmaps-plugins-voms
scas
"

dists="epel5 epel6 fc16 fc17"

rev=`svn info http://ndpfsvn.nikhef.nl/ro/mwsec/ | sed -n 's/^Revision: //p'`

for i in $components ; do
    for j in $dists ; do
	koji -c ${HOME}/.koji/nightly.conf build --nowait nightly-$j \
           svn+http://ndpfsvn.nikhef.nl/ro/mwsec\?packaging/fedora/branches/nightly-builds/$i/#$rev
    done
done

Each component directory in the nightly-builds branch contains the same set of files: a Makefile:

component = $(notdir $(shell pwd))

sources:
	./fetch-sources $(component)

And a fetch-sources script:

#!/bin/sh

set -e

component=$1

svn co http://ndpfsvn.nikhef.nl/ro/mwsec/trunk/${component}
cd ${component}
revision=`svn info | sed -n 's/^Last Changed Rev: //p'`

if [ -z $revision ]; then
   echo "Could not retrieve revision from svn info" >&2
   echo "svn info:" >&2
   svn info >&2
   exit 1
fi

./bootstrap
mkdir build
cd build 
../configure
make dist
mv ${component}-*.tar.gz ..
cd ..

version=`echo ${component}-*.tar.gz | sed -n "s/${component}-\\([0-9.]*\\)\\.tar\\.gz/\\1/p"`

if [ -z $version ]; then
   echo "Could not retrieve version from generated tarball" >&2
   echo "tarballs: " ${component}-*.tar.gz >&2
   exit 1
fi

sed -i -e "s/^Version: .*/Version: $version/" -e "s/^Release: .*/Release: 0.$revision%{?dist}/" ${component}.spec

The list of components misses just two packages: argus-pep-api-c, which is not under our active development, and xacml, which has some problems at this moment; gsoap-devel needs to be added to the build-srpm group (done) and as the nightly build does the bootstrapping as well, the autoconf for the epel5 tag is too old according to configure.in. Whether this is really the case remains to be seen.

Preventing repeat builds

Koji will refuse build artefacts that it already has. The nightly builds are versioned according to the svn revision of the latest update of the tree to build. That means that if no updates are done to the tree, there is no sense in repeating the build. The cron script should be enhanced to deal with this situation.

Troubleshooting

This section covers a couple of issues encountered in the wild, some of which weren't trivial to understand or solve.

low-level database access

Some koji commands to add things do not have a remove-things counterpart, such as add-group and add-group-pkg. If removal is needed, access to the PostgreSQL database is easy: change to user 'koji' on koji-hub.testbed and run psql.

libtoolize: can not copy

On fc17 libtoolize may fail with a strange error message:

libtoolize: can not copy `/usr/share/libtool/config/ltmain.sh' to `./'

which is due to 'tar' missing on the system. See bug https://bugzilla.redhat.com/show_bug.cgi?id=794675. The resolution is to include tar as a required package in the srpm-build group.

GenericError: failed to create repo (out of memory)

The regeneration of repositories which is done by kojira between builds is a memory-hungry task. As builders are configured by default to accept a load of 2 tasks, they may receive two createrepo/mergerepo jobs and run out of memory. If such tasks fail with a message like:

  GenericError: failed to create repo: /usr/bin/createrepo -vd
-o /var/lib/koji/tasks/257/257/repo -u http://koji-hub.testbed/kojifiles
-i /mnt/koji/repos/dist-fc17-build/45/i386/pkglist -g /mnt/koji/repos/dist-fc17-build/45/groups/comps.xml
--update --skip-stat /mnt/koji was killed by signal 9

(line breaks inserted for formatting, the error is on a single line) check /var/log/messages for signs of the OOM killer springing into action.

PYCURL ERROR 7 - "couldn't connect to host"

If all builds suddenly start failing for no apparent reason, and the error message from mock reads something like:

DEBUG util.py:257:  http://koji-hub.testbed/kojifiles/repos/dist-epel5-build/1472/i386/repodata/repomd.xml:
[Errno 14] PYCURL ERROR 7 - "couldn't connect to host"

(line break inserted for formatting) this is likely due to the fact that the squid service is not running anymore.