Difference between revisions of "Koji Testbed"
| Line 250: | Line 250: | ||
done | done | ||
done | done | ||
| + | |||
| + | Each component directory in the nightly-builds branch contains the same set of files: a Makefile: | ||
| + | component = $(notdir $(shell pwd)) | ||
| + | sources: | ||
| + | ./fetch-sources $(component) | ||
| + | |||
| + | And a fetch-sources script: | ||
| + | #!/bin/sh | ||
| + | |||
| + | set -e | ||
| + | |||
| + | component=$1 | ||
| + | |||
| + | svn co http://ndpfsvn.nikhef.nl/ro/mwsec/trunk/${component} | ||
| + | cd ${component} | ||
| + | revision=`svn info | sed -n 's/^Last Changed Rev: //p'` | ||
| + | |||
| + | if [ -z $revision ]; then | ||
| + | echo "Could not retrieve revision from svn info" >&2 | ||
| + | echo "svn info:" >&2 | ||
| + | svn info >&2 | ||
| + | exit 1 | ||
| + | fi | ||
| + | |||
| + | ./bootstrap | ||
| + | ./configure | ||
| + | make dist | ||
| + | version=`echo ${component}-*.tar.gz | sed -n "s/${component}-\\([0-9.]*\\)\\.tar\\.gz/\\1/p"` | ||
| + | |||
| + | if [ -z $version ]; then | ||
| + | echo "Could not retrieve version from generated tarball" >&2 | ||
| + | echo "tarballs: " ${component}-*.tar.gz >&2 | ||
| + | exit 1 | ||
| + | fi | ||
| + | |||
| + | mv ${component}-*.tar.gz .. | ||
| + | cd .. | ||
| + | |||
| + | sed -i -e "s/^Version: .*/Version: $version/" -e "s/^Release: .*/Release: 0.$revision%{?dist}/" ${component}.spec | ||
| + | |||
| + | |||
== Preventing repeat builds == | == Preventing repeat builds == | ||
Revision as of 10:50, 6 August 2012
Koji setup on the testbed
The Koji system is used by Fedora for builds of packages for Fedora Core and EPEL. It is open source and available as a standard package, and documentation for running a private installation is available.
Koji consists of a number of interrelated components:
- Koji Hub, the central XMLRPC service.
- A postgresql database as a backend for the hub
- Koji Web, a user interface front-end for the hub (other interaction is by the command-line client)
- Kojira, managing repository building and clean-up.
- One or more builders, on separate machines.
It is possible to install most of these components on separate machines, but our simple setup has only three:
- koji-hub.testbed, holding the hub, web, kojira and database
- koji-builder.testbed and koji-boulder.testbed, for building i386 and x86_64 packages.
The machines share access to a large NFS share on put.testbed:/mnt/put/koji for storing results and buildroots (the builders only have read-only access). Currently the machines are deployed on the M610 blades (blade13 and blade14) with the possibility for hot migration.
The installation mostly followed the very complete guidance on [1], choosing the SSL authentication option.
A note on memory consumption
The regeneration of repositories which is done by kojira between builds is a memory-hungry task. As builders are configured by default to accept a load of 2 tasks, they may receive two createrepo/mergerepo jobs and run out of memory. If such tasks fail with a message like:
GenericError: failed to create repo: /usr/bin/createrepo -vd -o /var/lib/koji/tasks/257/257/repo -u http://koji-hub.testbed/kojifiles -i /mnt/koji/repos/dist-fc17-build/45/i386/pkglist -g /mnt/koji/repos/dist-fc17-build/45/groups/comps.xml --update --skip-stat /mnt/koji was killed by signal 9
check /var/log/messages for signs of the OOM killer springing into action.
The builders are configured with 4096MB memory at the moment which should be enough.
SSL Authentication
The authentication system is picky when it comes to Common Names. For users, they have to be the same as user ids, and for servers (only koji-hub.testbed, really) it has to be the fully qualified host name.
A new CA was created for the purpose of handing out certificates (as reuse of existing certificates was hard because of the above restriction, but we may still find a way around that sometime). The CA installation is currently found on bleek.testbed:/srv/koji-pki.
For access to koji-web, the user needs to export the certificate file from there to a pkcs12 file and import it in the browser. For command-line koji use, put it in ~/.koji/clientcert.crt.
Building from git
This step still needs to be automated with a commit hook. The git repository on git://bleek.testbed/koji.git has one subdirectory per component, and each directory contains:
- a SPEC file
- a Makefile with a 'sources' target, which will fetch (with curl) the sources for the package.
After pushing the updates to this repository, record the commit hash and start the koji build like so (replace the hash with yours):
koji build dist-epel5 'git://bleek.testbed/koji.git?lcmaps/#7a2b193fdd8a26875a16303da609952cccc4cba8'
The format is particular to koji; it is parsed as an URL and that is why the ? and # are used to mark certain elements.
Adding tags, targets and repositories
See the main documentation on the Fedora wiki.
The builds are done against targets, which have source and target tags, which have buildroots populated from repositories. The initial buildroots are set up from virtual packages called 'build' and 'srpm-build' created in koji.
A dump of the history to see what steps are required
99 koji add-tag dist-epel5 100 koji add-tag --parent dist-epel5 --arches "i386 x86_64" dist-epel5-build 102 koji add-external-repo -t dist-epel5-build dist-epel5-external-repo http://mirrors.nl.eu.kernel.org/fedora-epel/5/\$arch/ 103 koji add-external-repo -t dist-epel5-build dist-epel5-centos5-repo http://spiegel.nikhef.nl/mirror/centos/5/os/\$arch/ 104 koji add-target dist-epel5 dist-epel5-build
These groups are going to be passed to 'yum groupinstall' when the buildroot is created.
106 koji add-group dist-epel5-build build 107 koji add-group dist-epel5-build srpm-build
They need to be populated
109 koji -s http://koji.fedoraproject.org/koji list-groups dist-epel5-build 110 koji -s https://koji.fedoraproject.org/koji list-groups dist-epel5-build
Clone from a typical value for the Fedora Koji systems. Be careful to exclude the fedpkg package which pulls in way too much! It includes mock which causes strange failures with conflicting group ids within the buildroot.
112 koji add-group-pkg dist-epel5-build build `cat epel5-build-pkgs ` 114 koji add-group-pkg dist-epel5-build srpm-build `cat epel5-srpm-build-pkgs `
Don't forget to add the package!
140 koji add-pkg --owner okoeroo dist-epel5 lcmaps 124 koji build dist-epel5 git://bleek.testbed/koji.git\?lcmaps/#a942e490ace1da950cedae2976bf59fa3b3f5038
The git repository must be added to the allowed repositories on the builder in kojid.conf.
Automated builds from tags in mwsec
Triggering a build in svn
the mwsec/packaging/fedora/tags directory is used for triggering builds in Koji. Each component has its own subdirectory, with the version number as a directory below that, so we have e.g.:
svn+ssh://svn@ndpfsvn.nikhef.nl/repos/mwsec/packaging/fedora/tags/lcmaps/1.5.5/
Within that directory, create a Makefile and a spec file. The spec file can be the same as the one from packaging/fedora/trunk for that component. The Makefile can be fairly generic, in most cases this should suffice:
dir = $(shell pwd) version = $(notdir $(dir)) component = lcmaps-plugins-verify-proxy sources: curl -O http://software.nikhef.nl/security/$(component)/$(component)-$(version).tar.gz sed -i -e 's/^Version: .*/Version: $(version)/' $(component).spec
Should a rebuild be necessary because of errors in Koji or in the spec file, update the Release field in the spec file as usual.
Scripts on the svn repository
The svn repository needs a post-commit hook to start koji builds automatically from svn. The svn account also needs a koji certificate with build rights. Network access to the koji-hub is by a DNAT proxy through bleek.
The following scripts are installed on sikkel.nikhef.nl:
/project/srv/svn/repos/mwsec/hooks -rwxr-xr-x 1 svn apache 1026 Jul 18 14:05 koji-build-trigger -rwxr-xr-x 1 svn apache 2717 Jul 19 14:27 post-commit
- post-commit:
REPOS="$1"
REV="$2"
echo "Commit revision $REV to $REPOS" >&2
PATH=/bin:/usr/bin
export PATH
# We are looking for additions (A) or updates (U) to a specific path to trigger
# a koji build. The 'svnlook changed' output may look like:
## A packaging/fedora/tags/lcmaps/
## A packaging/fedora/tags/lcmaps/1.2.3/
## A packaging/fedora/tags/lcmaps/1.2.3/Makefile
# And it's the middle one we trigger on.
svnlook changed -r $REV "$REPOS" | awk '$1 ~ /(A|U)/ && $2 ~ /packaging\/fedora\/tags\/[[:alnum:]-]+\/[[:alnum:].]+\/.*\.spec$/ {
split($2, pe, "/");
print pe[4], pe[5];
}' | while read component tag ; do
echo "component: $component; tag: $tag" >&2
if [ -n "$component" -a -n "$tag" ]; then
"$REPOS"/hooks/koji-build-trigger $component $tag $REV
fi
done
- koji-build-trigger:
#!/bin/sh
component="$1"
tag="$2"
revision="$3"
if [ -z "$component" ]; then
echo "Empty component specified for koji build." >&2
exit 1
fi
if [ -z "$tag" ]; then
echo "Empty tag specified for koji build." >&2
exit 1
fi
if [ -z "$revision" ]; then
echo "Empty revision specified for koji build." >&2
exit 1
fi
/usr/bin/koji -c /project/srv/svn/home/.koji/koji.conf build --nowait dist-epel5 svn+http://ndpfsvn.nikhef.nl/ro/mwsec\?packaging/fedora/tags/${component}/${tag}#$revision
/usr/bin/koji -c /project/srv/svn/home/.koji/koji.conf build --nowait dist-epel6 svn+http://ndpfsvn.nikhef.nl/ro/mwsec\?packaging/fedora/tags/${component}/${tag}#$revision
/usr/bin/koji -c /project/srv/svn/home/.koji/koji.conf build --nowait dist-fc16 svn+http://ndpfsvn.nikhef.nl/ro/mwsec\?packaging/fedora/tags/${component}/${tag}#$revision
/usr/bin/koji -c /project/srv/svn/home/.koji/koji.conf build --nowait dist-fc17 svn+http://ndpfsvn.nikhef.nl/ro/mwsec\?packaging/fedora/tags/${component}/${tag}#$revision
- koji.conf:
[koji] ;configuration for koji cli tool ;url of XMLRPC server server = https://bleek.nikhef.nl:8443/kojihub ;url of web interface weburl = https://bleek.nikhef.nl:8443/koji ;url of package download site topurl = https://bleek.nikhef.nl:8443/kojifiles/packages/ ;path to the koji top directory topdir = /mnt/koji ;configuration for Kerberos authentication ;the service name of the principal being used by the hub ;krbservice = host ;configuration for SSL authentication ;client certificate cert = ~/.koji/clientcert.crt ;certificate of the CA that issued the client certificate ca = ~/.koji/koji_ca_cert.crt ;certificate of the CA that issued the HTTP server certificate serverca = ~/.koji/koji_ca_cert.crt
- iptables rule on bleek:
iptables -t nat -A PREROUTING --proto tcp --dport 8443 --destination 194.171.96.17 -j DNAT --to-destination 10.198.8.7:443
Nightly builds from mwsec/trunk
The dedicated account kojinb for nightly builds is added on bleek.
At the moment nightly builds are run from a crontab of account dennisvd.
01 2 * * * ${HOME}/bin/koji-nightly
The koji-nightly script:
#!/bin/sh PATH=/bin:/usr/bin components=" lcas lcmaps xacml ees glexec jobrepository lcas-lcmaps-gt4-interface lcas-plugins-basic lcas-plugins-check-executable lcas-plugins-voms lcmaps-plugins-afs lcmaps-plugins-basic lcmaps-plugins-c-pep lcmaps-plugins-gums lcmaps-plugins-jobrep lcmaps-plugins-scas-client lcmaps-plugins-tracking-groupid lcmaps-plugins-verify-proxy lcmaps-plugins-voms scas " dists="epel5 epel6 fc16 fc17" rev=`svn info http://ndpfsvn.nikhef.nl/ro/mwsec/ | sed -n 's/^Revision: //p'` for i in $components ; do for j in $dists ; do koji -c ${HOME}/.koji/nightly.conf build --nowait nightly-$j svn+http://ndpfsvn.nikhef.nl/ro/mwsec\?packaging/fedora/branches/nightly-builds/$i/#$rev done done
Each component directory in the nightly-builds branch contains the same set of files: a Makefile:
component = $(notdir $(shell pwd)) sources: ./fetch-sources $(component)
And a fetch-sources script:
#!/bin/sh set -e component=$1 svn co http://ndpfsvn.nikhef.nl/ro/mwsec/trunk/${component} cd ${component} revision=`svn info | sed -n 's/^Last Changed Rev: //p'` if [ -z $revision ]; then echo "Could not retrieve revision from svn info" >&2 echo "svn info:" >&2 svn info >&2 exit 1 fi ./bootstrap ./configure make dist version=`echo ${component}-*.tar.gz | sed -n "s/${component}-\\([0-9.]*\\)\\.tar\\.gz/\\1/p"` if [ -z $version ]; then echo "Could not retrieve version from generated tarball" >&2 echo "tarballs: " ${component}-*.tar.gz >&2 exit 1 fi mv ${component}-*.tar.gz .. cd .. sed -i -e "s/^Version: .*/Version: $version/" -e "s/^Release: .*/Release: 0.$revision%{?dist}/" ${component}.spec
Preventing repeat builds
Koji will refuse build artefacts that it already has. The nightly builds are versioned according to the svn revision of the latest update of the tree to build. That means that if no updates are done to the tree, there is no sense in repeating the build. The cron script should be enhanced to deal with this situation.
odd details
On fc17 libtoolize may fail with a strange error message:
libtoolize: can not copy `/usr/share/libtool/config/ltmain.sh' to `./'
which is due to 'tar' missing on the system. See bug https://bugzilla.redhat.com/show_bug.cgi?id=794675. The resolution is to include tar as a required package in the srpm-build group.
