Difference between revisions of "Koji Testbed"
Line 177: | Line 177: | ||
fi | fi | ||
− | /usr/bin/koji -c /project/srv/svn/home/.koji/koji.conf build --nowait dist-epel5 svn+http://ndpfsvn.nikhef.nl/ro/mwsec\?packaging/fedora/tags/${component}/${tag}#$revision | + | /usr/bin/koji -c /project/srv/svn/home/.koji/koji.conf build --nowait dist-epel5 \ |
− | /usr/bin/koji -c /project/srv/svn/home/.koji/koji.conf build --nowait dist-epel6 svn+http://ndpfsvn.nikhef.nl/ro/mwsec\?packaging/fedora/tags/${component}/${tag}#$revision | + | svn+http://ndpfsvn.nikhef.nl/ro/mwsec\?packaging/fedora/tags/${component}/${tag}#$revision |
− | /usr/bin/koji -c /project/srv/svn/home/.koji/koji.conf build --nowait dist-fc16 svn+http://ndpfsvn.nikhef.nl/ro/mwsec\?packaging/fedora/tags/${component}/${tag}#$revision | + | /usr/bin/koji -c /project/srv/svn/home/.koji/koji.conf build --nowait dist-epel6 \ |
− | /usr/bin/koji -c /project/srv/svn/home/.koji/koji.conf build --nowait dist-fc17 svn+http://ndpfsvn.nikhef.nl/ro/mwsec\?packaging/fedora/tags/${component}/${tag}#$revision | + | svn+http://ndpfsvn.nikhef.nl/ro/mwsec\?packaging/fedora/tags/${component}/${tag}#$revision |
+ | /usr/bin/koji -c /project/srv/svn/home/.koji/koji.conf build --nowait dist-fc16 \ | ||
+ | svn+http://ndpfsvn.nikhef.nl/ro/mwsec\?packaging/fedora/tags/${component}/${tag}#$revision | ||
+ | /usr/bin/koji -c /project/srv/svn/home/.koji/koji.conf build --nowait dist-fc17 \ | ||
+ | svn+http://ndpfsvn.nikhef.nl/ro/mwsec\?packaging/fedora/tags/${component}/${tag}#$revision | ||
* koji.conf: | * koji.conf: |
Revision as of 10:47, 1 March 2013
Koji setup on the testbed
The Koji system is used by Fedora for builds of packages for Fedora Core and EPEL. It is open source and available as a standard package, and documentation for running a private installation is available.
Koji consists of a number of interrelated components:
- Koji Hub, the central XMLRPC service.
- A postgresql database as a backend for the hub
- Koji Web, a user interface front-end for the hub (other interaction is by the command-line client)
- Kojira, managing repository building and clean-up.
- One or more builders, on separate machines.
It is possible to install most of these components on separate machines, but our simple setup has only three:
- koji-hub.testbed, holding the hub, web, kojira and database
- koji-builder.testbed and koji-boulder.testbed, for building i386 and x86_64 packages.
- (temporarily, a third builder has been added: koji-monster.testbed as part of a performance evaluation. It's been taken out of the active pool)
The machines share access to a large NFS share on put.testbed:/mnt/put/koji for storing results and buildroots (the builders only have read-only access). Currently the machines are deployed on the M610 blades (blade13 and blade14) with the possibility for hot migration.
The installation mostly followed the very complete guidance on [1], choosing the SSL authentication option.
Using a squid proxy for external repositories
The builds use mock to install a base set of packages to fulfill the BuildRequires in the spec file. These packages come from CentOS and Fedora mirrors, and in order to reduce the load on the mirrors a local squid caching proxy is configured to store up to 100GB of popular packages. To use the proxy, set
yum_proxy = http://span.testbed:3128/
in /etc/kojid/kojid.conf on all builders.
memory and disk space requirement
The builders are configured with 4096MB memory at the moment which should be enough (but see the #Troubleshooting section below).
The disk space on the builders could easily run out, as there is no automatic clean-up of used build directories.
The builders have a tmpwatch script in /etc/cron.daily that erases old data (older than 4 hours)
/usr/bin/tmpwatch -caMf 4 /mnt/mock/
SSL Authentication
The authentication system is picky when it comes to Common Names. For users, they have to be the same as user ids, and for servers (only koji-hub.testbed, really) it has to be the fully qualified host name.
A new CA was created for the purpose of handing out certificates (as reuse of existing certificates was hard because of the above restriction, but we may still find a way around that sometime). The CA installation is currently found on bleek.testbed:/srv/koji-pki.
For access to koji-web, the user needs to export the certificate file from there to a pkcs12 file and import it in the browser. For command-line koji use, put it in ~/.koji/clientcert.crt.
Adding tags, targets and repositories
See the main documentation on the Fedora wiki.
The builds are done against targets, which have source and target tags, which have buildroots populated from repositories. The initial buildroots are set up from virtual packages called 'build' and 'srpm-build' created in koji.
A dump of the history to see what steps are required
99 koji add-tag dist-epel5 100 koji add-tag --parent dist-epel5 --arches "i386 x86_64" dist-epel5-build 102 koji add-external-repo -t dist-epel5-build dist-epel5-external-repo http://mirrors.nl.eu.kernel.org/fedora-epel/5/\$arch/ 103 koji add-external-repo -t dist-epel5-build dist-epel5-buildsys-macros-repo http://buildsys.fedoraproject.org/buildgroups/rhel5/\$arch/ 104 koji add-external-repo -t dist-epel5-build dist-epel5-centos5-repo http://spiegel.nikhef.nl/mirror/centos/5/os/\$arch/ 105 koji add-target dist-epel5 dist-epel5-build
NOTE: the buildsys external repo is ONLY needed on EPEL5 to provide the correct buildsys-macros package. The standard CentOS package causes the dist macro to expand to el5.centos. By providing this external Fedora repo and putting it before the centos5 repo, the correct buildsys-macros is installed. See also https://fedoraproject.org/wiki/Koji/KojiMisc#.25dist_tags. If needed (i.e. when the ordering turns out to be wrong) we can provide explicit priorities using a -p flag to add-external-repo. Check with
koji taginfo dist-epel5-build
These groups are going to be passed to 'yum groupinstall' when the buildroot is created.
106 koji add-group dist-epel5-build build 107 koji add-group dist-epel5-build srpm-build
They need to be populated. Clone from a typical value for the Fedora Koji systems, e.g.
109 koji -s http://koji.fedoraproject.org/kojihub/ list-groups dist-5E-epel-build
Use the output to produce a proper list of packages. Be careful to exclude the fedpkg package which pulls in way too much! It includes mock which causes strange failures with conflicting group ids within the buildroot.
112 koji add-group-pkg dist-epel5-build build `cat epel5-build-pkgs ` 114 koji add-group-pkg dist-epel5-build srpm-build `cat epel5-srpm-build-pkgs `
Don't forget to add the package!
140 koji add-pkg --owner okoeroo dist-epel5 lcmaps 124 koji build dist-epel5 git://bleek.testbed/koji.git\?lcmaps/#a942e490ace1da950cedae2976bf59fa3b3f5038
The git repository must be added to the allowed repositories on the builder in kojid.conf.
Automated builds from tags in mwsec
Building from git (deprecated)
This step still needs to be automated with a commit hook. The git repository on git://bleek.testbed/koji.git has one subdirectory per component, and each directory contains:
- a SPEC file
- a Makefile with a 'sources' target, which will fetch (with curl) the sources for the package.
After pushing the updates to this repository, record the commit hash and start the koji build like so (replace the hash with yours):
koji build dist-epel5 'git://bleek.testbed/koji.git?lcmaps/#7a2b193fdd8a26875a16303da609952cccc4cba8'
The format is particular to koji; it is parsed as an URL and that is why the ? and # are used to mark certain elements.
Triggering a build in svn
the mwsec/packaging/fedora/tags directory is used for triggering builds in Koji. Each component has its own subdirectory, with the version number as a directory below that, so we have e.g.:
svn+ssh://svn@ndpfsvn.nikhef.nl/repos/mwsec/packaging/fedora/tags/lcmaps/1.5.5/
Within that directory, create a Makefile and a spec file. The spec file can be the same as the one from packaging/fedora/trunk for that component. The Makefile can be fairly generic, in most cases this should suffice:
dir = $(shell pwd) version = $(notdir $(dir)) component = lcmaps-plugins-verify-proxy sources: curl -O http://software.nikhef.nl/security/$(component)/$(component)-$(version).tar.gz sed -i -e 's/^Version: .*/Version: $(version)/' $(component).spec
Should a rebuild be necessary because of errors in Koji or in the spec file, update the Release field in the spec file as usual.
Scripts on the svn repository
The svn repository needs a post-commit hook to start koji builds automatically from svn. The svn account also needs a koji certificate with build rights. Network access to the koji-hub is by a DNAT proxy through bleek.
The following scripts are installed on sikkel.nikhef.nl:
/project/srv/svn/repos/mwsec/hooks -rwxr-xr-x 1 svn apache 1026 Jul 18 14:05 koji-build-trigger -rwxr-xr-x 1 svn apache 2717 Jul 19 14:27 post-commit
- post-commit:
REPOS="$1" REV="$2" echo "Commit revision $REV to $REPOS" >&2 PATH=/bin:/usr/bin export PATH # We are looking for additions (A) or updates (U) to a specific path to trigger # a koji build. The 'svnlook changed' output may look like: ## A packaging/fedora/tags/lcmaps/ ## A packaging/fedora/tags/lcmaps/1.2.3/ ## A packaging/fedora/tags/lcmaps/1.2.3/Makefile # And it's the middle one we trigger on. svnlook changed -r $REV "$REPOS" | awk '$1 ~ /(A|U)/ && $2 ~ /packaging\/fedora\/tags\/[[:alnum:]-]+\/[[:alnum:].]+\/.*\.spec$/ { split($2, pe, "/"); print pe[4], pe[5]; }' | while read component tag ; do echo "component: $component; tag: $tag" >&2 if [ -n "$component" -a -n "$tag" ]; then "$REPOS"/hooks/koji-build-trigger $component $tag $REV fi done
- koji-build-trigger:
#!/bin/sh component="$1" tag="$2" revision="$3" if [ -z "$component" ]; then echo "Empty component specified for koji build." >&2 exit 1 fi if [ -z "$tag" ]; then echo "Empty tag specified for koji build." >&2 exit 1 fi if [ -z "$revision" ]; then echo "Empty revision specified for koji build." >&2 exit 1 fi /usr/bin/koji -c /project/srv/svn/home/.koji/koji.conf build --nowait dist-epel5 \ svn+http://ndpfsvn.nikhef.nl/ro/mwsec\?packaging/fedora/tags/${component}/${tag}#$revision /usr/bin/koji -c /project/srv/svn/home/.koji/koji.conf build --nowait dist-epel6 \ svn+http://ndpfsvn.nikhef.nl/ro/mwsec\?packaging/fedora/tags/${component}/${tag}#$revision /usr/bin/koji -c /project/srv/svn/home/.koji/koji.conf build --nowait dist-fc16 \ svn+http://ndpfsvn.nikhef.nl/ro/mwsec\?packaging/fedora/tags/${component}/${tag}#$revision /usr/bin/koji -c /project/srv/svn/home/.koji/koji.conf build --nowait dist-fc17 \ svn+http://ndpfsvn.nikhef.nl/ro/mwsec\?packaging/fedora/tags/${component}/${tag}#$revision
- koji.conf:
[koji] ;configuration for koji cli tool ;url of XMLRPC server server = https://bleek.nikhef.nl:8443/kojihub ;url of web interface weburl = https://bleek.nikhef.nl:8443/koji ;url of package download site topurl = https://bleek.nikhef.nl:8443/kojifiles/packages/ ;path to the koji top directory topdir = /mnt/koji ;configuration for Kerberos authentication ;the service name of the principal being used by the hub ;krbservice = host ;configuration for SSL authentication ;client certificate cert = ~/.koji/clientcert.crt ;certificate of the CA that issued the client certificate ca = ~/.koji/koji_ca_cert.crt ;certificate of the CA that issued the HTTP server certificate serverca = ~/.koji/koji_ca_cert.crt
- iptables rule on bleek:
iptables -t nat -A PREROUTING --proto tcp --dport 8443 --destination 194.171.96.17 -j DNAT --to-destination 10.198.8.7:443
Generating distribution repos from koji
To render the products of koji into something that can be installed by yum, we use the mash utility installed on koji-hub.testbed. The repositories are made using mash on an NFS share from bleek.nikhef.nl:
/srv/project/mwsec 10.198.8.7(rw,async,no_root_squash,insecure)
bleek.testbed:/srv/project/mwsec /mnt/mwsec nfs defaults 0 0
The output of mash should be send to /mnt/mwsec/mash. We create symlinks to the different distributions in /mnt/mwsec/rpm to allow syncing to software.nikhef.nl, the reason is that mash cannot easily put testing RPMs in ./testing/epel5/ and release RPMs in ./epel5/ since the directory is taken directly from the section name in the mwsec.mash config file:
lrwxrwxrwx rpm/epel5 -> ../mash/epel5-release lrwxrwxrwx rpm/epel6 -> ../mash/epel6-release lrwxrwxrwx rpm/fc16 -> ../mash/fc16-release lrwxrwxrwx rpm/fc17 -> ../mash/fc17-release drwxrwsr-x rpm/testing lrwxrwxrwx rpm/testing/epel5 -> ../../mash/epel5-testing lrwxrwxrwx rpm/testing/epel6 -> ../../mash/epel6-testing lrwxrwxrwx rpm/testing/fc16 -> ../../mash/fc16-testing lrwxrwxrwx rpm/testing/fc17 -> ../../mash/fc17-testing
For each distribution we create two additional tags -release and -testing:
koji add-tag --parent dist-epel5 dist-epel5-release koji add-tag --parent dist-epel5 dist-epel5-testing
and likewise for epel6, fc16 and fc17. Succesful builds can be added to either testing or release:
koji tag-pkg dist-epel5-release glexec-0.9.7-1.el5
Or moved from testing to release
koji move-pkg dist-epel5-testing dist-epel5-release lcmaps-1.5.7-1.el5
The mash configuration has one main config file and multiple distribution files: /etc/mash/mash.conf
[defaults] configdir = /etc/mash buildhost = http://koji-hub.testbed/kojihub repodir = file:///mnt/koji use_sqlite = True use_repoview = True
And /etc/mash/mwsec.mash containing the different distributions:
[epel5-release] rpm_path = %(arch)s repodata_path = %(arch)s/ source_path = SRPMS debuginfo = True # Do not create i386s in x86_64 multilib = False multilib_method = devel # Use old-style sha hash for EPEL5 only hash = sha tag = dist-epel5-release # Next one is needed to prevent mixing of testing and release. inherit = False strict_keys = False #keys = 22B3B81A, 1ACA3465, A82BA4B7, 069C8460, 97A1071F, E8E40FDE, 57BBCCBA, D22E77F2, 4EBFC273, 0B86274E, 6DF2196F, DF9B0AE9 # Note: repoviewurl used in latest-feed.xml for RPM location repoviewurl = /dist/mwsec/rpm/epel5/%(arch)s repoviewtitle = MWSEC for EPEL5 - %(arch)s arches = i386 x86_64 delta = False # Change distro_tags as fedora-release version gets bumped #distro_tags = cpe:/o:fedoraproject:fedora:17 rawhide hash_packages = False # This forces to also create older version of same package latest = False
The testing repository is the same except:
tag = dist-epel5-testing repoviewurl = /dist/mwsec/rpm/testing/epel5/%(arch)s repoviewtitle = MWSEC Testing for EPEL5 - %(arch)s
The epel6, fc16 and fc17 are the same except that they do not override the hash =
To run mash, simply use the following command, NOTE: it's best to have a umask 002 setting
for dist in epel{5,6} fc{16,17} ; do mash -o /mnt/mwsec/mash ${dist}-release mash -o /mnt/mwsec/mash ${dist}-testing done
Note: Mash completely removes and recreates the output directories! For epel5 we have old builds which should afterwards be merged back in, we use a tarball with symlinks in /mnt/mwsec/old_releases.tgz. The relative symlinks point to the actual files in /mnt/mwsec/mock/. The directory /mnt/mwsec/mock/ is a read-only NFS share on bleek.nikhef.nl:
/srv/project/mock 10.198.0.0/16(ro,async,root_squash,insecure)
bleek.testbed:/srv/project/mock /mnt/mock nfs defaults 0 0
By having the mock and mwsec directories at the same level, we can use relative symlinks from one to the other both on bleek.nikhef.nl and koji-hub.testbed.
See SAC_software_procedures#Updating the mash repositories for a practical procedure.
Nightly builds from mwsec/trunk
The dedicated account kojinb for nightly builds is added on bleek.
At the moment nightly builds are run from a crontab of account dennisvd.
01 2 * * * ${HOME}/bin/koji-nightly
The koji-nightly script:
#!/bin/sh PATH=/bin:/usr/bin components=" lcas lcmaps ees glexec jobrepository lcas-lcmaps-gt4-interface lcas-plugins-basic lcas-plugins-check-executable lcas-plugins-voms lcmaps-plugins-afs lcmaps-plugins-basic lcmaps-plugins-c-pep lcmaps-plugins-gums lcmaps-plugins-jobrep lcmaps-plugins-scas-client lcmaps-plugins-tracking-groupid lcmaps-plugins-verify-proxy lcmaps-plugins-voms scas " dists="epel5 epel6 fc16 fc17" rev=`svn info http://ndpfsvn.nikhef.nl/ro/mwsec/ | sed -n 's/^Revision: //p'` for i in $components ; do for j in $dists ; do koji -c ${HOME}/.koji/nightly.conf build --nowait nightly-$j \ svn+http://ndpfsvn.nikhef.nl/ro/mwsec\?packaging/fedora/branches/nightly-builds/$i/#$rev done done
Each component directory in the nightly-builds branch contains the same set of files: a Makefile:
component = $(notdir $(shell pwd)) sources: ./fetch-sources $(component)
And a fetch-sources script:
#!/bin/sh set -e component=$1 svn co http://ndpfsvn.nikhef.nl/ro/mwsec/trunk/${component} cd ${component} revision=`svn info | sed -n 's/^Last Changed Rev: //p'` if [ -z $revision ]; then echo "Could not retrieve revision from svn info" >&2 echo "svn info:" >&2 svn info >&2 exit 1 fi ./bootstrap mkdir build cd build ../configure make dist mv ${component}-*.tar.gz .. cd .. version=`echo ${component}-*.tar.gz | sed -n "s/${component}-\\([0-9.]*\\)\\.tar\\.gz/\\1/p"` if [ -z $version ]; then echo "Could not retrieve version from generated tarball" >&2 echo "tarballs: " ${component}-*.tar.gz >&2 exit 1 fi sed -i -e "s/^Version: .*/Version: $version/" -e "s/^Release: .*/Release: 0.$revision%{?dist}/" ${component}.spec
The list of components misses just two packages: argus-pep-api-c, which is not under our active development, and xacml, which has some problems at this moment; gsoap-devel needs to be added to the build-srpm group (done) and as the nightly build does the bootstrapping as well, the autoconf for the epel5 tag is too old according to configure.in. Whether this is really the case remains to be seen.
Preventing repeat builds
Koji will refuse build artefacts that it already has. The nightly builds are versioned according to the svn revision of the latest update of the tree to build. That means that if no updates are done to the tree, there is no sense in repeating the build. The cron script should be enhanced to deal with this situation.
Troubleshooting
This section covers a couple of issues encountered in the wild, some of which weren't trivial to understand or solve.
libtoolize: can not copy
On fc17 libtoolize may fail with a strange error message:
libtoolize: can not copy `/usr/share/libtool/config/ltmain.sh' to `./'
which is due to 'tar' missing on the system. See bug https://bugzilla.redhat.com/show_bug.cgi?id=794675. The resolution is to include tar as a required package in the srpm-build group.
GenericError: failed to create repo (out of memory)
The regeneration of repositories which is done by kojira between builds is a memory-hungry task. As builders are configured by default to accept a load of 2 tasks, they may receive two createrepo/mergerepo jobs and run out of memory. If such tasks fail with a message like:
GenericError: failed to create repo: /usr/bin/createrepo -vd -o /var/lib/koji/tasks/257/257/repo -u http://koji-hub.testbed/kojifiles -i /mnt/koji/repos/dist-fc17-build/45/i386/pkglist -g /mnt/koji/repos/dist-fc17-build/45/groups/comps.xml --update --skip-stat /mnt/koji was killed by signal 9
(line breaks inserted for formatting, the error is on a single line) check /var/log/messages for signs of the OOM killer springing into action.
PYCURL ERROR 7 - "couldn't connect to host"
If all builds suddenly start failing for no apparent reason, and the error message from mock reads something like:
DEBUG util.py:257: http://koji-hub.testbed/kojifiles/repos/dist-epel5-build/1472/i386/repodata/repomd.xml: [Errno 14] PYCURL ERROR 7 - "couldn't connect to host"
(line break inserted for formatting) this is likely due to the fact that the squid service is not running anymore.