Difference between revisions of "Koji Testbed"

From PDP/Grid Wiki
Jump to navigationJump to search
 
(61 intermediate revisions by 2 users not shown)
Line 11: Line 11:
 
* One or more builders, on separate machines.
 
* One or more builders, on separate machines.
  
−
It is possible to install most of these components on separate machines, but our simple setup has only three:
+
Linked to Koji in some functional way, but not part of its core, are some additional systems:
 +
* sigul, a secure gpg signing system to create signatures for rpms;
 +
* mash, a system to create yum repositories to release builds to the general public.
 +
 
 +
It is possible to install most of these components on separate machines, our setup is as follows:
  
 
* koji-hub.testbed, holding the hub, web, kojira and database
 
* koji-hub.testbed, holding the hub, web, kojira and database
−
* koji-builder.testbed and koji-boulder.testbed, for building i386 and x86_64 packages.
+
* koji-builder.testbed, a builder with 8 GB of memory which also does kojira for creating repositories
−
* (temporarily, a third builder has been added: koji-monster.testbed as part of a performance evaluation. It's been taken out of the active pool)
+
* koji-boulder.testbed, a normal builder with 4 GB of memory
 +
* put.testbed, which holds the shared NFS storage for all build artefacts
 +
* bleek.testbed, which holds the mash output before synchronising to software.nikhef.nl
 +
* koji-bridge.testbed, part of the sigul setup
 +
* sigul.testbed, the sigul signing server
  
 
The machines share access to a large NFS share on put.testbed:/mnt/put/koji for storing results and buildroots (the builders only have read-only
 
The machines share access to a large NFS share on put.testbed:/mnt/put/koji for storing results and buildroots (the builders only have read-only
Line 21: Line 29:
  
 
The installation mostly followed the very complete guidance on [https://fedoraproject.org/wiki/Koji/ServerHowTo], choosing the SSL authentication option.
 
The installation mostly followed the very complete guidance on [https://fedoraproject.org/wiki/Koji/ServerHowTo], choosing the SSL authentication option.
 +
 +
'''NOTE''' In december we upgraded to 1.8, and the database schema needed to change. See the migration documentation in /usr/share/doc/koji on koji-hub.testbed.
 +
--[[User:Dennisvd@nikhef.nl|Dennisvd@nikhef.nl]] ([[User talk:Dennisvd@nikhef.nl|talk]]) 10:01, 15 January 2014 (CET)
  
 
=== Using a squid proxy for external repositories ===
 
=== Using a squid proxy for external repositories ===
Line 30: Line 41:
 
=== memory and disk space requirement ===
 
=== memory and disk space requirement ===
  
−
The regeneration of repositories which is done by kojira between builds is a memory-hungry task. As builders are configured by default to accept a load of 2 tasks, they may receive two createrepo/mergerepo jobs and run out of memory. If such tasks fail with a message like:
 
−
  GenericError: failed to create repo: /usr/bin/createrepo -vd -o /var/lib/koji/tasks/257/257/repo -u http://koji-hub.testbed/kojifiles -i /mnt/koji/repos/dist-fc17-build/45/i386/pkglist -g /mnt/koji/repos/dist-fc17-build/45/groups/comps.xml --update --skip-stat /mnt/koji was killed by signal 9
 
−
check /var/log/messages for signs of the OOM killer springing into action.
 
  
−
The builders are configured with 4096MB memory at the moment which should be enough.
+
The basic builders are configured with 4096MB memory at the moment which should be enough (but see the [[#Troubleshooting]] section below). The createrepo and mergerepo processes can eat enormous amounts of RAM (they slurp in the external repositories with all of Fedora's packages), so koji-builder.testbed has been given 8 GB of memory and kojira is limited to using just this machine.
  
 
The disk space on the builders could easily run out, as there is no automatic clean-up of used build directories.
 
The disk space on the builders could easily run out, as there is no automatic clean-up of used build directories.
Line 41: Line 49:
 
  /usr/bin/tmpwatch -caMf 4 /mnt/mock/
 
  /usr/bin/tmpwatch -caMf 4 /mnt/mock/
  
−
Another approach is being tried at the moment on koji-monster.testbed, which runs every hour and deletes any directory that does not have the buildroot.lock file:
 
−
for i in /mnt/mock/* ; do
 
−
if [ ! -f $i/buildroot.lock ]; then
 
−
rm -rf $i
 
−
fi
 
−
done
 
  
 
== SSL Authentication ==
 
== SSL Authentication ==
Line 56: Line 58:
 
For access to koji-web, the user needs to export the certificate file from there to a pkcs12 file and import it in the browser. For command-line koji use, put it in ~/.koji/clientcert.crt.
 
For access to koji-web, the user needs to export the certificate file from there to a pkcs12 file and import it in the browser. For command-line koji use, put it in ~/.koji/clientcert.crt.
  
−
== Building from git ==
+
== Adding tags, targets and repositories ==
 +
 
 +
See the main documentation on [http://fedoraproject.org/wiki/Koji/ExternalRepoServerBootstrap the Fedora wiki].
 +
 
 +
We've had several experimental setups before coming to understand what everything really meant. So here is a bit of documentation on the terminology in use, and the apparent logic of the system.
 +
 
 +
=== Terminology ===
 +
 
 +
The most confusing word in Koji context is the word 'package', which means two related but very different things:
 +
# the base name of a package, as colloquially used in a sentence like: �Hey, is lcmaps installed on this box?�
 +
# the full ''name-version-release'' or NVR of the package, such as lcmaps-1.6.1-2.el6.
 +
 
 +
With that out of the way, the central structural element in Koji is the ''tag''. This is, of course, just a name, but many things can have tags attached to them. In fact, it is easier to reverse the idea of sticking a tag onto something and rather think of tags as pinboards where stuff is tacked onto. Here is a list of things that can go on tags:
 +
 
 +
* packages, in the first sense as described above
 +
* packages, in the second sense as described above
 +
* architectures, such as i386 and x86_64 (in fact, only those in our case)
 +
* groups (we'll get to those later)
 +
* external repositories
 +
* build targets (source or destination)
 +
* other tags called �parents�
 +
 
 +
The second most confusing word in Koji context is the word 'parent' and, along with that, the entire concept of inheritance. In the biological sense we can't choose, change, or add parents, but tags have no such limitations. While normally we inherit from our parents after they pass away, tags inherit everything from all of their parents (and parents' parents) immediately. Remember: in the graphical displays, Koji shows parents down and to the right in the inheritance tree.
 +
 
 +
The builds are done against build targets, which have source and target tags. The way this works in a nutshell is as follows:
 +
* the user issues the build of a package (1st sense) for a specific target
 +
* the target's ''build tag'' is chosen to create a build root
 +
* dependencies are resolved, hopefully
 +
* the build runs, and if all is well this results in a new artefact (a package in the second sense)
 +
* this package is tagged with the target's ''destination tag''
 +
 
 +
Resolving dependencies and setting up a build root works by combining external repositories and packages with the build tag, and installing a meta-package with 'yum groupinstall'. This is where the concept of ''groups'' comes in. There are two groups for the different phases of a build: <code>srpm-build</code> for the creation of source RPMs and <code>build</code> for the actual build.
 +
The groups as created in koji contain only package names.
 +
 
 +
=== Current setup ===
 +
 
 +
Fedora releases are built one on top another, and using inheritance makes perfect sense. But there is a catch. Builds, a.k.a. packages in the second sense, are inherited just the same. As are external repositories. We've experienced that our build tag for Fedora 18 collected everything from Fedora 17 and 16 (when we started), including all the repositories. This went crazy.
 +
 
 +
The solution is to be careful about what we inherit. The current setup is displayed best by the following pictures:
 +
 
 +
$ koji list-tag-inheritance --reverse f16
 +
f16 (7)  <- contains ONLY packages in the first sense
 +
  +-f17 (9)  <- so does this, by inheritance
 +
  |  +-f17-candidates (33)  <- this is the "build target" "destination tag"
 +
  |  +-f17-release (25)  <- this is used manually to mark a package (second sense) ready for shipping with mash
 +
  |  +-f17-testing (26)  <- idem, but the for the 'testing' repo
 +
  |  +-f18 (27)  <- only packages, first sense
 +
  |  |  +-f18-build (30)  <- the "build target" "build tag"; this has architectures and external repos
 +
  |  |  |  +-f18-candidates (31)  -< the "build target" "destination tag"
 +
  |  |  +-f19 (34)
 +
  |  |  |  +-f19-candidates (38)
 +
  |  |  |  +-f19-release (43)
 +
  |  |  |  +-f19-testing (44)
 +
  |  |  |  +-f19-build (36)
 +
  |  |  +-f18-release (39)
 +
  |  |  +-f18-testing (40)
 +
  |  +-f17-build (32)
 +
  +-f16-release (24)
 +
  +-f16-testing (23)
 +
  +-f16-build (45)
 +
 
 +
And this one:
 +
 
 +
$ koji list-tag-inheritance f19-build
 +
f19-build (36)  <- the "build target" "build tag"
 +
  +-f19 (34)  <- package names only (first sense)
 +
  |  +-f18 (27)  <- all the way
 +
  |    +-f17 (9)  <-  down to
 +
  |        +-f16 (7)  <- here
 +
  +-f19-basedeps (35)  <- these contain the 'groups'
 +
      +-f18-basedeps (28)  <- actually inherited from
 +
        +-f17-basedeps (10)  <- here
  
−
This step still needs to be automated with a commit hook. The git repository on git://bleek.testbed/koji.git has one subdirectory per component, and each directory contains:
+
A build for F19 ends up in the tag f19-candidates. From there, it needs to be manually moved. If it is a build dependency for other packages, it has to go to f19-build. Otherwise, it eventually goes to f19-testing and from there to f19-release, or directly to f19-release. A rotten product should have its candidacy stripped and ends up with no tags. In any case, no package should remain a candidate for too long.
−
* a SPEC file
+
 
−
* a Makefile with a 'sources' target, which will fetch (with curl) the sources for the package.
+
Because the *-builds don't inherit from each other, we have no pollution of external repositories or old builds for new Fedora releases.
−
After pushing the updates to this repository, record the commit hash and start the koji build like so (replace the hash with yours):
 
  
−
koji build dist-epel5 'git://bleek.testbed/koji.git?lcmaps/#7a2b193fdd8a26875a16303da609952cccc4cba8'
+
The situation for epel5 and epel6 is analogue.
  
−
The format is particular to koji; it is parsed as an URL and that is why the ? and # are used to mark certain elements.
+
=== Steps to follow if you need to do this from scratch ===
  
−
== Adding tags, targets and repositories ==
+
'''The following doesn't exactly reflect the current setup, but we'd do it this way if we had to start over.'''
  
−
See the main documentation on [http://fedoraproject.org/wiki/Koji/ExternalRepoServerBootstrap the Fedora wiki].
+
Create a tag to collect all our packages in.
 +
koji add-tag epel5
 +
koji add-pkg --owner <''your name''> epel5 lcmaps jobrepository ees glexec \
 +
          lcas-lcmaps-gt4-interface lcas-plugins-basic lcas-plugins-check-executable \
 +
          lcas-plugins-voms lcas lcmaps-plugins-afs lcmaps-plugins-basic lcmaps-plugins-c-pep lcmaps-plugins-gums \
 +
          lcmaps-plugins-jobrep lcmaps-plugins-scas-client lcmaps-plugins-tracking-groupid lcmaps-plugins-verify-proxy \
 +
          lcmaps-plugins-voms scas xacml argus-pep-api-c glexec-wn nagios-plugins-glexec nagios-plugins-ees mkgltempdir \
 +
          glexec-wrapper-scripts torque llrun saml2-xacml2-c-lib lcmaps-without-gsi mwsec-release
  
−
The builds are done against targets, which have source and target tags, which have buildroots populated from repositories. The initial buildroots are set up from virtual packages called 'build' and 'srpm-build' created in koji.
+
Another tag to use as a build target destination. All builds end up here, before they are moved to 'testing' or 'release' repositories.
 +
koji add-tag --parent epel5 epel5-candidates
  
−
A dump of the history to see what steps are required
+
These tags are used by the 'mash' tool to publish repositories.
 +
koji add-tag --parent epel5 epel5-testing
 +
koji add-tag --parent epel5 epel5-release
  
−
  99  koji add-tag dist-epel5
+
To resolve build dependencies, this tag is used as the build target ''source''.
−
  100 koji add-tag --parent dist-epel5 --arches "i386 x86_64" dist-epel5-build
+
  koji add-tag --parent epel5 --arches "i386 x86_64" epel5-build
−
  102  koji add-external-repo -t dist-epel5-build dist-epel5-external-repo http://mirrors.nl.eu.kernel.org/fedora-epel/5/\$arch/
 
−
  103  koji add-external-repo -t dist-epel5-build dist-epel5-centos5-repo http://spiegel.nikhef.nl/mirror/centos/5/os/\$arch/
 
−
  104  koji add-target dist-epel5 dist-epel5-build
 
  
−
These groups are going to be passed to 'yum groupinstall' when the buildroot is created.
+
A Koji build will first do a <nowiki>buildSRPMFromSCM</nowiki> job, which will run mock to generate a buildroot, inside of which 'yum groupinstall srpm-build' is run. Later, a <nowiki>buildArch</nowiki> job does the actual building, and here mock runs 'yum groupinstall build'.
 +
These groups must be created by us first, and populated with package names that we need to have available at this stage.
 +
koji add-group epel5-build build
 +
koji add-group epel5-build srpm-build
 +
 +
They need to be populated. Clone from a typical value for the Fedora Koji systems, e.g.
  
−
  106 koji add-group dist-epel5-build build
+
  koji add-group-pkg epel5-build build `koji -s http://koji.fedoraproject.org/kojihub/
−
  107 koji add-group dist-epel5-build srpm-build
+
      list-groups dist-5E-epel-build build | tail -n +2 | cut -d: -f1`
 +
Be careful to exclude the fedpkg package in srpm-build which pulls in way too much! It includes mock which causes strange failures with conflicting (unix) group ids within the buildroot.
 +
  koji add-group-pkg epel5-build srpm-build `koji -s http://koji.fedoraproject.org/kojihub/
 +
      list-groups dist-5E-epel-build srpm-build | tail -n +2 | cut -d: -f1 | grep -v fedpkg`
  
−
They need to be populated
+
As of EPEL7, there's fedpkg-minimal which should be safe (it only requires wget).
  
−
  109 koji -s http://koji.fedoraproject.org/koji list-groups dist-epel5-build
+
Add external repositories to populate the buildroot.
−
  110  koji -s https://koji.fedoraproject.org/koji list-groups dist-epel5-build
+
  koji add-external-repo -t epel5-build \
 +
      dist-epel5-external-repo http://mirrors.nl.eu.kernel.org/fedora-epel/5/\$arch/
 +
koji add-external-repo -t epel5-build dist-epel5-buildsys-macros-repo \
 +
      http://buildsys.fedoraproject.org/buildgroups/rhel5/\$arch/
 +
koji add-external-repo -t epel5-build dist-epel5-centos5-repo \
 +
      http://spiegel.nikhef.nl/mirror/centos/5/os/\$arch/
  
−
Clone from a typical value for the Fedora Koji systems. Be careful to exclude the fedpkg package which pulls in way too much! It includes mock which causes strange failures with conflicting group ids within the buildroot.
+
Define the build target based on these tags.
 +
koji add-target epel5 epel5-build epel5-candidates
  
−
  112  koji add-group-pkg dist-epel5-build build `cat epel5-build-pkgs `
+
'''NOTE''': the buildsys external repo is ONLY needed on EPEL5 to provide the correct buildsys-macros package. The standard CentOS package causes the dist macro to expand to el5.centos. By providing this external Fedora repo and putting it before the centos5 repo, the correct buildsys-macros is installed. See also [https://fedoraproject.org/wiki/Koji/KojiMisc#.25dist_tags https://fedoraproject.org/wiki/Koji/KojiMisc#.25dist_tags].
−
  114 koji add-group-pkg dist-epel5-build srpm-build `cat epel5-srpm-build-pkgs `
+
If needed (i.e. when the ordering turns out to be wrong) we can provide explicit priorities using a -p flag to add-external-repo. Check with
 +
  koji taginfo epel5-build
  
−
Don't forget to add the package!
 
  
−
  140  koji add-pkg --owner okoeroo dist-epel5 lcmaps
+
Since the srpm-build and build do not vary much between distributions,
−
  124  koji build dist-epel5 git://bleek.testbed/koji.git\?lcmaps/#a942e490ace1da950cedae2976bf59fa3b3f5038
+
we create a separate tag and use inheritance to save work for the Fedora branches.
 +
This does not hold true for EPEL branches, which are not inheriting from one another.
  
−
The git repository must be added to the allowed repositories on the builder in kojid.conf.
+
for i in 27 28 ; do
 +
    prev=`expr $i - 1`
 +
    koji add-tag --parent f$prev f$i
 +
    koji add-tag --parent f$prev-basedeps f${i}-basedeps
 +
    # koji add-tag --parent f$i --parent f$i-basedeps --arches "i386 x86_64" f${i}-build
 +
    koji add-tag --parent f$i --parent f${i}-basedeps --arches "x86_64" f${i}-build
 +
    for j in candidates testing release; do
 +
        koji add-tag --parent f$i f${i}-$j
 +
    done
 +
    koji add-external-repo -t f${i}-build dist-f${i}-base http://mirror.nl.leaseweb.net/fedora/linux/releases/${i}/Everything/\$arch/os/
 +
    # '''NOTE''': up to 27, we should leave out the ''Everything'' for the updates
 +
    koji add-external-repo -t f${i}-build dist-f${i}-updates http://mirror.nl.leaseweb.net/fedora/linux/updates/${i}/Everything/\$arch/
 +
    koji add-target f$i f${i}-build f${i}-candidates
 +
    # '''NOTE''': probably need to add priorities to the parents, e.g. using something like
 +
    koji edit-tag-inheritance --priority 2 f${i}-build f${i}-basedeps
 +
    koji add-tag-inheritance --priority 1 f${i}-build f${i}
 +
done
  
 
= Automated builds from tags in mwsec =
 
= Automated builds from tags in mwsec =
 +
 +
Koji can build from various revision control systems, such as git and SVN. The way this works is to give <code>koji build</code> a parameter in the form
 +
''(repository URL)''?''(component path)''#''(revision ID)''
 +
 +
The repository must be added to the allowed repositories on each builder (in kojid.conf):
 +
/etc/kojid/kojid.conf:
 +
...
 +
allowed_scms=bleek.testbed:/koji.git:no ndpfsvn.nikhef.nl:/ro/mwsec:no
 +
...
 +
 +
== Building from git (deprecated) ==
 +
 +
This step still needs to be automated with a commit hook. The git repository on git://bleek.testbed/koji.git has one subdirectory per component, and each directory contains:
 +
* a SPEC file
 +
* a Makefile with a 'sources' target, which will fetch (with curl) the sources for the package.
 +
After pushing the updates to this repository, record the commit hash and start the koji build like so (replace the hash with yours):
 +
 +
koji build dist-epel5 'git://bleek.testbed/koji.git?lcmaps/#7a2b193fdd8a26875a16303da609952cccc4cba8'
 +
 +
The format is particular to koji; it is parsed as an URL and that is why the ? and # are used to mark certain elements.
  
 
== Triggering a build in svn ==
 
== Triggering a build in svn ==
Line 111: Line 241:
  
 
Within that directory, create a Makefile and a spec file. The spec file can be the same as the one from packaging/fedora/trunk for that component.
 
Within that directory, create a Makefile and a spec file. The spec file can be the same as the one from packaging/fedora/trunk for that component.
−
The Makefile can be fairly generic, in most cases this should suffice:
+
The Makefile is generic, in most cases this should suffice:
−
  dir = $(shell pwd)
+
  cwd = $(abspath .)
−
  version = $(notdir $(dir))
+
  version = $(notdir $(cwd))
−
  component = lcmaps-plugins-verify-proxy
+
  component = $(notdir $(patsubst %/, %, $(dir $(cwd))))
 
   
 
   
 
  sources:
 
  sources:
Line 121: Line 251:
 
   
 
   
 
Should a rebuild be necessary because of errors in Koji or in the spec file, update the Release field in the spec file as usual.
 
Should a rebuild be necessary because of errors in Koji or in the spec file, update the Release field in the spec file as usual.
−
 
+
'''There is no way to remove or redo a successful build,''' if it is bad, just remove all tags.
−
 
 
  
 
== Scripts on the svn repository ==
 
== Scripts on the svn repository ==
Line 150: Line 279:
 
  # And it's the middle one we trigger on.
 
  # And it's the middle one we trigger on.
 
   
 
   
−
  svnlook changed -r $REV "$REPOS" | awk '$1 ~ /(A|U)/ && $2 ~ /packaging\/fedora\/tags\/[[:alnum:]-]+\/[[:alnum:].]+\/.*\.spec$/ {  
+
  svnlook changed -r $REV "$REPOS" | \
 +
    awk '$1 ~ /(A|U)/ && $2 ~ /packaging\/fedora\/tags\/[[:alnum:]-]+\/[[:alnum:].]+\/.*\.spec$/ {  
 
     split($2, pe, "/");
 
     split($2, pe, "/");
 
     print pe[4], pe[5];
 
     print pe[4], pe[5];
Line 182: Line 312:
 
  fi
 
  fi
 
   
 
   
−
  /usr/bin/koji -c /project/srv/svn/home/.koji/koji.conf build --nowait dist-epel5 svn+http://ndpfsvn.nikhef.nl/ro/mwsec\?packaging/fedora/tags/${component}/${tag}#$revision
+
  /usr/bin/koji -c /project/srv/svn/home/.koji/koji.conf build --nowait dist-epel5 \
−
  /usr/bin/koji -c /project/srv/svn/home/.koji/koji.conf build --nowait dist-epel6 svn+http://ndpfsvn.nikhef.nl/ro/mwsec\?packaging/fedora/tags/${component}/${tag}#$revision
+
    svn+http://ndpfsvn.nikhef.nl/ro/mwsec\?packaging/fedora/tags/${component}/${tag}#$revision
−
  /usr/bin/koji -c /project/srv/svn/home/.koji/koji.conf build --nowait dist-fc16 svn+http://ndpfsvn.nikhef.nl/ro/mwsec\?packaging/fedora/tags/${component}/${tag}#$revision
+
  /usr/bin/koji -c /project/srv/svn/home/.koji/koji.conf build --nowait dist-epel6 \
−
  /usr/bin/koji -c /project/srv/svn/home/.koji/koji.conf build --nowait dist-fc17 svn+http://ndpfsvn.nikhef.nl/ro/mwsec\?packaging/fedora/tags/${component}/${tag}#$revision
+
    svn+http://ndpfsvn.nikhef.nl/ro/mwsec\?packaging/fedora/tags/${component}/${tag}#$revision
 +
  /usr/bin/koji -c /project/srv/svn/home/.koji/koji.conf build --nowait dist-fc16 \
 +
    svn+http://ndpfsvn.nikhef.nl/ro/mwsec\?packaging/fedora/tags/${component}/${tag}#$revision
 +
  /usr/bin/koji -c /project/srv/svn/home/.koji/koji.conf build --nowait dist-fc17 \
 +
    svn+http://ndpfsvn.nikhef.nl/ro/mwsec\?packaging/fedora/tags/${component}/${tag}#$revision
  
 
* koji.conf:
 
* koji.conf:
Line 224: Line 358:
  
 
  iptables -t nat -A PREROUTING --proto tcp --dport 8443 --destination 194.171.96.17 -j DNAT --to-destination 10.198.8.7:443
 
  iptables -t nat -A PREROUTING --proto tcp --dport 8443 --destination 194.171.96.17 -j DNAT --to-destination 10.198.8.7:443
 +
 +
= Setting up sigul to sign RPMs  =
 +
 +
Before the RPMs that Koji produces are distributed, they should be signed. The YUM configuration can then be configured to use GPG to verify the origin of the RPMs upon installing.
 +
 +
Sigul integrates well with Koji, but unfortunately comes with little documentation. There is some [https://fedoraproject.org/wiki/Create_release_signing_key here].
 +
 +
The architecture of sigul is as follows:
 +
 +
[[File:Sigul-arch.png]]
 +
 +
The sigul server node holds the secret key(s) and must be secured very tightly. The sigul bridge node only allows connections from the client and the server, it will make no outgoing connections.
 +
 +
Sigul is available as a package in CentOS 6, and hold all three components. The client is installed on koji-hub.testbed; the bridge on koji-bridge.testbed and the server on sigul.testbed. The latter two cannot be accessed through ssh, only console access (wich 'virsh console' or on the virtual video console).
 +
 +
'''Note:''' the firewall on sigul and the sigul bridge should have a line to accept packets from eachother, as the stateful tracking doesn't work so well. The connection stays open for too long and the firewall loses the state.
 +
iptables -R INPUT 1 -p tcp -s 10.198.8.18 \! --syn -j ACCEPT
 +
TODO: write up how to set up a new key and put it to work in sigul.
  
 
= Generating distribution repos from koji =  
 
= Generating distribution repos from koji =  
  
−
To render the products of koji into something that can be installed by yum, we use the mash utility installed on koji-hub.testbed. The output of mash should be send to /mnt/koji/dists, and from there synced to software.nikhef.nl.
+
To render the products of koji into something that can be installed by yum, we use the mash utility installed on <tt>koji-hub.testbed</tt>. The repositories are made using mash on an NFS share from bleek.nikhef.nl:
 +
/srv/project/mwsec  10.198.8.7(rw,async,no_root_squash,insecure)
 +
 
 +
bleek.testbed:/srv/project/mwsec        /mnt/mwsec      nfs defaults  0 0
 +
 
 +
The output of mash should be send to <tt>/mnt/mwsec/mash</tt>. We create symlinks to the different distributions in <tt>/mnt/mwsec/rpm</tt> to allow syncing to <tt>software.nikhef.nl</tt>, the reason is that mash cannot easily put testing RPMs in <tt>./testing/epel5/</tt> and release RPMs in <tt>./epel5/</tt> since the directory is taken directly from the section name in the <tt>mwsec.mash</tt> config file:
 +
lrwxrwxrwx  rpm/epel5 -> ../mash/epel5-release
 +
lrwxrwxrwx  rpm/epel6 -> ../mash/epel6-release
 +
lrwxrwxrwx  rpm/fc16 -> ../mash/fc16-release
 +
lrwxrwxrwx  rpm/fc17 -> ../mash/fc17-release
 +
drwxrwsr-x  rpm/testing
 +
lrwxrwxrwx  rpm/testing/epel5 -> ../../mash/epel5-testing
 +
lrwxrwxrwx  rpm/testing/epel6 -> ../../mash/epel6-testing
 +
lrwxrwxrwx  rpm/testing/fc16 -> ../../mash/fc16-testing
 +
lrwxrwxrwx  rpm/testing/fc17 -> ../../mash/fc17-testing
  
−
The mash configuration has one main config file and one file per distribution: /etc/mash/mash.conf
+
For each distribution we create two additional tags -release and -testing:
 +
koji add-tag --parent dist-epel5 dist-epel5-release
 +
koji add-tag --parent dist-epel5 dist-epel5-testing
 +
and likewise for epel6, fc16 and fc17.
 +
Succesful builds can be added to either testing or release:
 +
koji tag-pkg dist-epel5-release glexec-0.9.7-1.el5
 +
Or moved from testing to release
 +
koji move-pkg dist-epel5-testing dist-epel5-release lcmaps-1.5.7-1.el5
 +
 
 +
The mash configuration has one main config file and multiple distribution files: /etc/mash/mash.conf
 
  [defaults]
 
  [defaults]
 
  configdir = /etc/mash
 
  configdir = /etc/mash
Line 237: Line 412:
 
  use_repoview = True
 
  use_repoview = True
  
−
And e.g. /etc/mash/epel5.mash (the other ones are similar):
+
And /etc/mash/mwsec.mash containing the different distributions:
−
  [epel5]
+
  [epel5-release]
−
  rpm_path = %(arch)s/os/Packages
+
  rpm_path = %(arch)s
−
  repodata_path = %(arch)s/os/
+
  repodata_path = %(arch)s/
−
  source_path = source/SRPMS
+
  source_path = SRPMS
 
  debuginfo = True
 
  debuginfo = True
−
  multilib = True
+
# Do not create i386s in x86_64
 +
  multilib = False
 
  multilib_method = devel
 
  multilib_method = devel
−
  tag = dist-epel5
+
# Use old-style sha hash for EPEL5 only
−
  inherit = True
+
hash = sha
 +
  tag = dist-epel5-release
 +
# Next one is needed to prevent mixing of testing and release.
 +
  inherit = False
 
  strict_keys = False
 
  strict_keys = False
−
  repoviewurl = http://koji-hub.testbed/repo-view/%(arch)s/os/
+
#keys = 22B3B81A, 1ACA3465, A82BA4B7, 069C8460, 97A1071F, E8E40FDE, 57BBCCBA, D22E77F2, 4EBFC273, 0B86274E, 6DF2196F, DF9B0AE9
−
  repoviewtitle = "MWSEC for EPEL5 - %(arch)s"
+
# Note: repoviewurl used in latest-feed.xml for RPM location
 +
  repoviewurl = /dist/mwsec/rpm/epel5/%(arch)s
 +
  repoviewtitle = MWSEC for EPEL5 - %(arch)s
 
  arches = i386 x86_64
 
  arches = i386 x86_64
 
  delta = False
 
  delta = False
 +
# Change distro_tags as fedora-release version gets bumped
 +
#distro_tags = cpe:/o:fedoraproject:fedora:17 rawhide
 
  hash_packages = False
 
  hash_packages = False
 +
# This forces to also create older version of same package
 +
latest = False
  
−
To run mash, simply use the following command:
+
The testing repository is the same except:
−
  for i in epel{5,6} fc{16,17} ; do mash -o /mnt/koji/dists $i ; done
+
  tag = dist-epel5-testing
 +
repoviewurl = /dist/mwsec/rpm/testing/epel5/%(arch)s
 +
repoviewtitle = MWSEC Testing for EPEL5 - %(arch)s
  
−
and rsync to software.nikhef.nl:
+
The epel6, fc16 and fc17 are the same except that they do not override the <tt>hash =</tt>
−
  rsync -a --delete /mnt/koji/dists/ software.nikhef.nl:/project/srv/www/site/software/html/experimental/mwsec/
+
 +
To run mash, simply use the following command, NOTE: it's best to have a umask 002 setting
 +
  for dist in epel{5,6} fc{16,17} ; do
 +
    mash -o /mnt/mwsec/mash ${dist}-release
 +
    mash -o /mnt/mwsec/mash ${dist}-testing
 +
done
  
−
= Nightly builds from mwsec/trunk =
+
'''Note''': Mash completely removes and recreates the output directories! For epel5 we have old builds which should afterwards be merged back in, we use a tarball with symlinks in <tt>/mnt/mwsec/old_releases.tgz</tt>. The relative symlinks point to the actual files in <tt>/mnt/mwsec/mock/</tt>.
 +
The directory <tt>/mnt/mwsec/mock/</tt> is a read-only NFS share on <tt>bleek.nikhef.nl</tt>:
 +
/srv/project/mock  10.198.0.0/16(ro,async,root_squash,insecure)
  
−
The dedicated account kojinb for nightly builds is added on bleek.
+
bleek.testbed:/srv/project/mock        /mnt/mock      nfs defaults  0 0
 +
By having the mock and mwsec directories at the same level, we can use relative symlinks from one to the other both on <tt>bleek.nikhef.nl</tt> and <tt>koji-hub.testbed</tt>.  
  
−
At the moment nightly builds are run from a crontab of account dennisvd.
+
See [[SAC_software_procedures#Updating the mash repositories]] for a practical procedure.
−
01 2 * * * ${HOME}/bin/koji-nightly
 
  
−
The koji-nightly script:
+
= Nightly builds from mwsec/trunk =
−
#!/bin/sh
 
−
 
−
PATH=/bin:/usr/bin
 
−
 
−
components="
 
−
lcas
 
−
lcmaps
 
−
ees
 
−
glexec
 
−
jobrepository
 
−
lcas-lcmaps-gt4-interface
 
−
lcas-plugins-basic
 
−
lcas-plugins-check-executable
 
−
lcas-plugins-voms
 
−
lcmaps-plugins-afs
 
−
lcmaps-plugins-basic
 
−
lcmaps-plugins-c-pep
 
−
lcmaps-plugins-gums
 
−
lcmaps-plugins-jobrep
 
−
lcmaps-plugins-scas-client
 
−
lcmaps-plugins-tracking-groupid
 
−
lcmaps-plugins-verify-proxy
 
−
lcmaps-plugins-voms
 
−
scas
 
−
"
 
−
 
−
dists="epel5 epel6 fc16 fc17"
 
−
 
−
rev=`svn info http://ndpfsvn.nikhef.nl/ro/mwsec/ | sed -n 's/^Revision: //p'`
 
−
 
−
for i in $components ; do
 
−
    for j in $dists ; do
 
−
koji -c ${HOME}/.koji/nightly.conf build --nowait nightly-$j svn+http://ndpfsvn.nikhef.nl/ro/mwsec\?packaging/fedora/branches/nightly-builds/$i/#$rev
 
−
    done
 
−
done
 
  
−
Each component directory in the nightly-builds branch contains the same set of files: a Makefile:
+
We no longer run nightly builds. That should move to a continuous integration system.
−
component = $(notdir $(shell pwd))
 
−
 
−
sources:
 
−
./fetch-sources $(component)
 
  
−
And a fetch-sources script:
+
= Troubleshooting =
−
#!/bin/sh
 
−
 
−
set -e
 
−
 
−
component=$1
 
−
 
−
svn co http://ndpfsvn.nikhef.nl/ro/mwsec/trunk/${component}
 
−
cd ${component}
 
−
revision=`svn info | sed -n 's/^Last Changed Rev: //p'`
 
−
 
−
if [ -z $revision ]; then
 
−
    echo "Could not retrieve revision from svn info" >&2
 
−
    echo "svn info:" >&2
 
−
    svn info >&2
 
−
    exit 1
 
−
fi
 
−
 
−
./bootstrap
 
−
./configure
 
−
make dist
 
−
version=`echo ${component}-*.tar.gz | sed -n "s/${component}-\\([0-9.]*\\)\\.tar\\.gz/\\1/p"`
 
−
 
−
if [ -z $version ]; then
 
−
    echo "Could not retrieve version from generated tarball" >&2
 
−
    echo "tarballs: " ${component}-*.tar.gz >&2
 
−
    exit 1
 
−
fi
 
−
 
−
mv ${component}-*.tar.gz ..
 
−
cd ..
 
−
 
−
sed -i -e "s/^Version: .*/Version: $version/" -e "s/^Release: .*/Release: 0.$revision%{?dist}/" ${component}.spec
 
  
−
The list of components misses just two packages: argus-pep-api-c, which is not under our active development, and xacml, which has some problems at this moment; gsoap-devel needs to be added to the build-srpm group (done) and as the nightly build does the bootstrapping as well, the autoconf for the epel5 tag is too old according to configure.in. Whether this is really the case remains to be seen.  
+
This section covers a couple of issues encountered in the wild, some of which weren't trivial to understand or solve.
  
−
== Preventing repeat builds ==
+
== low-level database access ==
  
−
Koji will refuse build artefacts that it already has. The nightly builds are versioned according to the svn revision of the latest update of the tree to build. That means that if no updates are done to the tree, there is no sense in repeating the build. The cron script should be enhanced to deal with this situation.
+
Some koji commands to add things do not have a remove-things counterpart, such as add-group and add-group-pkg. If removal is needed, access to the PostgreSQL database is easy: change to user 'koji' on koji-hub.testbed and run psql.
  
−
== odd details ==
+
== libtoolize: can not copy ==
  
 
On fc17 libtoolize may fail with a strange error message:
 
On fc17 libtoolize may fail with a strange error message:
Line 357: Line 480:
 
which is due to 'tar' missing on the system. See bug https://bugzilla.redhat.com/show_bug.cgi?id=794675.
 
which is due to 'tar' missing on the system. See bug https://bugzilla.redhat.com/show_bug.cgi?id=794675.
 
The resolution is to include tar as a required package in the srpm-build group.
 
The resolution is to include tar as a required package in the srpm-build group.
 +
 +
== GenericError: failed to create repo (out of memory) ==
 +
 +
The regeneration of repositories which is done by kojira between builds is a memory-hungry task. As builders are configured by default to accept a load of 2 tasks, they may receive two createrepo/mergerepo jobs and run out of memory. If such tasks fail with a message like:
 +
  GenericError: failed to create repo: /usr/bin/createrepo -vd
 +
-o /var/lib/koji/tasks/257/257/repo -u http://koji-hub.testbed/kojifiles
 +
-i /mnt/koji/repos/dist-fc17-build/45/i386/pkglist -g /mnt/koji/repos/dist-fc17-build/45/groups/comps.xml
 +
--update --skip-stat /mnt/koji was killed by signal 9
 +
(line breaks inserted for formatting, the error is on a single line)
 +
check /var/log/messages for signs of the OOM killer springing into action.
 +
 +
== PYCURL ERROR 7 - "couldn't connect to host" ==
 +
 +
If all builds suddenly start failing for no apparent reason, and the error message from mock reads something like:
 +
DEBUG util.py:257:  http://koji-hub.testbed/kojifiles/repos/dist-epel5-build/1472/i386/repodata/repomd.xml:
 +
[Errno 14] PYCURL ERROR 7 - "couldn't connect to host"
 +
(line break inserted for formatting)
 +
this is likely due to the fact that the [[#Using a squid proxy for external repositories|squid service]] is not running anymore.
 +
 +
 +
== Sigul signing seems to hang ==
 +
 +
We've seen the weird situation that the connection between the bridge and the server was semi-severed: the bridge still showed the connection in the output of netstat, but the server lost the connection. The resolution is to restart the server. If the bridge needs to be restarted, the server also needs a restart thereafter.
 +
 +
UPDATE: the cause of this problem lies with the connection being too quiet. Without keepalive the connection will be dropped. It's the stateful firewall's fault.

Latest revision as of 14:31, 13 February 2023

Koji setup on the testbed

The Koji system is used by Fedora for builds of packages for Fedora Core and EPEL. It is open source and available as a standard package, and documentation for running a private installation is available.

Koji consists of a number of interrelated components:

  • Koji Hub, the central XMLRPC service.
  • A postgresql database as a backend for the hub
  • Koji Web, a user interface front-end for the hub (other interaction is by the command-line client)
  • Kojira, managing repository building and clean-up.
  • One or more builders, on separate machines.

Linked to Koji in some functional way, but not part of its core, are some additional systems:

  • sigul, a secure gpg signing system to create signatures for rpms;
  • mash, a system to create yum repositories to release builds to the general public.

It is possible to install most of these components on separate machines, our setup is as follows:

  • koji-hub.testbed, holding the hub, web, kojira and database
  • koji-builder.testbed, a builder with 8 GB of memory which also does kojira for creating repositories
  • koji-boulder.testbed, a normal builder with 4 GB of memory
  • put.testbed, which holds the shared NFS storage for all build artefacts
  • bleek.testbed, which holds the mash output before synchronising to software.nikhef.nl
  • koji-bridge.testbed, part of the sigul setup
  • sigul.testbed, the sigul signing server

The machines share access to a large NFS share on put.testbed:/mnt/put/koji for storing results and buildroots (the builders only have read-only access). Currently the machines are deployed on the M610 blades (blade13 and blade14) with the possibility for hot migration.

The installation mostly followed the very complete guidance on [1], choosing the SSL authentication option.

NOTE In december we upgraded to 1.8, and the database schema needed to change. See the migration documentation in /usr/share/doc/koji on koji-hub.testbed. --Dennisvd@nikhef.nl (talk) 10:01, 15 January 2014 (CET)

Using a squid proxy for external repositories

The builds use mock to install a base set of packages to fulfill the BuildRequires in the spec file. These packages come from CentOS and Fedora mirrors, and in order to reduce the load on the mirrors a local squid caching proxy is configured to store up to 100GB of popular packages. To use the proxy, set 

yum_proxy = http://span.testbed:3128/

in /etc/kojid/kojid.conf on all builders.

memory and disk space requirement

The basic builders are configured with 4096MB memory at the moment which should be enough (but see the #Troubleshooting section below). The createrepo and mergerepo processes can eat enormous amounts of RAM (they slurp in the external repositories with all of Fedora's packages), so koji-builder.testbed has been given 8 GB of memory and kojira is limited to using just this machine.

The disk space on the builders could easily run out, as there is no automatic clean-up of used build directories.

The builders have a tmpwatch script in /etc/cron.daily that erases old data (older than 4 hours) 

/usr/bin/tmpwatch -caMf 4 /mnt/mock/


SSL Authentication

The authentication system is picky when it comes to Common Names. For users, they have to be the same as user ids, and for servers (only koji-hub.testbed, really) it has to be the fully qualified host name.

A new CA was created for the purpose of handing out certificates (as reuse of existing certificates was hard because of the above restriction, but we may still find a way around that sometime). The CA installation is currently found on bleek.testbed:/srv/koji-pki.

For access to koji-web, the user needs to export the certificate file from there to a pkcs12 file and import it in the browser. For command-line koji use, put it in ~/.koji/clientcert.crt.

Adding tags, targets and repositories

See the main documentation on the Fedora wiki.

We've had several experimental setups before coming to understand what everything really meant. So here is a bit of documentation on the terminology in use, and the apparent logic of the system.

Terminology

The most confusing word in Koji context is the word 'package', which means two related but very different things:

  1. the base name of a package, as colloquially used in a sentence like: �Hey, is lcmaps installed on this box?�
  2. the full name-version-release or NVR of the package, such as lcmaps-1.6.1-2.el6.

With that out of the way, the central structural element in Koji is the tag. This is, of course, just a name, but many things can have tags attached to them. In fact, it is easier to reverse the idea of sticking a tag onto something and rather think of tags as pinboards where stuff is tacked onto. Here is a list of things that can go on tags:

  • packages, in the first sense as described above
  • packages, in the second sense as described above
  • architectures, such as i386 and x86_64 (in fact, only those in our case)
  • groups (we'll get to those later)
  • external repositories
  • build targets (source or destination)
  • other tags called �parents�

The second most confusing word in Koji context is the word 'parent' and, along with that, the entire concept of inheritance. In the biological sense we can't choose, change, or add parents, but tags have no such limitations. While normally we inherit from our parents after they pass away, tags inherit everything from all of their parents (and parents' parents) immediately. Remember: in the graphical displays, Koji shows parents down and to the right in the inheritance tree.

The builds are done against build targets, which have source and target tags. The way this works in a nutshell is as follows:

  • the user issues the build of a package (1st sense) for a specific target
  • the target's build tag is chosen to create a build root
  • dependencies are resolved, hopefully
  • the build runs, and if all is well this results in a new artefact (a package in the second sense)
  • this package is tagged with the target's destination tag

Resolving dependencies and setting up a build root works by combining external repositories and packages with the build tag, and installing a meta-package with 'yum groupinstall'. This is where the concept of groups comes in. There are two groups for the different phases of a build: srpm-build for the creation of source RPMs and build for the actual build. The groups as created in koji contain only package names.

Current setup

Fedora releases are built one on top another, and using inheritance makes perfect sense. But there is a catch. Builds, a.k.a. packages in the second sense, are inherited just the same. As are external repositories. We've experienced that our build tag for Fedora 18 collected everything from Fedora 17 and 16 (when we started), including all the repositories. This went crazy.

The solution is to be careful about what we inherit. The current setup is displayed best by the following pictures: 

$ koji list-tag-inheritance --reverse f16
f16 (7)  <- contains ONLY packages in the first sense
  +-f17 (9)  <- so does this, by inheritance
  |  +-f17-candidates (33)  <- this is the "build target" "destination tag"
  |  +-f17-release (25)  <- this is used manually to mark a package (second sense) ready for shipping with mash
  |  +-f17-testing (26)  <- idem, but the for the 'testing' repo
  |  +-f18 (27)  <- only packages, first sense
  |  |  +-f18-build (30)  <- the "build target" "build tag"; this has architectures and external repos
  |  |  |  +-f18-candidates (31)  -< the "build target" "destination tag"
  |  |  +-f19 (34)
  |  |  |  +-f19-candidates (38)
  |  |  |  +-f19-release (43)
  |  |  |  +-f19-testing (44)
  |  |  |  +-f19-build (36)
  |  |  +-f18-release (39)
  |  |  +-f18-testing (40)
  |  +-f17-build (32)
  +-f16-release (24)
  +-f16-testing (23)
  +-f16-build (45)

And this one: 

$ koji list-tag-inheritance f19-build
f19-build (36)  <- the "build target" "build tag"
  +-f19 (34)  <- package names only (first sense)
  |  +-f18 (27)  <- all the way
  |     +-f17 (9)  <-  down to
  |        +-f16 (7)  <- here
  +-f19-basedeps (35)  <- these contain the 'groups'
     +-f18-basedeps (28)  <- actually inherited from
        +-f17-basedeps (10)  <- here

A build for F19 ends up in the tag f19-candidates. From there, it needs to be manually moved. If it is a build dependency for other packages, it has to go to f19-build. Otherwise, it eventually goes to f19-testing and from there to f19-release, or directly to f19-release. A rotten product should have its candidacy stripped and ends up with no tags. In any case, no package should remain a candidate for too long.

Because the *-builds don't inherit from each other, we have no pollution of external repositories or old builds for new Fedora releases.

The situation for epel5 and epel6 is analogue.

Steps to follow if you need to do this from scratch

The following doesn't exactly reflect the current setup, but we'd do it this way if we had to start over.

Create a tag to collect all our packages in. 

koji add-tag epel5
koji add-pkg --owner <your name> epel5 lcmaps jobrepository ees glexec \
         lcas-lcmaps-gt4-interface lcas-plugins-basic lcas-plugins-check-executable \
         lcas-plugins-voms lcas lcmaps-plugins-afs lcmaps-plugins-basic lcmaps-plugins-c-pep lcmaps-plugins-gums \
         lcmaps-plugins-jobrep lcmaps-plugins-scas-client lcmaps-plugins-tracking-groupid lcmaps-plugins-verify-proxy \
         lcmaps-plugins-voms scas xacml argus-pep-api-c glexec-wn nagios-plugins-glexec nagios-plugins-ees mkgltempdir \
         glexec-wrapper-scripts torque llrun saml2-xacml2-c-lib lcmaps-without-gsi mwsec-release

Another tag to use as a build target destination. All builds end up here, before they are moved to 'testing' or 'release' repositories. 

koji add-tag --parent epel5 epel5-candidates

These tags are used by the 'mash' tool to publish repositories. 

koji add-tag --parent epel5 epel5-testing
koji add-tag --parent epel5 epel5-release

To resolve build dependencies, this tag is used as the build target source. 

koji add-tag --parent epel5 --arches "i386 x86_64" epel5-build

A Koji build will first do a buildSRPMFromSCM job, which will run mock to generate a buildroot, inside of which 'yum groupinstall srpm-build' is run. Later, a buildArch job does the actual building, and here mock runs 'yum groupinstall build'. These groups must be created by us first, and populated with package names that we need to have available at this stage. 

koji add-group epel5-build build
koji add-group epel5-build srpm-build

They need to be populated. Clone from a typical value for the Fedora Koji systems, e.g. 

koji add-group-pkg epel5-build build `koji -s http://koji.fedoraproject.org/kojihub/
     list-groups dist-5E-epel-build build | tail -n +2 | cut -d: -f1`

Be careful to exclude the fedpkg package in srpm-build which pulls in way too much! It includes mock which causes strange failures with conflicting (unix) group ids within the buildroot. 

koji add-group-pkg epel5-build srpm-build `koji -s http://koji.fedoraproject.org/kojihub/
     list-groups dist-5E-epel-build srpm-build | tail -n +2 | cut -d: -f1 | grep -v fedpkg`

As of EPEL7, there's fedpkg-minimal which should be safe (it only requires wget).

Add external repositories to populate the buildroot. 

koji add-external-repo -t epel5-build \
      dist-epel5-external-repo http://mirrors.nl.eu.kernel.org/fedora-epel/5/\$arch/
koji add-external-repo -t epel5-build dist-epel5-buildsys-macros-repo \
      http://buildsys.fedoraproject.org/buildgroups/rhel5/\$arch/
koji add-external-repo -t epel5-build dist-epel5-centos5-repo \
      http://spiegel.nikhef.nl/mirror/centos/5/os/\$arch/

Define the build target based on these tags. 

koji add-target epel5 epel5-build epel5-candidates

NOTE: the buildsys external repo is ONLY needed on EPEL5 to provide the correct buildsys-macros package. The standard CentOS package causes the dist macro to expand to el5.centos. By providing this external Fedora repo and putting it before the centos5 repo, the correct buildsys-macros is installed. See also https://fedoraproject.org/wiki/Koji/KojiMisc#.25dist_tags. If needed (i.e. when the ordering turns out to be wrong) we can provide explicit priorities using a -p flag to add-external-repo. Check with 

koji taginfo epel5-build


Since the srpm-build and build do not vary much between distributions, we create a separate tag and use inheritance to save work for the Fedora branches. This does not hold true for EPEL branches, which are not inheriting from one another. 

for i in 27 28 ; do
    prev=`expr $i - 1`
    koji add-tag --parent f$prev f$i
    koji add-tag --parent f$prev-basedeps f${i}-basedeps
    # koji add-tag --parent f$i --parent f$i-basedeps --arches "i386 x86_64" f${i}-build
    koji add-tag --parent f$i --parent f${i}-basedeps --arches "x86_64" f${i}-build
    for j in candidates testing release; do
        koji add-tag --parent f$i f${i}-$j
    done
    koji add-external-repo -t f${i}-build dist-f${i}-base http://mirror.nl.leaseweb.net/fedora/linux/releases/${i}/Everything/\$arch/os/
    # NOTE: up to 27, we should leave out the Everything for the updates
    koji add-external-repo -t f${i}-build dist-f${i}-updates http://mirror.nl.leaseweb.net/fedora/linux/updates/${i}/Everything/\$arch/
    koji add-target f$i f${i}-build f${i}-candidates
    # NOTE: probably need to add priorities to the parents, e.g. using something like
    koji edit-tag-inheritance --priority 2 f${i}-build f${i}-basedeps
    koji add-tag-inheritance --priority 1 f${i}-build f${i}
done

Automated builds from tags in mwsec

Koji can build from various revision control systems, such as git and SVN. The way this works is to give koji build a parameter in the form 

(repository URL)?(component path)#(revision ID)

The repository must be added to the allowed repositories on each builder (in kojid.conf): 

/etc/kojid/kojid.conf:
...
allowed_scms=bleek.testbed:/koji.git:no ndpfsvn.nikhef.nl:/ro/mwsec:no
...

Building from git (deprecated)

This step still needs to be automated with a commit hook. The git repository on git://bleek.testbed/koji.git has one subdirectory per component, and each directory contains:

  • a SPEC file
  • a Makefile with a 'sources' target, which will fetch (with curl) the sources for the package.

After pushing the updates to this repository, record the commit hash and start the koji build like so (replace the hash with yours): 

koji build dist-epel5 'git://bleek.testbed/koji.git?lcmaps/#7a2b193fdd8a26875a16303da609952cccc4cba8'

The format is particular to koji; it is parsed as an URL and that is why the ? and # are used to mark certain elements.

Triggering a build in svn

the mwsec/packaging/fedora/tags directory is used for triggering builds in Koji. Each component has its own subdirectory, with the version number as a directory below that, so we have e.g.: 

svn+ssh://svn@ndpfsvn.nikhef.nl/repos/mwsec/packaging/fedora/tags/lcmaps/1.5.5/

Within that directory, create a Makefile and a spec file. The spec file can be the same as the one from packaging/fedora/trunk for that component. The Makefile is generic, in most cases this should suffice: 

cwd = $(abspath .)
version = $(notdir $(cwd))
component = $(notdir $(patsubst %/, %, $(dir $(cwd))))

sources:
	curl -O http://software.nikhef.nl/security/$(component)/$(component)-$(version).tar.gz
	sed -i -e 's/^Version: .*/Version: $(version)/' $(component).spec

Should a rebuild be necessary because of errors in Koji or in the spec file, update the Release field in the spec file as usual. There is no way to remove or redo a successful build, if it is bad, just remove all tags.

Scripts on the svn repository

The svn repository needs a post-commit hook to start koji builds automatically from svn. The svn account also needs a koji certificate with build rights. Network access to the koji-hub is by a DNAT proxy through bleek.

The following scripts are installed on sikkel.nikhef.nl: 

/project/srv/svn/repos/mwsec/hooks
-rwxr-xr-x 1 svn apache 1026 Jul 18 14:05 koji-build-trigger
-rwxr-xr-x 1 svn apache 2717 Jul 19 14:27 post-commit
  • post-commit:
REPOS="$1"
REV="$2"

echo "Commit revision $REV to $REPOS" >&2

PATH=/bin:/usr/bin
export PATH

# We are looking for additions (A) or updates (U) to a specific path to trigger
# a koji build. The 'svnlook changed' output may look like:
## A   packaging/fedora/tags/lcmaps/
## A   packaging/fedora/tags/lcmaps/1.2.3/
## A   packaging/fedora/tags/lcmaps/1.2.3/Makefile
# And it's the middle one we trigger on.

svnlook changed -r $REV "$REPOS" | \
    awk '$1 ~ /(A|U)/ && $2 ~ /packaging\/fedora\/tags\/[[:alnum:]-]+\/[[:alnum:].]+\/.*\.spec$/ { 
    split($2, pe, "/");
    print pe[4], pe[5];
}' | while read component tag ; do
	echo "component: $component; tag: $tag" >&2
    if [ -n "$component" -a -n "$tag" ]; then
	"$REPOS"/hooks/koji-build-trigger $component $tag $REV
    fi
done
  • koji-build-trigger:
#!/bin/sh

component="$1"
tag="$2"
revision="$3"

if [ -z "$component" ]; then
    echo "Empty component specified for koji build." >&2
    exit 1
fi

if [ -z "$tag" ]; then
    echo "Empty tag specified for koji build." >&2
    exit 1
fi

if [ -z "$revision" ]; then
    echo "Empty revision specified for koji build." >&2
    exit 1
fi

/usr/bin/koji -c /project/srv/svn/home/.koji/koji.conf build --nowait dist-epel5 \
    svn+http://ndpfsvn.nikhef.nl/ro/mwsec\?packaging/fedora/tags/${component}/${tag}#$revision
/usr/bin/koji -c /project/srv/svn/home/.koji/koji.conf build --nowait dist-epel6 \
    svn+http://ndpfsvn.nikhef.nl/ro/mwsec\?packaging/fedora/tags/${component}/${tag}#$revision
/usr/bin/koji -c /project/srv/svn/home/.koji/koji.conf build --nowait dist-fc16 \
    svn+http://ndpfsvn.nikhef.nl/ro/mwsec\?packaging/fedora/tags/${component}/${tag}#$revision
/usr/bin/koji -c /project/srv/svn/home/.koji/koji.conf build --nowait dist-fc17 \
    svn+http://ndpfsvn.nikhef.nl/ro/mwsec\?packaging/fedora/tags/${component}/${tag}#$revision
  • koji.conf:
[koji]

;configuration for koji cli tool

;url of XMLRPC server
server = https://bleek.nikhef.nl:8443/kojihub

;url of web interface
weburl = https://bleek.nikhef.nl:8443/koji

;url of package download site
topurl = https://bleek.nikhef.nl:8443/kojifiles/packages/

;path to the koji top directory
topdir = /mnt/koji

;configuration for Kerberos authentication

;the service name of the principal being used by the hub
;krbservice = host

;configuration for SSL authentication

;client certificate
cert = ~/.koji/clientcert.crt

;certificate of the CA that issued the client certificate
ca = ~/.koji/koji_ca_cert.crt

;certificate of the CA that issued the HTTP server certificate
serverca = ~/.koji/koji_ca_cert.crt
  • iptables rule on bleek:
iptables -t nat -A PREROUTING --proto tcp --dport 8443 --destination 194.171.96.17 -j DNAT --to-destination 10.198.8.7:443

Setting up sigul to sign RPMs

Before the RPMs that Koji produces are distributed, they should be signed. The YUM configuration can then be configured to use GPG to verify the origin of the RPMs upon installing.

Sigul integrates well with Koji, but unfortunately comes with little documentation. There is some here.

The architecture of sigul is as follows:

Sigul-arch.png

The sigul server node holds the secret key(s) and must be secured very tightly. The sigul bridge node only allows connections from the client and the server, it will make no outgoing connections.

Sigul is available as a package in CentOS 6, and hold all three components. The client is installed on koji-hub.testbed; the bridge on koji-bridge.testbed and the server on sigul.testbed. The latter two cannot be accessed through ssh, only console access (wich 'virsh console' or on the virtual video console).

Note: the firewall on sigul and the sigul bridge should have a line to accept packets from eachother, as the stateful tracking doesn't work so well. The connection stays open for too long and the firewall loses the state. 

iptables -R INPUT 1 -p tcp -s 10.198.8.18 \! --syn -j ACCEPT

TODO: write up how to set up a new key and put it to work in sigul.

Generating distribution repos from koji

To render the products of koji into something that can be installed by yum, we use the mash utility installed on koji-hub.testbed. The repositories are made using mash on an NFS share from bleek.nikhef.nl: 

/srv/project/mwsec  10.198.8.7(rw,async,no_root_squash,insecure)

bleek.testbed:/srv/project/mwsec        /mnt/mwsec      nfs defaults  0 0

The output of mash should be send to /mnt/mwsec/mash. We create symlinks to the different distributions in /mnt/mwsec/rpm to allow syncing to software.nikhef.nl, the reason is that mash cannot easily put testing RPMs in ./testing/epel5/ and release RPMs in ./epel5/ since the directory is taken directly from the section name in the mwsec.mash config file: 

lrwxrwxrwx   rpm/epel5 -> ../mash/epel5-release
lrwxrwxrwx   rpm/epel6 -> ../mash/epel6-release
lrwxrwxrwx   rpm/fc16 -> ../mash/fc16-release
lrwxrwxrwx   rpm/fc17 -> ../mash/fc17-release
drwxrwsr-x   rpm/testing
lrwxrwxrwx   rpm/testing/epel5 -> ../../mash/epel5-testing
lrwxrwxrwx   rpm/testing/epel6 -> ../../mash/epel6-testing
lrwxrwxrwx   rpm/testing/fc16 -> ../../mash/fc16-testing
lrwxrwxrwx   rpm/testing/fc17 -> ../../mash/fc17-testing

For each distribution we create two additional tags -release and -testing: 

koji add-tag --parent dist-epel5 dist-epel5-release
koji add-tag --parent dist-epel5 dist-epel5-testing

and likewise for epel6, fc16 and fc17. Succesful builds can be added to either testing or release: 

koji tag-pkg dist-epel5-release glexec-0.9.7-1.el5

Or moved from testing to release 

koji move-pkg dist-epel5-testing dist-epel5-release lcmaps-1.5.7-1.el5

The mash configuration has one main config file and multiple distribution files: /etc/mash/mash.conf 

[defaults]
configdir = /etc/mash
buildhost = http://koji-hub.testbed/kojihub
repodir = file:///mnt/koji
use_sqlite = True
use_repoview = True

And /etc/mash/mwsec.mash containing the different distributions: 

[epel5-release]
rpm_path = %(arch)s
repodata_path = %(arch)s/
source_path = SRPMS
debuginfo = True
# Do not create i386s in x86_64
multilib = False
multilib_method = devel
# Use old-style sha hash for EPEL5 only
hash = sha
tag = dist-epel5-release
# Next one is needed to prevent mixing of testing and release.
inherit = False
strict_keys = False
#keys = 22B3B81A, 1ACA3465, A82BA4B7, 069C8460, 97A1071F, E8E40FDE, 57BBCCBA, D22E77F2, 4EBFC273, 0B86274E, 6DF2196F, DF9B0AE9
# Note: repoviewurl used in latest-feed.xml for RPM location
repoviewurl = /dist/mwsec/rpm/epel5/%(arch)s
repoviewtitle = MWSEC for EPEL5 - %(arch)s
arches = i386 x86_64
delta = False
# Change distro_tags as fedora-release version gets bumped
#distro_tags = cpe:/o:fedoraproject:fedora:17 rawhide
hash_packages = False
# This forces to also create older version of same package
latest = False

The testing repository is the same except: 

tag = dist-epel5-testing
repoviewurl = /dist/mwsec/rpm/testing/epel5/%(arch)s
repoviewtitle = MWSEC Testing for EPEL5 - %(arch)s

The epel6, fc16 and fc17 are the same except that they do not override the hash =

To run mash, simply use the following command, NOTE: it's best to have a umask 002 setting 

for dist in epel{5,6} fc{16,17} ; do
    mash -o /mnt/mwsec/mash ${dist}-release
    mash -o /mnt/mwsec/mash ${dist}-testing
done

Note: Mash completely removes and recreates the output directories! For epel5 we have old builds which should afterwards be merged back in, we use a tarball with symlinks in /mnt/mwsec/old_releases.tgz. The relative symlinks point to the actual files in /mnt/mwsec/mock/. The directory /mnt/mwsec/mock/ is a read-only NFS share on bleek.nikhef.nl: 

/srv/project/mock   10.198.0.0/16(ro,async,root_squash,insecure)

bleek.testbed:/srv/project/mock         /mnt/mock       nfs defaults  0 0

By having the mock and mwsec directories at the same level, we can use relative symlinks from one to the other both on bleek.nikhef.nl and koji-hub.testbed.

See SAC_software_procedures#Updating the mash repositories for a practical procedure.

Nightly builds from mwsec/trunk

We no longer run nightly builds. That should move to a continuous integration system.

Troubleshooting

This section covers a couple of issues encountered in the wild, some of which weren't trivial to understand or solve.

low-level database access

Some koji commands to add things do not have a remove-things counterpart, such as add-group and add-group-pkg. If removal is needed, access to the PostgreSQL database is easy: change to user 'koji' on koji-hub.testbed and run psql.

libtoolize: can not copy

On fc17 libtoolize may fail with a strange error message: 

libtoolize: can not copy `/usr/share/libtool/config/ltmain.sh' to `./'

which is due to 'tar' missing on the system. See bug https://bugzilla.redhat.com/show_bug.cgi?id=794675. The resolution is to include tar as a required package in the srpm-build group.

GenericError: failed to create repo (out of memory)

The regeneration of repositories which is done by kojira between builds is a memory-hungry task. As builders are configured by default to accept a load of 2 tasks, they may receive two createrepo/mergerepo jobs and run out of memory. If such tasks fail with a message like: 

  GenericError: failed to create repo: /usr/bin/createrepo -vd
-o /var/lib/koji/tasks/257/257/repo -u http://koji-hub.testbed/kojifiles
-i /mnt/koji/repos/dist-fc17-build/45/i386/pkglist -g /mnt/koji/repos/dist-fc17-build/45/groups/comps.xml
--update --skip-stat /mnt/koji was killed by signal 9

(line breaks inserted for formatting, the error is on a single line) check /var/log/messages for signs of the OOM killer springing into action.

PYCURL ERROR 7 - "couldn't connect to host"

If all builds suddenly start failing for no apparent reason, and the error message from mock reads something like: 

DEBUG util.py:257:  http://koji-hub.testbed/kojifiles/repos/dist-epel5-build/1472/i386/repodata/repomd.xml:
[Errno 14] PYCURL ERROR 7 - "couldn't connect to host"

(line break inserted for formatting) this is likely due to the fact that the squid service is not running anymore.


Sigul signing seems to hang

We've seen the weird situation that the connection between the bridge and the server was semi-severed: the bridge still showed the connection in the output of netstat, but the server lost the connection. The resolution is to restart the server. If the bridge needs to be restarted, the server also needs a restart thereafter.

UPDATE: the cause of this problem lies with the connection being too quiet. Without keepalive the connection will be dropped. It's the stateful firewall's fault.

Retrieved from "https://wiki.nikhef.nl/grid/index.php?title=Koji_Testbed&oldid=10936"