Difference between revisions of "Koji Testbed"

From PDP/Grid Wiki
Jump to navigationJump to search
(35 intermediate revisions by 2 users not shown)
Line 29: Line 29:
  
 
The installation mostly followed the very complete guidance on [https://fedoraproject.org/wiki/Koji/ServerHowTo], choosing the SSL authentication option.
 
The installation mostly followed the very complete guidance on [https://fedoraproject.org/wiki/Koji/ServerHowTo], choosing the SSL authentication option.
 +
 +
'''NOTE''' In december we upgraded to 1.8, and the database schema needed to change. See the migration documentation in /usr/share/doc/koji on koji-hub.testbed.
 +
--[[User:Dennisvd@nikhef.nl|Dennisvd@nikhef.nl]] ([[User talk:Dennisvd@nikhef.nl|talk]]) 10:01, 15 January 2014 (CET)
  
 
=== Using a squid proxy for external repositories ===
 
=== Using a squid proxy for external repositories ===
Line 71: Line 74:
 
* packages, in the first sense as described above
 
* packages, in the first sense as described above
 
* packages, in the second sense as described above
 
* packages, in the second sense as described above
 +
* architectures, such as i386 and x86_64 (in fact, only those in our case)
 
* groups (we'll get to those later)
 
* groups (we'll get to those later)
 
* external repositories
 
* external repositories
 +
* build targets (source or destination)
  
  
 
The second most confusing word in Koji context is the word 'parent' and, along with that, the entire concept of inheritance. In the biological sense we can't choose, change, or add parents, but tags have no such limitations. While normally we inherit from our parents after they pass away, tags inherit everything from all of their parents (and parents' parents) immediately. Remember: in the graphical displays, Koji shows parents down and to the right in the inheritance tree.
 
The second most confusing word in Koji context is the word 'parent' and, along with that, the entire concept of inheritance. In the biological sense we can't choose, change, or add parents, but tags have no such limitations. While normally we inherit from our parents after they pass away, tags inherit everything from all of their parents (and parents' parents) immediately. Remember: in the graphical displays, Koji shows parents down and to the right in the inheritance tree.
  
The builds are done against targets, which have source and target tags, which have buildroots populated from repositories. The initial buildroots are set up from virtual packages called 'build' and 'srpm-build' created in koji.
+
The builds are done against build targets, which have source and target tags. The way this works in a nutshell is as follows:
 +
* the user issues the build of a package (1st sense) for a specific target
 +
* the target's ''build tag'' is chosen to create a build root
 +
* dependencies are resolved, hopefully
 +
* the build runs, and if all is well this results in a new artefact (a package in the second sense)
 +
* this package is tagged with the target's ''destination tag''
  
These groups can be inherited from earlier releases by using the 'add-tag-inheritance' command.
+
Resolving dependencies and setting up a build root works by combining external repositories and packages with the build tag, and installing a meta-package with 'yum groupinstall'. This is where the concept of ''groups'' comes in. There are two groups for the different phases of a build: <code>srpm-build</code> for the creation of source RPMs and <code>build</code> for the actual build.
koji add-tag-inheritance --maxdepth 1 --priority 2 dist-fc18-build dist-fc17-build
+
The groups as created in koji contain only package names.
  
(I'm not too sure what maxdepth and priority mean here; but Fedora's own koji lists the build group for FC18 as going all the way back to FC9.)
+
=== Current setup ===
 +
 
 +
Fedora releases are built one on top another, and using inheritance makes perfect sense. But there is a catch. Builds, a.k.a. packages in the second sense, are inherited just the same. As are external repositories. We've experienced that our build tag for Fedora 18 collected everything from Fedora 17 and 16 (when we started), including all the repositories. This went crazy.
 +
 
 +
The solution is to be careful about what we inherit. The current setup is displayed best by the following pictures:
 +
 
 +
$ koji list-tag-inheritance --reverse f16
 +
f16 (7)  <- contains ONLY packages in the first sense
 +
  +-f17 (9)  <- so does this, by inheritance
 +
  |  +-f17-candidates (33)  <- this is the "build target" "destination tag"
 +
  |  +-f17-release (25)  <- this is used manually to mark a package (second sense) ready for shipping with mash
 +
  |  +-f17-testing (26)  <- idem, but the for the 'testing' repo
 +
  |  +-f18 (27)  <- only packages, first sense
 +
  |  |  +-f18-build (30)  <- the "build target" "build tag"; this has architectures and external repos
 +
  |  |  |  +-f18-candidates (31)  -< the "build target" "destination tag"
 +
  |  |  +-f19 (34)
 +
  |  |  |  +-f19-candidates (38)
 +
  |  |  |  +-f19-release (43)
 +
  |  |  |  +-f19-testing (44)
 +
  |  |  |  +-f19-build (36)
 +
  |  |  +-f18-release (39)
 +
  |  |  +-f18-testing (40)
 +
  |  +-f17-build (32)
 +
  +-f16-release (24)
 +
  +-f16-testing (23)
 +
  +-f16-build (45)
 +
 
 +
And this one:
 +
 
 +
$ koji list-tag-inheritance f19-build
 +
f19-build (36)  <- the "build target" "build tag"
 +
  +-f19 (34)  <- package names only (first sense)
 +
  |  +-f18 (27)  <- all the way
 +
  |    +-f17 (9)  <-  down to
 +
  |        +-f16 (7)  <- here
 +
  +-f19-basedeps (35)  <- these contain the 'groups'
 +
      +-f18-basedeps (28)  <- actually inherited from
 +
        +-f17-basedeps (10)  <- here
 +
 
 +
A build for F19 ends up in the tag f19-candidates. From there, it needs to be manually moved. If it is a build dependency for other packages, it has to go to f19-build. Otherwise, it eventually goes to f19-testing and from there to f19-release, or directly to f19-release. A rotten product should have its candidacy stripped and ends up with no tags. In any case, no package should remain a candidate for too long.
 +
 
 +
Because the *-builds don't inherit from each other, we have no pollution of external repositories or old builds for new Fedora releases.
 +
 
 +
The situation for epel5 and epel6 is analogue.
  
 
=== Steps to follow if you need to do this from scratch ===
 
=== Steps to follow if you need to do this from scratch ===
  
A dump of the history to see what steps are required
+
'''The following doesn't exactly reflect the current setup, but we'd do it this way if we had to start over.'''
  
  99 koji add-tag dist-epel5
+
Create a tag to collect all our packages in.
  100 koji add-tag --parent dist-epel5 --arches "i386 x86_64" dist-epel5-build
+
  koji add-tag epel5
  102  koji add-external-repo -t dist-epel5-build \
+
  koji add-pkg --owner <''your name''> epel5 lcmaps jobrepository ees glexec \
      dist-epel5-external-repo http://mirrors.nl.eu.kernel.org/fedora-epel/5/\$arch/
+
          lcas-lcmaps-gt4-interface lcas-plugins-basic lcas-plugins-check-executable \
  103  koji add-external-repo -t dist-epel5-build dist-epel5-buildsys-macros-repo \
+
          lcas-plugins-voms lcas lcmaps-plugins-afs lcmaps-plugins-basic lcmaps-plugins-c-pep lcmaps-plugins-gums \
      http://buildsys.fedoraproject.org/buildgroups/rhel5/\$arch/
+
          lcmaps-plugins-jobrep lcmaps-plugins-scas-client lcmaps-plugins-tracking-groupid lcmaps-plugins-verify-proxy \
  104  koji add-external-repo -t dist-epel5-build dist-epel5-centos5-repo \
+
          lcmaps-plugins-voms scas xacml argus-pep-api-c glexec-wn nagios-plugins-glexec nagios-plugins-ees mkgltempdir \
      http://spiegel.nikhef.nl/mirror/centos/5/os/\$arch/
+
          glexec-wrapper-scripts torque llrun saml2-xacml2-c-lib lcmaps-without-gsi mwsec-release
  105  koji add-target dist-epel5 dist-epel5-build
+
 
'''NOTE''': the buildsys external repo is ONLY needed on EPEL5 to provide the correct buildsys-macros package. The standard CentOS package causes the dist macro to expand to el5.centos. By providing this external Fedora repo and putting it before the centos5 repo, the correct buildsys-macros is installed. See also [https://fedoraproject.org/wiki/Koji/KojiMisc#.25dist_tags https://fedoraproject.org/wiki/Koji/KojiMisc#.25dist_tags].
+
Another tag to use as a build target destination. All builds end up here, before they are moved to 'testing' or 'release' repositories.
If needed (i.e. when the ordering turns out to be wrong) we can provide explicit priorities using a -p flag to add-external-repo. Check with
+
koji add-tag --parent epel5 epel5-candidates
koji taginfo dist-epel5-build
 
  
These groups are going to be passed to 'yum groupinstall' when the buildroot is created.
+
These tags are used by the 'mash' tool to publish repositories.
 +
koji add-tag --parent epel5 epel5-testing
 +
koji add-tag --parent epel5 epel5-release
  
  106  koji add-group dist-epel5-build build
+
To resolve build dependencies, this tag is used as the build target ''source''.
  107 koji add-group dist-epel5-build srpm-build
+
  koji add-tag --parent epel5 --arches "i386 x86_64" epel5-build
  
 +
A Koji build will first do a <nowiki>buildSRPMFromSCM</nowiki> job, which will run mock to generate a buildroot, inside of which 'yum groupinstall srpm-build' is run. Later, a <nowiki>buildArch</nowiki> job does the actual building, and here mock runs 'yum groupinstall build'.
 +
These groups must be created by us first, and populated with package names that we need to have available at this stage.
 +
koji add-group epel5-build build
 +
koji add-group epel5-build srpm-build
 +
 
They need to be populated. Clone from a typical value for the Fedora Koji systems, e.g.
 
They need to be populated. Clone from a typical value for the Fedora Koji systems, e.g.
  
  109 koji -s http://koji.fedoraproject.org/kojihub/ list-groups dist-5E-epel-build
+
  koji add-group-pkg epel5-build build `koji -s http://koji.fedoraproject.org/kojihub/
 +
      list-groups dist-5E-epel-build build | tail -n +2 | cut -d: -f1`
 +
Be careful to exclude the fedpkg package in srpm-build which pulls in way too much! It includes mock which causes strange failures with conflicting (unix) group ids within the buildroot.
 +
koji add-group-pkg epel5-build srpm-build `koji -s http://koji.fedoraproject.org/kojihub/
 +
      list-groups dist-5E-epel-build srpm-build | tail -n +2 | cut -d: -f1 | grep -v fedpkg`
 +
 
 +
As of EPEL7, there's fedpkg-minimal which should be safe (it only requires wget).
  
Use the output to produce a proper list of packages. Be careful to exclude the fedpkg package which pulls in way too much! It includes mock which causes strange failures with conflicting group ids within the buildroot.
+
Add external repositories to populate the buildroot.
 +
koji add-external-repo -t epel5-build \
 +
      dist-epel5-external-repo http://mirrors.nl.eu.kernel.org/fedora-epel/5/\$arch/
 +
koji add-external-repo -t epel5-build dist-epel5-buildsys-macros-repo \
 +
      http://buildsys.fedoraproject.org/buildgroups/rhel5/\$arch/
 +
koji add-external-repo -t epel5-build dist-epel5-centos5-repo \
 +
      http://spiegel.nikhef.nl/mirror/centos/5/os/\$arch/
  
  112  koji add-group-pkg dist-epel5-build build `cat epel5-build-pkgs `
+
Define the build target based on these tags.
  114 koji add-group-pkg dist-epel5-build srpm-build `cat epel5-srpm-build-pkgs `
+
  koji add-target epel5 epel5-build epel5-candidates
  
Don't forget to add the package!
+
'''NOTE''': the buildsys external repo is ONLY needed on EPEL5 to provide the correct buildsys-macros package. The standard CentOS package causes the dist macro to expand to el5.centos. By providing this external Fedora repo and putting it before the centos5 repo, the correct buildsys-macros is installed. See also [https://fedoraproject.org/wiki/Koji/KojiMisc#.25dist_tags https://fedoraproject.org/wiki/Koji/KojiMisc#.25dist_tags].
 +
If needed (i.e. when the ordering turns out to be wrong) we can provide explicit priorities using a -p flag to add-external-repo. Check with
 +
koji taginfo epel5-build
  
  140  koji add-pkg --owner okoeroo dist-epel5 lcmaps
 
  124  koji build dist-epel5 git://bleek.testbed/koji.git\?lcmaps/#a942e490ace1da950cedae2976bf59fa3b3f5038
 
  
The git repository must be added to the allowed repositories on the builder in kojid.conf.
+
Since the srpm-build and build do not vary much between distributions,
 +
we create a separate tag and use inheritance to save work for the Fedora branches.
 +
This does not hold true for EPEL branches, which are not inheriting from one another.
 +
 
 +
for i in 27 28 ; do
 +
    prev=`expr $i - 1`
 +
    koji add-tag --parent f$prev f$i
 +
    koji add-tag --parent f$prev-basedeps f${i}-basedeps
 +
    # koji add-tag --parent f$i --parent f$i-basedeps --arches "i386 x86_64" f${i}-build
 +
    koji add-tag --parent f$i --parent f${i}-basedeps --arches "x86_64" f${i}-build
 +
    for j in candidates testing release; do
 +
        koji add-tag --parent f$i f${i}-$j
 +
    done
 +
    koji add-external-repo -t f${i}-build dist-f${i}-base http://mirror.nl.leaseweb.net/fedora/linux/releases/${i}/Everything/\$arch/os/
 +
    # '''NOTE''': up to 27, we should leave out the ''Everything'' for the updates
 +
    koji add-external-repo -t f${i}-build dist-f${i}-updates http://mirror.nl.leaseweb.net/fedora/linux/updates/${i}/Everything/\$arch/
 +
    koji add-target f$i f${i}-build f${i}-candidates
 +
    # '''NOTE''': probably need to add priorities to the parents, e.g. using something like
 +
    koji edit-tag-inheritance --priority 2 f${i}-build f${i}-basedeps
 +
    koji add-tag-inheritance --priority 1 f${i}-build f${i}
 +
done
  
 
= Automated builds from tags in mwsec =
 
= Automated builds from tags in mwsec =
 +
 +
Koji can build from various revision control systems, such as git and SVN. The way this works is to give <code>koji build</code> a parameter in the form
 +
''(repository URL)''?''(component path)''#''(revision ID)''
 +
 +
The repository must be added to the allowed repositories on each builder (in kojid.conf):
 +
/etc/kojid/kojid.conf:
 +
...
 +
allowed_scms=bleek.testbed:/koji.git:no ndpfsvn.nikhef.nl:/ro/mwsec:no
 +
...
  
 
== Building from git (deprecated) ==
 
== Building from git (deprecated) ==
Line 141: Line 241:
  
 
Within that directory, create a Makefile and a spec file. The spec file can be the same as the one from packaging/fedora/trunk for that component.
 
Within that directory, create a Makefile and a spec file. The spec file can be the same as the one from packaging/fedora/trunk for that component.
The Makefile can be fairly generic, in most cases this should suffice:
+
The Makefile is generic, in most cases this should suffice:
  dir = $(shell pwd)
+
  cwd = $(abspath .)
  version = $(notdir $(dir))
+
  version = $(notdir $(cwd))
  component = lcmaps-plugins-verify-proxy
+
  component = $(notdir $(patsubst %/, %, $(dir $(cwd))))
 
   
 
   
 
  sources:
 
  sources:
Line 151: Line 251:
 
   
 
   
 
Should a rebuild be necessary because of errors in Koji or in the spec file, update the Release field in the spec file as usual.
 
Should a rebuild be necessary because of errors in Koji or in the spec file, update the Release field in the spec file as usual.
 
+
'''There is no way to remove or redo a successful build,''' if it is bad, just remove all tags.
 
 
  
 
== Scripts on the svn repository ==
 
== Scripts on the svn repository ==
Line 260: Line 359:
 
  iptables -t nat -A PREROUTING --proto tcp --dport 8443 --destination 194.171.96.17 -j DNAT --to-destination 10.198.8.7:443
 
  iptables -t nat -A PREROUTING --proto tcp --dport 8443 --destination 194.171.96.17 -j DNAT --to-destination 10.198.8.7:443
  
= Signing RPMs with sigul =
+
= Setting up sigul to sign RPMs =
  
 
Before the RPMs that Koji produces are distributed, they should be signed. The YUM configuration can then be configured to use GPG to verify the origin of the RPMs upon installing.
 
Before the RPMs that Koji produces are distributed, they should be signed. The YUM configuration can then be configured to use GPG to verify the origin of the RPMs upon installing.
  
 +
Sigul integrates well with Koji, but unfortunately comes with little documentation. There is some [https://fedoraproject.org/wiki/Create_release_signing_key here].
 +
 +
The architecture of sigul is as follows:
 +
 +
[[File:Sigul-arch.png]]
 +
 +
The sigul server node holds the secret key(s) and must be secured very tightly. The sigul bridge node only allows connections from the client and the server, it will make no outgoing connections.
 +
 +
Sigul is available as a package in CentOS 6, and hold all three components. The client is installed on koji-hub.testbed; the bridge on koji-bridge.testbed and the server on sigul.testbed. The latter two cannot be accessed through ssh, only console access (wich 'virsh console' or on the virtual video console).
 +
 +
'''Note:''' the firewall on sigul and the sigul bridge should have a line to accept packets from eachother, as the stateful tracking doesn't work so well. The connection stays open for too long and the firewall loses the state.
 +
iptables -R INPUT 1 -p tcp -s 10.198.8.18 \! --syn -j ACCEPT
 +
TODO: write up how to set up a new key and put it to work in sigul.
  
 
= Generating distribution repos from koji =  
 
= Generating distribution repos from koji =  
Line 351: Line 463:
 
= Nightly builds from mwsec/trunk =
 
= Nightly builds from mwsec/trunk =
  
The dedicated account kojinb for nightly builds is added on bleek.
+
We no longer run nightly builds. That should move to a continuous integration system.
 
 
At the moment nightly builds are run from a crontab of account dennisvd.
 
01 2 * * * ${HOME}/bin/koji-nightly
 
 
 
The koji-nightly script:
 
#!/bin/sh
 
 
PATH=/bin:/usr/bin
 
 
components="
 
lcas
 
lcmaps
 
ees
 
glexec
 
jobrepository
 
lcas-lcmaps-gt4-interface
 
lcas-plugins-basic
 
lcas-plugins-check-executable
 
lcas-plugins-voms
 
lcmaps-plugins-afs
 
lcmaps-plugins-basic
 
lcmaps-plugins-c-pep
 
lcmaps-plugins-gums
 
lcmaps-plugins-jobrep
 
lcmaps-plugins-scas-client
 
lcmaps-plugins-tracking-groupid
 
lcmaps-plugins-verify-proxy
 
lcmaps-plugins-voms
 
scas
 
"
 
 
dists="epel5 epel6 fc16 fc17"
 
 
rev=`svn info http://ndpfsvn.nikhef.nl/ro/mwsec/ | sed -n 's/^Revision: //p'`
 
 
for i in $components ; do
 
    for j in $dists ; do
 
koji -c ${HOME}/.koji/nightly.conf build --nowait nightly-$j \
 
            svn+http://ndpfsvn.nikhef.nl/ro/mwsec\?packaging/fedora/branches/nightly-builds/$i/#$rev
 
    done
 
done
 
 
 
Each component directory in the nightly-builds branch contains the same set of files: a Makefile:
 
component = $(notdir $(shell pwd))
 
 
sources:
 
./fetch-sources $(component)
 
 
 
And a fetch-sources script:
 
#!/bin/sh
 
 
set -e
 
 
component=$1
 
 
svn co http://ndpfsvn.nikhef.nl/ro/mwsec/trunk/${component}
 
cd ${component}
 
revision=`svn info | sed -n 's/^Last Changed Rev: //p'`
 
 
if [ -z $revision ]; then
 
    echo "Could not retrieve revision from svn info" >&2
 
    echo "svn info:" >&2
 
    svn info >&2
 
    exit 1
 
fi
 
 
./bootstrap
 
mkdir build
 
cd build
 
../configure
 
make dist
 
mv ${component}-*.tar.gz ..
 
cd ..
 
 
version=`echo ${component}-*.tar.gz | sed -n "s/${component}-\\([0-9.]*\\)\\.tar\\.gz/\\1/p"`
 
 
if [ -z $version ]; then
 
    echo "Could not retrieve version from generated tarball" >&2
 
    echo "tarballs: " ${component}-*.tar.gz >&2
 
    exit 1
 
fi
 
 
sed -i -e "s/^Version: .*/Version: $version/" -e "s/^Release: .*/Release: 0.$revision%{?dist}/" ${component}.spec
 
 
 
The list of components misses just two packages: argus-pep-api-c, which is not under our active development, and xacml, which has some problems at this moment; gsoap-devel needs to be added to the build-srpm group (done) and as the nightly build does the bootstrapping as well, the autoconf for the epel5 tag is too old according to configure.in. Whether this is really the case remains to be seen.
 
 
 
== Preventing repeat builds ==
 
 
 
Koji will refuse build artefacts that it already has. The nightly builds are versioned according to the svn revision of the latest update of the tree to build. That means that if no updates are done to the tree, there is no sense in repeating the build. The cron script should be enhanced to deal with this situation.
 
  
 
= Troubleshooting =
 
= Troubleshooting =
Line 475: Line 498:
 
(line break inserted for formatting)
 
(line break inserted for formatting)
 
this is likely due to the fact that the [[#Using a squid proxy for external repositories|squid service]] is not running anymore.
 
this is likely due to the fact that the [[#Using a squid proxy for external repositories|squid service]] is not running anymore.
 +
 +
 +
== Sigul signing seems to hang ==
 +
 +
We've seen the weird situation that the connection between the bridge and the server was semi-severed: the bridge still showed the connection in the output of netstat, but the server lost the connection. The resolution is to restart the server. If the bridge needs to be restarted, the server also needs a restart thereafter.
 +
 +
UPDATE: the cause of this problem lies with the connection being too quiet. Without keepalive the connection will be dropped. It's the stateful firewall's fault.

Revision as of 18:34, 29 July 2018