Difference between revisions of "Koji Testbed"
From PDP/Grid Wiki
Jump to navigationJump to search(24 intermediate revisions by 2 users not shown) | |||
Line 29: | Line 29: | ||
The installation mostly followed the very complete guidance on [https://fedoraproject.org/wiki/Koji/ServerHowTo], choosing the SSL authentication option. | The installation mostly followed the very complete guidance on [https://fedoraproject.org/wiki/Koji/ServerHowTo], choosing the SSL authentication option. | ||
+ | |||
+ | '''NOTE''' In december we upgraded to 1.8, and the database schema needed to change. See the migration documentation in /usr/share/doc/koji on koji-hub.testbed. | ||
+ | --[[User:Dennisvd@nikhef.nl|Dennisvd@nikhef.nl]] ([[User talk:Dennisvd@nikhef.nl|talk]]) 10:01, 15 January 2014 (CET) | ||
=== Using a squid proxy for external repositories === | === Using a squid proxy for external repositories === | ||
Line 97: | Line 100: | ||
$ koji list-tag-inheritance --reverse f16 | $ koji list-tag-inheritance --reverse f16 | ||
f16 (7) <- contains ONLY packages in the first sense | f16 (7) <- contains ONLY packages in the first sense | ||
− | + | +-f17 (9) <- so does this, by inheritance | |
− | + | | +-f17-candidates (33) <- this is the "build target" "destination tag" | |
− | + | | +-f17-release (25) <- this is used manually to mark a package (second sense) ready for shipping with mash | |
− | + | | +-f17-testing (26) <- idem, but the for the 'testing' repo | |
− | + | | +-f18 (27) <- only packages, first sense | |
− | + | | | +-f18-build (30) <- the "build target" "build tag"; this has architectures and external repos | |
− | + | | | | +-f18-candidates (31) -< the "build target" "destination tag" | |
− | + | | | +-f19 (34) | |
− | + | | | | +-f19-candidates (38) | |
− | + | | | | +-f19-release (43) | |
− | + | | | | +-f19-testing (44) | |
− | + | | | | +-f19-build (36) | |
− | + | | | +-f18-release (39) | |
− | + | | | +-f18-testing (40) | |
− | + | | +-f17-build (32) | |
− | + | +-f16-release (24) | |
− | + | +-f16-testing (23) | |
− | + | +-f16-build (45) | |
And this one: | And this one: | ||
Line 120: | Line 123: | ||
$ koji list-tag-inheritance f19-build | $ koji list-tag-inheritance f19-build | ||
f19-build (36) <- the "build target" "build tag" | f19-build (36) <- the "build target" "build tag" | ||
− | + | +-f19 (34) <- package names only (first sense) | |
− | + | | +-f18 (27) <- all the way | |
− | + | | +-f17 (9) <- down to | |
− | + | | +-f16 (7) <- here | |
− | + | +-f19-basedeps (35) <- these contain the 'groups' | |
− | + | +-f18-basedeps (28) <- actually inherited from | |
− | + | +-f17-basedeps (10) <- here | |
A build for F19 ends up in the tag f19-candidates. From there, it needs to be manually moved. If it is a build dependency for other packages, it has to go to f19-build. Otherwise, it eventually goes to f19-testing and from there to f19-release, or directly to f19-release. A rotten product should have its candidacy stripped and ends up with no tags. In any case, no package should remain a candidate for too long. | A build for F19 ends up in the tag f19-candidates. From there, it needs to be manually moved. If it is a build dependency for other packages, it has to go to f19-build. Otherwise, it eventually goes to f19-testing and from there to f19-release, or directly to f19-release. A rotten product should have its candidacy stripped and ends up with no tags. In any case, no package should remain a candidate for too long. | ||
Line 158: | Line 161: | ||
A Koji build will first do a <nowiki>buildSRPMFromSCM</nowiki> job, which will run mock to generate a buildroot, inside of which 'yum groupinstall srpm-build' is run. Later, a <nowiki>buildArch</nowiki> job does the actual building, and here mock runs 'yum groupinstall build'. | A Koji build will first do a <nowiki>buildSRPMFromSCM</nowiki> job, which will run mock to generate a buildroot, inside of which 'yum groupinstall srpm-build' is run. Later, a <nowiki>buildArch</nowiki> job does the actual building, and here mock runs 'yum groupinstall build'. | ||
− | These groups must be created by us first, and populated with package names that we need to have available at this stage | + | These groups must be created by us first, and populated with package names that we need to have available at this stage. |
− | + | koji add-group epel5-build build | |
− | koji add-group epel5- | + | koji add-group epel5-build srpm-build |
− | koji add-group epel5- | + | |
− | |||
They need to be populated. Clone from a typical value for the Fedora Koji systems, e.g. | They need to be populated. Clone from a typical value for the Fedora Koji systems, e.g. | ||
− | koji add-group-pkg epel5- | + | koji add-group-pkg epel5-build build `koji -s http://koji.fedoraproject.org/kojihub/ |
list-groups dist-5E-epel-build build | tail -n +2 | cut -d: -f1` | list-groups dist-5E-epel-build build | tail -n +2 | cut -d: -f1` | ||
Be careful to exclude the fedpkg package in srpm-build which pulls in way too much! It includes mock which causes strange failures with conflicting (unix) group ids within the buildroot. | Be careful to exclude the fedpkg package in srpm-build which pulls in way too much! It includes mock which causes strange failures with conflicting (unix) group ids within the buildroot. | ||
− | koji add-group-pkg epel5- | + | koji add-group-pkg epel5-build srpm-build `koji -s http://koji.fedoraproject.org/kojihub/ |
list-groups dist-5E-epel-build srpm-build | tail -n +2 | cut -d: -f1 | grep -v fedpkg` | list-groups dist-5E-epel-build srpm-build | tail -n +2 | cut -d: -f1 | grep -v fedpkg` | ||
+ | |||
+ | As of EPEL7, there's fedpkg-minimal which should be safe (it only requires wget). | ||
Add external repositories to populate the buildroot. | Add external repositories to populate the buildroot. | ||
Line 187: | Line 191: | ||
− | + | Since the srpm-build and build do not vary much between distributions, | |
− | + | we create a separate tag and use inheritance to save work for the Fedora branches. | |
− | + | This does not hold true for EPEL branches, which are not inheriting from one another. | |
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | for i in | + | for i in 27 28 ; do |
prev=`expr $i - 1` | prev=`expr $i - 1` | ||
koji add-tag --parent f$prev f$i | koji add-tag --parent f$prev f$i | ||
− | koji add-tag --parent $i --arches "i386 x86_64" $i-build | + | koji add-tag --parent f$prev-basedeps f${i}-basedeps |
+ | # koji add-tag --parent f$i --parent f$i-basedeps --arches "i386 x86_64" f${i}-build | ||
+ | koji add-tag --parent f$i --parent f${i}-basedeps --arches "x86_64" f${i}-build | ||
for j in candidates testing release; do | for j in candidates testing release; do | ||
− | koji add-tag --parent $i $i-$j | + | koji add-tag --parent f$i f${i}-$j |
done | done | ||
− | koji add- | + | koji add-external-repo -t f${i}-build dist-f${i}-base http://mirror.nl.leaseweb.net/fedora/linux/releases/${i}/Everything/\$arch/os/ |
− | koji add-external-repo -t $i-build dist-$i- | + | # '''NOTE''': up to 27, we should leave out the ''Everything'' for the updates |
− | koji add-target $i $i-build $i-candidates | + | koji add-external-repo -t f${i}-build dist-f${i}-updates http://mirror.nl.leaseweb.net/fedora/linux/updates/${i}/Everything/\$arch/ |
+ | koji add-target f$i f${i}-build f${i}-candidates | ||
+ | # '''NOTE''': probably need to add priorities to the parents, e.g. using something like | ||
+ | koji edit-tag-inheritance --priority 2 f${i}-build f${i}-basedeps | ||
+ | koji add-tag-inheritance --priority 1 f${i}-build f${i} | ||
done | done | ||
Line 255: | Line 241: | ||
Within that directory, create a Makefile and a spec file. The spec file can be the same as the one from packaging/fedora/trunk for that component. | Within that directory, create a Makefile and a spec file. The spec file can be the same as the one from packaging/fedora/trunk for that component. | ||
− | The Makefile | + | The Makefile is generic, in most cases this should suffice: |
− | + | cwd = $(abspath .) | |
− | version = $(notdir $( | + | version = $(notdir $(cwd)) |
− | component = | + | component = $(notdir $(patsubst %/, %, $(dir $(cwd)))) |
sources: | sources: | ||
Line 373: | Line 359: | ||
iptables -t nat -A PREROUTING --proto tcp --dport 8443 --destination 194.171.96.17 -j DNAT --to-destination 10.198.8.7:443 | iptables -t nat -A PREROUTING --proto tcp --dport 8443 --destination 194.171.96.17 -j DNAT --to-destination 10.198.8.7:443 | ||
− | = | + | = Setting up sigul to sign RPMs = |
Before the RPMs that Koji produces are distributed, they should be signed. The YUM configuration can then be configured to use GPG to verify the origin of the RPMs upon installing. | Before the RPMs that Koji produces are distributed, they should be signed. The YUM configuration can then be configured to use GPG to verify the origin of the RPMs upon installing. | ||
+ | Sigul integrates well with Koji, but unfortunately comes with little documentation. There is some [https://fedoraproject.org/wiki/Create_release_signing_key here]. | ||
+ | |||
+ | The architecture of sigul is as follows: | ||
+ | |||
+ | [[File:Sigul-arch.png]] | ||
+ | |||
+ | The sigul server node holds the secret key(s) and must be secured very tightly. The sigul bridge node only allows connections from the client and the server, it will make no outgoing connections. | ||
+ | |||
+ | Sigul is available as a package in CentOS 6, and hold all three components. The client is installed on koji-hub.testbed; the bridge on koji-bridge.testbed and the server on sigul.testbed. The latter two cannot be accessed through ssh, only console access (wich 'virsh console' or on the virtual video console). | ||
+ | |||
+ | '''Note:''' the firewall on sigul and the sigul bridge should have a line to accept packets from eachother, as the stateful tracking doesn't work so well. The connection stays open for too long and the firewall loses the state. | ||
+ | iptables -R INPUT 1 -p tcp -s 10.198.8.18 \! --syn -j ACCEPT | ||
+ | TODO: write up how to set up a new key and put it to work in sigul. | ||
= Generating distribution repos from koji = | = Generating distribution repos from koji = | ||
Line 464: | Line 463: | ||
= Nightly builds from mwsec/trunk = | = Nightly builds from mwsec/trunk = | ||
− | + | We no longer run nightly builds. That should move to a continuous integration system. | |
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
= Troubleshooting = | = Troubleshooting = | ||
Line 588: | Line 498: | ||
(line break inserted for formatting) | (line break inserted for formatting) | ||
this is likely due to the fact that the [[#Using a squid proxy for external repositories|squid service]] is not running anymore. | this is likely due to the fact that the [[#Using a squid proxy for external repositories|squid service]] is not running anymore. | ||
+ | |||
+ | |||
+ | == Sigul signing seems to hang == | ||
+ | |||
+ | We've seen the weird situation that the connection between the bridge and the server was semi-severed: the bridge still showed the connection in the output of netstat, but the server lost the connection. The resolution is to restart the server. If the bridge needs to be restarted, the server also needs a restart thereafter. | ||
+ | |||
+ | UPDATE: the cause of this problem lies with the connection being too quiet. Without keepalive the connection will be dropped. It's the stateful firewall's fault. |