Difference between revisions of "Koji Testbed"

From PDP/Grid Wiki
Jump to navigationJump to search
Line 136: Line 136:
 
=== Steps to follow if you need to do this from scratch ===
 
=== Steps to follow if you need to do this from scratch ===
  
−
'''Unfortunately this is no longer current, as the setup changed several times.'''
+
'''The following doesn't exactly reflect the current setup, but we'd do it this way if we had to start over.'''
  
−
A dump of the history to see what steps are required
+
Create a tag to collect all our packages in.
 +
koji add-tag epel5
 +
koji add-pkg --owner <''your name''> epel5 lcmaps jobrepository ees glexec \
 +
          lcas-lcmaps-gt4-interface lcas-plugins-basic lcas-plugins-check-executable \
 +
          lcas-plugins-voms lcas lcmaps-plugins-afs lcmaps-plugins-basic lcmaps-plugins-c-pep lcmaps-plugins-gums \
 +
          lcmaps-plugins-jobrep lcmaps-plugins-scas-client lcmaps-plugins-tracking-groupid lcmaps-plugins-verify-proxy \
 +
          lcmaps-plugins-voms scas xacml argus-pep-api-c glexec-wn nagios-plugins-glexec nagios-plugins-ees mkgltempdir \
 +
          glexec-wrapper-scripts torque llrun saml2-xacml2-c-lib lcmaps-without-gsi mwsec-release
  
−
  koji add-tag epel5
+
Another tag to use as a build target destination. All builds end up here, before they are moved to 'testing' or 'release' repositories.
 +
koji add-tag --parent epel5 epel5-candidates
 +
 
 +
These tags are used by the 'mash' tool to publish repositories.
 +
koji add-tag --parent epel5 epel5-testing
 +
  koji add-tag --parent epel5 epel5-release
 +
 
 +
To resolve build dependencies, this tag is used as the build target ''source''.
 
  koji add-tag --parent epel5 --arches "i386 x86_64" epel5-build
 
  koji add-tag --parent epel5 --arches "i386 x86_64" epel5-build
 +
 +
A Koji build will first do a <nowiki>buildSRPMFromSCM</nowiki> job, which will run mock to generate a buildroot, inside of which 'yum groupinstall srpm-build' is run. Later, a <nowiki>buildArch</nowiki> job does the actual building, and here mock runs 'yum groupinstall build'.
 +
These groups must be created by us first, and populated with package names that we need to have available at this stage. Since these groups do not vary much between distributions, we create a separate tag and use inheritance to save work.
 +
koji add-tag epel5-basedeps
 +
koji add-group epel5-basedeps build
 +
koji add-group epel5-basedeps srpm-build
 +
koji add-tag-inheritance epel5-build epel5-basedeps
 +
They need to be populated. Clone from a typical value for the Fedora Koji systems, e.g.
 +
 +
koji add-group-pkg epel5-basedeps build `koji -s http://koji.fedoraproject.org/kojihub/
 +
      list-groups dist-5E-epel-build build | tail -n +2 | cut -d: -f1`
 +
Be careful to exclude the fedpkg package in srpm-build which pulls in way too much! It includes mock which causes strange failures with conflicting (unix) group ids within the buildroot.
 +
koji add-group-pkg epel5-basedeps srpm-build `koji -s http://koji.fedoraproject.org/kojihub/
 +
      list-groups dist-5E-epel-build srpm-build | tail -n +2 | cut -d: -f1 | grep -v fedpkg`
 +
 +
Add external repositories to populate the buildroot.
 
  koji add-external-repo -t epel5-build \
 
  koji add-external-repo -t epel5-build \
 
       dist-epel5-external-repo http://mirrors.nl.eu.kernel.org/fedora-epel/5/\$arch/
 
       dist-epel5-external-repo http://mirrors.nl.eu.kernel.org/fedora-epel/5/\$arch/
Line 148: Line 178:
 
  koji add-external-repo -t epel5-build dist-epel5-centos5-repo \
 
  koji add-external-repo -t epel5-build dist-epel5-centos5-repo \
 
       http://spiegel.nikhef.nl/mirror/centos/5/os/\$arch/
 
       http://spiegel.nikhef.nl/mirror/centos/5/os/\$arch/
 +
 +
Define the build target based on these tags.
 
  koji add-target epel5 epel5-build epel5-candidates
 
  koji add-target epel5 epel5-build epel5-candidates
 +
 
'''NOTE''': the buildsys external repo is ONLY needed on EPEL5 to provide the correct buildsys-macros package. The standard CentOS package causes the dist macro to expand to el5.centos. By providing this external Fedora repo and putting it before the centos5 repo, the correct buildsys-macros is installed. See also [https://fedoraproject.org/wiki/Koji/KojiMisc#.25dist_tags https://fedoraproject.org/wiki/Koji/KojiMisc#.25dist_tags].
 
'''NOTE''': the buildsys external repo is ONLY needed on EPEL5 to provide the correct buildsys-macros package. The standard CentOS package causes the dist macro to expand to el5.centos. By providing this external Fedora repo and putting it before the centos5 repo, the correct buildsys-macros is installed. See also [https://fedoraproject.org/wiki/Koji/KojiMisc#.25dist_tags https://fedoraproject.org/wiki/Koji/KojiMisc#.25dist_tags].
 
If needed (i.e. when the ordering turns out to be wrong) we can provide explicit priorities using a -p flag to add-external-repo. Check with
 
If needed (i.e. when the ordering turns out to be wrong) we can provide explicit priorities using a -p flag to add-external-repo. Check with
−
  koji taginfo dist-epel5-build
+
  koji taginfo epel5-build
  
−
These groups are going to be passed to 'yum groupinstall' when the buildroot is created.
 
  
−
koji add-group epel5-build build
+
For epel6, we can take a few shortcuts thanks to the power of inheritance.
−
koji add-group epel5-build srpm-build
 
  
−
They need to be populated. Clone from a typical value for the Fedora Koji systems, e.g.
+
Make epel6 a child of epel5 so it inherits all the packages.
 +
koji add-tag --parent epel5 epel6
 +
koji add-tag --parent epel6 --arches "i386 x86_64" epel6-build
 +
koji add-tag --parent epel6 epel6-candidates
 +
koji add-tag --parent epel6 epel6-testing
 +
koji add-tag --parent epel6 epel6-release
 +
koji add-tag --parent epel5-basedeps epel6-basedeps
  
−
  koji add-group-pkg epel5-build build `koji -s http://koji.fedoraproject.org/kojihub/
+
The external repos are simpler:
−
      list-groups dist-5E-epel-build build | tail -n +2 | cut -d: -f1`
+
  koji add-external-repo -t epel6-build \
−
Be careful to exclude the fedpkg package in srpm-build which pulls in way too much! It includes mock which causes strange failures with conflicting group ids within the buildroot.
+
      dist-epel6-external-repo http://mirror.nl.leaseweb.net/epel/6/$arch/
−
  koji add-group-pkg epel5-build srpm-build `koji -s http://koji.fedoraproject.org/kojihub/
+
  koji add-external-repo -t epel6-build dist-centos6-repo \
−
      list-groups dist-5E-epel-build srpm-build | tail -n +2 | cut -d: -f1 | grep -v fedpkg`
+
      http://spiegel.nikhef.nl/mirror/centos/6/os/\$arch/
 +
koji add-target epel6 epel6-build epel6-candidates
  
−
Don't forget to add each package!
+
For Fedora. It starts become a bit boring.
−
koji add-pkg --owner <''your name''> epel5 lcmaps jobrepository ees glexec \
 
−
          lcas-lcmaps-gt4-interface lcas-plugins-basic lcas-plugins-check-executable \
 
−
          lcas-plugins-voms lcas lcmaps-plugins-afs lcmaps-plugins-basic lcmaps-plugins-c-pep lcmaps-plugins-gums \
 
−
          lcmaps-plugins-jobrep lcmaps-plugins-scas-client lcmaps-plugins-tracking-groupid lcmaps-plugins-verify-proxy \
 
−
          lcmaps-plugins-voms scas xacml argus-pep-api-c glexec-wn nagios-plugins-glexec nagios-plugins-ees mkgltempdir \
 
−
          glexec-wrapper-scripts torque llrun saml2-xacml2-c-lib lcmaps-without-gsi mwsec-release
 
  
−
For epel6, we can take a shortcut.
+
koji add-tag --parent epel5 f17
 +
koji add-tag --parent f17 --arches "i386 x86_64" f17-build
 +
for i in candidates testing release; do
 +
    koji add-tag --parent f17 f17-$i
 +
done
 +
koji add-tag --parent epel5-basedeps f17-basedeps # may need adding/removing some group packages
 +
koji add-external-repo -t f17-build dist-f17-base http://mirror.nl.leaseweb.net/fedora/linux/releases/17/Everything/\$arch/os/
 +
koji add-target f17 f17-build f17-candidates
  
−
  koji add-tag --parent epel5 epel6
+
  for i in 18 19 20 ; do
−
koji add-tag --parent epel6 --arches "i386 x86_64" epel6-build
+
    prev=`expr $i - 1`
−
koji add-external-repo -t epel6-build \
+
    koji add-tag --parent f$prev f$i
−
      dist-epel5-external-repo http://mirrors.nl.eu.kernel.org/fedora-epel/5/\$arch/
+
    koji add-tag --parent $i --arches "i386 x86_64" $i-build
−
koji add-external-repo -t epel5-build dist-epel5-buildsys-macros-repo \
+
    for j in candidates testing release; do
−
      http://buildsys.fedoraproject.org/buildgroups/rhel5/\$arch/
+
        koji add-tag --parent $i $i-$j
−
koji add-external-repo -t dist-epel5-build dist-epel5-centos5-repo \
+
    done
−
      http://spiegel.nikhef.nl/mirror/centos/5/os/\$arch/
+
    koji add-tag --parent f$prev-basedeps $i-basedeps
−
koji add-target epel5 epel5-build epel5-candidates
+
    koji add-external-repo -t $i-build dist-$i-base http://mirror.nl.leaseweb.net/fedora/linux/releases/$i/Everything/\$arch/os/
 +
    koji add-target $i $i-build $i-candidates
 +
done
  
 
= Automated builds from tags in mwsec =
 
= Automated builds from tags in mwsec =

Revision as of 10:02, 1 August 2013

Retrieved from "https://wiki.nikhef.nl/grid/index.php?title=Koji_Testbed&oldid=9049"