Difference between revisions of "Koji Testbed"

From PDP/Grid Wiki
Jump to navigationJump to search
(12 intermediate revisions by 2 users not shown)
Line 100: Line 100:
 
  $ koji list-tag-inheritance --reverse f16
 
  $ koji list-tag-inheritance --reverse f16
 
  f16 (7)  <- contains ONLY packages in the first sense
 
  f16 (7)  <- contains ONLY packages in the first sense
   ??f17 (9)  <- so does this, by inheritance
+
   +-f17 (9)  <- so does this, by inheritance
   ? ??f17-candidates (33)  <- this is the "build target" "destination tag"
+
   | +-f17-candidates (33)  <- this is the "build target" "destination tag"
   ? ??f17-release (25)  <- this is used manually to mark a package (second sense) ready for shipping with mash
+
   | +-f17-release (25)  <- this is used manually to mark a package (second sense) ready for shipping with mash
   ? ??f17-testing (26)  <- idem, but the for the 'testing' repo
+
   | +-f17-testing (26)  <- idem, but the for the 'testing' repo
   ? ??f18 (27)  <- only packages, first sense
+
   | +-f18 (27)  <- only packages, first sense
   ? ? ??f18-build (30)  <- the "build target" "build tag"; this has architectures and external repos
+
   | | +-f18-build (30)  <- the "build target" "build tag"; this has architectures and external repos
   ? ? ? ??f18-candidates (31)  -< the "build target" "destination tag"
+
   | | | +-f18-candidates (31)  -< the "build target" "destination tag"
   ? ? ??f19 (34)
+
   | | +-f19 (34)
   ? ? ? ??f19-candidates (38)
+
   | | | +-f19-candidates (38)
   ? ? ? ??f19-release (43)
+
   | | | +-f19-release (43)
   ? ? ? ??f19-testing (44)
+
   | | | +-f19-testing (44)
   ? ? ? ??f19-build (36)
+
   | | | +-f19-build (36)
   ? ? ??f18-release (39)
+
   | | +-f18-release (39)
   ? ? ??f18-testing (40)
+
   | | +-f18-testing (40)
   ? ??f17-build (32)
+
   | +-f17-build (32)
   ??f16-release (24)
+
   +-f16-release (24)
   ??f16-testing (23)
+
   +-f16-testing (23)
   ??f16-build (45)
+
   +-f16-build (45)
  
 
And this one:
 
And this one:
Line 123: Line 123:
 
  $ koji list-tag-inheritance f19-build
 
  $ koji list-tag-inheritance f19-build
 
  f19-build (36)  <- the "build target" "build tag"
 
  f19-build (36)  <- the "build target" "build tag"
   ??f19 (34)  <- package names only (first sense)
+
   +-f19 (34)  <- package names only (first sense)
   ? ??f18 (27)  <- all the way
+
   | +-f18 (27)  <- all the way
   ?     ??f17 (9)  <-  down to
+
   |     +-f17 (9)  <-  down to
   ?       ??f16 (7)  <- here
+
   |       +-f16 (7)  <- here
   ??f19-basedeps (35)  <- these contain the 'groups'
+
   +-f19-basedeps (35)  <- these contain the 'groups'
       ??f18-basedeps (28)  <- actually inherited from
+
       +-f18-basedeps (28)  <- actually inherited from
         ??f17-basedeps (10)  <- here
+
         +-f17-basedeps (10)  <- here
  
 
A build for F19 ends up in the tag f19-candidates. From there, it needs to be manually moved. If it is a build dependency for other packages, it has to go to f19-build. Otherwise, it eventually goes to f19-testing and from there to f19-release, or directly to f19-release. A rotten product should have its candidacy stripped and ends up with no tags. In any case, no package should remain a candidate for too long.
 
A build for F19 ends up in the tag f19-candidates. From there, it needs to be manually moved. If it is a build dependency for other packages, it has to go to f19-build. Otherwise, it eventually goes to f19-testing and from there to f19-release, or directly to f19-release. A rotten product should have its candidacy stripped and ends up with no tags. In any case, no package should remain a candidate for too long.
Line 161: Line 161:
  
 
A Koji build will first do a <nowiki>buildSRPMFromSCM</nowiki> job, which will run mock to generate a buildroot, inside of which 'yum groupinstall srpm-build' is run. Later, a <nowiki>buildArch</nowiki> job does the actual building, and here mock runs 'yum groupinstall build'.
 
A Koji build will first do a <nowiki>buildSRPMFromSCM</nowiki> job, which will run mock to generate a buildroot, inside of which 'yum groupinstall srpm-build' is run. Later, a <nowiki>buildArch</nowiki> job does the actual building, and here mock runs 'yum groupinstall build'.
These groups must be created by us first, and populated with package names that we need to have available at this stage. Since these groups do not vary much between distributions, we create a separate tag and use inheritance to save work.  
+
These groups must be created by us first, and populated with package names that we need to have available at this stage.
koji add-tag epel5-basedeps
+
  koji add-group epel5-build build
  koji add-group epel5-basedeps build
+
  koji add-group epel5-build srpm-build
  koji add-group epel5-basedeps srpm-build
+
   
  koji add-tag-inheritance epel5-build epel5-basedeps
 
 
They need to be populated. Clone from a typical value for the Fedora Koji systems, e.g.
 
They need to be populated. Clone from a typical value for the Fedora Koji systems, e.g.
  
  koji add-group-pkg epel5-basedeps build `koji -s http://koji.fedoraproject.org/kojihub/
+
  koji add-group-pkg epel5-build build `koji -s http://koji.fedoraproject.org/kojihub/
 
       list-groups dist-5E-epel-build build | tail -n +2 | cut -d: -f1`
 
       list-groups dist-5E-epel-build build | tail -n +2 | cut -d: -f1`
 
Be careful to exclude the fedpkg package in srpm-build which pulls in way too much! It includes mock which causes strange failures with conflicting (unix) group ids within the buildroot.
 
Be careful to exclude the fedpkg package in srpm-build which pulls in way too much! It includes mock which causes strange failures with conflicting (unix) group ids within the buildroot.
  koji add-group-pkg epel5-basedeps srpm-build `koji -s http://koji.fedoraproject.org/kojihub/
+
  koji add-group-pkg epel5-build srpm-build `koji -s http://koji.fedoraproject.org/kojihub/
 
       list-groups dist-5E-epel-build srpm-build | tail -n +2 | cut -d: -f1 | grep -v fedpkg`
 
       list-groups dist-5E-epel-build srpm-build | tail -n +2 | cut -d: -f1 | grep -v fedpkg`
 +
 +
As of EPEL7, there's fedpkg-minimal which should be safe (it only requires wget).
  
 
Add external repositories to populate the buildroot.
 
Add external repositories to populate the buildroot.
Line 190: Line 191:
  
  
For epel6, we can take a few shortcuts thanks to the power of inheritance.
+
Since the srpm-build and build do not vary much between distributions,
 
+
we create a separate tag and use inheritance to save work for the Fedora branches.
Make epel6 a child of epel5 so it inherits all the packages.
+
This does not hold true for EPEL branches, which are not inheriting from one another.
koji add-tag --parent epel5 epel6
 
koji add-tag --parent epel6 --arches "i386 x86_64" epel6-build
 
koji add-tag --parent epel6 epel6-candidates
 
koji add-tag --parent epel6 epel6-testing
 
koji add-tag --parent epel6 epel6-release
 
koji add-tag --parent epel5-basedeps epel6-basedeps
 
 
 
The external repos are simpler:
 
koji add-external-repo -t epel6-build \
 
      dist-epel6-external-repo http://mirror.nl.leaseweb.net/epel/6/$arch/
 
koji add-external-repo -t epel6-build dist-centos6-repo \
 
      http://spiegel.nikhef.nl/mirror/centos/6/os/\$arch/
 
koji add-target epel6 epel6-build epel6-candidates
 
 
 
For Fedora. It starts to become a bit boring.
 
 
 
koji add-tag --parent epel5 f17
 
koji add-tag --parent f17 --arches "i386 x86_64" f17-build
 
for i in candidates testing release; do
 
    koji add-tag --parent f17 f17-$i
 
done
 
koji add-tag --parent epel5-basedeps f17-basedeps # may need adding/removing some group packages
 
koji add-external-repo -t f17-build dist-f17-base http://mirror.nl.leaseweb.net/fedora/linux/releases/17/Everything/\$arch/os/
 
koji add-target f17 f17-build f17-candidates
 
  
  for i in 18 19 20 ; do
+
  for i in 27 28 ; do
 
     prev=`expr $i - 1`
 
     prev=`expr $i - 1`
 
     koji add-tag --parent f$prev f$i
 
     koji add-tag --parent f$prev f$i
     koji add-tag --parent f$prev-basedeps f$i-basedeps
+
     koji add-tag --parent f$prev-basedeps f${i}-basedeps
     koji add-tag --parent f$i --parent f$i-basedeps --arches "i386 x86_64" f$i-build
+
     # koji add-tag --parent f$i --parent f$i-basedeps --arches "i386 x86_64" f${i}-build
 +
    koji add-tag --parent f$i --parent f${i}-basedeps --arches "x86_64" f${i}-build
 
     for j in candidates testing release; do
 
     for j in candidates testing release; do
         koji add-tag --parent f$i f$i-$j
+
         koji add-tag --parent f$i f${i}-$j
 
     done
 
     done
     koji add-external-repo -t f$i-build dist-f$i-base http://mirror.nl.leaseweb.net/fedora/linux/releases/$i/Everything/\$arch/os/
+
     koji add-external-repo -t f${i}-build dist-f${i}-base http://mirror.nl.leaseweb.net/fedora/linux/releases/${i}/Everything/\$arch/os/
     koji add-target f$i f$i-build $i-candidates
+
    # '''NOTE''': up to 27, we should leave out the ''Everything'' for the updates
 +
    koji add-external-repo -t f${i}-build dist-f${i}-updates http://mirror.nl.leaseweb.net/fedora/linux/updates/${i}/Everything/\$arch/
 +
     koji add-target f$i f${i}-build f${i}-candidates
 +
    # '''NOTE''': probably need to add priorities to the parents, e.g. using something like
 +
    koji edit-tag-inheritance --priority 2 f${i}-build f${i}-basedeps
 +
    koji add-tag-inheritance --priority 1 f${i}-build f${i}
 
  done
 
  done
  
Line 520: Line 503:
  
 
We've seen the weird situation that the connection between the bridge and the server was semi-severed: the bridge still showed the connection in the output of netstat, but the server lost the connection. The resolution is to restart the server. If the bridge needs to be restarted, the server also needs a restart thereafter.
 
We've seen the weird situation that the connection between the bridge and the server was semi-severed: the bridge still showed the connection in the output of netstat, but the server lost the connection. The resolution is to restart the server. If the bridge needs to be restarted, the server also needs a restart thereafter.
 +
 +
UPDATE: the cause of this problem lies with the connection being too quiet. Without keepalive the connection will be dropped. It's the stateful firewall's fault.

Revision as of 18:34, 29 July 2018