Difference between revisions of "SAC software procedures"

From PDP/Grid Wiki
Jump to navigationJump to search
 
(91 intermediate revisions by 4 users not shown)
Line 1: Line 1:
 
This page contains several procedures regarding the release process of the Grid Security Middleware.
 
This page contains several procedures regarding the release process of the Grid Security Middleware.
 +
 +
== Contributing features and bug fixes ==
 +
 +
Community contributions and bug fixes are welcome through our [https://bugzilla.nikhef.nl/ Bugzilla service]. Please [[Site_Access_Control#Contact|contact us]] for custom integration procedures.
  
 
== Source distribution release procedure ==
 
== Source distribution release procedure ==
Line 11: Line 15:
 
# BUGS: Updated the BUGS file in the component directory with known issues
 
# BUGS: Updated the BUGS file in the component directory with known issues
 
# NEWS Updated the NEWS file: write release notes like information. Hint which version solves the problem.
 
# NEWS Updated the NEWS file: write release notes like information. Hint which version solves the problem.
−
# ChangeLog: a "svn log > ChangeLog" for the time being.
+
# ChangeLog: a "svn update ; svn log -v > ChangeLog" for the time being.
 
# INSTALL: Update installation instruction file (if applicable)
 
# INSTALL: Update installation instruction file (if applicable)
 
# README: Update the README file with more useful information
 
# README: Update the README file with more useful information
Line 19: Line 23:
 
==== Extra info: Source repository details ====
 
==== Extra info: Source repository details ====
  
−
The main source repository for these components is https://ndpfsvn.nikhef.nl/cgi-bin/viewvc.cgi/mwsec/, which can be used with SVN in the following ways:
+
The main source repository for these components is https://ndpfsvn.nikhef.nl/viewvc/mwsec/, which can be used with SVN in the following ways:
  
 
SVN with SSH public key authentication (NB. username is ''always'' svn):
 
SVN with SSH public key authentication (NB. username is ''always'' svn):
Line 80: Line 84:
 
  svn co svn+ssh://svn@ndpfsvn.nikhef.nl/repos/mwsec/tags/glexec/0_8_10 glexec_0_8_10
 
  svn co svn+ssh://svn@ndpfsvn.nikhef.nl/repos/mwsec/tags/glexec/0_8_10 glexec_0_8_10
 
  cd glexec_0_8_10
 
  cd glexec_0_8_10
−
* Run the ./bootstrap on fc14.testbed
+
* Run the ./bootstrap on fedora20.testbed
 
  cd ${topdir}
 
  cd ${topdir}
 
  ./bootstrap
 
  ./bootstrap
−
* Run the ./configure on mwsecbuild.testbed
+
* Run the ./configure on mwsec-el6-64.testbed
 
  cd ${topdir}
 
  cd ${topdir}
 
  mkdir BUILD && cd BUILD
 
  mkdir BUILD && cd BUILD
Line 93: Line 97:
 
  sha1sum <component>-<version>.tar.gz > <component>-<version>.tar.gz.sha1
 
  sha1sum <component>-<version>.tar.gz > <component>-<version>.tar.gz.sha1
 
  sha256sum <component>-<version>.tar.gz > <component>-<version>.tar.gz.sha256
 
  sha256sum <component>-<version>.tar.gz > <component>-<version>.tar.gz.sha256
 +
For lazy people:
 +
for FILE in `ls *.tar.gz`; do echo $FILE; sha1sum $FILE > $FILE.sha1; sha256sum $FILE > $FILE.sha256; done
 
* Copy the tarball and the sha1 and sha256 checksum files to:
 
* Copy the tarball and the sha1 and sha256 checksum files to:
 
  software.nikhef.nl:/project/srv/www/site/software/html/security/<''component''>/release-candidate/
 
  software.nikhef.nl:/project/srv/www/site/software/html/security/<''component''>/release-candidate/
Line 100: Line 106:
 
== Generating RPM packages from the distributed tarballs ==
 
== Generating RPM packages from the distributed tarballs ==
  
−
The creation of RPM packages involves the following steps.
+
Release RPMs are build using our own koji instance, see [[Koji_Testbed]] for details on the Koji setup itself.
−
* Creating a source RPM from the source tarball(s) and SPEC file (on an EL5 machine, e.g. mwsecbuild.testbed);
 
−
* creating a binary RPM from the source RPM by running a mock build (on a reasonable modern mock >= 1.0.7);
 
−
* copying the resulting RPMs (source and binaries) to the mock repository on bleek.nikhef.nl.
 
−
 
 
−
For distributing the results more steps are necessary, see [[#Distribution release of RPMS|below]].
 
  
−
=== Procedure for building a new version or building a new release ===
+
'''Note''': information about how to use mock to make test-builds can be found on the [[Building RPMs using mock]] page.
  
 
The following steps need to be taken:
 
The following steps need to be taken:
  
 
* [[#Update the SPEC file|update the SPEC file]],
 
* [[#Update the SPEC file|update the SPEC file]],
−
* [[#Generating the source RPM|generating the source RPM]],
+
* [[#Starting the Koji build by creating a new tag for the SPEC file|Starting the Koji build by creating a new tag for the SPEC file]],
−
* [[#Generating the binary and source RPMs|generating the binary and source RPMs]].
+
* [[#Adding koji tags to the builds|Adding koji tags to the builds]],
 +
* [[#Updating the mash repositories|Updating the mash repositories]],
 +
* [[#Syncing with software.nikhef.nl|Syncing with software.nikhef.nl]].
  
−
==== Update the SPEC file ====
+
=== Update the SPEC file ===
  
 
<ul>
 
<ul>
−
<li>Check out the SPEC directory from the SVN repository (always use trunk):
+
<li> Either check out or update the SPEC files directories from the SVN repository
−
<pre>svn co svn+ssh://svn@ndpfsvn.nikhef.nl/repos/mwsec/packaging/fedora/trunk</pre>
+
<pre>
−
<li>Edit the spec file<br>''For a new version (the source has changed):''
+
svn co svn+ssh://svn@ndpfsvn.nikhef.nl/repos/mwsec/packaging/fedora
 +
cd fedora/trunk
 +
</pre>
 +
or
 +
<pre>
 +
cd fedora/trunk
 +
svn update
 +
</pre>
 +
<li> Edit the spec file<br>''For a new version (the source has changed):''
 
<ul><li> Edit the spec file by updating the ''Version:'' field to correspond to the new version.
 
<ul><li> Edit the spec file by updating the ''Version:'' field to correspond to the new version.
 
<li> '''Reset the Release: field to 1''', but don't delete <code>%{?dist}</code>.
 
<li> '''Reset the Release: field to 1''', but don't delete <code>%{?dist}</code>.
Line 129: Line 140:
 
</ul>
 
</ul>
  
−
==== Generating the source RPM ====
+
=== Starting the Koji build by creating a new tag for the SPEC file ===
  
−
The source RPM must be generated on an EL5 machine, because EL5 and older systems have a version of RPM that doesn't understand the new SHA1 checksum.
+
<ul>
 +
<li> Make sure the <tt>trunk</tt> and tags/&lang;component&rang; directories are up-to-date:
 +
<pre>
 +
cd fedora/trunk
 +
svn update
 +
cd ../tags/&lang;component&rang;
 +
svn update
 +
</pre>
 +
<li> Create a new tag:
 +
<pre>svn mkdir &lang;tag&rang;</pre>
 +
<li> Copy the spec file from the trunk directory
 +
<pre>svn copy ../../trunk/&lang;component&rang;.spec &lang;tag&rang;</pre>
 +
<li> Create a new Makefile (don't forget the <tt>svn add</tt> or <tt>svn copy</tt> from an older tag. It should contain something like
 +
<pre>
 +
cwd = $(abspath .)
 +
version = $(notdir $(cwd))
 +
component = $(notdir $(patsubst %/, %, $(dir $(cwd))))
  
−
Get the sources from the official software distribution:
+
sources:
−
wget <nowiki>http://software.nikhef.nl/security/</nowiki>''component''/''component-version''.tar.gz
+
        curl -LRO http://software.nikhef.nl/security/$(component)/$(component)-$(version).tar.gz
 +
        sed -i -e 's/^Version: .*/Version: $(version)/' $(component).spec
 +
</pre>
 +
<li>Commit will automatically trigger a koji build:
 +
<pre>svn commit &lang;tag&rang;</pre>
 +
</ul>
  
−
Get the SPEC file:
+
=== Adding koji tags to the builds ===
−
wget <nowiki>http://ndpfsvn.nikhef.nl/ro/mwsec/packaging/fedora/trunk/</nowiki>''component''.spec
 
  
−
Build the source RPM:
+
Once koji is finished (see e.g. <tt>koji list-tasks</tt> or use the web-interface via [http://koji-hub.testbed/koji/tasks http://koji-hub.testbed/koji/tasks]) the results should will be tagged as ''candidates''. When sufficiently satisfied they should be tagged as ''testing'' or ''release''. Once they are tagged as ''release'' they should also be tagged as ''build'', and will be used to build against (this will trigger a rebuild of the buildrepo):
−
  rpmbuild --define "_tmppath /tmp" --define "_sourcedir ." --define "_srcrpmdir ." --nodeps -bs ''component''.spec
+
First
 +
  component=glexec; version=0.9.8-1 # Example
 +
 +
for i in 6 7 8; do
 +
  koji move-pkg --nowait epel${i}-candidates epel${i}-testing ${component}-${version}.el${i}
 +
done
 +
for i in 30; do
 +
  koji move-pkg --nowait f${i}-candidates f${i}-testing ${component}-${version}.fc${i}
 +
done
  
−
This will leave the source RPM in the current directory. This needs to be transferred to the mock build machine.
+
==== Moving from testing to release ====
  
 +
Builds can be moved from testing to release using e.g.
 +
component=glexec; version=0.9.8-1 # Example
 +
 +
for i in 6 7 8; do
 +
  koji move-pkg --nowait epel${i}-testing epel${i}-release ${component}-${version}.el${i}
 +
  koji tag-pkg --nowait epel${i}-build ${component}-${version}.el${i}
 +
done
 +
for i in 30; do
 +
  koji move-pkg --nowait f${i}-testing f${i}-release ${component}-${version}.fc${i}
 +
  koji tag-pkg --nowait f${i}-build ${component}-${version}.fc${i}
 +
done
 +
Note that multiple components can be tagged using one <tt>koji tag-pkg</tt> command.
  
−
Alternatively one can use mock (see also [[#Generating the binary and source RPMs|Generating the binary and source RPMs]]):
+
For ''testing'' it is best to untag all older tags:
−
  mock -r epel-5-x86_64 --spec=''component''.spec --sources=. --resultdir=mock/result --buildsrpm
+
  for dist in epel6 epel7 epel8 f30; do
−
This will leave the source RPM in mock/result.
+
    koji untag-pkg --non-latest ${dist}-testing ${component}
 +
done
  
−
==== Generating the binary and source RPMs ====
+
After this, the mash and rsync steps described below have to be run.
  
−
Copy the source RPM to a relatively new mock build machine, e.g. CentOS 6, Fedora Core 14 or higher or Ubuntu :-).
+
=== Signing the release RPMs with sigul ===
  
−
The configuration to be used should be tuned to include the stable mwsec repository at  http://software.nikhef.nl/dist/redhat/el5/mwsec/, see [[mwsec mock setup]] for details.
+
When a new build has been tagged, it needs to be signed with the [http://software.nikhef.nl/dist/mwsec/RPM-GPG-KEY-MWSEC GPG key]  
 +
pub  4096R/980AC8BB 2012-03-21
 +
uid                  Nikhef Grid Security Middleware Signing Key <grid-mw-security@nikhef.nl>
 +
which resides on sigul.testbed.
  
−
Now run mock:
+
On koji-hub.testbed there is a handy script to sign all unsigned packages for a tag.
−
mock -D 'dist .el5' -r mwsec-el5-x86_64 --configdir=mock/config --resultdir=mock/result --rebuild ''component''.src.rpm
 
−
mock -D 'dist .el5' -r mwsec-el5-i386 --configdir=mock/config --resultdir=mock/result --rebuild ''component''.src.rpm
 
−
This creates source and binary RPMS in mock/result using the configuration setup from mock/config/
 
−
NOTE: the explicit setting of the %dist macro is needed to prevent it from becoming the new Centos standard .el5.centos.
 
  
−
On '''<tt>fc14.testbed</tt>''' these configs are installed in the standard directory, and the <tt>--configdir</tt> option should be left out.
+
sigulsign-mwsec.py --tag=epel5-release grid_mwsec
−
In addition, on this machine, two configs which also enable the mwsec testing repo have been installed: <tt>mwsec_testing-el5-x86_64</tt> and <tt>mwsec_testing-el5-i386</tt>. These should ''only'' be used for either testing purposes or in rare circumstances when multiple packages need updating at the same time.
 
  
−
==== Publishing the created binaries ====
+
Due to a weird bug it may happen some packages are *not* signed; in this case retrieve the list of packages to sign
 +
sigulsign-mwsec.py --tag=epel5-release --just-list grid_mwsec | sed 's/\.rpm$//' > to-sign
 +
and sign manually
 +
sigul sign-rpms --v3-signature --store-in-koji --koji-only grid_mwsec `cat to-sign`
  
−
The next steps are
+
The signing does not write RPMs to disk, it just imports the signatures into koji. Tell koji to write the RPMs now.
−
* upload the mock-created binary and source RPMS to the corresponding directories in bleek.nikhef.nl:/srv/project/mock/rhel5/
+
koji write-signed-rpm 980ac8bb glexec-0.8.9-1.el5
−
* create symlinks for the binary RPMS in bleek.nikhef.nl:/srv/project/mwsec/testing/el5/mwsec/
+
This step is not necessary when signing with the sigulsign-mwsec.py script.
−
* create a symlink for the source RPM in bleek.nikhef.nl:/srv/project/mwsec/el5/mwsec/
 
−
* after the new release has been certified, create symlinks for the binary RPMS in bleek.nikhef.nl:/srv/project/mwsec/el5/mwsec/, pointing to those in the testing repository.
 
−
See [[#Distribution release of RPMS|Distribution release of RPMS]] for further details.
 
  
−
=== Generating RPMs manually ===
+
=== Updating the mash repositories ===
  
−
The above procedure produces 'official' RPMS, in a reproducible way from a clean build environment. However, it is possible to generate RPM packages manually with a simple recipe.
+
When a new build has been tagged into either release and/or testing, the corresponding repositories need to be recreated using mash. This currently has to be done on <tt>koji-hub.testbed</tt>.
−
 
+
NOTE: any missing signed rpms (see the above procedure) will result in mash failure.
−
This procedure should work on a machine that matches the build target, i.e. CentOS 5 x86_64 or i386.
+
<ul>
−
 
+
<li> Set correct umask and run mash on all (or the relevant) repositories:
−
* Create a file called <code>.rpmmacros</code> with the following content:
+
<pre>
−
%_topdir  /home/''username''/rpmbuild
+
umask 0002; for dist in epel6 epel7 epel8 f30; do
−
%_tmppath /home/''username''/rpmbuild/tmp
+
    mash -o /mnt/mwsec/mash signed-${dist}-release
−
* Create a directory <code>rpmbuild</code> in your home directory, with the following subdirectories:
+
    mash -o /mnt/mwsec/mash signed-${dist}-testing
−
BUILD -> /tmp/''username''/rpmbuild/BUILD
+
done
−
RPMS
+
</pre>
−
SOURCES
+
</ul>
−
SPECS
 
−
SRPMS
 
−
tmp -> /tmp/''username''/rpmbuild/tmp
 
−
* create the tempdirs if necessary
 
−
mkdir -p /tmp/`id -un`/rpmbuild/{tmp,BUILD}
 
−
* Check out the SPEC directory
 
−
* test the build dependencies by running <code>rpmbuild ''specfile''</code>
 
−
* Install build dependencies as necessary
 
−
* Run <code>rpmbuild -ba ''specfile''</code> to generate binary and source RPMs.
 
−
 
 
−
=== Adding a new component and/or build target ===
 
−
 
 
−
This procedure is not yet worked out.
 
−
 
 
−
== Distribution release of RPMS ==
 
−
 
 
−
'''Warning: this procedure is not yet finalised.'''
 
−
 
 
−
The RPMS should all be on bleek.nikhef.nl at this time.
 
−
 
 
−
The generated RPMS should be distributed in a public, yum-installable repository. The repository is composed on bleek.nikhef.nl:/srv/project/mwsec with the following lay-out:
 
−
./el5
 
−
./el5/mwsec
 
−
./el5/mwsec/SRPMS            <- symlinks to mock
 
−
./el5/mwsec/SRPMS/repodata
 
−
./el5/mwsec/x86_64          <- symlinks to testing
 
−
./el5/mwsec/x86_64/repodata
 
−
./el5/mwsec/i386            <- symlinks to testing
 
−
./el5/mwsec/i386/repodata
 
−
./testing/
 
−
./testing/el5
 
−
./testing/el5/mwsec
 
−
./testing/el5/mwsec/i386            <- symlinks to mock
 
−
./testing/el5/mwsec/i386/repodata
 
−
./testing/el5/mwsec/x86_64          <- symlinks to mock
 
−
./testing/el5/mwsec/x86_64/repodata
 
  
−
The el5 and testing trees do not contain actual RPMS, but symlinks:
+
=== Syncing with software.nikhef.nl ===
−
* the testing RPMS are symlinks to the repective RPMS in mock
 
−
* the el5 binary RPMS are symlinks to the respective files in testing
 
−
* the el5 source RPMS are symlinks to the respective RPMS in mock
 
  
−
The noarch RPMS in mock are symlinked in both x86_64 and i386 (cf. the EPEL repository).
+
The repository on [http://software.nikhef.nl/dist/mwsec/rpm/ http://software.nikhef.nl/dist/mwsec/rpm/] is rsync-ed from <tt>bleek.nikhef.nl:/srv/project/mwsec/rpm/</tt> which contains only symlinks to the mash directories. This is necessary since mash cannot easily handle the combination epel5 testing/epel5. The directory layout is
 +
rpm/epel5 -> ../mash/signed-epel5-release
 +
rpm/epel6 -> ../mash/signed-epel6-release
 +
rpm/epel7 -> ../mash/signed-epel7-release
 +
rpm/epel8 -> ../mash/signed-epel8-release
 +
rpm/fc16 -> ../mash/fc16-release (out of support)
 +
rpm/f17 -> ../mash/signed-f17-release (out of support)
 +
rpm/fc17 -> f17 (original name retained, out of support)
 +
rpm/f18 -> ../mash/signed-f18-release (out of support)
 +
rpm/f19 -> ../mash/signed-f19-release (out of support)
 +
rpm/f20 -> ../mash/signed-f20-release (out of support)
 +
rpm/f21 -> ../mash/signed-f21-release (out of support)
 +
rpm/f22 -> ../mash/signed-f22-release (out of support)
 +
rpm/f23 -> ../mash/signed-f23-release (out of support)
 +
rpm/f24 -> ../mash/signed-f24-release (out of support)
 +
rpm/f25 -> ../mash/signed-f25-release (out of support)
 +
rpm/f26 -> ../mash/signed-f26-release (out of support)
 +
rpm/f27 -> ../mash/signed-f27-release (out of support)
 +
rpm/f28 -> ../mash/signed-f28-release (out of support)
 +
rpm/f29 -> ../mash/signed-f29-release (out of support)
 +
rpm/f30 -> ../mash/signed-f30-release
 +
rpm/testing/epel5 -> ../../mash/signed-epel5-testing
 +
rpm/testing/epel6 -> ../../mash/signed-epel6-testing
 +
rpm/testing/epel7 -> ../../mash/signed-epel7-testing
 +
rpm/testing/epel8 -> ../../mash/signed-epel8-testing
 +
rpm/testing/fc16 -> ../../mash/fc16-testing (out of support)
 +
rpm/testing/f17 -> ../../mash/fc17-testing (out of support)
 +
rpm/testing/fc17 -> f17 (original name retained, out of support)
 +
rpm/testing/f18 -> ../../mash/signed-fc18-testing (out of support)
 +
rpm/testing/f19 -> ../../mash/signed-fc19-testing (out of support)
 +
rpm/testing/f20 -> ../../mash/signed-fc20-testing (out of support)
 +
rpm/testing/f21 -> ../../mash/signed-fc21-testing (out of support)
 +
rpm/testing/f22 -> ../../mash/signed-fc22-testing (out of support)
 +
rpm/testing/f23 -> ../../mash/signed-fc23-testing (out of support)
 +
rpm/testing/f24 -> ../../mash/signed-fc24-testing (out of support)
 +
rpm/testing/f25 -> ../../mash/signed-fc25-testing (out of support)
 +
rpm/testing/f26 -> ../../mash/signed-fc26-testing (out of support)
 +
rpm/testing/f27 -> ../../mash/signed-fc27-testing (out of support)
 +
rpm/testing/f28 -> ../../mash/signed-fc28-testing (out of support)
 +
rpm/testing/f29 -> ../../mash/signed-fc29-testing (out of support)
 +
rpm/testing/f30 -> ../../mash/signed-fc30-testing
  
−
The layout of mock is
+
The rsync command to run from bleek.nikhef.nl is
−
  /srv/project/mock
+
  /srv/project/mwsec/rsync_software.sh
−
  /srv/project/mock/rhel5
+
or from koji-hub.testbed
−
  /srv/project/mock/rhel5/SRPMS
+
  /mnt/mwsec/rsync_software.sh
−
/srv/project/mock/rhel5/i386
+
or manually from bleek.nikhef.nl
−
/srv/project/mock/rhel5/noarch
+
  rsync --exclude=epel5/SPECS\* --del -rltpgv --copy-unsafe-links /srv/project/mwsec/rpm/ software.nikhef.nl:/project/srv/www/site/software/html/dist/mwsec/rpm
−
/srv/project/mock/rhel5/x86_64
+
''''NOTE'''': the / after the first rpm/ is necessary. Alternatively chdir to /srv/project/mwsec/rpm/ and use . instead.
  
−
After updating the el5 and/or testing directories, update the repodata:
+
Notes:
−
createrepo /srv/project/mwsec/el5/mwsec/SRPMS/
+
* RPMs will reside in either testing or the normal repository but not in both.
−
createrepo /srv/project/mwsec/el5/mwsec/i386/
+
* mash will copy RPMs from the koji builds found in <tt>/mnt/koji/packages</tt> (an NFS mount from put.testbed).
−
createrepo /srv/project/mwsec/el5/mwsec/x86_64/
+
* mash will copy noarch RPMs to both x86_64 and i386 (cf. the official EPEL repository).
−
createrepo /srv/project/mwsec/testing/el5/mwsec/i386/
 
−
createrepo /srv/project/mwsec/testing/el5/mwsec/x86_64/
 
−
chmod -R g+w /srv/project/mwsec
 
  
−
After this step, the repositories need to be synced with software.nikhef.nl:
+
Using the repository in a YUM repo file requires something like the following:
−
rsync -rltpgv --copy-unsafe-links /srv/project/mwsec/ software.nikhef.nl:/project/srv/www/site/software/html/dist/redhat/
 
−
 
−
Using this repository in a YUM repo file requires something like the following:
 
 
  [mwsec]
 
  [mwsec]
 
  name=Middleware Security Software
 
  name=Middleware Security Software
−
  baseurl=http://software.nikhef.nl/dist/redhat/el5/mwsec/$basearch
+
  baseurl=http://software.nikhef.nl/dist/mwsec/rpm/epel5/$basearch
−
  gpgcheck=0
+
  gpgcheck=1
−
 
+
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-MWSEC
−
== Generating Debian packages from the distributed source tarballs ==
+
 +
[mwsec-testing]
 +
name=Middleware Security Software testing
 +
baseurl=http://software.nikhef.nl/dist/mwsec/rpm/testing/epel5/$basearch
 +
enabled=0
 +
gpgcheck=1
 +
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-MWSEC
  
−
Debian packaging requires a <tt>debian/</tt> subdirectory in the root of the upstream source directory. The structure of this directory is described in the [http://www.debian.org/doc/manuals/maint-guide/ Debian New Maintainer's Guide]. For the SAC middleware, the debian packaging is [https://ndpfsvn.nikhef.nl/cgi-bin/viewvc.cgi/mwsec/packaging/debian/trunk/ maintained in SVN]. Debian does not prescribe the exact method of creating packages, as is the case with RPM, and several helper libraries have emerged over time. For the SAC middleware we've chosen [http://build-common.alioth.debian.org/cdbs-doc.html CDBS], the Common Debian Build System, which is a clean and modern solution that looks like it is becoming the standard for new packages.
+
=== UMD3 specific packages ===
  
−
=== Initial steps in Debianization ===
+
Some packages, such as the <tt>ees-pepd-oh</tt> are UMD3 specific, due to their dependencies. For this a separate infrastructure has been set up. The svn commit hook automatically runs the koji build for the correct distributions.
 +
For the subsequent steps, instead of epel5 and epel6, use umd3-epel5 and umd3-epel6. The following paragraphs give brief details.
  
−
The following steps are required only once for every source packaged to be debianized. All of this could be done manually, but it is a lot of hassle and prone to errors, so better leave it to the automated tools. Should any package decide to radically change it's nature (hopefully this will never happen!) it may be required to repeat these steps.
+
==== Adding koji tags to the UMD3 builds ====
 +
component=ees-pepd-oh; version=0.1.5-2 # Example
 +
 +
for i in 5 6; do
 +
    koji move-pkg --nowait umd3-epel${i}-candidates umd3-epel${i}-testing ${component}-${version}.el${i}
 +
done
  
−
'''Required System: Debian/Ubuntu. Required tools: dh_make.'''
+
==== Signing the UMD3 builds ====
 +
The signing command is basically the same
 +
sigulsign-mwsec.py --tag=umd3-epel5-release grid_mwsec
 +
etc.
  
−
* Unpack the upstream sources.
+
==== Updating the UMD3 mash repositories ====
−
* Run <code>dh_make -r cdbs</code>
+
The mash commands are also basically the same
−
* Review the contents of the <tt>debian/</tt> subdirectory.
+
umask 0002; for dist in umd3-epel5 umd3-epel6 ; do
−
* Create a subdirectory in SVN under mwsec/packaging/debian/trunk for the new package.
+
    mash -o /mnt/mwsec/mash signed-${dist}-release
−
* add the debian directory to this subdirectory
+
    mash -o /mnt/mwsec/mash signed-${dist}-testing
−
* set the svn:MergeWithUpstream property.
+
done
  
−
(TODO: review and verify this part!)
+
==== Syncing with software.nikhef.nl for UMD3 ====
 +
The layout is similar to the standard layout, except it is in a subdirectory of mwsec:
 +
umd3/rpm/epel5 -> ../../mash/signed-umd3-epel5-release
 +
umd3/rpm/epel6 -> ../../mash/signed-umd3-epel6-release
 +
umd3/rpm/testing/epel5 -> ../../../mash/signed-umd3-epel5-testing
 +
umd3/rpm/testing/epel6 -> ../../../mash/signed-umd3-epel6-testing
 +
The syncing script automatically syncs both the standard and the UMD3 repositories.
  
−
=== Setting up pbuilder for building Debian packages ===
+
The rsync command to run from bleek.nikhef.nl is
 +
/srv/project/mwsec/rsync_software.sh
 +
or from koji-hub.testbed
 +
/mnt/mwsec/rsync_software.sh
 +
or manually from bleek for the UMD3 repositories only
 +
rsync --del -rltpgv --copy-unsafe-links /srv/project/mwsec/umd3/ software.nikhef.nl:/project/srv/www/site/software/html/dist/mwsec/umd3
  
−
Debian's pbuilder is a system for building packages in a clean environment. It can use chroot or copy-on-write. In order to set it up so it can build for multiple distributions, some [http://wiki.debian.org/PbuilderTricks enhancements to the configuration] need to be applied.
 
  
−
Because pbuilder needs to be run as root, users are allowed to run /usr/sbin/pbuilder through <tt>sudo</tt>.
+
== Generating Debian packages from the distributed source tarballs ==
−
 
 
−
The main installation is on debian.testbed.
 
−
 
 
−
To accommodate Ubuntu builds, the ubuntu-archive-keyring package was installed manually from the Ubuntu archives. Otherwise debootstrap will fail.
 
−
 
 
−
To build a package, it should suffice to go to the SVN directory <tt>mwsec/packaging/debian/trunk/''component''/</tt> and run:
 
−
DIST=stable svn-buildpackage --svn-builder pdebuild --pbuilder cowbuilder
 
−
 
 
−
The ''cowbuilder'' will use the copy-on-write root in /var/cache/pbuilder/stable-amd64/base.cow. For other distributions just change <tt>DIST=stable</tt>, e.g. <tt>DIST=oneiric</tt> for the 11.10 Ubuntu release.
 
−
 
 
−
=== Creating a repository of packages to be downloaded ===
 
−
 
 
−
Debian offers several tools for generating repositories; some are meant for full-size Debian archives and others for simple, private use. For the SAC middleware the <tt>reprepro</tt> utility offers enough features while at the same time it is easy to install and maintain.
 
−
 
 
−
Basically follow the [http://www.debian-administration.org/articles/286 guidelines on how to set up a repository]. TODO: link the pbuilder output to this.
 
−
 
 
−
Put the build results in the 'incoming' directory. Use this in <tt>$HOME/.dput.cf</tt>
 
−
[mwsec]
 
−
method = local
 
−
incoming = /var/www/repos/apt/incoming
 
−
allow_unsigned_uploads = 1
 
−
run_dinstall = 0
 
  
−
And run dput:
+
''Main article: [[SAC Debian packaging]].''
  
−
dput mwsec /var/cache/pbuilder/stable-amd64/result/lcmaps_1.4.34-1_amd64.changes
+
With every new release of a software package, the Debian packages for various supported distributions (Debian as well as Ubuntu) need to be updated.
  
−
Process the 'incoming' directory (this should be automated sometime):
+
All of the software components that have already been debianized can be updated using the following steps. Of course, any serious changes in the software need to be reviewed carefully, especially when it comes to shared library symbols.
  
−
reprepro -b /var/www/repos/apt/debian processincoming squeeze lcmaps_1.4.34-1_amd64.changes
+
These steps assume that you've set up your personal Debian toolset.
  
−
(First it may be required to set the distribution in the changes file to 'squeeze' with the <tt>changestool</tt> command).
+
* Check out svn+ssh://svn@ndpfsvn.nikhef.nl/repos/mwsec/packaging/debian/trunk/''<component>''
 +
* uscan --destdir ../tarballs
 +
* dch -v ''<new version>''-1 -D UNRELEASED "New upstream release"
 +
* svn commit -m "new upstream release"
  
−
Sync the local repository to software.nikhef.nl:
+
Building e.g. for Debian squeeze
 +
* svn-buildpackage -S
 +
* cd ../build-area/
 +
* dpkg-source -x ''<component>''_''<version>''-1.dsc
 +
* cd ''<component>-<version>''
 +
* dch -D squeeze -b -v ''<version>''-1~bpo60+1 "Rebuild for squeeze"
 +
* dpkg-buildpackage -S
 +
* cd ..
 +
* cowpoke --dist=squeeze --arch=amd64 ''<component>''_''<version>''-1~bpo60+1.dsc
 +
* cowpoke --dist=squeeze --arch=i386 --dpkg-opts=-B ''<component>''_''<version>''-1~bpo60+1.dsc
  
−
rsync -rvP /var/www/repos/apt/debian/ software.nikhef.nl:/project/srv/www/site/software/html/dist/debian/
+
Putting the results in the main repository
 +
* ssh ref-debian6-64.testbed
 +
* dput mwsec-squeeze /var/cache/pbuilder/squeeze-amd64/results/''<component>''-''<version>''-1~bpo60+1_amd64.changes
 +
* dput mwsec-squeeze /var/cache/pbuilder/squeeze-i386/results/''<component>''-''<version>''-1~bpo60+1_i386.changes
 +
* rsync -a -vP --delete /var/local/debian-repository/debian/ software.nikhef.nl:/project/srv/www/site/software/html/dist/debian/

Latest revision as of 17:37, 25 March 2020

This page contains several procedures regarding the release process of the Grid Security Middleware.

Contributing features and bug fixes

Community contributions and bug fixes are welcome through our Bugzilla service. Please contact us for custom integration procedures.

Source distribution release procedure

This is the Software release procedure for Grid Security Middleware. This page describes that what to do when your software is ready for public distribution as source tarball. These tarballs can be used by anyone who likes to do their own local builds, but they are also used to generate RPM (Red Hat) and deb (Debian) packages.

Mandatory Check list

  1. make distcheck: sh bootstrap && ./configure && make distcheck must result in "Ready for distribution"
  2. Configure.ac: Update the version of the component in the configure.ac
  3. BUGS: Updated the BUGS file in the component directory with known issues
  4. NEWS Updated the NEWS file: write release notes like information. Hint which version solves the problem.
  5. ChangeLog: a "svn update ; svn log -v > ChangeLog" for the time being.
  6. INSTALL: Update installation instruction file (if applicable)
  7. README: Update the README file with more useful information
  8. Be sure to perform an svn ci here
  9. create a tag in the version control system.

Extra info: Source repository details

The main source repository for these components is https://ndpfsvn.nikhef.nl/viewvc/mwsec/, which can be used with SVN in the following ways:

SVN with SSH public key authentication (NB. username is always svn): 

svn+ssh://svn@ndpfsvn.nikhef.nl/repos/mwsec/

SVN with https access: 

https://ndpfsvn.nikhef.nl/repos/mwsec/

SVN with http access: 

http://ndpfsvn.nikhef.nl/ro/mwsec/

Creating a tag in Grid Security Middleware

We take the example for glexec, but this will be a valid set of sets for any component:

  • Go to the trunk, and make sure it's up to date:
# For glexec:
cd mwsec/trunk
svn update
cd ..
  • Check out the tags directory for the component exclusively:
svn co --depth=empty svn+ssh://svn@ndpfsvn.nikhef.nl/repos/mwsec/tags
cd tags
svn co --depth=immediates svn+ssh://svn@ndpfsvn.nikhef.nl/repos/mwsec/tags/glexec
cd glexec
  • Copy the HEAD from trunk into the tag directory, the directory name is only named by it's version:
svn copy ../../trunk/glexec 0_8_2
cd 0_8_2
  • Make a list of the used SVN externals. Resolve the externals as local files for each tag:
svn propget svn:externals > my_externals.txt
  • Make a list of the files associated to each external and remove the externals in the current SVN directory:
svn propdel svn:externals
  • For each entry in the my_externals.txt file and each file associated to them do something like:
rm -rf src/{safefile-1.0,environ,realpath,fileutil}
svn copy ../../../trunk/cgul/{safefile-1.0,environ,realpath,fileutil} src/
rm -rf m4
for i in project/*.m4 ; do mfour=`basename $i`;svn cp --parents ../../../trunk/m4/${mfour} m4/${mfour} ; done
rm my_externals.txt
cd ..
  • (easier than the above, just for the m4 macros which is the most common case, DO NOT USE FOR gLExec and EES:)
svn propdel svn:externals
rm -rf m4
for i in project/*.m4 ; do mfour=`basename $i`;svn cp --parents ../../../trunk/m4/${mfour} m4/${mfour} ; done
  • Commit!
svn commit

Release software as tarball

The software tarballs should be released and made available for download in http://software.nikhef.nl/security. The exact procedure is as follows:

  • Make a clean/fresh check out the tag:
cd $HOME/your_clean_dir/
svn co svn+ssh://svn@ndpfsvn.nikhef.nl/repos/mwsec/tags/glexec/0_8_10 glexec_0_8_10
cd glexec_0_8_10
  • Run the ./bootstrap on fedora20.testbed
cd ${topdir}
./bootstrap
  • Run the ./configure on mwsec-el6-64.testbed
cd ${topdir}
mkdir BUILD && cd BUILD
../configure

This is required to have consistency in the versions of the autotools that are used in the distributed software.

  • Create release tarbal:
make distcheck
  • generate a SHA1 and SHA256 checksum of the tarball and place it in a file with the same name but with the .sha1 extension.
sha1sum <component>-<version>.tar.gz > <component>-<version>.tar.gz.sha1
sha256sum <component>-<version>.tar.gz > <component>-<version>.tar.gz.sha256

For lazy people: 

for FILE in `ls *.tar.gz`; do echo $FILE; sha1sum $FILE > $FILE.sha1; sha256sum $FILE > $FILE.sha256; done
  • Copy the tarball and the sha1 and sha256 checksum files to:
software.nikhef.nl:/project/srv/www/site/software/html/security/<component>/release-candidate/
  • Verify that this tarball can be used for packaging, and that the packaged software works.
  • After sufficient certification, release the software by moving it to to the parent directory, and have the release manager sign the hashes by putting them in a GPG-signed e-mail to grid-mw-security@nikhef.nl.

Generating RPM packages from the distributed tarballs

Release RPMs are build using our own koji instance, see Koji_Testbed for details on the Koji setup itself.

Note: information about how to use mock to make test-builds can be found on the Building RPMs using mock page.

The following steps need to be taken:

Update the SPEC file

  • Either check out or update the SPEC files directories from the SVN repository 
    svn co svn+ssh://svn@ndpfsvn.nikhef.nl/repos/mwsec/packaging/fedora
cd fedora/trunk

    or 

    cd fedora/trunk
svn update
  • Edit the spec file
    For a new version (the source has changed):
    • Edit the spec file by updating the Version: field to correspond to the new version.
    • Reset the Release: field to 1, but don't delete %{?dist}.
    • Update build dependencies if necessary.
    For a new release (the source has not changed):
    • Edit the spec file by increasing the Release: field by 1, but don't delete %{?dist}.
  • Add a ChangeLog entry to reflect the changes in this release, briefly. See Fedora Guidelines for guidance.
  • Check in the spec file. Use the ChangeLog entry as the commit message.

Starting the Koji build by creating a new tag for the SPEC file

  • Make sure the trunk and tags/〈component〉 directories are up-to-date: 
    cd fedora/trunk
svn update
cd ../tags/〈component〉
svn update
  • Create a new tag: 
    svn mkdir 〈tag〉
  • Copy the spec file from the trunk directory 
    svn copy ../../trunk/〈component〉.spec 〈tag〉
  • Create a new Makefile (don't forget the svn add or svn copy from an older tag. It should contain something like 
    cwd = $(abspath .)
version = $(notdir $(cwd))
component = $(notdir $(patsubst %/, %, $(dir $(cwd))))

sources:
        curl -LRO http://software.nikhef.nl/security/$(component)/$(component)-$(version).tar.gz
        sed -i -e 's/^Version: .*/Version: $(version)/' $(component).spec
  • Commit will automatically trigger a koji build: 
    svn commit 〈tag〉

Adding koji tags to the builds

Once koji is finished (see e.g. koji list-tasks or use the web-interface via http://koji-hub.testbed/koji/tasks) the results should will be tagged as candidates. When sufficiently satisfied they should be tagged as testing or release. Once they are tagged as release they should also be tagged as build, and will be used to build against (this will trigger a rebuild of the buildrepo): First 

component=glexec; version=0.9.8-1 # Example

for i in 6 7 8; do
  koji move-pkg --nowait epel${i}-candidates epel${i}-testing ${component}-${version}.el${i}
done
for i in 30; do
  koji move-pkg --nowait f${i}-candidates f${i}-testing ${component}-${version}.fc${i}
done

Moving from testing to release

Builds can be moved from testing to release using e.g. 

component=glexec; version=0.9.8-1 # Example

for i in 6 7 8; do
  koji move-pkg --nowait epel${i}-testing epel${i}-release ${component}-${version}.el${i}
  koji tag-pkg --nowait epel${i}-build ${component}-${version}.el${i}
done
for i in 30; do
  koji move-pkg --nowait f${i}-testing f${i}-release ${component}-${version}.fc${i}
  koji tag-pkg --nowait f${i}-build ${component}-${version}.fc${i}
done

Note that multiple components can be tagged using one koji tag-pkg command.

For testing it is best to untag all older tags: 

for dist in epel6 epel7 epel8 f30; do
    koji untag-pkg --non-latest ${dist}-testing ${component}
done

After this, the mash and rsync steps described below have to be run.

Signing the release RPMs with sigul

When a new build has been tagged, it needs to be signed with the GPG key 

pub   4096R/980AC8BB 2012-03-21
uid                  Nikhef Grid Security Middleware Signing Key <grid-mw-security@nikhef.nl>

which resides on sigul.testbed.

On koji-hub.testbed there is a handy script to sign all unsigned packages for a tag. 

sigulsign-mwsec.py --tag=epel5-release grid_mwsec

Due to a weird bug it may happen some packages are *not* signed; in this case retrieve the list of packages to sign 

sigulsign-mwsec.py --tag=epel5-release --just-list grid_mwsec | sed 's/\.rpm$//' > to-sign

and sign manually 

sigul sign-rpms --v3-signature --store-in-koji --koji-only grid_mwsec `cat to-sign`

The signing does not write RPMs to disk, it just imports the signatures into koji. Tell koji to write the RPMs now. 

koji write-signed-rpm 980ac8bb glexec-0.8.9-1.el5

This step is not necessary when signing with the sigulsign-mwsec.py script.

Updating the mash repositories

When a new build has been tagged into either release and/or testing, the corresponding repositories need to be recreated using mash. This currently has to be done on koji-hub.testbed. NOTE: any missing signed rpms (see the above procedure) will result in mash failure.

  • Set correct umask and run mash on all (or the relevant) repositories: 
    umask 0002; for dist in epel6 epel7 epel8 f30; do
    mash -o /mnt/mwsec/mash signed-${dist}-release
    mash -o /mnt/mwsec/mash signed-${dist}-testing
done

Syncing with software.nikhef.nl

The repository on http://software.nikhef.nl/dist/mwsec/rpm/ is rsync-ed from bleek.nikhef.nl:/srv/project/mwsec/rpm/ which contains only symlinks to the mash directories. This is necessary since mash cannot easily handle the combination epel5 testing/epel5. The directory layout is 

rpm/epel5 -> ../mash/signed-epel5-release
rpm/epel6 -> ../mash/signed-epel6-release
rpm/epel7 -> ../mash/signed-epel7-release
rpm/epel8 -> ../mash/signed-epel8-release
rpm/fc16 -> ../mash/fc16-release (out of support)
rpm/f17 -> ../mash/signed-f17-release (out of support)
rpm/fc17 -> f17 (original name retained, out of support)
rpm/f18 -> ../mash/signed-f18-release (out of support)
rpm/f19 -> ../mash/signed-f19-release (out of support)
rpm/f20 -> ../mash/signed-f20-release (out of support)
rpm/f21 -> ../mash/signed-f21-release (out of support)
rpm/f22 -> ../mash/signed-f22-release (out of support)
rpm/f23 -> ../mash/signed-f23-release (out of support)
rpm/f24 -> ../mash/signed-f24-release (out of support)
rpm/f25 -> ../mash/signed-f25-release (out of support)
rpm/f26 -> ../mash/signed-f26-release (out of support)
rpm/f27 -> ../mash/signed-f27-release (out of support)
rpm/f28 -> ../mash/signed-f28-release (out of support)
rpm/f29 -> ../mash/signed-f29-release (out of support)
rpm/f30 -> ../mash/signed-f30-release
rpm/testing/epel5 -> ../../mash/signed-epel5-testing
rpm/testing/epel6 -> ../../mash/signed-epel6-testing
rpm/testing/epel7 -> ../../mash/signed-epel7-testing
rpm/testing/epel8 -> ../../mash/signed-epel8-testing
rpm/testing/fc16 -> ../../mash/fc16-testing (out of support)
rpm/testing/f17 -> ../../mash/fc17-testing (out of support)
rpm/testing/fc17 -> f17 (original name retained, out of support)
rpm/testing/f18 -> ../../mash/signed-fc18-testing (out of support)
rpm/testing/f19 -> ../../mash/signed-fc19-testing (out of support)
rpm/testing/f20 -> ../../mash/signed-fc20-testing (out of support)
rpm/testing/f21 -> ../../mash/signed-fc21-testing (out of support)
rpm/testing/f22 -> ../../mash/signed-fc22-testing (out of support)
rpm/testing/f23 -> ../../mash/signed-fc23-testing (out of support)
rpm/testing/f24 -> ../../mash/signed-fc24-testing (out of support)
rpm/testing/f25 -> ../../mash/signed-fc25-testing (out of support)
rpm/testing/f26 -> ../../mash/signed-fc26-testing (out of support)
rpm/testing/f27 -> ../../mash/signed-fc27-testing (out of support)
rpm/testing/f28 -> ../../mash/signed-fc28-testing (out of support)
rpm/testing/f29 -> ../../mash/signed-fc29-testing (out of support)
rpm/testing/f30 -> ../../mash/signed-fc30-testing

The rsync command to run from bleek.nikhef.nl is 

/srv/project/mwsec/rsync_software.sh

or from koji-hub.testbed 

/mnt/mwsec/rsync_software.sh

or manually from bleek.nikhef.nl 

rsync --exclude=epel5/SPECS\* --del -rltpgv --copy-unsafe-links /srv/project/mwsec/rpm/ software.nikhef.nl:/project/srv/www/site/software/html/dist/mwsec/rpm

'NOTE': the / after the first rpm/ is necessary. Alternatively chdir to /srv/project/mwsec/rpm/ and use . instead.

Notes:

  • RPMs will reside in either testing or the normal repository but not in both.
  • mash will copy RPMs from the koji builds found in /mnt/koji/packages (an NFS mount from put.testbed).
  • mash will copy noarch RPMs to both x86_64 and i386 (cf. the official EPEL repository).

Using the repository in a YUM repo file requires something like the following: 

[mwsec]
name=Middleware Security Software
baseurl=http://software.nikhef.nl/dist/mwsec/rpm/epel5/$basearch
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-MWSEC

[mwsec-testing]
name=Middleware Security Software testing
baseurl=http://software.nikhef.nl/dist/mwsec/rpm/testing/epel5/$basearch
enabled=0
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-MWSEC

UMD3 specific packages

Some packages, such as the ees-pepd-oh are UMD3 specific, due to their dependencies. For this a separate infrastructure has been set up. The svn commit hook automatically runs the koji build for the correct distributions. For the subsequent steps, instead of epel5 and epel6, use umd3-epel5 and umd3-epel6. The following paragraphs give brief details.

Adding koji tags to the UMD3 builds

component=ees-pepd-oh; version=0.1.5-2 # Example

for i in 5 6; do
    koji move-pkg --nowait umd3-epel${i}-candidates umd3-epel${i}-testing ${component}-${version}.el${i}
done

Signing the UMD3 builds

The signing command is basically the same 

sigulsign-mwsec.py --tag=umd3-epel5-release grid_mwsec

etc.

Updating the UMD3 mash repositories

The mash commands are also basically the same 

umask 0002; for dist in umd3-epel5 umd3-epel6 ; do
    mash -o /mnt/mwsec/mash signed-${dist}-release
    mash -o /mnt/mwsec/mash signed-${dist}-testing
done

Syncing with software.nikhef.nl for UMD3

The layout is similar to the standard layout, except it is in a subdirectory of mwsec: 

umd3/rpm/epel5 -> ../../mash/signed-umd3-epel5-release
umd3/rpm/epel6 -> ../../mash/signed-umd3-epel6-release
umd3/rpm/testing/epel5 -> ../../../mash/signed-umd3-epel5-testing
umd3/rpm/testing/epel6 -> ../../../mash/signed-umd3-epel6-testing

The syncing script automatically syncs both the standard and the UMD3 repositories.

The rsync command to run from bleek.nikhef.nl is 

/srv/project/mwsec/rsync_software.sh

or from koji-hub.testbed 

/mnt/mwsec/rsync_software.sh

or manually from bleek for the UMD3 repositories only 

rsync --del -rltpgv --copy-unsafe-links /srv/project/mwsec/umd3/ software.nikhef.nl:/project/srv/www/site/software/html/dist/mwsec/umd3


Generating Debian packages from the distributed source tarballs

Main article: SAC Debian packaging.

With every new release of a software package, the Debian packages for various supported distributions (Debian as well as Ubuntu) need to be updated.

All of the software components that have already been debianized can be updated using the following steps. Of course, any serious changes in the software need to be reviewed carefully, especially when it comes to shared library symbols.

These steps assume that you've set up your personal Debian toolset.

Building e.g. for Debian squeeze

  • svn-buildpackage -S
  • cd ../build-area/
  • dpkg-source -x <component>_<version>-1.dsc
  • cd <component>-<version>
  • dch -D squeeze -b -v <version>-1~bpo60+1 "Rebuild for squeeze"
  • dpkg-buildpackage -S
  • cd ..
  • cowpoke --dist=squeeze --arch=amd64 <component>_<version>-1~bpo60+1.dsc
  • cowpoke --dist=squeeze --arch=i386 --dpkg-opts=-B <component>_<version>-1~bpo60+1.dsc

Putting the results in the main repository

  • ssh ref-debian6-64.testbed
  • dput mwsec-squeeze /var/cache/pbuilder/squeeze-amd64/results/<component>-<version>-1~bpo60+1_amd64.changes
  • dput mwsec-squeeze /var/cache/pbuilder/squeeze-i386/results/<component>-<version>-1~bpo60+1_i386.changes
  • rsync -a -vP --delete /var/local/debian-repository/debian/ software.nikhef.nl:/project/srv/www/site/software/html/dist/debian/
Retrieved from "https://wiki.nikhef.nl/grid/index.php?title=SAC_software_procedures&oldid=10884"