SAC software procedures

This page contains several procedures regarding the release process of the Grid Security Middleware.

Source distribution release procedure

This is the Software release procedure for Grid Security Middleware. This page describes that what to do when your software is ready for public distribution as source tarball. These tarballs can be used by anyone who likes to do their own local builds, but they are also used to generate RPM (Red Hat) and deb (Debian) packages.

Mandatory Check list

1. make distcheck: sh bootstrap && ./configure && make distcheck must result in "Ready for distribution"
2. Configure.ac: Update the version of the component in the configure.ac
3. BUGS: Updated the BUGS file in the component directory with known issues
4. NEWS Updated the NEWS file: write release notes like information. Hint which version solves the problem.
5. ChangeLog: a "svn update ; svn log -v > ChangeLog" for the time being.
6. INSTALL: Update installation instruction file (if applicable)
7. README: Update the README file with more useful information
8. Be sure to perform an svn ci here
9. create a tag in the version control system.

Extra info: Source repository details

The main source repository for these components is https://ndpfsvn.nikhef.nl/viewvc/mwsec/, which can be used with SVN in the following ways:

SVN with SSH public key authentication (NB. username is always svn):

svn+ssh://svn@ndpfsvn.nikhef.nl/repos/mwsec/

SVN with https access:

https://ndpfsvn.nikhef.nl/repos/mwsec/

SVN with http access:

http://ndpfsvn.nikhef.nl/ro/mwsec/

Creating a tag in Grid Security Middleware

We take the example for glexec, but this will be a valid set of sets for any component:

• Go to the trunk, and make sure it's up to date:

# For glexec:
cd mwsec/trunk
svn update
cd ..

• Check out the tags directory for the component exclusively:

svn co --depth=empty svn+ssh://svn@ndpfsvn.nikhef.nl/repos/mwsec/tags
cd tags
svn co --depth=immediates svn+ssh://svn@ndpfsvn.nikhef.nl/repos/mwsec/tags/glexec
cd glexec

• Copy the HEAD from trunk into the tag directory, the directory name is only named by it's version:

svn copy ../../trunk/glexec 0_8_2
cd 0_8_2

• Make a list of the used SVN externals. Resolve the externals as local files for each tag:

svn propget svn:externals > my_externals.txt

• Make a list of the files associated to each external and remove the externals in the current SVN directory:

svn propdel svn:externals

• For each entry in the my_externals.txt file and each file associated to them do something like:

rm -rf src/{safefile-1.0,environ,realpath,fileutil}
svn copy ../../../trunk/cgul/{safefile-1.0,environ,realpath,fileutil} src/
rm -rf m4
for i in project/*.m4 ; do mfour=`basename $i`;svn cp --parents ../../../trunk/m4/${mfour} m4/${mfour} ; done
rm my_externals.txt
cd ..

• (easier than the above, just for the m4 macros which is the most common case, DO NOT USE FOR gLExec and EES:)

svn propdel svn:externals
rm -rf m4
for i in project/*.m4 ; do mfour=`basename $i`;svn cp --parents ../../../trunk/m4/${mfour} m4/${mfour} ; done

• Commit!

svn commit

Release software as tarball

The software tarballs should be released and made available for download in http://software.nikhef.nl/security. The exact procedure is as follows:

• Make a clean/fresh check out the tag:

cd $HOME/your_clean_dir/
svn co svn+ssh://svn@ndpfsvn.nikhef.nl/repos/mwsec/tags/glexec/0_8_10 glexec_0_8_10
cd glexec_0_8_10

• Run the ./bootstrap on fc17.testbed

cd ${topdir}
./bootstrap

• Run the ./configure on mwsecbuild.testbed

cd ${topdir}
mkdir BUILD && cd BUILD
../configure

This is required to have consistency in the versions of the autotools that are used in the distributed software.

• Create release tarbal:

make distcheck

• generate a SHA1 and SHA256 checksum of the tarball and place it in a file with the same name but with the .sha1 extension.

sha1sum <component>-<version>.tar.gz > <component>-<version>.tar.gz.sha1
sha256sum <component>-<version>.tar.gz > <component>-<version>.tar.gz.sha256

For lazy people:

for FILE in `ls *.tar.gz`; do echo $FILE; sha1sum $FILE > $FILE.sha1; sha256sum $FILE > $FILE.sha256; done

• Copy the tarball and the sha1 and sha256 checksum files to:

software.nikhef.nl:/project/srv/www/site/software/html/security/<component>/release-candidate/

• Verify that this tarball can be used for packaging, and that the packaged software works.
• After sufficient certification, release the software by moving it to to the parent directory, and have the release manager sign the hashes by putting them in a GPG-signed e-mail to grid-mw-security@nikhef.nl.

Generating RPM packages from the distributed tarballs

Note: this section describes the now-standard method using koji, how to use mock can be found on the Building RPMs using mock page.

Procedure for building a new version or building a new release

The following steps need to be taken:

• update the SPEC file,
• tagging the SPEC file,
• #Updating the mash repositories.

Update the SPEC file

• Check out or update the SPEC files directories from the SVN repository

  svn co svn+ssh://svn@ndpfsvn.nikhef.nl/repos/mwsec/packaging/fedora

• Go the

  trunk

  directory.

• svn update

• Edit the spec file
  For a new version (the source has changed):
  • Edit the spec file by updating the Version: field to correspond to the new version.
  • Reset the Release: field to 1, but don't delete %{?dist}.
  • Update build dependencies if necessary.
  For a new release (the source has not changed):
  • Edit the spec file by increasing the Release: field by 1, but don't delete %{?dist}.
• Add a ChangeLog entry to reflect the changes in this release, briefly. See Fedora Guidelines for guidance.
• Check in the spec file. Use the ChangeLog entry as the commit message.

Creating a new tag for the SPEC file / starting Koji build

• Go to the packaging/fedora/tags/<component> directory
• Update svn:

  svn update

• Create a new tag:

  svn mkdir <tag>

• Copy the spec file from the trunk directory

  svn copy ../trunk/<component>.spec <tag>

• Create a new Makefile (don't forget the svn add or svn copy from an older tag. It should contain something like

  dir = $(shell pwd)
  version = $(notdir $(dir))
  component = glexec
  
  sources:
          curl -O http://software.nikhef.nl/security/$(component)/$(component)-$(version).tar.gz
          sed -i -e 's/^Version: .*/Version: $(version)/' $(component).spec
  

• Commit will automatically trigger a koji build:

  svn commit

  Adding a new component and/or build target

  This procedure is not yet worked out.

  Distribution release of RPMS

  Warning: this procedure is not yet finalised.

  The RPMS should all be reachable on bleek.nikhef.nl at this time.

  The generated RPMS should be distributed in a public, yum-installable repository. The repository is composed on bleek.nikhef.nl:/srv/project/mwsec/rpm with the following lay-out:

  ./epel5
  ./epel5/SRPMS
  ./epel5/SRPMS/repodata
  ./epel5/i386
  ./epel5/i386/repodata
  ./epel5/i386/debug
  ./epel5/x86_64
  ./epel5/x86_64/repodata
  ./epel5/x86_64/debug
  ./testing
  ./testing/epel5
  ./testing/epel5/i386
  ./testing/epel5/i386/debug
  ./testing/epel5/i386/repodata
  ./testing/epel5/x86_64
  ./testing/epel5/x86_64/debug
  ./testing/epel5/x86_64/repodata
  

  and likewise for epel6, fc16 and fc17.

  • RPMs will reside in either testing or the normal repository but not in both.
  • Older RPMs are symlinked from the mock directories in /srv/project/mock/rhel5/.
    Newer RPMs will be copied from the koji builds found in /mnt/koji/dists/ (an NFS mount from put.testbed).
  • noarch RPMs are symlinked (mock) or copied (koji) to both x86_64 and i386 (cf. the EPEL repository).

  The layout of mock is

  /srv/project/mock
  /srv/project/mock/rhel5
  /srv/project/mock/rhel5/SRPMS
  /srv/project/mock/rhel5/i386
  /srv/project/mock/rhel5/noarch
  /srv/project/mock/rhel5/x86_64
  

  After updating the el5 and/or testing directories, update the repodata:

  umask 002
  createrepo /srv/project/mwsec/rpm/epel5/SRPMS
  createrepo /srv/project/mwsec/rpm/epel5/i386
  createrepo /srv/project/mwsec/rpm/epel5/x86_64
  createrepo /srv/project/mwsec/rpm/testing/epel5/SRPMS
  createrepo /srv/project/mwsec/rpm/testing/epel5/i386
  createrepo /srv/project/mwsec/rpm/testing/epel5/x86_64
  

  and likewise for the other distributions. Make sure the file permissions are ok:

  chmod -R ug+w,a+r /srv/project/mwsec
  

  After this step, the repositories need to be synced with software.nikhef.nl:

  rsync --exclude=rpm/epel5/SPECS\* --del -rltpgv --copy-unsafe-links /srv/project/mwsec/rpm software.nikhef.nl:/project/srv/www/site/software/html/dist/mwsec/
  

  Using this repository in a YUM repo file requires something like the following:

  [mwsec]
  name=Middleware Security Software
  baseurl=http://software.nikhef.nl/dist/mwsec/rpm/epel5/$basearch
  gpgcheck=0
  
  [mwsec-testing]
  name=Middleware Security Software testing
  baseurl=http://software.nikhef.nl/dist/mwsec/rpm/testing/epel5/$basearch
  enabled=0
  gpgcheck=0
  

  Generating Debian packages from the distributed source tarballs

  Main article: SAC Debian packaging.

  With every new release of a software package, the Debian packages for various supported distributions (Debian as well as Ubuntu) need to be updated.

  All of the software components that have already been debianized can be updated using the following steps. Of course, any serious changes in the software need to be reviewed carefully, especially when it comes to shared library symbols.

  These steps assume that you've set up your personal Debian toolset.

  • Check out svn+ssh://svn@ndpfsvn.nikhef.nl/repos/mwsec/packaging/debian/trunk/<component>
  • uscan --destdir ../tarballs
  • dch -v <new version>-1 -D UNRELEASED "New upstream release"
  • svn commit -m "new upstream release"

  Building e.g. for Debian squeeze

  • svn-buildpackage -S
  • cd ../build-area/
  • dpkg-source -x <component>_<version>-1.dsc
  • cd <component>-<version>
  • dch -D squeeze -b -v <version>-1~bpo60+1 "Rebuild for squeeze"
  • dpkg-buildpackage -S
  • cd ..
  • cowpoke --dist=squeeze --arch=amd64 <component>_<version>-1~bpo60+1.dsc
  • cowpoke --dist=squeeze --arch=i386 --dpkg-opts=-B <component>_<version>-1~bpo60+1.dsc

  Putting the results in the main repository

  • ssh ref-debian6-64.testbed
  • dput mwsec-squeeze /var/cache/pbuilder/squeeze-amd64/results/<component>-<version>-1~bpo60+1_amd64.changes
  • dput mwsec-squeeze /var/cache/pbuilder/squeeze-i386/results/<component>-<version>-1~bpo60+1_i386.changes
  • rsync -a -vP --delete /var/local/debian-repository/debian/ software.nikhef.nl:/project/srv/www/site/software/html/dist/debian/