Difference between revisions of "SAC software procedures"
| Line 149: | Line 149: | ||
<pre>svn mkdir <tag></pre> | <pre>svn mkdir <tag></pre> | ||
<li> Copy the spec file from the trunk directory | <li> Copy the spec file from the trunk directory | ||
| − | <pre>svn copy ../trunk/<component>.spec <tag></pre> | + | <pre>svn copy ../../trunk/<component>.spec <tag></pre> |
<li> Create a new Makefile (don't forget the <tt>svn add</tt> or <tt>svn copy</tt> from an older tag. It should contain something like | <li> Create a new Makefile (don't forget the <tt>svn add</tt> or <tt>svn copy</tt> from an older tag. It should contain something like | ||
<pre> | <pre> | ||
Revision as of 11:14, 22 March 2013
This page contains several procedures regarding the release process of the Grid Security Middleware.
Source distribution release procedure
This is the Software release procedure for Grid Security Middleware. This page describes that what to do when your software is ready for public distribution as source tarball. These tarballs can be used by anyone who likes to do their own local builds, but they are also used to generate RPM (Red Hat) and deb (Debian) packages.
Mandatory Check list
- make distcheck: sh bootstrap && ./configure && make distcheck must result in "Ready for distribution"
- Configure.ac: Update the version of the component in the configure.ac
- BUGS: Updated the BUGS file in the component directory with known issues
- NEWS Updated the NEWS file: write release notes like information. Hint which version solves the problem.
- ChangeLog: a "svn update ; svn log -v > ChangeLog" for the time being.
- INSTALL: Update installation instruction file (if applicable)
- README: Update the README file with more useful information
- Be sure to perform an svn ci here
- create a tag in the version control system.
Extra info: Source repository details
The main source repository for these components is https://ndpfsvn.nikhef.nl/viewvc/mwsec/, which can be used with SVN in the following ways:
SVN with SSH public key authentication (NB. username is always svn):
svn+ssh://svn@ndpfsvn.nikhef.nl/repos/mwsec/
SVN with https access:
https://ndpfsvn.nikhef.nl/repos/mwsec/
SVN with http access:
http://ndpfsvn.nikhef.nl/ro/mwsec/
Creating a tag in Grid Security Middleware
We take the example for glexec, but this will be a valid set of sets for any component:
- Go to the trunk, and make sure it's up to date:
# For glexec: cd mwsec/trunk svn update cd ..
- Check out the tags directory for the component exclusively:
svn co --depth=empty svn+ssh://svn@ndpfsvn.nikhef.nl/repos/mwsec/tags cd tags svn co --depth=immediates svn+ssh://svn@ndpfsvn.nikhef.nl/repos/mwsec/tags/glexec cd glexec
- Copy the HEAD from trunk into the tag directory, the directory name is only named by it's version:
svn copy ../../trunk/glexec 0_8_2 cd 0_8_2
- Make a list of the used SVN externals. Resolve the externals as local files for each tag:
svn propget svn:externals > my_externals.txt
- Make a list of the files associated to each external and remove the externals in the current SVN directory:
svn propdel svn:externals
- For each entry in the my_externals.txt file and each file associated to them do something like:
rm -rf src/{safefile-1.0,environ,realpath,fileutil}
svn copy ../../../trunk/cgul/{safefile-1.0,environ,realpath,fileutil} src/
rm -rf m4
for i in project/*.m4 ; do mfour=`basename $i`;svn cp --parents ../../../trunk/m4/${mfour} m4/${mfour} ; done
rm my_externals.txt
cd ..
- (easier than the above, just for the m4 macros which is the most common case, DO NOT USE FOR gLExec and EES:)
svn propdel svn:externals
rm -rf m4
for i in project/*.m4 ; do mfour=`basename $i`;svn cp --parents ../../../trunk/m4/${mfour} m4/${mfour} ; done
- Commit!
svn commit
Release software as tarball
The software tarballs should be released and made available for download in http://software.nikhef.nl/security. The exact procedure is as follows:
- Make a clean/fresh check out the tag:
cd $HOME/your_clean_dir/ svn co svn+ssh://svn@ndpfsvn.nikhef.nl/repos/mwsec/tags/glexec/0_8_10 glexec_0_8_10 cd glexec_0_8_10
- Run the ./bootstrap on fc17.testbed
cd ${topdir}
./bootstrap
- Run the ./configure on mwsecbuild.testbed
cd ${topdir}
mkdir BUILD && cd BUILD
../configure
This is required to have consistency in the versions of the autotools that are used in the distributed software.
- Create release tarbal:
make distcheck
- generate a SHA1 and SHA256 checksum of the tarball and place it in a file with the same name but with the .sha1 extension.
sha1sum <component>-<version>.tar.gz > <component>-<version>.tar.gz.sha1 sha256sum <component>-<version>.tar.gz > <component>-<version>.tar.gz.sha256
For lazy people:
for FILE in `ls *.tar.gz`; do echo $FILE; sha1sum $FILE > $FILE.sha1; sha256sum $FILE > $FILE.sha256; done
- Copy the tarball and the sha1 and sha256 checksum files to:
software.nikhef.nl:/project/srv/www/site/software/html/security/<component>/release-candidate/
- Verify that this tarball can be used for packaging, and that the packaged software works.
- After sufficient certification, release the software by moving it to to the parent directory, and have the release manager sign the hashes by putting them in a GPG-signed e-mail to grid-mw-security@nikhef.nl.
Generating RPM packages from the distributed tarballs
Release RPMs are build using our own koji instance, see Koji_Testbed for details on the Koji setup itself.
Note: information about how to use mock to make test-builds can be found on the Building RPMs using mock page.
The following steps need to be taken:
- update the SPEC file,
- Starting the Koji build by creating a new tag for the SPEC file,
- Adding koji tags to the builds,
- Updating the mash repositories,
- Syncing with software.nikhef.nl.
Update the SPEC file
- Either check out or update the SPEC files directories from the SVN repository
svn co svn+ssh://svn@ndpfsvn.nikhef.nl/repos/mwsec/packaging/fedora cd fedora/trunk
or
cd fedora/trunk svn update
- Edit the spec file
For a new version (the source has changed):- Edit the spec file by updating the Version: field to correspond to the new version.
- Reset the Release: field to 1, but don't delete
%{?dist}. - Update build dependencies if necessary.
- Edit the spec file by increasing the Release: field by 1, but don't delete
%{?dist}.
- Add a ChangeLog entry to reflect the changes in this release, briefly. See Fedora Guidelines for guidance.
- Check in the spec file. Use the ChangeLog entry as the commit message.
Starting the Koji build by creating a new tag for the SPEC file
- Make sure the trunk and tags/<component> directories are up-to-date:
cd fedora/trunk svn update cd ../tags/<component> svn update
- Create a new tag:
svn mkdir <tag>
- Copy the spec file from the trunk directory
svn copy ../../trunk/<component>.spec <tag>
- Create a new Makefile (don't forget the svn add or svn copy from an older tag. It should contain something like
dir = $(shell pwd) version = $(notdir $(dir)) component = glexec sources: curl -LRO http://software.nikhef.nl/security/$(component)/$(component)-$(version).tar.gz sed -i -e 's/^Version: .*/Version: $(version)/' $(component).spec - Commit will automatically trigger a koji build:
svn commit
Adding koji tags to the builds
Once koji is finished (see e.g. koji list-tasks or use the web-interface via http://koji-hub.testbed/koji/tasks) the results should be tagged as release or testing: Either
component=glexec; version=0.9.8-1 # Example
koji tag-pkg --nowait dist-epel5-testing ${component}-${version}.el5
koji tag-pkg --nowait dist-epel6-testing ${component}-${version}.el6
koji tag-pkg --nowait dist-fc16-testing ${component}-${version}.fc16
koji tag-pkg --nowait dist-fc17-testing ${component}-${version}.fc17
or
component=glexec; version=0.9.8-1 # Example
koji tag-pkg --nowait dist-epel5-release ${component}-${version}.el5
koji tag-pkg --nowait dist-epel6-release ${component}-${version}.el6
koji tag-pkg --nowait dist-fc16-release ${component}-${version}.fc16
koji tag-pkg --nowait dist-fc17-release ${component}-${version}.fc17
Note that multiple components can be tagged using one koji tag-pkg command.
For testing it is necessary to untag all older tags:
for dist in epel5 epel6 fc16 fc17 ; do
koji untag-pkg --non-latest dist-${dist}-testing ${component}
done
Moving from testing to release
Builds can be moved from testing to release using e.g.
koji move-pkg dist-epel5-testing dist-epel5-release lcmaps-1.5.7-1.el5
etc. After this, the mash and rsync steps described below have to be run.
Signing the release RPMs with sigul
When a new build has been tagged, it needs to be signed with the GPG key
pub 4096R/980AC8BB 2012-03-21 uid Nikhef Grid Security Middleware Signing Key <grid-mw-security@nikhef.nl>
which resides on sigul.testbed.
On koji-hub.testbed there is a handy script to sign all unsigned packages for a tag.
sigulsign-mwsec.py --tag=dist-epel5-release --just-sign grid_mwsec
Due to a weird bug it may happen some packages are *not* signed; in this case retrieve the list of packages to sign
sigulsign-mwsec.py --tag=dist-epel5-release --just-list grid_mwsec | sed 's/\.rpm$//' > to-sign
and sign manually
sigul sign-rpms --v3-signature --store-in-koji --koji-only grid_mwsec `cat to-sign`
The signing does not write RPMs to disk, it just imports the signatures into koji. Tell koji to write the RPMs now.
koji write-signed-rpm 980ac8bb glexec-0.8.9-1.el5
Updating the mash repositories
When a new build has been tagged into either release and/or testing, the corresponding repositories need to be recreated using mash. This currently has to be done on koji-hub.testbed. NOTE: at the moment mash is not yet configured to generate signed repositories, but we're going to switch real soon now and from that moment on any missing signed rpms (see the above procedure) will result in mash failure.
- Set correct umask:
umask 0002
- Run mash on all (or the relevant) repositories:
for dist in epel5 epel6 fc16 fc17 ; do mash -o /mnt/mwsec/mash ${dist}-release mash -o /mnt/mwsec/mash ${dist}-testing done
Syncing with software.nikhef.nl
The repository on http://software.nikhef.nl/dist/mwsec/rpm/ is rsync-ed from bleek.nikhef.nl:/srv/project/mwsec/rpm/ which contains only symlinks to the mash directories. This is necessary since mash cannot easily handle the combination epel5 testing/epel5. The directory layout is
rpm/epel5 -> ../mash/epel5-release rpm/epel6 -> ../mash/epel6-release rpm/fc16 -> ../mash/fc16-release rpm/fc17 -> ../mash/fc17-release rpm/testing/epel5 -> ../../mash/epel5-testing rpm/testing/epel6 -> ../../mash/epel6-testing rpm/testing/fc16 -> ../../mash/fc16-testing rpm/testing/fc17 -> ../../mash/fc17-testing
The rsync command to run from bleek.nikhef.nl is
/srv/project/mwsec/rsync_software.sh
or run manually
rsync --exclude=epel5/SPECS\* --del -rltpgv --copy-unsafe-links /srv/project/mwsec/rpm/ software.nikhef.nl:/project/srv/www/site/software/html/dist/mwsec/rpm
'NOTE': the / after the first rpm/ is necessary. Alternatively chdir to /srv/project/mwsec/rpm/ and use . instead.
Notes:
- RPMs will reside in either testing or the normal repository but not in both.
- mash will copy RPMs from the koji builds found in /mnt/koji/packages (an NFS mount from put.testbed).
- mash will copy noarch RPMs to both x86_64 and i386 (cf. the official EPEL repository).
Using the repository in a YUM repo file requires something like the following:
[mwsec] name=Middleware Security Software baseurl=http://software.nikhef.nl/dist/mwsec/rpm/epel5/$basearch gpgcheck=0 [mwsec-testing] name=Middleware Security Software testing baseurl=http://software.nikhef.nl/dist/mwsec/rpm/testing/epel5/$basearch enabled=0 gpgcheck=0
Generating Debian packages from the distributed source tarballs
Main article: SAC Debian packaging.
With every new release of a software package, the Debian packages for various supported distributions (Debian as well as Ubuntu) need to be updated.
All of the software components that have already been debianized can be updated using the following steps. Of course, any serious changes in the software need to be reviewed carefully, especially when it comes to shared library symbols.
These steps assume that you've set up your personal Debian toolset.
- Check out svn+ssh://svn@ndpfsvn.nikhef.nl/repos/mwsec/packaging/debian/trunk/<component>
- uscan --destdir ../tarballs
- dch -v <new version>-1 -D UNRELEASED "New upstream release"
- svn commit -m "new upstream release"
Building e.g. for Debian squeeze
- svn-buildpackage -S
- cd ../build-area/
- dpkg-source -x <component>_<version>-1.dsc
- cd <component>-<version>
- dch -D squeeze -b -v <version>-1~bpo60+1 "Rebuild for squeeze"
- dpkg-buildpackage -S
- cd ..
- cowpoke --dist=squeeze --arch=amd64 <component>_<version>-1~bpo60+1.dsc
- cowpoke --dist=squeeze --arch=i386 --dpkg-opts=-B <component>_<version>-1~bpo60+1.dsc
Putting the results in the main repository
- ssh ref-debian6-64.testbed
- dput mwsec-squeeze /var/cache/pbuilder/squeeze-amd64/results/<component>-<version>-1~bpo60+1_amd64.changes
- dput mwsec-squeeze /var/cache/pbuilder/squeeze-i386/results/<component>-<version>-1~bpo60+1_i386.changes
- rsync -a -vP --delete /var/local/debian-repository/debian/ software.nikhef.nl:/project/srv/www/site/software/html/dist/debian/
