SAC software procedures

From PDP/Grid Wiki
Jump to navigationJump to search

This page contains several procedures regarding the release process of the Grid Security Middleware.

Source distribution release procedure

This is the Software release procedure for Grid Security Middleware. This page describes that what to do when your software is ready for public distribution as source tarball. These tarballs can be used by anyone who likes to do their own local builds, but they are also used to generate RPM (Red Hat) and deb (Debian) packages.

Mandatory Check list

  1. make distcheck: sh bootstrap && ./configure && make distcheck must result in "Ready for distribution"
  2. Configure.ac: Update the version of the component in the configure.ac
  3. BUGS: Updated the BUGS file in the component directory with known issues
  4. NEWS Updated the NEWS file: write release notes like information. Hint which version solves the problem.
  5. ChangeLog: a "svn log > ChangeLog" for the time being.
  6. INSTALL: Update installation instruction file (if applicable)
  7. README: Update the README file with more useful information
  8. Be sure to perform an svn ci here
  9. create a tag in the version control system.

Extra info: Source repository details

The main source repository for these components is https://ndpfsvn.nikhef.nl/cgi-bin/viewvc.cgi/mwsec/, which can be used with SVN in the following ways:

SVN with SSH public key authentication (NB. username is always svn): 

svn+ssh://svn@ndpfsvn.nikhef.nl/repos/mwsec/

SVN with https access: 

https://ndpfsvn.nikhef.nl/repos/mwsec/

SVN with http access: 

http://ndpfsvn.nikhef.nl/ro/mwsec/

Creating a tag in Grid Security Middleware

We take the example for glexec, but this will be a valid set of sets for any component:

  • Go to the trunk, and make sure it's up to date:
# For glexec:
cd mwsec/trunk
svn update
cd ..
  • Check out the tags directory for the component exclusively:
svn co --depth=empty svn+ssh://svn@ndpfsvn.nikhef.nl/repos/mwsec/tags
cd tags
svn co --depth=immediates svn+ssh://svn@ndpfsvn.nikhef.nl/repos/mwsec/tags/glexec
cd glexec
  • Copy the HEAD from trunk into the tag directory, the directory name is only named by it's version:
svn copy ../../trunk/glexec 0_8_2
cd 0_8_2
  • Make a list of the used SVN externals. Resolve the externals as local files for each tag:
svn propget svn:externals > my_externals.txt
  • Make a list of the files associated to each external and remove the externals in the current SVN directory:
svn propdel svn:externals
  • For each entry in the my_externals.txt file and each file associated to them do something like:
rm -rf src/{safefile-1.0,environ,realpath,fileutil}
svn copy ../../../trunk/cgul/{safefile-1.0,environ,realpath,fileutil} src/
rm -rf m4
for i in project/*.m4 ; do mfour=`basename $i`;svn cp --parents ../../../trunk/m4/${mfour} m4/${mfour} ; done
rm my_externals.txt
cd ..
  • (easier than the above, just for the m4 macros which is the most common case, DO NOT USE FOR gLExec and EES:)
svn propdel svn:externals
rm -rf m4
for i in project/*.m4 ; do mfour=`basename $i`;svn cp --parents ../../../trunk/m4/${mfour} m4/${mfour} ; done
  • Commit!
svn commit

Release software as tarball

The software tarballs should be released and made available for download in http://software.nikhef.nl/security. The exact procedure is as follows on span/bleek:

  • Make a clean/fresh check out the tag:
cd $HOME/your_clean_dir/
svn co svn+ssh://svn@ndpfsvn.nikhef.nl/repos/mwsec/tags/glexec/0_8_10 glexec_0_8_10
cd glexec_0_8_10
  • Run the ./bootstrap on fc14.testbed
cd ${topdir}
./bootstrap
  • Run the ./configure on mwsecbuild.testbed
cd ${topdir}
mkdir BUILD
../configure

This is required to have consistency in the versions of the autotools that are used in the distributed software.

  • Create release tarbal:
make distcheck
  • generate a SHA1 and SHA256 checksum of the tarball and place it in a file with the same name but with the .sha1 extension.
sha1sum <component>-<version>.tar.gz > <component>-<version>.tar.gz.sha1
sha256sum <component>-<version>.tar.gz > <component>-<version>.tar.gz.sha256
  • Copy the tarball and the sha1 and sha256 checksum files to:
software.nikhef.nl:/project/srv/www/site/software/html/security/<component>/release-candidate/
  • Verify that this tarball can be used for packaging, and that the packaged software works.
  • After sufficient certification, release the software by moving it to to the parent directory, and have the release manager sign the hashes by putting them in a GPG-signed e-mail to grid-mw-security@nikhef.nl.

Generating RPM packages from the distributed tarballs

The creation of RPM packages involves the following steps.

  • Creating a source RPM from the source tarball(s) and SPEC file (on an EL5 machine, e.g. mwsecbuild.testbed);
  • creating a binary RPM from the source RPM by running a mock build (on a reasonable modern mock >= 1.0.7);
  • copying the resulting RPMs (source and binaries) to the mock repository on bleek.nikhef.nl.

For distributing the results more steps are necessary, see below.

Procedure for building a new version

If there is a new source release of a component, the following steps need to be taken:

Update the SPEC file

  • Check out the SPEC directory from the SVN repository (always use trunk):
svn co svn+ssh://svn@ndpfsvn.nikhef.nl/repos/mwsec/packaging/fedora/trunk
  • Edit the spec file by updating the Version: field to correspond to the new version.
  • Reset the Release: field to 1, but don't delete %{?dist}.
  • Update build dependencies if necessary.
  • Add a ChangeLog entry to reflect the changes in this release, briefly. See Fedora Guidelines for guidance.
  • Check in the spec file. Use the ChangeLog entry as the commit message.

Generating the source RPM

The source RPM must be generated on an EL5 machine, because EL5 and older systems have a version of RPM that doesn't understand the new SHA1 checksum.

Get the sources from the official software distribution: 

wget http://software.nikhef.nl/security/component/component-version.tar.gz

Get the SPEC file: 

wget http://ndpfsvn.nikhef.nl/ro/mwsec/packaging/fedora/trunk/component.spec

Build the source RPM: 

rpmbuild --define "_tmppath /tmp" --define "_sourcedir ." --define "_srcrpmdir ." --nodeps -bs component.spec

This will leave the source RPM in the current directory. This needs to be transferred to the mock build machine.

Generating the binary and source RPMs

Copy the source RPM to a relatively new mock build machine, e.g. CentOS 6, Fedora Core 14 or higher or Ubuntu :-). The configuration to be used should be tuned to include the stable mwsec repository at http://software.nikhef.nl/dist/redhat/el5/mwsec/: 

mkdir -p mock/config
ln -s /etc/mock/site-defaults.cfg mock/config/
ln -s /etc/mock/logging.ini mock/config/
ln -s mwsec-el5-x86_64.cfg mock/config/default.cfg
patch -o mock/config/mwsec-el5-i386.cfg /etc/mock/epel-5-i386.cfg << EOF
--- epel-5-i386.cfg     2011-08-05 13:50:28.036116189 +0200
+++ mwsec-el5-i386.cfg  2011-08-05 14:22:47.176301948 +0200
@@ -1,4 +1,4 @@
-config_opts['root'] = 'epel-5-i386'
+config_opts['root'] = 'mwsec-el5-i386'
 config_opts['target_arch'] = 'i386'
 config_opts['legal_host_arches'] = ('i386', 'i586', 'i686', 'x86_64')
 config_opts['chroot_setup_cmd'] = 'install buildsys-build'
@@ -41,6 +41,10 @@
 enabled=0
 mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=testing-epel5&arch=i386

+[mwsec]
+name=mwsec
+baseurl=http://software.nikhef.nl/dist/redhat/el5/mwsec/i386/
+
 [local]
 name=local
 baseurl=http://kojipkgs.fedoraproject.org/repos/dist-5E-epel-build/latest/i386/
EOF
patch -o mock/config/mwsec-el5-x86_64.cfg /etc/mock/epel-5-x86_64.cfg << EOF
--- epel-5-x86_64.cfg  2011-08-05 13:41:54.755416511 +0200
+++ mwsec-el5-x86_64.cfg       2011-08-05 13:47:51.501731049 +0200
@@ -1,4 +1,4 @@
-config_opts['root'] = 'epel-5-x86_64'
+config_opts['root'] = 'mwsec-el5-x86_64'
 config_opts['target_arch'] = 'x86_64'
 config_opts['legal_host_arches'] = ('x86_64',)
 config_opts['chroot_setup_cmd'] = 'install buildsys-build'
@@ -43,6 +43,15 @@
 enabled=0
 mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=testing-epel5&arch=x86_64
 
+[mwsec]
+name=mwsec
+baseurl=http://software.nikhef.nl/dist/redhat/el5/mwsec/x86_64/
+
+[mwsec_testing]
+name=mwsec-testing
+baseurl=http://software.nikhef.nl/dist/redhat/testing/el5/mwsec/x86_64/
+enabled=0
+
 [local]
 name=local
 baseurl=http://kojipkgs.fedoraproject.org/repos/dist-5E-epel-build/latest/x86_64/

EOF



mock -r nikhef-el5-x86_64 --confdir=mock/config --resultdir=mock/result --rebuild component.src.rpm

Procedure for updating a release

TO BE DONE

In the following cases a new RPM build needs to be done even if the sources did not change.

  1. There is a bug in the way the RPM was build (e.g. wrong build dependencies)
  2. There is a bug in the SPEC file, but the resulting RPM is already released
  3. A RPM-only patch or fix that is not used upstream is required

In this case, it is not necessary to download the source tarball as it should already be there. The procedure is as follows:

  • Check out the spec directory
svn co svn+ssh://svn@ndpfsvn.nikhef.nl/repos/mwsec/packaging/fedora/trunk
  • Edit the spec file by increasing the Release: field by 1, but don't delete %{?dist}.
  • Edit the spec file as necessary.
  • Add a ChangeLog entry to reflect the changes in this release.
  • Check in the spec file. Use the ChangeLog entry as the commit message.

All output should show up on 

span.nikhef.nl:/srv/project/rpmbuild/(S)RPMS/...

Generating RPMs manually

The above procedure produces 'official' RPMS, in a reproducible way from a clean build environment. However, it is possible to generate RPM packages manually with a simple recipe.

This procedure should work on a machine that matches the build target, i.e. CentOS 5 x86_64 or i386.

  • Create a file called .rpmmacros with the following content:
%_topdir  /home/username/rpmbuild
%_tmppath /home/username/rpmbuild/tmp
  • Create a directory rpmbuild in your home directory, with the following subdirectories:
BUILD -> /tmp/username/rpmbuild/BUILD
RPMS
SOURCES
SPECS
SRPMS
tmp -> /tmp/username/rpmbuild/tmp
  • create the tempdirs if necessary
mkdir -p /tmp/`id -un`/rpmbuild/{tmp,BUILD}
  • Check out the SPEC directory
  • test the build dependencies by running rpmbuild specfile
  • Install build dependencies as necessary
  • Run rpmbuild -ba specfile to generate binary and source RPMs.

Adding a new component and/or build target

This procedure is not yet worked out.

Distribution release of RPMS

Warning: this procedure is not yet finalised.

The RPMS should all be on bleek.nikhef.nl at this time.

The generated RPMS should be distributed in a public, yum-installable repository. The repository is composed on bleek.nikhef.nl:/srv/project/mwsec with the following lay-out: 

./el5
./el5/mwsec
./el5/mwsec/SRPMS            <- symlinks to mock
./el5/mwsec/SRPMS/repodata
./el5/mwsec/x86_64           <- symlinks to testing
./el5/mwsec/x86_64/repodata
./el5/mwsec/i386             <- symlinks to testing
./el5/mwsec/i386/repodata
./testing/
./testing/el5
./testing/el5/mwsec
./testing/el5/mwsec/i386            <- symlinks to mock
./testing/el5/mwsec/i386/repodata
./testing/el5/mwsec/x86_64          <- symlinks to mock
./testing/el5/mwsec/x86_64/repodata

The el5 and testing trees do not contain actual RPMS, but symlinks:

  • the testing RPMS are symlinks to the repective RPMS in mock
  • the el5 binary RPMS are symlinks to the respective files in testing
  • the el5 source RPMS are symlinks to the respective RPMS in mock

The noarch RPMS in mock are symlinked in both x86_64 and i386 (cf. the EPEL repository).

The layout of mock is 

/srv/project/mock
/srv/project/mock/rhel5
/srv/project/mock/rhel5/SRPMS
/srv/project/mock/rhel5/i386
/srv/project/mock/rhel5/noarch
/srv/project/mock/rhel5/x86_64

After updating the el5 and/or testing directories, update the repodata: 

createrepo /srv/project/mwsec/el5/mwsec/SRPMS/
createrepo /srv/project/mwsec/el5/mwsec/i386/
createrepo /srv/project/mwsec/el5/mwsec/x86_64/
createrepo /srv/project/mwsec/testing/el5/mwsec/i386/
createrepo /srv/project/mwsec/testing/el5/mwsec/x86_64/
chmod -R g+w /srv/project/mwsec

After this step, the repositories need to be synced with software.nikhef.nl: 

rsync -rltpgv --copy-unsafe-links /srv/project/mwsec/ software.nikhef.nl:/project/srv/www/site/software/html/dist/redhat/

Using this repository in a YUM repo file requires something like the following: 

[mwsec]
name=Middleware Security Software
baseurl=http://software.nikhef.nl/dist/redhat/el5/mwsec/$basearch
gpgcheck=0

Generating Debian packages from the distributed source tarballs

TODO: this section is coming soon.

Retrieved from "https://wiki.nikhef.nl/grid/index.php?title=SAC_software_procedures&oldid=5742"