User:Dennisvd@nikhef.nl/UserSpacePackageManagement

Managing software packages on the Grid is complicated, because there is no central control over the setup of the operating environment across clusters. Even in a relatively small, tightly coordinated collection of clusters such as BiG Grid, the offering of packages is not tailored to the user's needs.

There has always been the option for users (or rather, VOs) to place their own software in special locations on a per-cluster basis. This is a low-level fallback, basically offering just file system access and nothing else. To do proper package management in these so-called VO software areas is something that needs to be looked at.

There are various possibilities, but the one explored here concerns pkgsrc by NetBSD. It is a widely available open-source system that fetches and builds software from source whenever needed.

Easy installation

The pkgsrc-cmd.sh script (TODO: include link) is meant to be run in grid jobs by the VO software manager. The script can be used to initialize, check and install pkgsrc in the VO specific software area. Here is an example JDL file to initialize the system.

Executable = "pkgsrc-cmd.sh";
Arguments = "init";
Stdoutput = "stdout";
StdError = "stderror";
InputSandbox = "pkgsrc-cmd.sh";
OutputSandbox = {"stdout"};

Manual Installation steps

1. download pkgsrc (choose the most recent quarterly release; approximate size is 30MB)
2. Let's assume the the VO software area is /data/esia/pvier. Unpack the pkgsrc distribution there.

   tar xfz pkgsrc.tar.gz

   cd pkgsrc

3. bootstrap it for the local system. This creates a directory called 'work'.

   ./bootstrap --prefix /data/esia/pvier/pkg --unprivileged

   this will install the basics in prefix already.

   (I should try this on a minimalist system to see which build dependencies this brings).

4. Add the bin and sbin directories to the PATH and install any package

   export PATH=/data/esia/pvier/pkg/bin:/data/esia/pvier/pkg/sbin:$PATH

5. build something: cd lang/python26 && bmake
6. bmake install
7. bmake package

Bootstrapping

Trying to create a binary package of the bootstrapping phase, with ./bootstrap --gzip-binary-kit /tmp/pkgsrc-bootstrap.tar.gz --workdir /tmp/bootstrap --unprivileged --prefix=/tmp/foo

The resulting tarball can be deployed on another machine, the directories under ./tmp/foo should be moved to the real locations and etc/mk.conf should be edited to reflect the true setup. After that binary packages can be deployed.

Next: find out if moving packages to another prefix works.