Quattor package management with yum-based ncm-spma
Introduction
The update of ncm-spma, Quattor's package management component, to a version that uses yum instead SPMA and rpmt-py, has significantly simplified the configuration of package lists, changed the requirements on software repositories and changed the workflow of updating software.
The major advantage of the new component lies in the simplification of the software installation process. Where the original version required every package and all of its dependent packages to be exactly specified, often referred to as the RPM dependency hell, the new component only needs the packages that are desired (version can be specified if required, but that is not needed). The dependencies are handled automatically by yum, just as yum will take care of the selection of the optimal or most recent version of each package. Consequently, there will be less need for modifications in the software installation templates to upgrade and the trial-and-error approach to satisfy all dependencies will become an unpleasant relic of the past. Furthermore, since package versions do not need to be specified exactly any more, the painful process to generate templates for all package groups when a a new minor version of the distribution is released should also become simpler. Finally, since yum can calculate the minimal set of required packages, it is likely that fewer packages will be installed on the nodes. That implies a faster installation process and leaves fewer (possible unneeded) packages that may carry a vulnerability.
There is a modest price to pay for the simplification of the software templates: part of the complexity goes into maintaining the software repositories and one loses the compile-time verification of the presence of the packages to be installed in the available repositories. The latter may lead to unexpected problems when deploying, if none of the repositories holds the requested package.
One of Quattor's strengths was the ability to roll back an installation or configuration to a previous state. Performing a roll back to an earlier set of packages is not trivial when using yum. Nevertheless, it is very desirable not to lose this feature. By using snapshots of software repositories, it is possible to let yum consider an earlier set of packages when managing the installation, implying that downgrading to previous versions is required.
This article gives a description of the new situation as implemented at Nikhef.
Repository Management
Repositories are managed on the Quattor installation server stal.
There are two new aspects on repository management:
- Repositories have to be populated with care;
- Snapshotting of repositories to enable rolling back to an earlier set of packages and versions.
Populating Repositories with Packages
Before using yum, a Quattor software repository was simply a web page where a collection of packages was offered. The collection could consist of a mix of packages from different operating system versions or architectures. It was the administrator's responsibility to select a working mix of packages that could be found in the available repositories.
It is not so simple when using yum.
Yum will try to select the most recent version of a package. If two versions of the same package, intended for different distributions, exist in a repository, yum will always select the most recent, i.e., the lexically largest file name. For example, if a repository contains mypackage-1.0.el5.noarch.rpm and mypackage-1.0.el6.noarch.rpm, installing mypackage will always lead to selection of mypackage-1.0.el6.noarch.rpm, irrespective of the platform used. That can lead to very, very wrong dependency resolutions!
To prevent such problems, the repositories must be organized per distribution and optionally per architecture and may only contain packages corresponding to (i.e., built for) the related distribution and architecture. The following example presents the repositories for locally developed tools which are independent of the hardware architecture, nikhef-tools-el5 and nikhef-tools-el6:
nikhef
+-- tools/
+-- el5
+-- el6
To add new (versions of) packages to a repository, the above layout must be taken into consideration. Furthermore, new packages should be added to the repository mirror. The repository mirror is a dedicated directory (location) which acts as the source from which repository snapshots (see next section) are derived. This directory will contain all packages that are available within the context of the repository. The actual selection is made in the snapshots of the repository.
At the Quattor install server stal, the mirror directory is /project/mirror.
Snapshotting Repositories
To enable restoring a previous set of package versions, snapshots of software repositories are kept on the Quattor installation server. Per repository mirror location, there are one or more snapshots. Each snapshot location contains hard links to the packages present in the mirror location on a given moment. The snapshot location is converted into a yum repository by generating the repodata index files.
By convention, the repository snapshots are stored on the Quattor installation server under directory /project/repositories. The snapshot directory hierarchy is included in the periodic rsync-synchronisation to the slave servers behind alias stalkaars.
After adding a new package to a mirror, a new snapshot can be made with the script makesnapshot:
makesnapshot [options] <repository> <label>
The script will create a snapshot of the actual contents of the mirror repository. The snapshot will be named <label>. The details of the snapshot creation are described per repository a configuration file <repository>.conf in directory /project/quattor/etc/repository-snapshots/. This configuration file at least contains the source for the snapshot (by convention under the mirror location) and the destination (by convention under the snapshot location).
The script makesnapshot uses rsync to create hard links in the snapshot directory, which refer to the packages in the mirror directory. It is possible to modify the behavior of rsync by passing it some options via the arguments parameter.
For example, creating a snapshot with command makesnapshot nikhef-tools-el5 20130930 where the configuration file /project/quattor/etc/repository-snapshots/nikhef-tools-el5.conf is:
source = /project/mirror/nikhef/tools/el5 destination = /project/repositories/nikhef/tools/el5 arguments = --exclude nikhef-yaim-core-1.0.1-1.noarch.rpm
will create a new snapshot under directory /project/repositories/nikhef/tools/el5/20130930 with the actual contents of /project/mirror/nikhef/tools/el5, ignoring the presence of nikhef-yaim-core-1.0.1-1.noarch.rpm in the mirror directory.
It's a good idea to document what the reason is for creating a new snapshot and include hints as to what was added, removed or updated.
Package Selection
The selection of packages to be installed on a particular host is defined in the Quattor templates. Before using the yum-based SPMA, all packages had to be exactly specified, providing the name, version and architecture(s). Furthermore, SPMA required all dependencies of each package to be included in the list of packages to be installed.
For example, the installation of the grid middleware for the UMD-2 worker node required the specification of circa 80 packages:
"/software/packages"=pkg_repl("emi-wn","2.0.0-1.sl6","x86_64");
"/software/packages"=pkg_repl("a1_grid_env","3.0.2-1.sl6","x86_64");
"/software/packages"=pkg_repl("CGSI-gSOAP","1.3.5-2.el6","x86_64");
"/software/packages"=pkg_repl("classads","1.0.8-1.el6","x86_64");
"/software/packages"=pkg_repl("cleanup-grid-accounts","2.0.0-1","noarch");
"/software/packages"=pkg_repl("dcache-srmclient","2.1.1-1.el6","x86_64");
...
With the new SPMA, this has become much easier. Not only will yum select the most recent version of a package, assuming that a package's dependencies were correctly specified at build time, yum will also include all dependencies in the installation transaction.
The installation of the grid middleware for the UMD-2 worker node is now handled by a single line:
"/software/packages/{emi-wn}" = nlist();
Not only is the code in the Quattor template much shorter, and thus easier to maintain, it also requires less effort to update a package to the next version. The package version and its dependencies do not require modifications in the templates anymore; it is sufficient to change the repository snapshot selector so that it points to a newer version, as described in the next section.
Note that it is still possible to pin a package to a specific version. That can be achieved by using an old-style specification for (just!) the package under consideration:
"/software/packages"=pkg_repl("emi-wn","2.0.0-1.sl6","x86_64");
This will lead to installation of the RPM emi-wn-2.0.0-1.sl6.x86_64.rpm and its dependencies.
Repository Snapshot Selection
The selection of the snapshot for a given repository is handled in the Quattor templates. There are two changes with respect to the previous situation:
- The layout and contents of the repository templates have changed;
- There is a mechanism to select which snapshot to use for a given repository.
In the previous setup, each repository template pointed to a URL corresponding to the repository. Via an option in the build script makexprof, an index of all packages found under the URL was generated in the contents section of the repository template. At the end of the profile compilation process, the list of packages was validated against the list of packages available in the repositories. Compilation would fail if a package was present in the list for installation, whereas it was absent from the list of available packages.
In the current setup, a repository template points to a URL definition. However, part of the URL contains a place holder, which will later be expanded to the label of the snapshot.
structure template repository/centos64_x86_64_updates;
"name" = "centos64_x86_64_updates";
"owner" = "<SOME EMAIL ADDRESS>";
"protocols" = list(
nlist("name","http",
"url","http://stalkaars.nikhef.nl/repositories/centos/6/updates/%s")
);
The place holder %s is replaced by the snapshot label at the end of compilation.
The snapshot label is defined via a global variable REPOSITORY_SNAPSHOTS which must be of type nlist(). In this hash, the repository name is the key and the snapshot label is the value.
The snapshot label can be defined at 3 levels, in order of decreasing priority:
- per host, in the object template $L/cfg/clusters/<CLUSTER>/profiles/<HOSTNAME}.tpl;
- per cluster, in the template $L/cfg/clusters/<CLUSTER>/repository/config.tpl;
- globally, in the template $L/cfg/sites/ndpf/site/repository/default-snapshots.tpl.
To prevent accidental overwriting of earlier values, it is recommended to use the function add_repository_snapshot to add a snapshot label. This function will ignore a new value for the key if one was previously defined:
variable REPOSITORY_SNAPSHOTS = add_repository_snapshot( nlist(
"nikhef-tools-el5", "20130925",
"nikhef-tools-el6", "20130925",
) );
The repository templates no longer contain the index of packages and the option -u has been removed from the script makexprof. Consequently, compile-time validation of packages is no longer possible.
Workflow Installing or Updating a Package
To install or update a package on one host, all hosts in a cluster, or globally, proceeds as follows:
- Copy the package to the mirror location.
- Create a snapshot with a label.
- For installation, add the package to the Quattor templates. For an update, it may not be necessary.
- Change variable REPOSITORY_SNAPSHOTS at the appropriate level (host, cluster, globally) so that for the repository the new snapshot label is used.
- Compile & deploy.
External Documentation
Documentation from the Quattor website:
