How to work with our Quattor setup

This article describes how to perform basic and essential tasks with the current setup of Quattor at Nikhef.

Overview

Everyone who wants to make changes in the Quattor setup, should make them in his private working area. The working area can be created by checking out from the repository; see section Using the repository for details.

After making the changes in the private working copy, the templates should be compiled. How this is done is described in the section Compiling.

If the compilation was successful, i.e., without errors or warnings, the modified templates can be checked in into the Subversion repository. Note: modified templates should only be checked in if they are to be deployed immediately! Checking in templates but not deploying them could cause surprises for the next person to deploy his changes (well, that's what he might think!).

The deployment phase consists of a checkout on the Quattor server using a dedicated account, followed by a compilation and deployment procedure. The details of this process are given in section Deploying.

Using the repository

The Quattor setup, comprising the template hierarchy and the SCDB setup, are stored on the Subversion server ndpfsvn.nikhef.nl in repository ndpf. Access to this repository is limited to selected administrators. See the Wiki article on SVN access for details.

Checking out

Before checking out, ensure there is an ssh agent running that contains the key that is registered on the SVN server. To check out a working copy of the trunk revision under the current working directory, use the following command:

svn co svn+ssh://svn@ndpfsvn.nikhef.nl/repos/ndpf/nl.nikhef.ndpf.quattor-config/trunk

This will create a directory trunk in the working directory, which contains the checked out tree. The file system hierarchy under trunk includes the SCDB setup for Nikhef as well as the Quattor template hierarchy. After this check out, it should be possible to compile templates.

Updating

When a working copy exists, it can be synchronized with the last in the SVN repository. To retrieve the latest files, and merge them with local modifications if possible, the "svn up" command can be executed in the working directory or any of its child directories:

svn up 

This command will update the contents of the working directory and its child directories. If optional directories are specified, only the contents under those directories are updates.

Checking in

Checking in ("committing") the local changes into the SVN repository is done via the "svn commit" command:

svn commit 

The above command will commit all changes made under the current directory. It is also possible to specify which directories and/or files will be committed:

svn commit dir1 dir2 file3

will commit all changes in the hierarchies under dir1 and dir2, as well as in file file3.

Compiling

Compiling the Quattor setup consists of two phases:

1. Syntax checking

2. Translation and validation

The syntax check is performed on all *.tpl files under the base directory of the setup. That implies that any file with the file name extension .tpl should use a valid Pan syntax!

During the second phase, the compiler combines selected (valid) template files and performs some validity checks. These checks are defined in the template files and allow for compile-time checks of the values of paths in Pan. If there are no problems, the templates are compiled into an XML file. Assuming that an environment variable $L exists that points to the conf directory, the compiled templates are stored in the directory $L/build/xml.

The shell script makexprof can be used to compile profiles, provided that the directory $L/../bin is part of the environment variable PATH. Usage of the makexprof script:

makexprof [options] -f <facility-name> [template1] [template2] [...]

The mandatory -f with argument <facility-name> is required to select one of the facility names. Currently defined facilities: itb (Installation Test Bed), prd (Production cluster), opn (Optical Private Network) and generic (miscellaneous Quattor-managed servers). If one or more (object) template names are given, the compilation is only performed for the hosts under the given facility that match the specified names.

The following options can be used:

-u       Update repository templates before compilation
-d       Enable debug output
-h       Show usage information

Updating the repository templates is handled by the task update.rep.templates in the Ant build script. This action will refresh the contents of the generated templates that list all the packages that are present inthe software repositories. All packages that should be installed have to be present in at least one of these repositories. Using the -u option is identical to executing the command:

( cd $L ; ant update.rep.templates )

Deploying

The deployment phase should always immediately follow the check in of modified templates, not to surprise your colleagues at a later stage with the (rotten?) fruits of your efforts.

Unlike the other steps, the deployment step is executed with the ndpfmgr account on the Quattor server. Only selected persons can login to this account via ssh. It is strongly advised to use ssh key-forwarding when logging in. Currently, Quattor server stal should be used for deploying templates.

Start by updating to the last Quattor templates stored in Subversion:

cd $L/cfg
svn up

The templates should be compiled and the nodes should be triggered to refresh their local copies. This is handled by the command pushxprof (which is in fact a symlink to and supports the same arguments as makexprof):

pushxprof [options] -f <facility-name> [template1] [template2] [...]

Note that it may be required to use option -u to refresh the repository contents after new packages were added to the repositories. If the compilation step was successful, the newly compiled profiles were created in $L/build/xml and subsequently copied to the directory $L/deploy/xml. The contents of the latter are exposed via a web server to the nodes. Then all involved nodes are informed that a new profile exists. After receiving the notification, the nodes will download their new profiles and start changing the node configuration if needed.

It is possible to use makexprof as user ndpfmgr. However, this will only compile templates (resulting in XML files in $L/build/xml) and those compiled profiles will not be visible to the nodes!