Difference between revisions of "Installing updates: OS, CAs, Quattor, VL-e"

From PDP/Grid Wiki
Jump to navigationJump to search
m (Edited the "Synchronization of the local mirror" part.)
 
(24 intermediate revisions by 2 users not shown)
Line 31: Line 31:
  
 
At the moment, the following distributions are mirrored:
 
At the moment, the following distributions are mirrored:
* CentOS 4.8 i386  
+
* CentOS 4.9 i386  
* CentOS 4.8 x86_64
+
* CentOS 4.9 x86_64
* CentOS 5.4 x86_64
+
* CentOS 5.7 x86_64
  
 +
=== Generation of the OS update template for Quattor ===
 +
('''NOTE''': These article has been updated with the steps followed to upgrade from Centos56 to Cento57)
 +
 +
'''If you're just installing a new kernel, go directly to the step [http://www.nikhef.nl/pub/projects/grid/gridwiki/index.php/Installing_updates:_OS%2C_CAs%2C_Quattor%2C_VL-e#errata "errata"].'''
 +
 +
The current procedure for generating the OS updates is divided into four major steps:
 +
 +
* Create a new branch
 +
* buildOSTemplates: Script to produce OS templates from a Red Hat style OS distribution.
 +
* errata: This is similar to the description on [https://trac.lal.in2p3.fr/Quattor/wiki/DOC/OS/Errata OS Errata Management and Deployment]
 +
* Make manual changes in the necessary templates under the new branch.
 +
 +
==== Create a new branch ====
 +
 +
By using your local copy:
 +
 +
* cd $L/cfg/os
 +
* cp -a centos56_x86_64 centos57_x86_64
 +
 +
The copied branch still contains the directories that Subversion uses internally. They need to be removed:
 +
 +
* find $L/cfg/os/centos57_x86_64 -type d -name .svn -exec rm -rf {} \;
 +
 +
The templates under the directory ''$L/cfg/os/centos57_x86_64/rpms'' can also be deleted, as new ones will be generated by buildOSTemplates.
 +
 +
==== buildOSTemplates====
 +
 +
This is script is located under ''$L/bin'' and, as said, it will produce OS templates from a Red Hat style OS distribution. These are the relevant options to be used (For more information about other options, have a look at the script):
 +
 +
* '''--compsdir''' : name of directory under Distrib_Root containing ''comps.xml'' or file name passed with ''--compsxml''
 +
* '''--compsxml''' : name of xml file describing packages to be installed if different from ''comps.xml''
 +
* '''--debug''' : enable (very) verbose option in called tools. Use several times to increase verbosity.
 +
* '''--rpmdir''' : name of directory under Distrib_Root containing RPMs
 +
* '''--scdbroot''' : path of SCDB top-level directory (must contain ''externals/''). Default to current directory. As an alternative, define environment variable QUATTOR_SCDB_ROOT.
 +
* '''Distrib_Root''' : directory containing both a ''base'' and a ''RPMS'' directory.
 +
* '''Template_Dir''' : directory where to put the generated templates.
 +
 +
This is the generic script usage:
 +
 +
$L/bin/buildOSTemplates <OPTIONS> Distrib_Root Template_Dir
 +
 +
Example:
 +
 +
$L/bin/buildOSTemplates --compsdir os/x86_64/repodata --compsxml mycomps.xml --debug --rpmdir os/x86_64/CentOS --scdbroot $L /project/mirror/centos/5.7 $L/cfg/os/centos57_x86_64/rpms
 +
 +
==== errata ====
 +
 +
At host stal by using your personal account, run <tt>rpmErrata.pl</tt> to generate an errate template from the available OS updates:
 +
$L/../bin/rpmErrata.pl /project/mirror/centos/5.7/updates/x86_64/RPMS/ > \
 +
                        $L/cfg/os/centos57_x86_64/rpms/errata/<DATE>.tpl
 +
where <DATE> is (by convention) the date of generation, formatted as YYYYMMDD (e.g. 20120117).
 +
 +
The generated template contains only the last version of each package found in the update repository, except for the kernel packages. Only if the package was already installed before it will be updates. For kernel packages, a ''comment line'' for the last version is added. The kernel to be installed are selected somewhere else; see below for details.
 +
 +
The template sites/ndpf/site/os/errata-defaults.tpl defines the default errata versions to be deployed for a particular OS version:
 +
variable OS_ERRATA_TEMPLATE ?= nlist(
 +
  'centos57_x86_64',    'rpms/errata/20120117',
 +
  'centos49_x86_64',    'rpms/errata/20110520',
 +
  'centos49_i386',      'rpms/errata/20110520',
 +
);
 +
If needed, deviations from the defaults can be defined on a per-host basis in template sites/ndpf/site/os/version_db.tpl:
 +
variable NODE_OS_ERRATA_TEMPLATE ?= nlist(
 +
  escape('tbn13.nikhef.nl'),    'rpms/errata/20090101',
 +
);
 +
The definitions in NODE_OS_ERRATA_TEMPLATE take precedence over the defaults.
  
 +
For each errata template, there must be a corresponding initialization template at location:
 +
os/<OS_VERSION>/config/os/errata/20120117-init.tpl
  
=== Generation of the OS update template for Quattor ===
+
This template defines the (default) kernel version for the corresponding erratum level:
  
At host stal using your personal account, run <tt>rpmUpdates.pl</tt> to generate an update template from the available OS updates:
+
  $ cat os/centos57_x86_64/config/os/errata/20120117-init.tpl  
  $L/../bin/rpmUpdates.pl /project/mirror/centos/3.9/updates/i386/RPMS/ > /tmp/updates-centos39_i386.tpl
+
  template config/os/errata/20120117-init;
  $L/../bin/rpmUpdates.pl /project/mirror/centos/4.7/updates/i386/RPMS/ > /tmp/updates-centos47_i386.tpl
+
   
  $L/../bin/rpmUpdates.pl /project/mirror/centos/4.7/updates/x86_64/RPMS/ > /tmp/updates-centos47_x86_64.tpl
+
  variable OS_KERNEL_VERSION_ERRATA ?= nlist(
  $L/../bin/rpmUpdates.pl /project/mirror/centos/5.2/updates/i386/RPMS/ > /tmp/updates-centos52_i386.tpl
+
  'centos57',  '2.6.18-274.17.1.el5',
  $L/../bin/rpmUpdates.pl /project/mirror/centos/5.2/updates/x86_64/RPMS/ > /tmp/updates-centos52_x86_64.tpl
+
  );
  
The generated template will contain only the last version of each package found in the update repository, except for the kernel packages, for which all versions are added. By convention, we want Quattor only to keep the installation of the currently running kernel and the desired new kernel version. The other modules should thus be removed by hand or via the script $L/../bin/selectKernels.pl:
+
It is possible to override the kernel version via the variable KERNEL_VERSION_NUM. The selected kernel version is automatically added to the configuration for GRUB, so that it becomes the default for the next boot.
selectKernels.pl -k 2.6.9-78.0.8.EL -k 2.6.9-78.0.13.EL -o /tmp/new-updates.tpl -v /tmp/updates.tpl
 
(which deletes all kernels from /tmp/updates.tpl, except for 2.6.9-78.0.8.EL and 2.6.9-78.0.13.EL, stores the output in /tmp/new-updates.tpl and produces verbose output).
 
 
 
  
While editing the generated file, ensure that the template name is <tt>rpms/updates</tt>. Finally copy the edited file to <tt>$L/cfg/os/''<distribution>''/rpms/updates.tpl</tt>
+
==== Manual changes ====
  
If the target kernel version has changed, the value of the Pan variable KERNEL_VERSION_NUMBER has to be changed accordingly in the template <tt>$L/cfg/os/''<distribution>''/config/os/kernel_version_arch.tpl</tt>:
+
The RPMs included in the new templates could create dependencies and conflicts with the RPMs included in the templates that were copied from the old branch or with those included in the gLite templates. This is not a straight-forward operation and it needs some patient and time.
variable KERNEL_VERSION_NUMBER ?= "2.6.9-78.0.8.EL";
 
  
 
=== Generation of other templates ===
 
=== Generation of other templates ===
Line 72: Line 135:
  
 
Note: if a kernel update is installed, schedule a reboot of the node(s) to ensure that the desired kernel is loaded as soon as possible, to prevent surprises at a later stage.
 
Note: if a kernel update is installed, schedule a reboot of the node(s) to ensure that the desired kernel is loaded as soon as possible, to prevent surprises at a later stage.
 +
 +
 +
----
 +
[http://www.bronvanwelzijn.nl]

Latest revision as of 11:55, 25 January 2012

There are various kinds of packages that need to be updated frequently:

  • updates for the Operating System
  • updates for the gLite middleware
  • updates for the VL-e software
  • new versions of the CA packages
  • new Quattor releases

There is a lot of similarity in these updates. This article describes the procedures for the updates listed above, except for the gLite updates. Updating gLite packages is more complicated than the other updates; therefore, there is a dedicated article Updating Quattor-managed gLite servers.

Installing updates comprises the following steps:

  • Synchronization of the local mirror of the OS repository
  • Generation of Quattor update templates
  • Compilation and deployment

Synchronization of the local mirror

To synchronize the local mirror of the Operating System, execute the following script as user ndpfmgr at Quattor server stal:

mirror-centos

This will synchronize the local OS mirror at host stal with the OS mirror at spiegel.

To synchronize the local mirror of the CA mirror in /project/quattor/www/html/mirror/CA/, execute the following script as user ndpfmgr at Quattor server stal:

mirror_cas

To synchronize the local mirror of the Quattor software under /project/quattor/www/html/quattor/, execute the following script as user ndpfmgr at Quattor server stal:

mirror-quattor

To synchronize the local mirror of the VL-e PoC R3 mirror under /project/quattor/www/html/mirror/VL-e//R3/, execute the following script as user ndpfmgr at Quattor server stal:

mirror-vle

At the moment, the following distributions are mirrored:

  • CentOS 4.9 i386
  • CentOS 4.9 x86_64
  • CentOS 5.7 x86_64

Generation of the OS update template for Quattor

(NOTE: These article has been updated with the steps followed to upgrade from Centos56 to Cento57)

If you're just installing a new kernel, go directly to the step "errata".

The current procedure for generating the OS updates is divided into four major steps:

  • Create a new branch
  • buildOSTemplates: Script to produce OS templates from a Red Hat style OS distribution.
  • errata: This is similar to the description on OS Errata Management and Deployment
  • Make manual changes in the necessary templates under the new branch.

Create a new branch

By using your local copy:

  • cd $L/cfg/os
  • cp -a centos56_x86_64 centos57_x86_64

The copied branch still contains the directories that Subversion uses internally. They need to be removed:

  • find $L/cfg/os/centos57_x86_64 -type d -name .svn -exec rm -rf {} \;

The templates under the directory $L/cfg/os/centos57_x86_64/rpms can also be deleted, as new ones will be generated by buildOSTemplates.

buildOSTemplates

This is script is located under $L/bin and, as said, it will produce OS templates from a Red Hat style OS distribution. These are the relevant options to be used (For more information about other options, have a look at the script):

  • --compsdir : name of directory under Distrib_Root containing comps.xml or file name passed with --compsxml
  • --compsxml : name of xml file describing packages to be installed if different from comps.xml
  • --debug : enable (very) verbose option in called tools. Use several times to increase verbosity.
  • --rpmdir : name of directory under Distrib_Root containing RPMs
  • --scdbroot : path of SCDB top-level directory (must contain externals/). Default to current directory. As an alternative, define environment variable QUATTOR_SCDB_ROOT.
  • Distrib_Root : directory containing both a base and a RPMS directory.
  • Template_Dir : directory where to put the generated templates.

This is the generic script usage:

$L/bin/buildOSTemplates <OPTIONS> Distrib_Root Template_Dir

Example:

$L/bin/buildOSTemplates --compsdir os/x86_64/repodata --compsxml mycomps.xml --debug --rpmdir os/x86_64/CentOS --scdbroot $L /project/mirror/centos/5.7 $L/cfg/os/centos57_x86_64/rpms

errata

At host stal by using your personal account, run rpmErrata.pl to generate an errate template from the available OS updates:

$L/../bin/rpmErrata.pl /project/mirror/centos/5.7/updates/x86_64/RPMS/ > \
                        $L/cfg/os/centos57_x86_64/rpms/errata/<DATE>.tpl

where <DATE> is (by convention) the date of generation, formatted as YYYYMMDD (e.g. 20120117).

The generated template contains only the last version of each package found in the update repository, except for the kernel packages. Only if the package was already installed before it will be updates. For kernel packages, a comment line for the last version is added. The kernel to be installed are selected somewhere else; see below for details.

The template sites/ndpf/site/os/errata-defaults.tpl defines the default errata versions to be deployed for a particular OS version:

variable OS_ERRATA_TEMPLATE ?= nlist(
  'centos57_x86_64',    'rpms/errata/20120117',
  'centos49_x86_64',    'rpms/errata/20110520',
  'centos49_i386',      'rpms/errata/20110520',
);

If needed, deviations from the defaults can be defined on a per-host basis in template sites/ndpf/site/os/version_db.tpl:

variable NODE_OS_ERRATA_TEMPLATE ?= nlist(
  escape('tbn13.nikhef.nl'),    'rpms/errata/20090101',
);

The definitions in NODE_OS_ERRATA_TEMPLATE take precedence over the defaults.

For each errata template, there must be a corresponding initialization template at location:

os/<OS_VERSION>/config/os/errata/20120117-init.tpl

This template defines the (default) kernel version for the corresponding erratum level:

$ cat os/centos57_x86_64/config/os/errata/20120117-init.tpl 
template config/os/errata/20120117-init;

variable OS_KERNEL_VERSION_ERRATA ?= nlist(
  'centos57',   '2.6.18-274.17.1.el5',
);

It is possible to override the kernel version via the variable KERNEL_VERSION_NUM. The selected kernel version is automatically added to the configuration for GRUB, so that it becomes the default for the next boot.

Manual changes

The RPMs included in the new templates could create dependencies and conflicts with the RPMs included in the templates that were copied from the old branch or with those included in the gLite templates. This is not a straight-forward operation and it needs some patient and time.

Generation of other templates

Update templates for VL-e software can be generated using rpmUpdates.pl. The software update template is located under $L/cfg/grid/vle/<PoC-release>/rpms/<OS-version>/updates.tpl

The easiest way to deal with a set of new CAs, is to download the template from the CA site: [http://groep.web.cern.ch/groep/cadist/] (you need to go down into the directory corresponding to the new release and get the file pro_software_meta_lcg_CA.tpl). This file needs to be customized: the template's name should read common/security/lcg-CA and should eventually be saved as $L/cfg/grid/common/security/lcg-CA.tpl

To change the Quattor software that is installed, the new versions should be added by hand to the templates, to be found under $L/cfg/os/<distribution>/rpms/quattor.tpl

Compilation and Deployment

Perform a test compilation in your local environment and do not forget to refresh the repository templates:

makexprof -u -f itb

If this succeeds, commit the changes to SVN and follow the deployment procedure to install the updates.

Note: if a kernel update is installed, schedule a reboot of the node(s) to ensure that the desired kernel is loaded as soon as possible, to prevent surprises at a later stage.



[1]