Various local tools

This article present some local tools for use with the Torque resource manager. These tools are working with Torque version 2.3.8. Other versions may require modifications.

when_idle

Purpose: execute a command when a node has drained.

Description: this script is executed from the system epilogue script. At the end of each job, the script checks whether the following conditions are met:

• The node is offline and the offline comment matches a certain pattern (default: when_idle)
• The node is idle (contains no running batch jobs)

If the node is offline and idle, a script provided by the administrator can be executed if its md5sum is not present in an archive. This ensures that the scripts gets executed only once. After execution of the administrator's script, its md5sum is added to the archive and message is written to syslog. After a certain delay (15s), the node's status is cleared (i.e., no longer offline).

An additional init script is used when the node boots into run level 3, to check whether the node was offline with tag "when_idle". If that is the case, after a delay (600s) the node state is cleared and a message is sent to syslog. This handles the situation when the administrator's script reboots the node before the node's state could be cleared, which would keep the node offline.

Example: reboot the worker node when all batch jobs have finished, for example to load a new kernel. Invocation is simply executing /path/to/when_idle. The administrator's script would be:

# 2013-09-04
# Substitute <DATE> to force a different checksum
# Put the action here
reboot

prune_userprocs

Purpose: remove daemonized users processes after a batch job ends

Description: this script is executed from the system epilogue script. It searches for user processes that are left after their parent batch job ended. User processes started from a login or ssh session are excluded as well as processes running with a system account (uid below a certain level). This prevents stray processes from using resources meant for legitimate batch processes.

Example: kill all processes belonging to users with uid > 999 (-u) that are not part of a batch job (-a) with signal 9 (-k).

prune_userprocs -a -k 9 -u 999

mom-taskset

Purpose: allow the processes in a batch job to use exclusively use the cores corresponding to the allocated batch slots.

Description: this script is executed from the system prologue script (at present it is the prologue). It uses the linux taskset command to restrict the cores that the job's processes can access. If a job starts a lot of threads or child processes, those threads will compete with each other for CPU and not with processes belonging to other (users') jobs. The script can handle multicore jobs.

Assumption/limitation: this script assumes that the number of configured job slots on a worker node is smaller or equal to the number of cores. It is likely to behave unexpectedly (undesirably) when hyperthreading is enabled to allocate two "cores" to each job.

Example: the script works autonomously. Below the debug output is shown for a job to which 7 cores were allocated.

jobid=7168615.allier.nikhef.nl, slot=0,1,2,3,4,5,6
Setting taskset to CPU 0,1,2,3,4,5,6 for pid=1912
pid 1912's current affinity list: 0-7 pid 1912's new affinity list: 0-6