Difference between revisions of "Enabling multicore jobs and jobs requesting large amounts of memory"
| Line 31: | Line 31: | ||
By default, the batch system allocates 1 core and (optionally) a certain amount of memory to each job. It is reasonable that a multicore job requesting ''N'' cores can also access ''N'' times the amount of memory of a single job. Furthermore, when a job requests ''M'' times the amount of memory of a default jobs, that means that ''M''-1 other jobs cannot run on the machine. From resource point of view, the large-memory job is thus equivalent to a job requesting ''M'' cores. In the accounting system and resource usage counting, both types of jobs should be treated as if they are ''N'' or ''M'' simple jobs. | By default, the batch system allocates 1 core and (optionally) a certain amount of memory to each job. It is reasonable that a multicore job requesting ''N'' cores can also access ''N'' times the amount of memory of a single job. Furthermore, when a job requests ''M'' times the amount of memory of a default jobs, that means that ''M''-1 other jobs cannot run on the machine. From resource point of view, the large-memory job is thus equivalent to a job requesting ''M'' cores. In the accounting system and resource usage counting, both types of jobs should be treated as if they are ''N'' or ''M'' simple jobs. | ||
| − | The submit filter presented here scales the amount of memory available to a multicore job with the number of requested cores. The amount of memory per core is obtained from the queue definition (see subsection Setup | + | The submit filter presented here scales the amount of memory available to a multicore job with the number of requested cores. The amount of memory per core is obtained from the queue definition (see subsection Setup at the batch system), possibly capped by a maximum (which is also taken from the queue definition). '''Note:''' Torque allows to define defaults and maximum limits for physical memory (option ''mem''), virtual memory (option ''vmem''), virtual memory per process (''pvem''). The submit script will scale ''mem'' and ''vmem'' with the number of cores; the per-process limits ''pvmem'' and ''pmem'' scale with the maximum number of cores on any of the allocated physical machines. |
For large-memory jobs, the submit filter scales the number of allocated cores with the amount of requested memory divided by the amount of memory per single job; the number of cores is rounded up to the nearest integer. A large-memory jobs thus also gets allocated more than one core for computing. | For large-memory jobs, the submit filter scales the number of allocated cores with the amount of requested memory divided by the amount of memory per single job; the number of cores is rounded up to the nearest integer. A large-memory jobs thus also gets allocated more than one core for computing. | ||
Revision as of 16:41, 6 August 2012
This article describes
Introduction
Certain applications will benefit from access to more than core (logical CPU) on the same physical computer. Grid jobs that use more than one core are referred to as multicore jobs.
Other applications require a specific amount of memory to run efficiently or successfully. Such jobs are called large-memory jobs in this article (because they often require a higher-than-default amount of memory on the machine).
The Cream Computing Elements (CreamCEs) offer support for multicore jobs and large-memory jobs, although they need some additional configuration to forward the job requirements to the batch system.
This article describes the support of multicore jobs or large-memory jobs at the Cream Computing Elements at Nikhef. Section "System Configuration" describes the setup of the system. In section "Submitting multicore or (large) memory jobs", the information relevant to users of the Computing Elements is presented.
The information presented here is valid for the UMD-1 version of the CreamCE in combination with a batch system based on Torque 2.3. Other versions of the CreamCE (in particular the nearly unsupported gLite 3.2 version) may require different configuration. Other versions of the Torque batch system may work fine, although that hasn't been verified. Different batch systems fall outside the scope of this article.
System Configuration
Two services are involved in the submission of grid jobs requiring multiple cores or specific amounts of memory: the CreamCE and the Torque batch system. The CreamCE is the entry point for a grid job at a site. The CreamCE processes the resource requests and translates them into a format that is specific for the batch system implementation. The batch system can then allocate the requested resources.
==== Setup at the CreamCE
To support multicore jobs or memory requests, the CreamCE must recognize these requests and translate them into directives for the Torque batch system. In the implementation discussed here, the CreamCE needs 3 files:
- A script to process specific resource requests Media:pbs_local_submit_attributes.sh. This file should be installed as /usr/bin/pbs_local_submit_attributes.sh on the CreamCE. It processes memory resource requests.
- A configuration file to activate this submit filter Media:torque.cfg. This file should be stored as /var/spool/pbs/torque.cfg.
- A Torque submit filter to write the input for the batch system Media:torque_submit_filter.sh. This file should be installed as /usr/local/sbin/torque_submit_filter.sh (actually: the location specified in file torque.cfg above as value for SUBMITFILTER). The submit filter is discussed below in more detail.
The submit filter
The submit filter processes the initial input for the batch system. It inspects the requested resources and may change them before writing the (possibly modified) input to standard output (which is forwarded to the batch server).
By default, the batch system allocates 1 core and (optionally) a certain amount of memory to each job. It is reasonable that a multicore job requesting N cores can also access N times the amount of memory of a single job. Furthermore, when a job requests M times the amount of memory of a default jobs, that means that M-1 other jobs cannot run on the machine. From resource point of view, the large-memory job is thus equivalent to a job requesting M cores. In the accounting system and resource usage counting, both types of jobs should be treated as if they are N or M simple jobs.
The submit filter presented here scales the amount of memory available to a multicore job with the number of requested cores. The amount of memory per core is obtained from the queue definition (see subsection Setup at the batch system), possibly capped by a maximum (which is also taken from the queue definition). Note: Torque allows to define defaults and maximum limits for physical memory (option mem), virtual memory (option vmem), virtual memory per process (pvem). The submit script will scale mem and vmem with the number of cores; the per-process limits pvmem and pmem scale with the maximum number of cores on any of the allocated physical machines.
For large-memory jobs, the submit filter scales the number of allocated cores with the amount of requested memory divided by the amount of memory per single job; the number of cores is rounded up to the nearest integer. A large-memory jobs thus also gets allocated more than one core for computing.
After the number of cores has been established, the maximum CPU time available to the job will be computed. This limit is the product of the allocated number of cores and the default CPU time limit for a single core job. Note: although not required by Torque, the grid information system for certain VOs expects to find a CPU time limit. So not defining a CPU time limit will cause problems.
