Difference between revisions of "NDPFAccounting"
Line 12: | Line 12: | ||
= NDPF Local Accounting = | = NDPF Local Accounting = | ||
+ | |||
+ | The local accounting is the most important element, and must (and is :-) fully reliable, because it is used as the basis for the cost reimbursement for projects where we contribute in-kind contributions in the form of compute cycles. These data are collected (yearly) from the NDPF accounting database on a per-VO basis. | ||
+ | |||
+ | Data is inserted into this atabase on a daily basis. The records are (or should) inserted just after midnight, when the pbs/torque accounting files have been closed and are complete. Since the accounting is based on the "E" records in that file, we thus get all completed jobs. Jobs that are still running will ''not'' be accounted -- they will be files only as soon as they are finished. | ||
+ | |||
+ | == Master insertion == | ||
+ | Insertion in the database requires the collaboration of two components: | ||
+ | * the mapping from poolaccounts to grid user DNs | ||
+ | * the extraction of the pbs data from the accounting file, and linking the unix users of the facility to their grid credentials | ||
+ | |||
+ | At the moment, the grid group (FQAN) mappings are ''not'' part of this scheme, and only unix groups are stored in the database. The unix group-to-gridVO mapping is only done in the EGEE upload phase. This is partly historical, but since the VO-FQAN mapping side of the grid software is in constant flux anyway it is better like it is done now. | ||
+ | |||
+ | To ease the insertion, a meta-utility has been developed: <tt>ndpf-acctfeed</tt>. It is to be run on the PBS master (stro) every night | ||
+ | |||
+ | Usage: ./ndpf-acctfeed [-v] [-h] [-f] [--mapfile <poolmap>|--gridmapdir <dir>] | ||
+ | [--date|-d Ymd] [--nocapacity] [-n|--dryrun] | ||
+ | [--pbsaccdir dir] [--progacc command] [--progcapacity command] | ||
+ | |||
+ | and in accounting.ncm-cron.cron: | ||
+ | |||
+ | 15 0 * * * root (date --iso-8601=seconds --utc; PATH=/sbin:/bin:/usr/sbin:/usr/bin:/share/acct/bin; /share/acct/bin/merge_new_accdata) >> /var/log/accounting.ncm-cron.log 2>&1 | ||
+ | |||
+ | |||
+ | PS: the <tt>ndpf-acctfeed</tt> meta-tool replaces the historic 'merge_new_accdata' script. | ||
== Collecting data from PBS/Torque == | == Collecting data from PBS/Torque == | ||
The file <tt>/etc/pbsaccdb.conf</tt> must be present on the collecting system (i.e. the Torque server) and be formatted as described in [[NDPFAccouting_pbsaccdbconf]]. | The file <tt>/etc/pbsaccdb.conf</tt> must be present on the collecting system (i.e. the Torque server) and be formatted as described in [[NDPFAccouting_pbsaccdbconf]]. | ||
+ | |||
== Scaling and the GHzHourEquivalent == | == Scaling and the GHzHourEquivalent == | ||
Line 22: | Line 47: | ||
== Re-inserting historic data == | == Re-inserting historic data == | ||
− | |||
= EGEE uploads = | = EGEE uploads = |
Revision as of 10:04, 24 December 2008
Systems involved
vlaai | gridmapdir NFS | create poolmap files based on the gridmapdir state on a daily basis |
stro | Torque server | conversion from Torque accounting files and inserting these into the NDPF accounting database on fulda, using a poolmapfile (it will try to collect one automatically if it can mount the gridmapdir) |
klomp | MON box | conversion of data from accounting database to lcgrecords (UR-WG format), and upload through R-GMA |
NDPF Local Accounting
The local accounting is the most important element, and must (and is :-) fully reliable, because it is used as the basis for the cost reimbursement for projects where we contribute in-kind contributions in the form of compute cycles. These data are collected (yearly) from the NDPF accounting database on a per-VO basis.
Data is inserted into this atabase on a daily basis. The records are (or should) inserted just after midnight, when the pbs/torque accounting files have been closed and are complete. Since the accounting is based on the "E" records in that file, we thus get all completed jobs. Jobs that are still running will not be accounted -- they will be files only as soon as they are finished.
Master insertion
Insertion in the database requires the collaboration of two components:
- the mapping from poolaccounts to grid user DNs
- the extraction of the pbs data from the accounting file, and linking the unix users of the facility to their grid credentials
At the moment, the grid group (FQAN) mappings are not part of this scheme, and only unix groups are stored in the database. The unix group-to-gridVO mapping is only done in the EGEE upload phase. This is partly historical, but since the VO-FQAN mapping side of the grid software is in constant flux anyway it is better like it is done now.
To ease the insertion, a meta-utility has been developed: ndpf-acctfeed. It is to be run on the PBS master (stro) every night
Usage: ./ndpf-acctfeed [-v] [-h] [-f] [--mapfile <poolmap>|--gridmapdir <dir>] [--date|-d Ymd] [--nocapacity] [-n|--dryrun] [--pbsaccdir dir] [--progacc command] [--progcapacity command]
and in accounting.ncm-cron.cron:
15 0 * * * root (date --iso-8601=seconds --utc; PATH=/sbin:/bin:/usr/sbin:/usr/bin:/share/acct/bin; /share/acct/bin/merge_new_accdata) >> /var/log/accounting.ncm-cron.log 2>&1
PS: the ndpf-acctfeed meta-tool replaces the historic 'merge_new_accdata' script.
Collecting data from PBS/Torque
The file /etc/pbsaccdb.conf must be present on the collecting system (i.e. the Torque server) and be formatted as described in NDPFAccouting_pbsaccdbconf.