Using native APEL Client to report usage statistics from ARC CE
Installation
BLAH logger distributed along with nordugrid-arc-arex package since version 4.1.0, so with a latest version you already have logger installed.
For previous ARC versions you can install standalone package from Blackjack repos.
Install APEL parser and client from EMI repos:
yum -y localinstall http://emisoft.web.cern.ch/emisoft/dist/EMI/3/sl6/x86_64/base/emi-release-3.0.0-2.el6.noarch.rpm yum -y install apel-client apel-parsers
Install MySQL backend for APEL clients. MySQL can be installed on any host, running it on ARC CE host is not mandatory.
yum -y install mysql-server mysql chkconfig mysqld on service mysqld start mysql_secure_installation mysql -u root -p mysql> CREATE DATABASE apelclient; mysql> GRANT ALL PRIVILEGES ON apelclient.* TO apel@localhost IDENTIFIED BY 'myverysecureapelpassword'; mysql> FLUSH PRIVILEGES; mysql> quit
Provision APEL database with the required structures using the SQL file supplied with APEL packages:
mysql -u apel -p apelclient < /usr/share/apel/client.sql
Configuration
1. Configure ARC CE to call installed BLAH logger. In the /etc/arc.conf add the following configuration and restart A-REX:
authplugin="FINISHED timeout=10,onfailure=pass /usr/libexec/arc/arc-blahp-logger -I %I -U %u -L %C/job.%I.local -P %C/job.%I.proxy"
By default BLAH log is written to /var/log/arc/accounting/blahp.log-YYYYMMDD. The log prefix (without -YYYYMMDD) can be redefined with optional -p option.
CE ID is generated automatically and has a format host.fqdn:2811/nordugrid-torque-queue in accordance to GLUE1.3 publishing. Queue is added in runtime anyway, but prefix can be redefined with optional -c option.
To debug logger execution you can add -d 5 option and see all ARC logger output from all subsystems used.
2. Edit APEL parser configuration file (/etc/apel/parser.cfg) and specify DB access settings, batch system accounting logs path, and ARC CE BLAH plugin logs path:
[db] hostname = localhost port = 3306 name = apelclient username = apel password = myverysecureapelpassword [site_info] site_name = GOCBD-SITE-NAME lrms_server = some.fqdn.lrms.id [blah] enabled = true dir = /var/log/arc/accounting filename_prefix = blahp.log subdirs = false [batch] enabled = true reparse = false type = PBS dir = /path/to/torque/server_priv/accounting/ filename_prefix = 201 subdirs = false [logging] logfile = /var/log/apel/parser.log level = INFO console = false
Ensure that LRMS accounting logs are available on APEL node: configure file sharing (NFS?) accordingly is LRMS is not local to APEL node.
Run apelparser and make sure records are parsed successfully looking into parser.log and of course the database content :-)
If everything is OK, it's time to configure APEL client and broker network to use.
3. Configure APEL broker network client. Edit the /etc/apel/client.cfg:
[db] hostname = localhost port = 3306 name = apelclient username = apel password = myverysecureapelpassword [spec_updater] enabled = true site_name = GOCBD-SITE-NAME ldap_host = bdii.ha.grid.org.ua ldap_port = 2170 [joiner] enabled = true local_jobs = false [unloader] enabled = true dir_location = /var/spool/apel/ send_summaries = true withhold_dns = false interval = latest send_ur = false [ssm] enabled = true [logging] logfile = /var/log/apel/client.log level = INFO console = false
And finally SSM configuration should be specified in /etc/apel/sender.cfg:
[broker] # production bdii: ldap://bdii.ha.grid.org.ua:2170 network: PROD use_ssl: true # testing #host: test-msg02.afroditi.hellasgrid.gr #port: 6163 #use_ssl: false [certificates] certificate: /etc/grid-security/hostcert.pem key: /etc/grid-security/hostkey.pem capath: /etc/grid-security/certificates [messaging] # production destination: /queue/global.accounting.cpu.central # testing #destination: /queue/global.accounting.cputest.CENTRAL path: /var/spool/apel/outgoing [logging] logfile: /var/log/apel/ssmsend.log level: INFO console: false
4. Finally you need to create a CRON job to parse logs and sent records regularly. It can be simply /etc/cron.daily/apel that contains:
#!/bin/sh /usr/bin/apelparser /usr/bin/apelclient
Operational part to publish data to PROD Network
- declare "gLite-APEL" endpoing in GOCDB for grid-site (this step will add an endpoint DN to APEL's white-list and allows to publish records)
- use PROD brockers received from Top-BDII in SSM configuration (see above)
- check that data appeared in EGI accounting database: http://accounting.egi.eu/egi.php?ExecutingSite=GOCBD-SITE-NAME
- enable monitoring of EGI accounting database for your grid-site with Nagios tests: declare "APEL" endpoint in GOCDB
- Nagios check results will be browsable in a few days using URLs: http://goc-accounting.grid-support.ac.uk/rss/GOCBD-SITE-NAME_Sync.html and http://goc-accounting.grid-support.ac.uk/rss/GOCBD-SITE-NAME_Pub.html
- expect to see APEL "OK" green box on EGI Dashboard
To speed-up the process you can declare "APEL" and "gLite-APEL" endpoints simultaneously.
Running ARC CE along with CREAM
Not a problem, on your ARC CE you just does not need to parse LRMS logs (once they are already parsed on CREAM box) and does not need to run apelclient for the same reason. All you need is BLAH logger and APEL parser to parse only one more BLAH logs.
[batch] enabled = false
Several ARC CEs within the same grid site can be configured the same way.
Configuration for SLURM LRMS
- Please refer to the APEL Parsers System Administrator Guide (section 4.3) for instructions how to configure SLURM to write an accounting records for APEL parser.
- In APEL parser configuration specify SLURM type and corresponding accounting records path.
- Add and extra argument to ARC BLAH logger:
authplugin="FINISHED timeout=10,onfailure=pass /usr/libexec/arc/arc-blahp-logger -I %I -U %u -L %C/job.%I.local -P %C/job.%I.proxy -c host.fqdn:2811/nordugrid-SLURM"
BLAH logger for ARC < 4.1.0
For ARC 4.0.x install standalone package from Blackjack repo (EL6 only):
yum -y localinstall http://blackjack.grid.org.ua/pub/linux/rhel/6x/base/x86_64/blackjack-release-6-6.noarch.rpm yum -y install nordugrid-arc-blahp-logger
For ARC 3.x you need to specify version 3.0.0 directly (EL6 only):
yum -y install nordugrid-arc-blahp-logger-3.0.0
For previous ARC versions you can use compat BLAH logger.
BLAH logger also available for EL5 in Blackjack repo, for other OSes you can use sources.
Compat version of logger for older ARC CE versions
Compat version of logger has been written in BASH, depend on arcproxy tool and also available in Blackjack repo:
yum -y localinstall http://blackjack.grid.org.ua/pub/linux/rhel/6x/base/x86_64/blackjack-release-6-6.noarch.rpm yum -y install nordugrid-arc-blahp-logger-compat
The syntax of compat arc-blahp-logger invocation from A-REX is slightly differ from binary version:
authplugin="FINISHED timeout=10,onfailure=pass /usr/libexec/arc/arc-blahp-logger /var/log/arc/accounting/blahp.log- %W %C %I %u arc.univ.kiev.ua:2811/nordugrid-torque"
The last parameter is optional like -c in binary version.
