= Using native APEL Client to report usage statistics from ARC CE =
== Installation == 
BLAH logger distributed along with {{{nordugrid-arc-arex}}} package since version 4.1.0, so with a latest version you already have logger installed.

For previous ARC versions you can [#BLAHloggerforARC4.1.0 install standalone package from Blackjack repos].

Install APEL parser and client from EMI repos:
{{{
yum -y localinstall http://emisoft.web.cern.ch/emisoft/dist/EMI/3/sl6/x86_64/base/emi-release-3.0.0-2.el6.noarch.rpm
yum -y install apel-client apel-parsers
}}}

Install MySQL backend for APEL clients. MySQL can be installed on any host, running it on ARC CE host is not mandatory.
{{{
yum -y install mysql-server mysql
chkconfig mysqld on
service mysqld start
mysql_secure_installation
mysql -u root -p
mysql> CREATE DATABASE apelclient;
mysql> GRANT ALL PRIVILEGES ON apelclient.* TO apel@localhost IDENTIFIED BY 'myverysecureapelpassword';
mysql> FLUSH PRIVILEGES;
mysql> quit
}}}

Provision APEL database with the required structures using the SQL file supplied with APEL packages:
{{{
mysql -u apel -p apelclient < /usr/share/apel/client.sql
}}}

== Configuration ==

''1.'' Configure ARC CE to call installed BLAH logger. In the {{{/etc/arc.conf}}} add the following configuration and restart A-REX:
{{{
authplugin="FINISHED timeout=10,onfailure=pass /usr/libexec/arc/arc-blahp-logger -I %I -U %u -L %C/job.%I.local -P %C/job.%I.proxy"
}}}

By default BLAH log is written to {{{/var/log/arc/accounting/blahp.log-YYYYMMDD}}}. The log prefix (without {{{-YYYYMMDD}}}) can be redefined with optional {{{-p}}} option. 

CE ID is generated automatically and has a format {{{host.fqdn:2811/nordugrid-torque-queue}}} in accordance to GLUE1.3 publishing. Queue is added in runtime anyway, but prefix can be redefined with optional {{{-c}}} option.

To debug logger execution you can add {{{-d 5}}} option and see all ARC logger output from all subsystems used.

''2.'' Edit APEL parser configuration file ({{{/etc/apel/parser.cfg}}}) and specify DB access settings, batch system ''accounting'' logs path, and ARC CE BLAH plugin logs path:
{{{
[db]
hostname = localhost
port = 3306
name = apelclient
username = apel
password = myverysecureapelpassword

[site_info]
site_name = GOCBD-SITE-NAME
lrms_server = some.fqdn.lrms.id

[blah]
enabled = true
dir = /var/log/arc/accounting
filename_prefix = blahp.log
subdirs = false

[batch]
enabled = true
reparse = false
type = PBS
dir = /path/to/torque/server_priv/accounting/
filename_prefix = 201
subdirs = false

[logging]
logfile = /var/log/apel/parser.log
level = INFO
console = false
}}}

Ensure that LRMS accounting logs are available on APEL node: configure file sharing (NFS?) accordingly is LRMS is not local to APEL node.

Run {{{apelparser}}} and make sure records are parsed successfully looking into {{{parser.log}}} and of course the database content :-)

If everything is OK, it's time to configure APEL client and broker network to use.

''3.'' Configure APEL broker network client. Edit the {{{/etc/apel/client.cfg}}}:
{{{
[db]
hostname = localhost
port = 3306
name = apelclient
username = apel
password = myverysecureapelpassword

[spec_updater]
enabled = true
site_name = GOCBD-SITE-NAME
ldap_host = bdii.ha.grid.org.ua
ldap_port = 2170

[joiner]
enabled = true
local_jobs = false

[unloader]
enabled = true
dir_location = /var/spool/apel/
send_summaries = true
withhold_dns = false
interval = latest
send_ur = false

[ssm]
enabled = true

[logging]
logfile = /var/log/apel/client.log
level = INFO
console = false
}}}

And finally SSM configuration should be specified in {{{/etc/apel/sender.cfg}}}:
{{{
[broker]
# production
bdii: ldap://bdii.ha.grid.org.ua:2170
network: PROD
use_ssl: true
# testing
#host: test-msg02.afroditi.hellasgrid.gr
#port: 6163
#use_ssl: false

[certificates]
certificate: /etc/grid-security/hostcert.pem
key: /etc/grid-security/hostkey.pem
capath: /etc/grid-security/certificates

[messaging]
# production
destination: /queue/global.accounting.cpu.central
# testing
#destination: /queue/global.accounting.cputest.CENTRAL
path: /var/spool/apel/outgoing

[logging]
logfile: /var/log/apel/ssmsend.log
level: INFO
console: false
}}}

''4.'' Finally you need to create a CRON job to parse logs and sent records regularly. It can be simply {{{/etc/cron.daily/apel}}} that contains:
{{{
#!/bin/sh

/usr/bin/apelparser
/usr/bin/apelclient
}}}

== Operational part to publish data to PROD Network ==

 - declare "gLite-APEL" endpoing in GOCDB for grid-site (this step will add an endpoint DN to APEL's white-list and allows to  publish records)
 - use PROD brockers received from Top-BDII in SSM configuration (see above)
 - check that data appeared in EGI accounting database: http://accounting.egi.eu/egi.php?ExecutingSite=GOCBD-SITE-NAME
 - enable monitoring of EGI accounting database for your grid-site with Nagios tests: declare "APEL" endpoint in GOCDB
 - Nagios check results will be browsable in a few days using URLs: ​http://goc-accounting.grid-support.ac.uk/rss/GOCBD-SITE-NAME_Sync.html and ​http://goc-accounting.grid-support.ac.uk/rss/GOCBD-SITE-NAME_Pub.html
 - expect to see APEL "OK" green box on EGI Dashboard

To speed-up the process you can declare "APEL" and "gLite-APEL" endpoints simultaneously.

== Running ARC CE along with CREAM ==
Not a problem, on your ARC CE you just does not need to parse LRMS logs (once they are already parsed on CREAM box) and does not need to run {{{apelclient}}} for the same reason. All you need is BLAH logger and APEL parser to parse only one more BLAH logs.
{{{
[batch]
enabled = false
}}}

Several ARC CEs within the same grid site can be configured the same way.

== Configuration for SLURM LRMS ==
1. Please refer to the [https://twiki.cern.ch/twiki/pub/EMI/EMI3APELClient/APEL_Parsers_System_Administrator_Guide.pdf APEL Parsers System Administrator Guide] (section 4.3) for instructions how to configure SLURM to write an accounting records for APEL parser.

2. In APEL parser configuration specify {{{SLURM}}} type and corresponding accounting records path.

3. Add and extra argument to ARC BLAH logger:
{{{
authplugin="FINISHED timeout=10,onfailure=pass /usr/libexec/arc/arc-blahp-logger -I %I -U %u -L %C/job.%I.local -P %C/job.%I.proxy -c host.fqdn:2811/nordugrid-SLURM"
}}}


== BLAH logger for ARC < 4.1.0 ==

For ARC 4.0.x install standalone package from Blackjack repo (EL6 only):
{{{
yum -y localinstall http://blackjack.grid.org.ua/pub/linux/rhel/6x/base/x86_64/blackjack-release-6-6.noarch.rpm
yum -y install nordugrid-arc-blahp-logger
}}}

For ARC 3.x you need to specify version 3.0.0 directly (EL6 only):
{{{
yum -y install nordugrid-arc-blahp-logger-3.0.0
}}}

For previous ARC versions you can use [#CompatversionofloggerforolderARCCEversions compat BLAH logger].

BLAH logger also available for EL5 in Blackjack repo, for other OSes you can use [http://trac.grid.org.ua/repos/workarea/arc-misc/arc-blahp-logger/ sources].

=== Compat version of logger for older ARC CE versions ===

Compat version of logger has been written in BASH, depend on {{{arcproxy}}} tool and also available in Blackjack repo:
{{{
yum -y localinstall http://blackjack.grid.org.ua/pub/linux/rhel/6x/base/x86_64/blackjack-release-6-6.noarch.rpm
yum -y install nordugrid-arc-blahp-logger-compat
}}}

The syntax of compat {{{arc-blahp-logger}}} invocation from A-REX is slightly differ from binary version:
{{{
authplugin="FINISHED timeout=10,onfailure=pass /usr/libexec/arc/arc-blahp-logger /var/log/arc/accounting/blahp.log- %W %C %I %u arc.univ.kiev.ua:2811/nordugrid-torque"
}}}
The last parameter is optional like {{{-c}}} in binary version.