= Using native APEL Client to report usage statistics from ARC CE = == Installation (EL6) == Install ARC CE plugin to write logs for APEL BLAH parser from Blackjack repo: {{{ yum -y localinstall http://blackjack.grid.org.ua/pub/linux/rhel/6x/base/x86_64/blackjack-release-6-2.noarch.rpm yum -y install nordugrid-arc-blahp-logger }}} Install APEL parser and client from EMI repos: {{{ yum -y localinstall http://emisoft.web.cern.ch/emisoft/dist/EMI/3/sl6/x86_64/base/emi-release-3.0.0-2.el6.noarch.rpm yum -y install apel-client apel-parsers }}} Install MySQL backend for APEL clients. MySQL can be installed on any host, running it on ARC CE host is not mandatory. {{{ yum -y install mysql-server mysql chkconfig mysqld on service mysqld start mysql_secure_installation mysql -u root -p mysql> CREATE DATABASE apelclient; mysql> GRANT ALL PRIVILEGES ON apelclient.* TO apel@localhost IDENTIFIED BY 'myverysecureapelpassword'; mysql> FLUSH PRIVILEGES; mysql> quit }}} Provision APEL database with the required structures using the SQL file supplied with APEL packages: {{{ mysql -u apel -p apelclient < /usr/share/apel/client.sql }}} == Configuration == ''1.'' Configure ARC CE to call installed BLAH logger. In the {{{/etc/arc.conf}}} add the following configuration and restart A-REX: {{{ authplugin="FINISHED timeout=10,onfailure=pass /usr/libexec/arc/arc-blahp-logger -I %I -U %u -L %C/job.%I.local -P %C/job.%I.proxy" }}} By default BLAH log is written to {{{/var/log/arc/accounting/blahp.log-YYYYMMDD}}}. The log prefix (without {{{-YYYYMMDD}}}) can be redefined with optional {{{-p}}} option. CE ID is generated automatically and has a format {{{host.fqdn:2811/nordugrid-torque-queue}}} in accordance to GLUE1.3 publishing. Queue is added in runtime anyway, but prefix can be redefined with optional {{{-c}}} option. To debug logger execution you can add {{{-d 5}}} option and see all ARC logger output from all subsystems used. ''2.'' Edit APEL parser configuration file ({{{/etc/apel/parser.cfg}}}) and specify DB access settings, batch system ''accounting'' logs path, and ARC CE BLAH plugin logs path: {{{ [db] hostname = localhost port = 3306 name = apelclient username = apel password = myverysecureapelpassword [site_info] site_name = GOCBD-SITE-NAME lrms_server = some.fqdn.lrms.id [blah] enabled = true dir = /var/log/arc/accounting filename_prefix = blahp.log subdirs = false [batch] enabled = true reparse = false type = PBS dir = /path/to/torque/server_priv/accounting/ filename_prefix = 201 subdirs = false [logging] logfile = /var/log/apel/parser.log level = INFO console = false }}} Ensure that LRMS accounting logs are available on APEL node: configure file sharing (NFS?) accordingly is LRMS is not local to APEL node. Run {{{apelparser}}} and make sure records are parsed successfully looking into {{{parser.log}}} and of course the database content :-) If everything is OK, it's time to configure APEL client and broker network to use. ''3.'' Configure APEL broker network client. Edit the {{{/etc/apel/client.cfg}}}: {{{ [db] hostname = localhost port = 3306 name = apelclient username = apel password = myverysecureapelpassword [spec_updater] enabled = true site_name = GOCBD-SITE-NAME ldap_host = bdii.ha.grid.org.ua ldap_port = 2170 [joiner] enabled = true local_jobs = false [unloader] enabled = true dir_location = /var/spool/apel/ send_summaries = true withhold_dns = false interval = latest send_ur = false [ssm] enabled = true [logging] logfile = /var/log/apel/client.log level = INFO console = false }}} And finally SSM configuration should be specified in {{{/etc/apel/sender.cfg}}}: {{{ [broker] # production bdii: ldap://bdii.ha.grid.org.ua:2170 network: PROD use_ssl: true # testing #host: test-msg02.afroditi.hellasgrid.gr #port: 6163 #use_ssl: false [certificates] certificate: /etc/grid-security/hostcert.pem key: /etc/grid-security/hostkey.pem capath: /etc/grid-security/certificates [messaging] # production destination: /queue/global.accounting.cpu.central # testing #destination: /queue/global.accounting.cputest.CENTRAL path: /var/spool/apel/outgoing [logging] logfile: /var/log/apel/ssmsend.log level: INFO console: false }}} ''4.'' Finally you need to create a CRON job to parse logs and sent records regularly. It can be simply {{{/etc/cron.daily/apel}}} that contains: {{{ #!/bin/sh /usr/bin/apelparser /usr/bin/apelclient }}} == Operational part to publish data to PROD Network == - declare "gLite-APEL" endpoing in GOCDB for grid-site (this step will add an endpoint DN to APEL's white-list and allows to publish records) - use PROD brockers received from Top-BDII in SSM configuration (see above) - check that data appeared in EGI accounting database: http://accounting.egi.eu/egi.php?ExecutingSite=GOCBD-SITE-NAME - enable monitoring of EGI accounting database for your grid-site with Nagios tests: declare "APEL" endpoint in GOCDB - Nagios check results will be browsable in a few days using URLs: ​http://goc-accounting.grid-support.ac.uk/rss/GOCBD-SITE-NAME_Sync.html and ​http://goc-accounting.grid-support.ac.uk/rss/GOCBD-SITE-NAME_Pub.html - expect to see APEL "OK" green box on EGI Dashboard To speed-up the process you can declare "APEL" and "gLite-APEL" endpoints simultaneously. == Compat version of logger for older ARC CE versions == Compat version of logger has been written in BASH, depend on {{{arcproxy}}} tool and also available in Blackjack repo: {{{ yum -y localinstall http://blackjack.grid.org.ua/pub/linux/rhel/6x/base/x86_64/blackjack-release-6-2.noarch.rpm yum -y install nordugrid-arc-blahp-logger-compat }}} The syntax of compat {{{arc-blahp-logger}}} invocation from A-REX is slightly differ from binary version: {{{ authplugin="FINISHED timeout=10,onfailure=pass /usr/libexec/arc/arc-blahp-logger /var/log/arc/accounting/blahp.log- %W %C %I %u arc.univ.kiev.ua:2811/nordugrid-torque" }}} The last parameter is optional like {{{-c}}} in binary version. == Running ARC CE along with CREAM == Not a problem, on your ARC CE you just does not need to parse LRMS logs (once they are already parsed on CREAM box) and does not need to run {{{apelclient}}} for the same reason. All you need is BLAH logger and APEL parser to parse only one more BLAH logs. {{{ [batch] enabled = false }}}