Admin FAQ

Harvester Admin Tool

Harvester Admin Tool is available since Harvester version 0.0.20, which provides the command for harvester administrative operations.

Setup

After installation of Harvester, under {harvester_venv}/local/bin/ there is a script template harvester-admin.rpmnew.template . Copy it to be harvester-admin and modify it: Set the userName and VIRTUAL_ENV according to the user to run the admin tool and the harvester venv respectively. E.g. when venv directory is /usr :

# cp /usr/local/bin/harvester-admin.rpmnew.template /usr/local/bin/harvester-admin
# vim /usr/local/bin/harvester-admin

Usage

One can then run harvester-admin command. Option -h after any command/sub-command provides help message. Some examples below.

Show help:

# /usr/local/bin/harvester-admin -h
usage: harvester-admin [-h] [-v] {test,qconf,fifo} ...

positional arguments:
  {test,qconf,fifo}
    test                Test
    fifo                fifo related
    qconf               queue configuration

optional arguments:
  -h, --help            show this help message and exit
  -v, --verbose, --debug
                        Print more verbose output. (Debug mode !)

Admin tool test:

# /usr/local/bin/harvester-admin -v test
[2018-07-19 10:25:59,287 CRITICAL] Harvester Admin Tool: test CRITICAL
[2018-07-19 10:25:59,287 ERROR] Harvester Admin Tool: test ERROR
[2018-07-19 10:25:59,287 WARNING] Harvester Admin Tool: test WARNING
[2018-07-19 10:25:59,287 INFO] Harvester Admin Tool: test INFO
[2018-07-19 10:25:59,287 DEBUG] Harvester Admin Tool: test DEBUG
Harvester Admin Tool: test
[2018-07-19 10:25:59,287 DEBUG] ARGS: Namespace(debug=True, which='test') ; RESULT: None
[2018-07-19 10:25:59,288 DEBUG] Action completed in 0.001 seconds

Show help of qconf (queue configuration) sub-command:

# /usr/local/bin/harvester-admin qconf -h
usage: harvester-admin qconf [-h] {purge,list,dump} ...

positional arguments:
  {purge,list,dump}
    list             List queues. Only active queues listed by default
    dump             Dump queue configurations
    purge            Purge the queue thoroughly from harvester DB (Be careful
                     !!)

optional arguments:
  -h, --help         show this help message and exit

List all queue configurations in harvester:

# /usr/local/bin/harvester-admin qconf list -a
configID : queue name
--------- ------------
       2 : BNL_PROD
      33 : CERN-EXTENSION_HARVESTER
     101 : Taiwan-LCG2-HPC_Unified
     174 : BNL_PROD
...

FIFO

What is FIFO ?

FIFO in harvester is an optional feature that harvester agents can take advantage of message queue data structure. Main purpose FIFO is to reduce DB polling frequency and lower CPU usage of the node.

The FIFO has "Priority Queue" data structure. Different plugins can be chosen as fifo backend. Existing plugins: SQLite, Redis

So far, only monitor agent has option to enable FIFO. The monitor FIFO allow configurable priority of each PQ. Worker chunks with shorter fifoCheckInterval has higher priority and will be checked more frequently.

How to set up FIFO ?

Choose a FIFO backend and set up related service. Then, configure in harvester.

SQLite

Backend setup:

• Make sure sqlite3 is installed in OS.
• No special service configuration required.

Harvester configuration:

• In panda_harvester.cfg, [fifo] section, one should set fifoModule = pandaharvester.harvesterfifo.sqlite_fifo and fifoClass = SqliteFifo to use the SqliteFifo fifo plugin.

• The database_filename should be specified as the database filename for sqlite. This must be different from main Harvester DB and other fifo DBs if using sqlite.

• It is recommended to use placeholder $(AGENT) in filename to make different DBs for fifo of different agents.

• One can set DB file located in ramdisk for better performance.

• E.g.

  [fifo]
  fifoModule = pandaharvester.harvesterfifo.sqlite_fifo
  fifoClass = SqliteFifo
  database_filename = /dev/shm/$(AGENT)_fifo.db
  

Redis

Backend setup:

• Make sure the redis service is installed, configured, and running on the harvester node.

• Install python redis via pip within harvester python venv:

  $ pip install redis
  

Harvester configuration:

• In panda_harvester.cfg, [fifo] section, one should set fifoModule = pandaharvester.harvesterfifo.redis_fifo and fifoClass = RedisFifo to use the RedisFifo fifo plugin. This configuration is enough here if one sets up a standalone redis service on localhost with default port, without password nor db specified.

• If the redis service has specific host, port, db, and password, then respectively set redisHost, redisPort, redisDB, redisPassword accordingly.

• E.g.

  [fifo]
  fifoModule = pandaharvester.harvesterfifo.redis_fifo
  fifoClass = RedisFifo
  

Test and benchmark:

(TODO)