Moved config templates to a single directory

docs/manual/http_api.md

@@ -0,0 +1,122 @@
+HTTP API
+========
+
+This section covers the HTTP API used to interact with the Near-Line Data
+Store (NLDS), and the HTTP messages that must be sent to the API to carry out
+tasks with the NLDS.  It is divided upon task type:
+
+* [PUT / PUTLIST command](#put-command)
+* [PUTLIST command](#putlist-command)
+* [GET command](#get-command)
+* [GETLIST command](#getlist-command)
+* [DEL command](#del-command)
+* [DELLIST command](#dellist-command)
+
+### PUT / PUTLIST command
+
+The `PUT` and `PUTLIST` commands have been combined into a single HTTP API function, as transferring
+a single file is just the same as transferring a list of files - except there is only one filepath
+in the list.  Therefore, the filepath(s) are specified in the body of the HTTP message, and the other
+parameters, some of which are optional, are specified in the header / input parameter part of the message.
+Any optional metadata parameters, such as the holding label, id or tags are also specified in the body.
+
+| API endpoint | /files |
+|---|---|
+| HTTP method  | PUT |
+| Parameters   | transaction_id: `UUID` |
+|              | token: `str` (OAuth2 token) |
+|              | user: `str` |
+|              | group: `str` |
+|              | tenancy: `str` (Optional) |
+|              | job_label: `str` (Optional) |
+|              | access_key: `str` |
+|              | secret_key: `str` |
+| Body         | *JSON*: `Dict {"filelist" : List<str>, "label": str, "tag": Dict<str, str>, "holding_id": int}` |
+| Example      | `/files/put?transaction_id=1;user="bob";group="root"` |
+| Body example | `{"filelist" : ["file1", "file2", "file3"]}`|
+
+
+### GET command
+
+The `GET` command is no longer used by the NLDS client, but it is left in the HTTP API in case
+another application wishes to use a very simple GET function.  All of the parameters, some of which
+are optional, are specified in the header / input parameters.  This method does not support the
+full features of specifying a holding label, or id, or tags, as the `GET` HTTP method does not support
+a body.
+
+| API endpoint | /files |
+|---|---|
+| HTTP method  | GET |
+| Parameters   | transaction_id: `UUID` |
+|              | token: `str` (OAuth2 token) |
+|              | user: `str` |
+|              | group: `str` |
+|              | filepath: `str` |
+|              | target: `str` (Optional) |
+|              | tenancy: `str` (Optional) |
+|              | job_label: `str` (Optional) |
+|              | access_key: `str` |
+|              | secret_key: `str` |
+| Body         | none |
+| Example      | `GET /files/transaction_id=1;user="bob";group="root";filepath="myfile.txt"` |
+
+### GETLIST command
+
+The `GETLIST` command is used to get a list of files from the NLDS.  Of course, this list could have
+only a single file in it, meaning that the functionality overlaps with the `GET` command somewhat.
+The `GETLIST` uses the HTTP PUT method (rather than GET) so that a body can be supplied with the
+HTTP message.  It supports the full features of specifying a label, id or tags.
+
+| API endpoint | /files/getlist |
+|---|---|
+| HTTP method  | PUT |
+| Parameters   | transaction_id: `UUID` |
+|              | token: `str` (OAuth2 token) |
+|              | user: `str` |
+|              | group: `str` |
+|              | tenancy: `str` (Optional) |
+|              | target: `str` (Optional) |
+|              | job_label: `str` (Optional) |
+|              | access_key: `str` |
+|              | secret_key: `str` |
+| Body         | *JSON*  `Dict {"filelist" : List<str>, "label": str, "tag": Dict<str, str>, "holding_id": int}`|
+| Example      | `/files/getlist?transaction_id=1;user="bob";group="root";`|
+| Body example | `{"filelist" : ["file1", "file2", "file3"]}`|
+
+<!-- ### DEL command
+
+| API endpoint | /files |
+|---|---|
+| HTTP method  | DELETE |
+| Parameters   | transaction_id |
+|              | user |
+|              | group |
+|              | filepath |
+|              | tenancy |
+|              | access_key |
+|              | secret_key |
+| Body         | none |
+| Example      | `/files/transaction_id=1;user="bob";group="root";filepath="myfile.txt" `| -->
+
+### DEL / DELLIST command
+
+Like the `PUT` / `PUTLIST` command, the `DEL` / `DELLIST` commands are combined into a single HTTP 
+PUT method.  This is because deleting a single file is the same as deleting a list of files, just
+with a single file.  Like the `PUT` / `PUTLIST` command, the filelist is specified in the body,
+along with the metadata commands, with the input parameters / header passing the transaction id,
+user, group, etc.
+
+| API endpoint | /files/dellist |
+|---|---|
+| HTTP method  | PUT |
+| Parameters   | transaction_id: `UUID` |
+|              | token: `str` (OAuth2 token) |
+|              | user: `str` |
+|              | group: `str` |
+|              | tenancy: `str` (Optional) |
+|              | job_label: `str` (Optional) |
+|              | access_key: `str` |
+|              | secret_key: `str` |
+| Body         | *JSON*  `Dict {"filelist" : List<str>, "label": str, "tag": Dict<str, str>, "holding_id": int}`|
+| Example      | `/files/dellist?transaction_id=1;user="bob";group="root"`|
+| Body example | `{"filelist" : ["file1", "file2", "file3"]}`|

docs/manual/making_a_xrd_key.md

@@ -0,0 +1,28 @@
+Making a XrootD key
+===================
+
+To read or write files to antares tape via XrootD, a key needs to be generated
+and then passed to `xrdfs`, `xrdcp` or the Python libraries via two 
+environment variables.
+
+The key can be found in the `nlds-consumer-deploy` repository, in the
+`conf/archive_put/staging_secrets.yaml` file (amongst others).  It is the
+`tapeSecretSettings` Value.
+
+Once you have this secret, create the keytab file from it and change the 
+permissions (xrdfs will not let you use the key in any other state):
+
+    echo "key string from repo" > ~/.nlds-xrd.keytab
+    chmod g-r,o-r ~/.nlds-xrd.keytab
+
+This will allow the user to use `xrdfs` and `xrdcp`.  To allow the nlds to use
+tke key copy it to `/etc/nlds` and change ownership and permissions
+
+    sudo cp ~/.nlds-xrd.keytab /etc/nlds/nlds-xrd.keytab
+    sudo chown root /etc/nlds/nlds-xrd.keytab
+    sudo chmod g-r,o-r /etc/nlds/nlds-xrd.keytab
+
+Then set these environment variables:
+
+    export XrdSecPROTOCOL=sss
+    export XrdSecSSSKT=~/.nlds-xrd.keytab

docs/manual/microservices/template.md

@@ -0,0 +1,26 @@
+Microservices template
+======================
+
+Message input
+-------------
+
+| routing key | message contents |
+|-------------|------------------|
+
+Message mapping
+---------------
+
+| input key | output key |
+|-----------|------------|
+
+Message output
+--------------
+
+| routing key | message contents |
+|-------------|------------------|
+
+Process
+-------
+
+| api action | function |
+|------------|----------|

nlds_processors/catalog/catalog_worker.py

@@ -132,7 +132,6 @@ class CatalogConsumer(RMQC):
     _DB_ECHO = "echo"
     _DEFAULT_TENANCY = "default_tenancy"
     _DEFAULT_TAPE_URL = "default_tape_url"
-    _TARGET_AGGREGATION_SIZE = "target_aggregation_size"
 
     DEFAULT_CONSUMER_CONFIG = {
         _DB_ENGINE: "sqlite",
@@ -144,17 +143,13 @@ class CatalogConsumer(RMQC):
         },
         _DEFAULT_TENANCY: None,
         _DEFAULT_TAPE_URL: None,
-        _TARGET_AGGREGATION_SIZE: 5 * (1024**3),  # Default to 5 GB
     }
 
     def __init__(self, queue=DEFAULT_QUEUE_NAME):
         super().__init__(queue=queue)
 
         self.default_tape_url = self.load_config_value(self._DEFAULT_TAPE_URL)
         self.default_tenancy = self.load_config_value(self._DEFAULT_TENANCY)
-        self.target_aggregation_size = self.load_config_value(
-            self._TARGET_AGGREGATION_SIZE
-        )
 
         self.catalog = None
         self.tapelist = []

nlds_processors/index.py

@@ -102,7 +102,7 @@ def _scan(
         self.log("Starting index scan", RK.LOG_INFO)
 
         # Index the entirety of the passed filelist and check for permissions. The size
-        # of the packet will also be evaluated and used to send lists of roughly equal 
+        # of the packet will also be evaluated and used to send lists of roughly equal
         # size.
         self.index(filelist, rk_parts[0], body_json)
         self.log(f"Scan finished.", RK.LOG_INFO)
@@ -115,7 +115,7 @@ def callback(self, ch, method, properties, body, connection):
         self.log(
             f"Received from {self.queues[0].name} ({method.routing_key})",
             RK.LOG_DEBUG,
-            body_json=body_json
+            body_json=body_json,
         )
 
         # Check for system status