diff --git a/docs/manual/http_api.md b/docs/manual/http_api.md new file mode 100644 index 00000000..89fdc1fb --- /dev/null +++ b/docs/manual/http_api.md @@ -0,0 +1,122 @@ +HTTP API +======== + +This section covers the HTTP API used to interact with the Near-Line Data +Store (NLDS), and the HTTP messages that must be sent to the API to carry out +tasks with the NLDS. It is divided upon task type: + +* [PUT / PUTLIST command](#put-command) +* [PUTLIST command](#putlist-command) +* [GET command](#get-command) +* [GETLIST command](#getlist-command) +* [DEL command](#del-command) +* [DELLIST command](#dellist-command) + +### PUT / PUTLIST command + +The `PUT` and `PUTLIST` commands have been combined into a single HTTP API function, as transferring +a single file is just the same as transferring a list of files - except there is only one filepath +in the list. Therefore, the filepath(s) are specified in the body of the HTTP message, and the other +parameters, some of which are optional, are specified in the header / input parameter part of the message. +Any optional metadata parameters, such as the holding label, id or tags are also specified in the body. + +| API endpoint | /files | +|---|---| +| HTTP method | PUT | +| Parameters | transaction_id: `UUID` | +| | token: `str` (OAuth2 token) | +| | user: `str` | +| | group: `str` | +| | tenancy: `str` (Optional) | +| | job_label: `str` (Optional) | +| | access_key: `str` | +| | secret_key: `str` | +| Body | *JSON*: `Dict {"filelist" : List, "label": str, "tag": Dict, "holding_id": int}` | +| Example | `/files/put?transaction_id=1;user="bob";group="root"` | +| Body example | `{"filelist" : ["file1", "file2", "file3"]}`| + + +### GET command + +The `GET` command is no longer used by the NLDS client, but it is left in the HTTP API in case +another application wishes to use a very simple GET function. All of the parameters, some of which +are optional, are specified in the header / input parameters. This method does not support the +full features of specifying a holding label, or id, or tags, as the `GET` HTTP method does not support +a body. + +| API endpoint | /files | +|---|---| +| HTTP method | GET | +| Parameters | transaction_id: `UUID` | +| | token: `str` (OAuth2 token) | +| | user: `str` | +| | group: `str` | +| | filepath: `str` | +| | target: `str` (Optional) | +| | tenancy: `str` (Optional) | +| | job_label: `str` (Optional) | +| | access_key: `str` | +| | secret_key: `str` | +| Body | none | +| Example | `GET /files/transaction_id=1;user="bob";group="root";filepath="myfile.txt"` | + +### GETLIST command + +The `GETLIST` command is used to get a list of files from the NLDS. Of course, this list could have +only a single file in it, meaning that the functionality overlaps with the `GET` command somewhat. +The `GETLIST` uses the HTTP PUT method (rather than GET) so that a body can be supplied with the +HTTP message. It supports the full features of specifying a label, id or tags. + +| API endpoint | /files/getlist | +|---|---| +| HTTP method | PUT | +| Parameters | transaction_id: `UUID` | +| | token: `str` (OAuth2 token) | +| | user: `str` | +| | group: `str` | +| | tenancy: `str` (Optional) | +| | target: `str` (Optional) | +| | job_label: `str` (Optional) | +| | access_key: `str` | +| | secret_key: `str` | +| Body | *JSON* `Dict {"filelist" : List, "label": str, "tag": Dict, "holding_id": int}`| +| Example | `/files/getlist?transaction_id=1;user="bob";group="root";`| +| Body example | `{"filelist" : ["file1", "file2", "file3"]}`| + + + +### DEL / DELLIST command + +Like the `PUT` / `PUTLIST` command, the `DEL` / `DELLIST` commands are combined into a single HTTP +PUT method. This is because deleting a single file is the same as deleting a list of files, just +with a single file. Like the `PUT` / `PUTLIST` command, the filelist is specified in the body, +along with the metadata commands, with the input parameters / header passing the transaction id, +user, group, etc. + +| API endpoint | /files/dellist | +|---|---| +| HTTP method | PUT | +| Parameters | transaction_id: `UUID` | +| | token: `str` (OAuth2 token) | +| | user: `str` | +| | group: `str` | +| | tenancy: `str` (Optional) | +| | job_label: `str` (Optional) | +| | access_key: `str` | +| | secret_key: `str` | +| Body | *JSON* `Dict {"filelist" : List, "label": str, "tag": Dict, "holding_id": int}`| +| Example | `/files/dellist?transaction_id=1;user="bob";group="root"`| +| Body example | `{"filelist" : ["file1", "file2", "file3"]}`| \ No newline at end of file diff --git a/docs/manual/making_a_xrd_key.md b/docs/manual/making_a_xrd_key.md new file mode 100644 index 00000000..e01d3361 --- /dev/null +++ b/docs/manual/making_a_xrd_key.md @@ -0,0 +1,28 @@ +Making a XrootD key +=================== + +To read or write files to antares tape via XrootD, a key needs to be generated +and then passed to `xrdfs`, `xrdcp` or the Python libraries via two +environment variables. + +The key can be found in the `nlds-consumer-deploy` repository, in the +`conf/archive_put/staging_secrets.yaml` file (amongst others). It is the +`tapeSecretSettings` Value. + +Once you have this secret, create the keytab file from it and change the +permissions (xrdfs will not let you use the key in any other state): + + echo "key string from repo" > ~/.nlds-xrd.keytab + chmod g-r,o-r ~/.nlds-xrd.keytab + +This will allow the user to use `xrdfs` and `xrdcp`. To allow the nlds to use +tke key copy it to `/etc/nlds` and change ownership and permissions + + sudo cp ~/.nlds-xrd.keytab /etc/nlds/nlds-xrd.keytab + sudo chown root /etc/nlds/nlds-xrd.keytab + sudo chmod g-r,o-r /etc/nlds/nlds-xrd.keytab + +Then set these environment variables: + + export XrdSecPROTOCOL=sss + export XrdSecSSSKT=~/.nlds-xrd.keytab \ No newline at end of file diff --git a/docs/manual/manual.md b/docs/manual/manual.md new file mode 100644 index 00000000..e69de29b diff --git a/docs/manual/microservices/template.md b/docs/manual/microservices/template.md new file mode 100644 index 00000000..d20dcf04 --- /dev/null +++ b/docs/manual/microservices/template.md @@ -0,0 +1,26 @@ +Microservices template +====================== + +Message input +------------- + +| routing key | message contents | +|-------------|------------------| + +Message mapping +--------------- + +| input key | output key | +|-----------|------------| + +Message output +-------------- + +| routing key | message contents | +|-------------|------------------| + +Process +------- + +| api action | function | +|------------|----------| \ No newline at end of file diff --git a/docs/spec/uml/consumer_classes.png b/docs/spec/uml/consumer_classes.png new file mode 100644 index 00000000..4e618da9 Binary files /dev/null and b/docs/spec/uml/consumer_classes.png differ diff --git a/nlds/templates/server_config.j2 b/nlds/templates/server_config.j2 deleted file mode 100644 index cec783a0..00000000 --- a/nlds/templates/server_config.j2 +++ /dev/null @@ -1,131 +0,0 @@ -{ - "authentication" : { - "authenticator_backend" : "jasmin_authenticator", - "jasmin_authenticator" : { - "user_profile_url" : "{{ user_profile_url }}", - "user_services_url" : "{{ user_services_url }}", - "oauth_token_introspect_url" : "{{ token_introspect_url }}" - } - }, - "rabbitMQ": { - "user": "{{ rabbit_user }}", - "password": "{{ rabbit_password }}", - "heartbeat": "{{ rabbit_heartbeat }}", - "server": "{{ rabbit_server }}", - "admin_port": "{{ rabbit_port }}", - "vhost": "{{ rabbit_vhost }}", - "exchange": { - "name": "{{ rabbit_exchange_name }}", - "type": "{{ rabbit_exchange_type }}", - "delayed": "{{ rabbit_exchange_delayed }}" - }, - "queues": [ - { - "name": "{{ rabbit_queue_name }}", - "bindings": [ - { - "exchange": "{{ rabbit_exchange_name }}", - "routing_key": "{{ rabbit_queue_routing_key }}" - } - ] - } - ] - }, - "logging": { - "enable": "{{ logging_enable }}", - "log_level": "{{ logging_log_level }}", - "log_format": "{{ logging_log_format }}", - "add_stdout_fl": "{{ logging_add_stdout_fl }}", - "stdout_log_level": "{{ logging_stdout_log_level }}", - "log_files": "{{ logging_log_files }}", - "max_bytes": "{{ logging_max_bytes }}", - "backup_count": "{{ logging_backup_count }}" - }, - "nlds_q": { - "logging": "{{ nlds_q_logging_dict }}", - "print_tracebacks_fl": "{{ nlds_q_print_tracebacks_fl }}" - }, - "index_q": { - "logging": "{{ index_q_logging_dict }}", - "print_tracebacks_fl": "{{ index_q_print_tracebacks_fl }}", - "filelist_max_length": "{{ index_q_filelist_max_length }}", - "message_threshold": "{{ index_q_message_threshold }}", - "check_filesize_fl": "{{ index_q_check_filesize_fl }}", - "max_filesize": "{{ index_q_max_filesize }}" - }, - "catalog_q": { - "logging": "{{ catalog_q_logging_dict }}", - "print_tracebacks_fl": "{{ catalog_q_print_tracebacks_fl }}", - "db_engine": "{{ catalog_q_db_engine }}", - "db_options": { - "db_name" : "{{ catalog_q_db_options_db_name }}", - "db_user" : "{{ catalog_q_db_options_db_user }}", - "db_passwd" : "{{ catalog_q_db_options_db_passwd }}", - "echo": "{{ catalog_q_db_options_echo }}" - }, - "default_tenancy": "{{ catalog_q_default_tenancy }}", - "default_tape_url": "{{ catalog_q_default_tape_url }}" - }, - "monitor_q": { - "logging": "{{ monitor_q_logging_dict }}", - "print_tracebacks_fl": "{{ monitor_q_print_tracebacks_fl }}", - "db_engine": "{{ monitor_q_db_engine }}", - "db_options": { - "db_name" : "{{ monitor_q_db_options_db_name }}", - "db_user" : "{{ monitor_q_db_options_db_user }}", - "db_passwd" : "{{ monitor_q_db_options_db_passwd }}", - "echo": "{{ monitor_q_db_options_echo }}" - } - }, - "transfer_put_q": { - "logging": "{{ transfer_put_q_logging_dict }}", - "print_tracebacks_fl": "{{ transfer_put_q_print_tracebacks_fl }}", - "filelist_max_length": "{{ transfer_put_q_filelist_max_length }}", - "tenancy": "{{ transfer_put_q_tenancy }}", - "require_secure_fl": "{{ transfer_put_q_require_secure_fl }}" - }, - "transfer_get_q": { - "logging": "{{ transfer_put_q_logging_dict }}", - "print_tracebacks_fl": "{{ transfer_put_q_print_tracebacks_fl }}", - "filelist_max_length": "{{ transfer_put_q_filelist_max_length }}", - "tenancy": "{{ transfer_put_q_tenancy }}", - "require_secure_fl": "{{ transfer_put_q_require_secure_fl }}", - "chown_fl": "{{ transfer_put_q_chown_fl }}", - "chown_cmd": "{{ transfer_put_q_chown_cmd }}" - }, - "logging_q": { - "logging": "{{ logging_q_logging_dict }}", - "print_tracebacks_fl": "{{ logging_q_print_tracebacks_fl }}" - }, - "archive_put_q": { - "logging": "{{ archive_put_q_logging_dict }}", - "print_tracebacks_fl": "{{ archive_put_q_print_tracebacks_fl }}", - "tenancy": "{{ archive_put_q_tenancy }}", - "require_secure_fl": "{{ archive_put_q_require_secure_fl }}", - "tape_url": "{{ archive_put_q_tape_url }}", - "tape_pool": "{{ archive_put_q_tape_pool }}", - "query_checksum_fl": "{{ archive_put_q_query_checksum_fl }}", - "chunk_size": "{{ archive_put_q_chunk_size }}" - }, - "archive_get_q": { - "logging": "{{ archive_get_q_logging_dict }}", - "print_tracebacks_fl": "{{ archive_get_q_print_tracebacks_fl }}", - "tenancy": "{{ archive_get_q_tenancy }}", - "require_secure_fl": "{{ archive_get_q_require_secure_fl }}", - "tape_url": "{{ archive_get_q_tape_url }}", - "tape_pool": "{{ archive_get_q_tape_pool }}", - "query_checksum_fl": "{{ archive_get_q_query_checksum_fl }}", - "chunk_size": "{{ archive_get_q_chunk_size }}", - "prepare_requeue": "{{ archive_get_q_prepare_requeue }}" - }, - "rpc_publisher": { - "time_limit": "{{ rpc_publisher_time_limit }}", - "queue_exclusivity_fl": "{{ rpc_publisher_queue_exclusivity_fl }}" - }, - "cronjob_publisher": { - "access_key": "{{ cronjob_publisher_access_key }}", - "secret_key": "{{ cronjob_publisher_secret_key }}", - "tenancy": "{{ cronjob_publisher_tenancy }}", - "tape_url": "{{ cronjob_publisher_tape_url }}" - } -} diff --git a/nlds_processors/catalog/catalog_worker.py b/nlds_processors/catalog/catalog_worker.py index b58897f8..5525a833 100644 --- a/nlds_processors/catalog/catalog_worker.py +++ b/nlds_processors/catalog/catalog_worker.py @@ -132,7 +132,6 @@ class CatalogConsumer(RMQC): _DB_ECHO = "echo" _DEFAULT_TENANCY = "default_tenancy" _DEFAULT_TAPE_URL = "default_tape_url" - _TARGET_AGGREGATION_SIZE = "target_aggregation_size" DEFAULT_CONSUMER_CONFIG = { _DB_ENGINE: "sqlite", @@ -144,7 +143,6 @@ class CatalogConsumer(RMQC): }, _DEFAULT_TENANCY: None, _DEFAULT_TAPE_URL: None, - _TARGET_AGGREGATION_SIZE: 5 * (1024**3), # Default to 5 GB } def __init__(self, queue=DEFAULT_QUEUE_NAME): @@ -152,9 +150,6 @@ def __init__(self, queue=DEFAULT_QUEUE_NAME): self.default_tape_url = self.load_config_value(self._DEFAULT_TAPE_URL) self.default_tenancy = self.load_config_value(self._DEFAULT_TENANCY) - self.target_aggregation_size = self.load_config_value( - self._TARGET_AGGREGATION_SIZE - ) self.catalog = None self.tapelist = [] diff --git a/nlds_processors/index.py b/nlds_processors/index.py index bffdd96d..5f61cd8b 100644 --- a/nlds_processors/index.py +++ b/nlds_processors/index.py @@ -102,7 +102,7 @@ def _scan( self.log("Starting index scan", RK.LOG_INFO) # Index the entirety of the passed filelist and check for permissions. The size - # of the packet will also be evaluated and used to send lists of roughly equal + # of the packet will also be evaluated and used to send lists of roughly equal # size. self.index(filelist, rk_parts[0], body_json) self.log(f"Scan finished.", RK.LOG_INFO) @@ -115,7 +115,7 @@ def callback(self, ch, method, properties, body, connection): self.log( f"Received from {self.queues[0].name} ({method.routing_key})", RK.LOG_DEBUG, - body_json=body_json + body_json=body_json, ) # Check for system status diff --git a/nlds_processors/templates/processor_config.j2 b/nlds_processors/templates/processor_config.j2 deleted file mode 100644 index d3a5f049..00000000 --- a/nlds_processors/templates/processor_config.j2 +++ /dev/null @@ -1,18 +0,0 @@ -{ - "rabbitMQ": { - "queues": [ - { - "name": "{{ rabbit_queue_name }}", - "bindings": [ - { - "exchange": "{{ rabbit_exchange_name }}", - "routing_key": "{{ rabbit_routing_key }}" - } - ] - } - ] - }, - "{{ rabbit_queue_name }}": { - {{ rabbit_queue_options }} - } -} \ No newline at end of file diff --git a/nlds_processors/templates/queues/archive_del_q.j2 b/nlds_processors/templates/queues/archive_del_q.j2 deleted file mode 100644 index 52f7d89b..00000000 --- a/nlds_processors/templates/queues/archive_del_q.j2 +++ /dev/null @@ -1,15 +0,0 @@ -{ - "rabbitMQ": { - "queues": [ - { - "name": "archive_del_q", - "bindings": [ - { - "exchange": "{{ rabbit_exchange_name }}", - "routing_key": "*.archive-del.start" - } - ] - } - ] - } -} \ No newline at end of file diff --git a/nlds_processors/templates/queues/archive_put_q.j2 b/nlds_processors/templates/queues/archive_put_q.j2 deleted file mode 100644 index b176e165..00000000 --- a/nlds_processors/templates/queues/archive_put_q.j2 +++ /dev/null @@ -1,19 +0,0 @@ -{ - "rabbitMQ": { - "queues": [ - { - "name": "archive_get_q", - "bindings": [ - { - "exchange": "{{ rabbit_exchange_name }}", - "routing_key": "*.archive-put.start" - }, - { - "exchange": "{{ rabbit_exchange_name }}", - "routing_key": "*.archive-put.init" - } - ] - } - ] - } -} \ No newline at end of file diff --git a/nlds_processors/templates/queues/logging_q.j2 b/nlds_processors/templates/queues/logging_q.j2 deleted file mode 100644 index 04baa595..00000000 --- a/nlds_processors/templates/queues/logging_q.j2 +++ /dev/null @@ -1,15 +0,0 @@ -{ - "rabbitMQ": { - "queues": [ - { - "name": "logging_q", - "bindings": [ - { - "exchange": "{{ rabbit_exchange_name }}", - "routing_key": "*.log.*" - } - ] - } - ] - } -} diff --git a/nlds_processors/templates/queues/transfer_get_q.j2 b/nlds_processors/templates/queues/transfer_get_q.j2 deleted file mode 100644 index ae8b285e..00000000 --- a/nlds_processors/templates/queues/transfer_get_q.j2 +++ /dev/null @@ -1,19 +0,0 @@ -{ - "rabbitMQ": { - "queues": [ - { - "name": "transfer_get_q", - "bindings": [ - { - "exchange": "{{ rabbit_exchange_name }}", - "routing_key": "*.transfer-get.start" - }, - { - "exchange": "{{ rabbit_exchange_name }}", - "routing_key": "*.transfer-get.init" - } - ] - } - ] - } -} \ No newline at end of file diff --git a/nlds_processors/templates/queues/transfer_put_q.j2 b/nlds_processors/templates/queues/transfer_put_q.j2 deleted file mode 100644 index 9466715e..00000000 --- a/nlds_processors/templates/queues/transfer_put_q.j2 +++ /dev/null @@ -1,19 +0,0 @@ -{ - "rabbitMQ": { - "queues": [ - { - "name": "transfer_put_q", - "bindings": [ - { - "exchange": "{{ rabbit_exchange_name }}", - "routing_key": "*.transfer-put.start" - }, - { - "exchange": "{{ rabbit_exchange_name }}", - "routing_key": "*.transfer-put.init" - } - ] - } - ] - } -} \ No newline at end of file diff --git a/test_run/nlds-up b/nlds_utils/nlds-up similarity index 100% rename from test_run/nlds-up rename to nlds_utils/nlds-up diff --git a/nlds_processors/templates/queues/archive_get_q.j2 b/server_config/processors/archive_get_q.j2 similarity index 59% rename from nlds_processors/templates/queues/archive_get_q.j2 rename to server_config/processors/archive_get_q.j2 index 41d4e3ef..a66345df 100644 --- a/nlds_processors/templates/queues/archive_get_q.j2 +++ b/server_config/processors/archive_get_q.j2 @@ -1,4 +1,18 @@ -{ +{ + "archive_get_q": { + "tenancy": "{{ tenancy }}", + "tape_url": "{{ tape_url }}", + "tape_pool": {{ tape_pool|default(null) }}, + "chunk_size": {{ chunk_size|default(262144) }}, + "message_threshold": {{ filelist_threshold|default(10000)}}, + "filelist_max_length": {{ filelist_max_length|default(1000) }}, + "print_tracebacks_fl": {{ print_tracebacks|default(false) }}, + "require_secure_fl": {{ require_secure|default(false) }}, + "logging": { + "enable": true + "log_level" : "{{ log_level }}" + } + }, "rabbitMQ": { "queues": [ { diff --git a/server_config/processors/archive_put_q.j2 b/server_config/processors/archive_put_q.j2 new file mode 100644 index 00000000..ca5d85ab --- /dev/null +++ b/server_config/processors/archive_put_q.j2 @@ -0,0 +1,33 @@ +{ + "archive_put_q": { + "tenancy": "{{ tenancy }}", + "tape_url": "{{ tape_url }}", + "tape_pool": {{ tape_pool|default(null) }}, + "chunk_size": {{ chunk_size|default(262144) }}, + "message_threshold": {{ filelist_threshold|default(10000)}}, + "filelist_max_length": {{ filelist_max_length|default(1000) }}, + "print_tracebacks_fl": {{ print_tracebacks|default(false) }}, + "require_secure_fl": {{ require_secure|default(false) }}, + "logging": { + "enable": true + "log_level" : "{{ log_level }}" + } + }, + "rabbitMQ": { + "queues": [ + { + "name": "archive_put_q", + "bindings": [ + { + "exchange": "{{ rabbit_exchange_name }}", + "routing_key": "*.archive-put.start" + }, + { + "exchange": "{{ rabbit_exchange_name }}", + "routing_key": "*.archive-put.init" + } + ] + } + ] + } +} \ No newline at end of file diff --git a/nlds_processors/templates/queues/catalog_q.j2 b/server_config/processors/catalog_q.j2 similarity index 65% rename from nlds_processors/templates/queues/catalog_q.j2 rename to server_config/processors/catalog_q.j2 index 451e1952..5c4212e3 100644 --- a/nlds_processors/templates/queues/catalog_q.j2 +++ b/server_config/processors/catalog_q.j2 @@ -1,4 +1,19 @@ -{ +{ + "catalog_q":{ + "db_engine": "{{ db_engine }}", + "db_options": { + "db_name": "{{ db_name }}", + "db_user": "{{ db_user }}", + "db_passwd": "{{ db_passwd }}", + "echo": {{ db_echo|default(false) }} + }, + "logging":{ + "enable": true, + "log_level": "{{ log_level }}" + }, + "default_tenancy": "{{ tenancy }}", + "default_tape_url": "{{ default_tape_url }}" + }, "rabbitMQ": { "queues": [ { @@ -12,6 +27,10 @@ "exchange": "{{ rabbit_exchange_name }}", "routing_key": "*.catalog-get.list" }, + { + "exchange": "{{ rabbit_exchange_name }}", + "routing_key": "*.catalog-get.start" + }, { "exchange": "{{ rabbit_exchange_name }}", "routing_key": "*.catalog-del.start" @@ -30,7 +49,7 @@ }, { "exchange": "{{ rabbit_exchange_name }}", - "routing_key": "*.catalog-archive-remove.start" + "routing_key": "*.catalog-remove.start" } ] } diff --git a/nlds_processors/templates/queues/index_q.j2 b/server_config/processors/index_q.j2 similarity index 51% rename from nlds_processors/templates/queues/index_q.j2 rename to server_config/processors/index_q.j2 index 66dfb839..21c525b4 100644 --- a/nlds_processors/templates/queues/index_q.j2 +++ b/server_config/processors/index_q.j2 @@ -1,4 +1,15 @@ -{ +{ + "index_q":{ + "logging":{ + "enable": true, + "log_level" : "{{ log_level }}" + }, + "message_threshold": {{ filelist_threshold|default(10000)}}, + "filelist_max_length": {{ filelist_max_length|default(1000) }}, + "print_tracebacks_fl": {{ print_tracebacks|default(false) }}, + "check_filesize_fl": {{ check_filesize_fl|default(true)}}, + "max_filesize": {{ max_filesize|default(500000000)}} + }, "rabbitMQ": { "queues": [ { diff --git a/server_config/processors/logging_q.j2 b/server_config/processors/logging_q.j2 new file mode 100644 index 00000000..c6d19c87 --- /dev/null +++ b/server_config/processors/logging_q.j2 @@ -0,0 +1,25 @@ +{ + "logging": { + "enable": "{{ logging_enable|default(true) }}", + "log_level": "{{ logging_log_level }}", + "log_format": "{{ logging_log_format }}", + "add_stdout_fl": "{{ logging_add_stdout_fl }}", + "stdout_log_level": "{{ logging_stdout_log_level }}", + "log_files": "{{ logging_log_files }}", + "max_bytes": "{{ logging_max_bytes }}", + "backup_count": "{{ logging_backup_count }}" + }, + "rabbitMQ": { + "queues": [ + { + "name": "logging_q", + "bindings": [ + { + "exchange": "{{ rabbit_exchange_name }}", + "routing_key": "*.log.*" + } + ] + } + ] + } +} diff --git a/nlds_processors/templates/queues/monitor_q.j2 b/server_config/processors/monitor_q.j2 similarity index 57% rename from nlds_processors/templates/queues/monitor_q.j2 rename to server_config/processors/monitor_q.j2 index 99cd862d..3968e776 100644 --- a/nlds_processors/templates/queues/monitor_q.j2 +++ b/server_config/processors/monitor_q.j2 @@ -1,4 +1,17 @@ -{ +{ + "monitor_q":{ + "db_engine": "{{ db_engine }}", + "db_options": { + "db_name": "{{ db_name }}", + "db_user": "{{ db_user }}", + "db_passwd": "{{ db_passwd }}", + "echo": {{ db_echo|default(false) }} + }, + "logging":{ + "enable": true, + "log_level": "{{ log_level }}" + } + }, "rabbitMQ": { "queues": [ { diff --git a/nlds_processors/templates/queues/nlds_q.j2 b/server_config/processors/nlds_q.j2 similarity index 83% rename from nlds_processors/templates/queues/nlds_q.j2 rename to server_config/processors/nlds_q.j2 index 925c012a..0154c441 100644 --- a/nlds_processors/templates/queues/nlds_q.j2 +++ b/server_config/processors/nlds_q.j2 @@ -1,4 +1,10 @@ -{ +{ + "nlds_q":{ + "logging":{ + "enable": true, + "log_level": "{{ log_level }}" + } + }, "rabbitMQ": { "queues": [ { diff --git a/nlds_processors/templates/queues/transfer_del_q.j2 b/server_config/processors/transfer_del_q.j2 similarity index 59% rename from nlds_processors/templates/queues/transfer_del_q.j2 rename to server_config/processors/transfer_del_q.j2 index 7066a562..ffcd7385 100644 --- a/nlds_processors/templates/queues/transfer_del_q.j2 +++ b/server_config/processors/transfer_del_q.j2 @@ -1,4 +1,12 @@ -{ +{ + "transfer_del_q":{ + "tenancy": "{{ tenancy }}", + "require_secure_fl": {{ require_secure|default(false) }}, + "logging":{ + "enable": true, + "log_level" : "{{ log_level }}" + } + }, "rabbitMQ": { "queues": [ { diff --git a/server_config/processors/transfer_get_q.j2 b/server_config/processors/transfer_get_q.j2 new file mode 100644 index 00000000..db024a04 --- /dev/null +++ b/server_config/processors/transfer_get_q.j2 @@ -0,0 +1,32 @@ +{ + "transfer_get_q":{ + "tenancy": "{{ tenancy }}", + "filelist_max_length": {{ filelist_max_length|default(1000) }}, + "print_tracebacks_fl": {{ print_tracebacks|default(false) }}, + "check_filesize_fl": {{ check_filesize_fl|default(true) }}, + "require_secure_fl": {{ require_secure|default(false) }}, + "message_threshold": {{ filelist_threshold|default(10000)}}, + "filelist_max_length": {{ filelist_max_length|default(1000) }}, + "logging":{ + "enable": true, + "log_level" : "{{ log_level }}" + } + }, + "rabbitMQ": { + "queues": [ + { + "name": "transfer_get_q", + "bindings": [ + { + "exchange": "{{ rabbit_exchange_name }}", + "routing_key": "*.transfer-get.start" + }, + { + "exchange": "{{ rabbit_exchange_name }}", + "routing_key": "*.transfer-get.init" + } + ] + } + ] + } +} \ No newline at end of file diff --git a/server_config/processors/transfer_put_q.j2 b/server_config/processors/transfer_put_q.j2 new file mode 100644 index 00000000..30744c6e --- /dev/null +++ b/server_config/processors/transfer_put_q.j2 @@ -0,0 +1,32 @@ +{ + "transfer_put_q":{ + "tenancy": "{{ tenancy }}", + "filelist_max_length": {{ filelist_max_length|default(1000) }}, + "print_tracebacks_fl": {{ print_tracebacks|default(false) }}, + "check_filesize_fl": {{ check_filesize_fl|default(true) }}, + "require_secure_fl": {{ require_secure|default(false) }}, + "message_threshold": {{ filelist_threshold|default(10000)}}, + "filelist_max_length": {{ filelist_max_length|default(1000) }}, + "logging":{ + "enable": true, + "log_level" : "{{ log_level }}" + } + }, + "rabbitMQ": { + "queues": [ + { + "name": "transfer_put_q", + "bindings": [ + { + "exchange": "{{ rabbit_exchange_name }}", + "routing_key": "*.transfer-put.start" + }, + { + "exchange": "{{ rabbit_exchange_name }}", + "routing_key": "*.transfer-put.init" + } + ] + } + ] + } +} \ No newline at end of file diff --git a/server_config/server_config.j2 b/server_config/server_config.j2 new file mode 100644 index 00000000..269870df --- /dev/null +++ b/server_config/server_config.j2 @@ -0,0 +1,38 @@ +{ + "authentication" : { + "authenticator_backend" : "jasmin_authenticator", + "jasmin_authenticator" : { + "user_profile_url" : "{{ user_profile_url }}", + "user_services_url" : "{{ user_services_url }}", + "oauth_token_introspect_url" : "{{ token_introspect_url }}" + } + }, + "rabbitMQ": { + "user": "{{ rabbit_user }}", + "password": "{{ rabbit_password }}", + "heartbeat": "{{ rabbit_heartbeat }}", + "server": "{{ rabbit_server }}", + "admin_port": "{{ rabbit_port }}", + "vhost": "{{ rabbit_vhost }}", + "exchange": { + "name": "{{ rabbit_exchange_name }}", + "type": "{{ rabbit_exchange_type }}", + "delayed": "{{ rabbit_exchange_delayed }}" + }, + "queues": [ + { + {# queues are inserted here from the individual processors template files #} + } + ] + }, + "rpc_publisher": { + "time_limit": "{{ rpc_publisher_time_limit }}", + "queue_exclusivity_fl": "{{ rpc_publisher_queue_exclusivity_fl }}" + }, + "cronjob_publisher": { + "access_key": "{{ cronjob_publisher_access_key }}", + "secret_key": "{{ cronjob_publisher_secret_key }}", + "tenancy": "{{ cronjob_publisher_tenancy }}", + "tape_url": "{{ cronjob_publisher_tape_url }}" + } +} diff --git a/test_run/start_test_run.sh b/test_run/start_test_run.sh deleted file mode 100755 index 811fbb4b..00000000 --- a/test_run/start_test_run.sh +++ /dev/null @@ -1,9 +0,0 @@ -#! /usr/bin/env bash -if [[ ! -d ~/nlds_log ]] -then - mkdir ~/nlds_log -fi - -source $HOME/python-venvs/nlds-venv/bin/activate -# start a named screen session -screen -S nlds -c test_run/test_run.rc diff --git a/test_run/stop_test_run.sh b/test_run/stop_test_run.sh deleted file mode 100755 index 0c7f11c8..00000000 --- a/test_run/stop_test_run.sh +++ /dev/null @@ -1,3 +0,0 @@ -#! /usr/bin/env bash -screen -d nlds -screen -r nlds -X quit diff --git a/test_run/test_run.rc b/test_run/test_run.rc deleted file mode 100644 index 57e3ea1d..00000000 --- a/test_run/test_run.rc +++ /dev/null @@ -1,62 +0,0 @@ -# screen RC config file - -setenv PYTHON_DIR "$HOME/nlds-venv/bin" -setenv NLDS "$HOME/NLDS/nlds-server/nlds" -setenv NLDS_PROC "$NLDS/nlds_processors" - -# titles at the top -# left pos 1 -# create the indexer -screen -t "indexer" -exec "$PYTHON_DIR/python" "$NLDS_PROC/index.py" -split -v -focus - -# right pos 1 -# create the worker -screen -t "nlds_worker" -exec "$PYTHON_DIR/python" "$NLDS_PROC/nlds_worker.py" -split -focus - -# right pos 2 -# create the monitor -screen -t "monitor" -exec "$PYTHON_DIR/python" "$NLDS_PROC/monitor/monitor_worker.py" -split -focus - -# right pos 3 -# create the catalog -screen -t "catalog" -exec "$PYTHON_DIR/python" "$NLDS_PROC/catalog/catalog_worker.py" -split -focus - -# right pos 4 -# create the logger -# run the server via uvicorn -screen -t "server" -exec "$PYTHON_DIR/uvicorn" "nlds.main:nlds" "--reload" "--log-level=trace" "--port=8000" -focus left -split -focus - -# left pos 2 -# create the put_transfer -screen -t "put_transfer" -exec "$PYTHON_DIR/python" "$NLDS_PROC/transferers/put_transfer.py" -split -focus - -# left pos 3 -# create the get_transfer -screen -t "get_transfer" -exec "$PYTHON_DIR/python" "$NLDS_PROC/transferers/get_transfer.py" -split -focus - -# left pos 4 -screen -t "logger" -exec "$PYTHON_DIR/python" "$NLDS_PROC/logger.py" -