Upgrade dependencies & bump IFRS (#55)

* Upgrade dependencies * Bump IFRS version * Update IFRS readme * Update config docs IFRS * Update README ifrs * Update config files for all service --------- Co-authored-by: TheByronHimes <[email protected]>
ghga-de · Aug 12, 2024 · cffd7e9 · cffd7e9
1 parent 92e9ab3
commit cffd7e9
Show file tree

Hide file tree

Showing 27 changed files with 1,102 additions and 833 deletions.
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -49,13 +49,13 @@ repos:
       - id: no-commit-to-branch
         args: [--branch, dev, --branch, int, --branch, main]
   - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: v0.5.5
+    rev: v0.5.7
     hooks:
       - id: ruff
         args: [--fix, --exit-non-zero-on-fix]
       - id: ruff-format
   - repo: https://github.com/pre-commit/mirrors-mypy
-    rev: v1.11.0
+    rev: v1.11.1
     hooks:
       - id: mypy
         args: [--no-warn-unused-ignores]
diff --git a/lock/requirements-dev.txt b/lock/requirements-dev.txt
diff --git a/lock/requirements.txt b/lock/requirements.txt
diff --git a/services/dcs/README.md b/services/dcs/README.md
@@ -74,7 +74,7 @@ dcs --help
 ### Parameters
 
 The service requires the following configuration parameters:
-- **`unstaged_download_event_topic`** *(string)*: Name of the topic used for events indicating that a download was requested for a file that is not yet available in the outbox.
+- **`unstaged_download_event_topic`** *(string, required)*: Name of the topic used for events indicating that a download was requested for a file that is not yet available in the outbox.
 
 
   Examples:
@@ -84,7 +84,7 @@ The service requires the following configuration parameters:
   ```
 
 
-- **`unstaged_download_collection`** *(string)*: The type used for event indicating that a download was requested for a file that is not yet available in the outbox. The value should use hyphens in place of underscores if needed.
+- **`unstaged_download_collection`** *(string, required)*: The type used for event indicating that a download was requested for a file that is not yet available in the outbox. The value should use hyphens in place of underscores if needed.
 
 
   Examples:
@@ -94,7 +94,7 @@ The service requires the following configuration parameters:
   ```
 
 
-- **`files_to_delete_topic`** *(string)*: The name of the topic to receive events informing about files to delete.
+- **`files_to_delete_topic`** *(string, required)*: The name of the topic to receive events informing about files to delete.
 
 
   Examples:
@@ -108,7 +108,7 @@ The service requires the following configuration parameters:
 
 - **`service_name`** *(string)*: Default: `"dcs"`.
 
-- **`service_instance_id`** *(string)*: A string that uniquely identifies this instance across all instances of this service. A globally unique Kafka client ID will be created by concatenating the service_name and the service_instance_id.
+- **`service_instance_id`** *(string, required)*: A string that uniquely identifies this instance across all instances of this service. A globally unique Kafka client ID will be created by concatenating the service_name and the service_instance_id.
 
 
   Examples:
@@ -141,11 +141,11 @@ The service requires the following configuration parameters:
 
 - **`log_traceback`** *(boolean)*: Whether to include exception tracebacks in log messages. Default: `true`.
 
-- **`object_storages`** *(object)*: Can contain additional properties.
+- **`object_storages`** *(object, required)*: Can contain additional properties.
 
   - **Additional properties**: Refer to *[#/$defs/S3ObjectStorageNodeConfig](#%24defs/S3ObjectStorageNodeConfig)*.
 
-- **`files_to_register_topic`** *(string)*: The name of the topic to receive events informing about new files that shall be made available for download.
+- **`files_to_register_topic`** *(string, required)*: The name of the topic to receive events informing about new files that shall be made available for download.
 
 
   Examples:
@@ -155,7 +155,7 @@ The service requires the following configuration parameters:
   ```
 
 
-- **`files_to_register_type`** *(string)*: The type used for events informing about new files that shall be made available for download.
+- **`files_to_register_type`** *(string, required)*: The type used for events informing about new files that shall be made available for download.
 
 
   Examples:
@@ -165,7 +165,7 @@ The service requires the following configuration parameters:
   ```
 
 
-- **`download_served_event_topic`** *(string)*: Name of the topic used for events indicating that a download of a specified file happened.
+- **`download_served_event_topic`** *(string, required)*: Name of the topic used for events indicating that a download of a specified file happened.
 
 
   Examples:
@@ -175,7 +175,7 @@ The service requires the following configuration parameters:
   ```
 
 
-- **`download_served_event_type`** *(string)*: The type used for event indicating that a download of a specified file happened.
+- **`download_served_event_type`** *(string, required)*: The type used for event indicating that a download of a specified file happened.
 
 
   Examples:
@@ -185,7 +185,7 @@ The service requires the following configuration parameters:
   ```
 
 
-- **`file_registered_event_topic`** *(string)*: Name of the topic used for events indicating that a file has been registered for download.
+- **`file_registered_event_topic`** *(string, required)*: Name of the topic used for events indicating that a file has been registered for download.
 
 
   Examples:
@@ -195,7 +195,7 @@ The service requires the following configuration parameters:
   ```
 
 
-- **`file_registered_event_type`** *(string)*: The type used for event indicating that that a file has been registered for download.
+- **`file_registered_event_type`** *(string, required)*: The type used for event indicating that that a file has been registered for download.
 
 
   Examples:
@@ -205,7 +205,7 @@ The service requires the following configuration parameters:
   ```
 
 
-- **`file_deleted_event_topic`** *(string)*: Name of the topic used for events indicating that a file has been deleted.
+- **`file_deleted_event_topic`** *(string, required)*: Name of the topic used for events indicating that a file has been deleted.
 
 
   Examples:
@@ -215,7 +215,7 @@ The service requires the following configuration parameters:
   ```
 
 
-- **`file_deleted_event_type`** *(string)*: The type used for events indicating that a file has been deleted.
+- **`file_deleted_event_type`** *(string, required)*: The type used for events indicating that a file has been deleted.
 
 
   Examples:
@@ -225,7 +225,7 @@ The service requires the following configuration parameters:
   ```
 
 
-- **`kafka_servers`** *(array)*: A list of connection strings to connect to Kafka bootstrap servers.
+- **`kafka_servers`** *(array, required)*: A list of connection strings to connect to Kafka bootstrap servers.
 
   - **Items** *(string)*
 
@@ -264,7 +264,22 @@ The service requires the following configuration parameters:
   ```
 
 
-- **`db_connection_str`** *(string, format: password)*: MongoDB connection string. Might include credentials. For more information see: https://naiveskill.com/mongodb-connection-string/.
+- **`kafka_max_message_size`** *(integer)*: The largest message size that can be transmitted, in bytes. Only services that have a need to send/receive larger messages should set this. Exclusive minimum: `0`. Default: `1048576`.
+
+
+  Examples:
+
+  ```json
+  1048576
+  ```
+
+
+  ```json
+  16777216
+  ```
+
+
+- **`db_connection_str`** *(string, format: password, required)*: MongoDB connection string. Might include credentials. For more information see: https://naiveskill.com/mongodb-connection-string/.
 
 
   Examples:
@@ -274,7 +289,7 @@ The service requires the following configuration parameters:
   ```
 
 
-- **`db_name`** *(string)*: Name of the database located on the MongoDB server.
+- **`db_name`** *(string, required)*: Name of the database located on the MongoDB server.
 
 
   Examples:
@@ -284,7 +299,7 @@ The service requires the following configuration parameters:
   ```
 
 
-- **`drs_server_uri`** *(string)*: The base of the DRS URI to access DRS objects. Has to start with 'drs://' and end with '/'.
+- **`drs_server_uri`** *(string, required)*: The base of the DRS URI to access DRS objects. Has to start with 'drs://' and end with '/'.
 
 
   Examples:
@@ -339,7 +354,7 @@ The service requires the following configuration parameters:
   ```
 
 
-- **`ekss_base_url`** *(string)*: URL containing host and port of the EKSS endpoint to retrieve personalized envelope from.
+- **`ekss_base_url`** *(string, required)*: URL containing host and port of the EKSS endpoint to retrieve personalized envelope from.
 
 
   Examples:
@@ -349,7 +364,7 @@ The service requires the following configuration parameters:
   ```
 
 
-- **`presigned_url_expires_after`** *(integer)*: Expiration time in seconds for presigned URLS. Positive integer required. Exclusive minimum: `0`.
+- **`presigned_url_expires_after`** *(integer, required)*: Expiration time in seconds for presigned URLS. Positive integer required. Exclusive minimum: `0`.
 
 
   Examples:
@@ -379,7 +394,7 @@ The service requires the following configuration parameters:
   ```
 
 
-- **`auth_key`** *(string)*: The GHGA internal public key for validating the token signature.
+- **`auth_key`** *(string, required)*: The GHGA internal public key for validating the token signature.
 
 
   Examples:

diff --git a/services/dcs/config_schema.json b/services/dcs/config_schema.json
@@ -95,7 +95,7 @@
     }
   },
   "additionalProperties": false,
-  "description": "Modifies the orginal Settings class provided by the user",
+  "description": "Modifies the original Settings class provided by the user",
   "properties": {
     "unstaged_download_event_topic": {
       "description": "Name of the topic used for events indicating that a download was requested for a file that is not yet available in the outbox.",
@@ -301,6 +301,17 @@
       "title": "Generate Correlation Id",
       "type": "boolean"
     },
+    "kafka_max_message_size": {
+      "default": 1048576,
+      "description": "The largest message size that can be transmitted, in bytes. Only services that have a need to send/receive larger messages should set this.",
+      "examples": [
+        1048576,
+        16777216
+      ],
+      "exclusiveMinimum": 0,
+      "title": "Kafka Max Message Size",
+      "type": "integer"
+    },
     "db_connection_str": {
       "description": "MongoDB connection string. Might include credentials. For more information see: https://naiveskill.com/mongodb-connection-string/",
       "examples": [

diff --git a/services/dcs/example_config.yaml b/services/dcs/example_config.yaml
@@ -35,6 +35,7 @@ files_to_register_topic: internal-file-registry
 files_to_register_type: file_registered
 generate_correlation_id: true
 host: 127.0.0.1
+kafka_max_message_size: 1048576
 kafka_security_protocol: PLAINTEXT
 kafka_servers:
 - kafka:9092

diff --git a/services/ekss/README.md b/services/ekss/README.md
@@ -100,7 +100,7 @@ The service requires the following configuration parameters:
 
 - **`service_name`** *(string)*: Default: `"encryption_key_store"`.
 
-- **`service_instance_id`** *(string)*: A string that uniquely identifies this instance across all instances of this service. This is included in log messages.
+- **`service_instance_id`** *(string, required)*: A string that uniquely identifies this instance across all instances of this service. This is included in log messages.
 
 
   Examples:
@@ -133,7 +133,7 @@ The service requires the following configuration parameters:
 
 - **`log_traceback`** *(boolean)*: Whether to include exception tracebacks in log messages. Default: `true`.
 
-- **`vault_url`** *(string)*: URL of the vault instance to connect to.
+- **`vault_url`** *(string, required)*: URL of the vault instance to connect to.
 
 
   Examples:
@@ -191,7 +191,7 @@ The service requires the following configuration parameters:
   ```
 
 
-- **`vault_path`** *(string)*: Path without leading or trailing slashes where secrets should be stored in the vault.
+- **`vault_path`** *(string, required)*: Path without leading or trailing slashes where secrets should be stored in the vault.
 
 - **`vault_secrets_mount_point`** *(string)*: Name used to address the secret engine under a custom mount path. Default: `"secret"`.
 
@@ -354,7 +354,7 @@ The service requires the following configuration parameters:
   ```
 
 
-- **`server_private_key`** *(string, format: password)*: Base64 encoded server Crypt4GH private key.
+- **`server_private_key`** *(string, format: password, required)*: Base64 encoded server Crypt4GH private key.
 
 
   Examples:
@@ -364,7 +364,7 @@ The service requires the following configuration parameters:
   ```
 
 
-- **`server_public_key`** *(string)*: Base64 encoded server Crypt4GH public key.
+- **`server_public_key`** *(string, required)*: Base64 encoded server Crypt4GH public key.
 
 
   Examples:

diff --git a/services/ekss/config_schema.json b/services/ekss/config_schema.json
@@ -1,6 +1,6 @@
 {
   "additionalProperties": false,
-  "description": "Modifies the orginal Settings class provided by the user",
+  "description": "Modifies the original Settings class provided by the user",
   "properties": {
     "log_level": {
       "default": "INFO",

diff --git a/services/fins/README.md b/services/fins/README.md
@@ -45,7 +45,7 @@ fins --help
 ### Parameters
 
 The service requires the following configuration parameters:
-- **`files_to_delete_topic`** *(string)*: The name of the topic for events informing about files to be deleted.
+- **`files_to_delete_topic`** *(string, required)*: The name of the topic for events informing about files to be deleted.
 
 
   Examples:
@@ -59,7 +59,7 @@ The service requires the following configuration parameters:
 
 - **`service_name`** *(string)*: Default: `"fins"`.
 
-- **`service_instance_id`** *(string)*: A string that uniquely identifies this instance across all instances of this service. A globally unique Kafka client ID will be created by concatenating the service_name and the service_instance_id.
+- **`service_instance_id`** *(string, required)*: A string that uniquely identifies this instance across all instances of this service. A globally unique Kafka client ID will be created by concatenating the service_name and the service_instance_id.
 
 
   Examples:
@@ -92,7 +92,7 @@ The service requires the following configuration parameters:
 
 - **`log_traceback`** *(boolean)*: Whether to include exception tracebacks in log messages. Default: `true`.
 
-- **`kafka_servers`** *(array)*: A list of connection strings to connect to Kafka bootstrap servers.
+- **`kafka_servers`** *(array, required)*: A list of connection strings to connect to Kafka bootstrap servers.
 
   - **Items** *(string)*
 
@@ -131,7 +131,22 @@ The service requires the following configuration parameters:
   ```
 
 
-- **`db_connection_str`** *(string, format: password)*: MongoDB connection string. Might include credentials. For more information see: https://naiveskill.com/mongodb-connection-string/.
+- **`kafka_max_message_size`** *(integer)*: The largest message size that can be transmitted, in bytes. Only services that have a need to send/receive larger messages should set this. Exclusive minimum: `0`. Default: `1048576`.
+
+
+  Examples:
+
+  ```json
+  1048576
+  ```
+
+
+  ```json
+  16777216
+  ```
+
+
+- **`db_connection_str`** *(string, format: password, required)*: MongoDB connection string. Might include credentials. For more information see: https://naiveskill.com/mongodb-connection-string/.
 
 
   Examples:
@@ -141,7 +156,7 @@ The service requires the following configuration parameters:
   ```
 
 
-- **`db_name`** *(string)*: Name of the database located on the MongoDB server.
+- **`db_name`** *(string, required)*: Name of the database located on the MongoDB server.
 
 
   Examples:
@@ -151,7 +166,7 @@ The service requires the following configuration parameters:
   ```
 
 
-- **`file_registered_event_topic`** *(string)*: The name of the topic for events informing about new registered files for which the metadata should be made available.
+- **`file_registered_event_topic`** *(string, required)*: The name of the topic for events informing about new registered files for which the metadata should be made available.
 
 
   Examples:
@@ -161,7 +176,7 @@ The service requires the following configuration parameters:
   ```
 
 
-- **`file_registered_event_type`** *(string)*: The name of the type used for events informing about new registered files for which the metadata should be made available.
+- **`file_registered_event_type`** *(string, required)*: The name of the type used for events informing about new registered files for which the metadata should be made available.
 
 
   Examples:

diff --git a/services/fins/config_schema.json b/services/fins/config_schema.json
@@ -1,6 +1,6 @@
 {
   "additionalProperties": false,
-  "description": "Modifies the orginal Settings class provided by the user",
+  "description": "Modifies the original Settings class provided by the user",
   "properties": {
     "files_to_delete_topic": {
       "description": "The name of the topic for events informing about files to be deleted.",
@@ -119,6 +119,17 @@
       "title": "Generate Correlation Id",
       "type": "boolean"
     },
+    "kafka_max_message_size": {
+      "default": 1048576,
+      "description": "The largest message size that can be transmitted, in bytes. Only services that have a need to send/receive larger messages should set this.",
+      "examples": [
+        1048576,
+        16777216
+      ],
+      "exclusiveMinimum": 0,
+      "title": "Kafka Max Message Size",
+      "type": "integer"
+    },
     "db_connection_str": {
       "description": "MongoDB connection string. Might include credentials. For more information see: https://naiveskill.com/mongodb-connection-string/",
       "examples": [