Add email notifications channel

[davelopez]

Closes #17882

This PR adds a new email channel to the notifications system.

Backend changes

• Includes 2 database migrations to add 2 new columns to the notification table.
  • dispatched column: will determine if this notification has been "dispatched" through the additional communication channels. Any existing notification will set this value to "True" so users won't get emails retroactively.
  • galaxy_url column: allows to build URLs from the notification channel plugins that don't have access to the web request context.
• Move the creation of notifications (database objects) to a celery task if enabled. This can offload resolving the recipient users when we include complex combinations of groups and/or roles as recipients. We can also control if we want to explicitly run this synchronously when we don't use groups or roles.
• Adds a new periodic celery task (every 10 minutes by default) that will process non dispatched notifications and send them via all available channels. In-app notifications don't depend on this task so they will pop up in Galaxy at their due time. Other external channels, like email, will probably carry some delay depending on the dispatch_notifications_interval configuration option.
• Adds personalizable email templates for the currently existing types of notifications.

Frontend changes

• The notification preferences will display a new Email channel if the server "supports" it. The channels supported by the server are passed to the client via a new header named supported-channels when requesting api/notifications/preferences.
• Navigating to /user/notifications?preferences=true will display the preferences panel open. This makes it easier to link to the preferences from the email templates.

Sample Emails

Shared Item

Direct Message

See it in action

How to test the changes?

• I've included appropriate automated tests.
• This is a refactoring of components with existing test coverage.
• Instructions for manual testing are as follows:
  • Make sure enable_celery_tasks is set to true in your Galaxy config.
  • Set up the smtp_server in your config. I used https://github.com/rnwood/smtp4dev/
  • Play around with the email templates from lib/galaxy/config/templates/mail/notifications/ by copying them and setting the templates_dir accordingly.
  • Play around disabling the email channel for certain notifications in your preferences.
  • Play around scheduling notifications as an admin.

License

• I agree to license these and all my past contributions to the core galaxy codebase under the MIT license.

[davelopez]

It seems the generated schema name when using generics changes between different versions of Python... https://github.com/galaxyproject/galaxy/actions/runs/8596319163/job/23553029236?pr=17914 🤔

diff --git a/client/src/api/schema/schema.ts b/client/src/api/schema/schema.ts
index 8786ebbb1e..b621015166 100644
--- a/client/src/api/schema/schema.ts
+++ b/client/src/api/schema/schema.ts
@@ -5394,8 +5394,8 @@ export interface components {
             /** Tags */
             tags?: string[] | null;
         };
-        /** GenericNotificationCreateRequest[Annotated[int, BeforeValidator, BeforeValidator, PlainSerializer, WithJsonSchema, WithJsonSchema]] */
-        GenericNotificationCreateRequest_Annotated_int__BeforeValidator__BeforeValidator__PlainSerializer__WithJsonSchema__WithJsonSchema__: {
+        /** GenericNotificationCreateRequest[Annotated[int, BeforeValidator(func=<function <lambda> at 0x7ae0525f30d0>), BeforeValidator(func=<function ensure_valid_id at 0x7ae0525e61f0>), PlainSerializer(func=<function <lambda> at 0x7ae0525f3040>, return_type=<class 'str'>, when_used='json'), WithJsonSchema(json_schema={'type': 'string', 'example': '0123456789ABCDEF', 'pattern': '[0-9a-fA-F]+', 'minLength': 16}, mode='serialization'), WithJsonSchema(json_schema={'type': 'string', 'example': '0123456789ABCDEF', 'pattern': '[0-9a-fA-F]+', 'minLength': 16}, mode='validation')]] */
+        "GenericNotificationCreateRequest_Annotated_int__BeforeValidator_func__function__lambda__at_0x7ae0525f30d0____BeforeValidator_func__function_ensure_valid_id_at_0x7ae0525e61f0____PlainSerializer_func__function__lambda__at_0x7ae0525f3040___return_type__class__str____when_used__json____WithJsonSchema_json_schema___type____example____pattern__0-9a-fA-F______minLength___mode__serialization____WithJsonSchema_json_schema___type____example____pattern__0-9a-fA-F______minLength___mode__validation____": {
             /**
              * Notification
              * @description The notification to create. The structure depends on the category.
@@ -5405,10 +5405,10 @@ export interface components {
              * Recipients
              * @description The recipients of the notification. Can be a combination of users, groups and roles.
              */
-            recipients: components["schemas"]["GenericNotificationRecipients_Annotated_int__BeforeValidator__BeforeValidator__PlainSerializer__WithJsonSchema__WithJsonSchema__"];
+            recipients: components["schemas"]["GenericNotificationRecipients_Annotated_int__BeforeValidator_func__function__lambda__at_0x7ae0525f30d0____BeforeValidator_func__function_ensure_valid_id_at_0x7ae0525e61f0____PlainSerializer_func__function__lambda__at_0x7ae0525f3040___return_type__class__str____when_used__json____WithJsonSchema_json_schema___type____example____pattern__0-9a-fA-F______minLength___mode__serialization____WithJsonSchema_json_schema___type____example____pattern__0-9a-fA-F______minLength___mode__validation____"];
         };
-        /** GenericNotificationRecipients[Annotated[int, BeforeValidator, BeforeValidator, PlainSerializer, WithJsonSchema, WithJsonSchema]] */
-        GenericNotificationRecipients_Annotated_int__BeforeValidator__BeforeValidator__PlainSerializer__WithJsonSchema__WithJsonSchema__: {
+        /** GenericNotificationRecipients[Annotated[int, BeforeValidator(func=<function <lambda> at 0x7ae0525f30d0>), BeforeValidator(func=<function ensure_valid_id at 0x7ae0525e61f0>), PlainSerializer(func=<function <lambda> at 0x7ae0525f3040>, return_type=<class 'str'>, when_used='json'), WithJsonSchema(json_schema={'type': 'string', 'example': '0123456789ABCDEF', 'pattern': '[0-9a-fA-F]+', 'minLength': 16}, mode='serialization'), WithJsonSchema(json_schema={'type': 'string', 'example': '0123456789ABCDEF', 'pattern': '[0-9a-fA-F]+', 'minLength': 16}, mode='validation')]] */
+        "GenericNotificationRecipients_Annotated_int__BeforeValidator_func__function__lambda__at_0x7ae0525f30d0____BeforeValidator_func__function_ensure_valid_id_at_0x7ae0525e61f0____PlainSerializer_func__function__lambda__at_0x7ae0525f3040___return_type__class__str____when_used__json____WithJsonSchema_json_schema___type____example____pattern__0-9a-fA-F______minLength___mode__serialization____WithJsonSchema_json_schema___type____example____pattern__0-9a-fA-F______minLength___mode__validation____": {
             /**
              * Group IDs
              * @description The list of encoded group IDs of the groups that should receive the notification.
@@ -9673,7 +9673,7 @@ export interface components {
             variant: components["schemas"]["NotificationVariant"];
         };
         /** NotificationCreateRequestBody */
-        NotificationCreateRequestBody: components["schemas"]["GenericNotificationCreateRequest_Annotated_int__BeforeValidator__BeforeValidator__PlainSerializer__WithJsonSchema__WithJsonSchema__"];
+        NotificationCreateRequestBody: components["schemas"]["GenericNotificationCreateRequest_Annotated_int__BeforeValidator_func__function__lambda__at_0x7ae0525
f30d0____BeforeValidator_func__function_ensure_valid_id_at_0x7ae0525e61f0____PlainSerializer_func__function__lambda__at_0x7ae0525f3040___return_type__class__str___
_when_used__json____WithJsonSchema_json_schema___type____example____pattern__0-9a-fA-F______minLength___mode__serialization____WithJsonSchema_json_schema___type___
_example____pattern__0-9a-fA-F______minLength___mode__validation____"];
         /** NotificationCreatedResponse */
         NotificationCreatedResponse: {
             /**

Update

I fixed it with a workaround to not use Generics directly in the exposed API models because they will get this extremely long auto-generated name that even changes between different versions of Python... it is ugly. Still, the other result was uglier I think 😅

[davelopez]

I'm completely puzzled by this error in unit and package tests...

=========================== short test summary info ============================
ERROR lib/galaxy/schema/notifications.py - re.error: unbalanced parenthesis at position 434
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
=========== 25 deselected, 77 warnings, 1 error in 104.37s (0:01:44) ===========

I cannot see any unbalanced parenthesis in the whole file, let alone at position 434. In addition, this only fails consistently in Python 3.8 but not in Python 3.12 😵

am I missing something obvious?

[mvdbeek]

Arg, that happened to me to at one point and was a big mystery. Unfortunately I didn't remember what the problem was :(

[mvdbeek]

It might be an issue with how the generics are compiled ...

[mvdbeek]

This is what I see locally:

^\s*class\s*NotificationVariant\b
^\s*class\s*MandatoryNotificationCategory\b
^\s*class\s*PersonalNotificationCategory\b
^\s*class\s*MessageNotificationContentBase\b
^\s*class\s*ActionLink\b
^\s*class\s*BroadcastNotificationContent\b
^\s*class\s*MessageNotificationContent\b
^\s*class\s*NewSharedItemNotificationContent\b
^\s*class\s*NotificationResponse\b
^\s*class\s*UserNotificationResponse\b
^\s*class\s*BroadcastNotificationResponse\b
^\s*class\s*UserNotificationListResponse\b
^\s*class\s*BroadcastNotificationListResponse\b
^\s*class\s*NotificationStatusSummary\b
^\s*class\s*NotificationCreateData\b
^\s*class\s*GenericNotificationRecipients\b
^\s*class\s*GenericNotificationCreateRequest\b
^\s*class\s*GenericNotificationCreateRequest[int]\b
^\s*class\s*GenericNotificationRecipients[int]\b
^\s*class\s*GenericNotificationRecipients[Annotated[int, BeforeValidator(func=<function <lambda> at 0x1056794c0>), BeforeValidator(func=<function ensure_valid_id at 0x105679d30>), PlainSerializer(func=<function <lambda> at 0x105679280>, return_type=<class 'str'>, when_used='json'), WithJsonSchema(json_schema={'type': 'string', 'example': '0123456789ABCDEF', 'pattern': '[0-9a-fA-F]+', 'minLength': 16}, mode='serialization'), WithJsonSchema(json_schema={'type': 'string', 'example': '0123456789ABCDEF', 'pattern': '[0-9a-fA-F]+', 'minLength': 16}, mode='validation')]]\b

[davelopez]

It might be an issue with how the generics are compiled ...

Makes sense... It seems there are differences between Python versions around them as per the first issue I encountered with the client schema generation... 😞 I'll try to dig a bit more into it.

[mvdbeek]

I think https://github.com/davelopez/galaxy/blob/4575d1c47845d5b0a23e664b8d2390794f4dda0b/lib/galaxy/schema/invocation.py#L98-L102 was the solution when I ran into it.

[davelopez]

Thank you so much @mvdbeek! That was it!

I moved the logic to a GenericModel under a new generics module so it can be reused. I hope that is fine.

[davelopez]

I think this is finally ready for review so I updated the PR description.

I will follow up with some additional changes that don't quite fit here.

[jdavcs]

Looks great! Thank you!