Notification System for Dalgo

[fatchat]

Description

This project will build a notification system for Dalgo. Although this ticket is part of the DDP_backend repo, the selected contributor will work on the frontend webapp repo as well, and possibly on the prefect-proxy repo if required

Goals

• An interface for a Dalgo admin to post a notification to a user or to a set of users
• Support for multiple notification channels
• Email should be one channel
• Discord should be one channel
• Users should be able to choose their preferred channel
• Users should be able to turn off notifications to all channels
• Notifications should appear in the React frontend
• Indicator for unread notifications for a user in the frontend
• "Urgent" notifications should appear as a horizontal bar in the frontend e.g. for "System is going down in 10 minutes!" type of messages

Expected Outcome

In the backend:

• a Python interface to create new notifications, fetch unread notifications for a user, fetch the notification history and more
• an API to create new notifications (this API will be invoked by the dalgo-admin tool)
• a pub-sub model in which handlers for the various notification channels can subscribe and receive new notifications to be sent out
• handlers for Email and Discord to start off with

In the frontend

• Notifications in the toolbar (e.g. a "bell icon") showing number of unread
• User should be able to view their unread notifications by clicking on the icon
• A notification bar for "urgent" notifications, possibly with a red background or something similarly eye-catching
• A UX where a user can set and manage their notification preferences

Acceptance Criteria

• Text-only notifications are acceptable, but extra points for rich text (html, markdown)
• Admin should be able to send notifications to individual users
• Admin should be able to send notifications to a set of users (including all users)
• Admin should be able to send "urgent" notifications
• Users should receive or not receive channel notifications based on their preferences
• Users should see notifications in the frontend as described above
• Notification history should be stored in the database

Implementation Details

Libraries may be used but must be open-source
Our backend is Django and our frontend is NextJS

Mockups / Wireframes

None but we will decide on designs before the frontend work begins

Product Name

Dalgo

Project Name

Notification System for Dalgo

Organization Name:

Project Tech4Dev

Domain

Other

Tech Skills Needed:

Python, Django, JavaScript, NextJS

Mentor(s)

@fatchat

Complexity

High

Category

Integrations, Feature

Sub Category

API, Frontend, Backend

[Shruti3004]

Linking Installation guide here: https://github.com/DalgoT4D/DDP_backend?tab=readme-ov-file#setup-instructions

[venkatramanm]

@SaumyaSamagra , The requirements seem quite clear and it seems to be doable in the timeframe of the internship.
What is the pub-sub mechanism to be used, is not mentioned. Some clarity may be added or may need to be figured out earlier on on the project. That may be added to the skill set requirement as a preference.

[Snehil-Shah]

@fatchat is the plan to use something like RabbitMQ/Kafka for the pub-sub model?

[fatchat]

@Snehil-Shah I'm looking for a solution which uses an existing library. Having said that, I'd want to avoid a library which uses Kafka since that would be overkill. Rabbit/Zero/something small

The contributor would need to know how the notifications library is using its messaging components, but only for testing and debugging and not for extending them

[amansgith]

Hii @fatchat can "FastAPI Websocket Pub/Sub" package be used for it.. as its lightweight and maybe still effective for the pub sub model
Link

[ricky08sirus]

I want to contribute for this project as a part of DSP 2024.
please guide me with the procedure @fatchat .

[srihari77]

I want to contribute for this project as a part of DSP 2024.
please guide me with the procedure @fatchat .

[AkanshuAich]

Hii @fatchat,

I am writing to express my interest in contributing to this project as a part of DSP 2024. Having thoroughly reviewed the project, I am impressed by its objectives and it seeks the potential for great impact in industries.

With my background in Backend using Django , MERN with practicing hands on Machine learning, I believe I can make valuable contributions to both backend and frontend part. My experience includes several projects like Society-Expenditure Manager using Django and Real Estate using MERN, which I believe align well with the goals of your project.

I am particularly interested in fulfilling the requirements of the project and have some ideas on how to approach it effectively. I am committed to adhering to best practices, contributing high-quality code, and actively collaborating with the project maintainers and community.

I am excited about the opportunity to contribute to Notification System for Dalgo and help further its mission. I look forward to discussing potential contributions and how I can best support the project.

Please guide me with procedure and with all your knowledge and experience.

[fatchat]

hello all, thank you for your interest

please refer to the c4gt website / discord community on how to apply

and as much as i would love to, i won't be mentoring anyone during the application process... there are just too many people. whoever is selected will get two hours a week from me

thank you again

[SaumyaSamagra]

Hi Snehil, Please create a duplicate file and upload your proposal document <https://docs.google.com/document/d/1D2pLc_eP-8-waBnRuj57Tc_jQT7_-ZWRQCDjv6qtPTY/edit?userstoinvite=mohitdto0311%40gmail.com&sharingaction=manageaccess&role=writer> with the specified nomenclature as specified on Unstop <https://unstop.com/competitions/dedicated-mentoring-program-dmp-2024-code-for-govtech-932803?lb=7gWXVfKv> . For any questions please feel free to direct your queries at ***@***.*** and join the Discord server <https://bit.ly/C4GTJoinDiscordServer> Thank you Best, Saumya

…

On Sat, Mar 9, 2024 at 10:54 PM Snehil Shah ***@***.***> wrote: @fatchat <https://github.com/fatchat> is the plan to use something like RabbitMQ/Kafka for the pub-sub model? — Reply to this email directly, view it on GitHub <#442 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A7TNBGFBMMONBHM2M6RFR3LYXNATPAVCNFSM6AAAAABCTLBAG6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBWHEZDGMJWGM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Saumya Gupta Consultant | +91 9629770622 Samagra | Transforming Governance Address: 9/5 Sarvapriya Vihar, New Delhi-16 ***@***.***,77.1994501,17z/data=!3m1!4b1!4m5!3m4!1s0x390ce273981e5dfb:0x7bfceb198af17562!8m2!3d28.5566128!4d77.2016388?shorturl=1> Website: www.samagragovernance.in Follow Us: Linkedin <https://www.linkedin.com/company/samagra-transforming-governance/> | Youtube <https://www.youtube.com/channel/UCfkXErS-f87xUQkmSKSC8bg> | Instagram <https://www.instagram.com/samagragovernance/?hl=en>

[SaumyaSamagra]

Hi Aman, Please create a duplicate file and upload your proposal document <https://docs.google.com/document/d/1D2pLc_eP-8-waBnRuj57Tc_jQT7_-ZWRQCDjv6qtPTY/edit?userstoinvite=mohitdto0311%40gmail.com&sharingaction=manageaccess&role=writer> with the specified nomenclature as specified on Unstop <https://unstop.com/competitions/dedicated-mentoring-program-dmp-2024-code-for-govtech-932803?lb=7gWXVfKv> . For any questions please feel free to direct your queries at ***@***.*** and join the Discord server <https://bit.ly/C4GTJoinDiscordServer> Thank you Best, Saumya

…

On Sun, Mar 31, 2024 at 1:34 PM Aman Khan ***@***.***> wrote: Hii @fatchat <https://github.com/fatchat> can "FastAPI Websocket Pub/Sub" package be used for it.. as its lightweight and maybe still effective for the pub sub model Link <https://pypi.org/project/fastapi-websocket-pubsub/> — Reply to this email directly, view it on GitHub <#442 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A7TNBGHH3FZF3X42DW5Y2VLY267RJAVCNFSM6AAAAABCTLBAG6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMRYGU4TGOJRGU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Saumya Gupta Consultant | +91 9629770622 Samagra | Transforming Governance Address: 9/5 Sarvapriya Vihar, New Delhi-16 ***@***.***,77.1994501,17z/data=!3m1!4b1!4m5!3m4!1s0x390ce273981e5dfb:0x7bfceb198af17562!8m2!3d28.5566128!4d77.2016388?shorturl=1> Website: www.samagragovernance.in Follow Us: Linkedin <https://www.linkedin.com/company/samagra-transforming-governance/> | Youtube <https://www.youtube.com/channel/UCfkXErS-f87xUQkmSKSC8bg> | Instagram <https://www.instagram.com/samagragovernance/?hl=en>

[AbhimanyuSamagra]

Do not ask process related questions about how to apply and who to contact in the above ticket. The only questions allowed are about technical aspects of the project itself. If you want help with the process, you can refer instructions listed on Unstop and any further queries can be taken up on our Discord channel titled DMP queries.

[AkanshuAich]

@fatchat for the notification system using Django framework it will be more efficient to go with the third-party applications like Pusher, Firebase Cloud Messaging (FCM), Amazon Simple Notification Service (SNS), and Twilio. Celery with redis/RabbitMQ is also a good option as the background function and the notification system both can work simultaneously.

[RSN601KRI]

Hello, @fatchat , I would like to contribute to this project. I have the required skillset for it. Kindly guide me so that I can contribute to this project.

[amansgith]

Hi @fatchat can you help me with these things regarding the project of C4GT:

1. What problem exists in Dalgo or what will be enhanced by this system for which the notification system is to be developed?
2. Is any previous work related to the notification system already started or done?
3. As different channels like Email or discord have different methods for API usage and invocation method.. for handling these do we need to build a common platform so that new handles can be added in future?

Thanks

[fatchat]

@amansgith

1. We currently have notifications only for failures of flow-runs, and these only go to our Discord #ops channel. We would like to be able to send notifications to a user's email address, and let them configure this. More generally we would like to
   a. notify users when their connection schemas change
   b. notify users when the platform is going down for upgrades
   c. any other messages we would like to send our users

2. No

3. Yes, with limits. Right now we only want to be able to send text notifications, not html or rich-text or whatever else

[AbhimanyuSamagra]

Do not ask process related questions about how to apply and who to contact in the above ticket. The only questions allowed are about technical aspects of the project itself. If you want help with the process, you can refer instructions listed on Unstop and any further queries can be taken up on our Discord channel titled DMP queries. Here's a Video Tutorial on how to submit a proposal for a project.

[SachinSangaal]

Hello @fatchat I want to contribute for this project of C4GT 2024.
please guide me with the procedure.

[techiAlok]

Hello @fatchat i too want to contribute to this project of C4GT 2024.
please guide me to move further in the process.

[Rishikesh63]

@fatchat is DMP selection of DalgoT4D announced?

[fatchat]

Yes, announcements were made on Tuesday I believe

…

On Thu, May 30, 2024 at 5:10 PM Rishikesh Maddhesiya < ***@***.***> wrote: @fatchat <https://github.com/fatchat> is DMP selection of DalgoT4D announced? — Reply to this email directly, view it on GitHub <#442 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAQPOIAJDSNTL76723GNAELZE4F3NAVCNFSM6AAAAABCTLBAG6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMZZGM3DSMBQGA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[huzaifmalik786]

Weekly Learnings & Updates

Week 1

• Set up the DDP_backend project locally along with prefect-proxy and airbyte server.

Week 2

• Make a documentation file. [442]-Notification System for Dalgo Documentation #718
• Learn how to make migrations and revert them.
  python manage.py makemigrations ddpui
python manage.py migrate
python manage.py migrate ddpui 0077

Week 3

• Create notification, recipients, and user preference table models and make migrations.
• Create functions for managing user preferences.
• Create APIs for calling functions for managing user preferences

Issues:

• Create Model For Notification Table #674
• Create Model for User Notification Preferences Table #675
• Perform Migrations for Notification and Preferences Models #676
• Create a function for updating the User's Notification Preferences. #684
• Create a function for fetching the User's Notification Preferences #683
• Create a function for setting the User Notification Preferences #682
• Create APIs for the User's preference related functions.  #688

PR:

• [442]-Backend for Notification System (Phase 1) #730

Week 4

• Create functions for creating notification, deleting notification, fetching notification history, fetching notifications for a user, mark notification as read or unread.
• Create APIs for managing notification related functions.
• Create email and discord handlers.
• Create a celery worker for scheduling notification.

Issues:

• Enable login authorization for the APIs. #689
• Test all the functions through APIs. #690
• Create a Function to send a Notification #677
• Create a function for fetching all Notifications #678
• Create a function for fetching notifications for a specific user. #679
• Create a function for updating the read status of a notification #680
• Create a function for deleting Notification(s) #681

PR:

• [442] - Backend for Notification System (Phase 2) #744

Week 5

• Write pytests for notification APIs.
• Write pytests for notification preferences APIs.
• Write pytests for notification core functions.

Issues:

• Add pytests for notification APIs #746
• Add pytests for notification core functions #747
• Add pytests for user preferences APIs #745

PR:

• [442] - Backend for Notification System (Phase 2) #744

Week 6

• Create UI for user panel to see notifications.
• Integrate user panel with backend APIs.

Week 7

• Add an API for fetching unread notification count.
• Add tests for this API.
• Integrate it with user panel frontend.

Issues

• Create an API for fetching the unread notifications count #767

PR:

• get unread notification count api #766

Week 8

• Add a pop up to update user's notification preferences.
• Intergrate user preferences with backend APIs.

Week 9

• Planning and discussion for writing scripts to create and delete notifications
• Assigned to 3 different tickets other than this ticket.

Issues

• Move the Save Button webapp#1068

Week 9

• Make the save button floating when scrolling throughout the table.

Issues

• Move the Save Button webapp#1068

PR:

• search added + floating save button webapp#1119

Week 10

• Add column search in all the forms.

Issues

• Column search in all forms webapp#1065
• Add column-search in certain op-forms webapp#1040

PR:

• search added + floating save button webapp#1119

Week 11

• Explored the prefect-proxy repository and planned the changes required for the pagination to prefect flow run logs #789
• Made changes in the prefect-proxy repo to add pagination to logs.

PR:

• Updates for fetching flow run and task run logs prefect-proxy#154

Week 12

• Created a new API to fetch graph for a flow run.
• Added limit, offset and task_run_id to the logs API to fetch logs for a task run with pagination.

Issues:

• pagination to prefect flow run logs #789

PR:

• [442] - Pagination added for logs #832

Week 13

• Changes in the webapp application to apply pagination to the logs.

PR

• Pagination added for logs webapp#1143

Week 14

• Added scripts for creating and deleting notifications.
• Tested all the use cases.

PR

• [442] - Scripts for creating and deleting notification #847

[fatchat]

Weekly Goals

Week 1

• Talked about the project, architecture, approach

Week 2

• Set up the codebase locally.
• Make a documentation file for the ticket.
• Learn how to make migrations and revert them.

Week 3

• Create notification, recipients, and user preference table models. ([442]-Backend for Notification System (Phase 1) #730)
• Create functions for managing user preference. ([442]-Backend for Notification System (Phase 1) #730)

Week 4

• Create functions for handling fetch, create, delete, update read_status of notifications and sending notification through email and discord.

Week 5

• Manage scheduling of notifications.
• Wrote pytests for all backend functions

Week 6

• UI to view notifications

Week 7

• API to get unread count

Week 8

• Notification Preferences panel

Week 9

• Worked on search added + floating save button webapp#1119

Week 10

• Completed search added + floating save button webapp#1119

Week 11

• Updates for fetching flow run and task run logs prefect-proxy#154

Week 12

• [442] - Pagination added for logs #832

Week 13

• Pagination added for logs webapp#1143

Week 14

• Django management command to create a notification