Home
During Google Summer of Code 2024, I worked on the AI Emoji Embellisher App for Rocket.Chat. This app provides open-source LLM integrations for Rocket.Chat, allowing users to transform plain text messages into emojified and embellished text within Rocket.Chat channels.
This wiki is used to maintain a summary report of my GSoC work and as a quick guide for future GSoC aspirants.
Text embellishment is a natural language generation problem that aims to enhance the lexical and syntactic complexity of a text while retaining the same semantic information and meaning. This is often achieved by incorporating emojis, punctuation, and additional details to make the text more interesting and engaging. This wiki delves into the implementation details of the Embellisher App for Rocket.Chat. This app will enable users to generate emojified / embellished text messages from plain text messages they enter in the message box using open-source LLMs. Upon generation, the user will be notified with an interactive message that only the sender can see. Using the interactive action button, the user can copy, edit, redo, or send the embellished text message.
Text embellishment plays a vital role in enhancing communication, fostering engagement, conveying emotion, encouraging social sharing, and making written content more memorable and impactful. By leveraging creative elements effectively, writers can create content that not only informs but also entertains, inspires, and connects with readers. Thus, the importance of text embellishment lies in its ability to enhance communication.
During Google Summer of Code 2024, I focused on the following deliverables.
- 1. Slash Command Interface: Enables users to emojify and embellish their plain text messages using
/embellishslash command. - 2. Interactive Notified Messages: The LLM generated response is notified with block action buttons to perform forward, edit, redo and send operations.
- 3. Forward Functionality: Provides an interactive modal with an input field, to enter a channel name where a user wants to inject the LLM-generated response.
- 4. Edit and Send Functionality: Provides an interactive edit modal that allows users to edit the generated response and send it into the current channel.
- 5. Redo Functionality: Allows users to regenerate the LLM response using the redo modal to adjust the level of emojification and to provide any additional instruction prompts as needed.
- 6. Model Selection Settings: Enables users to select a model from the various in-house LLMs available for message embellishment. Alternatively, users can also provide access to their own LLM APIs in the app settings.
- 7. Use-case Selection Settings: Allows users to choose from a variety of use cases based on their communication needs, such as event promotions, marketing and sales, or customer support. Consequently, the app internally selects the appropriate prompts for the chosen use case.
- 8. Subcommand Interface: The
/embellish modeland/embellish helpsubcommands inform users of the current model in use and provide a brief overview of the app's functionality, respectively. - 9. On Installation Message: When users install the Embellisher app from the marketplace, they are introduced to the app by our bot. The welcome message explains the basics of the app, outlines the setup requirements, and provides guidance on how to get started.
- 1. LLM Response Streaming: Modification of the LLM inference handler to allow streamed response generation at the client side.
- 2. Marketplace Deployment: Publish the Embellisher App on the Rocket.Chat Apps Marketplace.
This section provides a video demonstration for each feature of the Embellisher App.
Upon installing the Embellisher App, users will receive a welcoming message that introduces them to its functionalities. This message includes an overview of the app's features and essential information to help them get started.
on-install.mov
Users can utilize the /embellish help command to quickly access information about all the available subcommands in our app. This command assists users by providing a list of commands along with explanations for each one."
helper-message.mov
This is the main slash command which enables users to emojify and embellish their plain text messages using /embellish followed by their text messages.
embellish-command.mov
These block action buttons are used to perform forward, edit, redo and send operations:
- Forward button provides an interactive modal with an input field, to enter a channel name where a user wants to inject the LLM-generated response.
- Edit and Send button provides an interactive edit modal that allows users to edit the generated response and send it into the current channel respectively.
- Redo button allows users to regenerate the LLM response using the redo modal to adjust the level of emojification and to provide any additional instruction prompts as needed.
block-actions.mov
Users can choose a model from the various in-house LLMs available for message embellishment. Additionally, users have the option to provide access to their own LLM APIs in the app settings. They can then verify their selected in-house LLM using the /embellish model command, which provides details about the chosen model.
model-selection.mov
Allows users to choose from a variety of use cases based on their communication needs, such as event promotions, marketing and sales, or customer support. Consequently, the app internally selects the appropriate prompts for the chosen use case. For example:
- The default
Text Communicationuse case offers a system prompt designed to emojify your text messages, enhancing your communication. - In contrast, the
Event Promotionsuse case provides a system prompt that not only emojifies but also embellishes and formats user messages in an appealing manner for business communication.
use-case.mov
💡 Contributions to Embellisher App
During the coding period, I divided my work into several modular features, each of which was tracked as an issue. Subsequently, each feature was implemented in its own git branch, and corresponding pull requests (PRs) were raised.
| PR Link | Description | Status |
|---|---|---|
| PR #1 | Base for the Embellisher App |  |
| PR #3 | [feat1]: Added slash command and LLM inference API | |
| PR #5 | [feat2]: Added app settings and system prompt config | |
| PR #7 | [feat3]: Added prompt persistence and initiator message | |
| PR #11 | [feat4]: Added block action handlers and send message function | |
| PR #12 | [feat5]: Added room interaction persistence and edit & redo modal | |
| PR #13 | [feat6]: Added edit and redo - block action and view submit handler | |
| PR #17 | [feat7]: Updated emojify guidelines and added emoji persistence | |
| PR #18 | [feat8]: Modified app settings and inference call for user hosted LLMs | |
| PR #19 | [feat9]: Added system prompts for more use cases | |
| PR #24 | [feat11]: Added Forward Functionality | |
| PR #25 | [enh]: Added app icon and Permission settings | |
| PR #26 | [feat12]: Added On Install and Helper Message | |
| Issue Link | Description | Status |
|---|---|---|
| Issue #2 | [feat1]: Slash Commands and LLM Inference |  |
| Issue #4 | [feat2]: App Settings and System Prompt configuration | |
| Issue #6 | [feat3]: Prompt Persistence and Initiator Message | |
| Issue #8 | [feat4]: Block Handlers and Send Functionality | |
| Issue #9 | [feat5]: Room Persistence and Edit / Redo Modal | |
| Issue #10 | [feat6]: Edit and Redo - Block Action and View Submit Handler | |
| Issue #14 | [feat7]: Improve Emojification Guidelines | |
| Issue #15 | [feat8]: App Settings and Fields for User Hosted LLMs | |
| Issue #16 | [feat9]: Additional Prompts for Customer Support and Healthcare use cases | |
| Issue #20 | [feat10]: Copy Action - Alternative Forward Functionality |  |
| Issue #21 | [feat11]: Forward Functionality - Block Action and View Submit Handler | |
| Issue #22 | [feat12]: On Install and Helper Messages | |
| Issue #23 | [enh]: Code Clean-up and UI Enhancements | |
This section details my key learnings and roadblocks I faced during the GSoC period.
-
LLM Inference Handlers: In the early stages, when I gained access to the in-house LLMs, testing each model individually proved to be tedious. I had to manually change the model settings in the inference handler code and redeploy the app each time, making the process time-consuming. Upon reviewing others' codebases, I discovered a solution - of creating a settings file containing all available models. This approach allowed me to switch models directly from the client side instead of altering the code and redeploying. The key takeaway here is the importance of asking questions and learning from others.
-
Room Interaction Storage: While working with the Apps Engine UI Kit modals, I encountered an issue where the required output was not produced upon clicking the submit button. I struggled with this problem for an extended period until I examined the Rocket.Chat GitHub App, which heavily utilized modals. Through this, I realized that room interactions need to be stored in persistence when dealing with modals.
-
Adjustment of Emojification Levels: When working on the redo functionality, my mentor advised me not to use model parameters like temperature and system prompts as inputs from users, as this could lead to a poor user experience. Instead, I initially mapped percentage values to the number of emojis using prompt designs, but this method was not sufficiently accurate. I then improved the approach by mapping percentage values to sentiments rather than emoji counts. For example, 0-20% would correspond to a "subtle" sentiment, while 40-60% would map to a "balanced" sentiment, resulting in more accurate outcomes. Additionally, when redoing a generated response with a different percentage, the output often did not meet expectations. I resolved this by modifying the prompts to consider the previous emojification percentage and comparing it with the current percentage requested by the user.
-
Forward Action Block: The forward action block opens a modal that requires access to all rooms a user is present in to inject messages into these rooms. However, even the rooms to which a user belongs were not accessible. The workaround for this was to allow users to directly specify the room name where they want to inject the message. This task initially posed a challenge, but with guidance from my mentor, I successfully implemented the functionality.
-
Copy Action Block: Since Rocket.Chat apps are deployed server-side, they cannot directly access client-side resources and APIs. The copy action requires access to client-side APIs, which is not feasible. As an alternative, I implemented a forward action block. The copy action would require some internal modifications in the apps engine which I intend to look into.
-
In-House Llama-70B Delayed Response: A major issue I faced was the delayed response when my app used the larger in-house model, Llama-70B. Due to the extended response time, the default app, Rocket.Cat, would notify an error, even though the required response would eventually be received after a few more seconds. The same issue arose when regenerating LLM responses with larger models; the modal interaction responder would trigger an error, but the result would still be obtained. The solution involved implementing LLM stream responses, which prevented errors and allowed me to receive responses in chunks, similar to how ChatGPT functions. However, in the Rocket.Chat apps system, notification messages cannot be deleted during a session, so after each chunk is received, a new notification appears instead of modifying the previous one. This is an issue I will continue to work on with my mentor.
Here to expresses my sincere gratitude to my mentors and provide my profile details for networking
I would like to express my sincere gratitude to my amazing mentors, Sing Li and Shubham Bhardwaj, for their invaluable guidance and support throughout my GSoC'24 journey 🚀. Their constant encouragement, technical expertise, and willingness to help have been instrumental in the success of this project. I feel incredibly fortunate to have had the opportunity to learn from such talented individuals. 🌟
I would also like to extend a heartfelt thank you to the entire Rocket.Chat developer community for their openness and support throughout the development process. Thank you both for your mentorship and for inspiring me to learn and grow beyond GSoC. 🙏
| Contributor | Sandeep Pillai |
|---|---|
| Organization | Rocket.Chat |
| Project | AI In-message Emoji Embellisher |
| GitHub | @sandeepB3 |
| Sandeep Pillai | |
| [email protected] | |
| Rocket.Chat | sandeep.pillai |
