SignGPT is a web application that converts input text prompts to sign language instructions using Llama 3 for Natural Language Processing (NLP) and Stable Diffusion XL for generating images based on the descriptions. The application is built using Flask, Python, HTML, CSS, and JavaScript.
- Converts text prompts to sign language instructions.
- Uses Llama 3 for NLP to generate descriptions of sign language instructions.
- Uses Stable Diffusion XL to generate images from descriptions.
- User-friendly web interface.
- Entering the prompt: You start by entering the prompt in the Text Area given on the home page. After you enter the prompt click the button to submit your request.
- Generating instructions: After you have submitted the request your prompt is sent to a remotely hosted Llama 3 model hosted using HuggingFace Spaces to convert the given prompt into sign language instructions.
- Receiving Prompts: After the prompts are converted to sign language instructions, these are further refined to extract just the instructions from the additional texts.
- Forwarding Prompts to Image Generation Model: Once we have the refined prompts we forward them to remotely hosted Stable Diffusion XL Model to generate and receive images.
- Displaying Images: Once we receive the generated images fromt the SDXL we pass them dynmaically from backend to frontend and display them with the prompts respectively.
- Python
- Flask
- Hugging Face Transformers
- Stable Diffusion XL
- HTML, CSS, JavaScript
- Llama 3 API from HuggingFace
- Stable Diffusion XL API from HuggingFace
- Flask Library
- HuggingFace Spaces
- HTML
- CSS
- JavaScript
The accuracy of the images and the prompts are totally dependent on the model used and the fine-tuning of the model. If you want better results from the model you can fine-tune the model on local storage and use them for better results.