Welcome to DocuThinker! This is a full-stack (FERN-Stack) application that integrates an AI-powered document processing backend with a React-based frontend. The app allows users to upload documents for summarization, generate key insights, and chat with an AI based on the document's content.
- 📖 Overview
- 🚀 Live Deployments
- ✨ Features
- ⚙️ Technologies
- 🖼️ User Interfaces
- 📂 Complete File Structure
- 🛠️ Getting Started
- 📋 API Endpoints
- 📱 Mobile App
- 📦 Containerization
- 🚀 Deployment
- ⚖️ Load Balancing & Caching
- 🔗 Jenkins Integration
- 🚢 Kubernetes Integration
- 🔧 Contributing
- 📝 License
- 📚 Additional Documentation
- 👨💻 Author
The DocuThinker app is designed to provide users with a simple, AI-powered document management tool. Users can upload PDFs or Word documents and receive summaries, key insights, and discussion points. Additionally, users can chat with an AI using the document's content for further clarification.
DocuThinker is created using the FERN-Stack architecture, which stands for Firebase, Express, React, and Node.js. The backend is built with Node.js and Express, integrating Firebase for user authentication and MongoDB for data storage. The frontend is built with React and Material-UI, providing a responsive and user-friendly interface.
It is currently deployed live on Vercel and Render. You can access the live app here.
We have deployed the entire app on Vercel and Render. You can access the live app here.
- Frontend: Deployed on Vercel.
- Backend: Deployed on Render. You can access the live backend here.
Feel free to explore the live app and test the functionalities. You can upload documents, generate summaries, and chat with an AI using the live app.
Note: The backend server may take a few seconds to wake up if it has been inactive for a while. The first API call may take a bit longer to respond. Subsequent calls should be faster as the server warms up. Additionally, the app is currently on the Free Tier of Render, so it may take longer to process your request since we are only allocated 512MB and 0.1 CPU.
Also: We are using Netlify for backups and testing. You can access the backup app here.
DocuThinker offers a wide range of features to help users manage and analyze their documents effectively. Here are some of the key features of the app:
- Document Upload & Summarization: Upload PDFs or Word documents for AI-generated summaries.
- Key Insights & Discussion Points: Generate important ideas and topics for discussion from your documents.
- AI Chat Integration: Chat with an AI using your document’s original context.
- Voice Chat with AI: Chat with an AI using voice commands for a more interactive experience.
- Sentiment Analysis: Analyze the sentiment of your document text for emotional insights.
- Multiple Language Support: Summarize documents in different languages for global users.
- Content Rewriting: Rewrite or rephrase document text based on a specific style or tone.
- Actionable Recommendations: Get actionable recommendations based on your document content.
- Bullet Point Summaries: Generate bullet point summaries for quick insights and understanding.
- Document Categorization: Categorize documents based on their content for easy organization.
- Profile Management: Update your profile information, social media links, and theme settings.
- User Authentication: Secure registration, login, and password reset functionality.
- Document History: View all uploaded documents and their details.
- Mobile App Integration: React Native mobile app for on-the-go document management.
- API Documentation: Swagger (OpenAPI) documentation for all API endpoints.
- Authentication Middleware: Secure routes with JWT and Firebase authentication middleware.
- Containerization: Dockerized the app with Docker & K8s for easy deployment and scaling.
- Continuous Integration: Automated testing and deployment with GitHub Actions & Jenkins.
- Frontend:
- React: JavaScript library for building user interfaces.
- Material-UI: React components for faster and easier web development.
- Axios: Promise-based HTTP client for making API requests.
- React Router: Declarative routing for React applications.
- Context API: State management for React applications.
- Tailwind CSS: Utility-first CSS framework for styling.
- Craco: Create React App Configuration Override for customizing Webpack.
- Webpack: Module bundler for JavaScript applications.
- Backend:
- Express: Web application framework for Node.js.
- Redis: In-memory data structure store for caching.
- Firebase Admin SDK: Firebase services for server-side applications.
- Node.js: JavaScript runtime for building scalable network applications.
- Firebase Authentication: Secure user authentication with Firebase.
- Firebase Auth JWT: Generate custom tokens for Firebase authentication.
- Middlewares: Firebase authentication middleware for securing routes and JWT middleware for token verification.
- AI/ML Services:
- Google Cloud Natural Language API: Machine learning models for text analysis.
- Google Gemini API: AI-powered chatbot for enhanced user interaction.
- Google AI Studio: Tools for building and deploying machine learning models.
- NLP: Natural Language Processing for customized chat/text analysis and summarization models.
- NER: Named Entity Recognition for identifying entities in text.
- POS Tagging: Part-of-Speech Tagging for analyzing word types in text.
- Database:
- MongoDB: NoSQL database for storing user data and documents.
- Firestore: Cloud Firestore for storing user data and documents.
- Redis: In-memory data structure store for caching.
- Mobile App:
- React Native: JavaScript framework for building mobile applications.
- Expo: Framework and platform for universal React applications.
- Firebase SDK: Firebase services for mobile applications.
- React Navigation: Routing and navigation for React Native apps.
- API Documentation:
- Swagger: OpenAPI documentation for all API endpoints.
- Containerization:
- Docker: Containerization platform for building, shipping, and running applications.
- Kubernetes: Container orchestration for automating deployment, scaling, and management.
- Load Balancing & Caching:
- NGINX: Web server for load balancing, reverse proxying, and caching.
- CI/CD & Deployment:
- GitHub Actions: Automated workflows for testing and deployment.
- Jenkins: Automation server for continuous integration and deployment.
- Render: Cloud platform for hosting and scaling web applications. (Used to deploy the backend)
- Vercel: Cloud platform for hosting and deploying web applications. (Used to deploy the frontend)
- Netlify: Cloud platform for hosting and deploying web applications. (Used as a backup)
DocuThinker features a clean and intuitive user interface designed to provide a seamless experience for users. The app supports both light and dark themes, responsive design, and easy navigation. Here are some screenshots of the app:
The DocuThinker app is organized into separate subdirectories for the frontend, backend, and mobile app. Each directory contains the necessary files and folders for the respective components of the app. Here is the complete file structure of the app:
DocuThinker-AI-App/
├── backend/
│ ├── ai_ml/
│ │ ├── perform_ner_pos.py # Named Entity Recognition and Part-of-Speech Tagging
│ │ ├── sen_analysis.py # Sentiment analysis for document text
│ │ ├── chat.js # Chatbot integration for AI chat functionality
│ │ ├── analyzer.js # Document analyzer for generating key ideas and discussion points
│ │ ├── textStatistics.js # Text statistics for analyzing document content
│ │ ├── documentClassifier.js # Document classifier for categorizing documents
│ │ ├── summarizer.js # Document summarizer for generating summaries
│ │ └── (and many more files...) # Additional AI/ML services
│ ├── middleware/
│ │ └── jwt.js # Authentication middleware with JWT for the app's backend
│ ├── controllers/
│ │ └── controllers.js # Controls the flow of data and logic
│ ├── models/
│ │ └── models.js # Data models for interacting with the database
│ ├── services/
│ │ └── services.js # Models for interacting with database and AI/ML services
│ ├── views/
│ │ └── views.js # Output formatting for success and error responses
│ ├── .env # Environment variables (git-ignored)
│ ├── firebase-admin-sdk.json # Firebase Admin SDK credentials (git-ignored)
│ ├── index.js # Main entry point for the server
│ ├── Dockerfile # Docker configuration file
│ ├── manage_server.sh # Shell script to manage and start the backend server
│ └── README.md # Backend README file
│
├── frontend/
│ ├── public/
│ │ ├── index.html # Main HTML template
│ │ └── manifest.json # Manifest for PWA settings
│ ├── src/
│ │ ├── assets/ # Static assets like images and fonts
│ │ │ └── logo.png # App logo or images
│ │ ├── components/
│ │ │ ├── ChatModal.js # Chat modal component
│ │ │ ├── Spinner.js # Loading spinner component
│ │ │ ├── UploadModal.js # Document upload modal component
│ │ │ ├── Navbar.js # Navigation bar component
│ │ │ ├── Footer.js # Footer component
│ │ │ └── GoogleAnalytics.js # Google Analytics integration component
│ │ ├── pages/
│ │ │ ├── Home.js # Home page where documents are uploaded
│ │ │ ├── LandingPage.js # Welcome and information page
│ │ │ ├── Login.js # Login page
│ │ │ ├── Register.js # Registration page
│ │ │ ├── ForgotPassword.js # Forgot password page
│ │ │ └── HowToUse.js # Page explaining how to use the app
│ │ ├── App.js # Main App component
│ │ ├── index.js # Entry point for the React app
│ │ ├── App.css # Global CSS 1
│ │ ├── index.css # Global CSS 2
│ │ ├── reportWebVitals.js # Web Vitals reporting
│ │ ├── styles.css # Custom styles for different components
│ │ └── config.js # Configuration file for environment variables
│ ├── .env # Environment variables file (e.g., REACT_APP_BACKEND_URL)
│ ├── package.json # Project dependencies and scripts
│ ├── craco.config.js # Craco configuration file
│ ├── Dockerfile # Docker configuration file
│ ├── manage_frontend.sh # Shell script for managing and starting the frontend
│ ├── README.md # Frontend README file
│ └── package.lock # Lock file for dependencies
│
├── mobile-app/ # Mobile app directory
│ ├── app/ # React Native app directory
│ ├── .env # Environment variables file for the mobile app
│ ├── app.json # Expo configuration file
│ ├── components/ # Reusable components for the mobile app
│ ├── assets/ # Static assets for the mobile app
│ ├── constants/ # Constants for the mobile app
│ ├── hooks/ # Custom hooks for the mobile app
│ ├── scripts/ # Scripts for the mobile app
│ ├── babel.config.js # Babel configuration file
│ ├── package.json # Project dependencies and scripts
│ └── tsconfig.json # TypeScript configuration file
│
├── kubernetes/ # Kubernetes configuration files
│ ├── manifests/ # Kubernetes manifests for deployment, service, and ingress
│ ├── backend-deployment.yaml # Deployment configuration for the backend
│ ├── backend-service.yaml # Service configuration for the backend
│ ├── frontend-deployment.yaml # Deployment configuration for the frontend
│ ├── frontend-service.yaml # Service configuration for the frontend
│ ├── firebase-deployment.yaml # Deployment configuration for Firebase
│ ├── firebase-service.yaml # Service configuration for Firebase
│ └── configmap.yaml # ConfigMap configuration for environment variables
│
├── nginx/
│ ├── nginx.conf # NGINX configuration file for load balancing and caching
│ └── Dockerfile # Docker configuration file for NGINX
│
├── images/ # Images for the README
├── .env # Environment variables file for the whole app
├── docker-compose.yml # Docker Compose file for containerization
├── jsconfig.json # JavaScript configuration file
├── package.json # Project dependencies and scripts
├── package-lock.json # Lock file for dependencies
├── postcss.config.js # PostCSS configuration file
├── tailwind.config.js # Tailwind CSS configuration file
├── render.yaml # Render configuration file
├── vercel.json # Vercel configuration file
├── manage_docuthinker.sh # Shell script for managing and starting the app (both frontend & backend)
├── jenkins_cicd.sh # Shell script for managing the Jenkins CI/CD pipeline
├── .gitignore # Git ignore file
├── LICENSE.md # License file for the project
├── README.md # Comprehensive README for the whole app
└── (and many more files...) # Additional files and directories not listed here
Ensure you have the following tools installed:
- Node.js (between v14 and v20)
- npm or yarn
- Firebase Admin SDK credentials
- Redis for caching
- MongoDB for data storage
- RabbitMQ for handling asynchronous tasks
- Docker for containerization (optional)
- Postman for API testing (optional)
- Expo CLI for running the mobile app
- Jenkins for CI/CD (optional)
- Kubernetes for container orchestration (optional)
- React Native CLI for building the mobile app
- Firebase SDK for mobile app integration
- Firebase API Keys and Secrets for authentication
- Expo Go app for testing the mobile app on a physical device
- Tailwind CSS for styling the frontend
- .env file with necessary API keys (You can contact me to get the
.envfile - but you should obtain your own API keys for production).
Additionally, basic fullstack development knowledge and AI/ML concepts are recommended to understand the app's architecture and functionalities.
-
Clone the repository:
git clone https://github.com/hoangsonww/DocuThinker-AI-App.git cd DocuThinker-AI-App/backend -
Navigate to the frontend directory:
cd frontend -
Install dependencies:
npm install
-
Start the Frontend React app:
npm start
-
Build the Frontend React app (for production):
npm run build
-
Alternatively, you can use
yarnto install dependencies and run the app:yarn install yarn start
-
Or, for your convenience, if you have already installed the dependencies, you can directly run the app in the root directory using:
npm run frontend
This way, you don't have to navigate to the
frontenddirectory every time you want to run the app. -
The app will run on
http://localhost:3000. You can access it in your browser.
Note that this is optional since we are deploying the backend on Render. However, you can (and should) run the backend locally for development purposes.
-
Navigate to the root (not
backend) directory:cd backend -
Install dependencies:
npm install
-
Start the backend server:
npm run server
-
The backend code is in the
backenddirectory. Feel free to explore the API endpoints and controllers.
Note: Be sure to use Node v.20 or earlier to avoid compatibility issues with Firebase Admin SDK.
- Navigate to the mobile app directory:
cd mobile-app - Install dependencies:
npm install
- Start the Expo server:
npx expo start
- Run the app on an emulator or physical device: Follow the instructions in the terminal to run the app on an emulator or physical device.
The backend of DocuThinker provides several API endpoints for user authentication, document management, and AI-powered insights. These endpoints are used by the frontend to interact with the backend server:
| Method | Endpoint | Description |
|---|---|---|
| POST | /register |
Register a new user in Firebase Authentication and Firestore, saving their email and creation date. |
| POST | /login |
Log in a user and return a custom token along with the user ID. |
| POST | /upload |
Upload a document for summarization. If the user is logged in, the document is saved in Firestore. |
| POST | /generate-key-ideas |
Generate key ideas from the document text. |
| POST | /generate-discussion-points |
Generate discussion points from the document text. |
| POST | /chat |
Chat with AI using the original document text as context. |
| POST | /forgot-password |
Reset a user's password in Firebase Authentication. |
| POST | /verify-email |
Verify if a user's email exists in Firestore. |
| GET | /documents/{userId} |
Retrieve all documents associated with the given userId. |
| GET | /documents/{userId}/{docId} |
Retrieve a specific document by userId and docId. |
| GET | /document-details/{userId}/{docId} |
Retrieve document details (title, original text, summary) by userId and docId. |
| DELETE | /delete-document/{userId}/{docId} |
Delete a specific document by userId and docId. |
| DELETE | /delete-all-documents/{userId} |
Delete all documents associated with the given userId. |
| POST | /update-email |
Update a user's email in both Firebase Authentication and Firestore. |
| POST | /update-password |
Update a user's password in Firebase Authentication. |
| GET | /days-since-joined/{userId} |
Get the number of days since the user associated with userId joined the service. |
| GET | /document-count/{userId} |
Retrieve the number of documents associated with the given userId. |
| GET | /user-email/{userId} |
Retrieve the email of a user associated with userId. |
| POST | /update-document-title |
Update the title of a document in Firestore. |
| PUT | /update-theme |
Update the theme of the app. |
| GET | /user-joined-date/{userId} |
Get date when the user associated with userId joined the service. |
| GET | /social-media/{userId} |
Get the social media links of the user associated with userId. |
| POST | /update-social-media |
Update the social media links of the user associated with userId. |
| POST | /update-profile |
Update the user's profile information. |
| POST | /update-document/{userId}/{docId} |
Update the document details in Firestore. |
| POST | /update-document-summary |
Update the summary of a document in Firestore. |
| POST | /sentiment-analysis |
Analyzes the sentiment of the provided document text |
| POST | /bullet-summary |
Generates a summary of the document text in bullet points |
| POST | /summary-in-language |
Generates a summary in the specified language |
| POST | /content-rewriting |
Rewrites or rephrases the provided document text based on a style |
| POST | /actionable-recommendations |
Generates actionable recommendations based on the document text |
More API endpoints will be added in the future to enhance the functionality of the app. Feel free to explore the existing endpoints and test them using Postman or Insomnia.
- Swagger Documentation: You can access the Swagger documentation for all API endpoints by running the backend server and navigating to
http://localhost:5000/api-docs. - Redoc Documentation: You can access the Redoc documentation for all API endpoints by running the backend server and navigating to
http://localhost:5000/api-docs/redoc.
For example, our API endpoints documentation looks like this:
- We use Node.js and Express to build the backend server for DocuThinker.
- The backend API is structured using Express and Firebase Admin SDK for user authentication and data storage.
- We use the MVC (Model-View-Controller) pattern to separate concerns and improve code organization.
- Models: Schema definitions for interacting with the database.
- Controllers: Handle the business logic and interact with the models.
- Views: Format the output and responses for the API endpoints.
- Services: Interact with the database and AI/ML services for document analysis and summarization.
- Middlewares: Secure routes with Firebase authentication and JWT middleware.
- The API endpoints are designed to be RESTful and follow best practices for error handling and response formatting.
- The Microservices Architecture is also used to handle asynchronous tasks and improve scalability.
- The API routes are secured using Firebase authentication middleware to ensure that only authenticated users can access the endpoints.
- The API controllers handle the business logic for each route, interacting with the data models and formatting the responses.
-
You can test the API endpoints using Postman or Insomnia. Simply make a POST request to the desired endpoint with the required parameters.
-
For example, you can test the
/uploadendpoint by sending a POST request with the document file as a form-data parameter. -
Feel free to test all the API endpoints and explore the functionalities of the app.
curl --location --request POST 'http://localhost:3000/register' \
--header 'Content-Type: application/json' \
--data-raw '{
"email": "[email protected]",
"password": "password123"
}'curl --location --request POST 'http://localhost:3000/upload' \
--header 'Authorization: Bearer <your-token>' \
--form 'File=@"/path/to/your/file.pdf"'The backend APIs uses centralized error handling to capture and log errors. Responses for failed requests are returned with a proper status code and an error message:
{
"error": "An internal error occurred",
"details": "Error details go here"
}The DocuThinker mobile app is built using React Native and Expo. It provides a mobile-friendly interface for users to upload documents, generate summaries, and chat with an AI. The mobile app integrates with the backend API to provide a seamless experience across devices.
Currently, it is in development and will be released soon on both the App Store and Google Play Store.
Stay tuned for the release of the DocuThinker mobile app!
Below is a screenshot of the mobile app (in development):
The DocuThinker app can be containerized using Docker for easy deployment and scaling. Follow these steps to containerize the app:
- Run the following command to build the Docker image:
docker compose up --build
- The app will be containerized and ready to run on port 3000.
You can also view the image in the Docker Hub repository here.
-
Install the Vercel CLI:
npm install -g vercel
-
Deploy the frontend:
vercel
-
Follow the instructions in your terminal to complete the deployment.
-
The backend can be deployed on platforms like Heroku, Render, or Vercel.
-
Currently, we are using Render to host the backend. You can access the live backend here.
-
Please note that we are currently on the Free Tier of Render. This means that the backend server may take a few seconds to wake up if it has been inactive for a while.
-
Therefore, the first API call may take a bit longer to respond. Subsequent calls should be faster as the server warms up. It is completely normal to take up to 2 minutes for the first API call to respond.
-
Also, the Free Tier of Render only allocates 512MB and 0.1 CPU. This may result in slower response times for API calls and document processing.
-
Additionally, during high traffic periods, the server may take longer to respond, although we have employed NGINX for load balancing and caching.
- We are using NGINX for load balancing and caching to improve the performance and scalability of the app.
- The NGINX configuration file is included in the repository for easy deployment. You can find the file in the
nginxdirectory. - Feel free to explore the NGINX configuration file and deploy it on your own server for load balancing and caching.
- NGINX can also be used for SSL termination, reverse proxying, and serving static files. More advanced configurations can be added to enhance the performance of the app.
- You can also use Cloudflare or AWS CloudFront for content delivery and caching to improve the speed and reliability of the app, but we are currently using NGINX for load balancing and caching due to costs and simplicity.
- For more information, refer to the NGINX Directory.
- The NGINX configuration file is included in the repository for easy deployment. You can find the file in the
- We are also using Docker with NGINX to deploy the NGINX configuration file and run the server in a containerized environment. The server is deployed and hosted on Render.
- Additionally, we are using Redis for in-memory caching to store frequently accessed data and improve the performance of the app.
- Redis can be used for caching user sessions, API responses, and other data to reduce the load on the database and improve response times.
- You can set up your own Redis server or use a managed service like Redis Labs or AWS ElastiCache for caching.
- We are using Jenkins for continuous integration and deployment. The Jenkins pipeline is set up to automatically test and deploy the app whenever changes are pushed to the main branch.
- The pipeline runs the tests, builds the app, and deploys it to Vercel and Render. Feel free to visit the pipeline at
Jenkinsfile. - The pipeline is triggered automatically whenever a new commit is pushed to the main branch.
- You can set up your own Jenkins pipeline to automate testing and deployment for your projects by following these commands and steps:
-
Install Jenkins:
brew install jenkins
-
Start Jenkins:
brew services start jenkins
-
Access Jenkins: Open your browser and go to
http://localhost:8080to access the Jenkins dashboard. -
Follow the instructions to set up Jenkins and create a new pipeline.
If successful, you should see the Jenkins pipeline running and deploying the app automatically whenever changes are pushed to the main branch. Here is an example:
- We are using Kubernetes for container orchestration and scaling. The app can be deployed on a Kubernetes cluster for high availability and scalability.
- The Kubernetes configuration files are included in the repository for easy deployment. You can find the files in the
kubernetesdirectory. - Feel free to explore the Kubernetes configuration files and deploy the app on your own Kubernetes cluster.
- You can also use Google Kubernetes Engine (GKE), Amazon EKS, or Azure AKS to deploy the app on a managed Kubernetes cluster.
We welcome contributions from the community! Follow these steps to contribute:
-
Fork the repository.
-
Create a new branch:
git checkout -b feature/your-feature
-
Commit your changes:
git commit -m "Add your feature" -
Push the changes:
git push origin feature/your-feature
-
Submit a pull request: Please submit a pull request from your forked repository to the main repository. I will review your changes and merge them into the main branch shortly.
Thank you for contributing to DocuThinker! 🎉
This project is licensed under the Creative Commons Attribution-NonCommercial License. See the LICENSE file for details.
The DocuThinker open-source project is for educational purposes only and should not be used for commercial applications. Feel free to use it for learning and personal projects!
- Son Nguyen - An aspiring Software Developer & Data Scientist
- Feel free to connect with me on LinkedIn.
- If you have any questions or feedback, please feel free to reach out to me at [email protected].
- Also, check out my portfolio for more projects and articles.
- If you find this project helpful or you learned something from the source code, consider giving it a star ⭐️. I would greatly appreciate it! 🚀
Happy Coding and Analyzing! 🚀
Created with ❤️ by Son Nguyen in 2024.