First stable release of LLM Doc Scraper, featuring a complete documentation scraping solution with LLM-powered summarization.
Added
- Initial release of LLM Doc Scraper
- Web scraping functionality with BeautifulSoup
- LLM-powered content summarization using OpenAI API
- Streamlit UI for easy interaction
- SQLite database for URL history
- Customizable filename generation
- Token counting with tiktoken
- Database migration system
- Model selection with persistence
- OpenAI API key management
- Progress tracking with duration display
Features
- 🌐 Intelligent web scraping with link detection
- 🤖 LLM-based content summarization
- 📝 Clean markdown output format
- 💾 Persistent URL history
- 📁 Smart file organization
- 🎯 Custom filename support
- 🖥️ User-friendly interface
- ⚡ Token counting and timing
- 🔄 Database migrations
- 🔑 Secure API key storage