Skip to content

Releases: vialcollet/llm-doc-scraper

v1.0.0 - First Stable Release

12 Dec 12:48
Compare
Choose a tag to compare

First stable release of LLM Doc Scraper, featuring a complete documentation scraping solution with LLM-powered summarization.

Added

  • Initial release of LLM Doc Scraper
  • Web scraping functionality with BeautifulSoup
  • LLM-powered content summarization using OpenAI API
  • Streamlit UI for easy interaction
  • SQLite database for URL history
  • Customizable filename generation
  • Token counting with tiktoken
  • Database migration system
  • Model selection with persistence
  • OpenAI API key management
  • Progress tracking with duration display

Features

  • 🌐 Intelligent web scraping with link detection
  • 🤖 LLM-based content summarization
  • 📝 Clean markdown output format
  • 💾 Persistent URL history
  • 📁 Smart file organization
  • 🎯 Custom filename support
  • 🖥️ User-friendly interface
  • ⚡ Token counting and timing
  • 🔄 Database migrations
  • 🔑 Secure API key storage