Skip to content

Latest commit

 

History

History
80 lines (55 loc) · 2.7 KB

README.md

File metadata and controls

80 lines (55 loc) · 2.7 KB
Raw

Substack2Markdown

Substack2Markdown is a Python tool for scraping free and premium Substack posts and saving them as Markdown files. It will save paid for content as long as you're subscribed to that substack. Most "save for later" apps (such as Pocket) don't save these posts, but with this script you can now browse and sort through these posts in a user-friendly HTML interface.

Substack2Markdown Interface

Once you run the script, it will create a folder named after the substack in /substack_md_files, and then begin to scrape the substack URL, converting the blog posts into markdown files. Once all the posts have been saved, it will generate an HTML file in /substack_html_pages directory that allows you to browse the posts.

You can either hardcode the substack URL and the number of posts you'd like to save into the top of the file, or specify them as command line arguments.

Features

  • Converts Substack posts into Markdown files.
  • Generates an HTML file to browse Markdown files.
  • Supports free and premium content (with subscription).
  • The HTML interface allows sorting essays by date or likes.

Installation

Clone the repo and install the dependencies:

git clone https://github.com/yourusername/substack_scraper.git
cd substack_scraper

# # Optinally create a virtual environment
# python -m venv venv
# # Activate the virtual environment
# .\venv\Scripts\activate  # Windows
# source venv/bin/activate  # Linux

pip install -r requirements.txt

For the premium scraper, update the config.py in the root directory with your Substack email and password:

EMAIL = "[email protected]"
PASSWORD = "your-password"

You'll also need Microsoft Edge installed for the Selenium webdriver.

Usage

Specify the Substack URL and the directory to save the posts to:

You can hardcode your desired Substack URL and the number of posts you'd like to save into the top of the file and run:

python substack_scraper.py

For free Substack sites:

python substack_scraper.py --url https://example.substack.com --directory /path/to/save/posts

For premium Substack sites:

python substack_scraper.py --url https://example.substack.com --directory /path/to/save/posts --premium

To scrape a specific number of posts:

python substack_scraper.py --url https://example.substack.com --directory /path/to/save/posts --number 5

Viewing Markdown Files in Browser

To read the Markdown files in your browser, install the Markdown Viewer browser extension.