Skip to content

PyPI Package (30,000+ downloads): scrapes Goodreads user data (profile, bookshelves, books, authors, etc.)

License

Notifications You must be signed in to change notification settings

YashTotale/goodreads-user-scraper

Repository files navigation

Goodreads Icon

Goodreads User Scraper

Scrape Goodreads User Data: Profile, Book Shelves, Books, Authors

Version  Downloads  Build 

Contents

Usage

Using pip:

pip install goodreads-user-scraper
goodreads-user-scraper --user_id <your id> --output_dir goodreads-data

Using pipx:

pipx run goodreads-user-scraper --user_id <your id> --output_dir goodreads-data

Arguments

--user_id

  • Description: The user whose data should be scraped. Find your user id using these directions.
  • Required: Yes

--output_dir

  • Description: The directory where all scraped data will be output.
  • Required: No
  • Default: goodreads-data

--skip_user_info

  • Description: Whether the script should skip scraping user information.
  • Required: No
  • Default: False

--skip_shelves

  • Description: Whether the script should skip scraping shelves.
  • Required: No
  • Default: False

--skip_authors

  • Description: Whether the script should skip scraping authors.
  • Required: No
  • Default: False

Troubleshooting

Ensure that your profile is viewable by anyone:

  1. Navigate to the Goodreads Account Settings page
  2. Click on the Account & notifications tab
  3. In the Privacy section, under Who Can View My Profile, select "anyone". Save your account settings.

Development

  1. Clone the GitHub repository

    git clone https://github.com/YashTotale/goodreads-user-scraper.git
  2. Run the install script

    sh scripts/install.sh
  3. Make changes

  4. Run the test script

    sh scripts/test.sh

Publishing

  1. Create .env

    TWINE_USERNAME=<foo>
    TWINE_PASSWORD=<bar>
    
  2. Run the publish script

    sh scripts/publish.sh <patch|minor|major>