This repository contains code to replicate the no-longer-available Toronto BookCorpus dataset. To this end, it scrapes and downloads books from Smashwords, the source of the original dataset. Similarly, all books are written in English and contain at least 20k words.
Replicating the Toronto BookCorpus dataset consists of three steps:
- Getting the URLs of the plaintext books to download (optional)
- Downloading the plaintext books
- Pre-processing the plaintext books
The first step is optional, as I have already provided a list of download URLS in data/book_urls.txt ready to use. Nonetheless, you can recreate this list as follows:
python src/get_book_urls.pyProvided you have a list of book URLs in data/book_urls.txt, you can download the plaintext books as follows:
python src/download_books.pyPlease note that you have to execute the above command multiple times (~30 times to be more precise), from multiple IP-addresses, as Smashwords (temporarily) blocks any IP-address after 500 downloads. If you know of a way to automate this through Python, please submit a PR!
After downloading the plaintext books, they need to be pre-processed in order to be a true replica of the Toronto BookCorpus dataset (sentence tokenized and one sentence per line). This can be accomplished as follows:
python src/preprocess_books.pyThis project builds upon bookcorpus.
Please read the Smashwords Terms of Service carefully. Furthermore, please use the code in this repository responsibly and adhere to any copyright (and related) laws. I am not responsible for any copyright / plagiarism / legal issues that may arise from using the code in this repository.
Replicate Toronto BookCorpus is open-source and licensed under GNU GPL, Version 3.