Usage:
python main.py [-j n]
where n is the number of threads you want to run in parallel.
Crawling custom URLs is no longer supported for security and quality reasons. To submit a domain to crawl, please visit https://mwmbl.org/app/domain-submissions/new
Clone this repo, install poetry if necessary, cd into the repo and type
poetry install
poetry shell
then the main.py command as documented above.