New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Allow user to choose which links to follow #3

Open

inglesp opened this issue Jun 9, 2016 · 1 comment

Owner

inglesp commented Jun 9, 2016

We currently follow all links, but in some cases this might not be appropriate

We should find a way to allow the user to configure which links to follow.

Owner Author

inglesp commented Jul 7, 2016

I've just added the follow_external_links option to crawl, which defaults to True. I think this addresses the likely main use case for this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment