New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Disallow: // shouldn't behave the same as Disallow: / #113

Open

wumpus opened this issue Aug 26, 2019 · 0 comments

Contributor

wumpus commented Aug 26, 2019

https://www.ibm.com/robots.txt has DIsallow:// and it appears that they expect it to not disallow everything. reppy treats it the same as Disallow:/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment