No explicit header set for the DOWNLOAD_SESSION

[rtibbles]

Observed behavior

The DOWNLOAD_SESSION that is used to download resources sets no explicit header - this proves to be an issue, for example, when downloading from wikimedia sites, because of their User Agent policy: https://meta.wikimedia.org/wiki/User-Agent_policy

Expected behavior

Ideally, we would follow the kind of User-Agent that the wikimedia policy spells out - we already retrieve the email for the user whose API token we are running with from Studio, so we should reuse this to set the header.

With that in place, we would then do the following for the User Agent:

f"Ricecooker/{ricecooker.__version__} bot ({user_email})"

User-facing consequences

Attempts to scrape without setting these headers may be treated as malicious.

Steps to reproduce

Attempt to download any file from wikimedia

Context

Ricecooker develop branch

[nikkuAg]

Hey, is this issue still open? I would like to work on this

[rtibbles]

Absolutely @nikkuAg - I will assign you, thanks for volunteering!

[MisRob]

Hi @nikkuAg, are you still planning to work on this?

[ghost]

Hey, is this issue still open? I would like to work on this

[AllanOXDi]

Hi @manic012 I have assigned it to you. Thanks for contributing

[adoo100]

Hi @rtibbles if this issue is still unresolved I would love to be a part and work in it :)

[akolson]

Hi @adoo100! Thank you! I have assigned it to you! Please feel free to reach out to @rtibbles incase it is not clear what needs to be done while resolving the issue.