Blacklist requests that are duplicates of existing resources or bound to fail

[Popolechien]

Following openzim/zimit#113, we should think about implementing a fairly easily editable list (hosted on drive.kiwix.org?) of blacklisted sites that can not be requested on zimit, e.g.

• kiwix.org subdomains (download and library);
• very large corporate websites (e.g. Facebook, Twitter, Reddit, Youtube, etc.)
• websites that have been scraped in the past and failed.

It's probably the matter of a separate ticket, but requests for websites we already have a scraper for (wikipedia, stackoverflow, etc.) should also be soft blocked and the user offered a direct link to the zim file.

[rgaudin]

Can you move your comment to #25 and close this? This is the scraper's repo.

[Popolechien]

@rgaudin Moved it but I'd keep it open as this ticket is a little bit different.

[rgaudin]

This one's better ; closing the other one but the problem raised there remains: where do we point to for stuff that we know exists?

[Popolechien]

Is your question "in case there are several versions of the same zim" (e.g., Wikipedia mini/nopic/maxi)?

The basic assumption here is that zimit provides a copy of the real thing, so we should send them the maxi zim file.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be now be reviewed manually. Thank you for your contributions.

[kelson42]

See also #33

[benoit74]

I've started to document blacklist I encounter during maintenance tasks at https://docs.google.com/spreadsheets/d/1mBjWT0hLmeg6EqT4nNEfCzLU8hGSzYs4IgbWDInhPqA/edit?gid=0#gid=0

[rgaudin]

Should we add a link to it to the routine? Should we count them in some way?

[benoit74]

Added the link to the routine, indeed it would help to have the link at hand.
About counting them, what would be the added value? (nothing against it, but I don't get why we would like to do this, and it seems to be cumbersome / complex to implement)

[rgaudin]

That's why I asked. The value would be to distinguish the importance between them ; should the eventual actions have to be prioritized

[Popolechien]

I've added two more to the list.
Which routine are we talking about?

[benoit74]

The weekly infra routine (manual checks we do every week to ensure infra is up and running)