Add rate limit option

[ocervell]

Add a rate limit option for controlling crawling speed (-rl, -rate-limit). You maybe can use https://github.com/projectdiscovery/ratelimit for this.

[edoardottt]

Hi @ocervell , there is this option available in cariddi, it's not a rate limiter..but similar:

  -d int
     Delay between a page crawled and another.

Seems this is in colly the way to provide rate limiting (https://go-colly.org/docs/examples/rate_limit/).

[ocervell]

Argh ... this is clearly not enough since it depends on the number of threads as well... adding a delay between requests is okay but not as precise as setting a global rate limit. I will investigate to see what we can do.