Slack Rate Limit Prevention

[ananyo141]

We have to proactively think about scenarios in the future where the application can reach the threshold of Slack's Fair Usage Policy (see the docs on rate limiting). As per testing, it is around 150 on local non-premium workspace. But there would be exponential backoff (less - around 90 on second time without cooloff once rate limit is hit)

Slack recommends a 1 request per second policy for best results. So the obvious solution can be to throttle requests to hit the API once every second, but there might be other ways to handle this. One caveat: the default frappe long worker has a hard limit of 25 minutes to run, so according to this 1500 requests is the limit.

After discussion with @niraj2477, we can have a combination of Job Batching + Cooloff + Failure Correction (task retry after cooldown period) to tackle the rate limiting, to dynamically handle and retry api calls upon rate limit.

NOTE: We are using throttling as of now (1 request/second)

[dhsathiya]

@ananyo141 Please check other open source Slack apps if they are using some other API endpoints or methods for this.