Add a random splay to repology API requests

[AMDmi3]

Describe the proposed change(s).

Below is a graph of requests rate to Repology API. Hourly peaks are caused by renovate bot. Though it's not critical, it would be better if the load was evenly distributed, improving high-percentile latencies and capacity.

FYI, some weeks ago I've rewritten API endpoint in Rust, which increased capacity tenfold and allowed to remove 1RPS rate limit for all renovate clients.

[rarkins]

Converted to discussion. First of all, thanks again for your continued support of Renovate users.

Is there any chance you could approximate the same type of graph for the Mend-hosted app alone?

I don't know how much of this traffic is by self-hosted users vs the Mend app. For the Mend app, I would have expected traffic to be evenly spread, because of time the lookups and cache expiries should approach randomness.

For self-hosted users, they probably set cron jobs to run on their repos every hour, or at the start of every x hours, and the lookups you see probably peak as a result of that, or partially as a result of that.

[AMDmi3]

Is there any chance you could approximate the same type of graph for the Mend-hosted app alone?

No. How can I tell it from self-hosted instances?

For self-hosted users, they probably set cron jobs to run on their repos every hour, or at the start of every x hours, and the lookups you see probably peak as a result of that, or partially as a result of that.

Makes sense. Is there a chance to add a dedicated method to call renovate bot from cron and mention it in the docs? I don't think that forcing it in any way is needed at this point as the peaks already are not too sharp, but it would help if at least some of new installations use that and randomize poll timing.

[AMDmi3]

That I understand. What I suggest is that there could be a cli flag which introduces random delay of 0÷3600 seconds, and it was recommended in the documentation to enable it when users run cli from cron. That way at least some newer self-hosted installations would enable it and distribute load. And existing users could be recommended to enable it if their instance does a lot of requests at the first minutes of an hour.

We could go further and make this flag required if cli is ran without tty, but as mentioned I don't feel like it makes sense right now. However, the existence of such a flag would be future-proof.

An example of such flag could be seen in freebsd-update utility (man) which

• runs normally (freebsd-update fetch) from tty
• refuses to run without tty, requiring either cron mode (freebsd-update cron, which is the same as freebsd-update fetch with a random delay), or --not-running-from-cron flag which overrides this behavior.

[rarkins]

Unfortunately I think such an idea would cause large problems. Controlling when renovate runs is a legitimate requirement. Eg some people may run it hourly and it takes ~50 minutes. Others may be paying for minutes and so renovate randomly delaying 30 minutes while idling may double their costs

[AMDmi3]

This mechanism is explicit, so the behavior shouldn't be a surprise for anybody, and non-mandatory.

[rarkins]

Sorry, I missed that you were suggesting it as opt-in. Unfortunately though my gut instinct is that almost none would enable it

[AMDmi3]

Well that's the possibility, that's why I don't insist. If there isn't much enthusiasm, let's close this.