Clarify what kind of cache in Renovate are used for and what is recommended for on-prem run from CI

[szpak]

Tell us more.

Renovate seems to use various cache types for various purpose. There is Redis/Sqlite, S3, local file cached and also cache used by the build systems (such as npm).

I remember, I was even improving the documentation years ago, but it would be good to create some page/space in the documentation to summarize it and show recommendations how to do it best.

From the documentation and a previous discussion, I know that:

S3 is a repo cache (Eg extracted dependencies) and not a lookup cache. Package caches are normally stored locally in disk by default, or you can specify a Redis URL to store them remotely

However, having both S3 and Redis configured, I still observe (v39.97.0 on-prem with GitLab CI), a few GBs in $RENOVATE_CACHE_DIR/others/npm and ~200MB extra in addition to npm (in $RENOVATE_CACHE_DIR).

The others/npm directory size increases dramatically (20GB+ after a few months) which affects time required to compress and upload stuff to GitLab CI cache. Someone suggested npm --cache npm/ cache verify which purges large part of it, however, at the cost of performance hit. Recently, I saw an idea to switch to sqlite by default, but I already use Redis 🤔 .

In the end, I would like to know, what are the benefits of caching file system cache (both Renovate and/or npm) on an CI executor, in addition to using S3 and Redis/Sqlite (because when I remove the CI cache completely, the execution time increases greatly in the first run, but maybe npm itself could be only removed? 🤔 )?

[rarkins]

I think you're right at a "meta" level that we should have a single page describing caching/performance improvements.

Answering specifically on one point, the "other/" directory is for each package manager's cache (npm, gradle, poetry, etc). In theory it does grow indefinitely over time. It's also hard to judge at which point it's performance improvement (e.g. not redownloading common npm or python packages for every second job) versus a burden due to the storage.