Persistent cache frequently runs out of inodes

[Nicolas-M]

When running Renovate CE 6.3.0 with the cachePersistence, our PVC always runs out of inodes over time. Once it reaches the max, Renovate stops working altogether. I have included a screenshot of the PVC usage below for reference. It does not seem like Renovate deletes any elements from its cache.

Our PVC is now 70 GB, which is well beyond the chart's 1GB default. We are running Renovate in AKS, using an azure-disk based storage class (ReadWriteOnce).

Grafana dashboard illustrating the PVC's space & inode usage sitting at 44.6 % & 100 % respectively. The usages climbed linearly.
Note: Although the screenshot is a bit older, the issue is still happening under Renovate-CE 6.3.0.

May be related to #313.

Thanks in advance for the help!

[rarkins]

Can you confirm which folders consume this disk space, and if they're all mapped to the host? Further, renovate should log debug information about the cache usage each run, although I'm on mobile and don't have the exact string handy right now

[Nicolas-M]

Thanks for the quick response! I had to delete the PVC to get my deployment running again, but I will report back once it starts to fill up.

[Nicolas-M]

As far as I can tell, the PVC gets correctly mounted to the host on /tmp/renovate. Please find a breakdown of the disk space usage of a few directories (where the percentage represent an approximate fraction of the current space used, not the total capacity):

• 75%: /tmp/renovate/cache/renovate/renovate-cache-v1/content-v2
• 13%: /tmp/renovate/cache/others
• 6%: /tmp/renovate/cache/renovate/renovate-cache-v1/index-v5
• 5%: /tmp/renovate/cache/containerbase

Looking at the logs when Renovate initializes upon pod creation, nothing stands out:

DEBUG: Using baseDir: /tmp/renovate
DEBUG: Using cacheDir: /tmp/renovate/cache
DEBUG: Using containerbaseDir: /tmp/renovate/cache/containerbase
DEBUG: Initializing Renovate internal cache into /tmp/renovate/cache/renovate/renovate-cache-v1

Here are the statistics for one of the jobs:

DEBUG: Package cache statistics (repository=myOrg/myRepository)
       "get": {"count": 8, "avgMs": 4, "medianMs": 2, "maxMs": 11},
       "set": {"count": 0}

Renovate seems to cleanup after every job as expected:

DEBUG: Checking file package cache for expired items
DEBUG: Deleted 4 of 8931 file cached entries in 15165ms
DEBUG: Renovate exiting
DEBUG: Renovate job finished
       "status": "success"

Unlikely related, but there is an undefined value in the branch summary output:

DEBUG: Branch summary (repository=myOrg/myRepository)
       "cacheModified": undefined,

Please let me know what other information would be useful in debugging this. Thanks again!

[rarkins]

Is that count of 8931 file cached entries align with your visibility of files/inodes?

BTW you might find it more efficient for multiple reasons and maybe fix this problem if you use a Redis cache instead of file cache.

[Nicolas-M]

Not at all. I ran the following command to count all the files recursively through directories in /tmp/renovate/cache/renovate/renovate-cache-v1/content-v2:

find . -maxdepth 1 -type d | while read -r dir
do printf "%s:\t" "$dir"; find "$dir" -type f | wc -l; done

The output of the command suggests I have 1,443,533 files, which is much larger than the 8693 file cached entries Renovate states in the logs.

I will start investigating deploying a Redis instance, though I am still curious as to what is going on with the file system cache.