Log2ram runs out of space

[anderl78]

Hello! Log2ram runs out of space after a few days. If i check the size of /var/log, it shows the expected size: ways smaller than the configured size of the ramdisk. Fir example /var/log is around 15 MB, the size of the ramdisk is 100 MB - but gets more and more full. Logrotate is already optimized. If i manually write the ramdisks content to disk, there is no difference in the usage of the ramdisk. The only way to fix it, is to reboot the system. System is a raspi 3b

[azlux]

Did you get an error message on the systemd service or the mount point is full ?
Second question, do you have rsync installed ?

Az

[anderl78]

Thank you for your answer! Rsync is installed and i get no errors on systemd service. The mount-point runs full. But if i look through it, a du gives me only a few mb used...

Just now (did areboot around 24 hours), i have a usage of 13MB of /var/log (du /var/log).
But a df shows me 29 mb used. And this 29 mb will increase day by day, till it reaches the maximum space available

[xtvdata]

Hi anderl78,
this is a a bit strange, however I doubt it is something related to log2ram.
log2ram just performs a synchronization of the contents in the RAM disk and in the persisted disk (e.g. SD card), thus it is not responsible of the the amount of data in /var/log, it just periodically ensures that what you have in RAM is also stored on the physical disk.

Typically if you have an ever growing /var/log it means that you have an issue either with:

• one or more application logging in debug mode (even tho this is not a reason for ever growing, since the logs should be rotated, but if the amount of debug is really high I might be that rotating and compressing log once per day is not enough) => if this is the case (see if the /var/log you have huge files) I'd suggest check if you really need debug level info
• rotation does not take into account part of the /var/log contents or rotation rules need to be fine-tuned (see previous point)

If the growing behavior is just inside a single 24hours the issue is most probably related to the first point, but if the growing goes on with the same trend after rotation it means it's a mix of the two.

However, I find strange that a reboot fixes the issue, as a clean reboot should not change the size of the logs... possibly there is something weird in /var/log. What happens if you just restart log2ram with the following command?

sudo systemctl restart log2ram.service

Does it produce any error?
All of the contents of /var/log are indeed synced with /var/hdd.log/?

Another strange thing is the fact that you have such a significant difference running du and df.
There could be a few reasons for this, including:

• df works in the same way with any user, but du ignores all the parts of the filesystem to which the current user is not authorized (ensure to run both commands as root)
• possibly (but I'm not sure) the difference is due to the fact you have a huge number of small files in some subdirectory of /var/log. du checks the size of each file ad sums it up dir by dir, however df checks the used space on a filesystem; every file is written to the filesystem in blocks and the minimum block size on ext4 is 1024, thus every file space occupancy is always rounded up to an integer number of kb... and if you have a large number of small files this could justify the difference.
• it might be (but this would be quite unusual) that you have some sparse files in /var/log. Sparse files' size is not directly correlated to actual disk usage, and in some cases my also lead to a strange behavior of rsync.

Just to give an idea on raspberries.
In all my regular RPIs, all of them with logging granularity and rotation optimized, /var/log contains between 200 and 400 files, the total size is about 5-6MB and difference between du and df is about 0.5%.
On the RPI I use to collect all logs from all devices (including routers, switches and other stuff), the total size of /var/log is about 50MB and the difference between du and df is about 5-6%. Obviously in this case the file count is much higher, including a number of very small files, thus the difference between du and df is justified.
To estimate the number of files in /var/log try to run:

sudo ls -lRA /var/log | wc -l

(I know it's an overestimation far from being precise, as it takes into account directories twice, there are empty lines, and bla bla..., but who cares, it gives you a very quick and dirty idea - just know that this file count is a bit overestimated...)

P.S.: having a limited amount of logs due to optimizations of logging granularity and rotation, might lead to the idea that using log2ram is an overkill.
Well it's not.
Indeed log2ram has a very strong point concerning SD preservation, but also it has a secondary benefit (which in my view is not secondary at all): it does significantly reduce general I/O operations on physical discs, and I/O is generally the most limited resource on a Raspberry Pi (with an SSD is much better, yet not stellar, due to USB3 limitations) and concurrency kills them when you are running on SDcard.

[anderl78]

Hey, sorry for my late answer. I had corona and was not able to do anything about this. Thank you very much for your answer!

Now i tried, as you suggested, a sudo systemctl restart log2ram.service. And after this, the usage seems to go back to normal... Dont know why!

[Phyks]

Hi,

It seems I had a similar issue. For some reason (buggy mail loop), my /var/log filled and log2ram was running out of space. I cleared up some space, resulting in ncdu indicating a total size of 100M for /var/log (300M configured in log2ram).

Still, this extra space was not reflected in df:

$ df -h
[...]
log2ram            300M    300M     0 100% /var/log

Looking at the systemd unit, it seems there is something missing and when the main process exits, the unit does not fall back to failed / stopped state nor systemd does automatically restart it:

$ sudo systemctl status log2ram
● log2ram.service - Log2Ram
     Loaded: loaded (/etc/systemd/system/log2ram.service; enabled; vendor preset: enabled)
     Active: active (exited) since Tue 2021-12-28 16:40:17 CET; 1 weeks 1 days ago
    Process: 24666 ExecReload=/usr/local/bin/log2ram write (code=exited, status=1/FAILURE)
   Main PID: 287 (code=exited, status=0/SUCCESS)
        CPU: 8.850s

A manual restart of log2ram did the trick though:

$ sudo systemctl restart log2ram
$ df -h
log2ram            300M    101M  200M  34% /var/log

Best

[ferrouswheel]

I also just experienced this. only 3M used on /var/log doing sudo du -hs but df shows it full and services fail due to being unable to write their logs...

restarting the log2ram service made df show the correct usage.

[azlux]

@ferrouswheel do you use rsync on log2ram ? Something weird about this issue.
And what's the output of :

du --apparent-size -s -h /var/log
du -s -h /var/log
du --apparent-size -s -h /var/hdd.log
du -s -h /var/hdd.log

[ferrouswheel]

My /etc/log2ram.conf has USE_RSYNC=false, but rsync is installed. Should I be using that?

Output of those commands is:

pi@raspberrypi:~ $ sudo du --apparent-size -s -h /var/log
2.8M	/var/log
pi@raspberrypi:~ $ sudo du -s -h /var/log
3.0M	/var/log
pi@raspberrypi:~ $ sudo du --apparent-size -s -h /var/hdd.log
2.7M	/var/hdd.log
pi@raspberrypi:~ $ sudo du -s -h /var/hdd.log
2.7M	/var/hdd.log

(I'll also run these commands if I experience the issue again)

[azlux]

The only thing I suppose to appear here is somehow you have sparse files ( amounts of space preallocated), and log2ram don't see it by default.
You can try to use rsync, but if it's because of some sparse file, this will change nothing.

[brainstain]

I'm also seeing this issue. I had rsync disabled, just enabled it. Restarting log2ram or the device cleared out the "ghost" space that was filling up the log2ram filesystem. Somewhat odd.

Let me know if there are any logs I can pull that would be helpful.

[vordenken]

I have the problem that log2ram runs out of space aswell. The big problem with this in my case is that if gets to 100% my raspberry pi loses its ethernet connection so I have to manually power it off/on to get it working again. I had this problem a while back and never got to the roots of this but now I installed log2ram a few weeks ago on a new pi and after a couple of days it started happening again.

Manually restarting the log2ram service doesn't do anything for me. I even increased the partition size via the log2ram config file to 80MB but it still got to 100% usage.
Running sudo du -hs /var/log/* | sort -h | tail -n 3 gives me this:

16M	/var/log/daemon.log.1
16M	/var/log/syslog.1
25M	/var/log/journal

I don't know why journal gets over 25MB because I set it to 20MB max like shown in the readme.
For now I have to remove log2ram from my pi because these outages are really impractical for me.
Let me know if you need any more information from me to maybe get this fixed.

Thanks! :)

[yurividal]

I have the problem that log2ram runs out of space aswell. The big problem with this in my case is that if gets to 100% my raspberry pi loses its ethernet connection so I have to manually power it off/on to get it working again. I had this problem a while back and never got to the roots of this but now I installed log2ram a few weeks ago on a new pi and after a couple of days it started happening again.

Manually restarting the log2ram service doesn't do anything for me. I even increased the partition size via the log2ram config file to 80MB but it still got to 100% usage. Running sudo du -hs /var/log/* | sort -h | tail -n 3 gives me this:

16M	/var/log/daemon.log.1
16M	/var/log/syslog.1
25M	/var/log/journal

I don't know why journal gets over 25MB because I set it to 20MB max like shown in the readme. For now I have to remove log2ram from my pi because these outages are really impractical for me. Let me know if you need any more information from me to maybe get this fixed.

Thanks! :)

I have the exact same problem. Did you ever find a solution to this?

[vordenken]

I have the problem that log2ram runs out of space aswell. The big problem with this in my case is that if gets to 100% my raspberry pi loses its ethernet connection so I have to manually power it off/on to get it working again. I had this problem a while back and never got to the roots of this but now I installed log2ram a few weeks ago on a new pi and after a couple of days it started happening again.

Manually restarting the log2ram service doesn't do anything for me. I even increased the partition size via the log2ram config file to 80MB but it still got to 100% usage. Running sudo du -hs /var/log/* | sort -h | tail -n 3 gives me this:

16M /var/log/daemon.log.1

16M /var/log/syslog.1

25M /var/log/journal

I don't know why journal gets over 25MB because I set it to 20MB max like shown in the readme. For now I have to remove log2ram from my pi because these outages are really impractical for me. Let me know if you need any more information from me to maybe get this fixed.

Thanks! :)

I have the exact same problem. Did you ever find a solution to this?

Unfortunately I did not...

[yurividal]

Seems like the logs are not getting properly recycled and journal logs get bigger than what they are supposed to.
My workaround was to add journal vacuum commands to run every hour to my crontab. I set vacuum anything older than 8hours

[trungklam]

I ran into this issue and managed to find the solution:

❯ df -h
Filesystem Size Used Avail Use% Mounted on
tmpfs 187M 15M 173M 8% /run
/dev/mmcblk0p2 59G 4.7G 52G 9% /
tmpfs 934M 0 934M 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
/dev/mmcblk0p1 253M 134M 120M 53% /boot/firmware
tmpfs 187M 4.0K 187M 1% /run/user/1000
log2ram 40M 40M 0 100% /var/log

After having a look on it ,the issue is the log of journald getting too big and it quickly fill up all the log2ram space. Solution ? Just clean it a bit:
sudo journalctl --vacuum-time=7d
sudo systemctl restart log2ram
The above cli will clean all the log older than 7 days. This issue can be prevented by set the max size for journald log and also increase the size for log2ram space. But imo juts clean journald logs regularly is good enough.

Result:

❯ df -h
Filesystem Size Used Avail Use% Mounted on
tmpfs 187M 15M 173M 8% /run
/dev/mmcblk0p2 59G 4.1G 52G 8% /
tmpfs 934M 0 934M 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
/dev/mmcblk0p1 253M 134M 120M 53% /boot/firmware
tmpfs 187M 4.0K 187M 1% /run/user/1000
log2ram 40M 8.8M 32M 22% /var/log

[yurividal]

I ran into this issue and managed to find the solution:

❯ df -h
Filesystem Size Used Avail Use% Mounted on
tmpfs 187M 15M 173M 8% /run
/dev/mmcblk0p2 59G 4.7G 52G 9% /
tmpfs 934M 0 934M 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
/dev/mmcblk0p1 253M 134M 120M 53% /boot/firmware
tmpfs 187M 4.0K 187M 1% /run/user/1000
log2ram 40M 40M 0 100% /var/log

After having a look on it ,the issue is the log of journald getting too big and it quickly fill up all the log2ram space. Solution ? Just clean it a bit: sudo journalctl --vacuum-time=7d sudo systemctl restart log2ram The above cli will clean all the log older than 7 days. This issue can be prevented by set the max size for journald log and also increase the size for log2ram space. But imo juts clean journald logs regularly is good enough.

Result:

❯ df -h
Filesystem Size Used Avail Use% Mounted on
tmpfs 187M 15M 173M 8% /run
/dev/mmcblk0p2 59G 4.1G 52G 8% /
tmpfs 934M 0 934M 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
/dev/mmcblk0p1 253M 134M 120M 53% /boot/firmware
tmpfs 187M 4.0K 187M 1% /run/user/1000
log2ram 40M 8.8M 32M 22% /var/log

This is not a fix for the issue, but rather a workaround. The root of the problem is that journal logs are getting bigger than what they should (they get bigger than what is specified on the journal config file), so, somehow the journal logs are not being properly rotated.

[balupton]

I fixed this error by doing sudo vim /etc/systemd/journald.conf with:

# https://www.freedesktop.org/software/systemd/man/journald.conf.html
SystemMaxUse=20M
RuntimeMaxUse=20M
MaxRetentionSec=1day
MaxFileSec=1day

Actually, a few days later and it still ran out of log space.

[camelcamro]

df -h
log2ram 50M 50M 0M 100% /var/log

hmm, for now, I'm doing it via a simple daily cronjob

#cleanup workaround for lo2ram
11 11 * * * sudo journalctl --vacuum-time=1d;sudo systemctl restart log2ram

still have no lcue what is really the problem, but a restart fix the issue for me.

and after: sudo systemctl restart log2ram
df -h
log2ram 50M 2,1M 48M 5% /var/log

[coaxk]

also seeing the same issue. jumping on the thread for updates

[thickconfusion]

Pi Zero W here. Running Pi Hole and log2ram. I have been fighting this a while, thinking more and more it's a Systemd journal bloating issue tangled with log2ram. Anyway, I found this as well. home-assistant/operating-system#2226

[ollie-campbell]

+1 here:

df -h
Filesystem      Size  Used Avail Use% Mounted on
....
log2ram          40M   40M     0 100% /var/log

sudo journalctl --vacuum-files=1
Vacuuming done, freed 0B of archived journals from /run/log/journal/11316449f0af4dd893e46267e666acbe.

sudo systemctl status log2ram.service

sudo systemctl status log2ram.service
● log2ram.service - Log2Ram
   Loaded: loaded (/etc/systemd/system/log2ram.service; enabled; vendor preset: enabled)
   Active: active (exited) since Wed 2023-06-21 09:38:47 BST; 4s ago
  Process: 734 ExecStart=/usr/local/bin/log2ram start (code=exited, status=0/SUCCESS)
 Main PID: 734 (code=exited, status=0/SUCCESS)

Jun 21 09:38:47 HOSTNAME log2ram[734]: apt/term.log.3.gz
Jun 21 09:38:47 HOSTNAME log2ram[734]: apt/term.log.4.gz
Jun 21 09:38:47 HOSTNAME log2ram[734]: apt/term.log.5.gz
Jun 21 09:38:47 HOSTNAME log2ram[734]: apt/term.log.6.gz
Jun 21 09:38:47 HOSTNAME log2ram[734]: apt/term.log.7.gz
Jun 21 09:38:47 HOSTNAME log2ram[734]: apt/term.log.8.gz
Jun 21 09:38:47 HOSTNAME log2ram[734]: private/
Jun 21 09:38:47 HOSTNAME log2ram[734]: sent 41,703,080 bytes  received 1,626 bytes  83,409,412.00 bytes/sec
Jun 21 09:38:47 HOSTNAME log2ram[734]: total size is 41,687,168  speedup is 1.00
Jun 21 09:38:47 HOSTNAME systemd[1]: Started Log2Ram.

df -h
Filesystem      Size  Used Avail Use% Mounted on
...
log2ram          40M   40M   52K 100% /var/log

journalctl -e

Jun 21 09:40:26 HOSTNAME rsyslogd[362]: action 'action-1-builtin:omfile' (module 'builtin:omfile') message lost, could not be processed. Check for additional error messages before this one. [v8.1901.0 try https://www.rsyslog.com/e/2027 ]
Jun 21 09:40:26 HOSTNAME rsyslogd[362]: file '8' write error: No space left on device [v8.1901.0 try https://www.rsyslog.com/e/2027 ]
Jun 21 09:40:26 HOSTNAME rsyslogd[362]: action 'action-1-builtin:omfile' (module 'builtin:omfile') message lost, could not be processed. Check for additional error messages before this one. [v8.1901.0 try https://www.rsyslog.com/e/2027 ]
Jun 21 09:40:26 HOSTNAME rsyslogd[362]: file '8' write error: No space left on device [v8.1901.0 try https://www.rsyslog.com/e/2027 ]
Jun 21 09:40:26 HOSTNAME rsyslogd[362]: action 'action-1-builtin:omfile' (module 'builtin:omfile') message lost, could not be processed. Check for additional error messages before this one. [v8.1901.0 try https://www.rsyslog.com/e/2027 ]
Jun 21 09:40:26 HOSTNAME rsyslogd[362]: file '8' write error: No space left on device [v8.1901.0 try https://www.rsyslog.com/e/2027 ]
Jun 21 09:40:26 HOSTNAME rsyslogd[362]: action 'action-1-builtin:omfile' (module 'builtin:omfile') message lost, could not be processed. Check for additional error messages before this one. [v8.1901.0 try https://www.rsyslog.com/e/2027 ]
Jun 21 09:40:26 HOSTNAME rsyslogd[362]: file '8' write error: No space left on device [v8.1901.0 try https://www.rsyslog.com/e/2027 ]