Free disk space

[y0d4a]

Hi,

I've been struggling to get Malcolm to run reliably on its own.
I keep running out of free space, primarily due to uploaded PCAP files.

In the configuration menu, I've set thresholds to monitor and manage free space when it drops below 80%, but it doesn't seem to be working.
The partition keeps filling up, and I can’t get Malcolm to run for more than two days before running into issues.

Any advice or suggestions would be greatly appreciated!

[mmguero]

Managing disk usage can be tricky because there are a lot of different components and different types of data in Malcolm that are all sort of "doing their own thing" when it comes to pruning old data to make sure there's room for the new.

Let's double check the settings you've got that have to do with this and then if we can identify which one(s) need to change you can either directly change them in the .env files or I can point you at the right question in the configuration wizard.

In your Malcolm directory, run this command to get the values for the variables controlling the storage management for the various aspects of Malcolm's data, and you'll see something like (although probably not exactly like) this example output:

$ grep -P "^\w*(PRUNE|CLEAN|FREE|MANAGE_PCAP)\w*\s*=" config/*.env
config/arkime.env:MANAGE_PCAP_FILES=false
config/arkime.env:ARKIME_FREESPACEG=10%
config/dashboards-helper.env:OPENSEARCH_INDEX_SIZE_PRUNE_LIMIT=0
config/dashboards-helper.env:OPENSEARCH_INDEX_SIZE_PRUNE_NAME_SORT=false
config/filebeat.env:FILEBEAT_CLEAN_INACTIVE=180m
config/filebeat.env:FILEBEAT_CLEAN_REMOVED=true
config/filebeat.env:LOG_CLEANUP_MINUTES=15
config/filebeat.env:ZIP_CLEANUP_MINUTES=20
config/zeek.env:EXTRACTED_FILE_PRUNE_THRESHOLD_MAX_SIZE=1TB
config/zeek.env:EXTRACTED_FILE_PRUNE_THRESHOLD_TOTAL_DISK_USAGE_PERCENT=0
config/zeek.env:EXTRACTED_FILE_PRUNE_INTERVAL_SECONDS=60

If you can post your results for that command we can go from there.

[y0d4a]

Yeah, i suppose that is the issue.
I tried couple of options, this is last conf. which is running:

config/arkime.env:MANAGE_PCAP_FILES=false
config/arkime.env:ARKIME_FREESPACEG=10%
config/dashboards-helper.env:OPENSEARCH_INDEX_SIZE_PRUNE_LIMIT=1.5TB
config/dashboards-helper.env:OPENSEARCH_INDEX_SIZE_PRUNE_NAME_SORT=false
config/filebeat.env:FILEBEAT_CLEAN_INACTIVE=180m
config/filebeat.env:FILEBEAT_CLEAN_REMOVED=true
config/filebeat.env:LOG_CLEANUP_MINUTES=360
config/filebeat.env:ZIP_CLEANUP_MINUTES=720
config/zeek.env:EXTRACTED_FILE_PRUNE_THRESHOLD_MAX_SIZE=1.5TB
config/zeek.env:EXTRACTED_FILE_PRUNE_THRESHOLD_TOTAL_DISK_USAGE_PERCENT=0
config/zeek.env:EXTRACTED_FILE_PRUNE_INTERVAL_SECONDS=300

I have 1.2T partition for malcolm folder (where is all data, including pcaps)

[mmguero]

Okay, if everything is on one partition we need to make a few adjustments. These can all be made in the configuration wizard, but I'll show you the values you'll want to change in the config files themselves first to get it working:

• MANAGE_PCAP_FILES needs to be true to allow the pruning process to delete older PCAP files to make room for newer ones
• ARKIME_FREESPACEG needs to be set to a value that means "Start deleting PCAP files when the free space drops below this amount;" in other words, setting it to 10% means "the oldest PCAP files will be deleted when my free space is below 10% of the drive" (or when free space is < ~120 gigabytes in your case, with a 1.2 TB partition)
• OPENSEARCH_INDEX_SIZE_PRUNE_LIMIT needs to be set to some lower value as well; this is the maximum size we're going to allow the opensearch indices to be before we start deleting old ones. If your partition is 1.2TB, I'd set this value to something like 750GB.
• EXTRACTED_FILE_PRUNE_THRESHOLD_MAX_SIZE is a threshold for extracted (carved) files. This probably doesn't need to be as large, you could set it to maybe 100GB.

So with those changes, extracted files will not consume more than 100GB, opensearch indices will not consume more than 700G, and PCAP deletion will occur when the free space drops below 10%, meaning you're going to end up with like 280G left for PCAPs.

To make those changes, stop malcolm, edit the .env files, and then start malcolm again. Or, if you're going through the configuration wizard these questions are all under Should Malcolm delete the oldest database indices and capture artifacts based on available storage?

[y0d4a]

Thank you for your time and assistance!
It seems the issue was related to the "MANAGE_PCAP_FILES" setting, which was set to false for some reason . This caused problems with the PCAP files consuming all available free space.
I followed your suggestion, restarted the Malcolm, and am now monitoring the situation.

If everything remains stable over the next 2–3 days, I’ll close the thread.

Wishing you all the best, cheers!

[mmguero]

Great! Make sure you set those other variables as well: even though you've address the issue with the PCAP files, you could also fill the disk up with the opensearch indexes or extracted files if OPENSEARCH_INDEX_SIZE_PRUNE_LIMIT and EXTRACTED_FILE_PRUNE_THRESHOLD_MAX_SIZE aren't set as well.

[y0d4a]

they are:
EXTRACTED_FILE_PRUNE_THRESHOLD_MAX_SIZE=150GB
OPENSEARCH_INDEX_SIZE_PRUNE_LIMIT=750GB

[y0d4a]

Hi, it happened again—I ran out of space... 😞
There's still 22GB left for now, but it won't last long.

$ du -h --max-depth=1 pcap

4.0K pcap/processed
1.2T pcap/upload
4.0K pcap/arkime-live
1.2T pcap

[y0d4a]

i done restart of malcolm, maybe that is the problem? i will now stop it all and start again..

[mmguero]

As pcaps are uploaded they should be moved from the pcap/upload directory to the pcap/processed directory; the code to do the cleanup of PCAPs only happens after they've sort of entered the system Arkime has been made aware of them. So for some reason them not being moved to the processed directory is a bit of a concern. If you clear the whole system out (./scripts/wipe, but be cautious as this will wipe out all of your existing processed data) or just remove some of the files from ./pcap/upload and start Malcolm up again, you should be able to try to get things moving after a startup by doing a touch pcap/upload/* which will hopefully signal to malcolm that those files are there and ready to be processed. Newly uploaded files shouldn't need that, but if you have ones left around in there after a restart that might help.

[y0d4a]

so, the flow looks like: suricata/zeek -> pcap upload -> arkime -> pcap processed ?
Yesterday, I stopped all containers and ran it again. I am now waiting to see the behavior of free space. I will let you know.

[mmguero]

Not exactly, if we're just talking about uploaded PCAP (not live capture) then it's pcap/upload -> pcap/processed -> PCAP goes three ways to be processed by zeek/suricata/arkime, kind of like this:

[y0d4a]

hm, look logs from pcap-monitor:

2024-11-22 10:51:38,585 WARN exited: watch-upload (exit status 1; not expected)
2024-11-22 10:51:39,587 INFO spawned: 'pcap-publisher' with pid 929
2024-11-22 10:51:39,588 INFO gave up: watch-upload entered FATAL state, too many start retries too quickly
Traceback (most recent call last):
File "/usr/local/bin/pcap_watcher.py", line 39, in
import watch_common
File "/usr/local/bin/watch_common.py", line 23, in
from watchdog.utils import WatchdogShutdown
ImportError: cannot import name 'WatchdogShutdown' from 'watchdog.utils' (/usr/local/lib/python3.11/dist-packages/watchdog/utils/init.py)

maybe there is the problem?

[mmguero]

Likely that second one is just left over from a previous update (and would go away if you ran docker system prune to clean up the dangling images).

However, if the one that is actually running is v24.07.0 (you could double-check this by running docker ps | grep malcolm-pcap-monitor while Malcolm is running, or grep image: docker-compose.yml) then I think the error about the watchdog library has definitely been fixed since July when that release was made. Looking at the releases page and searching for watchdog you'll see I've updated that library a few times since then. If that library weren't working then yeah, I could totally see those files not getting moved from upload to processed correctly, although I don't remember encountering that myself when I was testing v24.07.0.

We don't have a super-smooth upgrade process yet, but if I were you I'd try to get Malcolm updated to the latest release or do a clean install from the latest release, then go through the configuration steps again and see how it runs after that.

[y0d4a]

Again, get out of disk space :(

I cleaned all unused images etc.
Removed pcaps, restart malcolm.
all is running latest version:

in compose file there is tags for latest version:

grep -P "^\w*(PRUNE|CLEAN|FREE|MANAGE_PCAP)\w*\s*=" config/*.env

config/arkime.env:MANAGE_PCAP_FILES=true
config/arkime.env:ARKIME_FREESPACEG=20%
config/dashboards-helper.env:OPENSEARCH_INDEX_SIZE_PRUNE_LIMIT=750GB
config/dashboards-helper.env:OPENSEARCH_INDEX_SIZE_PRUNE_NAME_SORT=false
config/filebeat.env:FILEBEAT_CLEAN_INACTIVE=180m
config/filebeat.env:FILEBEAT_CLEAN_REMOVED=true
config/filebeat.env:LOG_CLEANUP_MINUTES=360
config/filebeat.env:ZIP_CLEANUP_MINUTES=720
config/zeek.env:EXTRACTED_FILE_PRUNE_THRESHOLD_MAX_SIZE=150GB
config/zeek.env:EXTRACTED_FILE_PRUNE_THRESHOLD_TOTAL_DISK_USAGE_PERCENT=0
config/zeek.env:EXTRACTED_FILE_PRUNE_INTERVAL_SECONDS=300

$ docker logs ids-pcap-monitor-1:

[y0d4a]

aha. now i see latest version is: 24.11, i will replace images and try with the
m

[mmguero]

Yeah, 24.07.0 is still the one from 4 months ago.

I'm going to be on vacation until December 2nd, but I will follow up here when I return.

[y0d4a]

wish you great time!