Curator not deleting closed indices, disk space maintained

[Jaap79]

Hello,

Question... I noticed that 1 of our 4 search nodes has a whole bunch of old closed indices (July this year). The disk space has reached the limit I set up in the global.sls file on the Manager. The disk space is, however, maintained at that limit so something must be functioning correctly..

However, I would expect that the oldest closed indices would be deleted. I also set the limit in global.sls to delete indices older than 90 days, which would mean that July would be deleted by now.

Any idea what I'm missing here? As stated, the disk space is not decreasing so somewhere a limit is being maintained...

[dougburks]

Are you using cross cluster search or actual Elastic clustering?

Have you reviewed all of the documentation here?
https://docs.securityonion.net/en/2.3/elasticsearch.html#deleting-indices

[Jaap79]

Hello Doug,

we’re using cross cluster search. If I read the info correctly, then so-(…)-closed-delete should delete closed indices based on the 90-day deletion limit that we’ve set up. However, I’m still seeing the July closed indices even today (November 16th).

[dougburks]

Cross cluster search deployments by default should delete based on the log_size_limit value. From https://docs.securityonion.net/en/2.3/elasticsearch.html#deleting-indices:

For standalone deployments and distributed deployments using cross cluster search, Elasticsearch indices are deleted based on the log_size_limit value in the minion pillar. If your open indices are using more than log_size_limit gigabytes, then Curator will delete old open indices until disk space is back under log_size_limit. If your total Elastic disk usage (both open and closed indices) is above log_size_limit, then so-curator-closed-delete will delete old closed indices until disk space is back under log_size_limit. so-curator-closed-delete does not use Curator because Curator cannot calculate disk space used by closed indices. For more information, see https://www.elastic.co/guide/en/elasticsearch/client/curator/current/filtertype_space.html.

[Jaap79]

Hi Doug,

It should - but how can I check if it actually does? We're still seeing closed indices from back in July while Curator should delete closed indices after 90 days. So-curator-closed-delete does not remove them either ...

I'm a bit puzzled because it would seem my disk space is full enough (around 100% of the log file limit reached, or very close to it) to warrant removing the closed indices. And even so, it should still remove them after the 90 days regardless of disk space, correct?