[BUG] High memory consumption

[Mavtti]

Describe the bug

We are using aws opensearch and since upgrading from 2.11 to 2.15, the memory usage increased and it slows down the whole cluser.
The biggest impact is on write operations which started to timeout after the upgrade.

I've seen this issue but I thought that the issue was supposed to be fixed in 2.15.

Upgrade happened on the 10th.

Max memory utilization more than often reaches 100% since then.

There is also an increase in Java Heap memory

Any idea ? Thx !

Related component

Other

To Reproduce

1. Upgrade to 2.15

Expected behavior

No issue when inserting data and lower memory consumption

Additional Details

No modification on our side on memory settings

[reta]

@Mavtti by any chance, could you capture class histogram to understand where the heap consumption is coming from? Thank you.

[Mavtti]

@reta Sorry I never had to do it before, would you mind explaining me how to do it ?

Btw, to mitigate the issue, we changed our instances type from t3 to m7g. It gave us more leeway.
But at the same time, we also saw an issue with an ever increasing memory just like this case.

[reta]

@reta Sorry I never had to do it before, would you mind explaining me how to do it ?

Thanks @Mavtti , there are multiple ways to do that (we just need to do this on any node that consumes more heap):

• using jcmd (https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr006.html)

jcmd <pid> GC.class_histogram

• using jmap (https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr014.html)

jmap  -histo <pid>

The Jackson issue should be fixed in 2.15.0 (and above)

[Mavtti]

Hey since I'm on managed aws opensearch, there is no way for me to run these commands.

And I saw that the jackson issue was supposed to be fixed but we actually are on 2.15 (OpenSearch_2_15_R20240904 to be exact) and still see this issue.

[reta]

Hey since I'm on managed aws opensearch, there is no way for me to run these commands.

Got it :(

And I saw that the jackson issue was supposed to be fixed but we actually are on 2.15 (OpenSearch_2_15_R20240904 to be exact) and still see this issue.

Correct, the heap consumption might be caused by another issue

[Mavtti]

So in the end, I can't give you more insight and there's not much you can do without it, correct ?

[reta]

So in the end, I can't give you more insight and there's not much you can do without it, correct ?

I think the best option we have is to engage with AWS support to ask for these details

[Mavtti]

Ok thx @reta for the answers !

[dblock]

Let's close it here. Do post what you find through working with AWS support please for the next person.

[Catch All Triage - 1, 2, 3, 4]