[BUG] Huge JVM Heap Usage

[artemus717]

Describe the bug

We are suffuring a huge jvm heap usage. Seemed like many fasterxml class in jvm and can not be GC normally.

Related component

Search

To Reproduce

Run OpenSearch for a while

Expected behavior

GC normally.

Additional Details

OpenSearch 2.14.0

128G Memory 64G JVM
/usr/share/opensearch/jdk/bin/java -Xshare:auto -Dopensearch.networkaddress.cache.ttl=60 -Dopensearch.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -XX:+ShowCodeDetailsInExceptionMessages -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dio.netty.allocator.numDirectArenas=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.security.manager=allow -Djav.locale.providers=SPI,COMPAT -Xms64g -Xmx64g -XX:+UseG1GC -XX:G1ReservePercent=25 -XX:InitiatingHeapOccupancyPercent=30 -Djava.io.tmpdir=/tmp/opensearch-15234535429733417150 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/lib/opensearch -XX:ErrorFile=/var/log/opensearch/hs_err_pid%p.log -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/opensearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m -Djava.security.manager=allow -Djava.util.concurrent.ForkJoinPool.common.threadFactory=org.opensearch.secure_sm.SecuredForkJoinWorkerThreadFactory -Dclk.tck=100 -Djdk.attach.allowAttachSelf=true -Djava.security.policy=file:///etc/opensearch/opensearch-performance-analyzer/opensearch_security.policy --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED -XX:MaxDirectMemorySize=34359738368 -Dopensearch.path.home=/usr/share/opensearch -Dopensearch.path.conf=/etc/opensearch -Dopensearch.distribution.type=rpm -Dopensearch.bundled_jdk=true -cp /usr/share/opensearch/lib/* org.opensearch.bootstrap.OpenSearch -p /var/run/opensearch/opensearch.pid --quiet

[artemus717]

FasterXML/jackson-core#1271

FasterXML/jackson-core#1266

[andrross]

OpenSearch 2.14 was released using Jackson 2.17.0. It is very possible that the issue here is related to the Jackson issue(s) posted above, which were fixed in Jackson 2.17.1.

We've already incorporated the new Jackson version (#13817) for the next release. We should probably consider patching the 2.14 release if this is as serious as it looks.

[reta]

We've already incorporated the new Jackson version (#13817) for the next release. We should probably consider patching the 2.14 release if this is as serious as it looks.

I will backport the 2.17.1 bump to 2.14.x and we could decide if patch release is necessary taking into account that 2.15.0 is pretty close

[peterzhuamazon]

Hi @reta @andrross ,

With 2.15.0 having the freeze date on 2024/06/07 (this Friday), do you guys think it would be the best to release 2.15.0 directly? Any arguments to have a 2.14.1 release now as it could take equally or just a bit less amount of time.

Also syncing @elfisher @Pallavi-AWS @getsaurabh02 into the conversation as we need to decide on this quick. Thanks!

[Pallavi-AWS]

Hi @reta @andrross ,

With 2.15.0 having the freeze date on 2024/06/07 (this Friday), do you guys think it would be the best to release 2.15.0 directly? Any arguments to have a 2.14.1 release now as it could take equally or just a bit less amount of time.

Also syncing @elfisher @Pallavi-AWS @getsaurabh02 into the conversation as we need to decide on this quick. Thanks!

If releasing 2.14.1 would take significant bandwidth, we can go with 2.15 directly and make sure that releases on time with all fixes included.

[artemus717]

After upgrade jackson-core to 2.17.1 on OpenSearch 2.14.0. JVM seemed normal now.

[getsaurabh02]

I agree. If there's no urgent need, considering it's likely a dependency issue rather than the code itself, upgrading to version 2.15 directly should be an option. We're only a few days away from the code freeze.

[peterzhuamazon]

Added label to move the fix to 2.15.0 release.
Based on the suggestions we will not release 2.14.1 for now.

Thanks.

[peternied]

[Triage - attendees 1 2 3 4 5 6 7]
@artemus717 Thanks for creating this issue