SIGSEGV Tomcat 10.1 Java 17 #3521
Comments
github-actions
bot
added
agent-java
community
|
Does it still happen if you use a different garbage collector? |
|
I didn't try it. My application uses G1GC Should i try SerialGC? |
|
On another enviroment my application uses SerialGC, but the problem still exists |
|
Any chance of producing a test we can reproduce? |
|
Can you post a couple of the other crashlogs too please |
|
Also just to confirm, you haven't enabled inferred spans? You can see the properties that are set in the startup info (agent output with lines including StartupInfo) |
|
I think that i didn't enable inferred spans
What is the most important for you in other crashlogs? Problematic frame? J 20648 c2 co.elastic.apm.agent.loginstr.reformatting.AbstractEcsReformattingHelper.onAppendEnter(Ljava/lang/Object;)Z (124 bytes) @ 0x00007f20ae058090 [0x00007f20ae057ce0+0x00000000000003b0] |
|
Are all the stacks the same in the crashlogs? If any are different we'd like to see any other stacks. Two suggestions we'd like you to try separately:
|
|
Ad.2 I added -XX:CompileCommand=exclude,co/elastic/apm/agent/collections/CachedKeyWeakConcurrentMap.getLookupKey Now the problematic frame is : # J 2489 c2 co.elastic.apm.agent.weakconcurrent.CachedLookupKey.get(Ljava/lang/Object;)Lco/elastic/apm/agent/weakconcurrent/CachedLookupKey; (16 bytes) @ 0x00007f1168fcfbc6 [0x00007f1168fcfac0+0x0000000000000106] ErrorFile in attachment |
|
Did you get any info from the JITC logging? Can we try more generic JITC disabling, by now adding to the command line It's not at all clear why this is crashing, the hotspot logging is itself crashing (hence the incomplete crashlog) but it's successfully seeing everything in the Elastic data structure, so it's something outside that, it's a process of narrowing down by excluding other things If you need an urgent workaround, you can disable the reformatting eg |
|
JITC log will be available soon... I Added below instruction to command line and now problematic frame is: New error file is available in attachment |
|
Yes, this is crashing inside the JVM native code in ObjectSynchronizer::FastHashCode. It's not at all clear why that would happen, we'll continue to look into it but I don't expect any quick solution, it's not even clear this is an agent bug. The best we can suggest is the workaround of disabling the log-reformatting given above. If you can create a standalone example that you can pass to us to reproduce this crash, that would be awesome! |
|
Hi @fr19k , We have another user with the same crash: Both crashes occur due to java.util logging reformatting on the same One thing I noticed is that both you and the other user specify the Could you maybe try again without that flag (so using the default logmanager) and check whether the error still occurs? |
|
OK, i will check your suggestion with log manager I can confirm that workaround with -Delastic.apm.disable_instrumentations=log-reformatting works properly (application is not crashing) |
|
What is the default java.util.logging.manager ? Tomcat seems to allows to overwrite java.util.logging.manager via LOGGING_MANAGER env, but i have specify proper class |
|
I think it should be Alternatively you might be able to simply remove the part where it configures |
|
If i set env variable LOGGING_MANAGER to java.util.logging.LogManager , application crashes at startup with below message |
|
Based on this SO thread it looks like you need to provide the full arguments for the If that doesn't work, I think we can also trick tomcat by setting As @jackshirazi mentioned earlier, it would most likely be easier for us to find the root cause if you could provide us an application / docker image to allow us to reproduce this crash ourselves. |
|
Hi @fr19k, Given that the crash seems quite fast to trigger here, we'd like to have a bit more information on your app/setup to enable us attempting to reproduce it. could you provide the following:
|
|
Hi,
Via env variables i'm passing ELASTIC_APM_SECRET_TOKEN, any other configs are passed through JVM variables
if i set LOGGING_MANAGER to, "-Djava.util.logging.manager=java.util.logging.LogManager" or "-Ddummy.prop=foobar" i receive many errors from tomcat like
My application uses logback for logging. The class java.util.logging is used only for tomcat part
My application crashes in every 30-60mins after normal traffic. When I disable log-reformatting, pod is stable (15days without restart) There was no intensive load tests, it's just normal production traffic. GcLogs looks good (there are no full gc pauses), throughput ~99.5% Below you can see a chart with typical(production) number of transactions |
|
Thanks for your feedback, with that we should be able to attempt reproducing it. One interesting part is that while the application uses logback for logging, the JUL logging API used by Tomcat should have very light usage in production. |
|
Logback is used due there is plugin(logback-logstash-encoder) to sending logs directly to logstash |
|
Hi @fr19k, an Oracle engineer has suggested running the test with -Xint - which although it would be much slower, could indicate whether or not this is a compiler bug. Would you be willing to run that test? It would likely need to run for much longer than previous tests |
|
Hi, no need to try this, we've reproduced the crash and found that disabling the c2 compiler (with |
|
To be closed after JVM release with https://bugs.openjdk.org/browse/JDK-8322726 fixed (we also expect a backport to 17 and 21, so wait for those for tracking purposes before closing) |
|
Hi @yuvalbar84 this is not a jar option, the c2 compiler is a JIT compiler in the JVM. Your options are any of these
|
|
Should be fixed now in JDK 17.0.12 according to the release notes. |
|
@JonasKunz thank you ! |
|
MInor correction: |
Hi,
Elastic Apm agent (version 1.46.0) causes SIGSEGV crash in JVM (17.0.10)
Then crash occures usually in 30min after kubernetes pod starts
My application is using tomcat 10.1.18 docker image
In attachment you can find Error Report from JVM
Application works properly with disabled/removed Elastic APM Agent
is there any workaround for that? maybe some fix?
hss_err_pid1.txt
The text was updated successfully, but these errors were encountered: