Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ERROR co.elastic.apm.agent.bci.IndyBootstrap - Advice threw an exception, this should never happen! #3530

Closed
dbalijapalli0264 opened this issue Feb 16, 2024 · 17 comments
Labels
agent-java community Issues and PRs created by the community triage

Comments

@dbalijapalli0264
Copy link

dbalijapalli0264 commented Feb 16, 2024

Describe the bug

Hello Elastic team,
Thank you for such an amazing product. I am opening this bug request to see if we can get some help from you on the error we are getting, while implementing elastic-apm-agent-1.45.0.jar or even the latest version elastic-apm-agent-1.47.0.jar. That's available on : https://mvnrepository.com/artifact/co.elastic.apm/elastic-apm-agent.

Our production log files are loaded with these 2 errors when we use the elastic-apm-agent-xxx.jar:
2024-02-15 20:12:34,254 [qtp292545106-1545] ERROR co.elastic.apm.agent.bci.IndyBootstrap - Advice threw an exception, this should never happen!
and
java.lang.ArrayIndexOutOfBoundsException: null

Steps to reproduce

Download teh api package from https://mvnrepository.com/artifact/co.elastic.apm/apm-agent-api/1.45.0 (or even 1.47.0)

  1. Update corresponding code block to extract following information and expose to GTM data layer

Add api jar dependency

import co.elastic.apm.api.ElasticApm;
import co.elastic.apm.api.Transaction;

Transaction transaction = ElasticApm.currentTransaction();

From transaction object we extract following attributes and store with “apmTransaction” key

traceId = transaction.traceId

spanId=transaction.ensureParentId()

sampled=transaction.sampled

Refer: https://www.elastic.co/guide/en/apm/agent/java/current/public-api.html

Expected behavior

Should not spam the logs with these errors:
ERROR co.elastic.apm.agent.bci.IndyBootstrap - Advice threw an exception, this should never happen!
AND
java.lang.ArrayIndexOutOfBoundsException: null

Debug logs

Click to expand
replace this line with your debug logs
@github-actions github-actions bot added agent-java community Issues and PRs created by the community triage labels Feb 16, 2024
@JonasKunz
Copy link
Contributor

Hi @dbalijapalli0264 , those two exceptions should also have stacktraces in addition to the exception messages.
Could you please provide the stacktraces?

@dbalijapalli0264
Copy link
Author

Apologies for not getting back earlier.
I don't see any stack trace for the error that says: "ERROR co.elastic.apm.agent.bci.IndyBootstrap - Advice threw an exception, this should never happen!"

But do see the below for the java.lang.ArrayIndexOutOfBoundsException: null

java.lang.ArrayIndexOutOfBoundsException: Index 34 out of bounds for length 34 at java.lang.AbstractStringBuilder.charAt(AbstractStringBuilder.java:353) ~[?:?] at java.lang.StringBuilder.charAt(StringBuilder.java:91) ~[?:?] at co.elastic.apm.agent.metrics.Labels$AbstractBase.hash(Labels.java:201) ~[elastic-apm-agent-1.43.0.jar:1.43.0] at co.elastic.apm.agent.metrics.Labels$AbstractBase.hashCode(Labels.java:135) ~[elastic-apm-agent-1.43.0.jar:1.43.0] at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936) ~[?:?] at co.elastic.apm.agent.metrics.MetricRegistry.getOrCreateMetricSet(MetricRegistry.java:233) ~[elastic-apm-agent-1.43.0.jar:1.43.0] at co.elastic.apm.agent.metrics.MetricRegistry.updateTimer(MetricRegistry.java:218) ~[elastic-apm-agent-1.43.0.jar:1.43.0] at co.elastic.apm.agent.impl.transaction.Transaction.trackMetrics(Transaction.java:525) ~[elastic-apm-agent-1.43.0.jar:1.43.0] at co.elastic.apm.agent.impl.transaction.Transaction.afterEnd(Transaction.java:284) ~[elastic-apm-agent-1.43.0.jar:1.43.0] at co.elastic.apm.agent.impl.transaction.AbstractSpan.end(AbstractSpan.java:580) ~[elastic-apm-agent-1.43.0.jar:1.43.0] at co.elastic.apm.agent.impl.transaction.AbstractSpan.end(AbstractSpan.java:557) ~[elastic-apm-agent-1.43.0.jar:1.43.0] at co.elastic.apm.agent.servlet.ServletTransactionHelper.onAfter(ServletTransactionHelper.java:248) ~[elastic-apm-agent-1.43.0.jar:1.43.0] at co.elastic.apm.agent.servlet.ServletApiAdvice.onExitServlet(ServletApiAdvice.java:253) ~[elastic-apm-agent-1.43.0.jar:1.43.0] at co.elastic.apm.agent.servlet.JavaxServletApiAdvice.onExitServletService(JavaxServletApiAdvice.java:45) ~[elastic-apm-agent-1.43.0.jar:1.43.0] at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1602) ~[people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548) ~[people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) ~[people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600) ~[people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) ~[people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) ~[people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434) ~[people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) ~[people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501) ~[people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) ~[people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) ~[people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349) ~[people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:763) ~[people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at com.about.mantle.spring.jetty.JettyBaseConfig$GlobeGzipHandler.handle(JettyBaseConfig.java:436) ~[people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:722) ~[people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at com.about.mantle.spring.jetty.JettyBaseConfig$GlobeGzipHandler.handle(JettyBaseConfig.java:436) ~[people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at com.about.mantle.spring.jetty.MantleInstrumentedHandler.handle(MantleInstrumentedHandler.java:92) ~[people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at org.eclipse.jetty.server.Server.handle(Server.java:516) ~[people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:400) ~[people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:645) [people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:392) [people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) [people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) [people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) [people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) [people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) [people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) [people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) [people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) [people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409) [people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) [people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) [people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at java.lang.Thread.run(Thread.java:833) [?:?]

@dbalijapalli0264
Copy link
Author

stack trace right before the error:
java.lang.ArrayIndexOutOfBoundsException

error.exception.stacktrace
[
{
"exclude_from_grouping": false,
"filename": "DebugHttpServletRequest.java",
"classname": "com.about.globe.core.http.DebugHttpServletRequest",
"line": {
"number": 166
},
"function": "getParameterMap",
"module": "com.about.globe.core.http"
},
{
"exclude_from_grouping": false,
"library_frame": true,
"filename": "HttpRequestContextContributor.java",
"classname": "com.about.mantle.http.HttpRequestContextContributor",
"line": {
"number": 188
},
"function": "contribute",
"module": "com.about.mantle.http"
},
{
"exclude_from_grouping": false,
"filename": "RequestContextFactoryImpl.java",
"classname": "com.about.globe.core.http.impl.RequestContextFactoryImpl",
"line": {
"number": 24
},
"function": "create",
"module": "com.about.globe.core.http.impl"
},
{
"exclude_from_grouping": false,
"library_frame": true,
"filename": "MantleDebugRequestHandlerMethods.java",
"classname": "com.about.mantle.handlers.methods.MantleDebugRequestHandlerMethods",
"line": {
"number": 176
},
"function": "debugSupportMethod",
"module": "com.about.mantle.handlers.methods"
},
{
"exclude_from_grouping": false,
"library_frame": true,
"filename": "MantleDebugRequestHandlerMethods.java",
"classname": "com.about.mantle.handlers.methods.MantleDebugRequestHandlerMethods",
"line": {
"number": 147
},
"function": "debugModels",
"module": "com.about.mantle.handlers.methods"
},
{
"exclude_from_grouping": false,
"library_frame": true,
"filename": "DebugModelsController.java",
"classname": "com.about.mantle.endpoint.controllers.DebugModelsController",
"line": {
"number": 23
},
"function": "doGet",
"module": "com.about.mantle.endpoint.controllers"
},
{
"exclude_from_grouping": false,
"library_frame": true,
"filename": "AbstractMantleEndpointController.java",
"classname": "com.about.mantle.endpoint.controllers.AbstractMantleEndpointController",
"line": {
"number": 73
},
"function": "doService",
"module": "com.about.mantle.endpoint.controllers"
},
{
"exclude_from_grouping": false,
"library_frame": true,
"filename": "AbstractMantleEndpointController.java",
"classname": "com.about.mantle.endpoint.controllers.AbstractMantleEndpointController",
"line": {
"number": 114
},
"function": "handleRequest",
"module": "com.about.mantle.endpoint.controllers"
},
{
"exclude_from_grouping": false,
"library_frame": true,
"filename": "SimpleControllerHandlerAdapter.java",
"classname": "org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter",
"line": {
"number": 52
},
"function": "handle",
"module": "org.springframework.web.servlet.mvc"
},
{
"exclude_from_grouping": false,
"library_frame": true,
"filename": "DispatcherServlet.java",
"classname": "org.springframework.web.servlet.DispatcherServlet",
"line": {
"number": 1060
},
"function": "doDispatch",
"module": "org.springframework.web.servlet"
},
{
"exclude_from_grouping": false,
"library_frame": true,
"filename": "DispatcherServlet.java",
"classname": "org.springframework.web.servlet.DispatcherServlet",
"line": {
"number": 962
},
"function": "doService",
"module": "org.springframework.web.servlet"
},
{
"exclude_from_grouping": false,
"library_frame": true,
"filename": "FrameworkServlet.java",
"classname": "org.springframework.web.servlet.FrameworkServlet",
"line": {
"number": 1006
},
"function": "processRequest",
"module": "org.springframework.web.servlet"
},
{
"exclude_from_grouping": false,
"library_frame": true,
"filename": "FrameworkServlet.java",
"classname": "org.springframework.web.servlet.FrameworkServlet",
"line": {
"number": 898
},
"function": "doGet",
"module": "org.springframework.web.servlet"
},
{
"exclude_from_grouping": false,
"library_frame": true,
"filename": "HttpServlet.java",
"classname": "javax.servlet.http.HttpServlet",
"line": {
"number": 687
},
"function": "service",
"module": "javax.servlet.http"
},
{
"exclude_from_grouping": false,
"library_frame": true,
"filename": "FrameworkServlet.java",
"classname": "org.springframework.web.servlet.FrameworkServlet",
"line": {
"number": 883
},
"function": "service",
"module": "org.springframework.web.servlet"
},
{
"exclude_from_grouping": false,
"library_frame": true,
"filename": "HttpServlet.java",
"classname": "javax.servlet.http.HttpServlet",
"line": {
"number": 790
},
"function": "service",
"module": "javax.servlet.http"
},
{
"exclude_from_grouping": false,
"library_frame": true,
"filename": "ServletHolder.java",
"classname": "org.eclipse.jetty.servlet.ServletHolder",
"line": {
"number": 799
},
"function": "handle",
"module": "org.eclipse.jetty.servlet"
},
{
"exclude_from_grouping": false,
"library_frame": true,
"filename": "ServletHandler.java",
"classname": "org.eclipse.jetty.servlet.ServletHandler$ChainEnd",
"line": {
"number": 1631
},
"function": "doFilter",
"module": "org.eclipse.jetty.servlet"
},
{
"exclude_from_grouping": false,
"library_frame": true,
"filename": "WebSocketUpgradeFilter.java",
"classname": "org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter",
"line": {
"number": 228
},
"function": "doFilter",
"module": "org.eclipse.jetty.websocket.server"
},
{
"exclude_from_grouping": false,
"library_frame": true,
"filename": "FilterHolder.java",
"classname": "org.eclipse.jetty.servlet.FilterHolder",
"line": {
"number": 193
},
"function": "doFilter",
"module": "org.eclipse.jetty.servlet"
},
{
"exclude_from_grouping": false,
"library_frame": true,
"filename": "ServletHandler.java",
"classname": "org.eclipse.jetty.servlet.ServletHandler$Chain",
"line": {
"number": 1601
},
"function": "doFilter",
"module": "org.eclipse.jetty.servlet"
},
{
"exclude_from_grouping": false,
"library_frame": true,
"filename": "ContentCachingRequestFilter.java",
"classname": "com.about.mantle.web.filter.ContentCachingRequestFilter",
"line": {
"number": 23
},
"function": "doHttpFilter",
"module": "com.about.mantle.web.filter"
},
{
"exclude_from_grouping": false,
"filename": "AbstractCoreFilter.java",
"classname": "com.about.globe.core.web.filter.AbstractCoreFilter",
"line": {
"number": 37
},
"function": "doFilter",
"module": "com.about.globe.core.web.filter"
},
{
"exclude_from_grouping": false,
"library_frame": true,
"filename": "FilterHolder.java",
"classname": "org.eclipse.jetty.servlet.FilterHolder",
"line": {
"number": 193
},
"function": "doFilter",
"module": "org.eclipse.jetty.servlet"
},
{
"exclude_from_grouping": false,
"library_frame": true,
"filename": "ServletHandler.java",
"classname": "org.eclipse.jetty.servlet.ServletHandler$Chain",
"line": {
"number": 1601
},
"function": "doFilter",
"module": "org.eclipse.jetty.servlet"
},
{
"exclude_from_grouping": false,
"library_frame": true,
"filename": "XContentTypeOptionsFilter.java",
"classname": "com.about.mantle.web.filter.XContentTypeOptionsFilter",
"line": {
"number": 24
},
"function": "doHttpFilter",
"module": "com.about.mantle.web.filter"
},
{
"exclude_from_grouping": false,
"library_frame": true,
"filename": "AbstractHttpFilter.java",
"classname": "com.about.mantle.web.filter.AbstractHttpFilter",
"line": {
"number": 31
},
"function": "doFilter",
"module": "com.about.mantle.web.filter"
},
{
"exclude_from_grouping": false,
"library_frame": true,
"filename": "FilterHolder.java",
"classname": "org.eclipse.jetty.servlet.FilterHolder",
"line": {
"number": 193
},
"function": "doFilter",
"module": "org.eclipse.jetty.servlet"
},
{
"exclude_from_grouping": false,
"library_frame": true,
"filename": "ServletHandler.java",
"classname": "org.eclipse.jetty.servlet.ServletHandler$Chain",
"line": {
"number": 1601
},
"function": "doFilter",
"module": "org.eclipse.jetty.servlet"
},
{
"exclude_from_grouping": false,
"library_frame": true,
"filename": "MalformedURLFilter.java",
"classname": "com.about.mantle.web.filter.MalformedURLFilter",
"line": {
"number": 29
},
"function": "doHttpFilter",
"module": "com.about.mantle.web.filter"
},
{
"exclude_from_grouping": false,
"filename": "AbstractCoreFilter.java",
"classname": "com.about.globe.core.web.filter.AbstractCoreFilter",
"line": {
"number": 37
},
"function": "doFilter",
"module": "com.about.globe.core.web.filter"
},
{
"exclude_from_grouping": false,
"library_frame": true,
"filename": "FilterHolder.java",
"classname": "org.eclipse.jetty.servlet.FilterHolder",
"line": {
"number": 193
},
"function": "doFilter",
"module": "org.eclipse.jetty.servlet"
},
{
"exclude_from_grouping": false,
"library_frame": true,
"filename": "ServletHandler.java",
"classname": "org.eclipse.jetty.servlet.ServletHandler$Chain",
"line": {
"number": 1601
},
"function": "doFilter",
"module": "org.eclipse.jetty.servlet"
},
{
"exclude_from_grouping": false,
"library_frame": true,
"filename": "ExternalServiceProxyFilter.java",
"classname": "com.about.mantle.web.filter.ExternalServiceProxyFilter",
"line": {
"number": 36
},
"function": "doHttpFilter",
"module": "com.about.mantle.web.filter"
},
{
"exclude_from_grouping": false,
"filename": "AbstractCoreFilter.java",
"classname": "com.about.globe.core.web.filter.AbstractCoreFilter",
"line": {
"number": 37
},
"function": "doFilter",
"module": "com.about.globe.core.web.filter"
},
{
"exclude_from_grouping": false,
"library_frame": true,
"filename": "FilterHolder.java",
"classname": "org.eclipse.jetty.servlet.FilterHolder",
"line": {
"number": 193
},
"function": "doFilter",
"module": "org.eclipse.jetty.servlet"
},
{
"exclude_from_grouping": false,
"library_frame": true,
"filename": "ServletHandler.java",
"classname": "org.eclipse.jetty.servlet.ServletHandler$Chain",
"line": {
"number": 1601
},
"function": "doFilter",
"module": "org.eclipse.jetty.servlet"
},
{
"exclude_from_grouping": false,
"library_frame": true,
"filename": "ResponseHeadersFilter.java",
"classname": "com.about.mantle.web.filter.ResponseHeadersFilter",
"line": {
"number": 33
},
"function": "doFilter",
"module": "com.about.mantle.web.filter"
},
{
"exclude_from_grouping": false,
"library_frame": true,
"filename": "FilterHolder.java",
"classname": "org.eclipse.jetty.servlet.FilterHolder",
"line": {
"number": 193
},
"function": "doFilter",
"module": "org.eclipse.jetty.servlet"
},
{
"exclude_from_grouping": false,
"library_frame": true,
"filename": "ServletHandler.java",
"classname": "org.eclipse.jetty.servlet.ServletHandler$Chain",
"line": {
"number": 1601
},
"function": "doFilter",
"module": "org.eclipse.jetty.servlet"
},
{
"exclude_from_grouping": false,
"library_frame": true,
"filename": "SafelistHttpMethodFilter.java",
"classname": "com.about.mantle.web.filter.SafelistHttpMethodFilter",
"line": {
"number": 23
},
"function": "doHttpFilter",
"module": "com.about.mantle.web.filter"
},
{
"exclude_from_grouping": false,
"library_frame": true,
"filename": "AbstractHttpFilter.java",
"classname": "com.about.mantle.web.filter.AbstractHttpFilter",
"line": {
"number": 31
},
"function": "doFilter",
"module": "com.about.mantle.web.filter"
},
{
"exclude_from_grouping": false,
"library_frame": true,
"filename": "FilterHolder.java",
"classname": "org.eclipse.jetty.servlet.FilterHolder",
"line": {
"number": 193
},
"function": "doFilter",
"module": "org.eclipse.jetty.servlet"
},
{
"exclude_from_grouping": false,
"library_frame": true,
"filename": "ServletHandler.java",
"classname": "org.eclipse.jetty.servlet.ServletHandler$Chain",
"line": {
"number": 1601
},
"function": "doFilter",
"module": "org.eclipse.jetty.servlet"
},
{
"exclude_from_grouping": false,
"library_frame": true,
"filename": "EnsureHttpsFilter.java",
"classname": "com.about.mantle.web.filter.EnsureHttpsFilter",
"line": {
"number": 33
},
"function": "doHttpFilter",
"module": "com.about.mantle.web.filter"
},
{
"exclude_from_grouping": false,
"filename": "AbstractCoreFilter.java",
"classname": "com.about.globe.core.web.filter.AbstractCoreFilter",
"line": {
"number": 37
},
"function": "doFilter",
"module": "com.about.globe.core.web.filter"
},
{
"exclude_from_grouping": false,
"library_frame": true,
"filename": "FilterHolder.java",
"classname": "org.eclipse.jetty.servlet.FilterHolder",
"line": {
"number": 193
},
"function": "doFilter",
"module": "org.eclipse.jetty.servlet"
},
{
"exclude_from_grouping": false,
"library_frame": true,
"filename": "ServletHandler.java",
"classname": "org.eclipse.jetty.servlet.ServletHandler$Chain",
"line": {
"number": 1601
},
"function": "doFilter",
"module": "org.eclipse.jetty.servlet"
},
{
"exclude_from_grouping": false,
"library_frame": true,
"filename": "TraceFilter.java",
"classname": "com.about.mantle.web.filter.TraceFilter",
"line": {
"number": 43
},
"function": "doFilter",
"module": "com.about.mantle.web.filter"
},
{
"exclude_from_grouping": false,
"library_frame": true,
"filename": "FilterHolder.java",
"classname": "org.eclipse.jetty.servlet.FilterHolder",
"line": {
"number": 193
},
"function": "doFilter",
"module": "org.eclipse.jetty.servlet"
}
]

@JonasKunz
Copy link
Contributor

Are you using Transaction.setName anywhere in you code?
The shown error indicates a race condition:

  • The transaction name is internally stored in a StringBuilder
  • When the transaction ends, a hashcode is computed for the transaction name (stored in the StringBuilder)
  • If now the transaction name is updated concurrently after the transaction has ended, that can lead to the ArrayIndexOutOfBounds exception you are seeing

@dbalijapalli0264
Copy link
Author

Thank you @JonasKunz, looking into it now. Will get back with the outcome.

@SylvainJuge
Copy link
Member

Hi @dbalijapalli0264, there hasn't been any update since your last message, is this issue still happening in your environment ?

@jackshirazi
Copy link
Contributor

Closing this, if it's still an issue, comment or reopen it as needed

@dbalijapalli0264
Copy link
Author

Hello, Sorry for lack of response, we had to roll back the changes as this exception was overloading our production error logs. We are back testing it now. and i do see that the exception is still happening.

Here are the exceptions, these both exceptions happen together when they happen.

co.elastic.apm.agent.bci.IndyBootstrap

message:2024-05-09 18:20:20,202 [qtp2056288620-1138] ERROR co.elastic.apm.agent.bci.IndyBootstrap - Advice threw an exception, this should never happen!

java.lang.ArrayIndexOutOfBoundsException

java.lang.ArrayIndexOutOfBoundsException: Index 34 out of bounds for length 34 at java.lang.AbstractStringBuilder.charAt(AbstractStringBuilder.java:353) ~[?:?] at java.lang.StringBuilder.charAt(StringBuilder.java:91) ~[?:?] at co.elastic.apm.agent.metrics.Labels$AbstractBase.hash(Labels.java:201) ~[elastic-apm-agent-1.47.0.jar:1.47.0] at co.elastic.apm.agent.metrics.Labels$AbstractBase.hashCode(Labels.java:135) ~[elastic-apm-agent-1.47.0.jar:1.47.0] at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936) ~[?:?] at co.elastic.apm.agent.metrics.MetricRegistry.getOrCreateMetricSet(MetricRegistry.java:233) ~[elastic-apm-agent-1.47.0.jar:1.47.0] at co.elastic.apm.agent.metrics.MetricRegistry.updateTimer(MetricRegistry.java:218) ~[elastic-apm-agent-1.47.0.jar:1.47.0] at co.elastic.apm.agent.impl.transaction.Transaction.trackMetrics(Transaction.java:525) ~[elastic-apm-agent-1.47.0.jar:1.47.0] at co.elastic.apm.agent.impl.transaction.Transaction.afterEnd(Transaction.java:284) ~[elastic-apm-agent-1.47.0.jar:1.47.0] at co.elastic.apm.agent.impl.transaction.AbstractSpan.end(AbstractSpan.java:578) ~[elastic-apm-agent-1.47.0.jar:1.47.0] at co.elastic.apm.agent.impl.transaction.AbstractSpan.end(AbstractSpan.java:548) ~[elastic-apm-agent-1.47.0.jar:1.47.0] at co.elastic.apm.agent.servlet.ServletTransactionHelper.onAfter(ServletTransactionHelper.java:248) ~[elastic-apm-agent-1.47.0.jar:1.47.0] at co.elastic.apm.agent.servlet.ServletApiAdvice.onExitServlet(ServletApiAdvice.java:253) ~[elastic-apm-agent-1.47.0.jar:1.47.0] at co.elastic.apm.agent.servlet.JavaxServletApiAdvice.onExitServletService(JavaxServletApiAdvice.java:45) ~[elastic-apm-agent-1.47.0.jar:1.47.0]

@dbalijapalli0264
Copy link
Author

dbalijapalli0264 commented May 9, 2024

Could we reopen this issue please? SylvainJuge or jackshirazi

@SylvainJuge
Copy link
Member

Hi @dbalijapalli0264 , as Jonas asked previously here do you know if the transaction name is modified using Transaction.setName

@SylvainJuge
Copy link
Member

@dbalijapalli0264 to be a bit more clear here, we need more information about your context in order to make any progress here, otherwise there is no point in keeping this open if we don't have any actionable insights on the things we might need to fix here.

@dbalijapalli0264
Copy link
Author

dbalijapalli0264 commented May 20, 2024

Hello, and apologies, that it took me a while before i got back the information, here is the information you requested.

@SylvainJuge or @jackshirazi
your question: Are you using Transaction.setName --- yes we are setting the transaction name. Bt we are setting the name before the transaction ends. The transaction name is not updated concurrently after the transaction has ended.

Below is the snippet of our code. The backend to frontend correlation works fine(attached an image, please see), but, our logs are over loaded with the ArrayIndexOutOfBounds and co.elastic.apm.agent.bci.IndyBootstrap errors, which is not acceptable.

(PS: Please note in the code below, we don't set the transaction name for the deferred calls. )


import co.elastic.apm.api.ElasticApm;
import co.elastic.apm.api.Transaction;
import com.about.mantle.model.gtm.ApmTransaction;

public ApmTransaction createApmTransaction(RequestContext rc) {
if (rc.isDeferred()) {
    return null;
}

ApmTransaction.Builder builder = new ApmTransaction.Builder();
try {
    Transaction apmTransaction = ElasticApm.currentTransaction();
    builder = builder.traceId(apmTransaction.getTraceId()).spanId(apmTransaction.ensureParentId()).sampled(apmTransaction.isSampled());
    apmTransaction.setName(rc.getTemplateName());
} catch (Exception e) {
    log.error("Error adding elastic apm transaction to gtm model", e);
}
return builder.build();
}

From the attached screenshot, as you see, we are just trying to set the template name to the transaction. without the above code to set transaction name, the template name is not being set.
Screenshot 2024-05-20 at 9 43 52 AM

If this is not the proper way to do it, could you please let me know how to do it so that we have a proper correlation from backend to frontend.
(PS: Its been very frustrating that we are not able to get this work fine despite all our efforts to have a proper correlated transaction.)

@JonasKunz
Copy link
Contributor

But we are setting the name before the transaction ends. The transaction name is not updated concurrently after the transaction has ended.

Unfortunately that is the only possible explanation for the error you are seeing I can come up with right now. Also we have never seen this error occur before. Could you remove all calls to Transaction.setName and check whether the error still occurs?

@dbalijapalli0264
Copy link
Author

Thanks you for the response @JonasKunz, Checking with our team internally, will get back to you.

@dbalijapalli0264
Copy link
Author

We are trying to remove the Transaction.setName after running a set of tests, please keep this ticket active for a bit longer.

@paulbandler
Copy link

Was there any closure on this issue? I've just come across the same exception. My application is a grails application. We have no APM API calls in our application, buy I believe the apm agent is 'grails aware' in that it identifies transactions with grails controller entry points, so perhaps it is internally setting the transaction name as mentioned above?

This has started to happen as I was increasing the load today.

@jackshirazi
Copy link
Contributor

please open a discussion item at our forum

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
agent-java community Issues and PRs created by the community triage
Projects
None yet
Development

No branches or pull requests

5 participants