Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

not receiving Data at elastic Side #318

Open
ahmedhkhalil opened this issue May 21, 2024 · 20 comments
Open

not receiving Data at elastic Side #318

ahmedhkhalil opened this issue May 21, 2024 · 20 comments

Comments

@ahmedhkhalil
Copy link

Dears,
we are trying to instrument flutter application using andorid sdk native agent from Elastic we have followed below documentation
https://www.elastic.co/observability-labs/blog/monitoring-android-applications-apm
there is no data appeared on elastic side despite below :
on Android Side we are receiving below warning:
D/ELASTIC_AGENT(19073): Ignoring central config sync request
I/ELASTIC_AGENT(19073): Central config returned max age is null
I/ELASTIC_AGENT(19073): Scheduling next central config poll
D/ELASTIC_AGENT(19073): Next central config poll in 60 seconds
D/ELASTIC_AGENT(19073): Starting span: 'SdkSpan{traceId=e08badad1310d7d54bfb90c72ddd7a82, spanId=a87fee2a479b8368, parentSpanContext=ImmutableSpanContext{traceId=e08badad1310d7d54bfb90c72ddd7a82, spanId=0de1adc07d3f86e1, traceFlags=01, traceState=ArrayBasedTraceState{entries=[]}, remote=false, valid=true}, name=Paused, kind=INTERNAL, attributes=AttributesMap{data={network.carrier.icc=eg, session.id=802fa057-7ba2-4c1f-9146-dcc37aecdf4e, network.carrier.mcc=602, network.connection.type=wifi, type=mobile, network.carrier.name=Orange EG, network.carrier.mnc=01, fragmentName=FlutterFragment}, capacity=128, totalAddedValues=9}, status=ImmutableStatusData{statusCode=OK, description=}, totalRecordedEvents=0, totalRecordedLinks=0, startEpochNanos=1716282098617757968, endEpochNanos=0}', within context: '{opentelemetry-trace-span-key=SdkSpan{traceId=e08badad1310d7d54bfb90c72ddd7a82, spanId=0de1adc07d3f86e1, parentSpanContext=ImmutableSpanContext{traceId=00000000000000000000000000000000, spanId=0000000000000000, traceFlags=00, traceState=ArrayBasedTraceState{entries=[]}, remote=false, valid=false}, name=Paused, kind=INTERNAL, attributes=AttributesMap{data={network.carrier.icc=eg, activityName=MainActivity, session.id=802fa057-7ba2-4c1f-9146-dcc37aecdf4e, network.carrier.mcc=602, network.connection.type=wifi, type=mobile, network.carrier.name=Orange EG, network.carrier.mnc=01, screen.name=MainActivity}, capacity=128, totalAddedValues=11}, status=ImmutableStatusData{statusCode=OK, description=}, totalRecordedEvents=2, totalRecordedLinks=0, startEpochNanos=1716282098614000000, endEpochNanos=0}}'

and on Elastic APM fleet Server i recieve below :
./elastic-agent-20240429-336.ndjson:{"log.level":"info","@timestamp":"2024-04-29T13:17:36.477Z","message":"request ok","component":{"binary":"apm-server","dataset":"elastic_
agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"url.original":"/","http.request.method":"POST","user_agent.original":"OTel-OTLP-Exporter-J
ava/1.34.1","http.request.id":"f4c43c38-86f1-4750-928e-cfb726ab38ea","log.origin":{"file.line":62,"file.name":"middleware/log_middleware.go","function":"github.com/elastic/a
pm-server/internal/beater/api.apmMiddleware.LogMiddleware.func1.1"},"log.logger":"request","source.address":"","event.duration":66983,"http.request.body.bytes":
804,"http.response.status_code":200,"ecs.version":"1.6.0","service.name":"apm-server","ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-05-21T09:58:24.366Z","message":"not modified","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"url.original":"/config/v1/agents?service.name=####&service.environment=######","log.origin":{"file.line":62,"file.name":"middleware/log_middleware.go","function":"github.com/elastic/apm-server/internal/beater/api.apmMiddleware.LogMiddleware.func1.1"},"service.name":"apm-server","http.request.body.bytes":0,"http.response.status_code":304,"user_agent.original":"Dalvik/2.1.0 (Linux; U; Android 14; SM-S918B Build/UP1A.231005.007)","http.request.id":"13f77e5b-4f72-44b1-9c9a-dc23757123dd","event.duration":61349,"log.logger":"request","http.request.method":"GET","source.address":"197.38.152.50","ecs.version":"1.6.0","ecs.version":"1.6.0"}

Despite that no data shown in elastic or kibana and service not appeared

@LikeTheSalad
Copy link
Contributor

Hi @ahmedhkhalil

Please bear in mind that the Elastic agent is meant to be used for native Android projects only, and not for hybrid ones such as Flutter. However, given that Flutter projects contain an Android project inside, I would assume that the Elastic agent could be implemented directly into those to cover the code that's specifically written within said Android projects, though we haven't tested such a use case.

Based on the logs that you've shared, it seems like you've already implemented the Elastic agent within your Flutter's Android subproject and that some spans have been created as well, however, when it comes to the server logs, it doesn't seem like those belong to your Android application, this is because there are some missing attributes that we send in every span from our Android agent, and also because the value for the service.name ones that I can see is "apm-server".

Because of the above, we should start by verifying that your spans are reaching your APM Server. We can start by checking that your APM Server URL and secretToken values are configured within your Android application project. You can find both, the URL and the secretToken of your APM server, in Kibana, by opening up the menu and going to: Management->Integrations->APM - and then scrolling down until you see the following values:

Kibana APM values

After finding your APM Server URL and secretToken values, you should verify that you're setting them in your Android project, either via the Elastic Android Gradle plugin, as shown here, or, in case you're not using the plugin, you can also set them up manually as explained here.

@ahmedhkhalil
Copy link
Author

Dear,
thanks alot for your reply but as i have remove service.name and enviroment.name for security reason but i can confirm to you that below log is from APM server Logs it self which mean that connectivity is ok and elastic APM server is able to recieve event successfully with correct environment and service name , bbut issue is that this event is not parsed anywhere in elastic search it self that what i am understand , note also i have server EUM application running on same environment with different service name and all of them working fine
{"log.level":"info","@timestamp":"2024-05-21T09:58:24.366Z","message":"not modified","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"url.original":"/config/v1/agents?service.name=####&service.environment=######","log.origin":{"file.line":62,"file.name":"middleware/log_middleware.go","function":"github.com/elastic/apm-server/internal/beater/api.apmMiddleware.LogMiddleware.func1.1"},"service.name":"apm-server","http.request.body.bytes":0,"http.response.status_code":304,"user_agent.original":"Dalvik/2.1.0 (Linux; U; Android 14; SM-S918B Build/UP1A.231005.007)","http.request.id":"13f77e5b-4f72-44b1-9c9a-dc23757123dd","event.duration":61349,"log.logger":"request","http.request.method":"GET","source.address":"197.38.152.50","ecs.version":"1.6.0","ecs.version":"1.6.0"}

@LikeTheSalad
Copy link
Contributor

Got it, although it's still confusing because, apart from the removed service.name and environment.name attrs, there are some other missing attrs that we send from the Android agent, such as network.connection.type, or activityName, which, even though both show up in the logs that you've shared from your application, it seems like those are missing from the server logs, which is why I'm wondering if maybe the logs that you are seeing in your server might come from a different source rather than the Elastic Android agent.

It would be helpful to confirm that the server endpoint/auth values are the ones that show up in Kibana, as per my instructions above, just to discard that it might not be that kind of issue. Also, could you confirm which version of the Elastic Stack are you using? That could help to narrow down the possible causes of this issue.

@ahmedhkhalil
Copy link
Author

yes i confirm that , server url is correct and i am not using auth token in my environment , my elastic version is latest 8.13.4

@LikeTheSalad
Copy link
Contributor

Thank you for confirming. If that's the case, then it doesn't look like there are too many possible causes for this issue, at least on the Elastic agent configuration side of things. Having said that, I'd like to confirm a couple of things, does your APM Server URL starts with http or https? And, have you added any configuration changes into your app regarding the export protocol, as explained here?

@ahmedhkhalil
Copy link
Author

ahmedhkhalil commented May 22, 2024

APM endpoint is HTTP but Public URL is behind proxy which make ssl temination for requests from https to http
yes i changed endpoint from otel to http as below
private val apmConfiguration: ElasticApmConfiguration = ElasticApmConfiguration.builder()
.setExportProtocol(ExportProtocol.HTTP)
.setLibraryLoggingPolicy(LoggingPolicy.enabled(LogLevel.DEBUG))
.setServiceName("-Test-mum")
.setServiceVersion("1.0.4")
.setDeploymentEnvironment(deploymentEnvironment)
.build()

@LikeTheSalad
Copy link
Contributor

Thank you.

Usually, requests to endpoints starting with http:// tend to have issues due to Android's SSL/TLS default config. For testing purposes, I'd recommend enabling cleartext traffic, as explained here, just to discard that we're not dealing with an SSL/TLS related issue.

@ahmedhkhalil
Copy link
Author

in android endpoint is https not http , as android refuse to send requests to http
ok when i enabling clear text what do you like me to capture

@LikeTheSalad
Copy link
Contributor

Hi @ahmedhkhalil - Thank you for your patience. I just checked and it seems like there's an issue in our APM Server when using HTTP to export Android telemetry. I'm currently taking a look into it to gather more details, though, in the meantime, if you switch to ExportProtocol.GRPC, you should be able to see your Android APM data in Kibana.

I'll keep adding updates here as soon as I can find more info on the issue with HTTP.

@ahmedhkhalil
Copy link
Author

hi @LikeTheSalad when we tried grpc we was reciving below error message
E/ELASTIC_AGENT( 9960): java.io.IOException: unexpected end of stream on com.android.okhttp.Address@e77c591c
E/ELASTIC_AGENT( 9960): at com.android.okhttp.internal.http.Http1xStream.readResponse(Http1xStream.java:203)
E/ELASTIC_AGENT( 9960): at com.android.okhttp.internal.http.Http1xStream.readResponseHeaders(Http1xStream.java:129)
E/ELASTIC_AGENT( 9960): at com.android.okhttp.internal.http.HttpEngine.readNetworkResponse(HttpEngine.java:750)
E/ELASTIC_AGENT( 9960): at com.android.okhttp.internal.http.HttpEngine.readResponse(HttpEngine.java:622)
E/ELASTIC_AGENT( 9960): at com.android.okhttp.internal.huc.HttpURLConnectionImpl.execute(HttpURLConnectionImpl.java:475)
E/ELASTIC_AGENT( 9960): at com.android.okhttp.internal.huc.HttpURLConnectionImpl.getResponse(HttpURLConnectionImpl.java:411)
E/ELASTIC_AGENT( 9960): at com.android.okhttp.internal.huc.HttpURLConnectionImpl.getHeaders(HttpURLConnectionImpl.java:167)
E/ELASTIC_AGENT( 9960): at com.android.okhttp.internal.huc.HttpURLConnectionImpl.getHeaderField(HttpURLConnectionImpl.java:211)
E/ELASTIC_AGENT( 9960): at com.android.okhttp.internal.huc.DelegatingHttpsURLConnection.getHeaderField(DelegatingHttpsURLConnection.java:191)
E/ELASTIC_AGENT( 9960): at com.android.okhttp.internal.huc.HttpsURLConnectionImpl.getHeaderField(HttpsURLConnectionImpl.java:30)
E/ELASTIC_AGENT( 9960): at co.elastic.apm.android.sdk.internal.features.centralconfig.fetcher.CentralConfigurationFetcher.fetch(CentralConfigurationFetcher.java:78)
E/ELASTIC_AGENT( 9960): at co.elastic.apm.android.sdk.internal.features.centralconfig.CentralConfigurationManager.sync(CentralConfigurationManager.java:80)
E/ELASTIC_AGENT( 9960): at co.elastic.apm.android.sdk.internal.features.centralconfig.poll.ConfigurationPollManager.onTaskRun(ConfigurationPollManager.java:67)
E/ELASTIC_AGENT( 9960): at co.elastic.apm.android.sdk.internal.services.periodicwork.ManagedPeriodicTask.runTask(ManagedPeriodicTask.java:37)
E/ELASTIC_AGENT( 9960): at co.elastic.apm.android.sdk.internal.services.periodicwork.PeriodicWorkService.run(PeriodicWorkService.java:90)
E/ELASTIC_AGENT( 9960): at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:487)
E/ELASTIC_AGENT( 9960): at java.util.concurrent.FutureTask.run(FutureTask.java:264)
E/ELASTIC_AGENT( 9960): at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:307)
E/ELASTIC_AGENT( 9960): at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
E/ELASTIC_AGENT( 9960): at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:644)
E/ELASTIC_AGENT( 9960): at java.lang.Thread.run(Thread.java:1012)
E/ELASTIC_AGENT( 9960): Caused by: java.io.EOFException: \n not found: size=0 content=...
E/ELASTIC_AGENT( 9960): at com.android.okhttp.okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:202)
E/ELASTIC_AGENT( 9960): at com.android.okhttp.internal.http.Http1xStream.readResponse(Http1xStream.java:188)

@LikeTheSalad
Copy link
Contributor

It seems like those logs are related to central config polling, which is a separate request from the traces exporting one. Could you verify if you can see your app listed in Kibana -> Observability -> APM -> Services?

@ahmedhkhalil
Copy link
Author

Dears,
i tried below setup with grpc
APM local end point is http and it's public access without any load balancer in middle , i am able to recieve session and service appear in Kibana services tab
but when ever i put haproxy in middle to make ssl termination and send traffic to backend either grpc or http , no data is received on elastic and response from apm server is 400 and error message "invalid character "
i suspect that there is issue from APM side
image

@georgesamirmansour
Copy link

E/AndroidRuntime( 9291): java.lang.NoClassDefFoundError: Failed resolution of: Lio/opentelemetry/instrumentation/api/internal/SemconvStability;
E/AndroidRuntime( 9291): at io.opentelemetry.instrumentation.api.instrumenter.http.HttpClientAttributesExtractorBuilder.buildNetExtractor(HttpClientAttributesExtractorBuilder.java:134)
E/AndroidRuntime( 9291): at io.opentelemetry.instrumentation.api.instrumenter.http.HttpClientAttributesExtractor.(HttpClientAttributesExtractor.java:100)
E/AndroidRuntime( 9291): at io.opentelemetry.instrumentation.api.instrumenter.http.HttpClientAttributesExtractorBuilder.build(HttpClientAttributesExtractorBuilder.java:129)
E/AndroidRuntime( 9291): at io.opentelemetry.instrumentation.okhttp.v3_0.internal.OkHttpInstrumenterFactory.create(OkHttpInstrumenterFactory.java:56)
E/AndroidRuntime( 9291): at io.opentelemetry.instrumentation.library.okhttp.v3_0.internal.OkHttp3Singletons.lambda$static$2(OkHttp3Singletons.java:35)
E/AndroidRuntime( 9291): at io.opentelemetry.instrumentation.library.okhttp.v3_0.internal.OkHttp3Singletons.$r8$lambda$u63Q7IRr8X_9Th6yCeJtzlM1cyI(OkHttp3Singletons.java:0)
E/AndroidRuntime( 9291): at io.opentelemetry.instrumentation.library.okhttp.v3_0.internal.OkHttp3Singletons$$ExternalSyntheticLambda0.get(R8$$SyntheticClass:0)
E/AndroidRuntime( 9291): at io.opentelemetry.instrumentation.library.okhttp.v3_0.internal.CachedSupplier.get(CachedSupplier.java:31)
E/AndroidRuntime( 9291): at io.opentelemetry.instrumentation.library.okhttp.v3_0.internal.OkHttp3Singletons.lambda$static$5(OkHttp3Singletons.java:83)
E/AndroidRuntime( 9291): at io.opentelemetry.instrumentation.library.okhttp.v3_0.internal.OkHttp3Singletons.$r8$lambda$FBKUum8dYLg-qIeXrCw3rmWpOr8(OkHttp3Singletons.java:0)
E/AndroidRuntime( 9291): at io.opentelemetry.instrumentation.library.okhttp.v3_0.internal.OkHttp3Singletons$$ExternalSyntheticLambda3.get(R8$$SyntheticClass:0)
E/AndroidRuntime( 9291): at io.opentelemetry.instrumentation.library.okhttp.v3_0.internal.CachedSupplier.get(CachedSupplier.java:31)
E/AndroidRuntime( 9291): at io.opentelemetry.instrumentation.library.okhttp.v3_0.internal.LazyInterceptor.intercept(LazyInterceptor.java:27)
E/AndroidRuntime( 9291): at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
E/AndroidRuntime( 9291): at io.opentelemetry.instrumentation.library.okhttp.v3_0.internal.OkHttp3Singletons.lambda$static$4(OkHttp3Singletons.java:76)
E/AndroidRuntime( 9291): at io.opentelemetry.instrumentation.library.okhttp.v3_0.internal.OkHttp3Singletons.$r8$lambda$663SEQRE2c1G7-M_pfa4eU-NLQ4(OkHttp3Singletons.java:0)
E/AndroidRuntime( 9291): at io.opentelemetry.instrumentation.library.okhttp.v3_0.internal.OkHttp3Singletons$$ExternalSyntheticLambda2.intercept(R8$$SyntheticClass:0)
E/AndroidRuntime( 9291): at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
E/AndroidRuntime( 9291): at io.opentelemetry.instrumentation.library.okhttp.v3_0.internal.OkHttp3Singletons.lambda$static$3(OkHttp3Singletons.java:65)
E/AndroidRuntime( 9291): at io.opentelemetry.instrumentation.library.okhttp.v3_0.internal.OkHttp3Singletons.$r8$lambda$gJVQNJFg1SbPjywQG7gDltoRIL0(OkHttp3Singletons.java:0)
E/AndroidRuntime( 9291): at io.opentelemetry.instrumentation.library.okhttp.v3_0.internal.OkHttp3Singletons$$ExternalSyntheticLambda1.intercept(R8$$SyntheticClass:0)
E/AndroidRuntime( 9291): at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
E/AndroidRuntime( 9291): at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201)
E/AndroidRuntime( 9291): at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:517)
E/AndroidRuntime( 9291): at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
E/AndroidRuntime( 9291): at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:644)
E/AndroidRuntime( 9291): at java.lang.Thread.run(Thread.java:1012)
E/AndroidRuntime( 9291): Caused by: java.lang.ClassNotFoundException: Didn't find class "io.opentelemetry.instrumentation.api.internal.SemconvStability" on path: DexPathList[[zip file "/data/app/~~ytm5u3m8tCcFs9-hsUwAjQ==/app.t2.availo-1LyuzmDaaTFr8EbTXOK-Jg==/base.apk"],nativeLibraryDirectories=[/data/app/~~ytm5u3m8tCcFs9-hsUwAjQ==/app.t2.availo-1LyuzmDaaTFr8EbTXOK-Jg==/lib/arm64, /data/app/~~ytm5u3m8tCcFs9-hsUwAjQ==/app.t2.availo-1LyuzmDaaTFr8EbTXOK-Jg==/base.apk!/lib/arm64-v8a, /system/lib64, /system_ext/lib64]]
E/AndroidRuntime( 9291): at dalvik.system.BaseDexClassLoader.findClass(BaseDexClassLoader.java:259)
E/AndroidRuntime( 9291): at java.lang.ClassLoader.loadClass(ClassLoader.java:379)
E/AndroidRuntime( 9291): at java.lang.ClassLoader.loadClass(ClassLoader.java:312)

@ahmedhkhalil
Copy link
Author

hi @LikeTheSalad above is commnet from my college show exception happening when we try to use new SDK v.18 that you just released

@LikeTheSalad
Copy link
Contributor

Hi @georgesamirmansour - Thank you for your feedback. I've created this new issue #321 to discuss more about the error that you've experienced, as it's a different one than the one being discussed here.

@LikeTheSalad
Copy link
Contributor

Dears, i tried below setup with grpc APM local end point is http and it's public access without any load balancer in middle , i am able to recieve session and service appear in Kibana services tab but when ever i put haproxy in middle to make ssl termination and send traffic to backend either grpc or http , no data is received on elastic and response from apm server is 400 and error message "invalid character " i suspect that there is issue from APM side image

Hi @ahmedhkhalil - It seems like there might be some issues with your HAProxy setup when communicating to the APM Server. The changes I've made in the new version 0.18.0 are only to fix the HTTP exporting, however, based on your message, it seems like this proxy issue is also happening when using HTTP as well, and it also seems like the Android agent exporting is working properly when there's no proxy set in between. Therefore, I'd suggest you would be to raise this issue in the APM Server repo since they will be able to provide better support for APM Server-related queries.

@ahmedhkhalil
Copy link
Author

@LikeTheSalad issue her is that EUM Data Is received successfully without any issue , as i mentioned issue may be on how Android Agent is sending data

@LikeTheSalad
Copy link
Contributor

@LikeTheSalad issue her is that EUM Data Is received successfully without any issue , as i mentioned issue may be on how Android Agent is sending data

Just to confirm, this is my understanding:

  • Elastic Agent -> Server = works ✅
  • Elastic Agent -> HAProxy -> Server = does not work ⛔️

Is that correct?

@ahmedhkhalil
Copy link
Author

Elastic Agent -> Server = works ✅ yes but HTTP traffic only
Elastic Agent -> HAProxy -> Server = does not work ⛔️ over https "from Agent to haproxy https and from haproxy to agent http"

@LikeTheSalad
Copy link
Contributor

I see, thank you. So it seems like you're trying to connect to HAProxy over HTTPS. I think we can verify a couple of things to make sure what's going on:

  • To discard issues with the protocol, I think it might be worth checking both GRPC and HTTP as export protocols from the agent, HTTP should work well starting from version 0.18.0 of the Elastic agent.
  • If you're getting logs that contain the following error: javax.net.ssl.SSLHandshakeException: java.security.cert.CertPathValidatorException or any SSL/TLS related issue, then it seems like your HAProxy instance might not be using trusted CAs by the Android OS.

Those are the possible issues that come to my mind, although there might be specific HAProxy-related configs that I'm not aware of which could be causing other issues. In any case, I think it's worth checking the above items in detail to discard potential root causes. Please bear in mind that the SSL/TLS configuration used by the Elastic Agent is the same as the one provided within the Android OS and there's no custom SSL config provided by the Agent. If you happen to have SSL cert issues, you should take a look at Android's official documentation on how to address them.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants