Sampling of APM Log Messages in Kafka Provider

[peterfouquet0001]

Is your feature request related to a problem?

We are using the java apm agent 1.43.0.
The agent logs every kafka receive message event into the apm logfile and produces to many load on the apm server.
The setting ELASTIC_APM_TRANSACTION_SAMPLE_RATE reduce only the attributes in the log event, but not the count of the log events.
We need an additional property for sampling apm kafka events to log only a promille of the kafka events to apm.

Describe the solution you'd like

We need an additional property for sampling apm kafka events to log only a promille of the kafka events to apm.

For example
ELASTIC_APM_SAMPLE_RATE_TRANSACTION_TYPE_MESSAGING=0.001 and ELASTIC_APM_SAMPLE_RATE_TRANSACTION_TYPE_REQUEST=0.1

Describe alternatives you've considered

Additional context

The problem is the count of log entries for transaction type messaging

[jackshirazi]

The agent logs every kafka receive message event into the apm logfile

Not by default - or if so it's a bug. Do you configure your app/agent to log these or are you saying the agent is logging all these by default?

[peterfouquet0001]

We are using the apm defaults.

[JonasKunz]

Are you referring to textual log messages being generated in the APM-agent logs?
Or are you actually referring to the number of transactions / spans being generated and sent to the APM-server?

[peterfouquet0001]

Im referring to the number of textual log messages, because or application processed more than 1.000.000.000 Kafka Message per Day, an we will only log a small sample (e.g. only every thousandth event) in apm.

[JonasKunz]

Unless you configured the apm agent log level to DEBUG or higher, there should be no log messages generated by the APM Agent. Can you show some example log messages?

[peterfouquet0001]

Sorry,
i told about the log messages created by the apm agent for each processed kafka message, see attached examle
apm.txt

[JonasKunz]

Okay, so you are seeing too many transactions. The problem is that your Kafka Transaction is not the root of the trace: It has a parent. ELASTIC_APM_TRANSACTION_SAMPLE_RATE only applies to root transactions and the resulting sampling rate is passed downstream via the kafka message.
You can do one of the following two things to fix this:

• Configure ELASTIC_APM_TRANSACTION_SAMPLE_RATE on the service sending your Kafka messages (assuming that that is the root of the trace)
• On your service receiving the messages set ELASTIC_APM_TRACE_CONTINUATION_STRATEGY to restart and configure ELASTIC_APM_TRANSACTION_SAMPLE_RATE. This will however cause distributed tracing to cease working.

[jackshirazi]

I'm assuming this resolved the concern, feel free to comment further if needed but I'm closing this for now