"Output Events Rate" in stack monitoring is always zero

[axw]

APM Server version (apm-server version): 8.3.0-BC4

Description of the problem including expected versus actual behavior:

"Output Events Rate" in stack monitoring is always zero.

Steps to reproduce:

1. Start 8.3.0-BC4 with stack monitoring enabled.
2. Send some events, check that they show up in the APM UI.
3. Navigate to stack monitoring, observe the "Output Events Rate" chart is always reporting zero.

[axw]

Hmm, I just reconfigured the integration with expvar enabled, and now it's working. Maybe there's race condition?

[axw]

Happened again after upgrading from 8.2.3 to 8.3.0-BC4. Initially the output was zero, after reconfiguring the integration (this time changing the event rate limit), the output went non-zero.

[axw]

This is apparently still an issue, at least in system tests, as seen here:

https://apm-ci.elastic.co/blue/organizations/jenkins/apm-server%2Fapm-server-mbp%2FPR-9014/detail/PR-9014/1/pipeline/

[lahsivjar]

I haven't been able to reproduce this exact error. However, due to the way our instrumentation works it is possible that after a reload event the old modelindexer is still receiving data while the instrumentation has moved to the new modelindexer. This is due to the fact that we wait for the old modelindexer to gracefully shutdown however, we switch the monitoring to new modelindexer before the old one exits.

The above will result in the instrumentation data to report 0 until the old indexer shuts down.

[simitt]

Moving this to backlog since we haven't spend more time recently to track this down.

[tegenterter]

It appears that this bug lead up to an incident (https://github.com/elastic/cloud/issues/110723) and should be prioritized

[simitt]

Moved it into the 8.7 milestone again to be picked up and verified if this is still a bug in current versions.

[axw]

I don't recall if this has already been ruled out, but I realise now that I never wrote down on this issue a possible contributing factor: every time we reconfigure the server, we create a new libbeat monitoring registry:

apm-server/internal/beater/beater.go

Line 628 in 32a167b

libbeatMonitoringRegistry := monitoring.Default.NewRegistry("libbeat")

[lahsivjar]

Hmm, nice catch. I don't remember any conversation around this so I think this hasn't been ruled out.

[endorama]

I was looking at this today and I have 2 questions:

1. how can I send some test data?
2. my first hint at this would be to try reusing the libbeatMonitoringRegistry instead of creating it anew like it is done for the output registry

   apm-server/internal/beater/beater.go

   Lines 634 to 639 in 32a167b

   	outputRegistry := stateRegistry.GetRegistry("output")
   	if outputRegistry != nil {
   	outputRegistry.Clear()
   	} else {
   	outputRegistry = stateRegistry.NewRegistry("output")
   	}

   What do you think?

[axw]

how can I send some test data?

You could use https://github.com/elastic/apm-server/tree/main/systemtest/cmd/sendotlp to send test data to APM Server

my first hint at this would be to try reusing the libbeatMonitoringRegistry instead of creating it anew like it is done for the output registry

You could try, but I don't think that will work. There are assumptions about there being a 1:1 relationship between metrics and outputs, e.g. here:

apm-server/internal/beater/beater.go

Lines 688 to 696 in 9880622

	// Install our own libbeat-compatible metrics callback which uses the docappender stats.
	// All the metrics below are required to be reported to be able to display all relevant
	// fields in the Stack Monitoring UI.
	monitoring.NewFunc(libbeatMonitoringRegistry, "output.write", func(_ monitoring.Mode, v monitoring.Visitor) {
	v.OnRegistryStart()
	defer v.OnRegistryFinished()
	v.OnKey("bytes")
	v.OnInt(appender.Stats().BytesTotal)
	})

[carsonip]

#14337 may be related in a sense that reloads (which may take a long time) need to be handled carefully.