[Stack Monitoring] Review diff between documented and shipped fields

[miltonhultgren]

Both #143337 and elastic/beats#33104 have highlighted that there are differences between which fields we document and which fields we actually ship.
So far it seems that the drift is due to misses in the copy-to-event schemas used in each metricset.

As a starting point, we should go through each metricset in each module we own and document the difference in fields documents and fields shipped.
Then create follow up tickets to bring these into alignment.

[elasticmachine]

Pinging @elastic/infra-monitoring-ui (Team:Infra Monitoring UI)

[matschaffer]

@klacabane called out some details in this comment about his experiences doing mapping alignment.

[matschaffer]

We might want to pull this into scope for #120415 or whatever the follow-up epics for packages/integrations will be.

[klacabane]

The tooling around integrations enable robust and testable mappings so it makes a great candidate to be the source of truth from where we can sync other mappings (metrics-* > .monitoring-* > metricbeat-*)

A mechanism that detects changes between mappings and automatically corrects them could be useful. Simple version of that mechanism could be code running against the local repositories containing static mappings (integrations, elasticsearch and metricbeat). It would parse the integrations mappings and sync the other repos with the deltas found. For that to happen we'll need to build the mappings from the package's yml fields definitions. Given packages have specification that allows referencing to dependencies for fields definition, it might not be that easy (and wise) to do it ourself, so unless that mapping building logic can be reused it'd be probably better to get the mappings from a running elasticsearch.

One option to extract the mappings is the esArchiver. The esArchiver format is also used to load data for integrations tests. Right now most e2e tests run against mappings cached in the kibana repo, and these mappings are naturally outdated at this point. This is out of this task scope, but I'm thinking that whenever we update the mappings with this process we could also replace them in the automated test suites given we have a known directory structure

[miltonhultgren]

I just want to clarify also that in this case there wasn't any issue with the mappings but rather the document shipped.

So while I totally agree that it would be great to have tools to detect drifts in mappings between collection modes and have contract tests for those or automatically update those mappings/test resources, what we also need is some way to check "does this document conform to the expected mapping within this collection mode".

[klacabane]

package testing answer that question partially by validating that if a field exists in a document it should have a corresponding mapping defined, but not the other way around. So I guess an audit of every API hit by these packages is one suggestion ? We'd have to consider that some fields are only returned under certain condition/architecture.

API contracts from the producer would be lovely but considering we're hitting unsupported/unofficial apis it's fair that they don't exist

[miltonhultgren]

To add just one more drop of challenge, we tend to also translate the API response and most misses are in that translation.
So the mapping is correct, and the API "config" is correct so the right response is coming from the API but our event transformation logic is bugged.

[matschaffer]

As @crespocarlos mentioned in retro yesterday, it could be that elastic-package system tests are the best route out of these sorts of issues. Of course, they mainly apply only to agent/integration flows. But perhaps with that in place we can automate the other mappings from the stack tests as a source-of-truth.