[Response Ops][Alerting] Possible "experimental" ECS fields are not backwards compatible for alerts as data

[ymao1]

There was a recent PR to update the ECS field mappings from 8.6 to the latest that we noticed caused an error when updating existing alerts as data indices due to this error:

[2023-10-13T11:06:44.254-04:00][ERROR][plugins.alerting] ResponseError: illegal_argument_exception
	Root causes:
		illegal_argument_exception: can't merge a non-nested mapping [faas.trigger] with a nested mapping
    at KibanaTransport.request (/Users/ying/Code/kibana_prs/node_modules/@elastic/transport/src/Transport.ts:535:17)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)

It looks like there was an intentional change to the ECS faas.trigger field to change it from nested to object. Reading the issue, it seems that this change was approved because at the time the faas.trigger field was experimental. We've left a question to the ECS team to ask if there's a way to differentiate experimental ECS fields from non-experimental fields, but in the meantime, we are left with a lot of alerts as data indices in the wild with the experimental nested mapping type.

In order to move foward with the ECS update, which we've seen some customer requests for have been asking for, we should figure out how to handle this

Options

• Omit this problematic field from the mapping - this is the easiest but will probably cause issues in the future
• Update the code to either skip updating the underlying mapping of existing concrete indices entirely or skip if there's a failure, with the understanding that the AAD indices should rollover and get the updated mapping in the future. This might cause mapping conflicts across indices in an alias or data stream. We need to understand what the behavior is in the alerts table if we go this route
• Investigate whether this problem would go away if we switched to dynamic mappings for ECS as suggested here: Replace ECS static mapping with ECS dynamic mapping for AAD indices #168497

[elasticmachine]

Pinging @elastic/response-ops (Team:ResponseOps)

[mikecote]

After discussing with @ymao1 @pmuellr @kobelb @doakalexi we will go with the following option:

Omit this problematic field from the mapping - this is the easiest but will probably cause issues in the future

While we omit this field, we should also omit all other ECS experimental fields to prevent this situation for now.

[ymao1]

According to the ECS RFC stages doc, any RFC in stage 3 is considered GA.

Here are the fields from RFCs in generated/ecs_flat.yml that are not yet in stage 3:

https://github.com/elastic/ecs/blob/main/rfcs/text/0002-rfc-environment.md - marked as beta
https://github.com/elastic/ecs/blob/main/rfcs/text/0009-data_stream-fields.md - not marked as beta
https://github.com/elastic/ecs/blob/main/rfcs/text/0015-create-file-elf.md - not marked as beta
https://github.com/elastic/ecs/blob/main/rfcs/text/0027-faas-fields.md - not marked as beta
https://github.com/elastic/ecs/blob/main/rfcs/text/0034-device-fields.md - not marked as beta
https://github.com/elastic/ecs/blob/main/rfcs/text/0035-tty-output.md - marked as beta
https://github.com/elastic/ecs/blob/main/rfcs/text/0037-host-metrics.md - 2 fields added, not marked as beta
https://github.com/elastic/ecs/blob/main/rfcs/text/0040-volume-device.md - not marked as beta

Seems like fields make it into the normal (not experimental) generated ECS file when they reach stage 2, where iterative but not breaking changes might be expected but they're still only recommended for experimental usage. There is a beta designation` but doesn't look like all fields in stage 2 mark themselves as beta.

[ymao1]

Opened an issue in the ECS repo: elastic/ecs#2294