[EPM] Dynamic mapping not being created for fields.yml entry

[andrewkroh]

Kibana version: 8.10.2

Describe the bug:

Fleet is not creating dynamic_templates entries in the component template for all fields that are declared as dynamic fields (using a wildcard name and type: object). The problem appears to only occur when two fields are declared that seem to have a hierarchical relationship.

I suspect that Fleet is not making a distinction between dynamic and static field mappings when considering what fields can be removed from the template because they are non-leaf fields. That logic should not apply to dynamic fields. In this case we saw Item.* be dropped and only Item.*.* landed in the final template.

Steps to reproduce:

Create a package with these fields. Install it. Then check the Elasticsearch component template.

# fields.yml
---
   - name: o365.audit.Item.*
      type: object
      object_type: keyword
      object_type_mapping_type: '*'
   - name: o365.audit.Item.*.*
      type: object
      object_type: keyword
      object_type_mapping_type: '*'

Generated template:

"dynamic_templates": [
        {
          "o365.audit.Item.*.*": {
            "path_match": "o365.audit.Item.*.*",
            "mapping": {
              "type": "keyword"
            }
          }
        },
]

Expected behavior:

"dynamic_templates": [
        {
          "o365.audit.Item.*": {
            "path_match": "o365.audit.Item.*",
            "mapping": {
              "type": "keyword"
            }
          }
        },
        {
          "o365.audit.Item.*.*": {
            "path_match": "o365.audit.Item.*.*",
            "mapping": {
              "type": "keyword"
            }
          }
        },
]

Any additional context:

• o365: fix mappings for dynamically mapped fields integrations#7988

[elasticmachine]

Pinging @elastic/fleet (Feature:EPM)

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[nchaulet]

@andrewkroh What the goal of the Item.* mapping, it seems to me it will result in a mapping where we will not be able to write to Item.*.* the Item.* object_type seems not correct to me

For example when testing those mappings

I think with that recent change #169981 made by @jsoriano only the o365.audit.Item.*.* mapping should be needed.

Testing it this give the following dynamic_templates

 "dynamic_templates": [
        {
          "o365.audit.Item.*.*": {
            "mapping": {
              "type": "keyword"
            },
            "match_mapping_type": "*",
            "path_match": "o365.audit.Item.*.*"
          }
        },
        {
          "o365.audit.Item.*": {
            "mapping": {
              "type": "object",
              "dynamic": true
            },
            "match_mapping_type": "object",
            "path_match": "o365.audit.Item.*"
          }
        }
      ]

[andrewkroh]

I think you are right. I don't see a need for both o365.audit.Item.* and o365.audit.Item.*.* unless o3365.audit.Item had subobjects: false.

[nchaulet]

@andrewkroh it do not seems o3365.audit.Item has subobjects: false should we close that issue and create a new one to fix the o365 issue?