UI for adding fields aliases for ECS

[ruflin]

Idea Description

With the introduction of ECS users will want to map their existing data to ECS. Assuming a user has field source_ip but ECS is source.ip, he will want to create an alias from source.ip to source_ip. With the Elasticsearch API it's possible to append mapping to existing indices (see example on the bottom).

As normally not only 1 field has to be mapped, I would hope for an easy way to specify the alias name, the path (field) it should point to and the indices it should be applied to. I could imagine this as a table with two columns for the field and alias name. On the bottom or top a field for the index names.

Elasticsearch commands

The necessary commands in Elasticsearch to apply alias b to field a across several indices looks today as following:

PUT test-1
{
  "mappings": {
    "_doc": {
      "properties": {
        "a": {
          "type": "keyword"
        }
      }
    }
  }
}

PUT test-2
{
  "mappings": {
    "_doc": {
      "properties": {
        "a": {
          "type": "keyword"
        }
      }
    }
  }
}

PUT foo
{
  "mappings": {
    "_doc": {
      "properties": {
        "a": {
          "type": "keyword"
        }
      }
    }
  }
}

PUT test-*,foo/_mapping/_doc 
{
  "properties": {
    "b": {
      "type": "alias",
      "path": "a"
    }
  }
}

Additional ideas

These are additional ideas which could make this feature even nicer:

• Upload of a csv file with the mapping inside is possible to fill the table. This would allow us to provide predefined migration files for certain tools where we know the old format.
• Export / save a migration to be reused later (could be an export to csv)
• Allow do delete aliases (I think this we require changes in ES as it's not possible today)

[elasticmachine]

Pinging @elastic/kibana-operations

[tylersmalley]

@ruflin we're working on trying to get this in for 6.6 and it would be part of the 7.0 Upgrade Assistant. Our goal is for plugins to be able to extend what fields need to be aliased - then present a UI to the user for effected indices and allow them click a button to add the aliases.

Do you have an example of all the mappings you're wanting to perform?

[ruflin]

A concrete list of aliases we will need can be found here: https://github.com/elastic/beats/pull/9283/files#diff-3a5cfd1aae7e3e54c908e482c7d7af26R63

To clarify: My request for this feature is broader then just the upgrade assistant but also have it available as an independent tool. Assuming someone ingest Bro logs in JSON format with fluentd and wants to map them to ECS, he could use this tool also in 7.2.

[tylersmalley]

@ruflin, since Beats decided not to go down this path - can this issue now be closed?

[ruflin]

This feature request is not specific to Beats but is broader related to alias management especially in the context of ECS. A large portion of the data that comes into Elasticsearch in the near future will not be in ECS but mapping it to ECS would be very powerful. This is where this feature comes into play. I would still hope this feature is happening :-)

[tylersmalley]

Ok, since this is more of an Elasticsearch management thing - I am going to move this issue over to that team for triage.

[elasticmachine]

Pinging @elastic/es-ui

[cjcenizal]

@ruflin I created #57061 to track a feature for allowing users to edit an index, including its mappings. This will be a step towards the features envisaged in this issue.

[elasticmachine]

Pinging @elastic/kibana-management (Team:Kibana Management)