[Fleet] Provide the capability for Fleet to install integrations assets on a remote cluster

[nimarezainia]

When the remote Elasticsearch cluster is added as an output, install the integrations remotely on these clusters. Currently users have to do this manually and keep track. We can at minimum take care of installing the ingest pipelines on the remote clusters.

Implementation plan

• [Fleet] Add feature to synchronize integrations on remote ES clusters #192361
• [Fleet] Report failures of integration synchronization in Fleet UI #192363
• [Fleet] Add ability to view and remove integration synchronization of remote outputs in Fleet Settings #192362
• the ability to synchronize integrations should be under the Enterprise license

[nimarezainia]

Proposal:

Once the user enables the toggle switch to synchronize assets, go through the same workflow that currently exists in the integrations app that allows users to install an integration (integrations --> integration --> settings --> install integration assets) in order to install the assets on reote. To achieve this the Kibana API key will be required which the user has to provide OR we provide some information for the user to get the API). Is there anything else required from the user to install the assets remotely?

Allow the user to signify their intent to install assets remotely:

Allow the user to see at a global level which remote ES outputs the assets are being synchronized across.
The remote clusters can be deleted from this list (in which case the assets also need to be uninstalled). If the user needs to exapnd the set of remote clusters, they would need to go to the output itself and do that - mainly because other information is required for the installation (like api keys).

fyi: @kpollich based on our conversation.

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[juliaElastic]

I think this makes sense, providing an API key to the remote kibana should be enough to keep package assets in sync.
We would have to store this API key as an output secret in Remote ES output.

As for the implementation, we would have to add to the package install/update/uninstall logic to call the install API on the remote clusters where synchronising is enabled. We can use the /api/fleet/epm/packages/<pkgName>/<pkgVersion> API to install package assets without integration policies.

We should have a way of notifying users if the synchronisation failed for a package or kibana cannot connect to the remote kibana. Something similar to what we did for remote output health, where we regularly ping the remote output (in fleet-server) and store the result in a data stream for the UI to read from.

[nimarezainia]

I think this makes sense, providing an API key to the remote kibana should be enough to keep package assets in sync. We would have to store this API key as an output secret in Remote ES output.

As for the implementation, we would have to add to the package install/update/uninstall logic to call the install API on the remote clusters where synchronising is enabled. We can use the /api/fleet/epm/packages/<pkgName>/<pkgVersion> API to install package assets without integration policies.

We should have a way of notifying users if the synchronisation failed for a package or kibana cannot connect to the remote kibana. Something similar to what we did for remote output health, where we regularly ping the remote output (in fleet-server) and store the result in a data stream for the UI to read from.

Great. This logic exists in the integrations tab that allows asset installation/uninstallation.
Could we guide the user on where to get that kibana API key? The only way this could happen is for the user to get it from the console I believe.

[juliaElastic]

Great. This logic exists in the integrations tab that allows asset installation/uninstallation. Could we guide the user on where to get that kibana API key? The only way this could happen is for the user to get it from the console I believe.

They could get it from the Kibana UI as well, there is an API key creation feature under Security.

We could show a console request to copy out with minimum privileges required to install packages. They would require the manage_own_api_key role to create the API key.

 POST /_security/api_key
   {
     "name": "integration_sync", 
     "role_descriptors": {
       "integration_writer": { 
         "cluster": [],
        "indices":[],
        "applications": [{
           "application": "kibana-.kibana",
             "privileges": ["feature_fleet.all", "feature_fleetv2.agent_policies_all"],
             "resources": ["*"]
         }]
        }
     }
   }

# response
{
  "id": "yOr4VJEBDVUCDmN-I7gB",
  "name": "integration_sync",
  "api_key": "zuoRjN9xSv-4XiqJUlEc8Q",
  "encoded": "<redacted>"
}

Then the encoded value should be copied and pasted as the API key.

[nimarezainia]

Prefer this to be an action they can perform from within the remote ES flyout. (toggle an option to keep the integrations in synch)

[juliaElastic]

@nimarezainia Created implementation issues and moved this to Ready.