[NL-to-ESQL] Retrieve context from cluster / indices

[pgayvallet]

To improve the ESQL task, we could try to retrieve some context from the customer cluster.

We could for example fetch the list of all indices / aliases / datastreams the current user has access to, and then ask the LLM to guess, based on the user query, which one(s) are the most likely to be the target of the query. Then from there we could retrieve the mappings and provide it to the LLM, so that it has better knowledge of the existing fields.

The major downside here would be that we would be performing more LLM calls during the task, impacting the overall generation time. So we would probably need to make that opt-in by the caller.

One other option could be to let the caller pass the information regarding which index/index-patterns should be targeted, and we would then retrieve the mappings from that information. This could make a lot of sense for features like discover or visualize, when the application that would call the ESQL generation knows which index/indices the user is currently visualizing.

[elasticmachine]

Pinging @elastic/appex-ai-infra (Team:AI Infra)