[Obs Logs Explorer / Logs+] Move Logs ML jobs from the Infra plugin

[Kerry350]

⚠️ In a draft format, and requires further discussion, opened based on requests.

Summary

At the moment the code for creating the Logs ML jobs (for anomaly detection and categorisation) and the UIs to visualise these live inside of the Infra plugin, and the infra plugin / LogsUI features are disabled within the serverless context. This issue covers moving the job creation (flyout components) and relevant APIs to another plugin that is accessible within serverless. logs_shared could potentially house these components and APIs. The Observability Logs Explorer could then utilise these components to offer job creation in the explorer context. Or we could even create a new plugin / package for these. Logs UI would also have to consume these from the new location (or potentially we add some duplication, or maybe if things move to the Obs Log Explorer we just remove them from the LogsUI).

We would need to give consideration to where to visualise these results (previously LogsUI had two dedicated pages), do we move these pages to the Observability Log Explorer? Does AIOps somehow take over this for serverless?

Code locations

• The results APIs live here
• The validation APIs live here
• Components exist here with log_analysis prefix, here for the UIs - log entry rate and log entry categories
• Types and some helpers in common
• Hooks for interacting with ML, handling the modules etc live here

Considerations

• In infra some of the module code for handling jobs is shared between Logs and Metrics. We should be careful to ensure nothing breaks here on the generic level.
• Things like "job is outdated" checks rely on the "source configuration" (Log View) settings that are embedded in the job at creation time. The Log View saved object is disabled in serverless, however we still register a default. Again, care should be taken here to ensure this functionality works, or is removed, or is based on environment.

[ruflin]

Key is to make sure the logs jobs work with Logs Explorer and don't have a dependency on the Logs UI.

There are also the ml modules for logs: https://github.com/elastic/kibana/tree/main/x-pack/plugins/ml/server/models/data_recognizer/modules It must be ensured either the existing or new ones work based on data from the Logs Explorer. Currently it seems no patterns are inside the modules.

[Kerry350]

@ruflin

There are also the ml modules for logs: https://github.com/elastic/kibana/tree/main/x-pack/plugins/ml/server/models/data_recognizer/modules

Yeah, the logs_ui_analysis and logs_ui_categories are the relevant jobs.

It must be ensured either the existing or new ones work based on data from the Logs Explorer. Currently it seems no patterns are inside the modules.

Yeah, I think this is where most consideration is needed. There are no hardcoded patterns in the modules because in the Logs UI we determined which indices to use based on the source configuration / log view from the settings page, and then injected these when calling the setup APIs. We then also stored these under custom_settings in the job to ensure we could check when something was outdated etc. In the Logs UI there was this nice 1:1 mapping between the settings and the jobs, in the Log Explorer I'm not 100% sure how this would look. Of course we have a "selection" within the Log Explorer, but I'm not sure if we want to be able to create jobs just based on selections.

For context, this is what actually gets sent when setting up a module from the Logs UI:

Here indexPatternName comes from the Logs UI settings, and then there's a filter as I selected a specific dataset.

We'd need to figure out how to best map this to Explorer (or use something else entirely).

[weltenwort]

@Kerry350 thanks for writing this up. I agree that it would likely be possible to extract the job creation fly-outs and result pages from the infra plugin and re-use them in both places, potentially backed by slightly different ml modules.

Where we extract them to exactly depends on whether we want the ml modules to be only available in the context of the observability solution or also outside. @ruflin, what's your take on this? Is there chance that we want the job definitions to be available in the other serverless project types?

Another thing to consider is that we now have an ad-hoc categorization aggregation workflow for text fields right in the Log Explorer and Discover. So do we still need to carry over the categorization job type and results page or is that feature sufficient?

[mwtyang]

Another option is to create and add these O11y anomaly detection jobs as preconfigured jobs in the ML menu link. See below for the rough mockup.

cc @arisonl @grabowskit

[ruflin]

Where we extract them to exactly depends on whether we want the ml modules to be only available in the context of the observability solution or also outside. @ruflin, what's your take on this? Is there chance that we want the job definitions to be available in the other serverless project types?

I expect this to also show up in the security projects.

Another thing to consider is that we now have an ad-hoc categorization aggregation workflow for text fields right in the Log Explorer and Discover. So do we still need to carry over the categorization job type and results page or is that feature sufficient?

Didn't fully get this.

index pattern

Would it be possible that we start with logs-*-* as a hardcoded index pattern?

Anomaly detection

I don't think it should be either / or. If we have the jobs, show them in all the places.

[jgowdyelastic]

As an FYI, as it's probably relevant to this discussion, I'm currently working on a PR to allow us to add tags to a module.
In serverless this will allow us to only show modules which are relevant to the current project. E.g. Security modules won't appear in Observability.

[weltenwort]

That would be very useful, thanks @jgowdyelastic!

[weltenwort]

@ruflin, thanks for the response. I have some follow-up or clarification questions 🤔

I expect this to also show up in the security projects.

Do you mean the just the modules? Because there's no log explorer in the security project from where we could offer the set-up fly-outs or the result pages.

Didn't fully get this.

This was about the categorization job. When we implemented this in the Logs UI, the only way to categorize message was continuously as a "side-effect" of anomaly detection jobs. Nowadays we have the "log pattern analysis" feature in the unified field list, which is backed by the new categories aggregation. The question was whether it's worth migrating the job setup and results UI for that categorization if we already have inherited a similar feature from Discover.

Would it be possible that we start with logs-- as a hardcoded index pattern?

Yes, that would be possible.

[ruflin]

Do you mean the just the modules? Because there's no log explorer in the security project from where we could offer the set-up fly-outs or the result pages.

Not now, but I expect this to change. With the change from James we will be able to enable it for security when we get there.

The question was whether it's worth migrating the job setup and results UI for that categorization if we already have inherited a similar feature from Discover.

Sounds like migration doesn't make sense in this case but it would still be good to improve the UX for logs around it.

[weltenwort]

Sounds like migration doesn't make sense in this case but it would still be good to improve the UX for logs around it.

We could try to expose a trigger to analyze the message field more prominently in the UI.

[weltenwort]

Not now, but I expect this to change. With the change from James we will be able to enable it for security when we get there.

Sorry for not expressing myself clearly enough. It's trivial to also expose the ML modules in the security project. But I'm wondering how the custom set-up fly-out and result page would work in security.

With "I expect this to change" do you mean that the full log explorer might become available in security?

[ruflin]

With "I expect this to change" do you mean that the full log explorer might become available in security?

Yes, eventually.

[weltenwort]

Thanks, that is a very important constraint, which we need to take into account in many places. So far it was built under the assumption that it's observability-specific.

[elasticmachine]

Pinging @elastic/ml-ui (:ml)