[ML] Anomaly Explorer: Display markers for scheduled events in distribution type anomaly charts

[rbrtj]

Summary

Fix for #129304

Previously, for distribution type charts, we displayed calendar event markers only for anomalous data points. The changes improve the display of event markers for such chart types, including showing calendar event markers even when there is no underlying data point.

Scenario	Before	After
Rare chart
Population chart
Single metric chart (no difference)

[elasticmachine]

Pinging @elastic/ml-ui (:ml)

[rbrtj]

It is a small PR, but the changes may have some implications, so I've tagged 3 people.

[walterra]

Nit: Maybe we can think of another name for this flag since this isn't really about fake or dummy data but just about the schedule event marker. Something like isScheduledEventMarkerWithoutData but less wordy 😅 . Maybe isScheduledEventOnly/isScheduleEventMarker?

[rbrtj]

This property isn't really needed so I got rid of it 👍

[walterra]

For rare jobs with just a few entities we show labels and with the update in this PR the scheduled events end up as labeled undefined at the bottom of the chart. To fix this you will probably need to extend the marker data with the fieldname/value of the entity. I wonder if you add that if the custom sorting will still be necessary?

[walterra]

The above is a bit tricky to reproduce, I wasn't able to come up with a job that produced these results with Kibana sample data. I tweak the logs sample dataset to come up with these results. Here's what you need to do in Kibana Dev Console:

# This returns docs that are part of rare log messages in the sample data set
GET kibana_sample_data_logs/_search?q=cgi-bin

# Grab then one of the returned docs and index them another time but just with a different response code.
# Make sure to grab your own doc from the result above because the dataset creates dynamic time stamps.
PUT kibana_sample_data_logs/_doc/asdf406?op_type=create
{
          "agent": "Mozilla/5.0 (X11; Linux i686) AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.50 Safari/534.24",
          "bytes": 1831,
          "clientip": "30.156.16.164",
          "extension": "",
          "geo": {
            "srcdest": "US:IN",
            "src": "US",
            "dest": "IN",
            "coordinates": {
              "lat": 55.53741389,
              "lon": -132.3975144
            }
          },
          "host": "elastic-elastic-elastic.org",
          "index": "kibana_sample_data_logs",
          "ip": "30.156.16.163",
          "machine": {
            "ram": 9663676416,
            "os": "win xp"
          },
          "memory": 73240,
          "message": "30.156.16.163 - - [2018-09-01T12:44:34.756Z] \"GET /cgi-bin/mj_wwwusr HTTP/1.1\" 404 1831 \"-\" \"Mozilla/5.0 (X11; Linux i686) AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.50 Safari/534.24\"",
          "phpmemory": 73240,
          "referer": "http://www.elastic-elastic-elastic.com/success/timothy-l-kopra",
          "request": "/cgi-bin/mj_wwwusr",
          "response": 406,
          "tags": [
            "success",
            "info"
          ],
          "@timestamp": "2024-08-10T12:44:34.756Z",
          "url": "https://elastic-elastic-elastic.org/cgi-bin/mj_wwwusr",
          "utc_time": "2024-08-10T12:44:34.756Z",
          "event": {
            "dataset": "sample_web_logs"
          },
          "bytes_gauge": 1831,
          "bytes_counter": 79545760
        }

To identify the time to be used for the scheduled event go to log rate analysis and then create a scheduled event some days before the spike:

Again, you will have different timestamps here because of the dynamically generated dataset.

Once the calendar with the scheduled event is set up, you can create a rare job on field response.keyword to reproduce.

[rbrtj]

For rare jobs with just a few entities we show labels and with the update in this PR the scheduled events end up as labeled undefined at the bottom of the chart. To fix this you will probably need to extend the marker data with the fieldname/value of the entity. I wonder if you add that if the custom sorting will still be necessary?

I've added a const entity value for markers data, also hidden the Y-axis label for markers, as the tooltip already displays events details.
Custom sorting is still needed tho, as by default we're sorting by data values length, and the event markers should be displayed at the bottom IMO

[peteharverson]

I am still seeing undefined appear in the axis labels for a rare job:

You can reproduce with the filebeat_ecs data, using the 'rare' job wizard to create a job with this config:

    "bucket_span": "3h",
    "detectors": [
      {
        "detector_description": "rare by \"http.request.method\"",
        "function": "rare",
        "by_field_name": "http.request.method",
        "detector_index": 0
      }
    ],
    "influencers": [
      "http.request.method"
    ],

[rbrtj]

Hmm, I can't seem to reproduce it to have an undefined label with the same dataset and job config:

[peteharverson]

Retested @rbrtj and I can no longer reproduce this with the undefined label!

[peteharverson]

Tested latest changes and LGTM

[walterra]

LGTM, great work!

[kibana-ci]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 6932c44

Failed CI Steps

• Jest Tests #1

Test Failures

• [job] [logs] Jest Tests #1 / Responder offline callout should be visible when agent type is sentinel_one and host is offline

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
ml	2036	2037	+1

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
ml	4.6MB	4.6MB	+235.0B

History

• 💔 Build #234467 failed e60fc2a
• 💛 Build #233934 was flaky 8f891b5
• 💛 Build #233732 was flaky 01c4d41
• 💚 Build #232796 succeeded dacd7d0

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @rbrtj

[kibanamachine]

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation