[SLO] Exclude stale slos from healthy count on overview

[justinkambic]

Summary

Resolves #198911.

The result is achieved by nesting a new filter agg inside the existing HEALTHY agg to remove any stale SLOs from the ultimate result.

This required a modification of the parsing code on the ES response to include a new not_stale key. The original success total is preserved in the doc_count of that agg, but is no longer referenced.

The filter for the not_stale agg I have added is the logical inverse of the filter we're using to determine stale SLOs:

{
  "range": {
    "summaryUpdatedAt": {
      "gte": "now-48h"
    }
  }
}

Reviewer note: I also changed the spelling of a UI component, should be a completely transparent change.

Example

Before

This is my local running on main:

After

This is my local running on this PR branch:

Proof query works

You can replicate these results by including a similar agg on a query against SLO data. I added a terms agg to the stale agg to determine how many SLOs I need to remove. The number of HEALTHY SLOs showing up in stale should match the difference between the total doc_count from healthy and the doc_count in the not_stale sub-aggregation.

Query

You can run this example aggs:

{
  "aggs": {
    "stale": {
      "filter": {
        "range": {
          "summaryUpdatedAt": {
            "lt": "now-48h"
          }
        }
      },
      "aggs": {
        "by_status": {
          "terms": {
            "field": "status"
          }
        }
      }
    },
    "healthy": {
      "filter": {
        "term": {
          "status": "HEALTHY"
        }
      },
      "aggs": {
        "not_stale": {
          "filter": {
            "range": {
              "summaryUpdatedAt": {
                "gte": "now-48h"
              }
            }
          }
        }
      }
    }
  }
}

Relevant output

Here's a subset of my example query output. You can see that stale.by_status.buckets[1] contains a total of 2 docs, which is the difference between healthy.doc_count and healthy.not_stale.doc_count.

{
  "stale": {
    "doc_count": 7,
    "by_status": {
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 0,
      "buckets": [
        {
          "key": "VIOLATED",
          "doc_count": 5
        },
        {
          "key": "HEALTHY",
          "doc_count": 2
        }
      ]
    }
  },
  "healthy": {
    "doc_count": 9,
    "not_stale": {
      "doc_count": 7
    }
  }
}

[elasticmachine]

Pinging @elastic/obs-ux-management-team (Team:obs-ux-management)

[github-actions]

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

• /oblt-deploy : Deploy a Kibana instance using the Observability test environments.
• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

[shahzad31]

now that i am thinking, i think same should be subtracted from degrading and violated SLOs.

[justinkambic]

We talked about this offline and agreed we can apply this filtering further up rather than adding sub-aggregations for all of the non-stale filters. I'll ping when this is done.

[kdelemme]

👍🏻

[kdelemme]

I think we need to keep using the settings, and also if we can double check the usage of worst. I have the feeling it is not used and could be 🔪
Otherwise, looks good to me.

[kdelemme]

Just one thing to cleanup but otherwise 👍🏻

[elasticmachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 92033ef
• Kibana Serverless Image: docker.elastic.co/kibana-ci/kibana-serverless:pr-201027-92033efbd663

Failed CI Steps

• FTR Configs #9

Test Failures

• [job] [logs] FTR Configs #9 / management Index patterns on aliases discover verify hits should be able to discover and verify no of hits for alias2

Metrics [docs]

✅ unchanged

History

• 💔 Build #254175 failed 566e2ac
• 💚 Build #254089 succeeded 060ebcc
• 💚 Build #253710 succeeded 1186417
• 💚 Build #253250 succeeded 1bfa7e7
• 💔 Build #252909 failed 5ad2aa9

cc @kdelemme @justinkambic

[kibanamachine]

Starting backport for target branches: 8.17, 8.x

https://github.com/elastic/kibana/actions/runs/12034819890

[kibanamachine]

💚 All backports created successfully

Status	Branch	Result
✅	8.17
✅	8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation