[Stack Monitoring] CPU Usage duration for alerting is not correctly used in the elastic query

[zazabe]

Kibana version: 7.12.0 / 7.15.1

Elasticsearch version: 7.12.0 / 7.15.1

Server OS version: Elastic Cloud on AWS

Original install method (e.g. download page, yum, from source, etc.): Elastic Cloud on AWS

Describe the bug:

We customized the CPU Usage period and threshold to 50% over 1 hour, but the alert is still triggered when the CPU usage is above 50% for less than 1h.

• CPU Alert config
• Node CPU utilization during this period
• Alert in slack (UTC+2)

Expected behavior:

The CPU Usage alert should trigger only if the CPU usage is above 50% in average for 1 hour.

In depth:

I executed the query produced by fetch_cpu_usage_node_stats manually, I can use either a period of 5min or 1h, the result is still 58% of CPU usage:

• query for 1h = 58.69%
• query for 5min = 58.98%

The query should be redesigned, because the CPU alert is not correctly calculating the CPU usage for different period based on derivative values when monitoring.ui.container.elasticsearch.enabled = true.

The root-cause is the date_histogram aggregation, for example:

• a CPU alert triggering if the usage is >50% for 1 hour is executed at 2021-10-26 08:00:10.000
• there are 2 items in the histogram: 2021-10-26 07:00:00.000 and 2021-10-26 08:00:00.000
• only the last item is used by the alert algorithm, so from 08:00:00 to 08:00:10
  In this case, if the CPU usage is above the threshold just for few seconds at around 08:00, the alert will trigger.

Another problem with this query: it's not returning derivative values if the time-range is set at a rounded second, probably because there's only one item in the date_histogram aggregation.

[zazabe]

@chrisronline fyi

Tested with kibana 7.15.1 / elasticsearch 7.15.1, same problem:

• CPU Alert config
• Node CPU utilization during this period
  Alert in slack (UTC+2)

[elasticmachine]

Pinging @elastic/infra-monitoring-ui (Team:Infra Monitoring UI)

[chrisronline]

Tagging the proper team here, I'm no longer working on this.

[simianhacker]

@zazabe Thanks for filling this issue and detailing the derivative issues. Are you running in a containerized environment?

[zazabe]

Are you running in a containerized environment?

Yes, we are using Elastic Cloud on AWS.

---

BTW, we solved the problem by using a watcher, it might be useful to fix the problem:

CPU usage watcher

{
  "trigger": {
    "schedule": {
      "interval": "5m"
    }
  },
  "input": {
    "search": {
      "request": {
        "search_type": "query_then_fetch",
        "indices": [
          ".monitoring-es-*"
        ],
        "rest_total_hits_as_int": true,
        "body": {
          "aggs": {
            "nodes": {
              "aggs": {
                "cpu_usage": {
                  "avg_bucket": {
                    "buckets_path": "histo>container_cpu_usage",
                    "gap_policy": "skip"
                  }
                },
                "histo": {
                  "aggs": {
                    "avg_cfs_quota_micros": {
                      "avg": {
                        "field": "node_stats.os.cgroup.cpu.cfs_quota_micros"
                      }
                    },
                    "container_cpu_usage": {
                      "bucket_script": {
                        "buckets_path": {
                          "avg_cfs_quota_micros": "avg_cfs_quota_micros.value",
                          "delta_cpuacct_usage_nanos": "delta_cpuacct_usage_nanos.normalized_value",
                          "delta_number_of_elapsed_periods": "delta_number_of_elapsed_periods.normalized_value"
                        },
                        "script": "(params.delta_cpuacct_usage_nanos / ( params.delta_number_of_elapsed_periods * params.avg_cfs_quota_micros * 1000))"
                      }
                    },
                    "delta_cpuacct_usage_nanos": {
                      "derivative": {
                        "buckets_path": "max_cpuacct_usage_nanos",
                        "gap_policy": "skip",
                        "unit": "1s"
                      }
                    },
                    "delta_number_of_elapsed_periods": {
                      "derivative": {
                        "buckets_path": "max_number_of_elapsed_periods",
                        "gap_policy": "skip",
                        "unit": "1s"
                      }
                    },
                    "max_cpuacct_usage_nanos": {
                      "max": {
                        "field": "node_stats.os.cgroup.cpuacct.usage_nanos"
                      }
                    },
                    "max_number_of_elapsed_periods": {
                      "max": {
                        "field": "node_stats.os.cgroup.cpu.stat.number_of_elapsed_periods"
                      }
                    }
                  },
                  "date_histogram": {
                    "field": "timestamp",
                    "fixed_interval": "1m"
                  }
                }
              },
              "terms": {
                "field": "source_node.name"
              }
            }
          },
          "query": {
            "bool": {
              "filter": [
                {
                  "range": {
                    "timestamp": {
                      "gte": "{{ctx.trigger.scheduled_time}}||-{{ctx.metadata.period}}"
                    }
                  }
                },
                {
                  "term": {
                    "type": "node_stats"
                  }
                }
              ]
            }
          },
          "size": 0
        }
      }
    }
  },
  "condition": {
    "script": {
      "source": "return ctx.payload.aggregations.nodes.buckets.stream().anyMatch(it -> it.cpu_usage.value > ctx.metadata.cpu_usage_max);",
      "lang": "painless"
    }
  },
  "actions": {
    "slack_1": {
      "slack": {
        "message": {
          "from": "kibana-production",
          "to": [
            "#kibana-production"
          ],
          "text": """CPU usage above {{ctx.payload.cpu_usage_max_percent}}% during the last {{ctx.metadata.period}} for node(s): 
{{#ctx.payload.nodes}}- {{node_name}}: {{cpu_usage_percent}}%
{{/ctx.payload.nodes}}"""
        }
      }
    }
  },
  "metadata": {
    "period": "60m",
    "cpu_usage_max": 0.5
  },
  "transform": {
    "script": {
      "source": "['cpu_usage_max_percent': ctx.metadata.cpu_usage_max * 100, 'nodes': ctx.payload.aggregations.nodes.buckets.stream().filter(it -> it.cpu_usage.value > ctx.metadata.cpu_usage_max).map(it -> ['node_name':it.key,'cpu_usage_percent':Math.round(it.cpu_usage.value * 100)]).collect(Collectors.toList())];",
      "lang": "painless"
    }
  }
}

[simianhacker]

One problem with this rule is that the date_histogram interval is the same size as the time range of the query. This will give you inconsistent results due to how the date_histogram bucketing works and when the query runs. Sometimes you will get enough buckets for the derivatives and other times you won't. The other issue is that the date_histogram will create buckets at the end of the time range that go past the data available so that last bucket is not a complete representation of the whole minute.

A better way to write this would be to query for 2x the interval and then use 2 filter aggregations with a range filter for the previous bucket and the current bucket; then calculate the derivative manually. For example, if the user asks for the last 5 minutes, query the last 10 minutes.

BTW... this only applies when monitoring.ui.container.elasticsearch.enabled: true is set.

[miltonhultgren]

We've reverted the changes introduced to attempt to fix this issue.