Skip to content

Commit

Permalink
Support extra label selectors on mimir.rules.kubernetes (#1773)
Browse files Browse the repository at this point in the history 
* support extra labels on mimir rules queries

* Add tests and expose query matchers to users.

* Docs

* Changelog

* Apply suggestions from code review

Co-authored-by: Clayton Cornell <[email protected]>

---------

Co-authored-by: Clayton Cornell <[email protected]>
  • Loading branch information
@thampiotr @clayton-cornell
thampiotr and clayton-cornell authored Sep 30, 2024
1 parent 65b9f44 commit 8f1be0e
Show file tree
Hide file tree
Showing 7 changed files with 447 additions and 119 deletions.
5 changes: 5 additions & 0 deletions CHANGELOG.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,11 @@ Main (unreleased)
- Add the function `path_join` to the stdlib. (@wildum)
- Add support to `loki.source.syslog` for the RFC3164 format ("BSD syslog"). (@sushain97)

### Enhancements

- The `mimir.rules.kubernetes` component now supports adding extra label matchers
to all queries discovered via `PrometheusRule` CRDs. (@thampiotr)

### Bugfixes

- Update windows_exporter from v0.27.2 vo v0.27.3: (@jkroepke)
Expand Down
132 changes: 87 additions & 45 deletions docs/sources/reference/components/mimir/mimir.rules.kubernetes.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -50,25 +50,25 @@ mimir.rules.kubernetes "LABEL" {

`mimir.rules.kubernetes` supports the following arguments:

Name | Type | Description | Default | Required
-------------------------|---------------------|--------------------------------------------------------------------------------------------------|---------------|---------
`address` | `string` | URL of the Mimir ruler. | | yes
`tenant_id` | `string` | Mimir tenant ID. | | no
`use_legacy_routes` | `bool` | Whether to use [deprecated][gem-2_2] ruler API endpoints. | false | no
`prometheus_http_prefix` | `string` | Path prefix for [Mimir's Prometheus endpoint][gem-path-prefix]. | `/prometheus` | no
`sync_interval` | `duration` | Amount of time between reconciliations with Mimir. | "5m" | no
`mimir_namespace_prefix` | `string` | Prefix used to differentiate multiple {{< param "PRODUCT_NAME" >}} deployments. | "alloy" | no
`bearer_token_file` | `string` | File containing a bearer token to authenticate with. | | no
`bearer_token` | `secret` | Bearer token to authenticate with. | | no
`enable_http2` | `bool` | Whether HTTP2 is supported for requests. | `true` | no
`follow_redirects` | `bool` | Whether redirects returned by the server should be followed. | `true` | no
`proxy_url` | `string` | HTTP proxy to send requests through. | | no
`no_proxy` | `string` | Comma-separated list of IP addresses, CIDR notations, and domain names to exclude from proxying. | | no
`proxy_from_environment` | `bool` | Use the proxy URL indicated by environment variables. | `false` | no
`proxy_connect_header` | `map(list(secret))` | Specifies headers to send to proxies during CONNECT requests. | | no
`external_labels` | `map(string)` | Labels to add to each rule. | `{}` | no

At most, one of the following can be provided:
| Name | Type | Description | Default | Required |
|--------------------------|---------------------|--------------------------------------------------------------------------------------------------|---------------|----------|
| `address` | `string` | URL of the Mimir ruler. | | yes |
| `tenant_id` | `string` | Mimir tenant ID. | | no |
| `use_legacy_routes` | `bool` | Whether to use [deprecated][gem-2_2] ruler API endpoints. | false | no |
| `prometheus_http_prefix` | `string` | Path prefix for [Mimir's Prometheus endpoint][gem-path-prefix]. | `/prometheus` | no |
| `sync_interval` | `duration` | Amount of time between reconciliations with Mimir. | "5m" | no |
| `mimir_namespace_prefix` | `string` | Prefix used to differentiate multiple {{< param "PRODUCT_NAME" >}} deployments. | "alloy" | no |
| `bearer_token_file` | `string` | File containing a bearer token to authenticate with. | | no |
| `bearer_token` | `secret` | Bearer token to authenticate with. | | no |
| `enable_http2` | `bool` | Whether HTTP2 is supported for requests. | `true` | no |
| `follow_redirects` | `bool` | Whether redirects returned by the server should be followed. | `true` | no |
| `proxy_url` | `string` | HTTP proxy to send requests through. | | no |
| `no_proxy` | `string` | Comma-separated list of IP addresses, CIDR notations, and domain names to exclude from proxying. | | no |
| `proxy_from_environment` | `bool` | Use the proxy URL indicated by environment variables. | `false` | no |
| `proxy_connect_header` | `map(list(secret))` | Specifies headers to send to proxies during CONNECT requests. | | no |
| `external_labels` | `map(string)` | Labels to add to each rule. | `{}` | no |

At most, one of the following can be provided:
- [`bearer_token` argument](#arguments).
- [`bearer_token_file` argument](#arguments).
- [`basic_auth` block][basic_auth].
Expand Down Expand Up @@ -105,17 +105,19 @@ This is useful if you configure Mimir to use a different [prefix][gem-path-prefi
The following blocks are supported inside the definition of
`mimir.rules.kubernetes`:

Hierarchy | Block | Description | Required
-------------------------------------------|------------------------|----------------------------------------------------------|---------
rule_namespace_selector | [label_selector][] | Label selector for `Namespace` resources. | no
rule_namespace_selector > match_expression | [match_expression][] | Label match expression for `Namespace` resources. | no
rule_selector | [label_selector][] | Label selector for `PrometheusRule` resources. | no
rule_selector > match_expression | [match_expression][] | Label match expression for `PrometheusRule` resources. | no
basic_auth | [basic_auth][] | Configure basic_auth for authenticating to the endpoint. | no
authorization | [authorization][] | Configure generic authorization to the endpoint. | no
oauth2 | [oauth2][] | Configure OAuth2 for authenticating to the endpoint. | no
oauth2 > tls_config | [tls_config][] | Configure TLS settings for connecting to the endpoint. | no
tls_config | [tls_config][] | Configure TLS settings for connecting to the endpoint. | no
| Hierarchy | Block | Description | Required |
|--------------------------------------------|--------------------------|----------------------------------------------------------|----------|
| extra_query_matchers | [extra_query_matchers][] | Additional label matchers to add to each query. | no |
| extra_query_matchers > matcher | [matcher][] | A label matcher to add to query. | no |
| rule_namespace_selector | [label_selector][] | Label selector for `Namespace` resources. | no |
| rule_namespace_selector > match_expression | [match_expression][] | Label match expression for `Namespace` resources. | no |
| rule_selector | [label_selector][] | Label selector for `PrometheusRule` resources. | no |
| rule_selector > match_expression | [match_expression][] | Label match expression for `PrometheusRule` resources. | no |
| basic_auth | [basic_auth][] | Configure basic_auth for authenticating to the endpoint. | no |
| authorization | [authorization][] | Configure generic authorization to the endpoint. | no |
| oauth2 | [oauth2][] | Configure OAuth2 for authenticating to the endpoint. | no |
| oauth2 > tls_config | [tls_config][] | Configure TLS settings for connecting to the endpoint. | no |
| tls_config | [tls_config][] | Configure TLS settings for connecting to the endpoint. | no |

The `>` symbol indicates deeper levels of nesting. For example,
`oauth2 > tls_config` refers to a `tls_config` block defined inside
Expand All @@ -127,16 +129,38 @@ an `oauth2` block.
[tls_config]: #tls_config-block
[label_selector]: #label_selector-block
[match_expression]: #match_expression-block
[extra_query_matchers]: #extra_query_matchers-block
[matcher]: #matcher-block

### extra_query_matchers block

The `extra_query_matchers` block has no attributes. It contains zero or more [matcher][] blocks.
These blocks allow you to add extra label matchers to all queries that are discovered by `mimir.rules.kubernetes`
component. The algorithm of adding the label matchers to queries is the same as the one provided by
`promtool promql label-matchers set` command in [promtool](https://github.com/prometheus/prometheus/tree/main/cmd/promtool).

### matcher block

The `matcher` block describes a label matcher that will be added to each query found in `PrometheusRule` CRDs.

The following arguments are supported:

| Name | Type | Description | Default | Required |
|--------------|----------|----------------------------------------------------|---------|----------|
| `name` | `string` | Name of the label to match. | | yes |
| `match_type` | `string` | The type of match. One of `=`, `!=`, `=~` and `!~`. | | yes |
| `value` | `string` | Value of the label to match. | | yes |


### label_selector block

The `label_selector` block describes a Kubernetes label selector for rule or namespace discovery.

The following arguments are supported:

Name | Type | Description | Default | Required
---------------|---------------|---------------------------------------------------|-----------------------------|---------
`match_labels` | `map(string)` | Label keys and values used to discover resources. | `{}` | yes
| Name | Type | Description | Default | Required |
|----------------|---------------|---------------------------------------------------|---------|----------|
| `match_labels` | `map(string)` | Label keys and values used to discover resources. | `{}` | yes |

When the `match_labels` argument is empty, all resources will be matched.

Expand All @@ -146,11 +170,11 @@ The `match_expression` block describes a Kubernetes label match expression for r

The following arguments are supported:

Name | Type | Description | Default | Required
-----------|----------------|----------------------------------------------------|---------|---------
`key` | `string` | The label name to match against. | | yes
`operator` | `string` | The operator to use when matching. | | yes
`values` | `list(string)` | The values used when matching. | | no
| Name | Type | Description | Default | Required |
|------------|----------------|------------------------------------|---------|----------|
| `key` | `string` | The label name to match against. | | yes |
| `operator` | `string` | The operator to use when matching. | | yes |
| `values` | `list(string)` | The values used when matching. | | no |

The `operator` argument should be one of the following strings:

Expand Down Expand Up @@ -204,13 +228,13 @@ actually exist.

## Debug metrics

Metric Name | Type | Description
----------------------------------------------|-------------|-------------------------------------------------------------------------
`mimir_rules_config_updates_total` | `counter` | Number of times the configuration has been updated.
`mimir_rules_events_total` | `counter` | Number of events processed, partitioned by event type.
`mimir_rules_events_failed_total` | `counter` | Number of events that failed to be processed, partitioned by event type.
`mimir_rules_events_retried_total` | `counter` | Number of events that were retried, partitioned by event type.
`mimir_rules_client_request_duration_seconds` | `histogram` | Duration of requests to the Mimir API.
| Metric Name | Type | Description |
|-----------------------------------------------|-------------|--------------------------------------------------------------------------|
| `mimir_rules_config_updates_total` | `counter` | Number of times the configuration has been updated. |
| `mimir_rules_events_total` | `counter` | Number of events processed, partitioned by event type. |
| `mimir_rules_events_failed_total` | `counter` | Number of events that failed to be processed, partitioned by event type. |
| `mimir_rules_events_retried_total` | `counter` | Number of events that were retried, partitioned by event type. |
| `mimir_rules_client_request_duration_seconds` | `histogram` | Duration of requests to the Mimir API. |

## Example

Expand Down Expand Up @@ -253,6 +277,24 @@ mimir.rules.kubernetes "default" {
}
```

This example adds label matcher `{cluster=~"prod-.*"}` to all the queries discovered by `mimir.rules.kubernetes`.

```alloy
mimir.rules.kubernetes "default" {
address = "GRAFANA_CLOUD_METRICS_URL"
extra_query_matchers {
matcher {
name = "cluster"
match_type = "=~"
value = "prod-.*"
}
}
}
```

If a query in the form of `up != 1` is found in `PrometheusRule` CRDs,
it will be modified to `up{cluster=~"prod-.*"} != 1` before sending it to Mimir.

The following example is an RBAC configuration for Kubernetes. It authorizes {{< param "PRODUCT_NAME" >}} to query the Kubernetes REST API:

```yaml
Expand Down
Loading

0 comments on commit 8f1be0e

Please sign in to comment.