got Unauthorized by "alerts" to get ".es-query" rule after elk upgrade

[turbotankist]

Kibana version:
8.11.4

Elasticsearch version:
8.11.4

Server OS version:
eks

Original install method (e.g. download page, yum, from source, etc.):
eck-operator 2.10.0

Describe the bug:
After upgradin elastic and kibana from 8.6.2 to 8.11.4 I can't get or update kibana rules.

{
  "statusCode": 403,
  "error": "Forbidden",
  "message": """Unauthorized by "alerts" to get ".es-query" rule"""
}

Expected behavior:

Screenshots (if relevant):

Any additional context:
Some of rules works and I can edit them, Some doesn't. There is no any correlation.

[elasticmachine]

Pinging @elastic/response-ops (Team:ResponseOps)

[turbotankist]

I can fix alerts with query

POST /.kibana_alerting_cases/_update/alert:5a49f000-a3b0-11ed-a3ac-8d8ea114e6eb
{ "doc": {
    "alert":{
      "consumer":"logs"
    }
  }
}

Failed alerts has consumer: "alerts" value.
But we have 500 alerts

[turbotankist]

I've fixed all rules by this

POST .kibana_alerting_cases/_update_by_query
{
  "script": {
    "source": "ctx._source.alert.consumer = 'logs'",
    "lang": "painless"
  },
  "query": {
    "match": {
      "alert.consumer": "alerts"
    }
  }
}

But after this command I need to update rules ApiKeys

[EyalWork]

Hey, any news about this? I'm experiencing something very similar and not sure how to proceed.

[cnasikas]

Hey @EyalWork! Could you provide more information about your situation? Which type of rules are affected? For which permissions it does not work for you?

[EyalWork]

Well, I've recently updated my VM based cluster from 8.6.0 to 8.11.3, and ever since I've had specific rules I cant access - meaning I can edit, I can see them in discover by reading the ".kibana_alerting_cases_8.11.3", but I cant seem to use the built in UI to view the rule run history etc.
The only thing they all have in common is the "alert.consumer" field has "discover" as its value. The thing that makes it weirder is that its in 3 specific Spaces, and in all my other Spaces the rules that have the same value work perfectly fine.
I will try to fix them by changing the value manually, but since I don't know what caused this, or what changing the value actually means, I'm afraid its a matter of time until the rest of my spaces fall for the same issue.
Should I fix this manually? or is this something that was fixed in a newer version?

[EyalWork]

f

I've fixed all rules by this

POST .kibana_alerting_cases/_update_by_query
{
  "script": {
    "source": "ctx._source.alert.consumer = 'logs'",
    "lang": "painless"
  },
  "query": {
    "match": {
      "alert.consumer": "alerts"
    }
  }
}

But after this command I need to update rules ApiKeys

I ended up using this to fix this issue, will most likely update my version regardless in the upcoming weeks.
I would still love to hear what the issue is, or what the best way to approach it is.
Thank you all for your time, hope we all have a wonderful week.

[cnasikas]

Hey @EyalWork! Sorry for the late reply. There is a known bug (#184595) for rules with the consumer set as discover.

The thing that makes it weirder is that its in 3 specific Spaces, and in all my other Spaces the rules that have the same value work perfectly fine

This indeed is weird. It should at least be consistent in all spaces.

Should I fix this manually? or is this something that was fixed in a newer version?

I would advise you to go through the official support channels. Unfortunately, the bug (only for consumer: discover) is not yet fixed. We are currently working on it.

I would still love to hear what the issue is, or what the best way to approach it is.

I will try to reproduce it and come back with an update and a workaround. @turbotankist @EyalWork Could you please tell me what Kibana privileges (how the role is configured) the user you logged in has? It will help a lot with the investigation.

[EyalWork]

Hi!

Could you please tell me what Kibana privileges (how the role is configured) the user you logged in has?

In my case, I've tried with numerous users, from users who originally made the alert to a superuser to eventually using the elastic user.

This indeed is weird. It should at least be consistent in all spaces.

Knowing that i should change it off "discover" regardless of space, What does alert.consumer even stand for? knowing that I can just change it to logs is an easy fix, but I still don't know what the effects of that are.

So far, I've only fixed it in 1 space where my alerts were most important, but I'm willing to cooperate to the best of my abilities to try and help understand and solve this issue.

[bplies-ATX]

Our teams just started noticing this today on Elastic stack 8.12.2 . Same errors as others have already described. Unable to delete, disable, or save an edit of the Alert rule - even as superuser.

[EyalWork]

Our teams just started noticing this today on Elastic stack 8.12.2 . Same errors as others have already described. Unable to delete, disable, or save an edit of the Alert rule - even as superuser.

You should check what the value of the "alert.consumer" field is on the rules that aren't working, In my case all the rules that didn't work had the same value, so changing that using the POST http request from the post's author.

Don't forget you have to update your API keys after this, I used a script to update them in mass, but I sadly don't have it with me as I work in an air gapped environment.

Also, the rules technically still work from what I've tested, its just that you can't really see their run history\edit them.

[cnasikas]

Hey all! The bug regarding the .es-query rule type and the consumer discover is fixed by #192321 and will be available from 8.15.3+.

What does alert.consumer even stand for? knowing that I can just change it to logs is an easy fix, but I still don't know what the effects of that are.

The consumer is an attribute used by the Alerting RBAC model. It defines who owns the rule and the alerts procured by the rule. At the moment of writing, the consumer is coupled with a feature privilege. This means that the consumer should be a valid feature ID. Assuming a user has access only to the "Logs" feature privilege the the user will have access to all rules and alerts with the logs consumer because it has access to the "Logs" feature privilege (the feature ID is logs). It will not have access to rules and alerts with the stackAlerts for example. The alerts consumer is treated specially by the framework and it will authorized based on the rule type which is also tightened to a feature ID. We are in the process of a series of improvements in the authorization model. Our goal is for users and API consumers to not have to understand the underlying complexities of the Alerting RBAC model.

[cnasikas]

@turbotankist In your case it seems that you have .es-query rules and the consumer alerts. Is this correct? Could you please provide more details about the privileges the user had? Also from which page in the UI you tried to access the rule?

[turbotankist]

It's been 10 months. I can't give information about the privileges now. The cluster was installed via eck-operator. Nothing has been specifically changed in the kibana permissions.

[cnasikas]

@turbotankist Sorry for the time it took for this issue. Is there anything I can help with the current issue?