[ResponseOps][Alerting] classify API unauthorized errors as user error

[pmuellr]

Saw a number of messages of the following form in a QA project:

error(s): ResponseError: security_exception
	Root causes:
		security_exception: action [indices:data/read/search] is unauthorized for API key id [redacted] of user [redacted], this action is granted by the index privileges [read,all]

These are logged with tags: framework-error

These feel like they should be user error. It appears these set our rule run failure SLO off.

[elasticmachine]

Pinging @elastic/response-ops (Team:ResponseOps)

[pmuellr]

Was asked to check if these show up in alerting framework messages or just in the SIEM-specific messages logged by the rule executor.

Both. The ones from the SIEM-specific rule executors are noisy, but aren't specifically considered "errors" at all by the framework, they're just messages logged by the executor.

However, those rule executions do seem to be returned as failed by the executor, so we see the following logged when the rule completes:

message: Executing Rule siem.queryRule:<rule-id> has resulted in the following error(s): ResponseError: security_exception\n\tRoot causes:\n\t\tsecurity_exception: action [indices:data/read/search] is unauthorized for API key id [<api-key-id>] of user [<user-id>], this action is granted by the index privileges [read,all]

tags: <rule-id>, siem.queryRule, rule-run-failed, framework-error

So seems like we should get the SIEM rule to return these as user-error, unless we can figure out how to do this ourselves, by using our wrapped ES client library. Seems like that should be our first approach, since it would end up working for all rule types - see if we can determine this at the framework level.