[alerting] apiKey invalidation task gets "unable to decrypt" error

[pmuellr]

Kibana version: 7.13.1

Describe the bug:

Seen in logs:

Error executing alerting apiKey invalidation task: Unable to decrypt attribute \"apiKeyId\""}

Steps to reproduce:

1. unknown

The task body is here:

kibana/x-pack/plugins/alerting/server/invalidate_pending_api_keys/task.ts

Lines 118 to 186 in 3a434d7

	return ({ taskInstance }: RunContext) => {
	const { state } = taskInstance;
	return {
	async run() {
	let totalInvalidated = 0;
	const configResult = await config;
	try {
	const [
	{ savedObjects, http },
	{ encryptedSavedObjects, security },
	] = await coreStartServices;
	const savedObjectsClient = savedObjects.getScopedClient(
	getFakeKibanaRequest(http.basePath.serverBasePath),
	{
	includedHiddenTypes: ['api_key_pending_invalidation'],
	excludedWrappers: ['security'],
	}
	);
	const encryptedSavedObjectsClient = encryptedSavedObjects.getClient({
	includedHiddenTypes: ['api_key_pending_invalidation'],
	});
	const configuredDelay = configResult.invalidateApiKeysTask.removalDelay;
	const delay = timePeriodBeforeDate(new Date(), configuredDelay).toISOString();
	let hasApiKeysPendingInvalidation = true;
	const PAGE_SIZE = 100;
	do {
	const apiKeysToInvalidate = await savedObjectsClient.find<InvalidatePendingApiKey>({
	type: 'api_key_pending_invalidation',
	filter: `api_key_pending_invalidation.attributes.createdAt <= "${delay}"`,
	page: 1,
	sortField: 'createdAt',
	sortOrder: 'asc',
	perPage: PAGE_SIZE,
	});
	totalInvalidated += await invalidateApiKeys(
	logger,
	savedObjectsClient,
	apiKeysToInvalidate,
	encryptedSavedObjectsClient,
	security
	);
	hasApiKeysPendingInvalidation = apiKeysToInvalidate.total > PAGE_SIZE;
	} while (hasApiKeysPendingInvalidation);
	return {
	state: {
	runs: (state.runs || 0) + 1,
	total_invalidated: totalInvalidated,
	},
	schedule: {
	interval: configResult.invalidateApiKeysTask.interval,
	},
	};
	} catch (e) {
	logger.warn(`Error executing alerting apiKey invalidation task: ${e.message}`);
	return {
	state: {
	runs: (state.runs || 0) + 1,
	total_invalidated: totalInvalidated,
	},
	schedule: {
	interval: configResult.invalidateApiKeysTask.interval,
	},
	};
	}
	},
	};

Not clear to me how this could happen, I guess we need to look for some race conditions? Also checking if there is a multi-tenant story going on, or if multiple Kibanas with different encryption keys could be using the same elasticsearch.

I advised the user to look for saved objects of the type api_key_pending_invalidation, presumably the problematic key keeps running into this problem, and is one of the oldest.

Even if we don't find the source of the method, I'm guessing there are some changes that we can make:

• identify the SO id of the offending document, so it can be explicitly deleted
• from looking at the code, I'm guessing that any problem like this will cause the entire batch of keys to have their processing stopped; that is, I suspect one bad document is preventing lots of API keys from being deleted. Seems possible that if the bad doc is one of the first returned from the find() call, there are lots of API keys that aren't being deleted, that should be.

[elasticmachine]

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

[pmuellr]

Was wondering - haven't really looked - to see if we can identify the source of the API key that fails decryption - specifically to the alerting rule. We may not be providing that info today in the invalidation docs. And not completely sure it would be actionable. It was a previous API key for a rule. The rule might not even exist :-). I have a feeling like it could be useful in some sort of diagnosis though.

[a1exus]

I'm getting following message as well:

Failed to decrypt "apiKeyId" attribute: Unsupported state or unable to authenticate data

version 7.17.4 & 7.17.5

[zakaria-effyis]

I'm getting the following message as well:

[2022-10-21T12:51:56.999+00:00][ERROR][plugins.encryptedSavedObjects] Failed to decrypt "apiKey" attribute: Unsupported state or unable to authenticate data
[2022-10-21T12:51:57.012+00:00][ERROR][plugins.alerting] Executing Rule default:apm.transaction_duration:8b945ea0-3b5c-11ed-a085-f9a9951ec658 has resulted in Error: Unable to decrypt attribute "apiKey" 

but in version 8.4.1 running on ECK

[rgarcia89]

Same here with Kibana v8.11.3 deployed using ECK v2.10.0

[2023-12-27T09:58:45.240+00:00][ERROR][plugins.encryptedSavedObjects] Failed to decrypt "apiKeyId" attribute: Unsupported state or unable to authenticate data
[2023-12-27T09:58:45.241+00:00][ERROR][plugins.encryptedSavedObjects] Failed to decrypt "apiKeyId" attribute: Unsupported state or unable to authenticate data
[2023-12-27T09:58:45.263+00:00][ERROR][plugins.encryptedSavedObjects] Failed to decrypt "apiKeyId" attribute: Unsupported state or unable to authenticate data
[2023-12-27T09:58:45.267+00:00][ERROR][plugins.encryptedSavedObjects] Failed to decrypt "apiKeyId" attribute: Unsupported state or unable to authenticate data

[a1exus]

• workaround: re-create by removing existing saved objects
• right way: set apiKeyId in config, this way Kibana won't generate new one and it won't mismatch w/ existing key (encryption)

[rgarcia89]

@a1exus do you know if this is related to some specific saved objects? And also if there is a way to figure out which they are?

[a1exus]

@a1exus do you know if this is related to some specific saved objects? And also if there is a way to figure out which they are?

my guess it's not a single object, it's all.. hence re-creating all objects should help, and setting up an apikey so it won't change next time you restart..

[yogeshkumark]

Which is the stable version to work in ELK?