[discuss] Adding a new Core encryption service

[pgayvallet]

Discussion started in #178304:

We were discussing, a few weeks ago with @azasypkin, about how nice it would be to have a centralized way to manage encryption within Kibana. Something like an encryption service that could define a master key that consumers needing to use encryption could use as a base for their own keys (while also exposing the corresponding APIs to encrypt/decrypt based on our needs). It would allow to avoid manually defining the keys for all parts of Kibana that need to use encryption, (while still allowing the option for customers that really need/want it).

I'm opening this issue to discuss about what this new encryption service could look like, in term of features, responsibilities and APIs.

[elasticmachine]

Pinging @elastic/kibana-security (Team:Security)

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[pgayvallet]

cc @azasypkin @legrego and more globally @elastic/kibana-security, curious to have your thought on this, given you're probably all have a better vision than we do on what should a global/core encryption service look like.

[pgayvallet]

Just from our previous discussions, in terms of what the service could help us with:

• Encrypted saved objects: we could finally get rid of the encryption extension and manage encryption more directly from the SOR itself.

• Encrypted UI settings (Encrypted UI Settings #178304): it could allow managing encrypted setting values without the need of yet another inversion of control to delegate the encryption to a plugin

• Centralized "master" encryption key: it could allow customers to only specify one "master" encryption key, and have all Kibana features that needs to rely on encryption to use it to create sub keys (or use it directly?)

• Provide high-level APIs to manage encryption and decryption for Kibana features that need it (note: could/should probably be exposed as static utilities - maybe it's done already, even?)

What did I forget?

[legrego]

@pgayvallet your list looks good to me. To make Centralized "master" encryption key explicit, this would currently cover:

1. Session encryption
2. ESO encryption
3. Reporting encryption

One other use case I thought of is user preferences. I fully expect that solution teams will eventually need a way to securely store user-specific settings. Having a core encryption service would facilitate this as well.

[azasypkin]

What did I forget?

The list seems to cover the majority of the benefits I can think of at the moment, thanks! The only two things that might be worth mentioning explicitly, even though it's implied, are that 1) we'll hopefully have a single approach to rotate the encryption keys (declaratively via config and programmatically via APIs) and 2) detect encryption misconfigurations more easily.

I fully expect that solution teams will eventually need a way to securely store user-specific settings. Having a core encryption service would facilitate this as well.

++, I've already heard about the use cases that could benefit from this (e.g., user-level integrations with external systems like ChatGPT or CoPilot that might require us to store user-specific external credentials).

[legrego]

Next steps from our latest Core/Security sync:

We agreed that @elastic/kibana-security would produce an initial RFC, with Core contributing support for the API design.

[pgayvallet]

@legrego do you know already would be on charge of the RFC in the Security team? I'd gladly be the main contact point on Core's side.

[legrego]

@legrego do you know already would be on charge of the RFC in the Security team? I'd gladly be the main contact point on Core's side.

@pgayvallet no, we haven't planned this work yet. Thanks for offering to be our point of contact, we'll be sure to coordinate with you