PSMDB-1283 Added KMIP key state polling (#896)

* PSMDB-1283 Added KMIP key state polling

modified:   docs/kmip.md

docs/data-at-rest-encryption.md

@@ -4,23 +4,28 @@ Data at rest encryption for the WiredTiger storage engine in MongoDB was
 introduced in MongoDB Enterprise version 3.2 to ensure that encrypted data
 files can be decrypted and read by parties with the decryption key.
 
-!!! important
-
-    You can only enable data at rest encryption and provide all encryption settings on an empty database, when you start the mongod instance for the first time. You cannot enable or disable encryption while the Percona Server for MongoDB server is already running and / or has some data. Nor can you change the effective encryption mode by simply restarting the server. Every time you restart the server, the encryption settings must be the same.
-
 ## Differences from upstream
 
 The data encryption at rest in Percona Server for MongoDB is introduced in version 3.6 to be compatible with data encryption at rest interface in MongoDB. In the current release of Percona Server for MongoDB, the data encryption at rest does not include support for Amazon AWS key management service. Instead, Percona Server for MongoDB is [integrated with HashiCorp Vault](vault.md). 
 
 Starting with release 5.0.7-6, Percona Server for MongoDB supports the secure transfer of keys using [Key Management Interoperability Protocol (KMIP)](kmip.md). This allows users to store encryption keys in their favorite KMIP-compatible key manager when they set up encryption at rest.
 
-Two types of keys are used for data at rest encryption:
+
+## Workflow
+
+!!! important
+
+    You can only enable data at rest encryption and provide all encryption settings on an empty database, when you start the mongod instance for the first time. You cannot enable or disable encryption while the Percona Server for MongoDB server is already running and / or has some data. Nor can you change the effective encryption mode by simply restarting the server. Every time you restart the server, the encryption settings must be the same.
+
+Each node of Percona Server for MongoDB generates a random, individual key for every database. It encrypts every database with an individual key and puts those keys into the special, so-called key database. Then each node of Percona Server for MongoDB randomly generates a unique master encryption key and encrypts the key database with this key. 
+
+Thus, two types of keys are used for data at rest encryption:
 
 * Database keys to encrypt data. They are stored internally, near the data that they encrypt.
 
 * The master key to encrypt database keys. It is kept separately from the data and database keys and requires external management.
 
-To manage the master key, use one of the supported key management options:
+To manage the master encryption key, use one of the supported key management options:
 
 * Integration with an external key server (recommended). Percona Server for MongoDB is [integrated with HashiCorp Vault](vault.md) for this purpose and supports the secure transfer of keys using [Key Management Interoperability Protocol (KMIP)](kmip.md).
 
@@ -29,6 +34,7 @@ To manage the master key, use one of the supported key management options:
 Note that you can use only one of the key management options at a time. However, you can switch from one management option to another (e.g. from a keyfile to HashiCorp Vault). Refer to [Migrating from Key File Encryption to HashiCorp Vault Encryption](encryption-mode-switch.md) section for details.
 
 
+
 ## Important configuration options
 
 Percona Server for MongoDB supports the `encryptionCipherMode` option where you choose one of the following cipher modes:

docs/install/minor-upgrade.md

@@ -1,5 +1,7 @@
 # Minor upgrade of Percona Server for MongoDB
 
+If you are using data-at-rest-encryption with KMIP server, check the [upgrade considerations](../kmip.md#upgrade-considerations)
+
 To upgrade Percona Server for MongoDB to the latest version, follow these steps:
 
 

docs/install/upgrade-from-44.md

@@ -1,11 +1,15 @@
 # Upgrading from Percona Server for MongoDB 4.4 to 5.0
 
-To upgrade Percona Server for MongoDB to version 5.0, you must be running version 4.4. Upgrades from earlier versions are not supported.
+## Considerations
 
-Before upgrading your production Percona Server for MongoDB deployments, test all your applications
+1. To upgrade Percona Server for MongoDB to version 5.0, you must be running version 4.4. Upgrades from earlier versions are not supported.
+
+2. Before upgrading your production Percona Server for MongoDB deployments, test all your applications
 in a testing environment to make sure they are compatible with the new version.
 For more information, see [Compatibility Changes in MongoDB 5.0](https://docs.mongodb.com/manual/release-notes/5.0-compatibility/)
 
+3. If you are using data-at-rest-encryption with KMIP server, check the [upgrade considerations](../kmip.md#upgrade-considerations)
+
 We recommend to upgrade Percona Server for MongoDB from official Percona repositories using [`percona-release` repository management tool](https://docs.percona.com/percona-software-repositories/index.html) and
 the corresponding package manager for your system. 
 

docs/kmip.md

@@ -8,15 +8,52 @@ KMIP enables the communication between key management systems and the database s
 
 * Streamlines encryption key management
 * Eliminates redundant key management processes
+* Reduces the mean time to resolve (MTTR) compromised encryption key incidents via [key state polling](#key-state-polling)
+
+???+ admonition "Version changes"
+
+     The following table lists the changes in the KMIP implementation in Percona Server for MongoDB and the versions that introduced those changes:     
+
+     |Version         | Description |
+     |----------------|-------------|
+     | [5.0.8-7](release_notes/5.0.8-7.md)| [Master key rotation](#key-rotation).|
+     | [5.0.9-8](release_notes/5.0.9-8.md)| [Support for multiple KMIP servers for failover](#support-for-multiple-kmip-servers).|
+     | [5.0.11-10](release_notes/5.0.11-10.md)| [Changed handling of `kmipKeyIdentifier` option](#optional-key-identifier).|
+     | [5.0.28-24](release_notes/5.0.28-24.md)| [Key state polling](#key-state-polling).|
+
+
+## Support for multiple KMIP servers
 
 Starting with version 5.0.9-8, you can specify multiple KMIP servers for failover. On startup, Percona Server for MongoDB connects to the servers in the order listed and selects the one with which the connection is successful.
 
+## Optional key identifier
+
 Starting with version 5.0.11-10, the `kmipKeyIdentifier` option is no longer mandatory. When left blank, the database server creates a key on the KMIP server and uses that for encryption. When you specify the identifier, the key with such an ID must exist on the key storage.
 
 !!! note
 
     Starting with version 5.0.17-14, the master key is stored in a raw-byte format. If you set up Percona Server for MongoDB 5.0.17-14 with data-at-rest encryption using KMIP and wish to downgrade to some previous version, this downgrade is not possible via binary replacement. Consider using the [logical backup and restore via Percona Backup for MongoDB](https://docs.percona.com/percona-backup-mongodb/usage/start-backup.html) for this purpose.
 
+## Key rotation
+
+Starting with version 5.0.8-7, the support for [master key rotation](https://www.mongodb.com/docs/manual/tutorial/rotate-encryption-key/#kmip-master-key-rotation) is added. This enables users to comply with data security regulations when using KMIP.
+
+## Key state polling
+
+When a Percona Server for MongoDB node generates a new master encryption key, it registers the key on the KMIP server with the `Pre-Active` state. Starting with version 5.0.28-24, Percona Server for MongoDB automatically activates the master encryption key and periodically checks (polls) its state. If a master encryption key for a node is not in the `Active` state, the node reports an error and shuts down. This process helps security engineers identify the nodes that require out-of-schedule master key rotation.
+
+Key state polling es enabled by default and is regulated by these configuration file options: `kmip.activateKeys` and `kmip.keyStatePollingSeconds`.
+
+The following diagram illustrates the master key lifecycle with key state polling:
+
+![image](_images/kmip-state-polling.png)
+
+The master key state polling functionality is particularly useful in cluster deployments with hundreds of nodes. If some master keys are compromised, security engineers change their state from `Active` so that the nodes encrypted with these keys identify themselves. This approach allows the security engineers to rotate master keys only on the affected nodes instead of the entire cluster, thus reducing the mean time to resolve (MTTR) compromised encryption key incidents.
+
+!!! admonition "See also"
+
+    Percona Blog: [Improve the Security of a Percona Server for MongoDB Deployment with KMIP Key State Polling]() by Konstantin Trushin.
+
 ## KMIP parameters
 
 | Configuration file | {{ optionlink('security.kmip.serverName') }}|
@@ -73,11 +110,22 @@ Starting with version 5.0.11-10, the `kmipKeyIdentifier` option is no longer man
 | **Type**            | int|
 | **Description**     | The time to wait for the response from the KMIP server. Min value: 1000. Default: 5000. <br><br>If the `connectRetries` setting is specified, the `mongod` waits up to the value specified with `connectTimeoutMS` for each retry.|
 
-# Key rotation
+| Configuration file  | {{optionlink('security.kmip.activateKeys')}}|
+|-------------------- | --------------------|
+| **Command line**    | `kmipActivateKeys`|
+| **Type**            | boolean|
+| **Description**     | When enabled, Percona Server for MongoDB activates a newly created master encryption key or verifies that the existing master key is in the Active state at startup. It also initiates the key state polling. Enabled by default. Available starting with version 5.0.28-24.|
+
+| Configuration file  | {{optionlink('security.kmip.keyStatePollingSeconds')}}|
+|-------------------- | --------------------|
+| **Command line**    | `kmipKeyStatePollingSeconds`|
+| **Type**            | int|
+| **Description**     | The period in seconds to check the state of the master encryption key. Default: 900. If the master encryption key is not in the Active state, the node logs the error and shuts down. Available starting with version 5.0.28-24.|
+
 
-Starting with release 5.0.8-7, the support for [master key rotation](https://www.mongodb.com/docs/manual/tutorial/rotate-encryption-key/#kmip-master-key-rotation) is added. This enables users to comply with data security regulations when using KMIP.
 
-# Configuration
+
+## Configuration
 
 ### Considerations
 
@@ -111,7 +159,9 @@ $ mongod --enableEncryption \
   --kmipKeyIdentifier <kmip_identifier>
 ```
 
-# Minor upgrade of Percona Server for MongoDB to version 5.0.11-10 and higher
+## Upgrade considerations
+
+### To version 5.0.11-10 and higher
 
 With the `kmipKeyIdentifier` option becoming optional in version 5.0.11-10, the standard upgrade procedure doesn’t work if you are upgrading from version 5.0.10-9 and earlier.
 
@@ -122,3 +172,11 @@ This section provides upgrade instructions from Percona Server for MongoDB 5.0.1
 For a single-node deployment, use the `mongodump` / `mongorestore` tools to make a backup before the update and to restore from it after binaries are updated.
 
 For replica sets, data must be re-encrypted with the **new** key during the upgrade. Go through the [encrypting existing data steps](https://www.mongodb.com/docs/v5.0/tutorial/configure-encryption/#std-label-encrypt-existing-data)  but perform the [minor upgrade](install/upgrade-from-mongodb.md#minor-upgrade) between steps 1 and 2 to replace the `mongod` binary.
+
+### To version 5.0.28-24 and higher
+
+Percona Server for MongoDB 5.0.28 and subsequent versions tolerate already existing `Pre-Active` master keys as follows: if at startup Percona Server for MongoDB detects that the data directory is encrypted with an existing master key in the `Pre-Active` state, it logs a warning and continues to operate as usual. In that case, Percona Server for MongoDB does not do periodic key state polling regardless the value specified for the [`kmipKeyStatePollingSeconds`](#security-kmip-keystatepollingseconds) option. [Read more about key state polling](#key-state-polling).
+
+We recommend to either rotate a master encryption key or manually change the existing key to the Active state. You can also explicitly set the [`security.kmip.activateKeys`](#security-kmip-activatekeys) configuration file option to ensure that only the active keys are used. This one-time operation smooths the major upgrade flow.
+
+

mkdocs-base.yml

@@ -92,8 +92,8 @@ markdown_extensions:
   pymdownx.details: {}
   pymdownx.mark: {}
   pymdownx.smartsymbols: {}
-  pymdownx.tabbed: 
-      {alternate_style: true}
+  pymdownx.tabbed: {}
+#      {alternate_style: true}
   pymdownx.tilde: {}
   pymdownx.superfences: 
     custom_fences:

mkdocs-pdf.yml

@@ -7,4 +7,13 @@ copyright: Percona LLC, © 2024
 
 markdown_extensions:
   pymdownx.tabbed: {}
-  admonition: {}
+  admonition: {}
+
+extra_css:
+  - https://unicons.iconscout.com/release/v3.0.3/css/line.css
+  - https://cdnjs.cloudflare.com/ajax/libs/font-awesome/4.4.0/css/font-awesome.min.css
+  - css/percona.css
+  - css/nocopy.css
+  - css/extra.css
+  - css/mongodb.css
+  - css/landing.css