Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document how to perform a restore from k8up backup #90

Merged
merged 2 commits into from
Aug 26, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
133 changes: 133 additions & 0 deletions docs/modules/ROOT/pages/how-tos/restore-from-backup.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,133 @@
= Restore from Backup

When configuring component-vault with `backup.enabled = true`, the component sets up regular backups using k8up.
This how-to explains how Vault can be restored from such a backup.

== Information on the Vault Unseal Key and Root Token

include::partial$vault-warning.adoc[]

== Prerequisites

* `restic` - command line tool to access backups made by k8up
* `vault` - command line tool to interact with Vault
* `kubectl`
* Write access to the cluster's tenant repository
* Read access to the restic repository in which k8up stored the Vault backups
* The Vault instance's *unseal key* and *root token* - these must be backed up manually; they are not part of the automated k8up backup.

== Procedure

=== 1. Set up new Vault instance
HappyTetrahedron marked this conversation as resolved.
Show resolved Hide resolved

. Add the `vault` application to your cluster configuration.
.. If your old instance of vault is still running on the cluster, you can use component instantiation to create a second instance by adding `vault as new-vault` to your application list, and configuring it under `new_vault`.
. Initially disable backups by setting `.backups.enabled` to `false`
. Compile and push the cluster config and wait for Vault to start.

=== 2. Retrieve the Vault snapshot

. Set up the restic credentials (values correspond to the component parameters `backup.bucket` and `backup.password`)
+
[source,shell]
----
export AWS_ACCESS_KEY_ID="S3_KEY" # from component configuration: backup.bucket.accesskey
export AWS_SECRET_ACCESS_KEY="S3_SECRET" # from component confiugration: backup.bucket.secretkey
export RESTIC_REPOSITORY="s3:https://path.to.my/bucket"
export RESTIC_PASSWORD="RESTIC_REPO_KEY" # from component configuration: backup.password
----
. Retrieve the latest Vault snapshot to your local disk
+
[source,shell]
----
mkdir restore
restic restore --target restore/ latest
----
. Verify the snapshot file
+
[source,shell]
----
ls restore
# This should show a file named "[instance name]-backup.snapshot"
----

=== 3. Restore the snapshot

. Expose the Vault pod
+
[source,shell]
----
kubectl port-forward -n $VAULT_INSTANCE_NAME ${VAULT_INSTANCE_NAME}-0 8200
----
. In a separate terminal, prepare the environment to access Vault
+
[source,shell]
----
# Get root token to log in
export VAULT_TOKEN="$(kubectl get secret -n $VAULT_INSTANCE_NAME ${VAULT_INSTANCE_NAME}-seal -ojsonpath='{.data.vault-root}' | base64 -d)"
export VAULT_ADDR="http://127.0.0.1:8200"
----
. Restore the backup
+
[source,shell]
----
vault operator raft snapshot restore -force restore/${VAULT_INSTANCE_NAME}-backup.snapshot
----

=== 4. Unseal Vault

If you were logged into the Vault UI, you should have gotten logged out now.
This is expected.

. Open your browser at http://localhost:8200
. Use the *Vault Unseal Key* of the Vault instance you've just restored to unseal Vault
. Use the *Vault root token* of the Vault instance you've just restored to log in with the `Token` method
. Verify that the restore worked, and secrets are now restored in Vault.

[IMPORTANT]
====
The unseal key and root token of the Vault instance you're restoring need to have been stored separately.
Without them, the restore procedure cannot be completed.
====

=== 5. Update the Vault Secret

NOTE: Without this step, your Vault instance will not be able to auto-unseal.

. Encode the Vault credentials
+
[source,shell]
----
export VAULT_UNSEAL_KEY="OLD_UNSEAL_KEY"
export VAULT_ROOT_TOKEN="OLD_ROOT_TOKEN"

echo -n "$VAULT_UNSEAL_KEY" | base64 -w0
echo -n "$VAULT_ROOT_TOKEN" | base64 -w0
----
. Update the Vault secret
+
[source,shell]
----
kubectl edit secret -n ${VAULT_INSTANCE_NAME} ${VAULT_INSTANCE_NAME}-seal
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd be nice to have a copy paste-able command here since updating secrets is a PITA. Especally if you encode a newline into the base64 by accident.

----
. Update the `vault-root` and `vault-unseal-0` keys to reflect the values you have just encoded
. Save the secret
. Verify that auto-unseal works:
.. Restart all vault pods simultaneously:
+
[source,shell]
----
kubectl delete pod -n $VAULT_INSTANCE_NAME ${VAULT_INSTANCE_NAME}-{0..2}
----
.. Expose the Vault UI
+
[source,shell]
----
kubectl port-forward -n $VAULT_INSTANCE_NAME ${VAULT_INSTANCE_NAME}-0 8200
----
.. Verify that http://localhost:8200[the Vault UI] does not prompt you for the unseal key


=== 6. Cleanup

. Reenable `backups.enabled` in the component configuration
6 changes: 6 additions & 0 deletions docs/modules/ROOT/pages/references/parameters.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -252,6 +252,12 @@ default:: `true`

Whether to do backups using k8up.

[WARNING]
.Manual Setup Required
====
include::partial$vault-warning.adoc[]
====

== `backup.schedule`

[horizontal]
Expand Down
1 change: 1 addition & 0 deletions docs/modules/ROOT/partials/nav.adoc
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
* xref:index.adoc[Home]
* xref:tutorials/install.adoc[Installation]
* xref:how-tos/lieutenant.adoc[Lieutenant Integration]
* xref:how-tos/restore-from-backup.adoc[Restore Vault from Backup]
* xref:references/parameters.adoc[Parameters]
8 changes: 8 additions & 0 deletions docs/modules/ROOT/partials/vault-warning.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
`component-vault` leverages k8up's application specific backups to create Vault snapshots.
Restoring such a snapshot is only possible by providing the Vault https://developer.hashicorp.com/vault/docs/concepts/seal[unseal key] and https://developer.hashicorp.com/vault/docs/concepts/tokens#root-tokens[root token].
As these are sensitive secrets, they are not backed up as part of the component's automated backup process.

When setting up Vault, you must store the Vault unseal key and root token in a safe location in order to be able to restore backups.

In a running Vault instance, both of these secrets can be found in the `[instance name]-seal` secret in the Vault instance's namespace.
In order to have a reliable Vault backup, this secret must be backed up separately to a safe location.
Loading