Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document how to perform a restore from k8up backup #90

Merged
merged 2 commits into from
Aug 26, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
117 changes: 117 additions & 0 deletions docs/modules/ROOT/pages/how-tos/restore-from-backup.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,117 @@
= Restore from Backup

When configuring component-vault with `backup.enabled = true`, the component sets up regular backups using k8up.
This how-to explains how Vault can be restored from such a backup.

== Information on the Vault Unseal Key and Root Token

include::partial$vault-warning.adoc[]

== Prerequisites

* `restic` - command line tool to access backups made by k8up
* `vault` - command line tool to interact with Vault
* `kubectl`
* Write access to the cluster's tenant repository
* Read access to the restic repository in which k8up stored the Vault backups
* The Vault instance's *unseal key* and *root token* - these must be backed up manually; they are not part of the automated k8up backup.

== Procedure

=== 1. Set up new Vault instance
HappyTetrahedron marked this conversation as resolved.
Show resolved Hide resolved

* Add the `vault` application to your cluster configuration
HappyTetrahedron marked this conversation as resolved.
Show resolved Hide resolved
* Initially disable backups by setting `.backups.enabled` to `false`
* Compile and push the cluster config and wait for Vault to start.

=== 2. Retrieve the Vault snapshot

* Set up the restic credentials (values correspond to the component parameters `backup.bucket` and `backup.password`)
+
[source,shell]
----
export AWS_ACCESS_KEY_ID="S3_KEY backup.bucket.accesskey"
HappyTetrahedron marked this conversation as resolved.
Show resolved Hide resolved
export AWS_SECRET_ACCESS_KEY="S3_SECRET backup.bucket.secretkey"
export RESTIC_REPOSITORY="s3:https://path.to.my/bucket"
export RESTIC_PASSWORD="RESTIC_REPO_KEY backup.password"
----
* Retrieve the latest Vault snapshot to your local disk
+
[source,shell]
----
mkdir restore
restic restore --target restore/ latest
----
* Verify the snapshot file
+
[source,shell]
----
ls restore
# This should show a file named "[instance name]-backup.snapshot"
----

=== 3. Restore the snapshot

* Expose the Vault pod
+
[source,shell]
----
kubectl port-forward -n $VAULT_INSTANCE_NAME ${VAULT_INSTANCE_NAME}-0 8200
----
* In a separate terminal, prepare the environment to access Vault
+
[source,shell]
----
# Get root token to log in
export VAULT_TOKEN="$(kubectl get secret -n $VAULT_INSTANCE_NAME ${VAULT_INSTANCE_NAME}-seal -ojsonpath='{.data.vault-root}' | base64 -d)"
export VAULT_ADDR="http://127.0.0.1:8200"
----
* Restore the backup
+
[source,shell]
----
vault operator raft snapshot restore -force restore/${VAULT_INSTANCE_NAME}-backup.snapshot
----

=== 4. Unseal Vault

If you were logged into the Vault UI, you should have gotten logged out now.
This is expected.

* Open your browser at http://localhost:8200
* Use the *Vault Unseal Key* of the Vault instance you've just restored to unseal Vault
* Use the *Vault root token* of the Vault instance you've just restored to log in with the `Token` method
* Verify that the restore worked, and secrets are now restored in Vault.

[IMPORTANT]
====
The unseal key and root token of the Vault instance you're restoring need to have been stored separately.
Without them, the restore procedure cannot be completed.
====

=== 5. Update the Vault Secret

NOTE: Without this step, your Vault instance will not be able to auto-unseal.

* Encode the Vault credentials
+
[source,shell]
----
export VAULT_UNSEAL_KEY="OLD_UNSEAL_KEY"
export VAULT_ROOT_TOKEN="OLD_ROOT_TOKEN"

echo -n "$VAULT_UNSEAL_KEY" | base64 -w0
echo -n "$VAULT_ROOT_TOKEN" | base64 -w0
----
* Update the Vault secret
+
[source,shell]
----
kubectl edit secret -n ${VAULT_INSTANCE_NAME} ${VAULT_INSTANCE_NAME}-seal
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd be nice to have a copy paste-able command here since updating secrets is a PITA. Especally if you encode a newline into the base64 by accident.

----
* Update the `vault-root` and `vault-unseal-0` keys to reflect the values you have just encoded
* Save the secret
HappyTetrahedron marked this conversation as resolved.
Show resolved Hide resolved

=== 6. Cleanup

* Reenable `backups.enabled` in the component configuration
6 changes: 6 additions & 0 deletions docs/modules/ROOT/pages/references/parameters.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -252,6 +252,12 @@ default:: `true`

Whether to do backups using k8up.

[WARNING]
.Manual Setup Required
====
include::partial$vault-warning.adoc[]
====

== `backup.schedule`

[horizontal]
Expand Down
1 change: 1 addition & 0 deletions docs/modules/ROOT/partials/nav.adoc
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
* xref:index.adoc[Home]
* xref:tutorials/install.adoc[Installation]
* xref:how-tos/lieutenant.adoc[Lieutenant Integration]
* xref:how-tos/restore-from-backup.adoc[Restore Vault from Backup]
* xref:references/parameters.adoc[Parameters]
8 changes: 8 additions & 0 deletions docs/modules/ROOT/partials/vault-warning.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
`component-vault` leverages k8up's application specific backups to create Vault snapshots.
Restoring such a snapshot is only possible by providing the Vault unseal key and root token.
HappyTetrahedron marked this conversation as resolved.
Show resolved Hide resolved
As these are sensitive secrets, they are not backed up as part of the component's automated backup process.

When setting up Vault, you must store the Vault unseal key and root token in a safe location in order to be able to restore backups.

In a running Vault instance, both of these secrets can be found in the `[instance name]-seal` secret in the Vault instance's namespace.
In order to have a reliable Vault backup, this secret must be backed up separately to a safe location.
Loading