diff --git a/docs/guides/mysql/pitr/_index.md b/docs/guides/mysql/pitr/_index.md index 00d65c58c5..5e06fcfef5 100644 --- a/docs/guides/mysql/pitr/_index.md +++ b/docs/guides/mysql/pitr/_index.md @@ -7,4 +7,4 @@ menu: parent: guides-mysql weight: 42 menu_name: docs_{{ .version }} ---- \ No newline at end of file +--- diff --git a/docs/guides/mysql/pitr/restic/archiver.md b/docs/guides/mysql/pitr/restic/archiver.md new file mode 100644 index 0000000000..b4dcc8b4d6 --- /dev/null +++ b/docs/guides/mysql/pitr/restic/archiver.md @@ -0,0 +1,468 @@ +--- +title: Continuous Archiving and Point-in-time Recovery Using Restic Driver +menu: + docs_{{ .version }}: + identifier: restic + name: Restic + parent: pitr-mysql + weight: 10 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + +> New to KubeDB? Please start [here](/docs/README.md). + +# KubeDB MySQL - Continuous Archiving and Point-in-time Recovery Using Restic Driver + +Here, we will demonstrate how to use KubeDB to provision a MySQL database with continuous archiving capabilities, also show point-in-time restoration. + +This process utilizes [Percona XtraBackup](https://www.percona.com/mysql/software/percona-xtrabackup), a robust tool for taking physical backups of MySQL databases, ensuring data integrity and consistency. We use XtraBackup as the base backup for our MySQL archiver. +Let's explore how we use XtraBackup in MySQL database archiving. +## Before You Begin + +To get started with archiving MySQL using Percona XtraBackup, you’ll need a Kubernetes cluster with the kubectl command-line tool configured to interact with it. If you don’t already have a cluster, you can easily create one using [kind](https://kind.sigs.k8s.io/docs/user/quick-start/). + +Next, install the `KubeDB` operator in your cluster by following the steps outlined [here](/docs/setup/README.md). + +To install the `KubeStash` operator in your cluster, follow the steps outlined [here](https://github.com/kubestash/installer/tree/master/charts/kubestash). + +To keep things isolated, this tutorial uses a separate namespace called `demo` throughout this tutorial. + +```bash +$ kubectl create ns demo +namespace/demo created +``` +> Note: The yaml files used in this tutorial are stored in [docs/guides/mysql/pitr/restic/yamls](https://github.com/kubedb/docs/tree/{{< param "info.version" >}}/docs/guides/mysql/pitr/restic/yamls) folder in GitHub repository [kubedb/docs](https://github.com/kubedb/docs). + +## Continuous Archiving +Continuous archiving involves making regular copies (or "archives") of the MySQL transaction log files.To ensure continuous archiving to a remote location we need prepare `BackupStorage`,`RetentionPolicy`,`MySQLArchiver` for the KubeDB Managed MySQL Databases. + +### BackupStorage +BackupStorage is a CR provided by KubeStash that can manage storage from various providers like GCS, S3, and more. +Here we are using AWS s3 bucket. +```yaml +apiVersion: storage.kubestash.com/v1alpha1 +kind: BackupStorage +metadata: + name: storage + namespace: demo +spec: + storage: + provider: s3 + s3: + endpoint: s3.amazonaws.com + bucket: mysql-xtrabackup + region: us-east-1 + prefix: my-demo + secretName: s3-secret + usagePolicy: + allowedNamespaces: + from: All + deletionPolicy: WipeOut +``` +Note: Before applying this yaml, verify that a bucket named `mysql-archiver` is already created on your bucket provider. + +```bash + $ kubectl apply -f backupstorage.yaml + backupstorage.storage.kubestash.com/storage created +``` + +### secrets for backup-storage +```yaml +apiVersion: v1 +kind: Secret +type: Opaque +metadata: + name: s3-secret + namespace: demo +stringData: + AWS_ACCESS_KEY_ID: "*************26CX" + AWS_SECRET_ACCESS_KEY: "************jj3lp" + AWS_ENDPOINT: s3.amazonaws.com +``` + +```bash + $ kubectl apply -f storage-secret.yaml + secret/s3-secret created +``` + +### Retention policy +RetentionPolicy is a CR provided by KubeStash that allows you to set how long you'd like to retain the backup data. + +```yaml +apiVersion: storage.kubestash.com/v1alpha1 +kind: RetentionPolicy +metadata: + name: mysql-retention-policy + namespace: demo +spec: + maxRetentionPeriod: "30d" + successfulSnapshots: + last: 10 + failedSnapshots: + last: 2 +``` +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/mysql/pitr/restic/yamls/retention-policy.yaml +retentionpolicy.storage.kubestash.com/mysql-retention-policy created +``` + +### MySQLArchiver +MySQLArchiver is a CR provided by KubeDB for managing the archiving of MySQL binlog files and performing physical backups + +```yaml +apiVersion: archiver.kubedb.com/v1alpha1 +kind: MySQLArchiver +metadata: + name: mysqlarchiver-sample + namespace: demo +spec: + pause: false + databases: + namespaces: + from: Selector + selector: + matchLabels: + kubernetes.io/metadata.name: demo + selector: + matchLabels: + archiver: "true" + retentionPolicy: + name: mysql-retention-policy + namespace: demo + encryptionSecret: + name: "encrypt-secret" + namespace: "demo" + fullBackup: + driver: "Restic" + scheduler: + successfulJobsHistoryLimit: 1 + failedJobsHistoryLimit: 1 + schedule: "0 0 * * *" + sessionHistoryLimit: 2 + manifestBackup: + scheduler: + successfulJobsHistoryLimit: 1 + failedJobsHistoryLimit: 1 + schedule: "0 0 * * *" + sessionHistoryLimit: 2 + backupStorage: + ref: + name: "storage" + namespace: "demo" +``` + +### EncryptionSecret + +```yaml +apiVersion: v1 +kind: Secret +type: Opaque +metadata: + name: encrypt-secret + namespace: demo +stringData: + RESTIC_PASSWORD: "changeit" +``` + +```bash + $ kubectl create -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/mysql/pitr/restic/yamls/encryptionSecret.yaml + secret/encrypt-secret created + + $ kubectl create -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/mysql/pitr/restic/yamls/mysqlarchiver.yaml + mysqlarchiver.archiver.kubedb.com/mysqlarchiver-sample created +``` + +# Deploy MySQL +We are now ready with the setup for continuous MySQL archiving. We will deploy a MySQL object that references the MySQL archiver object. + +```yaml +apiVersion: kubedb.com/v1 +kind: MySQL +metadata: + name: mysql + namespace: demo + labels: + archiver: "true" +spec: + version: "8.2.0" + replicas: 3 + topology: + mode: GroupReplication + storageType: Durable + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + deletionPolicy: WipeOut +``` + +```bash + $ kubectl create -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/mysql/pitr/restic/yamls/mysql.yaml + mysql.kubedb.com/mysql created + ``` + +```bash +$ kubectl get pod -n demo +NAME READY STATUS RESTARTS AGE +mysql-0 2/2 Running 0 15m +mysql-1 2/2 Running 0 15m +mysql-2 2/2 Running 0 15m + +``` + +Once the MySQL database is ready and backup storage is prepared, the MySQL Archiver object will trigger the KubeDB Operator to create a sidekick pod. Subsequently, the KubeStash Operator will generate a full backup along with a manifest backup. + +```bash +$ kubectl get pod -n demo +NAME READY STATUS RESTARTS AGE +mysql-0 2/2 Running 0 15m +mysql-1 2/2 Running 0 15m +mysql-2 2/2 Running 0 15m +mysql-archiver-full-backup-1733120326-4n8nd 0/1 Completed 0 10m +mysql-archiver-manifest-backup-1733120326-9gw4f 0/1 Completed 0 10m +mysql-sidekick 1/1 Running 0 10m +retention-policy-mysql-archiver-full-backup-1733120326-7rx9t 0/1 Completed 0 9m31s +retention-policy-mysql-archiver-manifest-backup-1733120326l79mb 0/1 Completed 0 9m56s +``` + +Here, + +`mysql-sidekick` pod is responsible for uploading binlog files + +`mysql-archiver-full-backup-1733120326-4n8nd` pod is responsible for creating the base backup of MySQL. + +`mysql-archiver-manifest-backup-1733120326-9gw4f` is the pod of the manifest backup related to MySQL object. + +`retention-policy-mysql-archiver-full-backup-1733120326-7rx9t` will automatically clean up previous full-backup snapshots according to the rules defined in the `mysql-retention-policy` custom resource (CR). + +`retention-policy-mysql-archiver-manifest-backup-1733120326l79mb` will automatically clean up previous manifest-backup snapshots according to the rules specified in the `mysql-retention-policy` custom resource (CR). + +### Validate BackupConfiguration and BackupSession + +```bash + +$ kubectl get backupconfigurations -n demo + +NAME PHASE PAUSED AGE +mysql-archiver Ready 14m + +$ kubectl get backupsession -n demo +NAME INVOKER-TYPE INVOKER-NAME PHASE DURATION AGE +mysql-archiver-full-backup-1733120326 BackupConfiguration mysql-archiver Succeeded 50s 14m +mysql-archiver-manifest-backup-1733120326 BackupConfiguration mysql-archiver Succeeded 25s 14m + +$ kubectl get repository.storage.kubestash.com -n demo +NAME INTEGRITY SNAPSHOT-COUNT SIZE PHASE LAST-SUCCESSFUL-BACKUP AGE +mysql-full true 1 2.073 KiB Ready 14m 14m +mysql-manifest true 1 2.073 KiB Ready 14m 14m + +``` + +## Data Insert and Switch Binlog File +After each and every binlog switch the binlog files will be uploaded to backup storage + +```bash +$ kubectl exec -it -n demo mysql-0 -- bash + +bash-4.4$ mysql -uroot -p$MYSQL_ROOT_PASSWORD + +mysql> create database hello; + +mysql> use hello; + +mysql> CREATE TABLE `demo_table`( + -> `id` BIGINT(20) NOT NULL, + -> `name` VARCHAR(255) DEFAULT NULL, + -> PRIMARY KEY (`id`) + -> ); + +mysql> INSERT INTO `demo_table` (`id`, `name`) + -> VALUES + -> (1, 'John'), + -> (2, 'Jane'), + -> (3, 'Bob'), + -> (4, 'Alice'), + -> (5, 'Charlie'), + -> (6, 'Diana'), + -> (7, 'Eve'), + -> (8, 'Frank'), + -> (9, 'Grace'), + -> (10, 'Henry'); + + +mysql> select now(); ++---------------------+ +| now() | ++---------------------+ +| 2024-12-02 06:38:42 | ++---------------------+ ++---------------------+ + +mysql> select count(*) from demo_table; ++----------+ +| count(*) | ++----------+ +| 10 | ++----------+ + +``` + +> At this point We have 10 rows in our newly created table `demo_table` on database `hello` + +## Point-in-time Recovery +Point-In-Time Recovery allows you to restore a MySQL database to a specific point in time using the archived transaction logs. This is particularly useful in scenarios where you need to recover to a state just before a specific error or data corruption occurred. +Let's say accidentally our db drops the the table `demo_table` and we want to restore that. + +```bash +$ kubectl exec -it -n demo mysql-0 -- bash + +mysql> drop table demo_table; + +mysql> flush logs; + +``` +We can't restore from a full backup since at this point no full backup was perform. so we can choose a specific time in which time we want to restore.We can get the specific time from the binlog that archived in the backup storage . Go to the binlog file and find where to store. You can parse binlog-files using `mysqlbinlog`. + +For the demo I will use the previous time we get from `select now()` + +```bash +mysql> select now(); ++---------------------+ +| now() | ++---------------------+ +| 2024-12-02 06:38:42 | ++---------------------+ +``` + +### ReplicationStrategy + +The ReplicationStrategy determines how MySQL restores are managed when using the Restic driver in a group replication setup. We support three strategies: `none`, `sync`, and `fscopy`, with `none` being the default. + +To configure the desired strategy, set the `spec.init.archiver.replicationStrategy` field in your MySQL Database manifest. These strategies are applicable only when restoring a MySQL database in group replication mode. + +**Strategies Overview:** + +***none*** + +Each MySQL replica independently restores the base backup and binlog files. After completing the restore process, the replicas individually join the replication group. + +***sync*** + +The base backup and binlog files are restored exclusively on pod-0. Other replicas then synchronize their data by leveraging the MySQL clone plugin to replicate from pod-0. + +***fscopy*** + +The base backup and binlog files are restored on pod-0. The data is then copied from pod-0's data directory to the data directories of other replicas using file system copy. Once the data transfer is complete, the group replication process begins. + +Please note that `fscopy` does not support cross-zone operations. + +Choose the replication strategy that best fits your restoration and replication requirements. On this demonstration, we have used the `sync` replication strategy. + +### Restore MySQL +```yaml +apiVersion: kubedb.com/v1 +kind: MySQL +metadata: + name: restore-mysql + namespace: demo +spec: + init: + archiver: + replicationStrategy: sync + encryptionSecret: + name: encrypt-secret + namespace: demo + fullDBRepository: + name: mysql-full + namespace: demo + recoveryTimestamp: "2024-12-02T06:38:42Z" + version: "8.2.0" + replicas: 3 + topology: + mode: GroupReplication + storageType: Durable + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + deletionPolicy: WipeOut + +``` + +```bash + $ kubectl create -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/mysql/pitr/restic/yamls/mysql-restore.yaml + mysql.kubedb.com/restore-mysql created + ``` + +**Check for Restored MySQL** + +```bash +$ kubectl get pod -n demo +data-restore-mysql-0-pvc-restorer-5vtj7 0/1 Completed 0 7m40s +restore-mysql-0 2/2 Running 0 6m40s +restore-mysql-1 2/2 Running 0 5m37s +restore-mysql-2 2/2 Running 0 5m21s +restore-mysql-binlog-restorer-0 0/2 Completed 0 5m58s +restore-mysql-manifest-restorer-pzx5z 0/1 Completed 0 6m54s +``` + +The pod `data-restore-mysql-0-pvc-restorer-5vtj7` is responsible for restoring the base backup. + +The pod `restore-mysql-binlog-restorer-0` is responsible for restoring the binlog file. + +```bash +$ kubectl get mysql -n demo +NAME VERSION STATUS AGE +mysql.kubedb.com/mysql 8.2.0 Ready 32m +mysql.kubedb.com/restore-mysql 8.2.0 Ready 2m53s +``` + +**Validating Data on Restored MySQL** + +```bash +$ kubectl exec -it -n demo restore-mysql-0 -- bash +bash-4.4$ mysql -uroot -p$MYSQL_ROOT_PASSWORD + +mysql> use hello + +mysql> select count(*) from demo_table; ++----------+ +| count(*) | ++----------+ +| 10 | ++----------+ +1 row in set (0.00 sec) + +``` + +**so we are able to successfully recover from a disaster** + + +## Cleaning up + +To cleanup the Kubernetes resources created by this tutorial, run: + +```bash +$ kubectl delete -n demo mysql/mysql +$ kubectl delete -n demo mysql/restore-mysql +$ kubectl delete -n demo backupstorage.storage.kubestash.com/storage +$ kubectl delete -n demo mysqlarchiver/mysqlarchiver-sample +$ kubectl delete ns demo +``` + +## Next Steps + +- Learn about [backup and restore](/docs/guides/mysql/backup/stash/overview/index.md) MySQL database using Stash. +- Learn about initializing [MySQL with Script](/docs/guides/mysql/initialization/script_source.md). +- Learn about [custom MySQLVersions](/docs/guides/mysql/custom-versions/setup.md). +- Want to setup MySQL cluster? Check how to [configure Highly Available MySQL Cluster](/docs/guides/mysql/clustering/ha_cluster.md) +- Monitor your MySQL database with KubeDB using [built-in Prometheus](/docs/guides/mysql/monitoring/using-builtin-prometheus.md). +- Monitor your MySQL database with KubeDB using [Prometheus operator](/docs/guides/mysql/monitoring/using-prometheus-operator.md). +- Detail concepts of [MySQL object](/docs/guides/mysql/concepts/mysql.md). +- Use [private Docker registry](/docs/guides/mysql/private-registry/using-private-registry.md) to deploy MySQL with KubeDB. +- Want to hack on KubeDB? Check our [contribution guidelines](/docs/CONTRIBUTING.md). \ No newline at end of file diff --git a/docs/guides/mysql/pitr/restic/yamls/backupstorage-restricted-ns.yaml b/docs/guides/mysql/pitr/restic/yamls/backupstorage-restricted-ns.yaml new file mode 100644 index 0000000000..a865afebbf --- /dev/null +++ b/docs/guides/mysql/pitr/restic/yamls/backupstorage-restricted-ns.yaml @@ -0,0 +1,32 @@ +apiVersion: storage.kubestash.com/v1alpha1 +kind: BackupStorage +metadata: + name: storage + namespace: demo +spec: + storage: + provider: s3 + s3: + endpoint: s3.amazonaws.com + bucket: mysql-xtrabackup + region: us-east-1 + prefix: my-demo + secretName: s3-secret + usagePolicy: + allowedNamespaces: + from: All + deletionPolicy: WipeOut + # for restricted namespace + runtimeSettings: + pod: + securityContext: + runAsUser: 65535 + runAsNonRoot: true + seccompProfile: + type: RuntimeDefault + container: + securityContext: + allowPrivilegeEscalation: false + capabilities: + drop: + - ALL \ No newline at end of file diff --git a/docs/guides/mysql/pitr/restic/yamls/backupstorage.yaml b/docs/guides/mysql/pitr/restic/yamls/backupstorage.yaml new file mode 100644 index 0000000000..c7f0835bac --- /dev/null +++ b/docs/guides/mysql/pitr/restic/yamls/backupstorage.yaml @@ -0,0 +1,18 @@ +apiVersion: storage.kubestash.com/v1alpha1 +kind: BackupStorage +metadata: + name: storage + namespace: demo +spec: + storage: + provider: s3 + s3: + endpoint: s3.amazonaws.com + bucket: mysql-xtrabackup + region: us-east-1 + prefix: my-demo + secretName: s3-secret + usagePolicy: + allowedNamespaces: + from: All + deletionPolicy: WipeOut diff --git a/docs/guides/mysql/pitr/yamls/encryptionSecret.yaml b/docs/guides/mysql/pitr/restic/yamls/encryptionSecret.yaml similarity index 100% rename from docs/guides/mysql/pitr/yamls/encryptionSecret.yaml rename to docs/guides/mysql/pitr/restic/yamls/encryptionSecret.yaml diff --git a/docs/guides/mysql/pitr/restic/yamls/mysql-restore.yaml b/docs/guides/mysql/pitr/restic/yamls/mysql-restore.yaml new file mode 100644 index 0000000000..cf0eca6973 --- /dev/null +++ b/docs/guides/mysql/pitr/restic/yamls/mysql-restore.yaml @@ -0,0 +1,28 @@ +apiVersion: kubedb.com/v1 +kind: MySQL +metadata: + name: restore-mysql + namespace: demo +spec: + init: + archiver: + replicationStrategy: sync + encryptionSecret: + name: encrypt-secret + namespace: demo + fullDBRepository: + name: mysql-full + namespace: demo + recoveryTimestamp: "2024-12-02T06:38:42Z" + version: "8.2.0" + replicas: 3 + topology: + mode: GroupReplication + storageType: Durable + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + deletionPolicy: WipeOut \ No newline at end of file diff --git a/docs/guides/mysql/pitr/restic/yamls/mysql.yaml b/docs/guides/mysql/pitr/restic/yamls/mysql.yaml new file mode 100644 index 0000000000..2a48a283af --- /dev/null +++ b/docs/guides/mysql/pitr/restic/yamls/mysql.yaml @@ -0,0 +1,24 @@ +apiVersion: kubedb.com/v1 +kind: MySQL +metadata: + name: mysql + namespace: demo + labels: + archiver: "true" +spec: + version: "8.2.0" + replicas: 3 + topology: + mode: GroupReplication + storageType: Durable + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + archiver: + ref: + name: mysqlarchiver-sample + namespace: demo + deletionPolicy: WipeOut \ No newline at end of file diff --git a/docs/guides/mysql/pitr/restic/yamls/mysqlarchiver.yaml b/docs/guides/mysql/pitr/restic/yamls/mysqlarchiver.yaml new file mode 100644 index 0000000000..bae04e2c0b --- /dev/null +++ b/docs/guides/mysql/pitr/restic/yamls/mysqlarchiver.yaml @@ -0,0 +1,39 @@ +apiVersion: archiver.kubedb.com/v1alpha1 +kind: MySQLArchiver +metadata: + name: mysqlarchiver-sample + namespace: demo +spec: + pause: false + databases: + namespaces: + from: Selector + selector: + matchLabels: + kubernetes.io/metadata.name: demo + selector: + matchLabels: + archiver: "true" + retentionPolicy: + name: mysql-retention-policy + namespace: demo + encryptionSecret: + name: "encrypt-secret" + namespace: "demo" + fullBackup: + driver: "Restic" + scheduler: + successfulJobsHistoryLimit: 1 + failedJobsHistoryLimit: 1 + schedule: "0 0 * * *" + sessionHistoryLimit: 2 + manifestBackup: + scheduler: + successfulJobsHistoryLimit: 1 + failedJobsHistoryLimit: 1 + schedule: "0 0 * * *" + sessionHistoryLimit: 2 + backupStorage: + ref: + name: "storage" + namespace: "demo" \ No newline at end of file diff --git a/docs/guides/mysql/pitr/restic/yamls/retention-policy.yaml b/docs/guides/mysql/pitr/restic/yamls/retention-policy.yaml new file mode 100644 index 0000000000..1cacd4ce20 --- /dev/null +++ b/docs/guides/mysql/pitr/restic/yamls/retention-policy.yaml @@ -0,0 +1,11 @@ +apiVersion: storage.kubestash.com/v1alpha1 +kind: RetentionPolicy +metadata: + name: mysql-retention-policy + namespace: demo +spec: + maxRetentionPeriod: "30d" + successfulSnapshots: + last: 10 + failedSnapshots: + last: 2 diff --git a/docs/guides/mysql/pitr/archiver.md b/docs/guides/mysql/pitr/volumesnapshot/archiver.md similarity index 60% rename from docs/guides/mysql/pitr/archiver.md rename to docs/guides/mysql/pitr/volumesnapshot/archiver.md index 4be52657b9..659f02da9e 100644 --- a/docs/guides/mysql/pitr/archiver.md +++ b/docs/guides/mysql/pitr/volumesnapshot/archiver.md @@ -2,8 +2,8 @@ title: Continuous Archiving and Point-in-time Recovery menu: docs_{{ .version }}: - identifier: pitr-mysql-archiver - name: Overview + identifier: volumesnapshot + name: VolumeSnapshot parent: pitr-mysql weight: 10 menu_name: docs_{{ .version }} @@ -12,9 +12,9 @@ section_menu_id: guides > New to KubeDB? Please start [here](/docs/README.md). -# KubeDB MySQL - Continuous Archiving and Point-in-time Recovery +# KubeDB MySQL - Continuous Archiving and Point-in-time Recovery using VolumeSnapshot -Here, will show you how to use KubeDB to provision a MySQL to Archive continuously and Restore point-in-time. +Here, will show you how to use KubeDB to provision a MySQL to Archive continuously, also show point-in-time restoration. ## Before You Begin @@ -24,8 +24,6 @@ Now,install `KubeDB` operator in your cluster following the steps [here](/docs/s To install `KubeStash` operator in your cluster following the steps [here](https://github.com/kubestash/installer/tree/master/charts/kubestash). -To install `SideKick` in your cluster following the steps [here](https://github.com/kubeops/installer/tree/master/charts/sidekick). - To install `External-snapshotter` in your cluster following the steps [here](https://github.com/kubernetes-csi/external-snapshotter/tree/release-5.0). To keep things isolated, this tutorial uses a separate namespace called `demo` throughout this tutorial. @@ -34,9 +32,9 @@ To keep things isolated, this tutorial uses a separate namespace called `demo` t $ kubectl create ns demo namespace/demo created ``` -> Note: The yaml files used in this tutorial are stored in [docs/guides/mysql/remote-replica/yamls](https://github.com/kubedb/docs/tree/{{< param "info.version" >}}/docs/guides/mysql/remote-replica/yamls) folder in GitHub repository [kubedb/docs](https://github.com/kubedb/docs). +> Note: The yaml files used in this tutorial are stored in [docs/guides/mysql/pitr/volumesnapshot/yamls](https://github.com/kubedb/docs/tree/{{< param "info.version" >}}/docs/guides/mysql/pitr/volumesnapshot/yamls) folder in GitHub repository [kubedb/docs](https://github.com/kubedb/docs). -## continuous archiving +## Continuous Archiving Continuous archiving involves making regular copies (or "archives") of the MySQL transaction log files.To ensure continuous archiving to a remote location we need prepare `BackupStorage`,`RetentionPolicy`,`MySQLArchiver` for the KubeDB Managed MySQL Databases. ### BackupStorage @@ -46,27 +44,28 @@ BackupStorage is a CR provided by KubeStash that can manage storage from various apiVersion: storage.kubestash.com/v1alpha1 kind: BackupStorage metadata: - name: linode-storage + name: storage namespace: demo spec: storage: provider: s3 s3: - bucket: mehedi-mysql-wal-g - endpoint: https://ap-south-1.linodeobjects.com - region: ap-south-1 - prefix: backup - secretName: storage + endpoint: s3.amazonaws.com + bucket: mysql-archiver + region: us-east-1 + prefix: my-demo + secretName: s3-secret usagePolicy: allowedNamespaces: from: All - default: true deletionPolicy: WipeOut ``` +Note: Before applying this yaml, verify that a bucket named `mysql-archiver` is already created on your bucket provider. + ```bash $ kubectl apply -f backupstorage.yaml - backupstorage.storage.kubestash.com/linode-storage created + backupstorage.storage.kubestash.com/storage created ``` ### secrets for backup-storage @@ -75,17 +74,17 @@ apiVersion: v1 kind: Secret type: Opaque metadata: - name: storage + name: s3-secret namespace: demo stringData: AWS_ACCESS_KEY_ID: "*************26CX" AWS_SECRET_ACCESS_KEY: "************jj3lp" - AWS_ENDPOINT: https://ap-south-1.linodeobjects.com + AWS_ENDPOINT: s3.amazonaws.com ``` ```bash $ kubectl apply -f storage-secret.yaml - secret/storage created + secret/s3-secret created ``` ### Retention policy @@ -100,7 +99,7 @@ metadata: spec: maxRetentionPeriod: "30d" successfulSnapshots: - last: 100 + last: 10 failedSnapshots: last: 2 ``` @@ -125,16 +124,16 @@ spec: from: Selector selector: matchLabels: - kubernetes.io/metadata.name: demo + kubernetes.io/metadata.name: demo selector: matchLabels: archiver: "true" retentionPolicy: name: mysql-retention-policy namespace: demo - encryptionSecret: + encryptionSecret: name: "encrypt-secret" - namespace: "demo" + namespace: "demo" fullBackup: driver: "VolumeSnapshotter" task: @@ -148,12 +147,12 @@ spec: manifestBackup: scheduler: successfulJobsHistoryLimit: 1 - failedJobsHistoryLimit: 1 + failedJobsHistoryLimit: 1 schedule: "*/30 * * * *" sessionHistoryLimit: 2 backupStorage: ref: - name: "linode-storage" + name: "storage" namespace: "demo" ``` @@ -172,12 +171,15 @@ stringData: ``` ```bash - $ kubectl create -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/mysql/pirt/yamls/mysqlarchiver.yaml + $ kubectl create -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/mysql/pitr/volumesnapshot/yamls/encryptionSecret.yaml + secret/encrypt-secret created + + $ kubectl create -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/mysql/pitr/volumesnapshot/yamls/mysqlarchiver.yaml mysqlarchiver.archiver.kubedb.com/mysqlarchiver-sample created - $ kubectl create -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/mysql/pirt/yamls/encryptionSecret.yaml + ``` -## Ensure volumeSnapshotClass +## Ensure VolumeSnapshotClass ```bash $ kubectl get volumesnapshotclasses @@ -205,8 +207,10 @@ $ kubectl apply -f volumesnapshotclass.yaml volumesnapshotclass.snapshot.storage.k8s.io/longhorn-snapshot-vsc unchanged ``` +Note: Ensure that the VolumeSnapshotClass is provisioned with the same storage class driver used for provisioning your MySQL database. In our case, we are using the `longhorn` storageclass as our database provisioner, with the driver set to `driver.longhorn.io`. + # Deploy MySQL -So far we are ready with setup for continuously archive MySQL, We deploy a mysqlql referring the MySQL archiver object +We are now ready with the setup for continuous MySQL archiving. We will deploy a MySQL object that references the MySQL archiver object. ```yaml apiVersion: kubedb.com/v1 @@ -217,8 +221,6 @@ metadata: labels: archiver: "true" spec: - authSecret: - name: my-auth version: "8.2.0" replicas: 3 topology: @@ -238,45 +240,69 @@ spec: deletionPolicy: WipeOut ``` +```bash +$ kubectl get pod -n demo +NAME READY STATUS RESTARTS AGE +mysql-0 2/2 Running 0 28h +mysql-1 2/2 Running 0 28h +mysql-2 2/2 Running 0 28h + +``` + +Once the MySQL database is ready and backup storage is prepared, the MySQL Archiver object will trigger the KubeDB Operator to create a sidekick pod. Subsequently, the KubeStash Operator will generate a full backup along with a manifest backup. ```bash $ kubectl get pod -n demo -NAME READY STATUS RESTARTS AGE -mysql-0 2/2 Running 0 28h -mysql-1 2/2 Running 0 28h -mysql-2 2/2 Running 0 28h -mysql-backup-config-full-backup-1703680982-vqf7c 0/1 Completed 0 28h -mysql-backup-config-manifest-1703680982-62x97 0/1 Completed 0 28h -mysql-sidekick 1/1 Running 0 28h +NAME READY STATUS RESTARTS AGE +mysql-0 2/2 Running 0 28h +mysql-1 2/2 Running 0 28h +mysql-2 2/2 Running 0 28h +mysql-archiver-full-backup-1733206003-hq4pb 0/1 Completed 0 28h +mysql-archiver-manifest-backup-1733206003-q78jj 0/1 Completed 0 28h +mysql-sidekick 1/1 Running 0 28h +retention-policy-mysql-archiver-full-backup-1733206003-b2b42 0/1 Completed 0 28h +retention-policy-mysql-archiver-manifest-backup-1733206003skwqc 0/1 Completed 0 28h + ``` -`mysql-sidekick` is responsible for uploading binlog files +`mysql-sidekick` pod is responsible for uploading binlog files + +`mysql-backup-config-full-backup-1703680982-vqf7c` is the pod of volumes levels backups for MySQL. + +`mysql-backup-config-manifest-1703680982-62x97` is the pod of the manifest backup related to MySQL object -`mysql-backup-config-full-backup-1703680982-vqf7c` are the pod of volumes levels backups for MySQL. +`retention-policy-mysql-archiver-full-backup-1733206003-b2b42` will automatically clean up previous full-backup of volumesnapshots according to the rules defined in the `mysql-retention-policy` custom resource (CR). -`mysql-backup-config-manifest-1703680982-62x97` are the pod of the manifest backup related to MySQL object +`retention-policy-mysql-archiver-manifest-backup-1733206003skwqc` will automatically clean up previous manifest-backup snapshots according to the rules specified in the `mysql-retention-policy` custom resource (CR). -### validate BackupConfiguration and VolumeSnapshots + + +### Validate BackupConfiguration and VolumeSnapshots ```bash $ kubectl get backupconfigurations -n demo NAME PHASE PAUSED AGE -mysql-backup-config Ready 2m43s +mysql-archiver Ready 2m43s $ kubectl get backupsession -n demo -NAME INVOKER-TYPE INVOKER-NAME PHASE DURATION AGE -mysql-backup-config-full-backup-1702388088 BackupConfiguration mysql-backup-config Succeeded 74s -mysql-backup-config-manifest-1702388088 BackupConfiguration mysql-backup-config Succeeded 74s +NAME INVOKER-TYPE INVOKER-NAME PHASE DURATION AGE +mysql-archiver-full-backup-1733206003 BackupConfiguration mysql-backup-config Succeeded 74s +mysql-archiver-manifest-backup-1733206003 BackupConfiguration mysql-backup-config Succeeded 74s kubectl get volumesnapshots -n demo NAME READYTOUSE SOURCEPVC SOURCESNAPSHOTCONTENT RESTORESIZE SNAPSHOTCLASS SNAPSHOTCONTENT CREATIONTIME AGE mysql-1702388096 true data-mysql-1 1Gi longhorn-snapshot-vsc snapcontent-735e97ad-1dfa-4b70-b416-33f7270d792c 2m5s 2m5s + +$ kubectl get repository.storage.kubestash.com -n demo +NAME INTEGRITY SNAPSHOT-COUNT SIZE PHASE LAST-SUCCESSFUL-BACKUP AGE +mysql-full true 1 2.073 KiB Ready 2m43s 2m43s +mysql-manifest true 1 2.073 KiB Ready 2m43s 2m43s ``` -## data insert and switch wal -After each and every wal switch the wal files will be uploaded to backup storage +## Data Insert and Switch Binlog File +After each and every binlog switch the binlog files will be uploaded to backup storage ```bash $ kubectl exec -it -n demo mysql-0 -- bash @@ -310,11 +336,7 @@ mysql> select now(); +---------------------+ | now() | +---------------------+ -| 2023-12-28 17:10:54 |mysql> select now(); -+---------------------+ -| now() | -+---------------------+ -| 2023-12-28 17:10:54 | +| 2024-12-03 06:09:34 | +---------------------+ +---------------------+ @@ -331,7 +353,7 @@ mysql> select count(*) from demo_table; ## Point-in-time Recovery Point-In-Time Recovery allows you to restore a MySQL database to a specific point in time using the archived transaction logs. This is particularly useful in scenarios where you need to recover to a state just before a specific error or data corruption occurred. -Let's say accidentally our dba drops the the table tab_1 and we want to restore. +Let's say accidentally our db drops the table `demo_table` and we want to restore that. ```bash $ kubectl exec -it -n demo mysql-0 -- bash @@ -351,11 +373,38 @@ mysql> select now(); +---------------------+ | now() | +---------------------+ -| 2023-12-28 17:10:54 | +| 2024-12-03 06:09:34 | +---------------------+ ``` -### Restore MySQL +### ReplicationStrategy + +The ReplicationStrategy determines how MySQL restores are managed when using the VolumeSnapshot. We support three strategies: `none`, `sync`, and `fscopy`, with `none` being the default. + +To configure the desired strategy, set the `spec.init.archiver.replicationStrategy` field in your configuration. These strategies are applicable only when restoring a MySQL database in group replication mode. + +**Strategies Overview:** + +***none*** + +Each MySQL replica independently restores the base backup volumesnapshot and binlog files. After completing the restore process, the replicas individually join the replication group. + +***sync*** + +The base backup volumesnapshot and binlog files are restored exclusively on pod-0. Other replicas then synchronize their data by leveraging the MySQL clone plugin to replicate from pod-0. + +***fscopy*** + +The base backup and binlog files are restored on pod-0. The data is then copied from pod-0's data directory to the data directories of other replicas using file system copy. Once the data transfer is complete, the group replication process begins. Please note that `fscopy` does not support cross-zone operations. + +***clone*** + +If you have a different type of base backup(ex: VolumeSnapshot, Restic), the clone process will ensure that the VolumeSnapshot is restored as the base backup. Each MySQL replica independently restores the base backup volumesnapshot and binlog files. After completing the restore process, the replicas individually join the replication group. + + +Choose the replication strategy that best fits your restoration and replication requirements. On this demonstration, we have used the sync replication strategy. + +### Restore MySQL ```yaml apiVersion: kubedb.com/v1 kind: MySQL @@ -365,13 +414,14 @@ metadata: spec: init: archiver: + replicationStrategy: sync encryptionSecret: name: encrypt-secret namespace: demo fullDBRepository: - name: mysql-repository + name: mysql-full namespace: demo - recoveryTimestamp: "2023-12-28T17:10:54Z" + recoveryTimestamp: "2024-12-03T06:09:34Z" version: "8.2.0" replicas: 3 topology: @@ -383,7 +433,7 @@ spec: - ReadWriteOnce resources: requests: - storage: 10Gi + storage: 1Gi deletionPolicy: WipeOut ``` @@ -408,8 +458,8 @@ restore-mysql-restoresession-lk6jq 0/1 Completed 0 ```bash $ kubectl get mysql -n demo NAME VERSION STATUS AGE -mysql 8.2.0 Ready 28h -restore-mysql 8.2.0 Ready 5m37s +mysql 8.2.0 Ready 28h +restore-mysql 8.2.0 Ready 5m37s ``` **Validating data on Restored MySQL** @@ -432,6 +482,8 @@ mysql> select count(*) from demo_table; **so we are able to successfully recover from a disaster** + + ## Cleaning up To cleanup the Kubernetes resources created by this tutorial, run: @@ -439,8 +491,8 @@ To cleanup the Kubernetes resources created by this tutorial, run: ```bash $ kubectl delete -n demo mysql/mysql $ kubectl delete -n demo mysql/restore-mysql -$ kubectl delete -n demo backupstorage -$ kubectl delete -n demo mysqlarchiver +$ kubectl delete -n demo backupstorage/storage +$ kubectl delete -n demo mysqlarchiver/mysqlarchiver-sample $ kubectl delete ns demo ``` diff --git a/docs/guides/mysql/pitr/volumesnapshot/yamls/backupstorage-restricted-ns.yaml b/docs/guides/mysql/pitr/volumesnapshot/yamls/backupstorage-restricted-ns.yaml new file mode 100644 index 0000000000..ad178cffdd --- /dev/null +++ b/docs/guides/mysql/pitr/volumesnapshot/yamls/backupstorage-restricted-ns.yaml @@ -0,0 +1,32 @@ +apiVersion: storage.kubestash.com/v1alpha1 +kind: BackupStorage +metadata: + name: storage + namespace: demo +spec: + storage: + provider: s3 + s3: + endpoint: s3.amazonaws.com + bucket: mysql-archiver + region: us-east-1 + prefix: my-demo + secretName: s3-secret + usagePolicy: + allowedNamespaces: + from: All + deletionPolicy: WipeOut + # for restricted namespace + runtimeSettings: + pod: + securityContext: + runAsUser: 65535 + runAsNonRoot: true + seccompProfile: + type: RuntimeDefault + container: + securityContext: + allowPrivilegeEscalation: false + capabilities: + drop: + - ALL \ No newline at end of file diff --git a/docs/guides/mysql/pitr/volumesnapshot/yamls/backupstorage.yaml b/docs/guides/mysql/pitr/volumesnapshot/yamls/backupstorage.yaml new file mode 100644 index 0000000000..d5c693628b --- /dev/null +++ b/docs/guides/mysql/pitr/volumesnapshot/yamls/backupstorage.yaml @@ -0,0 +1,18 @@ +apiVersion: storage.kubestash.com/v1alpha1 +kind: BackupStorage +metadata: + name: storage + namespace: demo +spec: + storage: + provider: s3 + s3: + endpoint: s3.amazonaws.com + bucket: mysql-archiver + region: us-east-1 + prefix: my-demo + secretName: s3-secret + usagePolicy: + allowedNamespaces: + from: All + deletionPolicy: WipeOut diff --git a/docs/guides/mysql/pitr/volumesnapshot/yamls/encryptionSecret.yaml b/docs/guides/mysql/pitr/volumesnapshot/yamls/encryptionSecret.yaml new file mode 100644 index 0000000000..4eb0c25bdb --- /dev/null +++ b/docs/guides/mysql/pitr/volumesnapshot/yamls/encryptionSecret.yaml @@ -0,0 +1,8 @@ +apiVersion: v1 +kind: Secret +type: Opaque +metadata: + name: encrypt-secret + namespace: demo +stringData: + RESTIC_PASSWORD: "changeit" diff --git a/docs/guides/mysql/pitr/volumesnapshot/yamls/mysql-restore.yaml b/docs/guides/mysql/pitr/volumesnapshot/yamls/mysql-restore.yaml new file mode 100644 index 0000000000..fce3703974 --- /dev/null +++ b/docs/guides/mysql/pitr/volumesnapshot/yamls/mysql-restore.yaml @@ -0,0 +1,29 @@ +apiVersion: kubedb.com/v1 +kind: MySQL +metadata: + name: restore-mysql + namespace: demo +spec: + init: + archiver: + replicationStrategy: sync + encryptionSecret: + name: encrypt-secret + namespace: demo + fullDBRepository: + name: mysql-full + namespace: demo + recoveryTimestamp: "2024-12-03T06:09:34Z" + version: "8.2.0" + replicas: 3 + topology: + mode: GroupReplication + storageType: Durable + storage: + storageClassName: "longhorn" + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + deletionPolicy: WipeOut \ No newline at end of file diff --git a/docs/guides/mysql/pitr/volumesnapshot/yamls/mysql.yaml b/docs/guides/mysql/pitr/volumesnapshot/yamls/mysql.yaml new file mode 100644 index 0000000000..06fe8780e9 --- /dev/null +++ b/docs/guides/mysql/pitr/volumesnapshot/yamls/mysql.yaml @@ -0,0 +1,25 @@ +apiVersion: kubedb.com/v1 +kind: MySQL +metadata: + name: mysql + namespace: demo + labels: + archiver: "true" +spec: + version: "8.2.0" + replicas: 3 + topology: + mode: GroupReplication + storageType: Durable + storage: + storageClassName: "longhorn" + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 10Gi + archiver: + ref: + name: mysqlarchiver-sample + namespace: demo + deletionPolicy: WipeOut \ No newline at end of file diff --git a/docs/guides/mysql/pitr/yamls/mysqlarchiver.yaml b/docs/guides/mysql/pitr/volumesnapshot/yamls/mysqlarchiver.yaml similarity index 97% rename from docs/guides/mysql/pitr/yamls/mysqlarchiver.yaml rename to docs/guides/mysql/pitr/volumesnapshot/yamls/mysqlarchiver.yaml index 4fe652feed..e12cd519bd 100644 --- a/docs/guides/mysql/pitr/yamls/mysqlarchiver.yaml +++ b/docs/guides/mysql/pitr/volumesnapshot/yamls/mysqlarchiver.yaml @@ -38,5 +38,5 @@ spec: sessionHistoryLimit: 2 backupStorage: ref: - name: "linode-storage" + name: "storage" namespace: "demo" \ No newline at end of file diff --git a/docs/guides/mysql/pitr/yamls/retentionPolicy.yaml b/docs/guides/mysql/pitr/volumesnapshot/yamls/retentionPolicy.yaml similarity index 93% rename from docs/guides/mysql/pitr/yamls/retentionPolicy.yaml rename to docs/guides/mysql/pitr/volumesnapshot/yamls/retentionPolicy.yaml index 7e08dccb08..1cacd4ce20 100644 --- a/docs/guides/mysql/pitr/yamls/retentionPolicy.yaml +++ b/docs/guides/mysql/pitr/volumesnapshot/yamls/retentionPolicy.yaml @@ -6,6 +6,6 @@ metadata: spec: maxRetentionPeriod: "30d" successfulSnapshots: - last: 100 + last: 10 failedSnapshots: last: 2 diff --git a/docs/guides/mysql/pitr/yamls/voluemsnapshotclass.yaml b/docs/guides/mysql/pitr/volumesnapshot/yamls/voluemsnapshotclass.yaml similarity index 100% rename from docs/guides/mysql/pitr/yamls/voluemsnapshotclass.yaml rename to docs/guides/mysql/pitr/volumesnapshot/yamls/voluemsnapshotclass.yaml diff --git a/docs/guides/mysql/pitr/yamls/backupstorage.yaml b/docs/guides/mysql/pitr/yamls/backupstorage.yaml deleted file mode 100644 index 686860f4b0..0000000000 --- a/docs/guides/mysql/pitr/yamls/backupstorage.yaml +++ /dev/null @@ -1,19 +0,0 @@ -apiVersion: storage.kubestash.com/v1alpha1 -kind: BackupStorage -metadata: - name: linode-storage - namespace: demo -spec: - storage: - provider: s3 - s3: - bucket: mehedi-pg-wal-g - endpoint: https://ap-south-1.linodeobjects.com - region: ap-south-1 - prefix: backup - secretName: storage - usagePolicy: - allowedNamespaces: - from: All - default: true - deletionPolicy: WipeOut \ No newline at end of file